gsave newpath 20 20 moveto 20 220 lineto 220 220 linetop 220 20 lineto
closepath 2 setlinewidth gsave .4 setgray fill grestore stroke grestore ∎

11institutetext: Chen Li, Xintong Li, Md Rahaman, Xiaoqi Li and Jian Wu
22institutetext: Microscopic Image and Medical Image Analysis Group, College
of Medicine and Biological Information Engineering, Northeastern University,
110169, Shenyang, PR China
Chen Li 22email<EMAIL_ADDRESS>33institutetext: Xiaoyan Li, Hongzan
Sun, Hong Zhang and Yong Zhang 44institutetext: China Medical University,
110122, Shenyang, China 55institutetext: Yudong Yao 66institutetext:
Department of Electrical and Computer Engineering, Stevens Institute of
Technology, Hoboken, NJ 07030, USA 77institutetext: Marcin Grzegorzek
88institutetext: Institute of Medical Informatics, University of Luebeck,
Luebeck, Germany

# A Comprehensive Review of Computer-aided Whole-slide Image Analysis: from
Datasets to Feature Extraction, Segmentation, Classification and Detection
Approaches

Chen Li  Xintong Li  Md Rahaman  Xiaoyan Li  Hongzan Sun  Hong Zhang  Yong
Zhang  Xiaoqi Li  Jian Wu  Yudong Yao  Marcin Grzegorzek

(Received: date / Accepted: date)

###### Abstract

With the development of computer-aided diagnosis (CAD) and image scanning
technology, _Whole-slide Image_ (WSI) scanners are widely used in the field of
pathological diagnosis. Therefore, WSI analysis has become the key to modern
digital pathology. Since 2004, WSI has been used more and more in CAD. Since
machine vision methods are usually based on semi-automatic or fully automatic
computers, they are highly efficient and labor-saving. The combination of WSI
and CAD technologies for segmentation, classification, and detection helps
histopathologists obtain more stable and quantitative analysis results, save
labor costs and improve diagnosis objectivity. This paper reviews the methods
of WSI analysis based on machine learning. Firstly, the development status of
WSI and CAD methods are introduced. Secondly, we discuss publicly available
WSI datasets and evaluation metrics for segmentation, classification, and
detection tasks. Then, the latest development of machine learning in WSI
segmentation, classification, and detection are reviewed continuously.
Finally, the existing methods are studied, the applicability of the analysis
methods are analyzed, and the application prospects of the analysis methods in
this field are forecasted.

###### Keywords:

Whole-slide image analysis computer-aided diagnosis feature extraction image
segmentation image classification object detection

##  1 Introduction

###  1.1 Brief Knowledge of Whole-slide Imaging Technique

_Whole-slide Image_ (WSI), also generally mention as “virtual microscopy”,
purposes to imitate typical light microscopy in a computer-generated model
Farahani-2015-WSIP . People usually think of whole-slide imaging as an image
acquisition method. It is possible to transform the whole glass slide into a
digital form Janabi-2012-WSIA . Furthermore, the “digital slides” are used for
humans observation or performing them to automated image analysis
Pantanowitz-2011-RTCS .

The processing of whole-slide imaging is performed by the WSI system. A WSI
system has a scanner, networked computer(s), and possibly a server or cloud
solution for storage, display (e.g. tablet, etc.), and compatible software for
image creation, management, and analysis gabry-2014-WSIW  saco-2016-CSWS
pantanowitz-2013-VWSI . The first part applies technical hardware (scanner) to
digitize glass slides, generates a sizable classical digital image (so-called
“digital slide”) accordingly. The second part exploits technical software (ie,
virtual slide viewer) to view and/or analyze these huge digital images
Weinstein-2009-OTVM . WSI devices have different looks and performance, but
overall, the WSI scanner includes the following parts: an optical microscope
system with a camera, an acquisition system, computer hardwares/softwares,
scanning softwares, and a digital slide viewer. Supplemental components
include the feeder or image processing systems Farahani-2015-WSIP .

Figure 1: The workflow of the proposed whole-slide imaging. (1) is the glass
slide Amin-2008-AWSI . (2) whole mount glass slides. (3) whole slide imaging
scanner. (4) digital whole image Farahani-2015-WSIP .

As shown in Fig. 1, (1) is the pathologic biopsy, (2) is the whole mount glass
slides, (3) is whole slide imaging scanner, (4) is the obtained WSIs. The
optical microscope system is the essential part of the WSI scanner, especially
the lens optics and the camera because it can determine the quality of the
images. Charged coupled device (CCD) sensors on cameras that can convert
analog signals into digital signals. There are two major methods of slide
acquisition. One is area scanning, the other is line scanning. The area
scanner moves on the sample block by block and section by section, that is,
after stopping at each position to capture an image, it is repositioned to the
next position. The line scanner is smooth and continuous movement and fast
scanning Higgins-2015-ACDP . After choosing a range of interest on a slide,
adjust focus, and scan the slide Amin-2008-AWSI . If WSI scanners have a
Z-stacking facility (scan slides at different focal planes along the vertical
Z-axis and stack images on top of each other to produce composite multi-planar
images Farahani-2015-WSIP ), they can better center on particular areas of
interest gabry-2014-WSIW . Owing to the images generated by the WSI systems
are large, the visual field of a computer should be bigger than the visual
field of a traditional microscope over four times Rojo-2006-CCCA .

With the accelerating development of science and technology, the WSI system
has progressed rapidly. WSI offers higher quality and resolution images with
annotation Pantanowitz-2011-RTCS . The scanner with fast scanning speed has
improved image quality and reduced storage costs Janabi-2012-WSIA . The
digital approach also can reduce the time of transporting glass slides and the
risk of breakage and fading Ghaznavi-2013-DIPW  Camparo-2012UWSI
Webster-2014-WSIA . Moreover, the digital slides do not deteriorate over time
saco-2016-CSWS .

WSI infuses into many fields such as E-education, virtual workshops, and
pathology aspect. Now, there is a growing need for pathology to improve
quality, patient safety, and diagnostic accuracy. These causes and economic
pressures to consolidate and centralize diagnostic services Ghaznavi-2013-DIPW
. Moreover, WSI can boost distinct pathology practices, so it is generally
used in pathology Cornish-2012-WSIR . Digital pathology networks based on WSI
systems can solve some difficult problems with pathology. For example, WSI can
be explored by several observers from different areas at the same time.
Discussions using WSI can save the time needed for transferring glass slides
to distant places for attaining second minds and teleconsultation
Janabi-2012-WSIA  Webster-2014-WSIA  Janabi-2012-WSIF . WSI equivalently
broadens the scope of cytopathology where virtual slides are used for numerous
intents like telecytology, quality activities (e.g. archiving and proficiency
testing), and education (e.g. virtual atlases) gabry-2014-WSIW . It will also
let pathologists become more efficient, precise, and creative at quantifying
prognostic biomarkers like HER2/neu (c-erbB-2). But also, crucially, WSI
develops CAD in combination with the continually developing computer
artificial intelligence, big data, and cloud technology. Nowadays, WSI
technology is very advance and offers the pathology community novel clinical,
nonclinical, and research image-related applications Farahani-2015-WSIP .

###  1.2 The Development of WSI Analysis

The traditional pathological section analysis method requires specially
trained pathologists to look for areas of interest under the microscope one by
one, and then analyze and diagnose based on professional knowledge.
Traditional manual analysis of pathological images has many drawbacks and
problems. There are no quantitative indicators, so the qualitative analysis
results cannot be reproduced Singh-2010-BCDC . Moreover, most doctors have
tight working conditions, heavy workload, and time pressure. In this case, the
human cognitive process is easily disturbed, leading to incomplete diagnosis
and misdiagnosis Goggin-2007-CDSS . Although traditional slide analysis is
accurate, it can be deeply personal. It is available for the same person to
evaluate a slide one day and to get different conclusions the following week.
Besides, the procedure is a challenging and time-consuming task
Higgins-2015-ACDP . Therefore, CAD is a more efficient, accurate, and
intuitive method.

The computer-aided reading slide can help pathologists improve diagnosis
accuracy and detection rate and reduce the overall misdiagnosis rate.
Moreover, the computer is not affected by fatigue and human error and provides
better assistance to doctors Goggin-2007-CDSS  Fazal-2018-PPFR . It is also a
valuable tool to reduce the workload of clinicians Lisboa-2006-UANN . While
reducing pathologists’ workload and improving efficiency, it can also perform
intuitive quantitative analysis of pathological conditions. These are better
than manual reading slides. Computer-aided viewing of WSI is now rapidly
developing. WSI provides the pathology field unique clinical, nonclinical, and
analysis of image-related applications Farahani-2015-WSIP .

In recent years, the pathological WSI analysis performed by CAD doctors, it
has been widely used in different cancer fields (ie, breast cancer, prostate
cancer, gastric cancer, neuroblastoma). The scope of applications focuses on
disease classification, early screening, tissue localization, and benign and
malignant diagnosis. Common tasks with CAD include classification,
segmentation, and detection.

For example, in the work of Huang-2017-AHGP , automatic detection and
sequencing system based on Gleason pattern recognition is proposed for the
automatic detection of high-grade prostate cancer. In the field of breast
cancer, the work of Mehta-2018-LSBB  makes the segmentation of WSI images of
breast biopsy with biologically significant tissue markers. The study of
Korbar-2017-DLCC  trains a modified version of the residual network (ResNet)
to classify different types of colorectal polyps on WSIs. At present, the
development trend of computer-aided viewing of WSIs is shown in Fig. 2.

Figure 2: The development trend of computer-aided viewing of WSIs. The
horizontal direction shows the time. The vertical direction shows the
cumulative number of related works in each year.

As shown in Fig. 2, as the years are getting closer and closer, technology
continues to advance. There are more and more cases of using computers to
assist diagnosis. The number of cases in the three main applications of
classification, segmentation, and detection has increased year by year. The
number of cases in other applications are growing, such as retrieval
Ma-2016-BHIR , localization Alomari-2009-LTHR . Beginning in 2008, CAD viewing
WSI has helped pathologists begin to realize it significantly. Since 2014,
there has been an increasing trend in the number of computer-aided
pathologists diagnosed with WSI. Gradually by 2020, the growth rate of CAD has
increased, reflecting the vigorous development of this technology.

Besides, to explain and clarify computer-aided pathologists’ work context in
viewing WSI, an organization chart is shown in Fig. 3. The figure shows the
general process of CAD and processing WSI. It shows seven important steps in
the histopathology image analysis system, including data acquisition, image
presentation, image preprocessing, feature extraction, data post-processing,
classifier design, and system evaluation.

Figure 3: The organization chart of histopathology WSI analysis using
computer-aided analysis approaches in this paper.

In Fig. 3, histopathological data initially obtained from the medical field,
2-D or 3-D digital microscopic images are first captured by various imaging
equipment (e.g., optical microscopy), and then saved in a specific color space
(e.g., Red, Green and Blue (RGB) color space). The 3rd step is the image pre-
processing step, the properties of images are improved by dataset
augmentation, segmentation, and so on, which is an importation preparation for
the feature extraction step. In the next step, feature extraction is
implemented, where the image can be represented by its attributes (shape,
texture, and color features), or layout features (global and local features),
or extraction style (manual or automatic). These feature extraction categories
are not separate, but can be converted into other categories using appropriate
methods. After that, the post-processing step takes the responsibility to
enhance the extracted features, where filter, morphological processing,
normalization are always used. Also, the classifier can be classified as
shallow or deep according to its learning structure. Finally, various
numerical and intuitive methods are used to evaluate the classification
system, such as classification accuracy, classification error rate,
sensitivity, and specificity. Besides, each step is not independent, but is
closely connected with other steps through information feedback. Therefore,
the entire CAD viewing WSI system is an organic whole C.Li-2016-CBMI .

###  1.3 Motivation of This Review

Now WSI technology has applications in many fields. For example, to perform
preliminary diagnosis of surgical pathology, and perform intraoperative frozen
section diagnosis through remote consultation huang-2018-TCFS  boyce-2017-AUVW
, and seek expert advice without incurring international transportation costs
or delays Boroujeni-2016-WSIH . WSI also provides advantages in tumor
diagnosis, prognosis, and targeted therapy. It can also facilitate teachers
and students in teaching Ghaznavi-2013-DIPW . Therefore, the research field of
WSI analysis through CAD systems is significant. To the best of our knowledge,
there exist some survey papers that summarize WSI analysis (e.g., the reviews
in Pantanowitz-2011-RTCS , Gurcan-2009-HIAA ; Kothari-2013-PIIQ ;
Veta-2014-BCHI ; Sharma-2015-ARGB ; Li-2018-LSRM ; Komura-2018-MLMH ;
Chang-2019-AIIP ; Nichols-2019-MLAA ; Wang-2019-PIAU ; Dimitriou-2019-DLWS ;
Kumar-2020-WSIP ). In the following part, the summary of survey papers related
to the WSI analysis is presented.

The survey of Pantanowitz-2011-RTCS  reviews the current status of WSI
pathology, including supervision and verification, remote and routine
pathological diagnosis, educational use, implementation issues, and cost-
benefit analysis of WSI in routine clinical practice. However, this article
only focuses on the application of CAD systems in WSI analysis. This review
rarely mentions this, and only 12 references are about WSI.

The survey of Gurcan-2009-HIAA  reviews the latest CAD techniques for digital
histopathology. This article also briefly introduces new image analysis
technologies developed and applied in the United States and Europe for some
specific histopathology-related problems. More than 130 papers on CAD have
been summarized and only three articles are about WSI.

The survey of Kothari-2013-PIIQ  reviews the WSI informatics method of
histopathology, related challenges and future research opportunities. However,
this article reviews image quality control, feature extraction of image
attributes captured at pixels, object, and semantic levels, image features for
predictive modeling, and data and information visualization for diagnostic or
predictive applications. It does not discuss the entire process of CAD and the
viewing of WSI. More than 130 papers have been summarized. However, only three
articles are about WSI.

The survey of Veta-2014-BCHI  reviews the analysis methods of
histopathological images of breast cancer, introduces the process of tissue
preparation, staining, and slide digitization, and then discusses different
image processing techniques and applications, from tissue staining analysis to
CAD, and the prognosis of breast cancer patients. Although the
histopathological images discussed in the article are WSIs, they are only
about breast cancer and not comprehensive. More than 110 papers have been
summarized. However, only four articles are about WSI.

The survey of Sharma-2015-ARGB  provides a comprehensive overview of the
graph-based methods explored so far in digital histopathology. More than 170
papers have been summarized. However, only four articles are about WSI.

The survey of Li-2018-LSRM  reviews the latest methods of large-scale medical
image analysis, which are mainly based on computer vision, machine learning,
and information retrieval. Then, they comprehensively reviewed the algorithms
and technologies related to the main processes in the pipeline, including
feature representation, feature indexing, and search. However, WSI appears
only in the sample dataset, and no actual analysis is performed. Of the more
than 250 papers summarized in this paper, only three mention WSI.

The survey of Komura-2018-MLMH  introduces the application of digital
pathological image analysis using machine learning algorithms, solve some
specific analysis problems, and propose possible solutions. However, there are
only 11 articles related to WSI on the topics we are interested in. More than
120 papers have been summarized. But only 11 articles are about WSI.

The survey of Chang-2019-AIIP  introduces the general situation of artificial
intelligence, a brief history in the medical field and the latest developments
in pathology, and the future prospects of pathology driven by it. This review
only briefly mentions WSI in the part of the pathology application imaging and
example datasets. Of the more than 70 papers summarized in this paper, only
four mention WSI.

The survey of Kumar-2020-WSIP  introduces the technical aspects of WSI, its
application in diagnostic pathology, training and research, and its prospects.
It highlights the benefits, limitations, and challenges of delaying the use of
this technology in daily practice. But this article only focuses on computer-
aided pathologists to view WSI and its application in diagnosis, which are not
discussed in this review. Of the 50 references, 20 are about WSI.

From the existing review papers mentioned above, we can find that many
researchers are concerned about the current status and development trend of
WSI technology itself, and hundreds of related works have been systematically
summarized and discussed in those review papers. However, all these survey
papers use WSI format datasets as examples only, and do not aim to introduce
the detailed introduction of computer-aided pathologists to review WSI
technology. Therefore, we present this review paper to analyze all related
works using CAD combined with WSI in the past few decades. This survey
summarizes more than 210 related works from 2004 to 2020. The audience for
this review is related researchers in the field of medical imaging and medical
professionals.

###  1.4 Structure of This Review

This structure of this paper is as follows: Sec. 2 summarizes the related
datasets and commonly used evaluation methods. Sec. 3 illustrates frequently
used feature extraction methods. Sec. 4, 5, and 6 present the related work of
segmentation, classification, and detection using WSI and CAD technology.
After the overview of different works, the most frequently used approaches are
analyzed in Sec. 7. Finally, Sec. 8 concludes this review with prospective
future direction.

##  2 Datasets and Evaluation Methods

In this section, we have discussed some commonly used datasets and evaluation
metrics for the classification, segmentation, and detection tasks.

###  2.1 Publicly Available Datasets about WSI

To better analyze the CAD using WSI technology, we have summarized some
frequently used publicly available datasets in our study. Tab. 1 shows the
necessary information of these datasets. Two of the most commonly used
datasets are The Cancer Genome Atlas (TCGA) TCGA  and the Camelyon datasets
Litjens-2018-1HSS . Both datasets are often used for classification and
detection. At the same time, we find two WSI datasets named TUPAC16
Veta-2019-PBTP  and Kimia Path24 Babaie-2017-CRDP . TUPAC dataset is widely
used, such as mitosis detection, prediction of breast tumor proliferation,
automatic scoring (classification), and so on. Kimia Path24 is often used for
classification and retrieval Babaie-2017-CRDP  Kumar-2018-DBFR . The basic
information of the common datasets are shown in Table. 1.

Table 1: The basic information of the publicly available used datasets. Databases | Year | Field | Number of images or or size
---|---|---|---
TCGA | 2006 | Cancer related | Over 470 TB
NLST Pathology Images | 2009 | Lung | Around 1250 H&E slides
BreakHis | 2015 | Breast cancer | 9,109 microscopic images
TUPAC16 | 2016 | Tumor mitosis | Around 821 H&E slides
Camelyon | 2017 | Breast cancer | Around 3TB
Kimia Path24 | 2017 | Pathology Images | 24 WSIs

####  2.1.1 TCGA database

TCGA is a project jointly launched by The National Cancer Institute (NCI) and
the National Human Genome Research Institute (NHGRI) in 2006 TCGA . It
contains clinical data, genome variation, mRNA expression, miRNA expression,
methylation and other data of various human cancers (including subtypes of
tumors). The database is designed to use high-throughput genomic analysis
techniques to help people developing a better understanding of cancer and
improve the ability to prevent, diagnose, and treatment Tomczak-2015-CGAI .
While TCGA main work focuses on genomics and clinical data, it also
accumulates a large number of WSIs in patient’s tissue. Since WSI datasets are
much larger than other datasets, to facilitate viewing, David et al.
Gutman-2013-CDSA  proposes an integrated network platform named Cancer Digital
Slide Archive (CDSA) to accommodate all WSI in TCGA. Since the dataset
contains many types of cancers, it has a wide range of uses. Fig. 4 below is
an example of WSIs in an adrenal cortical carcinoma in the TCGA database.

Figure 4: An example of WSI in an adrenal cortical carcinoma in the TCGA
database TCGA .

####  2.1.2 Camelyon Database

The Camelyon Challenge is hosted by International Symposium on Biomedical
Imaging (ISBI) Litjens-2018-1HSS . The whole competition dataset (Camelyon16,
Camelyon17 ) are derived from sentinel lymph nodes of breast cancer patients
contains WSIs of Hematoxylin and Eosin (H&E) stained node sections
Bejnordi-2017-DADL Bandi-2018-DIMC . Therefore, the Camelyon dataset is
suitable for the automatic detection and classification of breast cancer in
WSI. The data of Camelyon16 are from the Radboud University Medical Centre and
the University of Utrecht Medical Centre. The Camelyon16 dataset is composed
of 170 phase I lymph node WSIs (100 normals and 70 metastatics) and 100 Phase
II WSIs (60 normals and 40 metastatics), and the test dataset consisted of 130
WSIs from two universities. The Camelyon16 dataset is used as training values
for the evaluation of Camelyon17. Fig. 5 is a pathological picture of a lymph
node in Camelyon. The left side belongs to normal cell tissue, and the right
cell has been swallowed and occupied by cancer cells.

Figure 5: A pathological image of a lymph node in Camelyon Litjens-2018-1HSS .
The left side belongs to normal cell tissue, and the right cell has been
swallowed and occupied by cancer cells.

####  2.1.3 TUPAC16 Database

The TUPAC16 challenge is held in the context of the MICCAI Veta-2019-PBTP .
TUPAC16 main challenge dataset consists of 821 TCGA WSIs with two types of
tumor proliferation data. 500 for training, 321 for testing. In addition to
the main challenge dataset, there are two secondary datasets (area of interest
and mitotic detection). The area of interest auxiliary dataset contains 148
cases that are randomly selected from the training dataset. The mitotic test
dataset consisted of WSIs of 73 breast cancer cases from three pathological
centers. Of the 73 cases, 23 are AMIDA13 challenge Veta-2015-AAMD . The
remaining 50 cases previously used to assess the interobserver agreement for
mitosis counting are from two other pathology centers in the Netherlands. So
the dataset is mainly used for automatic detection of tumor mitosis or other
regions of interest(ROI). Fig. 6 shows some examples of mitosis maps in H&E
breast cancer slices, with green arrows marking mitosis.

Figure 6: Some examples of mitosis diagrams in H&E breast cancer slices in the
TUPAC16 Veta-2019-PBTP , with green arrows marking mitosis.

####  2.1.4 Kimia Path24 Database

This dataset is consciously and manually selected from 350 WSIs from different
body parts so that the 24 WSIs clearly represented different texture patterns.
So this dataset is more like a computer vision dataset (as opposed to a
pathology dataset) because visual attention is spent on the diversity of
patterns rather than on anatomy and malignancy Babaie-2017-CRDP . Therefore,
this dataset is mainly used for classification and retrieval of
histopathological images. The 24 WSIs thumbnails in this dataset are shown in
Fig. 7.

Figure 7: 24 WSIs thumbnails in Kimia Path24 database Babaie-2017-CRDP .

###  2.2 Evaluation Method

This subsection introduces the evaluation methods of classification,
segmentation, and detection algorithms and related formulas.

####  2.2.1 Basic Evaluation Indexs

The confusion matrix is used to observe the performance of the model in each
category, and the probability of each category can be calculated. The specific
style of the confusion matrix is shown in Tab. 2.

Table 2: Confusion matrix of basic evaluation indexs. Data Class | Classified as Pos | Classified as Neg
---|---|---
Pos | True Positive(TP) | False Negative(FN)
Neg | False positive(FP) | True Negative(TN)

According to the confusion matrix, the True Positive Rate (TPR) can be defined
as TP/(TP + FN), which represents the proportion of the actual positive
instances in all positive instances of the positive class predicted by the
classifier, the False Postive Rate (FPR) can be defined as FP/(FP+TN), which
represents the proportion of the actual negative instances in all negative
instances of the positive class predicted by the classifier. It can be seen
that the mathematical expressions of the following evaluation metrics are
shown in Table. 3 Sokolova-2009-SAPM .

Table 3: Evaluation metrics. _Acc_ , _P_ , _R_ , _Se_ , _Sp_ and _F1_ denote accuracy, precision, recall, sensitivity, specificity and F1 score, respectively. Assessments | Formula | Assessments | Formula
---|---|---|---
_Acc_ | $\frac{TP+TN}{TP+TN+FP+FN}$ | _Se_ | $\frac{TP}{TP+FN}$
_P_ | $\frac{TP}{TP+FP}$ | _Sp_ | $\frac{TN}{TN+FP}$
_R_ | $\frac{TP}{TP+FN}$ | _F1_ | $2\texttimes\frac{P\texttimes R}{P+R}$

####  2.2.2 Evaluation of Segmentation Methods

Image segmentation Haralick-1985-IST  is the segmentation of images with
existing targets and precise boundaries. The commonly used indicators are
accuracy, precision, recall, F-measure, sensitivity, and specificity. These
metrics we have discussed in Sec. 2.2.1 and their mathematical expressions are
given Tab. 2. Dice co-efficient (D) and Jaccard index (J) are popular
segmentation evaluation indexes in recent years. Dice co-efficient (D)
represents the ratio of the area intersected by two individuals to the total
area, that is, the similarity between ground truth and the segmentation result
graph. If the segmentation is perfect, the value is 1. Then, if S stands for
the segmentation result graph and G stands for ground truth, the expression of
Dice co-efficient (D) is given in Eq. (1).

$D(S,G)=\frac{2|A\cap G|}{|A|+|G|}$ (1)

Jaccard Index (J) represents the intersection ratio of two individuals, which
is similar to Dice co-efficient. The formula is given in Eq. (2).

$J(S,G)=\frac{|A\cap G|}{|A\cup G|}$ (2)

####  2.2.3 Evaluation of Classification Methods

Classification Kamavisdar-2013SICA  is the operation of determining the
properties of objects in the image. In the field of digital histopathology we
studied, some are the classification of cancer Petushi-2006-LSCH , some are
the operation of selecting ROI Swiderska-2015-TMMH , and some are the
identification of cancer regions Doyle-2010-BBMC . The purpose of
classification is achieved by the constructed classifier. The performance
indicators used to evaluate these classifiers are critical to the final
results. Accuracy is the most commonly used indicators to evaluate
classifiers. Precision, recall, sensitivity, specificity, and F1 score are
widely used to evaluate classifiers. Accuracy,precision, recall, F-measure,
sensitivity, and specificity we have discussed in Sec. 2.2.1 and their
mathematical expressions are given in Tab. 2. With the continuous improvement
of classification requirements in practical applications, ROC (Receiver
Operating Characteristic), AUC (Area Under ROC Curve), a non-traditional
measurement standard, have emerged. ROC is a curve drawn on a two-dimensional
plane with FPR as the abscissa and TPR as the ordinate. It can reflect the
sensitivity and specificity of the continuous variables as a comprehensive
indicator. It can also solve the problem of class imbalance in the actual
dataset. AUC quantifies the area under the ROC curve into a numerical value to
make the results more intuitive.

####  2.2.4 Evaluation of Detection Methods

Detection Pal-1993-RIST  is another common task in analyzing histopathological
WSIs. Detection is not only to determine the attributes of the region
identified in WSI, but also to identify and obtain more detailed results.
Because of the similarity between testing and classification, most of the
evaluation indexes are the same as the classification, including accuracy,
precision, recall, F-measure, sensitivity, and specificity that we have
discussed in Sec. 2.2.1. However, in WSI detection, it is difficult to locate,
determine and quantify multiple lesions. Therefore, FROC (Free Receiver
Operating Characteristic Curve) Egan-1961-OCSD  is proposed to evaluate the
detection results. FROC curve is a small variation of the ROC curve. It is a
curve drawn on a two-dimensional plane with FP as the horizontal coordinate
and TPR as the vertical coordinate. This allows the detection of multiple
lesion areas on a single WSI.

###  2.3 Summary

According to the review above, we can see that the commonly used public
datasets are TCGA, TUPAC16, and Kimia Path24 for the classification,
segmentation, and detection of histopathological images using the combination
of WSI technology and CAD with the brief introduction. Also, the evaluation
indicators of these three tasks. The basic commonly used evaluation indicators
are accuracy, precision, recall, sensitivity, and specificity. In terms of
classification, there are comprehensive indicators such as AUC. Dice co-
efficient and Jaccard index in segmentation indicators have become popular in
recent years, and FROC in detection indicators can be used for positioning,
qualitative and quantitative analysis of multiple lesions.

##  3 Feature Extraction

Traditional image feature extraction is generally divided into three steps:
preprocessing, feature extraction, and feature processing. Then using machine
learning methods to segment and classify the features. The purpose of
preprocessing is to eliminate interference factors and highlight
characteristic information. The main methods are: image standardization
Shen-2003-PSBM  (adjust the image size); image normalization Khan-2014-NMAS
(adjust the image center of gravity to 0). The main purpose of feature
processing is to eliminate features with a small amount of information and
reduce the amount of calculation. The common feature processing method is
principal components analysis Jhajharia-2016-NNBB .

Among them, feature extraction is a crucial step. Converting input data into a
set of features is called feature extraction Ping-2013-RIFE . The main goal of
feature extraction is to obtain the most relevant information from the
original data and represent the information in a lower-dimensional space
Kumar-2014-DRFE . Therefore, in this section, we mainly summarize the features
extracted in WSI for CAD. The types of extracted features are shown in Fig. 8.

Figure 8: The types of feature extraction methods.

###  3.1 Traditional Feature Extraction

In the process of segmentation, classification, or detection combined with CAD
and WSI technology, the commonly used extracted features include color
features, texture features and shape features.

####  3.1.1 Color Featur Extraction

Color is an important feature, which is widely used for image representation
Kodituwakku-2004-CCFI . The color of an image is invariant to rotation,
translation, or scaling. Color characteristics are defined according to a
specific color space or model Ping-2013-RIFE . Many color spaces are used in
the literature such as RGB Roullier-2010-GMRS , HSV ((Hue Saturation Value))
Mercan-2016-LDRR , and LAB Mercan-2016-MIML . Common color features include
color histograms, color moments, and color coherence vector(CCV)
Pass-1997-CIUC . Among the papers we summarized, 24 papers used color features
Kong-2009-CAEN ; Roullier-2010-GMRS ; Samsi-2012-ECFA ; Kothari-2012-BIMP ;
Akakin-2012-CBMI ; Collins-2013-AMAU ; Nayak-2013-CTHS ; Veta-2013-DMFB ;
Kothari-2013-ETFA ; Homeyer-2013-PQNH ; Hiary-2013-SLWS ; Bautista-2014-CSWS ;
Mercan-2014-LDRR ; Yeh-2014-MSDP ; Litjens-2015-ADPC ; Weingant-2015-EPTC ;
Li-2015-FRID ; Mercan-2016-LDRR ; Barker-2016-ACBT ; Mercan-2016-MIML ;
Brieu-2016-SSMS ; Mercan-2017-MIML ; Cruz-2018-HTAS ; Morkunas-2018-MLBC .

##### RGB-based Color Features

We have found 12 studies that utilized RGB feature extraction technique
Kong-2009-CAEN ; Roullier-2010-GMRS ; Akakin-2012-CBMI ; Collins-2013-AMAU ;
Veta-2013-DMFB ; Homeyer-2013-PQNH ; Hiary-2013-SLWS ; Bautista-2014-CSWS ;
Yeh-2014-MSDP ; Litjens-2015-ADPC ; Cruz-2018-HTAS ; Morkunas-2018-MLBC .

The color features extracted by Kong-2009-CAEN  are combined with the color
and entropy information extracted from the RGB image channel. In
Roullier-2010-GMRS , the vector extracted with the RGB feature indicates that
the feature vector of each pixel is the local entropy of the red-green
difference calculated in the square neighborhood around the pixel. In
Akakin-2012-CBMI , the mean and standard deviations are calculated as first-
order and second-order statistical features from the three RGB channels, and
there are six features. The author in Collins-2013-AMAU  extracts core RGB
features. The color features of Veta-2013-DMFB  are described by the average,
standard deviation, minimum and maximum values of the three color channels in
the RGB color space in the candidate area and two other areas. In Fig. 10 an
example of features distribution image showing the spatial distribution of the
cell nuclear diameter in Yeh-2014-MSDP . The work of Homeyer-2013-PQNH
extracts the PVS function of each R, G, and B channel (8 pixel value
statistics). PVS is composed of the minimum, maximum, sum, average, and
standard deviation of the constituent pixels, and the lower quartile, median
and upper quartile are composed of values in a specific color channel. Fig. 9
shows the RGB feature extraction and classification in Homeyer-2013-PQNH .

Figure 9: An image after RGB feature extraction and classification in
Homeyer-2013-PQNH . This figure corresponds to Fig.2 (2) in the original
paper.

The author in Hiary-2013-SLWS  extracts the variance in each color channel of
RGB: s2R, s2G, s2B, the variance (maximum value) between the peaks of each
color channel s2. The work of Bautista-2014-CSWS  uses color saturation and
RGB color. Yeh-2014-MSDP  extracts color features from the RGB channels. Fig.
10 shows an example of feature distribution of the cell nuclear in diameter
Yeh-2014-MSDP .

Figure 10: An example of feature distribution image in Yeh-2014-MSDP . This
figure corresponds to Fig.10 in the original paper.

The author in Litjens-2015-ADPC  extracts RGB histogram, and overlay features.
Cruz-2018-HTAS  extracts the first-order statistics of 14 color channels, and
the color histogram of each RGB channel is 8×38 bin histogram. In
Morkunas-2018-MLBC , for each 2D superpixel (for example, grayscale
superpixel), two statistics are calculated of mean and standard deviation of
the pixel value. For 3D superpixels (such as RGB superpixels), eight
statistics are calculated the mean and standard deviation of the pixel value
for each color channel and each RGB superpixel, then this as a color function.

##### HSV-based Color Features

There are four papers on color feature extraction based on HSV Samsi-2012-ECFA
; Akakin-2012-CBMI ; Homeyer-2013-PQNH ; Bautista-2014-CSWS .

In Samsi-2012-ECFA , the extracted color feature is the hue channel converted
from the HSV color space of the original image. In Akakin-2012-CBMI , from the
three HSV channels, the average and standard deviation are calculated as
first-order and second-order statistical features, and a total of six features
are extracted. Homeyer-2013-PQNH  extracts the PVS function of each H, S and V
channel. Bautista-2014-CSWS  uses color saturation and value and RGB color as
a function in HSV color space.

##### LAB-based Color Features

There are five papers on color feature extraction based on LAB Kong-2009-CAEN
; Akakin-2012-CBMI ; Mercan-2014-LDRR ; Mercan-2016-LDRR ; Mercan-2017-MIML .

In Kong-2009-CAEN , the extracted color features are composed of color and
entropy information extracted from LAB image channels. In Akakin-2012-CBMI ,
from the three channels of CIELAB, the average and standard deviation are
calculated as first-order and second-order statistical features, and a total
of six features are extracted. Mercan-2014-LDRR  extracted the color histogram
calculated in LAB space. Fig. 11 shows the color histogram mentioned in the
paper. Cutting the WSI into a patch is a normal operation in the image
processing process. (a,b,c,f,g,h) in Fig. 11 are the image blocks after the
WSI slice.

Figure 11: (a) Original $120\times 120$ pixel patch, (b) deconvolved color
channel that shows good contrast for nuclei dyed with haematoxylin, (c)
deconvolved color channel that shows good contrast for eosin dye, (d) LBP
histogram calculated on haematoxylin channel, (e) LBP histogram calculated on
eosin channel,(f, g, h) L, a, and b channels of the image in LAB color
space,(i, j, k) color histograms of L, A and B channels. At the end, LBP
histograms on two channels are concatenated to produce the first set of
features, color histograms on LAB channels are concatenated to produce the
second set of features. These figure are from Mercan-2014-LDRR . This figure
corresponds to Fig.3 in the original paper.

The color feature of Mercan-2016-LDRR  is the LAB histogram of the color.
Mercan-2017-MIML  uses the color histogram calculated for each channel in the
CIE-Lab space as the color feature.

##### Others Color Features

Other papers related to color feature extraction total of six Nayak-2013-CTHS
; Kothari-2013-ETFA ; Hiary-2013-SLWS ; Weingant-2015-EPTC ; Li-2015-FRID ;
Barker-2016-ACBT ; Brieu-2016-SSMS .

Nayak-2013-CTHS  and Kothari-2013-ETFA  extract the color information in WSIs
as features. Hiary-2013-SLWS  extracts the average value of the variance,
saturation, brightness of HSI and the hue value of the color model $\mu$.
Weingant-2015-EPTC  extracts the color channel histogram as the color feature.
Li-2015-FRID  extracts the histogram of the three-channel HSD color model as
color features. Brieu-2016-SSMS  extracts the color information in WSI as
features. Barker-2016-ACBT  extracts rough color features. The rough feature
is the use of feature analysis of the diversity of rough areas in WSI to
roughly characterize their shape, color, and texture. The fine feature refers
to the more comprehensive image features extracted from the slice to express
deeper features. The feature histogram extracts in Barker-2016-ACBT  is shown
in Fig. 12.

Figure 12: Histogram distribution of features used in Elastic Net models,
showing the number of models in which features of a given class appear. The
number of features is normalized based on the total number of features
represented for each class. Bar patterns represent the feature class. Most
features for each class appear in all, or nearly all models, as expected if
they have diagnostic value. This figure is from Barker-2016-ACBT . This figure
corresponds to Fig.7 in the original paper.

From the above review, we can see that in terms of color feature extraction,
the RGB color features are used most frequently, focusing on the period from
2009 to 2018. Then HSV color feature and LAB (CIE-LAB) color feature. The
dates are 2012 to 2014 and 2009 to 2017. In other papers, HSI and HSD color
models are used as color characteristics.

####  3.1.2 Texture Featur Extraction

The texture feature describes the surface properties of the object
corresponding to the image or image area. Unlike the color feature, the
texture feature is not based on the feature of pixels. It needs to be
calculated in the area containing multiple pixels. Texture feature is an
effective method when judging images with large differences in thickness and
density. However, when the thickness, density, feature are easy to distinguish
between the information. However, difficult for the usual texture features to
accurately reflect the differences between the textures with different human
visual perception.

Commonly used texture information description methods are: statistical methods
( such as gray-level co-occurrence matrix (GLCM) Mohanaiah-2013-ITFE ),
geometric methods (such as voronoi checkerboard feature method
Tuceryan-1990-TSUV ), model methods (such as random fields Chellappa-1985-CTUG
), and signal processing methods (such as wavelet transform Pichler-1996-CTFE
). Among the papers we summarized, 57 papers used texture features
Diamond-2004-UMCT ; Petushi-2006-LSCH ; Sertel-2008-CAPN ; Sertel-2009-CCCS ;
Kong-2009-CAEN ; Sertel-2009-CPNW ; Roullier-2010-MEBC ; Doyle-2010-BBMC ;
Difranco-2011-ESWS ; Kong-2011-IMAG ; Roullier-2011-MRGA ; Grunkin-2011-PCIA ;
Nguyen-2011-PCDF ; Akakin-2012-CBMI ; Sharma-2012-DSHI ; Nayak-2013-CTHS ;
Jiao-2013-CCDU ; Veta-2013-DMFB ; Kothari-2013-ETFA ; Kong-2013-MBMA ;
Homeyer-2013-PQNH ; Hiary-2013-SLWS ; Mercan-2014-LDRR ; Apou-2014-FSTC ;
Bejnordi-2015-MSSC ; Sharma-2015-ABND ; Swiderska-2015-CMSA ;
Weingant-2015-EPTC ; Li-2015-FRID ; Zhang-2015-FGHI ; Cooper-2015-NGPA ;
Apou-2015-SWSI ; Peikari-2015-TDRR ; Mercan-2016-LDRR ; Barker-2016-ACBT ;
Bejnordi-2016-ADDW ; Zhao-2016-AGEW ; Harder-2016-CFCG ; Shirinifard-2016-DPAU
; Gadermayr-2016-DACC ; Leo-2016-ESHF ; Mercan-2016-MIML ; Brieu-2016-SSMS ;
Saltz-2017-CSSG ; Mercan-2017-MIML ; Hu-2017-DLBC ; Bejnordi-2017-DADL ;
Valkonen-2017-MDWS ; Mirschl-2018-DLCI ; Xu-2018-AACM ; Yoshida-2018-AHCWG ;
Han-2018-ACDL ; Morkunas-2018-MLBC ; Simon-2018-MRLF ; Klimov-2019-WSIB .

##### Local Binary Pattern-based Texture Features

There are 14 papers on texture feature extraction based on local binary
pattern (LBP) Sertel-2008-CAPN ; Sertel-2009-CPNW ; Roullier-2010-MEBC ;
Roullier-2011-MRGA ; Homeyer-2013-PQNH ; Mercan-2014-LDRR ; Bejnordi-2015-MSSC
; Mercan-2016-LDRR ; Mercan-2016-MIML ; Gadermayr-2016-DACC ; Babaie-2017-CRDP
; Mercan-2017-MIML ; Bejnordi-2017-DADL ; Simon-2018-MRLF . In
Sertel-2008-CAPN ; Sertel-2009-CPNW ; Roullier-2010-MEBC ; Roullier-2011-MRGA
; Homeyer-2013-PQNH ; Bejnordi-2015-MSSC ; Mercan-2016-LDRR ; Mercan-2016-MIML
; Babaie-2017-CRDP ; Mercan-2017-MIML , texture histograms of LBP features are
extracted as texture features. Fig. 13 shows the calculation of the LBP
feature of a given pixel in Sertel-2009-CPNW .

Figure 13:  (a) The conventional LBP operator; (b) Circular pattern used to
compute rotation invariant uniform patterns. This figure corresponds to Fig.5
in Sertel-2009-CPNW .

Texture features compute using LBP for small image patches are extracted from
Mercan-2014-LDRR . In Gadermayr-2016-DACC , multi-resolution LBP is extracted
as the texture feature. In Bejnordi-2017-DADL , different structural texture
features are extracted, including LBP features. In Simon-2018-MRLF , LBP, MRC
LBP feature are extracted as texture features.

##### Haralick-based Texture Features

Five papers involve texture feature extraction based on Haralick
Diamond-2004-UMCT ; Sertel-2008-CAPN ; Kong-2009-CAEN ; Leo-2016-ESHF ;
Hu-2017-DLBC . In Diamond-2004-UMCT , 4 Haralick features are the most
suitable for discriminating stroma from PCA. Haralick features are extracted
as texture features in Sertel-2008-CAPN ; Kong-2009-CAEN ; Leo-2016-ESHF . In
Hu-2017-DLBC , extracts 57 subcellular location features, including Haralick
texture features and DNA overlapping the features (experiments). The
experimental images in Leo-2016-ESHF  and the extracted haralick features are
shown in Fig. 14.

Figure 14:  (a, b) is the experimental image, (c, d) is the Haralick intensity
texture. This figure corresponds to Fig.2 in Leo-2016-ESHF .

##### GLCM-based Texture Features

Eight papers involve texture feature extraction based on GLCM Sertel-2009-CCCS
; Doyle-2010-BBMC ; Difranco-2011-ESWS ; Jiao-2013-CCDU ; Hiary-2013-SLWS ;
Weingant-2015-EPTC ; Bejnordi-2016-ADDW ; Morkunas-2018-MLBC .

In Sertel-2009-CCCS , the extraction of the mean and variance of the range of
values within the local neighborhoods and entropy and homogeneity of co-
occurrence histograms as texture features. Co-occurrence features are
extracted as texture features from Doyle-2010-BBMC  and Difranco-2011-ESWS .
16 features such as mean and variance are combined to form a group of 18
features Jiao-2013-CCDU . In Hiary-2013-SLWS , co-occurrence matrix,
correlation, and energy are extracted as texture features. In
Weingant-2015-EPTC , GLCM features are extracted. In Bejnordi-2016-ADDW , co-
occurrence matrix statistics are extracted for each hyperpixel. In
Morkunas-2018-MLBC , 2D hyper-pixel texture is obtained by using the spatial
grayscale symbiosis matrix and 1px displacement vector of 3D hyper-pixel. From
the co-occurrence matrix, the second moment of angle, contrast, correlation,
sum of squares, deficit moment, average, sum variance, sum entropy, entropy,
difference variance, difference entropy, and correlation information measures
1 and 2 are calculated. The average value of the 13 parameters obtained is the
characteristic descriptor.

##### Filter and Scale-invariant Feature Transform(SIFT)-based Texture
Features

There are five related papers on filter and SIFT based texture feature
extraction Nguyen-2011-PCDF ; Peikari-2015-TDRR ; Bejnordi-2016-ADDW ;
Bejnordi-2017-DADL ; Valkonen-2017-MDWS .

In Nguyen-2011-PCDF , first-order statistics, second-order statistics, and
gabor filter features are used as texture features. In Peikari-2015-TDRR , a
Gauss-like texture filter is applied to extract texture features. Fig. 15
shows the uniform distribution of histogram filter response in
Peikari-2015-TDRR .

Figure 15:  Patches of three different tissue types (in RGB) corresponding to
(a) fat, (b) stroma, and (c) epithelial cells -top- along with their
normalized luminance channel images (after converting them from RGB to LAB)
-middle-, and maximal filter responses after convolving Gaussian-like filters
at all directions of onescale (($\sigma$x, $\sigma$y) = (1,3)) -bottom-, and
plot showing their histogram of filter response magnitudes. This figure
corresponds to Fig.3 in Peikari-2015-TDRR .

In Bejnordi-2016-ADDW , the gray histogram statistics extracted from the
filter bank response for each hyperpixel. Different structural texture
features, such as SIFT features, are extracted from Bejnordi-2017-DADL . The
vlfeat implementation of MSER (Maximally Stable Extremal Regions) and SIFT is
used by extracting from Valkonen-2017-MDWS .

##### Others Texture Features

There are a total of 24 related papers based on the extraction of other
texture features Petushi-2006-LSCH ; Kong-2011-IMAG ; Grunkin-2011-PCIA ;
Akakin-2012-CBMI ; Sharma-2012-DSHI ; Nayak-2013-CTHS ; Kothari-2013-ETFA ;
Kong-2013-MBMA ; Apou-2014-FSTC ; Swiderska-2015-CMSA ; Li-2015-FRID ;
Zhang-2015-FGHI ; Cooper-2015-NGPA ; Apou-2015-SWSI ; Barker-2016-ACBT ;
Shirinifard-2016-DPAU ; Gadermayr-2016-DACC ; Brieu-2016-SSMS ;
Mirschl-2018-DLCI ; Xu-2018-AACM ; Yoshida-2018-AHCWG ; Han-2018-ACDL ;
Klimov-2019-WSIB . Texture Parameters: DNM1, DNM2,DNM3, DNM1-2, DNM1-3,
DNM2-3, DNM2-2-3, DT,DN are extracted from Petushi-2006-LSCH . The feature
vector extracted in Petushi-2006-LSCH  is shown in Fig. 16.

Figure 16:  Tissue microtextures identified using image processing. The first
three columns show examples of cell nuclei belonging to nucleus morphology
categories NM1, NM2, and NM3, respectively. The textures identified as ECM and
AT represent, respectively the collagen-rich stroma, and fat and tissue-devoid
regions of the slide. This figure corresponds to Fig.2 in Petushi-2006-LSCH .

The texture is applied to the cytoplasmic region around the nucleus in
Kong-2011-IMAG . In Grunkin-2011-PCIA , texture features are used to identify
areas with high or low intensity variability in the image. Average, standard
deviation, contrast, correlation, energy, entropy, and uniformity are
extracted from Akakin-2012-CBMI  as texture features. Texton-based texture is
extracted from Sharma-2012-DSHI . In Nayak-2013-CTHS , texture information is
used as feature. In Kothari-2013-ETFA , quantitative image features are
extracted to capture its texture. Nuclear texture features are extracted from
the chromatin content and distribution in Kong-2013-MBMA . Each area is tagged
according to its texture description in Apou-2014-FSTC . Intensity on the
basis of histograms of the sum and difference images are extracted as texture
features in Swiderska-2015-CMSA . In Li-2015-FRID , texture features are
extracted for classification. In Zhang-2015-FGHI , the texture feature is
extracted from the cell image and compressed into a binary code. These
compressed features are stored in a hash table that allows constant time
access across many images. In Cooper-2015-NGPA , the texture features of each
nucleus are extracted. In Apou-2015-SWSI , the area is rendered using a
manually positioned texture unit. The Fig. 17 shows the procedural structure
and texture rendered in Apou-2015-SWSI .

Figure 17:  Procedural structure and texture: (a) Procedural 3D model of a
lobular epithelial layer. (b) Slice of (a) showing 2D lobular patterns. (c)
Procedural cell rendering (random layout, arbitrary colors). This figure
corresponds to Fig.5 in Apou-2015-SWSI .

Barker-2016-ACBT , use the riesz texture features. The texture features of
slides stained with Ki67 are extracted from Shirinifard-2016-DPAU . The author
of Gadermayr-2016-DACC  extracts Histograms of Oriented Gradients (HOG), and
Fisher Vectors (FV) as texture features. The kernel texture features are
extracted in Brieu-2016-SSMS  and Saltz-2017-CSSG . In Mirschl-2018-DLCI ,
texture features are extracted from the ROI. In Xu-2018-AACM , the texture of
the subregion is extracted for statistical analysis and classification. In
Yoshida-2018-AHCWG , the standard deviation (variance) within the area defined
by the contour is used as the texture feature. Han-2018-ACDL , extract the
first-and second-order texture features. A total of 166 texture features are
extracted from the convolved hematoxylin (nuclear staining) channel in
Klimov-2019-WSIB .

####  3.1.3 Shape Featur Extraction

The shape feature is just what the name suggests. Under normal circumstances,
there are two ways to represent shape features. One is contour features, and
the other is regional features. The contour feature of the image is mainly for
the outer boundary of the image, and the regional feature of the image is
related to the entire shape area Zhang-2004-RSRD . Commonly used shape feature
extraction methods include boundary feature method (such as hough transform
method Philip-1994-FHSF ), geometric parameter method (such as moment, area,
circumference Huang-1997-OLSV ), fourier descriptors Zhang-2012-MFDS , and
other methods. There are 15 papers that use shape features among the papers we
have summarized Diamond-2004-UMCT ; Kong-2011-IMAG ; Grunkin-2011-PCIA ;
Lu-2012-ASAE ; Kothari-2012-BIMP ; Veta-2012-PVAE ; Lopez-2013-ABDM ;
Collins-2013-AMAU ; Veta-2013-DMFB ; Kothari-2013-ETFA ; Kong-2013-MBMA ;
Cooper-2015-NGPA ; Barker-2016-ACBT ; Saltz-2017-CSSG ; Xu-2018-AACM .

##### Basic Geometric Parameter-based Shape Feature

Among the papers that used shape feature extraction, seven papers extracted
basic geometric shape features Lu-2012-ASAE ; Kothari-2012-BIMP ;
Veta-2012-PVAE ; Collins-2013-AMAU ; Veta-2013-DMFB ; Kong-2013-MBMA ;
Grunkin-2011-PCIA . The feature extraction steps in six of the papers are all
used to classify, segment, or detect the task before it is used to better
represent the image. In Lu-2012-ASAE , the major axis length to minor axis
length ratio of a best-fit ellipse is extracted as the shape feature to
eliminate false regions. In Kothari-2012-BIMP , eosinophilic-object shape
features (pixel area, elliptical area, major-minor axes lengths, eccentricity,
boundary fractal, bending energy, convex hull area, solidity, perimeter, and
count) are extracted. The author in Veta-2012-PVAE  extracts two morphometric
features, the mean nuclear area and standard deviation of the nuclear area,
using a fully automatic segmentation method on WSIs. The author in
Collins-2013-AMAU  extracts basic morphologic features and calculates its odds
ratio for malignant tumors. The author in Veta-2013-DMFB  extracts
compactness, eccentricity, firmness, and sphericity as shape features. The
author in Kong-2013-MBMA  extractes perimeter, eccentricity, circularity,
major axis length, minor axis length as geometric shape feature. Fig. 18 shows
the morphological characteristic spectrum of the image in Kong-2013-MBMA . The
seventh paper Grunkin-2011-PCIA  is the morphological feature extraction for
post-processing.

Figure 18:  Image analysis enables the reliable and objective characterization
of tissues through a process of image normalization, image segmentation, and
feature extraction.

##### Other Shape Features

In the early days, some existing library functions and third-party existing
functions were usually used to directly extract features. In Diamond-2004-UMCT
; Kong-2011-IMAG ; Saltz-2017-CSSG ; Xu-2018-AACM , the nuclear morphological
features of the nuclei in WSI are extracted as the shape features of the
images.

Over time, many other advanced extraction methods have emerged. There are also
four papers on other shape feature extraction Lopez-2013-ABDM ;
Kothari-2013-ETFA ; Cooper-2015-NGPA ; Barker-2016-ACBT . Lopez-2013-ABDM
extractes multiple sharpness features. In Kothari-2013-ETFA , the author
extractes 461 quantitative image features capturing the texture, color, shape,
and topological properties of a histopathological image. Cooper-2015-NGPA
extracts precise quantitative morphometric features. There are shape features
in the core feature group extracted by Barker-2016-ACBT .

###  3.2 Deep Learning Feature Extraction

Convolutional Neural Network (CNN) is widely used to extract the deep learning
features in various WSI analysis tasks. In the papers, a total of 53 papers
used CNN for deep learning feature extraction Puerto-2016-DPAW ;
Sharma-2016-DCNN ; Wang-2016-DLIM ; Geccer-2016-DCBC ;
Sirinukunwattana-2016-LSDL ; Hou-2016-PCNN ; Sheikhzadeh-2016-ALMB ;
Cruz-2017-ARIB ; Wollmann-2017-AGBC ; Araujo-2017-CBCH ; Bejnordi-2017-CASC ;
Sharma-2017-DCNN ; Korbar-2017-DLCC ; Jimenez-2017-DMCR ; Xu-2017-LSTH ;
Korbar-2017-LUHD ; Ghosh-2017-SLCA ; Das-2017-CHWS ; Cui-2018-DLAO ;
Zanjani-2018-CDHW ; Courtiol-2018-CDLH ; Kumar-2018-DBFR ; Bychkov-2018-DLTA ;
Gecer-2018-DCCW ; Ren-2018-DPCP ; Tellez-2018-GWSI ;
Sirinukunwattana-2018-IWSS ; Kwok-2018-MCBC ; Das-2018-MILD ; Lin-2018-SFDS ;
Campanella-2018-TSDM ; Jiang-2018-TMDD ; Wang-2018-WSLW ; Shou-2018-WSIC ;
Tellez-2018-WSMD ; Bertram-2019-LSDM ; Li-2019-AMRM ; Liu-2019-AIBC ;
Campanella-2019-CGCP ; Yue-2019-CCOP ; Maksoud-2019-CCOR ; Bilaloglu-2019-EPCW
; Lin-2019-FSFD ; Yu-2019-LSGC ; Sanghvi-2019-PAIA ; Wang-2019-RRMI ;
Kohlberger-2019-WSIF ; Xu-2019-MTPW ; Chen-2020-ITWS ; Sornapudi-2020-CWSH ;
Pantanowitz-2020-AIAP .

The basic configuration of CNN is the convolutional layer, pooling layer and
fully connected layer Rahaman-2020-ICSC . These three layers can be stacked.
Take the input of the previous layer as the output of the next layer, and
finally get $N$ feature maps Rawat-2017-DCNN  with very low dimensions.
Because it is an end-to-end learning model, it can learn more fully and
extract features better Zhiqiang-2017-ROBC . The convolutional layer acts as a
feature extractor, and the neurons in the convolutional layer are arranged
into feature maps. Since different feature maps in the same convolution have
different weights, $N$ features can be extracted at each position
Lecun-1998-GLAD  Lecun-2015-DL .

##### Deep Learning Features of the VGG Series

In CNN, several classical improved network structures are often applied to
extract deep features on WSI. VGGNet is an improvement based on the original
framework of Krizhevsky-2017-ICDC . The full name of VGG is Visual Geometry
Group, which belongs to the Department of Science and Engineering of Oxford
University. It can be applied to face recognition, image classification, etc.
VGGNet increases the network depth by adding more convolutional layers and
fixing other parameters of the network framework Simonyan-2014-VDCN . All
layers use $3\times 3$ convolution filters that there are fewer parameters and
lower cost. Among the papers we have summarized, the papers that use VGGNet to
extract deep learning features are Kumar-2018-DBFR ; Bychkov-2018-DLTA ;
Campanella-2018-TSDM ; Wang-2018-WSLW ; Li-2019-AMRM ; Yue-2019-CCOP ;
Lin-2019-FSFD . The process of VGG extracting features in Bychkov-2018-DLTA
is shown in Fig. 19.

Figure 19:  The VGG-16 network produces a high-dimensional feature vector for
each individual tile from an input image. This figure corresponds to Fig.1 in
Bychkov-2018-DLTA .

##### Deep Learning Features of the ResNet Series

ResNet is another widely used CNN structure. The full name of ResNet is
Residual Network and the proponent is Balduzzi D. ResNet has pushed deep
learning to a new level, reducing the error rate to a level lower than that of
humans for the first time. The residual module in ResNet makes the network
deeper, but with lower complexity. It also makes the network easier to
optimize and solves the problem of vanishing gradient He-2016-DRLI . The
bottlen neck layer in ResNet uses 1$\texttimes$1 networks, which expands the
dimension of the featuremap and greatly reduces the amount of calculation
He-2016-IMDR . Among the papers we reviewed, the papers that use ResNet to
extract deep learning features include Korbar-2017-DLCC ; Korbar-2017-LUHD ;
Kwok-2018-MCBC ; Campanella-2018-TSDM ; Jiang-2018-TMDD ; Shou-2018-WSIC ;
Bertram-2019-LSDM ; Xu-2019-MTPW .

##### Deep Learning Features of the U-net Series

The full name of U-net is Unity Networking. it is a network architecture
established to solve the problem of medical image segmentation. This structure
is based on FCN (Fully Convolutional Neural Network). It adds an upsampling
stage and adds many feature channels, allowing more original image texture
information in high-resolution layers, using valid for convolution throughout,
ensuring that the results obtained are based on no missing context features
Ronneberger-2015-UCNB . In the paper we summarized, the use of U-net for deep
learning feature extraction have Bandi-2017-CDMT ; Seth-2019-ALBD ;
Seth-2019-ASDW ; Feng-2020-DLMA .

##### Other Deep Learning Feature

There are other improved structures based on CNN, such as GoogLeNet
Wang-2016-DLIM . The Google Academic team carefully prepares GoogLeNet to
participate in the ILSVRC 2014 competition. The main idea is to approximate
the optimal sparse structure by building a dense block structure to improve
performance without increasing the amount of calculation. The initial version
of GoogLeNet appeared in Szegedy-2015-GDC .

There are also other improved structures based on Recurrent Neural Networks
(RNN) Maksoud-2019-CCOR  and LSTM Ren-2018-DPCP , which are used to extract
deep learning features. RNN appeared in the 1980s, and its prototype has seen
in the Hopfield neural network model proposed by American physicists in 1982
Hopfield-1982-NNPS . RNN has a strong processing ability for variable length
sequence data. Therefore, it is very effective for data with time-series
characteristics and can mine time series information and language information
in the data. The Long Short-Term Memory (LSTM) model appeared because of the
drawbacks of RNN, and LSTM can solve the problem of gradient disappearance in
RNN Hochreiter-1997-LSTM .

###  3.3 Summary

It can be seen from the content we reviewed above, in the traditional feature
extraction, color feature, texture feature, and shape feature are the three
most commonly used features. The texture feature is the most used. In the
papers we summarized, from 2004 to 2019, a total of 51 papers used texture
features. The second is color features, which are generally based on the three
color spaces of RGB, HSV, and LAB. Among them, the RGB color space is the most
commonly used. The least applied is the shape feature. For more details and
analysis in this regard, see the detailed introduction in the following
chapters.

Over times, the level of science and technology has also continuously
improved. As can be seen from the papers we summarized, since 2016, deep
learning features have been gradually applied to this day. The specific deep
learning network architecture will be introduced in a separate method analysis
later. Table. LABEL:CDFE is a summary of the CAD methods used for feature
extraction in WSI.

Table 4: Summary of the CAD methods used for feature extraction in WSI (Traditional (T)). Feature Type | Method | Reference | Year | Team | Details
---|---|---|---|---|---
T | Color |  Kong-2009-CAEN  | 2009 | J Kong, O Sertel |  | Combined Color and entropy
---
information extracted from RGB
and LAB image channels
T | Color |  Roullier-2010-GMRS  | 2010 | V. Roullier | RGB feature vector
T | Color |  Samsi-2012-ECFA  | 2012 | Siddharth Samsi | Hue channel for HSV color space conversion
T | Color |  Kothari-2012-BIMP  | 2012 | Sonal Kothari |  | RGB, CIELAB, HSV, mean and standard
---
deviation in each channel in three spaces
T | Color |  Akakin-2012-CBMI  | 2012 | Hatice Cinar Akakin | Nuclear RGB features
T | Color |  Collins-2013-AMAU  | 2013 | Brian T. Collins | color information
T | Color |  Nayak-2013-CTHS  | 2013 | Nandita Nayak |  | The average, standard deviation,
---
minimum and maximum values of
each color channel
in the RGB space
T | Color |  Veta-2013-DMFB  | 2013 | M. Veta | $\backslash$
T | Color |  Kothari-2013-ETFA  | 2013 | Kothari S | PVS for each R, G, B and H, S, V channel
T | Color |  Homeyer-2013-PQNH  | 2013 | André Homeyer |  Variance of RGB, $/mu$ hue of HIS
T | Color |  Hiary-2013-SLWS  | 2013 | Hazem Hiary |  | RGB color and color saturation
---
and value in HSV space
T | Color |  Bautista-2014-CSWS  | 2014 | Pinky A. Bautista |  | RGB color and color saturation
---
and value in HSV space RGB
T | Color |  Mercan-2014-LDRR  | 2014 | Ezgi Mercan | Lab space color histogram
T | Color |  Yeh-2014-MSDP  | 2014 | Fang‑Cheng Yeh | RGB color channels
T | Color |  Litjens-2015-ADPC  | 2015 | Litjens, G | RGB histogram features
T | Color |  Weingant-2015-EPTC  | 2015 | Michaela Weingant | color channel histogram
T | Color |  Li-2015-FRID  | 2015 | Ruoyu Li |  | Histogram of the three-channel
---
HSD color model
T | Color |  Mercan-2016-LDRR  | 2016 | Mercan E | LAB histograms for color
T | Color |  Barker-2016-ACBT  | 2016 | Barker J | $\backslash$
T | Color |  Mercan-2016-MIML  | 2016 | Mercan, C | LAB histograms for Color
T | Color |  Brieu-2016-SSMS  | 2016 | Brieu N, Pauly O | $\backslash$
T | Color |  Mercan-2017-MIML  | 2017 | Caner Mercan | color histogram of each channel in CIE-LAB space
T | Color |  Cruz-2018-HTAS  | 2018 | Angel Cruz-Roa |  | color Histograms 8 × 3 8-bin histogram
---
for each RGB channel
T | Color |  Morkunas-2018-MLBC  | 2018 | Morkūnas M |  | The average and standard deviation
---
of the pixel values of each
color channel and its RGB superpixels.
T | Texture |  Diamond-2004-UMCT  | 2004 | JamesDiamondPhDa | $\backslash$
T | Texture |  Petushi-2006-LSCH  | 2006 | Sokol Petushi |  | 9 texture parameters: DNM1, DNM2, DNM3,
---
DNM1-2, DNM1-3, DNM2-3, DNM1-2-3, DT, DN
T | Texture |  Sertel-2008-CAPN  | 2008 | Olcay Sertel | LBP features,Haralick features
T | Texture |  Sertel-2009-CCCS  | 2009 | O Sertel | $\backslash$
T | Texture |  Kong-2009-CAEN  | 2009 | J Kong, O Sertel | Four textural Haralick features
T | Texture |  Sertel-2009-CPNW  | 2009 | O Sertel, J Kong | LBP features
T | Texture |  Roullier-2010-MEBC  | 2010 | Vincent Roullier | LBP histogram
T | Texture |  Difranco-2011-ESWS  | 2011 | Matthew D. DiFranco |  | Feature maps(the mean and standard from HS),
---
Co-occurrence texture features
T | Texture |  Kong-2011-IMAG  | 2011 | Jun Kong | Texture and gradient features
T | Texture |  Roullier-2011-MRGA  | 2011 | Vincent Roullier | LBP
T | Texture |  Grunkin-2011-PCIA  | 2011 | Michael Grunkin | $\backslash$
T | Texture |  Nguyen-2011-PCDF  | 2011 | Kien Nguyen | Gabor filter features
T | Texture |  Doyle-2010-BBMC  | 2012 | Scott Doyle | Co-occurrence Features
T | Texture |  Akakin-2012-CBMI  | 2012 | Hatice Cinar Akakin |  | Mean, standard deviation, contrast,
---
correlation, energy,
entropy and uniformity
T | Texture |  Sharma-2012-DSHI  | 2012 | Harshita Sharma | Texton-based texture
T | Texture |  Nayak-2013-CTHS  | 2013 | Nandita Nayak | $\backslash$
T | Texture |  Jiao-2013-CCDU  | 2013 | Liping Jiao |  | Combine mean, variance
---
and other 17 features which
are extracted by GLCM method
T | Texture |  Veta-2013-DMFB  | 2013 | M. Veta | $\backslash$
T | Texture |  Kothari-2013-ETFA  | 2013 | Kothari S | $\backslash$
T | Texture |  Kong-2013-MBMA  | 2013 | Jun Kong | $\backslash$
T | Texture |  Homeyer-2013-PQNH  | 2013 | André Homeyer | $\backslash$
T | Texture |  Hiary-2013-SLWS  | 2013 | Hiary H, Alomari R S | $\backslash$
T | Texture |  Apou-2014-FSTC  | 2014 | Apou G, Naegel B | $\backslash$
T | Texture |  Mercan-2016-LDRR  | 2014 | Ezgi Mercan | LBP
T | Texture |  Bejnordi-2015-MSSC  | 2015 | Bejnordi, B. E | LBP
T | Texture |  Sharma-2015-ABND  | 2015 | Harshita Sharma | GLCM Features
T | Texture |  Swiderska-2015-CMSA  | 2015 | Zaneta Swiderska | $\backslash$
T | Texture |  Weingant-2015-EPTC  | 2015 | Michaela Weingant | GLCM features
T | Texture |  Li-2015-FRID  | 2015 | Ruoyu Li | $\backslash$
T | Texture |  Zhang-2015-FGHI  | 2015 | Xiaofan Zhang | $\backslash$
T | Texture |  Cooper-2015-NGPA  | 2015 | Lee AD Cooper | $\backslash$
T | Texture |  Apou-2015-SWSI  | 2015 | Gregory Apou | $\backslash$
T | Texture |  Peikari-2015-TDRR  | 2015 | Peikari M | $\backslash$
T | Texture |  Mercan-2016-LDRR  | 2016 | Mercan E | LBP
T | Texture |  Barker-2016-ACBT  | 2016 | Barker J | $\backslash$
T | Texture |  Bejnordi-2016-ADDW  | 2016 | Bejnordi B E | $\backslash$
T | Texture |  Zhao-2016-AGEW  | 2016 | Zhao Y | GLCM
T | Texture |  Harder-2016-CFCG  | 2016 | Harder N | Co-occurrence Feature
T | Texture |  Shirinifard-2016-DPAU  | 2016 | Shirinifard A | $\backslash$
T | Texture |  Gadermayr-2016-DACC  | 2016 | Gadermayr M | HOG,LBP,FV
T | Texture |  Leo-2016-ESHF  | 2016 | Leo P, Lee G | Haralick features
T | Texture |  Mercan-2016-MIML  | 2016 | Mercan, C | LBP
T | Texture |  Brieu-2016-SSMS  | 2016 | Brieu N, Pauly O | $\backslash$
T | Texture |  Saltz-2017-CSSG  | 2017 | Saltz, J | $\backslash$
T | Texture |  Babaie-2017-CRDP  | 2017 | Babaie M | LBP
T | Texture |  Hu-2017-DLBC  | 2017 | Hu J X | Haralick texture
T | Texture |  Bejnordi-2017-DADL  | 2017 | Bejnordi B E | SIFT,LBP,GLCM
T | Texture |  Valkonen-2017-MDWS  | 2017 | Valkonen M | VLFeat implementation of MSER and SIFT
T | Texture |  Mirschl-2018-DLCI  | 2018 | Jeffrey J. Nirschl | $\backslash$
T | Texture |  Xu-2018-AACM  | 2018 | Hongming Xu | $\backslash$
T | Texture |  Yoshida-2018-AHCWG  | 2018 | Hiroshi Yoshida | $\backslash$
T | Texture |  Han-2018-ACDL  | 2018 | W. Han | $\backslash$
T | Texture |  Morkunas-2018-MLBC  | 2018 | Morkūnas M | $\backslash$
T | Texture |  Mercan-2017-MIML  | 2018 | Caner Mercan | $\backslash$
T | Texture |  Simon-2018-MRLF  | 2018 | Olivier Simon | LBP,mrcLBP feature
T | Texture |  Klimov-2019-WSIB  | 2019 | S Klimov | $\backslash$
T | Shape |  Diamond-2004-UMCT  | 2004 | JamesDiamondPhDa | $\backslash$
T | Shape |  Kong-2011-IMAG  | 2011 | Jun Kong | $\backslash$
T | Shape |  Grunkin-2011-PCIA  | 2011 | Michael Grunkin | $\backslash$
T | Shape |  Lu-2012-ASAE  | 2012 | Cheng Lu | $\backslash$
T | Shape |  Kothari-2012-BIMP  | 2012 | Sonal Kothari | $\backslash$
T | Shape |  Veta-2012-PVAE  | 2012 | Mitko Veta | $\backslash$
T | Shape |  Lopez-2013-ABDM  | 2013 | Lopez X M | Multiple sharpness features
T | Shape |  Collins-2013-AMAU  | 2013 | Brian T. Collins | $\backslash$
T | Shape |  Veta-2013-DMFB  | 2013 | M. Veta | $\backslash$
T | Shape |  Kothari-2013-ETFA  | 2013 | Kothari S | $\backslash$
T | Shape |  Kong-2013-MBMA  | 2013 | Jun Kong | $\backslash$
T | Shape |  Cooper-2015-NGPA  | 2015 | Lee AD Cooper | $\backslash$
T | Shape |  Barker-2016-ACBT  | 2016 | Barker J | $\backslash$
T | Shape |  Wang-2016-DLIM  | 2016 | Dayong Wang Aditya Khosla | $\backslash$
T | Shape |  Saltz-2017-CSSG  | 2017 | Saltz, J | $\backslash$
T | Shape |  Xu-2018-AACM  | 2018 | Hongming Xu | $\backslash$
DL | CNN |  Puerto-2016-DPAW  | 2016 | Puerto M | $\backslash$
DL | CNN |  Sharma-2016-DCNN  | 2016 | Sharma H | $\backslash$
DL | CNN |  Wang-2016-DLIM  | 2016 | Dayong Wang Aditya Khosla | GoogLeNet
DL | CNN |  Geccer-2016-DCBC  | 2016 | Geçer B | $\backslash$
DL | CNN |  Sirinukunwattana-2016-LSDL  | 2016 | Sirinukunwattana K | NEP coupled with CNN
DL | CNN |  Hou-2016-PCNN  | 2016 | Hou L | $\backslash$
DL | CNN |  Sheikhzadeh-2016-ALMB  | 2016 | Sheikhzadeh F | CNN,FCN
DL | CNN |  Cruz-2017-ARIB  | 2017 | Cruz-Roa A | $\backslash$
DL | CNN |  Wollmann-2017-AGBC  | 2017 | Wollmann T | DNN
DL | CNN |  Araujo-2017-CBCH  | 2017 | Araújo T | Patch-wise trained CNN
DL | CNN |  Bandi-2017-CDMT  | 2017 | Bándi P | FCN,U-net
DL | CNN |  Bejnordi-2017-CASC  | 2017 | Bejnordi B E | Context-aware stacked CNN
DL | CNN |  Sharma-2017-DCNN  | 2017 | Sharma H | Selected self-designed CNN architecture
DL | CNN |  Korbar-2017-DLCC  | 2017 | Korbar B | A modified version of a ResNet architecture
DL | CNN |  Jimenez-2017-DMCR  | 2017 | Jimenez-del-Toro O | $\backslash$
DL | CNN |  Xu-2017-LSTH  | 2017 | Xu Y | $\backslash$
DL | CNN |  Korbar-2017-LUHD  | 2017 | Korbar B | ResNet
DL | CNN |  Ghosh-2017-SLCA  | 2017 | Ghosh A | Deep convolutional network
DL | CNN |  Das-2017-CHWS  | 2017 | Das K, Karri S P K | self-designed CNN
DL | CNN |  Cui-2018-DLAO  | 2018 | Cui Y | FCN
DL | CNN |  Zanjani-2018-CDHW  | 2018 | Farhad Ghazvinian Zanjani |  | Apply CRFs over latent spaces
---
of a trained deep CNN
DL | CNN |  Courtiol-2018-CDLH  | 2018 | Courtiol P | ResNet
DL | CNN |  Kumar-2018-DBFR  | 2018 | Meghana Dinesh Kumar | VGG,AlexNet
DL | CNN |  Bychkov-2018-DLTA  | 2018 | Dmitrii Bychkov | VGG
DL | CNN |  Gecer-2018-DCCW  | 2018 | Baris Gecer | FCN
DL | RNN |  Ren-2018-DPCP  | 2018 | Jian Ren | LSTM
DL | CNN |  Tellez-2018-GWSI  | 2018 | David Tellez | Self-designed CNN
DL | CNN |  Sirinukunwattana-2018-IWSS  | 2018 | K Sirinukunwattana | Self-designed CNN
DL | CNN |  Kwok-2018-MCBC  | 2018 | Scotty Kwok | Inception-Resnet-v2
DL | CNN |  Das-2018-MILD  | 2018 | Kausik Das | A MIL framework for CNN
DL | CNN |  Lin-2018-SFDS  | 2018 | Huangjing Lin | FCN
DL | CNN |  Campanella-2018-TSDM  | 2018 | Campanella G | VGG and ResNet
DL | CNN |  Jiang-2018-TMDD  | 2018 | Jiang S | ResNet
DL | CNN |  Wang-2018-WSLW  | 2018 | X Wang | VGG
DL | CNN |  Shou-2018-WSIC  | 2018 | Junni Shou | DenseNet
DL | CNN |  Tellez-2018-WSMD  | 2018 | David Tellez | $\backslash$
DL | CNN |  Seth-2019-ALBD  | 2019 | Nikhil Seth | U-Net
DL | CNN |  Bertram-2019-LSDM  | 2019 | Christof A. Bertram | ResNet
DL | CNN |  Li-2019-AMRM  | 2019 | Jiayun Li | VGG
DL | CNN |  Liu-2019-AIBC  | 2019 | Liu Y | Inception
DL | CNN |  Campanella-2019-CGCP  | 2019 | Gabriele Campanella | A MIL framework for CNN
DL | CNN |  Yue-2019-CCOP  | 2019 | Xingzhi Yue | VGG
DL | RNN |  Maksoud-2019-CCOR  | 2019 | Sam Maksoud | LSTM
DL | CNN |  Bilaloglu-2019-EPCW  | 2019 | S Bilaloglu | PathCNN(Self-designed)
DL | CNN |  Lin-2019-FSFD  | 2019 | Huangjing Lin | VGG
DL | CNN |  Yu-2019-LSGC  | 2019 | H Yu, X Zhang | DNN
DL | CNN |  Sanghvi-2019-PAIA  | 2019 | Adit B. Sanghvi, MISM | $\backslash$
DL | $\backslash$ |  Wang-2019-RRMI  | 2019 | Shujun Wang | RMDL
DL | CNN |  Kohlberger-2019-WSIF  | 2019 | Kohlberger T | ConvFocus(based CNN)
DL | CNN |  Seth-2019-ASDW  | 2019 | Seth N, Akbar S | U-net
DL | CNN |  Xu-2019-MTPW  | 2019 | Xu J | ResNet and DenseNet
DL | CNN |  Feng-2020-DLMA  | 2020 | Feng Y, Hafiane A | U-net
DL | CNN |  Chen-2020-ITWS  | 2020 | Chen P, Shi X | DeepCIN(CNN,BLSTM)
DL | CNN |  Sornapudi-2020-CWSH  | 2020 | Sornapudi S, Addanki R | $\backslash$
DL | CNN |  Pantanowitz-2020-AIAP  | 2020 | Pantanowitz L | Self-designed CNN
Table 4: Continue: Summary of the CAD methods used for feature extraction in
WSI.

##  4 Segmentation Methods

In recent years, with the increasing size and quantity of medical images,
computers must facilitate processing and analysis. In particular, computer
algorithms for delineating anatomical structures and other areas of interest
are becoming increasingly important in assisting and automating specific
histopathological tasks. These algorithms are called image segmentation
algorithms Pham-2000-CMMI .

Image segmentation refers to the process of dividing a digital image into
multiple segments, namely a set of pixels. The pixels in a region are similar
according to some homogeneity criteria (such as color, intensity, or texture),
to locate and identify objects and boundaries in the image Gonzalez-2007-IP .
The practical applications of image segmentation include: filtering noise
images, medical applications (locating tumors and other pathologies, measuring
tissue volume, computer-guided surgery, diagnosis, treatment planning,
anatomical structure research) Patil-2013-MISR , locating objects in satellite
images (roads, forests, etc.), facial recognition, fingerprint recognition,
etc. The selection of segmentation techniques and the level of segmentation
depends on the specific type of image and the characteristics of the problem
being considered Dass-2012-IST1 .

In the process of medical image segmentation, the details required in the
segmentation process largely depend on the clinical application of the
problems Masood-2015-SMIS  Zuva-2011-ISAT . The purpose of segmentation is to
improve the visualization process to deal with the detection process more
effectively. Medical image segmentation is faced with many problems because
the quality of the segmentation process is affected Shrimali-2009-CTSM . When
there is noise in the image, there will be uncertainty, which makes it
difficult to classify the image Birkfellner-2016-AMIP . The reason is that the
intensity value of the pixel has been modified due to noise in the image. Such
a change in pixel intensity value will disturb the uniformity of the image
intensity range Al-2010-CSRN . Therefore, to deal with this uncertainty, image
segmentation plays a crucial role in medical diagnostic systems He-2013-MIS .

As a crucial step in CAD pathologists, segmentation techniques have flourished
in recent years. As shown in Fig. 2, from 2010 to 2020, the number of papers
using segmentation WSI technology to assist doctors in diagnosis has increased
from 2 to 28. According to the papers we have reviewed, segmentation is
divided into five different techniques including thresholding-based, region-
based, graph-based, clustering-based, deep learning, and other image
segmentation methods. Its composition and structure diagram are shown in
Figure. 20.

Figure 20: The composition of the segmentation method in WSI.

###  4.1 Thresholding based Segmentation Method

Threshold segmentation is a classic method in image segmentation. It uses the
difference in grayscale between the target and the background to be extracted
in the image, and divides the pixel level into several categories by setting a
threshold to achieve the separation of the target and the background
Fu-1981-SIS ; Haralick-1985-IST . The threshold segmentation method is simple
to calculate, and can always use closed and connected boundaries to define
non-overlapping regions. Images with a strong contrast between the target and
the background can provide better segmentation effect Sahoo-1988-STT .

Among them, the selection of the optimal threshold is a significant issue.
Commonly used threshold selection methods are: manual experience selection
method, histogram method Rosenfeld-1983-HCAA , maximum between-class variance
method (OTSU) Ohtsu-1978-DLST , adaptive threshold method. Because the
threshold segmentation method is simple to implement, the amount of
calculation is small, and the performance is relatively stable, it has become
the most in image segmentation. As the basic and most widely used segmentation
technology, it has been used in many fields. Among the reviewed papers, five
are based on threshold based segmentation Kong-2011-IMAG ; Lu-2012-ASAE ;
Shu-2013-SOCN ; Vo-2016-CWSI ; Arunachalam-2017-CAIS .

In Kong-2011-IMAG , an image analysis tool for segmentation and
characterization of cell nuclei is developed. The microscopic images of
glioblastoma from the TCGA project are used. To reliably identify cell nuclei,
a fast hybrid gray-scale reconstruction algorithm is applied to the image to
normalize the background area degraded by artifacts produced by tissue
preparation and scanning Vincent-1993-MGRI . This operation separates the
foreground from the normalized background and allows simple threshold
processing to identify the nucleus.

In Lu-2012-ASAE , a computer-aided technique is proposed for segmentation and
analysis of the whole slide skin histopathological images. Before using the
segmentation technique, determine the single-color channel that provides good
discrimination information between the epidermis and dermis regions. Then
multi-resolution image analysis is used in the proposed segmentation
technique. First, a low-resolution image of the WSI is generated. Then, the
global threshold method and shape analysis to segment low-resolution images
are used. Based on the segmented skin area, the layout of the skin is
determined, and a high-resolution image block of the skin is generated for
further manual or automatic analysis. Experiments on 16 different whole slide
skin images show that the technology has high performance, $92\%$ sensitivity,
$93\%$ accuracy, and $97\%$ specificity are achieved.

In Shu-2013-SOCN , a new method is proposed to segment severely aggregated
overlapping cores. The proposed method first involves applying a combination
of global and local thresholds to extract foreground regions.

In Vo-2016-CWSI  and Vo-2019-MCMH , a highly scalable and cost-effective image
analysis framework based on MapReduce is proposed, and a cloud-based
implementation is provided. The framework adopts a grid-based overlap
segmentation scheme and provides parallelization of image segmentation based
on MapReduce. In the segmentation step, a threshold method is applied to
segment the nucleus.

In Arunachalam-2017-CAIS , the segmentation of tumor and non-tumor areas on
the WSIs datasets of osteosarcoma histopathology. The method in this article
combines pixel-based and object-based methods, using tumor attributes, such as
nucleus clusters, density, and circularity, and using multi-threshold Otsu
segmentation technology to further classify tumor regions as live and
inactive. The pan-fill algorithm clusters similar pixels into cell objects and
calculates the cluster data to analyze the studied area further. The final
experimental results show that for all the sampled datasets used, the accuracy
of the method in question in identifying live tumors and coagulative necrosis
is $100\%$, while the accuracy of fibrosis and acellular/low cell tumors is
about $90\%$. The WSI effect after multi-threshold Otsu segmentation is shown
in Figure. 21.

Figure 21:  Otsu output showing more blue color. This figure corresponds to
Fig.3 in Arunachalam-2017-CAIS .

###  4.2 Region-based Segmentation Method

Region-based segmentation is a kind of segmentation techniques based on
directly finding the region. In fact, similar to the boundary-based image
segmentation technology, it uses the similarity of the object’s gray
distribution and background. Generally, region-based image segmentation
methods contain two categories: watershed segmentation and region growing.

##### Watershed Segmentation

The watershed algorithm draws on the theory of morphology and is a region-
based image segmentation algorithm. In this method, an image is regarded as a
topographic map, and the gray value corresponds to the height of the terrain.
High gray values correspond to mountains, and low gray values correspond to
valleys. If rain falls on the surface, the low-lying area is a basin, and the
ridge between the basins is called a watershed. Watershed is equivalent to an
adaptive multi-threshold segmentation algorithm Vincent-1991-WDSE .

In Kong-2011-IMAG , overlapping nuclei are separated using the watershed
method. In Veta-2012-PVAE , an automatic cell nucleus segmentation algorithm
is used to extract size-related morphometric features of cell nuclei and
analyze their prognostic value in male breast cancer. The segmentation process
consists of four main steps: preprocessing, watershed segmentation controlled
by multi-scale markers, post-processing, and merging of multi-scale results.
The overall process of this automatic segmentation method is shown in the
Figure. 22. In Veta-2013-ANSH , the same automatic segmentation method as in
Veta-2012-PVAE  is used in H&E stained breast cancer histopathology images.

Figure 22: Overview of the automatic nuclei segmentation method. This figure
corresponds to Fig.1 in Veta-2012-PVAE .

In Shu-2013-SOCN , to segment the overlapping nuclei gathered in the
foreground region, seed markers are obtained using morphological filtering and
intensity-based region growth. Then the seed watershed and separate the
aggregated nuclei are applied. Finally, a post-processing step of identifying
positive nuclear pixels is added to eliminate false pixels. Some segmentation
results are shown in Fig. 23. In Vo-2016-CWSI  and Vo-2019-MCMH , watershed
technology is used to separate overlapping nuclei in objects is used.

Figure 23: Some segmentation results. This figure corresponds to Fig.7 in
Shu-2013-SOCN .

##### Region Growing

Region growing is an image segmentation method of serial region segmentation.
Region growth refers to starting from a certain pixel and gradually adding
neighboring pixels according to specific criteria. When certain conditions are
met, the regional growth is terminated, that is, The region’s growth depends
on the selection of the initial point (seed point), growth criteria, and
termination conditions Hojjatoleslami-1998-RGNA .

The region growing is relatively a common method. It can achieve the best
performance when there is no prior knowledge available, and it can be used to
segment more complex images. However, the regional growth method is iterative,
and space and time costs are relatively high Adams-1994-SRG . Among the WSI-
based segmentation tasks we have summarized, there is one paper related to
region growth Shu-2013-SOCN .

###  4.3 Graph-based Segmentation Method

Graph-based segmentation is a classic image segmentation algorithm. The
algorithm is a greedy clustering algorithm based on the graph. Its advantages
include simple implementation and faster speed Felzenszwalb-2004-EGIS . Many
popular algorithms are based on this method Van-2011-SSSO .

Graph-based segmentation first expresses image as a graph in graph theory, so
that, each point in a pixel is regarded as a vertex $v_{i}\in V$, and each
pixel and 8 adjacent pixels (eight neighborhoods) form a graph edge $e_{i}\in
E$, so a graph $G=(V,E)$ is constructed. The weight of each side of the graph
is the relationship between the pixel and the neighboring pixels, which
expresses the similarity between the neighboring pixels. Treat each node
(pixel) as a single area, and then merge it according to the parameters of the
area and the internal difference to get the final segmentation West-1996-IGH .

Because WSIs are usually very large, they are stored as pyramids of tiled
images, so that they can be processed in a hierarchical manner, that is, low-
resolution to determine the area of interest, high-resolution image
classification, top-down segmentation. Among the papers we reviewed, there are
three papers that combine graph-based segmentation methods with multi-
resolution Roullier-2011-MRGA ; Roullier-2010-GMRS ; Roullier-2010-MEBC .

In Roullier-2010-GMRS , Roullier-2010-MEBC  and Roullier-2011-MRGA , the
mitosis in WSI of breast tissue is extracted. The image is simplified by
discrete regularization, and clustering is performed by unsupervised 2-mean
clustering. Clustering is performed in a specific area divided at the previous
resolution. The obtained clusters are expanded with finer resolution levels
through pixel duplication and refined in specific areas. At the last
resolution level, the mitotic figure is extracted. The segmentation result of
mitosis in Roullier-2010-GMRS  is shown in the Figure. 24.

Figure 24: The composition of the segmentation method in WSI. This figure
corresponds to Fig.3 in Roullier-2010-GMRS

###  4.4 Clustering-based Segmentation Method

Clustering is based on the relationship between each pixel and neighboring
pixels. If a pixel is similar in color, texture, or gray to its neighboring
pixels, then they will be merged into the same class.

$k$-means algorithm is more commonly used in clustering algorithms. The
cluster center point is obtained by calculating the average of all pixels in
each cluster in the sample. The basic working principle of the $k$-means
algorithm is to receive the parameter $k$ input by the user, divide the given
$n$ data sample points into $k$ groups on average, take the input $k$ points
as the cluster centers to be converged, and calculate the other clusters. The
Euclidean distance from the sampling point to the $k$ convergence centers. And
compare the distance between all sampling points and the convergence center
point. The classification is made by comparing the minimum form of Euclidean
distance. Then after repeated iterations, the mean value of $k$ clusters is
successively obtained. Until the performance criterion function, clustering is
the best, the overall error is the smallest, and the best clustering effect is
obtained Macqueen-1967-SMCA .

Among the papers we reviewed, the papers on $k$-means clustering and
segmentation are Hiary-2013-SLWS  and Arunachalam-2017-CAIS . In
Hiary-2013-SLWS , the author uses $k$-means unsupervised learning for WSI
segmentation, which produces a highly robust correctness result equivalent to
supervised learning, which is $95.5\%$ accuracy. In Arunachalam-2017-CAIS ,
the $k$-means clustering technique with color normalization is used for tumor
separation.

###  4.5 Deep Learning based Segmentation

Among the papers we reviewed, 10 papers used deep learning methods for WSI
segmentation Bandi-2017-CDMT ; Xu-2017-LSTH ; Dong-2018-RANT ; Cui-2018-DLAO ;
Sirinukunwattana-2018-IWSS ; Mehta-2018-LSBB ; Vo-2019-MCMH ; Seth-2019-ALBD ;
Seth-2019-ASDW ; Feng-2020-DLMA .

In Bandi-2017-CDMT , two different CNN structures, FCN and U-net, are used to
segment and accurately identify tissue slices. Here, the two methods are
compared with the traditional foreground extraction (FESI) algorithm based on
structural information. These three methods are applied to 54 WSIs, and the
average value of the Yakoka index and the standard deviation of the Yakoka
index are used for evaluation. The final U-net result is the best (Jaccard
index is 0.937). The qualitative effects of different algorithms are shown in
Figure. 25.

Figure 25: Qualitative results for the different algorithms. This figure
corresponds to Fig.2 in Bandi-2017-CDMT

In Xu-2017-LSTH , CNN-based ImageNet is used to extract features and convert
them into histopathological images. And Support Vector Machine (SVM) is used
to define segmentation as a classification problem. This method is applied to
the digital pathology and colon cancer dataset of the MICCAI 2014 Challenge,
and finally won the first place with $84\%$ test data accuracy.

In Dong-2018-RANT , a simple and effective framework called Reinforced Auto-
Zoom Net (RAZN) is proposed, which considers the accurate and fast prediction
of breast cancer segmentation. RAZN learns a strategy network to decide
whether to zoom on a given area of interest. Because the amplification action
is selective, RAZN is robust to unbalanced and noisy ground truth labels and
effectively reduces overfitting. Finally, the method is evaluated on the
public breast cancer dataset. It can be seen from the experimental results
that RAZN is superior to single-scale and multi-scale baseline methods, and
obtains better accuracy with lower inference cost.

In Cui-2018-DLAO , an automatic end-to-end deep neural network algorithm is
proposed to segment the single core. The kernel-boundary model is introduced
to predict the kernel and its boundary simultaneously using FCN. Given the
color normalized image, the model directly outputs the estimated kernel map
and boundary map. After post-processing, the final segmented core is produced.
A method for extracting and assembling overlapping blocks is designed to
seamlessly predict the cores in a large WSI. The final result proves the
effectiveness of the data expansion method for cell nucleus segmentation
tasks. The experiment shows that this method is superior to the prior art
method and it is possible to accurately segment WSI within an acceptable time.

In Sirinukunwattana-2018-IWSS , different architectures are systematically
compared to evaluate how the inclusion of multi-scale information affects
segmentation performance. The architectures are shown in Figure. 26. It uses a
public breast cancer dataset and a locally collected prostate cancer dataset.
The result shows that the visual environment and scale play a vital role in
the classification of histological images.

Figure 26: Used architectures. This figure corresponds to Fig.2 in
Sirinukunwattana-2018-IWSS

In Mehta-2018-LSBB , a new encoder-decoder architecture is proposed to solve
the semantic segmentation problem of breast biopsy WSI. The designed new
architecture contains four new functions: (1) Input Perceptual Coding Block
(IA-RCU), which enhances the input inside the encoder to compensate for the
loss of information due to down-sampling operations, (2) densely connected
decoding network and (3) additional sparsely connected decoding network to
effectively combine the multi-level features aggregated by the encoder, and
(4) a multi-resolution network for context-aware learning, which uses densely
connected fusion blocks to combine different resolutions rate output. The
architecture is shown in Figure. 27. The result after segmentation is shown in
Figure. 28.

Figure 27: Multi-resolution encoder-decoder network structure diagram. This
figure corresponds to Fig.4 in Mehta-2018-LSBB .

Figure 28: Segmentation result. The first line describes aggressive cases,
while the second line describes benign cases. This figure corresponds to Fig.9
in Mehta-2018-LSBB .

In Seth-2019-ALBD  and Seth-2019-ASDW , several U-net architectures-DCNN
designed to output probability maps are trained to segment ductal carcinoma
in-situ (DCIS) WSI in wireless sensor networks and verified the minimum
required to achieve excellent accuracy at the slide level good patch vision.
U-net is trained five times to achieve the best test results (DSC = 0.771, F1
= 0.601), which means that U-net benefits from seeing wider contextual
information.

In Feng-2020-DLMA , a multi-scale image processing method is proposed to solve
the segmentation problem of liver cancer in histopathological images. These
eight networks are compared, and then the network most suitable for liver
cancer segmentation is selected. Through a comprehensive comparison of
performance, U-net is selected. The local color normalization method of
pathological images is used to solve the influence of the background, and then
a seven-layer Gaussian pyramid representation is established for each WSI to
obtain a multi-scale image set. The trained U-net is fine-tuned at each level
to obtain an independent model. Then, shift cropping and weighted overlap are
used in the prediction process to solve block continuity. Finally, the
predicted image is mapped back to the original size, and a voting mechanism is
proposed to combine the multi-scale predicted images. The experimental data
are the verification images of the 2019 MICCAI PAIP Challenge. The evaluation
results show that this algorithm is better than other algorithms.

###  4.6 Other Segmentation Methods

In Bergmeir-2012-SCCN , a new algorithm for segmenting cell nuclei is used.
Before determining the precise shape of the cell nucleus by the elastic
segmentation algorithm, the proposed algorithm uses a voting scheme and prior
knowledge to locate the cell nucleus. After removing noise through the mean
shift and median filtering, the Canny edge detection algorithm is used to
extract edges. Since the nucleus is observed to be surrounded by cytoplasm,
its shape is roughly elliptical, and the edges adjacent to the background are
removed. The random hough transform of the ellipse finds the candidate kernel,
and then processes it through the level set algorithm. The algorithm is tested
and compared with other algorithms in a database containing 207 images
obtained from two different microscope slides, and the results passed the
positive predictive value (PPV) and TPR. The high value of, is displayed,
resulting in a high measurement value of $96.15\%$.

In Apou-2014-FSTC , a fast segmentation method based on WSI is proposed. Due
to the large size of the WSI, a set of horizontal and vertical optimal paths
that follow the high gradient of the image are used to segment the image so
that the relevant segmentation of the image is provided in an effective
manner. Then other subsequent steps are executed. The schematic diagram of the
segmentation process is shown in Figure. 29.

Figure 29: Path computation in horizontal and vertical strips, leading to an
image partition. This figure corresponds to Fig.4 in Apou-2014-FSTC

In Zhang-2015-FGHI , a robust segmentation method is developed to accurately
delineate ROI (eg, cells) using hierarchical voting and repelling active
contours. Its segmentation is based on active contours with repelling terms
Cohen-1991-OACM . The exclusion term is used to prevent the contour lines from
intersecting and merging. Based on the detection result, the circle is
associated with each detected cell as the initial contour. The final
segmentation result is shown in Figure. 30.

Figure 30: Segmentation results of different methods on a randomly picked
patch. From left to right: original image and ours. This figure corresponds to
Fig.3 in Zhang-2015-FGHI

.

In Li-2015-FRID , a new ROI search and segmentation algorithm based on
superpixels is proposed. First, the initial recognition of the ROI is obtained
by gathering superpixels at low magnification. Then, by marking the
corresponding pixels, the superpixels are mapped to the higher magnification
image. This process is repeated several times until the segmentation is
stable. This method is different from the previous classic segmentation
methods based on superpixels Achanta-2012-SSCS ; Yamaguchi-2014-EJSO . This
algorithm provides image segmentation with topology preservation.

In Brieu-2016-SSMS , threshold processing is performed on the foreground
posterior image to detect the foreground area, and spot detection algorithms
such as MSER and Gaussian difference are used further to identify the
brightness of the image. According to the classification function combining
suitable candidate parameters are selected. The remaining area is further
divided by calculating the minimum path of the posterior mapping between the
concave points and checking the goodness of fit of the candidate area in turn.
The schematic diagram after segmentation is shown in Figure. 31.

Figure 31: The schematic diagram after segmentation is shown in the figure.
Left: H&E (a)-(b), IHC-nuclei (c), and CD8 (d) regions; Center: posterior maps
for marked nuclei and cells (blue channel if applicable), for homogeneous
nuclei (red channel) and for textured nuclei (green channel); Right: outline
of the segmentation result. This figure corresponds to Fig.3 in
Brieu-2016-SSMS .

###  4.7 Summary

As can be seen from the content we reviewed above, machine learning is used in
combination with WSI technology to assist diagnosis. In the field of image
segmentation, used methods include thresholding-based segmentation, region-
based segmentation, graph-based segmentation, clustering-based segmentation,
deep learning-based segmentation, and other segmentation methods.

We can know that the graph-based segmentation method is a more classic image
algorithm, so it became popular earlier, and many algorithms are based on the
graph-based segmentation method. Thresholding-based segmentation methods are
often used in WSI in combination with the watershed algorithm in region-based
segmentation methods. In the papers we reviewed, three of them used both
methods at the same time. Then clustering-based segmentation methods and some
other segmentation methods appeared to be used. Until 2017, deep learning
methods are widely used, and the application of deep learning to WSI
segmentation began to get good results. Among them, the U-net architecture
based on multi-resolution has been used many times. Table. LABEL:SCMS is a
summary of the CAD method for segmentation technology in WSI.

Table 5: Summary of the CAD methods used for segmentation in WSI. Method | Reference | Year | Team | Details
---|---|---|---|---
Thresholding |  Kong-2011-IMAG  | 2011 | Jun Kong | Simple threshold watershed method
Thresholding |  Lu-2012-ASAE  | 2012 | Cheng Lu | Otsu threshold
Thresholding |  Shu-2013-SOCN  | 2013 | Jie Shu |  | Automatic threshold method
---
(foreground and background
classification), region
growth, watershed
segmentation (seriously
clustered overlapping
cores for segmentation)
Thresholding |  Vo-2016-CWSI  | 2016 | Vo H |  | Parallelization of
---
image segmentation based
on MapReduce
Thresholding |  Arunachalam-2017-CAIS  | 2017 | Arunachalam H B |  | The combination of pixel-based and
---
object-based methods (k-means and
non-tumor images, and multi-level
Otsu threshold segmentation)
Region-based(Watered) |  Kong-2011-IMAG  | 2011 | Jun Kong |  | Simple threshold
---
watershed method
Region-based(Watered) |  Veta-2012-PVAE  | 2012 | Mitko Veta |  | Watershed segmentation
---
controlled by
multi-scale markers
Region-based(Watered) |  Veta-2013-ANSH  | 2013 | Veta M |  | Marker-controlled
---
watershed segmentation,
with multiple scales and
different markers
(automatic nuclear segmentation)
Region-based(Watered) |  Shu-2013-SOCN  | 2013 | Jie Shu |  | Automatic threshold method
---
(foreground and background
classification), region growth,
watershed segmentation
(seriously clustered overlapping
cores for segmentation)
Region-based(Watered) |  Vo-2016-CWSI  | 2016 | Vo H |  | Parallelization of Image
---
Segmentation Based
on MapReduce
Region-based(Watered) |  Vo-2019-MCMH  | 2019 | Hoang Vo |  | Core segmentation
---
(MapReduce architecture)
Region-based(Region growing) |  Shu-2013-SOCN  | 2013 | Jie Shu |  | Automatic threshold method
---
(foreground and background
classification), region growth,
watershed segmentation (seriously
clustered overlapping cores
for segmentation)
Graph-based |  Roullier-2010-GMRS  | 2010 | V. Roullier |  | Integrates graph-based
---
segmentation (discrete semi-
supervised clustering)
and multi-resolution
segmentation (cluster
space refinement)
Graph-based |  Roullier-2010-MEBC  | 2010 | Vincent Roullier |  | Integrates graph-based
---
segmentation (discrete semi-
supervised clustering)
and multi-resolution
segmentation (cluster
space refinement)
Graph-based |  Roullier-2011-MRGA  | 2011 | Vincent Roullier |  | Multi-resolution segmentation
---
method (regularization framework,
histogram construction,
histogram 3 mean clustering,
partition and spatial refinement)
Clustering |  Hiary-2013-SLWS  | 2013 | Hazem Hiary |  $k$-means clustering
Clustering |  Arunachalam-2017-CAIS  | 2017 | Arunachalam H B |  | The combination of pixel
---
-based and object-based methods
(k-means and multi-level
Otsu threshold)
Deep Learning |  Bandi-2017-CDMT  | 2017 | Bándi P |  | Using FCN and U-net
---
to organize background
segmentation
Deep Learning |  Xu-2017-LSTH  | 2017 | Xu Y | SVM-CNN
Deep Learning |  Dong-2018-RANT  | 2018 | Nanqing Dong |  | High-resolution image
---
semantic segmentation
framework-Reinforced
Auto-Zoom Net (RAZN) (FCN)
Deep Learning |  Cui-2018-DLAO  | 2018 | Cui Y |  | Supervised FCN method
---
for nuclear segmentation
in histopathological images
Deep Learning |  Sirinukunwattana-2018-IWSS  | 2018 | K Sirinukunwattana |  | Combining multiple
---
CNNs of different
scales with LSTM
Deep Learning |  Mehta-2018-LSBB  | 2018 | Sachin Mehta |  | Multi-resolution
---
encoder-decoder network
semantic segmentation
Deep Learning |  Vo-2019-MCMH  | 2019 | Hoang Vo |  | Core segmentation
---
(MapReduce architecture)
Deep Learning |  Seth-2019-ALBD  | 2019 | Nikhil Seth |  | Multi-resolution
---
U-net architecture
Deep Learning |  Seth-2019-ASDW  | 2019 | Seth N, Akbar S |  | Multi-resolution
---
U-net architecture
Deep Learning |  Feng-2020-DLMA  | 2020 | Feng Y, Hafiane A |  | Multi-resolution
---
seven-layer pyramid U-net
Other |  Bergmeir-2012-SCCN  | 2012 | Christoph Bergmeir |  | Random Hough transform
---
ellipse fitting cell
nucleus to segment the nucleus
(level set algorithm)
Other |  Apou-2014-FSTC  | 2014 | Apou G, Naegel B |  | The path calculation in
---
the horizontal and
vertical strips
Other |  Zhang-2015-FGHI  | 2015 | Xiaofan Zhang |  | (Robust segmentation method)
---
Use Euclidean distance.
Based on active contours
with repelling terms
Other |  Li-2015-FRID  | 2015 | Ruoyu Li and Junzhou Huang |  | The superpixels are clustered
---
at low magnification to
obtain ROI. Then, the
superpixels are mapped
to the higher magnification
image. This process is
repeated several times.
Other |  Brieu-2016-SSMS  | 2016 | Brieu N, Pauly O |  | It is further divided
---
by calculating the
minimum path of the
posterior mapping between
the concave points and
checking the goodness of
fit of the candidate
area in turn
Table 5: Continue: Summary of the CAD methods used for segmentation in WSI.

##  5 Classification Methods

Image classification, as the name suggests, is to have a fixed set of
classification labels, and then for the input image, find a classification
label from the classification label set, and finally assign the classification
label to the input image. It is at the heart of computer vision and is the
most fundamental issue that forms the basis for other computer vision tasks
such as positioning, detection, and segmentation Rawat-2017-DCNN
Karpathy-2015-DVSA . It is widely used in practice. While a simple task for
humans, it can be challenging for computer systems. Many seemingly different
problems in computer vision (such as object detection and segmentation) can be
reduced to image classification.

In the analysis of pathological images, the most studied task is CAD. It also
helps the pathologist make a diagnosis. The diagnostic process is the task of
mapping one or more WSIs to a disease category. Since errors produced by
machine learning systems are different from those produced by human
pathologists Wang-2016-DLIM , the use of computer-aided design systems can
improve classification accuracy Komura-2018-MLMH .

In recent years, due to the progress of computer technology, histopathological
image classification has gradually become a research hotspot in the field of
medical image processing. To the human anatomy area and the pathological
changes area to carry on the accurate classification, may the maximum degree
doctor accurate, the rapid diagnosis condition. This is of great significance
to the further diagnosis of doctors and the further treatment of patients.

In the papers we summarized, there are around 54 articles from 2004 to 2020 on
image classification using WSI techniques to assist pathologists in diagnosis.
We can see from the development trend in Fig. 2 that the application of
classification has increased. This reflects the wide application of
classification technology. From these papers, we can briefly summarize the
classification methods they applied including traditional machine learning
algorithms, deep learning algorithm, and other methods.

###  5.1 Traditional Machine Learning based Classification Method

Among the papers we reviewed, there are 18 papers involve the classification
of WSI by using traditional machine learning algorithms for CAD.

##### SVM-based Classification Method

SVM is a type of supervised machine learning technique. It was first published
in 1963 by Vladimir N. Vapnik and Alexander Y. Lerner Vapnik-1963-RPHG . It
uses the hypothesis space of linear functions in hyperspace Roweis-2000-NDRL
and trains with the learning algorithm of optimization theory, which realizes
the learning bias derived from statistical learning theory. The purpose of
classification by SVM is to find an effective computational method to learn
good separation hyperplanes in hyperspace Kumar-2012-RCIC . SVM is designed
for binary classification. when an SVM is applied to a multi-class
classification problem, it internally splits the task into multiple binary
classification problems and uses several SVMs to solve them Kamath-2018-CSTM
Madzarov-2009-MCSC . A total of ten papers in our review involve the use of
SVMs for WSI classification.

In Difranco-2011-ESWS , a tile-based approach is proposed to generate
clinically relevant probability maps of prostate cancer in prostate WSIs. The
probability of cancer existence is calculated from the response of each
classifier in the ensemble. Before classification, texture feature extraction
and spatial filtering are performed. The classification is then performed
using either an RF or an SVM (linear and radial kernels). Different feature
subsets and different subsampled training data strategies are used for
performance comparison. The final best classification result is obtained by
Radial Basis Function (RBF) kernel SVM, which reports an AUC value of
$95.50\%$. The final heat map is shown in Figure. 32.

Figure 32: This figure shows the utility of using classification probabilities
for heat map visualization at the tile level. There is a good difference
between strong positive and negative results (red versus blue). This figure
corresponds to Fig.16 in Difranco-2011-ESWS .

In Nayak-2013-CTHS , a method of automatically learning features from
unlabeled images is proposed for WSI classification. The first step is to
learn the dictionary from the unlabeled images. Then the sparse automatic
encoder is used to learn the function. An automatic encoder consists of three
parts: an encoder, a dictionary, and a set of codes. The encoder is used to
train the classifier on a small amount of label data. Multi-class
regularization support vector classification is used, the regularization
parameter is 1, and the polynomial kernel is 3. In terms of data, two datasets
from (i) glioblastoma multiforme (GBM) and (ii) clear cell renal cell
carcinoma (KIRC) of TCGA are used. The classification accuracy rates of
$84.3\%$ and $80.9\%$ are obtained respectively. The classification results of
heterogeneous GBM tissue sections are shown in Figure. 33.

Figure 33: Two examples of classification results of a heterogeneous GBM
tissue sections. The left and right images correspond to the original and
classification results, respectively. Color coding is black (tumor), pink
(necrosis), and green (transition to necrosis). This figure corresponds to
Fig.4 in Nayak-2013-CTHS .

In Jiao-2013-CCDU , an SVM based classifier is used to classify colon cancer
WSI and non-colon cancer WSI. Then, 18 simple features (such as gray-level
mean and gray-level variance) and 16 texture features extracted by the GLCM
method are selected as the feature set. The final result shows that when all
features are used, the mean values of accuracy, recall and F-measure are
$96.67\%$, $83.33\%$, and $89.51\%$, respectively.

In Lu-2015-AADS , a technology for automatically diagnosing skin using WSI is
proposed. There are five steps: epidermal segmentation, keratinocyte
segmentation, melanocyte detection, feature construction and classification.
Based on the segmented ROI, the spatial distribution and morphological
features are constructed. A multi-class SVM classifier classifies these
features representing skin tissue. It provides about $90\%$ accuracy in the
classification of melanoma, and normal skin.

In Weingant-2015-EPTC , the WSI of prostate tumors is classified. First,
adaptive cell density estimation is introduced, and H&E staining normalization
is applied in the supervised classification framework to improve the
robustness of the classifier. RF feature selection, class-balanced training
sample sub-sampling, and SVM classification are used to predict high- and low-
grade prostate cancer on image slices. AUC measured the classification
performance to get 0.703 for HG-PCA and 0.705 for LG-PCA. The results proved
the effectiveness of cell density and staining standardization for prostate
WSIs classification.

In Peikari-2015-TDRR , a texture analysis technique is proposed to simplify
the processing of H&E stained WSIs by classifying clinically important areas.
The specific method is to randomly select image blocks from the entire tissue
area, divide them into small blocks, and perform Gaussian texture filtering on
them. The texture filter responses for each texture are combined and
statistical indicators are obtained from the histogram of the response. Then,
the visual word bag pipeline is used to combine extracted features to form a
word histogram for each image block. The SVM classifier is trained using the
computed lexical histograms to distinguish clinically relevant and unrelated
plaques. Finally, the ROC is 0.87. It can be proved that texture features can
be used to classify important areas in WSI.

In Gadermayr-2016-DACC , to improve the generalization ability of the
classification model for kidney WSI, a domain adaptive method is proposed, in
which the classifier is trained on the data from the source domain to present
a small number of user-labeled samples from the target domain. Efficient
linear SVM is used to avoid waiting time during interaction. In a comparison
between interactive and non-interactive domain adaptation, it is observed that
interactive domain adaptation has a positive effect on the classification
performance.

In Shukla-2017-CFRI , SVM is used to predict the presence of cancer by WSI of
sentinel lymph nodes, and an equivalent fuzzy model is developed to improve
the interpretability. The SVM model consists of 50 support vectors, and the
accuracy rate is $94.59\%$. This uses all 50 support vectors to create a fuzzy
rule-based model (FRBS) equivalent to two IF-then rules, which categorizes
each WSI as non-cancerous and cancerous. FRBS achieved an accuracy rate of
$91.89\%$. Experiments show that the work of SVM can be accurately represented
in a human interpretable method.

In Xu-2018-AACM , an automatic WSI analysis and classification technology for
melanocyte tumors is proposed. First, the skin epidermis and dermis regions
are segmented through a multi-resolution frame. Next, an epidermal analysis is
performed, in which a set of epidermal features reflecting the nucleus
morphology and spatial distribution are calculated. While performing epidermal
analysis, dermal analysis is also performed, in which the dermal cell nucleus
is segmented, and a set of structural and cytological features are calculated.
Finally, by using a multi-class SVM with extracted epidermal and dermal
features, the skin melanocyte images are classified into different categories.
It is known from the experimental results that the classification accuracy of
this technology is over $95\%$.

##### Random Forest-based Classification Method

RF is a popular machine learning algorithm, often used in classification tasks
in various fields Cutler-2007-RFCE ; Ghimire-2010-CLCC ; Gislason-2006-RFLC ;
Guo-2011-RALM ; Chen-2005-PPPI ; Ozccift-2011-RFEC ; Seera-2014-HISM ;
Titapiccolo-2013-AIMS . RF is a collection of tree structure classifiers
Breiman-2001-RF . Each tree relies on the value of a randomly selected vector
distributed in the same way among all trees in the forest Masetic-2016-CHFD .
Each tree in the forest will vote once, assigning each input to the most
likely category label. This method is fast and robust to noise, and is a
successful ensemble that can identify nonlinear patterns in data. It can
easily handle numerical and categorical data Titapiccolo-2013-AIMS . One of
the main advantages of RF is that even if more trees are added to the forest,
it causes overfitting Chaudhary-2016-IRFC . In the papers we have summarized,
there are five papers mainly using RF classifiers for related WSI
classification.

In Homeyer-2013-PQNH , it is mainly to quantify the necrotic part in
histological WSIs. First, the threshold principle is used to eliminate the
background. Color and texture features are then extracted. After the feature
classification step is performed, another post-processing step is performed to
process the misclassified isolated tiles, that is, using the principle of
context classification and spatial context information to re-evaluate all
uncertainly classified slices. The sections are then merged, and each section
is identified by merging all the non-background slices joined by the edges.
Finally, the results are evaluated. Naive Bayes classifier, $k$-nearest
neighbor ($k$NN), and RF classifier are used in the post-processing
classifier. From the experimental results, it can be known that the RF
classifier has the best results, and the HSV-based features are better than
RGB.

In Sharma-2016-DCNN , the main task is to automatically classify cancer from
histopathological stomach images. The classification performance of
traditional image analysis methods and deep learning methods are
quantitatively compared. First, the data is augmented. In traditional image
analysis methods, the classifier used is a RF classifier.

Metastatic breast cancer is identified in Wang-2016-DLIM . First, the
background is automatically detected based on the threshold. And by comparing
GoogLeNet, AlexNet, VGG16 and FaceNet. GoogLeNet is selected to generate tumor
probability heatmaps, and a RF classifier is used to classify metastatic WSIs
and negative WSIs, and finally the AUC of 0.925 is obtained.

In Jamaluddin-2017-TDWS , normal sections and tumor sections from histological
images of lymph node tissue are classified. The first step is to remove
unnecessary information. The CNN model is then reconstructed or trained to
segment the tumor area. The specific details are in Sec. 6. After the tumor
area is segmented, the features are extracted, and the RF classifier is used
for classification. Finally the result with an AUC score of 0.94 is obtained.

In Klimov-2019-WSIB , the risk of recurrence of DCIS is classified. First of
all, the color normalization and down sampling tasks is performed. Secondly,
the texture features are extracted. Then, input these features into the RF
classifier to predict the high and low risk of recurrence. It is convenient
for the doctor to give the corresponding diagnosis and treatment plan. The
final result shows that the classifier significantly predicts the 10-year risk
of recurrence during training (accuracy = 0.87, sensitivity = 0.71, and
specificity = 0.91).

##### Others Traditional Machine Learning Classification Method

In addition to the two commonly used classification methods of SVM and RF,
some other methods are also used to classify histopathological WSI, such as
Bayesian classifier Domingos-1997-OSBC  and $k$NN Hastie-1996-DANN
classifier.

In Sertel-2008-CAPN ; Sertel-2009-CPNW , a system has been developed for
quantitative analysis of neuroblastoma WSIs, which included stromal rich and
stromal poor. The developed method is based on the Gaussian pyramid method
with multi-resolution. WSIs include non-overlapping image blocks and parallel
processing of image blocks, which is carried out by the parallel computing
module developed previously. Then, the texture features are extracted and the
optimal subset is executed. Next, the feature selection of the sequential
float forward selection (SFFS) method is adopted, and the confidence degree is
calculated by $k$NN classification. If the confidence level falls below the
set threshold, switch to a higher resolution. The experimental results show
that the overall classification accuracy is $95\%$, and the calculation amount
is reduced by $60\%$.

In Kong-2009-CAEN , WSI is graded for neuroblastoma biopsy. The texture
features obtained from the segmentation components of the tissue are extracted
and processed by an automatic classifier group. This automatic classifier
group Multiple Classifiers: $k$NN, linear discriminant analysis (LDA) & $k$NN,
LDA & nearest mean (NM), correlation LDA (CORRLDA) & $k$NN, CORRLDA & NM, LDA
& Bayesian and SVM with a linear kernel. The output of multiple classifiers is
then selected using a simple two-step classifier combination mechanism
consisting of voting and weighting processes. The automatic classifier group
is trained in multi-resolution frame with different levels of differentiation.
The trained classification system is tested on 33 WSIs. Finally, the
classification accuracy is $87.88\%$.

In Doyle-2010-BBMC , prostate cancer regions in WSIs are identified. WSIs are
first decomposed into an image pyramid with multiple resolution levels. Areas
identified as cancer by a Bayesian classifier at a lower resolution are then
identified at a higher resolution. At each resolution level, the AdaBoost
integration method is used to select 10 image features from more than 900
first order statistical, second order co-occurrence and Gaborfilter feature
groups. The experimental result shows that compared with other classifiers,
the Bayesian classifier produces higher AUC and precision.

###  5.2 Deep Learning based Classification Method

In this section, the relevant contents of using deep learning algorithms to
classify histopathological images with WSIs are summarized.

In Arevalo-2015-UFLF , basal cell carcinoma WSIs are studied as an integrated
unsupervised characteristic. First, a set of feature detectors are learned
from a set of patches randomly sampled in the image set. The detectors will
capture the most common patterns by simulating the automatic encoder neural
network. The image is then represented using the convolution or BOF method.
This representation is achieved using the feature detector learned in the
previous step. Next, the representation obtained from the convolution or BOF
method to train the binary classification model, the softmax regression
classifier. Basal cell carcinoma includes different categories of cancer and
non-cancer carcinoma. The final result of the system is shown in Figure. 34.
The best results in AUC are obtained, which are superior to the most advanced
$7\%$ and $98.1\%$.

Figure 34: Outputs produced by the system for different cancer and non-cancer
input images (red stain indicates cancer regions, blue stain indicates normal
regions). This figure corresponds to Table.2 in Arevalo-2015-UFLF .

In Barker-2016-ACBT , brain tumor types in WSIs are automatically classified.
The main method is to analyze local features from coarse to fine in
pathological images. Firstly, the diversity of rough areas in WSIs is
analyzed, including the spatial local characteristics of shape, color, and
texture of WSIs. Then the clustering-based method is used to create the
representative group. The individual representative tiles in each group are
then concretely analyzed. An Elastic Net classifier produces the diagnostic
decision value for tile to obtain the WSI level diagnosis. Finally, 302 cases
of brain cancer included automatically two possible diagnoses (glioblastoma
multiforme (182 cases) and glioblastoma low grade (120 cases)) to evaluate our
method with an accuracy rate of $93.1\%$ ($P\ll 0.001$).

In Geccer-2016-DCBC  and Gecer-2018-DCCW , deep convolutional networks are
used to detect and classify WSIs ROI in breast cancer. The detection part is
described in detail in Sec. 6. In this part, the classification is mainly
introduced, that is, the ROI include five diagnostic categories. The
classifier is designed based on CNN, which uses the features learned by CNN to
classify the detected ROI. The CNN structure diagram designed is shown in
Figure. 35. Then the post-processing of WSI classification is carried out.
WSIs are classified according to the prediction of most categories in the
remaining cancer areas. The results show that the efficiency is improved by
about 6.6 times with sufficient accuracy.

Figure 35: Designs of the CNN. This figure corresponds to Figure.4.6 in
Geccer-2016-DCBC .

In Sirinukunwattana-2016-LSDL , it mainly includes two parts, namely, the
detection and classification of cell nuclei. The detection part proposes a
spatially constrained CNN for nuclear detection. The details are in Sec. 6.
Classification uses a new NEP combined with CNN. None of the proposed
detection and classification methods require nuclear segmentation. And the
proposed method is applied to colorectal adenocarcinoma WSIs, and the final
classification result obtains a higher F1 score. The final detection and
classification results are shown in the Figure. 36.

Figure 36: The detection and classification are the result graphs under 20
times magnification. This figure corresponds to Figure.9.(c) in
Sirinukunwattana-2016-LSDL .

In Hou-2016-PCNN , an expectation maximization method is proposed. The spatial
relationship of patches is used to locate the distinguished patches robustly.
This method is applied to the subtype classification of glioma and non-small
cell lung cancer. The classification module uses patch-level CNN and trains a
decision fusion model as a two-level model. The first level (patch-level)
model is based on expectation maximization, combined with CNN output patch-
level prediction. In the second level (image-level), the patch-level predicted
histogram is input to the image-level multiple logistic regression or SVM.

In Babaie-2017-CRDP , a new dataset, Kimia Path24, is introduced for image
classification and retrieval in digital pathology. The WSIs of 24 different
textures are generated to test patches. Especially, the patch classification
is based on LBP histograms, bag-of-words approach, and CNN.

In Araujo-2017-CBCH , a method to classify WSI for breast biopsies using CNN
is proposed. The proposed network architecture can retrieve information on
different scales. First, the image includes 12 non-overlapping patches, and
then the piece-by-piece training CNN and CNN+SVM classifier are used to
calculate the patch-level probability. Finally, one of three different patch
probability fusion methods is used to obtain image classification results.
These three methods are majority voting (choosing the most common patch as the
image tag), maximum probability (choosing the patch category with high
probability as the image tag), and probability (the category with the largest
sum of patch-level probability). The final results classify the images into
four categories: invasive, in situ, benign and normal. The proposed system
achieves the overall sensitivity of about $81\%$ of cancer patch
classification.

In Bejnordi-2017-CASC , a context-aware stacked CNN is proposed to classify
WSIs as normal/benign, DCIS, and invasive ductal carcinoma (IDC). The first is
to train a CNN and use high pixel resolution to capture cell level
information. The characteristic responses generated by the model are then
inputted to the second CNN and superimposed on the first CNN. The system had
an AUC of 0.962 for binary classification of non-malignant and malignant
slides, and a three-level accuracy rate of $81.3\%$ for normal/benign
classification of WSIs, DCIS, and IDC.

In Sharma-2017-DCNN , gastric cancer WSI is automatically classified. The
traditional image analysis method and deep learning method are proposed and
compared quantitatively. In the traditional analysis method, GLCM, Gabor
filter bank response, LBP histogram, gray level histogram, HSV histogram, and
RGB histogram are used for classification and RF. In terms of the deep
learning method, AlexNet is proposed as a deep convolution framework. The
structure of the network is shown in Figure. 37. According to the experiment,
the overall classification accuracy of the cancer classification proposed by
AlexNet is 0.6990, and the overall classification accuracy of necrosis
detection is 0.8144.

Figure 37: Proposed AlexNet architecture. This figure corresponds to Figure.4
in Sharma-2017-DCNN .

In Korbar-2017-DLCC , different types of colorectal polyps on WSIs are
classified to help pathologists diagnose them. Here is a modified version of
the ResNet structure. The whole WSI is divided into patches and then applied
to the ResNet. If at least five patches on a WSI are recognized as this class,
the average confidence level is $70\%$. If there is no cancer type in the
patch, the WSI is considered normal. Finally, the accuracy is $93.0\%$. The
recall is $88.3\%$; F1 scores $88.8\%$.