∎

11institutetext: Xiaoqi Li, Chen Li, Xintong Li, Jian Wu 22institutetext:
Microscopic Image and Medical Image Analysis Group, College of Medicine and
Biological Information Engineering, Northeastern University, Shenyang, 110169,
China
Corresponding author: Chen Li, E-mail<EMAIL_ADDRESS>33institutetext: Md Mamunur Rahaman School of Computer Science and
Engineering, University of New South Wales, Sydney, NSW 2052, Australia
44institutetext: Xiaoyan Li, Hongzan Sun 55institutetext: China Medical
University, Shenyang, 110001, China
66institutetext: Marcin Grzegorzek 77institutetext: Institute of Medical
Informatics, University of Luebeck, Luebeck, Germany

# What Can Machine Vision Do for Lymphatic Histopathology Image Analysis: A
Comprehensive Review

Xiaoqi Li  Haoyuan Chen  Chen Li  Md Mamunur Rahaman  Xintong Li  Jian Wu
Xiaoyan Li  Hongzan Sun  Marcin Grzegorzek

(Received: date / Accepted: date)

###### Abstract

Over the past ten years, machine vision (MV) algorithms for image analysis
have been developing rapidly with computing power. At the same time,
histopathological slices can be stored as digital images. Therefore, MV
algorithms can provide diagnostic references to doctors. In particular, the
continuous improvement of deep learning algorithms has further improved the
accuracy of MV in disease detection and diagnosis. This paper reviews the
application of image processing techniques based on MV in lymphoma
histopathological images in recent years, including segmentation,
classification and detection. Finally, the current methods are analyzed, some
potential methods are proposed, and further prospects are made.

###### Keywords:

Machine vision lymphatic histopathology image computer-aided diagnosis image
preprocessing image segmentation feature extraction classification detection

##  1 Introduction

###  1.1 Basic Clinical Information about Lymph

Lymph is also known as lymphatic fluid, a colorless and transparent liquid in
humans and animals. It contains lymphocytes and is partly formed by tissue
fluid infiltrating into the lymphatic vessels. Lymph exists in all parts of
the human body and plays an important role in the immune system. The lymph
nodes are organized as lymphoid organs, containing lymphocytes within a fine
reticular stroma [1]. The lymphatic system consists of a network of vessels,
termed lymphatics, lymph nodes and lymphoid organs. It is an important part of
the circulatory and immune systems and plays an important role in homeostasis
by controlling extracellular fluid volume and combating infection [2]. For
example, the lymphatic system can drain the accumulated fluid to restore
normal fluid circulation if the body tissue swells in injury.

Malignant lymphoma, also known as lymph cancer, is a malignant tumor that
originates in the lymph nodes and tissues of organs, which is caused by the
malignant proliferation of lymphocytes and tissue cells. According to the
different pathological characteristics, malignant lymphoma includes two
categories: Hodgkin’s lymphoma (HL) and Non-Hodgkin’s lymphoma (NHL). HL
accounts for about 10$\%$ of all lymphomas, and NHL accounts for 90$\%$ of the
remaining lymphomas [3]. According to World Health Organization’s (WHO), HL
includes two categories: classic Hodgkin’s lymphoma (CHL) and nodular
lymphocyte-predominant Hodgkin’s lymphoma (NLPHL). HL is one of the most
common lymphomas in the Western world, with approximately 3 cases per 100,000
persons every year [4]. In 2015, there were approximately 9,050 new HL cases
in the United States [5]. The exact cause of HL is unclear, but the factors
that increase the risk of HL include exposure to viral infection, family
factors, and immunosuppression [6]. Nevertheless, HL is a curable malignant
tumor [7], today’s medical technology has made great progress, soaring with
the possibility of a cure for patients with HL. Currently, over 80$\%$ of
patients newly diagnosed with HL can be cured [6]. NHL is a group of
heterogeneous malignant tumors of the lymphatic system. NHL results in many
deaths worldwide, and its incidence has been rising. According to the WHO
classification of blood and lymphoid tumors, NHL includes B-cell and T-cell
tumors. B-cell lymphoma accounts for about 90$\%$ of all lymphomas [8]. In the
United States, among new cancers, NHL accounts for 5$\%$ of men, and 4$\%$ of
women each year, and causes 5$\%$ of all cancer deaths [9]. Although some
cases are related to immunodeficiency, autoimmunity, or viral infection, the
causes of NHL are unclear in most cases [10].

Although some of the causes of lymphoma as mentioned above are not clear,
pathologists and doctors have not given up exploring the causes of lymphoma.
Traditionally, lymphoma is diagnosed by surgical excision biopsy [11]. Biopsy
refers to the use of local excision [12], forceps [13], fine-needle aspiration
[14], scraping [15] to obtain diseased tissue from the living body for
pathological diagnosis. Besides, due to the symptoms of some lymphatic tissue
diseases are not typical, and imageology lacks specificity, a biopsy is
frequently required for clinical diagnosis. The diagnosis of NHL is most
easily established by examining the tissue obtained by open biopsy of the
involved site [16], and the initial diagnosis of HL can only be made by biopsy
[17]. For example, lymphatic mapping with sentinel lymph node biopsy is widely
used to reduce complications associated with axillary lymph node dissection in
low-risk breast cancer patients [18]. Diffuse large B-cell lymphoma (DLBCL) is
diagnosed ideally by biopsy of abnormally enlarged and suspected lymph nodes
during clinical examination and radiographic imaging [19].

In addition to the biopsy mentioned above, the commonly used methods in the
diagnosis of lymphoma include histopathological examination and
immunohistochemistry (IHC). In histopathological observation, the removed
diseased tissues are fixed with formalin solution and embedded in paraffin to
make tissue sections [20]. And then after different staining, the tissues
sections are observed with an optical microscope. By analyzing and
synthesizing the characteristics of the diseased tissue, the pathological
diagnosis of the disease can be made. The most commonly used staining method
for tissue sections is hematoxylin-eosin (H$\&$E) staining, which is usually
the only staining method used in lymph node diagnosis [21]. Moreover, most
H$\&$E stained tissue sections can be used for initial analysis [22], the
pathological evaluation of sentinel nodes also includes H$\&$E staining [23].
H$\&$E staining can help detect cells that are not typical, and determine
their histological affiliation [24]. Currently, the intraoperative
histopathological examination of sentinel lymph nodes is mainly based on
frozen sections stained with H$\&$E [25]. If the histopathological observation
failed to make a diagnosis or requires further research, observation
techniques such as IHC can be assisted. IHC is very helpful to confirm the
diagnosis with a high sensitivity [24] [26]. The presence of lymph node
metastasis is one of the most important prognostic factors and treatment
options for many kinds of cancer [27]. However, sometimes micrometastasis in
lymph nodes cannot be detected only based on H$\&$E staining, and can only be
detected after IHC investigation on paraffin-embedded tissues [24] [28]. For
example, in [29], 20$\%$ of breast cancer patients have occult lymph node
metastasis detected by IHC, and only 7$\%$ have been detected by H$\&$E
staining. Therefore, the laboratory should control the quality of reagents and
equipment to ensure that histopathological observations such as H$\&$E
staining and IHC can make a correct diagnosis of lymphoma.

To make the above content more intuitive, we can observe the acquisition
process of the histopathological image in Fig. 1. First, a biological sample
is taken from a living body. Then, the biopsy is fixed to avoid chemical
changes in the tissue [30]. After that, the tissue is cut into sections to be
placed on a glass slide for staining. Then the stained tissue is covered on
the slide with a thin piece of plastic or glass, which can protect the tissue
and facilitate observation under a microscope. Finally, the slide is observed
and digitized with a microscope.

Figure 1: The process of histopathology image acquisition. ((a)-(f) are taken
from http://library.med.utah.edu/WebPath/HISTHTML/HISTOTCH/HISTOTCH.html. (g)
corresponds to Fig.1 (a) in [31])

In this paragraph, we introduce the morphological features of some typical
lymphomas. Because every type of lymphoma has a characteristic morphology, so
many lymphomas can be accurately subcategorized based on morphological
characteristics [32]. Pathologist’s morphological analysis (i.e., tissue
structure, cell shape, and staining characteristics) is still the gold
standard for the current routine lymph node histopathological staging and
grading [33] [34]. Moreover, because the prognosis of patients with lymphoma
varies greatly, the morphology of lymphoma is the main determinant of
treatment outcome and prognosis [35]. In NHL, follicular lymphoma (FL) is the
common type of B-cell lymphoma (As shown in Fig. 2 (a)). FL cells usually show
a distinct follicular-like growth pattern, with follicles similar in size and
shape, with unclear boundaries. Tumorous follicles are mainly composed of
centrocyte (CC) and centroblast (CB) in different proportions. The CC of FL is
small in size and medium in large, with irregular nuclei and inconspicuous
nucleoli. The CB has a large volume, 2 to 3 times larger than normal
lymphocytes. DLBCL is a diffusely hyperplastic large B cell malignant tumor.
The cell morphology of DLBCL is diverse, the nucleus is round or oval, with
single or multiple nucleoli (As shown in Fig. 2 (b)). In HL, NLPHL presents a
vaguely large nodular shape, and the background structure is a large spherical
network composed of follicular dendritic cells (As shown in Fig. 2 (c)).

Figure 2: Pathological pictures of three types of lymphomas (H$\&$E stain).
(a) corresponds to Fig. 6.67 (b) in [36]. (b) corresponds to Fig. 6.61 in
[37]. (c) corresponds to Fig. 3.1 in [38].

Early detection of lymphoma can greatly increase the possibility of cure, so a
fast and reliable way is needed for early detection and screening. Cytological
and histological techniques are generally used to examine biopsy tissue [39].
However, these techniques have several limitations. For instance, they have a
high subjectivity, because the result of diagnosis depends on the
pathologists’ opinion. Therefore, different pathologists may have different
diagnostic results when examining the same tissue sample [40]. In addition,
the examination process for biopsy tissue is very time-consuming [41]. At the
same time, problems with the preparation of slides and differences in staining
can also cause pathologists to make mistakes when analyzing diseases. For
example, lymph node metastasis is one of the most important factors for the
prognosis of breast cancer. The status of regional lymph nodes is generally
assessed through sentinel lymph node (SLN) surgery. The SLN is removed,
histopathologically processed and examined by a pathologist. This cumbersome
examination process is likely to cause small metastases to be missed [42]
[43]. Due to the increase in the incidence of lymphoma and the treatment of
specific patients, the diagnosis and grading of lymphoma have become more and
more complicated [44]. Nowadays, pathologists must examine a large number of
slides, and their histopathological cancer diagnosis workload and complexity
have increased dramatically. Finally, the more serious problem is that the
number of pathologists has decreased a lot in the past ten years [45]. The
author of [46] estimates that the number of pathologists in the United States
will decrease from 5.7 to 3.7 per 100,000 in the next 20 years. Therefore,
histopathological examination technology and biopsy technology for lymphoma
urgently need to be improved. This is not only the liberation of the
pathologist’s hands and eyes but also the responsibility for the patient.

###  1.2 The Development of Machine Vision in the Diagnosis and Treatment of
Lymphoma

Many kinds of diseases can be evaluated and diagnosed by analyzing cells and
tissues. As mentioned above, most of the analysis is performed manually by
pathologists and the accuracy of the results largely depends on the
proficiency of pathologists. Besides, with the increasing workload and limited
number of pathologists, _Computer-aided Diagnosis_ (CAD) systems become a
major research subject [47]. A CAD system generally consists of three parts
[48]. First, CAD systems segment the object from the background tissue.
Second, CAD systems extract features related to the classification. Finally, a
diagnosis result based on the extracted features is obtained from a
classifier. In recent years, CAD has become a major research direction of
medical image analysis. Usually, the medical images that require computer
assistance are histopathological slides. Compared with pathologists, the
analysis of diseases by CAD systems is performed quickly, and usually, only
one sample image is needed to get accurate results [49]. With CAD systems,
pathological images can be automatically classified. This can enable doctors
to be more efficient in diagnosing diseases, and the diagnosis results can be
more accurate and objective [50] [51]. Today, CAD is used in many fields of
medicine. For example, CAD systems play a very important role in the early
detection of breast cancer [52], lung cancer diagnosis [53], and arrhythmia
detection [54]. In [55], a CAD system is mainly used in cancer diagnosis and
prognosis. [56] describes a CAD system that can classify stromal development
and neuroblastoma grades.

A CAD system is composed of image preprocessing, segmentation, feature
extraction, feature reduction, detection and classification, and post-
processing modules (as shown in Fig. 3).

Figure 3: Computer-assisted diagnostic-system flowchart.

In CAD systems, MV is an important research direction. For example, in [57], a
CAD system based on deep learning is developed to locate and differentiate
metastatic lymph nodes of thyroid cancer using ultrasound images, a thriving
example of MV as a high-sensitivity screening tool. In recent years, MV has
developed rapidly and has received much attention, and it also represents the
most cutting-edge research direction of deep learning. For example, from the
earliest ImageNet large-scale visual recognition competition (ILSVRC) [58] to
the AlphaGo man-machine battles [59], we have seen the powerful vitality and
potential of computers in the field of vision processing. These superhuman
performances and results can be achieved because of the development and
improvement of deep learning and its deep neural network technology [60]. It
can be seen that deep learning is driving the development of MV and artificial
intelligence (AI) at high speed, and it is also changing our lives. At
present, MV has been used in agriculture [61], medicine [62], military fields
[63] and various industrial applications. The current trends in the
application of MV in the diagnosis and treatment of lymphoma are roughly shown
in Fig. 4.

Figure 4: The development trend of the applications of MV in the diagnosis and
treatment of lymphoma. The horizontal axis represents the year, and the
vertical axis represents the number of studies.

As shown in Fig. 4, the applications of MV in the diagnosis and treatment of
lymphoma mainly include segmentation, classification, and detection. Besides,
the number of applications as well as tasks has increased, such as
quantitative analysis [64], analysis of characteristics of lymphoma [65] and
so on.

###  1.3 Motivation of this Review

This paper comprehensively reviews the MV methods for lymphatic histopathology
image analysis. The motivation of this study is to study the popular
technologies and trends of MV and to explore the future potential of MV in the
diagnosis and treatment of lymphoma. In the course of our work, we found that
some survey papers are related to the applications of MV in this area.

The survey of[66] in 2004 reviews various image analysis methods which involve
pathological image segmentation, cell detection and classification tasks.
However, only one reference in this paper mentions the methods of extracting
features for lymphoma classification.

The survey of[67] in 2006 reviews the machine learning (ML) methods used in
cancer prediction and prognosis. The author summarizes the percentage of
papers using different ML methods in predicting lymphoma in a histogram. These
ML methods include artificial neural networks (ANN), support vector machines
(SVM), clustering, and so on. The paper does not review the applications of MV
in lymphatic histopathology image analysis (LHIA).

The survey of[68] in 2008 summarizes the current techniques for content-based
medical image retrieval. The author mainly reviews retrieval techniques based
on features such as color, texture, shape, and spatial relationship. However,
only one reference in this paper mentions that the shape of various nucleus
segmented from original leukocyte images can be characterized by Fourier
descriptors for efficient retrieval of chronic lymphocytic leukemia (CLL),
follicular center cell lymphoma and mantle cell lymphoma (MCL).

The survey of[69] in 2009 reviews the latest developments in CAD technology
for digital histopathology including histopathological image preprocessing,
feature extraction and selection, image detection and segmentation. This paper
reviews about 120 references, but only one summarizes in detail that combining
statistical texture features with low-level color texture features can improve
the classification performance of FL.

The survey of[70] in 2012 reviews the segmentation and classification methods
in histological images. More than 34 papers have been summarized, but only 4
papers are related to lymphoma.

The survey of[71] in 2012 summarizes the methods of histopathological images
including pre-processing, segmentation, feature extraction and dimensionality
reduction, and post-processing. However, the paper does not summarize the
processing methods in LHIA, only one example in the paper is related to FL
grading.

The survey of[72] in 2013 reviews the applications of content-based medical
image retrieval technology in 2D images, multi-dimensional images and medical
multi-modal images. But the images discussed in this paper are not the
histopathological images that we are interested in, and only 1 reference is
related to lymph nodes.

The survey of[73] in 2013 reviews the development trends of various cell
nucleus detection, segmentation, feature calculation and classification
techniques for histopathological images in cancer detection and grading.
However, the paper only shows lymphocyte nuclei in one picture. More than 100
papers are summarized in this paper, but only 2 papers related to lymphoma are
summarized.

The survey of[74] in 2014 reviews the current techniques and applications for
describing the visual content of histopathological images. An overview of the
process of obtaining histopathological images is described, including biopsy,
fixation, sectioning, cutting, staining, coverslipping and digitalization.
This article also summarizes the steps of automatic histopathology image
analysis including image preprocessing, feature extraction, and pattern
recognition using machine learning methods. At the same time, this paper also
reviews in detail four different feature extraction schemes and the
applications of automatic histopathology image analysis in four different
medical fields. But among the more than 70 references in the paper, only 5 are
related to lymphoma.

The survey of[75] in 2016 summarizes the applications of deep learning
technology in medical images. An overview of the applications of convolutional
neural networks (CNNs) and other deep learning methods in medical tasks is
presented. These tasks include lesion detection, image segmentation, shape
modeling, and image registration. However, only 3 of the more than 30
references in the paper are related to lymph node detection and the images
used are CT scan images.

The survey of[76] in 2016 reviews the developments of computer image analysis
tools used in predictive modeling of digital pathology images through
detection, feature extraction, segmentation and tissue classification. This
paper summarizes more than 20 references, only 2 are related to lymph.

The survey of[77] in 2017 reviews various image preprocessing, segmentation,
feature extraction, feature reduction, and classification techniques used in
histopathological image analysis. More than 150 references are summarized, but
only 11 are related to lymphoma.

The survey of[78] in 2017 reviews the tasks of deep learning technology in
medical image analysis in different fields. These tasks mainly include
detection, segmentation, classification, retrieval, registration, image
generation and enhancement. However, only 11 are related to lymphoma among
more than 350 references in the paper.

The survey of[79] in 2017 reviews the ML algorithms used in medical image
analysis, especially CNNs. The important research fields and applications of
medical image segmentation, detection, localization, classification and
registration are covered in the paper. However, only 1 is related to the
detection of lymph node enlargement among more than 120 references in the
paper, and the images used are CT slices that we are not interested in.

The survey of[80] in 2018 reviews the applications of AI and deep learning
technology in the diagnosis of breast pathology, as well as other recent
developments in digital image analysis. However, only 8 are related to lymph
among more than 180 references in the paper.

The survey of[81] in 2019 reviews the applications of AI in digital pathology,
including tumor diagnosis and prognosis, and hand-crafted feature-based ML
methods. At the same time, this paper also summarizes the applications of
several deep neural networks to the segmentation, detection, and
classification of pathological images. These Deep neural network approaches
include CNNs, fully convolutional networks (FCN), recurrent neural networks
(RNN), generative adversarial networks (GAN). However, only 14 are related to
lymph among more than 180 references in the paper. The survey of[82] in 2019
reviews the development and current status of deep learning in cancer
diagnosis involving radiology and pathology. However, only 5 are related to
lymph among about 90 references in the paper.

The survey of[83] in 2020 reviews the application of ANNs in breast
histopathology images, including image classification, segmentation, and
detection. However, only 3 are related to lymph among more than 150 references
in the paper.

According to the existing survey papers described above, we can find that none
of the survey papers contributed to a comprehensive summary of the
applications of MV in LHIA. Therefore, this study summarizes more than 170
works from 1999 to 2020 to fill the gap of MV in this field. The works
summarized in this paper are scoured from arXiv, IEEE, Springer, Elsevier, and
so on. The screening process of the whole work is shown in Fig. 5. Twenty-six
keywords are used to search for papers we are interested in, and a total of
2,175 papers are collected. After two rounds of screening, we retain 171
papers, including 135 papers on lymphoma histopathology image analysis and 36
survey papers.

Figure 5: The flowchart of the paper searching and screening process.

###  1.4 Structure of this Review

The content introduced in each section of this paper is as follows: Sec. 2
introduces commonly used datasets and evaluation methods. Sec.3 introduces
commonly used preprocessing methods, and Sec.4 introduces commonly used image
segmentation methods. Sec.5, Sec.6, Sec.7 introduce the use of CAD technology
for feature extraction, classification, and detection methods. Sec.8 analyses
commonly used methods and potential methods, and finally Sec.9 summarizes this
article and made further prospects for future work.

##  2 Datasets and Evaluation Methods

In the applications of MV in analysing histopathological images of lymphoma,
we have discussed some publicly available datasets that are frequently used.
Besides, we listed evaluation metrics used in segmentation, classification,
and detection tasks.

###  2.1 Publicly Available Datasets

Four publicly available datasets are discussed in this section; among them,
three of them are Camelyon16 [84], Camelyon17 [84], IICUB-2008 [85]. and the
fourth dataset is collected from the National Cancer Institute (NCI) [86] and
the National Institute on Aging (NIA) [86] and the National Institute on Aging
(NIA) [87]. Table. 1 shows the basic information of the datasets.

Table 1: available lymph histopathology image datasets. Datasets | Year | Staining | Task | Number of images
---|---|---|---|---
Camelyon16 | 2016 | H$\&$E | Detection |  | 270 images for training,
---
130 images for testing
Camelyon17 | 2017 | H$\&$E |  | Detection and
---
classification
| 500 images for training,
---
500 images for testing
IICBU-2008 | 2008 | H$\&$E | Classification | 113 CLL, 139 FL, 122 MCL
the NCI and NIA | $\backslash$ | H$\&$E | Classification | 12 CLL, 62 FL, 99 MCL

The Camelyon16 challenge is organized by International Symposium on Biomedical
Imaging (ISBI) in 2016. The purpose of Camelyon16 is to advance the algorithms
for automated detection of cancer metastases in H$\&$E stained lymph node
images. This competition has a high clinical significance in reducing the
workload of pathologists and the subjectivity in the diagnosis. The data used
in Camelyon16 consists of 400 Whole-slide images (WSIs) of SLN, which are
collected in Radboud University Medical Center (Radboudumc) (Nijmegen, the
Netherlands), and the University Medical Center Utrecht (Utrecht, the
Netherlands). The training dataset includes 270 WSIs, and the test dataset
includes 130 WSIs. In Fig. 6, (a), (b) and (c) are WSIs at low, medium and
high resolution respectively [84].

Figure 6: The WSIs in Camelyon16 dataset.

The Camelyon17 is the second challenge launched by the Diagnostic Image
Analysis Group (DIAG) and Department of Pathology in Radboudumc. The purpose
of Camelyon17 is to advance the algorithms for automated detection and
classification of cancer metastases in H$\&$E stained lymph node images. Like
Camelyon 16, Camalyon 17 also has a high clinical significance. The training
dataset includes 500 WSIs, and the test dataset includes 500 WSIs (as shown in
Fig. 6).

The IICBU-2008 dataset is proposed by [85] in 2008 to provide free access of
biological image datasets for computer experts, it includes nine small
datasets containing different organs or cells. Three types of malignant
lymphoma are proposed in the dataset containing lymphoma. There are 113 CLL
images, 139 FL images and 122 MCL images (as shown in Fig. 7). The images are
stained with H$\&$E and usually used for classification such as [88] [89].

Figure 7: The histopathological images in IICBU-2008 dataset.

The dataset used in [90] and [91] is from the NCI and NIA, contains 173
histological NHL images which are comprised of 12 CLL images, 62 FL images and
99 MCL images (as shown in Fig.8).

Figure 8: The histopathological images in the NCI and NIA dataset.

###  2.2 Evaluation Method

This section introduces some commonly used evaluation metrics in the
classification, segmentation, and detection tasks of lymphoma histopathology
images.

####  2.2.1 Some Basic Metrics

Here, we presented several commonly used evaluation metrics. Suppose we want
to classify two types of samples, recorded as positive and negative,
respectively: True positive (TP), which is the number of actually positive
samples classified as positive by the classifier. False-positive (FP) is the
number of negative samples classified as positive by the classifier. False-
negative (FN) is the number of samples that are actually positive but
classified as negative by the classifier. True negative (TN) is the number of
negative samples classified as negative by the classifier. Table. 2 shows the
confusion matrix of the above four indexes. The value in the diagonal of the
confusion matrix is the number of correct classifications for each category.

Table 2: Confusion Matrix. P represents positive, N represents negative. | Classification Result
---|---
Actual Result |  | P | N
P | TP | FN
N | FP | TN

####  2.2.2 Evaluation Criteria for Classification models

In classification tasks, when a classifier is trained with enough training
samples, the next step is to provide test samples to check whether the test
samples are correctly classified. Then, a method to evaluate and quantify the
results of this classification is necessary. Table. 3 shows the commonly used
evaluation metrics in classification tasks.

Accuracy (ACC) is the most commonly used evaluation metric, which is the
proportion of the number of correctly classified samples among all samples. A
well-performing classifier generally has a high ACC. Precision indicates the
proportion of samples classified as positive that are actually positive.
Sensitivity (SEN) represents the proportion of all positive examples that are
correctly classified and measures the classifier’s ability to recognize
positive examples. Specificity (SPE) represents the proportion of all negative
examples that are correctly classified and measures the classifier’s ability
to recognize negative examples. SEN and SPE are also called true positive
rates (TPR) and true negative rates (TNR). Recall measures how many positive
samples are classified as positive. It can be seen that a Recall is equal to
SEN in Table 3. F1-score is often used as the final evaluation method in some
multi-classification tasks. It is the harmonic average of precision and
recall. In addition, Receiver Operating Characteristic Curve (ROC curve) can
also be used to evaluate the quality of a classifier. TPR is used as the
ordinate in the ROC curve, and one minus TNR is also called false positive
rate (FPR) as the abscissa. The closer the ROC curve is to the upper left
corner, the better the classifier performance. The classifier can also be
evaluated by the area under the ROC curve (Area Under Curve, AUC). The larger
the AUC, the more reliable the classifier model. In addition, some metrics
such as positive predictive value (PPV) and negative predictive value (NPV)
are also used in [92]. PPV represents the proportion of true positive samples
among all samples classified as positive and is given as TP/(TP+FP). NPV
represents the proportion of true negative samples among all samples
classified as negative and is given as TN/(TN+FN).

Table 3: Evaluation metrics for classification tasks. Assessments | Formula
---|---
ACC | $\frac{TP+TN}{TP+TN+FP+FN}$
Precision | $\frac{TP}{TP+FP}$
SEN | $\frac{TP}{TP+FN}$
SPE | $\frac{TN}{TN+FP}$
Recall | $\frac{TP}{TP+FN}$
F1-score | $\frac{2TP}{2TP+FN+FP}$

####  2.2.3 Evaluation Criteria for Segmentation Methods

Segmentation can be used to detect the region of interest (ROI) in
histopathological images. In tasks of histopathology image analysis, ROIs
include tissue components such as lymphocytes and cell nuclei. Segmentation is
the process to divide ROIs from the tissue. Since segmentation is generally
used to prepare for subsequent detection and classification, so it is
necessary to evaluate the segmentation result by appropriate metrics.

A segmentation algorithm is often measured by detection accuracy and
segmentation accuracy. Detection accuracy is used to measure the ability of
the segmentation algorithm in identifying ROIs, segmentation accuracy is used
to measure the similarity between the ROIs obtained in the segmentation
algorithm and the regions in the ground truth images [77]. The detection
accuracy metrics commonly used in segmentation are discussed in Sec. 2.2.2
such as Precision, Recall, F-measure (namely F1-score). The segmentation
accuracy metrics commonly used in segmentation include Dice coefficient
(DICE), Hausdorff distance (HD), mean absolute distance (MAD) [93], Jaccard
index (JAC) [94] and so on. The following are the formulas for the above
metrics. $S_{g}$ is a set of points ($g1,g2,...,gn$) constituting the
automatically segmented contour, $S_{t}$ is a set of points ($t1,t2,...,tn$)
constituting the ground truth contour.

1) DICE represents the ratio of the area where the two regions intersect to
the total area and is given by

${\rm DICE}=\frac{2|S_{g}\cap S_{t}|}{|S_{g}|+|S_{t}|}$ (1)

2) HD between $S_{g}$ and $S_{t}$ is the maximum among the minimum distance
computed from each point of $S_{g}$ to each point of $S_{t}$. HD is defined by

${\rm HD}={\rm max}(h(S_{g},S_{t}),h(S_{t},S_{g}))$ (2)

$h(S_{g},S_{t})=\mathop{{\rm max}}_{g\in S_{g}}\mathop{{\rm max}}_{t\in
S_{t}}\parallel g-t\parallel$ (3)

3) MAD between $S_{g}$ and $S_{t}$ is the mean of the minimum distance
computed from each point of $S_{g}$ to each point of $S_{t}$. MAD is defined
by

${\rm MAD}=\frac{1}{n}{\rm\Sigma}_{g=1}^{n}\\{{\rm min}_{t\in
S_{t}}\\{h(g,t)\\}\\}$ (4)

4) JAC represents the intersection of two regions divided by their union and
is defined by

${\rm JAC}=\frac{|S_{g}\cap S_{t}|}{|S_{g}\cup S_{t}|}$ (5)

####  2.2.4 Evaluation Criteria for Detection Methods

Because most of the detection tasks are completed by classification, many
metrics in the classification tasks can also be used to evaluate the results
of the detection tasks. Such as ACC, Confusion matrix, Precision, Recall, ROC
curve, AUC, SEN, SPE in Sec. 2.2.2. In addition, free response operating
characteristic (FROC) is used to evaluate the result in [95], dice similarity
coefficient (DSC) (as shown in Eq. 6) is used to evaluate the result in [96]
and AI (Artificial Intelligence) score is used to evaluate the probability of
lymph node metastasis in [97].

${\rm DSC}=\frac{2{\rm TP}}{2{\rm TP}+{\rm FN}+{\rm FP}}$ (6)

###  2.3 Summary

In summary, we introduce the datasets commonly used in tasks of LHIA, among
which the commonly used data sets are Camelyon16, Camelyon17 and IICBU-2008.
Next, we summarize the evaluation metrics commonly used in classification,
segmentation and detection tasks. We find that the commonly used metrics in
classification and detection tasks are ACC, SEN, SPE, AUC. DICE and HD are the
commonly used metrics in segmentation tasks.

##  3 Image Preprocessing

In order to obtain the expected results in tasks of LHIA, the images must be
in good quality. Due to the difference in staining, the histopathological
images obtained from the biopsy sample may show uneven staining and blurring
cases [77]. In addition, the images can be simplified by removing irrelevant
information and highlighting useful information for tasks such as feature
extraction, segmentation and detection can be carried out smoothly. Therefore,
the images need to be preprocessed by resizing, denoising, and normalization
methods. In this section, we introduce the preprocessing methods commonly used
in tasks of LHIA, which are based on color, filter, threshold, morphology,
histogram and other methods.

###  3.1 Color-based Preprocessing Techniques

At present, some CAD methods in tasks of LHIA convert images from RGB color
space to other color spaces. Since the histopathological images show multiple
colors, the RGB color space is not enough to describe these images. Therefore,
the RGB images need to be converted to other color spaces to get better
quantification and description of colors. In addition to the conversion of
various color spaces, color-based preprocessing methods also include image
grayscale, color deconvolution, color normalization, color equalization and
enhancement, and extraction of mean value or maximum and minimum of R, G and B
channel. The commonly used color-based preprocessing methods on LHIA as shown
in Table. 4.

In [98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 49, 110, 111,
89, 65, 112], $L^{*}a^{*}b^{*}$ color space is used on LHIA for
$L^{*}a^{*}b^{*}$ allowing color changes to be compatible with differences in
visual perception. In [113, 114, 49, 110], $L^{*}u^{*}v^{*}$ color space is
used on LHIA for $L^{*}u^{*}v^{*}$ color space are uniform in perception. In
[115, 116, 117, 105, 118, 107, 119, 120, 121, 122, 123, 49, 124, 125, 110,
111], HSV color space is used on LHIA for HSV separating the intensity from
the color information. In [103], HSI color space is used on LHIA for HSI is
hue based color space similar to HSV. In [49, 110], Ycbcr color space is used
on LHIA. In [110], YIQ color space is used on LHIA. In [111], YUV color space
is used on LHIA. Ycbcr, YIQ and YUV are luminance based color spaces. In
[126], a new color space is defined which is called MDC (the most discriminant
color space). MDC is a linear combination of RGB color space, the texture
features extracted from MDC can separate classes optimally. In [104, 127,
128], color deconvolution is used to separate H and E overlapped channels into
individual channels. Fig. 9 shows an example of application in color
deconvolution. In [119, 129, 130, 131, 132, 111, 133], RGB images are
converted into grayscale images because the variations of color and intensity
may hamper the classification performance, grayscale features need to be
calculated, noise and small details in RGB images need to be removed and
suppressed and the data in the individual channel is easier to process after
image grayscale. In [134, 135], the color saturation of the original image has
increased after color equalization and enhancement, color augmentation is used
to reduce the effect of outlying colors. In [110, 91, 135, 128], color
normalization is used to improve the quality of images. In [136, 137, 138],
the R channel is extracted from RGB color space because the R channel has the
greatest contrast compared to the image background.

Table 4: Color-based Preprocessing Techniques. Reference | Year | Team | Details
---|---|---|---
[113] | 2005 | W Chen |  RGB to $L^{*}u^{*}v^{*}$
[98] | 2006 | J Angulo |  RGB to $L^{*}a^{*}b^{*}$
[99] | 2008 | O Sertel |  RGB to $L^{*}a^{*}b^{*}$
[115] | 2008 | A Basavanhally | RGB to HSV
[116] | 2008 | O Sertel | RGB to HSV
[100] | 2009 | O Sertel |  RGB to $L^{*}a^{*}b^{*}$
[101] | 2009 | AN Basavanhally |  RGB to $L^{*}a^{*}b^{*}$
[102] | 2009 | K Belkacem-Boussaid |  RGB to $L^{*}a^{*}b^{*}$
[103] | 2010 | K Belkacem-Boussaid |  RGB to $L^{*}a^{*}b^{*}$, HSI
[114] | 2010 | O Sertel |  RGB to $L^{*}u^{*}v^{*}$
[104] | 2010 | NV Orlov |  |  RGB to $L^{*}a^{*}b^{*}$,
---
color deconvolution
[139] | 2010 | O Sertel |  | RGB to a 1-D unitone image and further
---
normalize the unitone image to the [0,1] range
[117] | 2010 | S Samsi | RGB to HSV
[105] | 2010 | K Belkacem-Boussaid |  RGB to HSV, $L^{*}a^{*}b^{*}$
[118] | 2010 | J Han | RGB to HSV
[106] | 2010 | K Belkacem-Boussaid |  RGB to $L^{*}a^{*}b^{*}$
[126] | 2011 | H Kong | RGB to MDC
[107] | 2012 | HC Akakin |  RGB to HSV, $L^{*}a^{*}b^{*}$
[119] | 2012 | S Samsi | RGB to grayscale, HSV
[108] | 2012 | B Oztan |  RGB to $L^{*}a^{*}b^{*}$
[129] | 2013 | E Acar | RGB to grayscale
[120] | 2014 | T Ishikawa | RGB to HSV
[130] | 2014 | E Michail | RGB to grayscale
[121] | 2014 | YL Kuo | RGB to HSV
[131] | 2014 | K Dimitropoulos | RGB to grayscale
[122] | 2015 | MD Zarella | RGB to HSV
[123] | 2015 | MFA Fauzi | RGB to HSV
[49] | 2015 | C Di Ruberto |  RGB to HSV, $L^{*}a^{*}b^{*}$, $L^{*}u^{*}v^{*}$, Ycbcr
[140] | 2016 | J Chen | Stain normalization
[124] | 2016 | D Wang | RGB to HSV
[125] | 2016 | R Chen | RGB to HSV
[134] | 2016 | N Codella |  | Color equalization and enhancement,
---
relative color enhancement
[110] | 2017 | TAA Tosta |  |  RGB to HSV, $L^{*}a^{*}b^{*}$, $L^{*}u^{*}v^{*}$, Ycbcr, YIQ.
---
color normalization
[136] | 2017 | TAA Tosta | Extract the R channel from RGB model
[127] | 2017 | BB Cheikh | Color deconvolution
[132] | 2018 | H Jiang | RGB to grayscale
[91] | 2018 | MG Ribeiro | Color normalization
[137] | 2018 | TAA Tosta | Extract the R channel from RGB model
[111] | 2019 | H Zhu |  RGB to HSV, $L^{*}a^{*}b^{*}$, YUV, grayscale
[89] | 2019 | J Bai |  RGB to the blue ratio gray space, $L^{*}a^{*}b^{*}$
[65] | 2019 | AS Martins |  RGB to $L^{*}a^{*}b^{*}$
[133] | 2019 | P Bándi | RGB to grayscale
[135] | 2020 | N Hashimoto | Color normalization, color augmentation
[128] | 2020 | F Bianconi | Color deconvolution, color normalization
[112] | 2021 | AS Martins |  RGB to $L^{*}a^{*}b^{*}$
[138] | 2021 | TA Azevedo Tosta | Extract the R channel from RGB model

Figure 9: (a) MCL lymphoma sample stained with H$\&$E. (b) H channel. (c) E
channel. This figure corresponds to Fig. 4 in [104].

###  3.2 Filter-based Preprocessing Techniques

Filtering can smooth images and remove uninteresting parts of the images such
as noise, artifacts, and so on. The commonly used filter-based preprocessing
methods are shown in Table. 5.

In [117, 141, 125, 64, 142, 143, 133], a median filter is used to smooth
texture variations, remove artifacts and noise, homogenize color intensity. An
example using median filter is shown in Fig. 10. In [118, 144, 145, 130, 131,
64, 146, 110, 136, 137, 138], Gaussian filter is used to remove noise and
small details, convolve input images, smooth images, homogenize color
intensity, adjust contrast. In [106], a matching filter is used for removing
noise, flattening background and enhancing the contours of the follicle region
to homogenize images. In [134], region filter is used to remove areas with a
surface area that is less than 10 pixels. In [147], the 2D order-statistics
filter is used to remove noise and discrete points.

Figure 10: The median filter in preprocessing. (a) and (b) corresponds to Fig. 1 (a) and Fig. 1 (d) respectively in [125]. Table 5: Filter-based Preprocessing Techniques. Reference | Year | Team | Details
---|---|---|---
[117] | 2010 | S Samsi | median filter
[118] | 2010 | J Han | Gaussian filters
[106] | 2010 | K Belkacem-Boussaid | matching filter
[141] | 2012 | M Oger | median filter
[144] | 2012 | A Schmitz | Gaussian filtering
[145] | 2013 | T Schäfer | Gaussian filter
[130] | 2014 | E Michail | Gaussian filtering
[131] | 2014 | K Dimitropoulos | Gaussian filtering
[125] | 2016 | R Chen | median filter
[64] | 2016 | P Shi | median filter, Gaussian filter
[134] | 2016 | N Codella | region filter
[44] | 2016 | G Litjens | for separating background from tissue
[142] | 2017 | T. Wollmann | median filter
[146] | 2017 | A ES Negm | Gaussian filter
[110] | 2017 | TAA Tosta | Gaussian filter
[136] | 2017 | TAA Tosta | Gaussian filter
[147] | 2018 | MZ do Nascimento | 2D order-statistics filter
[143] | 2018 | T Wollmann | median filter
[137] | 2018 | TAA Tosta | Gaussian filter
[133] | 2019 | P Bándi | median filter
[138] | 2021 | TA Azevedo Tosta | Gaussian filter

###  3.3 Threshold-based Preprocessing Techniques

Thresholding can separate the area of interest in an image from the
background. In RGB images, the brightnesses of R, G and B commonly are used to
be a threshold. Table. 6 shows the references using threshold-based
preprocessing techniques.

In [148, 149], the mean brightness of R component is selected to be a
threshold of the binary image. The pixels whose values are less than the
threshold are set to 1 and the rest is set to 0. In [103, 124, 125, 134, 150,
151, 95, 133], the Otsu threshold method is used to calculate the optimal
threshold of each channel in color space, binarize images, segment color
channels, remove the background regions of images. The detection result is
shown in Fig. 11 after the preprocessing including conversion of color space
and Otsu thresholding, the tissue regions are surrounded by a green curve. In
[144, 145, 120, 123, 146, 152], a threshold is selected to separate the
background area and tissue area to highlight the tissue area. In [153], the
intensity threshold is used to remove non-candidate regions for nucleoli are
dark and non-candidate regions are bright.

Figure 11: Visualization of tissue area detection during preprocessing. The figure corresponds to Figure 1 in [124]. Table 6: Threshold-based Preprocessing Techniques. Reference | Year | Team | Details
---|---|---|---
[148] | 2007 | M Zorman |  | The mean brightness value of the R
---
component of the RGB image is used as the threshold
[103] | 2010 | K Belkacem-Boussaid | Otsu
[149] | 2011 | M Zorman |  | The mean brightness value of the R
---
component of the RGB image is used as the threshold
[144] | 2012 | A Schmitz | Background thresholding
[145] | 2013 | T Schäfer | a Threshold for background elimination
[120] | 2014 | T Ishikawa |  | Extracting background area by
---
binarization with threshold 30
[153] | 2014 | E Michail | Intensity thresholding
[121] | 2014 | YL Kuo | The threshold is used to create the binary image
[131] | 2014 | K Dimitropoulos |  | A threshold (empirically set to 0.37) on
---
the normalized R component of the RGB image
[123] | 2015 | MFA Fauzi | A local thresholding operation
[124] | 2016 | D Wang |  | Otsu is used to computed the optimal
---
threshold values in H, S, V channels
[125] | 2016 | R Chen |  | Otsu’s binarization is
---
performed on the S channel
[134] | 2016 | N Codella | Otsu thresholding technique
[150] | 2017 | K Xiao | Otsu is used to binarize the image
[142] | 2017 | T. Wollmann | Color thresholding
[146] | 2017 | A ES Negm | A threshold for background elimination
[127] | 2017 | BB Cheikh |  | Extracting nuclear objects
---
by image thresholding
[151] | 2018 | Y Li | Otsu is used to exclude the background
[147] | 2018 | MZ do Nascimento |  | Contrast limited adaptive
---
histogram equalisation (CLAHE)
[143] | 2018 | T Wollmann | Color thresholding
[95] | 2018 | H Lin | Otsu is used to determine the adaptive threshold
[152] | 2018 | A BenTaieb |  | Removing background pixels using a
---
threshold on the pixel intensity values
[133] | 2019 | P Bándi | Otsu’s adaptive threshold

###  3.4 Morphology-based Preprocessing Techniques

Morphological operations can smooth the image contour, suppress the noise,
fill in small holes and connect broken areas. There are four basic operations,
namely dilation, erosion, opening operation and closing operation. Table. 7
shows the references using morphology-based preprocessing techniques.

In [103, 93, 154, 133], opening and closing operations are used to recover the
shape of tissue, fill the holes in the segmented binary images. In [154, 153,
134, 150], dilation and erosion operators are used to connect edges, remove
noise and black holes. In [154], the preprocessing process includes removing
noise, smoothing images by using disc smooth operator, making edges continuous
by using canny edge detector, opening and closing operations, dilation
operator, filling connected components using hole filling method, normalizing
the maximum area of the connected area in images. Fig. 12 shows the results of
preprocessing.

Figure 12: The results of preprocessing. The figure corresponds to Fig. 5 in [154]. Table 7: Morphology-based Preprocessing Techniques. Reference | Year | Team | Details
---|---|---|---
[103] | 2010 | K Belkacem-Boussaid | Opening and closing
[93] | 2010 | J Cheng | Hole-filling algorithm, opening
[155] | 2010 | U Neuman | Removing small spots
[106] | 2010 | K Belkacem-Boussaid |  | Eliminating undesired areas
---
and filling holes
[154] | 2013 | BS Sandhya |  | Disc smooth operator, opening,
---
closing, dilation,filling holes
[153] | 2014 | E Michail | Dilation
[121] | 2014 | YL Kuo | Hasan hole filling algorithm
[134] | 2016 | N Codella | Disk dilation of 2 pixels
[150] | 2017 | K Xiao | Erosion, dilation
[127] | 2017 | BB Cheikh. | Removing small object
[133] | 2019 | P Bándi | Closing, opening, hole filling

###  3.5 Histogram-based Preprocessing Techniques

The histogram shows the total number of pixels in each gray level in the
images. Histogram equalization, histogram normalization and histogram
stretching are commonly used to enhance image contrast. Table. 8 shows the
references using histogram-based preprocessing techniques.

In [117, 126, 141, 130, 131, 134, 110, 136, 137, 156, 128, 138], histogram
equalization is used to increase the contrast of images, better identify the
tissue regions, enhance the differences between background and tissue,
normalize the color distribution of slides under different staining and
lighting conditions to obtain a uniformly distributed histogram. As shown in
the first row of Fig. 13, the histopathological images of FL show inconsistent
colors caused by inconsistent lighting conditions, the second row shows more
consistent images after histogram equalization. In [107], the co-occurrence
histogram is normalized to extract texture features. In [127], histogram
stretching is used to solve the problem of changes in hematoxylin
concentration after color deconvolution.

Figure 13: First row: the original images of FL. Second row: the images after histogram equalization. The figure corresponds to Fig. 1 in [126]. Table 8: Histogram-based Preprocessing Techniques. Reference | Year | Team | Details
---|---|---|---
[117] | 2010 | S Samsi | Histogram equalization
[126] | 2011 | H Kong | Histogram equalization
[141] | 2012 | M Oger | Adaptive equalization of histogram
[107] | 2012 | HC Akakin | Normalizing co-occurrence histograms
[130] | 2014 | E Michail | Histogram equalization
[131] | 2014 | K Dimitropoulos | Histogram equalization
[134] | 2016 | N Codella | Histogram equalization
[110] | 2017 | TAA Tosta | Histogram equalization
[136] | 2017 | TAA Tosta | Histogram equalization
[127] | 2017 | BB Cheikh | Histogram stretching
[137] | 2018 | TAA Tosta | Histogram equalization
[156] | 2019 | UV Somaratne | Histogram equalization
[128] | 2020 | F Bianconi | Histogram equalization
[138] | 2021 | TA Azevedo Tosta | Histogram equalization

###  3.6 Other Preprocessing Techniques

In addition to the above-mentioned preprocessing methods, some papers design
different image preprocessing methods in LHIA.

In [94], a method is used to segment and detect lymphocytes in HER2+ breast
cancer (BC) histopathology images. In this scheme, an EM-based (expectation
Maximization) segmentation provides a specific initialization for enhancing
the performance of active contours. Fig. 14 shows the results obtained by EM.

Figure 14: (a) Original HER2+ BC histopathology image with corresponding
(b)-(e) class binarized scenes. The figure corresponds to Fig. 3 in [94].

In [88] and [157], a framework is designed for histology images
classification. In the feature preparation component of the framework, each
image is divided into 25 blocks that are equal-sized (as shown in Fig. 15).
The purposes of the step include decreasing space complexity for feature
extraction, the local features extracted by blocks can improve the results
than the global features extracted without blocks and different features
useful for classification in different regions of the image can be extracted.
In [118, 120, 150, 158, 132, 152, 159, 96, 160, 161, 162, 163, 135], the
original images are cropped into smaller patches. In [158], 40,000 patches are
extracted from the training set because the process of eliminating black
borders, noise and white background area would increase the unnecessary time.
In [152], all sides around tissue are cropped for reducing the computational
amount. In [162], 2048 pixel $\times$ 2048 pixel sized patches are cropped in
the center of each annotation of WSI at a magnification of $\times$5, and 2048
pixel $\times$ 2848 pixel sized patches are cropped in the periphery of each
annotation of WSI at a magnification of $\times$20 and$\times$40 (as shown in
Fig. 16). Next, 128 pixel $\times$ 128 pixel sized patches are cropped from
2048 pixel $\times$ 2018 pixel sized patches at a magnification of $\times$5
during the training phase and test phase, and 64 pixel $\times$ 64 pixel sized
patches are cropped from 2048 pixel $\times$ 2018 pixel sized patches at
magnifications of $\times$20 and $\times$40 during the training phase and test
phase (as shown in Fig. 17).

Figure 15: Division of the image to 25 blocks. This figure corresponds to
Figure 2 in [88].

Figure 16: Procedure for creating image patches. The figure corresponds to
Fig. 1 in [162].

Figure 17: Preprocessing during the training and testing phase. The figure
corresponds to Fig. 2 in [162].

In [155], a method is designed for segmenting images of IHC stained FL tissue
sections. Color separation and rescaling are used in the step of
preprocessing. First, the brown in an image is separated by replacing all
pixels outside the defined range of brown with white pixels (as shown in Fig.
18 (b)). Second, the Gaussian pyramid is used for rescaling the image (as
shown in Fig. 18 (b), upper left, black box).

Figure 18: The preprocessing steps. The figure corresponds to Fig.3 in [155].

Since the contours of the images may be irregular, the Fourier shape
descriptors are used to smooth and underline better the contours in [106, 126,
164, 119].

In [106], the Gaussian filtering is firstly used to convolve the images. Next,
the pixels marked as background in the original image are removed and the
pixels marked as tissue are left by background thresholding. Finally, a region
labeling method is used to make the remaining pixels are connected. Fig. 19
shows the effects of the preprocessing. In [145] and [146], the preprocessing
methods are similar to [106]. Firstly, a weighted averaging filter with
weights according to a Gaussian distribution is used to smooth images.
Secondly, an intensity threshold obtained from the histogram of the grayscale
image is used to separate background points and object points, and the gray
scale image is generated by averaging the RGB values of each point. Finally, a
region-based segmentation is used to eliminate small isolated tissue areas and
connect the object points.

Figure 19: In Box 1, the background pixels are removed. In Box 2, tissue
fragments are eliminated. The figure corresponds to Figure. 2 in [106].

In [153], the preprocessing steps include: (1) The Gray-Level Run Length
algorithm and the intensity histogram of nucleus are used to encapsulate the
information that the CBs are of a higher non-uniformity than non-CBs. (2) The
intensity threshold is used to eliminate the areas without nucleus. (3)
Circular Hough Transform is used to images to detect nucleus. (4) Dilation is
used to keep the proximate area of the nucleus.

In [121], the preprocessing steps include: (1) Sigmoid contrast enhancement is
used to strengthen the contrast of the image. (2) The enhanced image is
converted to grayscale by using the V-component of HSV space transferred from
RGB space. (3) The threshold for creating the binary image is chosen by
extending the original Otsu’s method. (4) The holes in images after watershed
segmentation are filled by Hasan hole filling algorithm. Fig. 20 shows the
effect of preprocessing.

Figure 20: The effect of preprocessing in [121]. The figure corresponds to
Fig. 3 in [121].

In [165, 166, 167, 156, 96, 45, 168, 163, 135], data augmentation methods are
used to avoid issue of biases and overfitting. The methods include left-right
flip, rotation, perturbing color, adding jitter, scaling, shear, flipping
(both horizontal and vertical), mirror operation. In [163], the patches are
rotated by 0 and 90 degrees after cropping. In [135], the patches are rotated
to augment data when the number of patches in a WSI is less than 3,000.

###  3.7 Summary

In summary, we presented methods used in the preprocessing techniques of LHIA.
AS shown in Fig. 21, a total of 46 papers used colour-based preprocessing
methods, 23 papers used threshold-based methods, 11 papers used morphology-
based method, and 14 papers used histogram-based methods. The colour-based
methods are the most commonly used method in preprocessing of LHIA, and the
HSV colour space is the most commonly used. In addition, the images are
frequently cropped into smaller patches and data augmentation methods are also
commonly used.

Figure 21: Statistical graphs of four preprocessing methods.

##  4 Image Segmentation

In this section, we have summarized the segmentation tasks in LHIA.
Segmentation is a crucial step in image processing applications, which
identifies ROIs, including nuclei, glands, and lymphocytes. The ROIs include
follicular regions, lymphocytes, CBs, centrocytes, and neoplastic cells in
lymphoma images. It is essential to correlate accurate identification of ROIs
with pathologies [169].

###  4.1 Threshold-based Segmentation Methods

Image thresholding is a widely used segmentation technique that generates
binary images by selecting a reasonable threshold. The threshold is determined
by using the difference in grayscale characteristics between the ROIs and
background tissue in an image. The commonly used threshold-based segmentation
methods in LHIA include intensity thresholding, Otsu thresholding and adaptive
thresholding. Table. 9 shows the references using threshold-based segmentation
methods.

In[148], symbol-based ML method is used for segmenting FL images. The focus is
on identifying lymphomas by finding follicles from microscopy images of FL. In
the first stage, image preprocessing and feature extraction are implemented,
then different rough set methods are used for pixel classification in the
second stage. The mean brightness value of the R component of RGB image is
used as a threshold for image binarization. Two decision trees are selected
and a best overall accuracy of 86.65$\%$ is obtained from the first decision
tree on 3627 training objects and 1813 test objects.

In[99, 100], since the red blood cells and the background area in the FL
images show relatively consistent patterns, thus they are segmented by setting
a threshold for the intensity value in the RGB color space. Fig. 22 shows the
segmentation results of [99].

Figure 22: From left to right are grade I, grade II, grade III of FL
respectively. Blue corresponds to nuclei, cyan corresponds to the cytoplasm,
yellow corresponds to extracellular, red corresponds to the background and
gray corresponds to red blood cells. The figure corresponds to Fig. 1 in [99].

In[102, 130, 131, 134], Otsu thresholding is used in the segmentation step. In
[102], after the conversion of RGB color space to $L^{*}a^{*}b^{*}$ color
space, the Otsu thresholding method is used to segment the background tissues
for extracting ROIs in images. In [130, 131], a threshold of 0.37 is used to
remove red blood cells, then Otsu thresholding is used to segment nuclei from
extra-cellular and background tissues. Fig. 23 shows the result after Otsu
thresholding segmentation. In [134], Otsu thresholding is applied to the
H$\&$E segmentation. Eosin segmentation and hematoxylin segmentation are
realized by setting the highest threshold and the lowest threshold
respectively.

Figure 23: The result after Otsu thresholding. (a) and (b) respectively
correspond (a) and (e) of Fig. 2 in [130].

In[155], a novel color image segmentation is used to help pathologists to
investigate prognostic factors of FL. The segmentation method includes
preprocessing, adaptive threshold, watershed and post-processing. The method
is tested on 50 images and compared with the results of manual counting. The
method only uses a mean time of 18 seconds which is the mean time of an image
proceeding, and 100$\%$ objects are classified in the method.

In[170], global thresholding and local thresholding are evaluated in 35 blood
smear test images of acute lymphoblastic leukemia (ALL). Compared with ground
truth, local thresholding obtains the mean Zijdenbos Similarity Index (ZSI),
precision and recall is 92.5$\%$, 91.79$\%$ and 94.03$\%$ respectively, and
global thresholding obtains the mean ZSI, precision and recall are 30.72$\%$,
23.38$\%$ and 99.39$\%$ respectively. The results show that global
thresholding can be an alternative method when the number of white blood cell
pixels and the number of other pixels reach a balance in an image.

Table 9: Threshold-based Methods for Segmentation. Reference | Year | Team | Segmentation Method
---|---|---|---
[148] | 2007 | M Zorman | Thresholding using mean brightness value
[99] | 2008 | O Sertel | Thresholding the intensity values of RGB color space
[102] | 2009 | K Belkacem-Boussaid | Otsu thresholding
[100] | 2009 | O Sertel | Thresholding the intensity values of RGB color space
[155] | 2010 | U Neuman | An adaptive threshold
[130] | 2014 | E Michail |  | Red blood cells are removed by thresholding
---
with value of 0.37; Otsu is used to segment nuclei
[131] | 2014 | K Dimitropoulos |  | Red blood cells are removed by thresholding
---
with value of 0.37; Otsu is used to segment nuclei
[134] | 2016 | N Codella | Otsu thresholding
[171] | 2017 | C Senaras |  | Optimal adaptive thresholding method,
---
a parameter free elliptical arc and line segment detector,
Simple Linear Iterative Clustering
[170] | 2020 | EP Mandyartha | Global and adaptive thresholding

###  4.2 Clustering-based Segmentation Methods

In the clustering algorithm, the goal is to group similar things together. The
clustering algorithm usually does not use training data for learning, called
unsupervised learning in ML. Contrary to supervised segmentation methods,
unsupervised segmentation does not train a classifier. Unsupervised
segmentation groups the pixels of an image into several clusters based on
similar pixels with a property (color, texture, intensity value, etc.). The
commonly used clustering-based segmentation method is $k$-means clustering.
Table. 10 shows the references using clustering-based segmentation methods.

$k$-means clustering is an iterative process of splitting $n$ objects into $k$
clusters. First, $k$ cluster center points are obtained by calculating the
mean of pixels in each cluster. Second, the distance between each object and
cluster center points is calculated. Third, an object is assigned to a cluster
with minimum distance. Fourth, the cluster center points are recomputed
through averaging all objects of a cluster. Finally, the best clustering
result is obtained through repeated iterations.

In[99, 100, 108], in an FL image, the components including nuclei, cytoplasm
and extracellular are segmented by $k$-means clustering. Fig. 22 (Sec. 4.1)
shows the segmentation results by $k$-means clustering. In[116], $k$-means
clustering is used to segment each cell into nuclei, cytoplasm and inter-
nuclei materials. In [117, 119], $k$-means is used to group four clusters
including background, cells stained blue, follicle regions in the darker brown
regions and the regions which show light brown. Fig. 24 shows the example of a
scatter plot of test image after $k$-means clustering.

Figure 24: Example of scatter plot of test image after clustering. The figure
corresponds to Fig. 3 in [117].

In[118], $k$-means clustering is applied to group three structures including
mantle zone, germinal centers and inter-follicular regions. Besides, $k$-means
clustering is again used for a finer segmentation.

In[64], a hierarchical $k$-means clustering method is proposed for segmenting
FL images. First, $k$-means clustering segments H$\&$E images into nuclei,
cytoplasm and extracellular space. However, only core parts of nuclei are
segmented, and most nuclei are segmented into cytoplasm because of low
contrasts of outer edge in nuclei. Therefore, a second $k$-means clustering,
namely hierarchical clustering, is used to segment the regions which are not
nuclei into three parts (the three parts are described in the first
clustering).

In[172], a method is proposed for FL grading. The method segments nuclei
through using a locally defined clustering approach to identify the shape of
nuclei. In the experimentation, 25 images are used to be training dataset 85
images are used to be test dataset. The method can be used to grade FL and
accurately identify grade III.

Table 10: Clustering-based Methods for Segmentation. Reference | Year | Team | Segmentation Method
---|---|---|---
[99] | 2008 | O Sertel |  $k$-means clustering
[116] | 2008 | O Sertel |  $k$-means clustering, eccentricity
[100] | 2009 | O Sertel |  $k$-means clustering
[117] | 2010 | S Samsi |  $k$-means clustering
[118] | 2010 | J Han |  $k$-means clustering
[119] | 2012 | S Samsi |  $k$-means clustering
[108] | 2012 | B Oztan |  $k$-means clustering
[64] | 2016 | P Shi | Hierarchical $k$-means clustering

###  4.3 Region-based Segmentation Methods

A watershed algorithm is a segmentation method based on region, which segments
ROIs by finding watershed lines. In the watershed algorithm, grey-level images
are considered a topographic. The grey level of a pixel corresponds to its
elevation, and the high grey level corresponds to a mountain, the low grey
level corresponds to the valley. When the water level rises to a certain
height, the water will overflow the current valley. A dam can be built on the
watershed to avoid the collection of water in two valleys. Therefore, the
image is divided into 2-pixel sets; one is the valley pixel set submerged by
water, and the other is the watershed line pixel set. In the end, the lines
formed by these dams partition the entire image to achieve the segmentation
result of the image.

In[155], a watershed is used to segment nuclei as the second step of
segmentation because the white components in the first step of segmentation
(adaptive threshold map) are sometimes not related to only one nucleus. Fig.
25 shows the segmentation result of a watershed algorithm.

Figure 25: The segmentation result of watershed algorithm. The figure
corresponds to Fig. 3 in [155].

In[117], multiple follicles may merge into one region in the process of
$k$-means clustering. Therefore, the iterative watershed algorithm is used to
split the overlapping regions until the segmentation ends. Fig. 26 shows the
process of the iterative watershed algorithm.

Figure 26: The process of iterative watershed algorithm. (a), (b) and (c)
corresponds the first, second and third iteration. The figure corresponds to
Fig. 4 in [117].

In[122], a method for predicting axillary lymph nodes is proposed. In the
segmentation step, watershed transform is applied to segment candidate nuclei
detected by SVM.

###  4.4 Deep Learning based Segmentation Methods

In addition to some traditional segmentation methods,[173, 133, 96] use deep
learning segmentation techniques in LHIA.

In[173], four deep learning methods are used in automatically detecting
lymphocytes of histopathology images. The four deep learning methods are: (1)
patch classification using FCN; (2) U-net, which is used for segment cells;
(3) detection using You Only Look Once (YOLO); (4) prediction using Locality
Sensitive Method (LSM). The dataset consists of breast cancer, colon cancer
and prostate cancer images, which includes 37 training images, 6 validation
images and 40 testing images. Finally, U-net achieves the highest performance
of an F1-score equalling 0.78. Besides, U-net achieves a higher agreement ($k$
= 0.72) with manual evaluation, compared with pathologists ($k$ = 0.64).

In[133], a fully convolutional neural network (FCNN) is used for
differentiating tissue and background regions for avoiding low-efficient
scanning in empty background areas. There are seven convolutional layers, the
first six convolutional layers have a ReLU activation function and the last
convolutional layer has a softmax activation function. FCNN can achieve
similar performance with U-net in segmentation. This dataset contains WSIs of
organs in 10 categories, each with 10 WSIs, for a total of 100 WSIs. There are
10 CK8-18 stained and 10 H$\&$E stained lymph node WSIs. FCNN is compared with
three traditional methods, achieves Dice scores from 0.97 to 0.98 in a
resolution-agnostic network, a Dice score of 0.97 and sensitivity of 0.98 on
eight testing images.

In[96], a method is developed, which is used to automatically detect follicles
in CD8 stained FL images. The method uses U-net to segment follicles in the
whole WSI. The method is trained in more than 1000 follicles and is tested on
eight whole digital slides. Finally, the method achieves an average Dice
similarity coefficient equalling 85.6$\%$ compared with pathologists. Fig. 27
shows the segmentation results using U-net.

Figure 27: (a) and (c) are original images. (b) and (d) are the segmented
results, the green regions are the detected follicles. The figure corresponds
to Figure 4 in [96].

###  4.5 Other Segmentation Methods

In addition to the above segmentation methods, some other segmentation methods
also achieve good results.

In[106], a technique is developed to segment the follicular regions in the
H$\&$E stained FL images. The technique is based on an active contour model
and is initialized through users manually selecting seed points in follicular
regions. The technique uses matched filter to remove noises and flatten the
background of L channel after the color space conversion of RGB to
$L^{*}a^{*}b^{*}$. The technique is tested on 40 FL images and the mean
accuracy is 0.71, the standard deviation is 0.12.

In[174], an algorithm is proposed to separate touching cells. In the
algorithm, the outer boundary is delineated by a L2E robust estimation and a
robust gradient vector flow snake, then concave points on the inner edges and
boundary are detected automatically. These points construct a concave vertex
graph. Finally, the touching cells are separated by recursively calculating
the optimal path in the concave vertex graph after minimizing a cost function.
The algorithm is tested on two datasets including 207 touching images and 3898
images and an average accuracy of 90.1$\%$ is obtained. Fig. 28 shows the
process of constructing the concave vertex graph.

Figure 28: The process of constructing the concave vertex graph. (a) Original
image with yellow boundary. (b) Detection of high curvature points. (c)
Detection of concave points. (d) Detection of inner edges. (e) The original
image is superimposed by outer boundary, concave vertices and inner edges. (f)
The concave vertex graph. These figures (a)-(f) correspond to (a)-(f) of Fig.
3 in [174].

In[115], a segmentation method is proposed to automatically detect
lymphocytes, which consists of a Bayesian classifier and template matching.
the segmentation algorithm takes advantage of the strong hematoxylin staining
of lymph infiltration through the Bayesian classifier, template matching can
be used to discard the false positive results because of the size and shape of
the lymphocyte. Fig. 29 shows the result of the proposed segmentation method.

Figure 29: (a) Original image. (b) Voronoi diagram. (c) Delaunay
triangulation. (d) Minimum spanning tree showed in segmented lymphocytes. The
figure corresponds to Fig. 3 in [115].

In[114], an image analysis system is proposed to quantitatively evaluate
digitized FL tissue slides. In the segmentation step, the tissue is segmented
into cytological components by the mean-shift algorithm for segmenting
individual cells. The mean-shift algorithm does not need to know the number of
clusters and does not control the shape of clusters. Then, morphological
operations are used in post-processing. Fig. 30 shows the process of
segmentation.

Figure 30: (a) Original image. (b) The segmentation result of mean-shift. (c)
The final result. The figure corresponds to Figure 1 and Figure 2 in [114].

In[93], a multi-phase level set framework is used to initial segmentation of
cells. The method avoids the vacuum and overlap problems. In the method, $n$ =
4 phase segmentation is selected to separate the ROIs for there are four
regions with different intensity levels in original images.

In[139], a computer-aided detection system is proposed, which is used to
automatically detect CB cells in FL images. In the segmentation step, the RGB
images are converted into 1-D unitone images by Principal Components Analysis
(PCA). Then, the unitone images are normalized to a range of [0, 1]. Next, the
individual cells are segmented through two steps: (1) expectation, which is
used to calculate the likelihood relative to the current estimation. (2)
maximization, which is used to maximize the expected log likelihood.

In[126], an integrated framework is proposed, which comprises an algorithm of
supervised cell-image segmentation and a method of splitting touching-cell. In
this method, the cell tissue and extracellular tissue are segmented. The
color-texture feature extracted from the local neighborhood of each pixel is
used as the input of the classification algorithm. Local Fourier transform
(LFT) texture features are extracted from MDC color space (described in Sec.
3.1). In the segmentation part, the boundaries are smoothed by the Fourier
shape descriptor after differentiating between a touching-cell clump and a
single non-touching cell. The approach is tested on 750 positive and 750
negative training patches and each image achieves a total error rate of
5.25$\%$.

In[164], a new algorithm is proposed, which is applied for splitting
touching/overlapping cells. Similar to the segmentation part of [126], the
boundaries are smoothed by Fourier shape descriptor after differentiating
between a touching-cell clump and a single non-touching cell. Then, a novel
iterative splitting algorithm is only used to split the touching-cell clump.
Finally, the splitting method is tested on 21 FL images and obtain an average
error rate which equals 5.2$\%$.

In[141], a framework is proposed, which is used to segment follicular regions
before histological grading. First, the mask of follicular boundaries is
generated. Then, the boundaries are mapped onto the corresponding registered
images. The framework is tested on 12 H$\&$E and IHC stained FL images.
Finally, the maximum average of sensitivity, specificity, conformity and JAC
is 0.935, 0.843, 0.198 and 0.57 respectively.

In[146], a system is introduced, which can automatically differentiate the
categories for diffuse lymphoma cells. In the segmentation step, an image
pixel is segmented into multiple pixel regions through the minimum variance
quantization approach.

In[110], an unsupervised method used for segmenting the nuclear components is
proposed. The method is based on fuzzy 3-partition entropy and genetic
algorithm. The genetic algorithm initial population is needed to define, then
the images after preprocessed are calculated and normalized, the fuzzy
3-partition entropy approach is used to define the values of each individual.
The dataset used in the segmentation method includes 12 CLL, 62 FL and 99 MCL
images. The mean value of accuracy of the method is 81.48$\%$.

In[137], an unsupervised segmentation algorithm using a genetic algorithm is
proposed for obtaining object diagnoses, which is applied to segment nuclei in
CLL and FL images. The algorithm segments nuclei through evaluating different
genetic algorithms fitness functions. SVM is used to classify using the
features extracted from segmented regions. The algorithm is tested on 12 CLL
and 62 FL images and obtains an accuracy of classification equalling
98.14$\%$.

In[127], a fast superpixel segmentation algorithm is applied. The number of
expected superpixels is set to 3500 and the compact factor is set to 35.
First, the color deconvolution algorithm (described in Sec. 3.1) is applied to
differentiate nuclear regions. Second, histogram stretching is used in the
hematoxylin channel of the output of the color deconvolution algorithm. Third,
the nuclei are extracted by image thresholding followed by filtering and small
object removal. Finally, a superpixel segmentation algorithm is used because
some segmented nuclei may overlap. The segmentation result is shown in Fig.
31.

Figure 31: The result of segmentation. The figure corresponds to Fig. 1 in
[127].

In[175], a new framework is presented, which is developed to detect and
classify CBs in FL images stained with H$\&$E and PAX5. An energy minimization
approach based on graph cuts is used in the segmentation of nuclei in PAX5
stained images. The approach is a kind of unsupervised approach. The process
of segmentation is a labeling problem, the label stands for different
cytological components. After the nuclei segmentation, there are many
overlapping nuclei that need to be split. First, the spurious seeds are
identified and discarded. Then, each nucleus is detected by using an
ellipsoidal model. Finally, a validation criterion is applied to identify
spurious seeds.

###  4.6 Summary

From the above summary of the segmentation methods in LHIA, the threshold-
based segmentation methods and $k$-means clustering-based segmentation methods
in the traditional methods are more commonly used. In terms of deep learning,
U-net is an efficient segmentation method, and FCNN can also achieve similar
performance to U-net with low computational cost. In addition to the
traditional and deep learning segmentation methods, some other segmentation
methods also achieve good performance.

##  5 Feature Extraction

In image processing tasks, it is expected to extract features from the output
of the image segmentation. Feature extraction is a crucial step in
histopathology image analysis, which is a process of extracting features of
the object from segmented images. The extracted features are generally used in
classification or detection tasks. The set which consists of all quantified
features is defined as a feature set or a feature vector. The features are
defined as original characteristics or attributes of an image, which include
visual features and statistical features. Visual features generally include
color, texture, shape, etc. Statistical features need to be measured manually,
which include spectrum, histogram, etc. The extraction methods are crucial for
the following classification and detection tasks. This section introduces the
commonly used features and methods of extracting features in LHIA. Table. 11
and Table. 12 show the commonly used feature extraction methods, including
traditional and deep learning-based methods.

###  5.1 Visual Features Extraction

The visual features of an image include color, shape, texture.

####  5.1.1 Color Features Extraction

Color features belong to the internal features of an image and describe the
surface properties of images. Color features are widely used in CAD system.

In[148, 149], the contours of a follicle are different from the other
follicles in the image, so the method of extracting features focuses on the
distinction. First, the R component of RGB color space is extracted for it has
the best contrast compared with the background areas. The mean brightness of
the component is used as a threshold for image binarization. Because the
density of white pixels in follicle contours is relatively high, so a method
of measuring density is proposed. Then, four transformation matrices are
obtained, which can provide a feature vector for each pixel of the original
image. In the method, the transform matrices are obtained through scanning the
binary image obtained from thresholding on column-by-column and row-by-row.

RGB color space is a commonly used color space for color feature extraction.
The references using RGB color space include [107, 129]. In [64], RGB color
space is used to extract color intensities. Fig. 32 shows the distribution map
of pixels in feature space after first clustering. In addition, the mean and
standard deviation of a single pixel is also extracted in each color channel
of RGB. In [128], the color histograms are extracted in RGB color space.

Figure 32: (a) Original image. (b) The red part represents extracellular
spaces, the green part represents cytoplasm and the blue part represents
nuclei. RGB intensities represent the coordinate axes. The figure corresponds
to Figure 2 in [64].

In[116, 117, 118, 141, 107, 122], HSV color space are used to extract color
features. In [117, 119], the pixel values in the H channel are extracted for
clustering.

In[114], the $L^{*}u^{*}v^{*}$ is used to construct feature space.

In[105], the L channel of $L^{*}a^{*}b^{*}$ is used to extract morphological
features. In [107], the color features extracted from $L^{*}a^{*}b^{*}$
represent images intensity and color information of images.

In[88], each image is divided into 25 blocks, the color features are extracted
from these blocks. The extracted color features include color dominant, color
histogram and color moment. Same as [88], [157] first divides the original
image into 25 blocks and a total of 505 visual features are extracted from
each block. The extracted color features include color dominant, color moment,
color histogram and edge histogram.

In[126], LFT is used to extract color features from RGB color space.

In[123], a color histogram with 64-dimension is extracted from the
64$\times$64 blocks of 3771 CBs and 4000 non-CBs. In [128], color histogram of
three dimension and marginal color histogram of one dimensions are extracted
in RGB color space.

####  5.1.2 Texture Features Extraction

In[99], a new method applied to grade FL is proposed. This method improves the
_Gray Level Co-occurrence Matrix_ (GLCM) by using self-organizing feature maps
(SOFMs). After segmentation, SOFM extracts features of non-linear color
quantization and color-texture. Color quantization is a process of
constructing a color space through selecting several representative colors
from the set of the whole RGB color space. The texture features are extracted
by using GLCM from the quantized color space. This method is compared with
gray-level and uniform and color quantization method, Fig. 33 shows the
compare results. From the figure, the proposed method provides a better
enhanced image that can produce more detailed texture features. The method is
tested on 510 images through $k$-fold cross validation. Finally, a Bayesian
classifier is applied, and an overall correct classification rate of 88.9$\%$
and 100$\%$ sensitivity are obtained.

In[100], first, $k$-means clustering is used to segmentation (in Sec. 4.2).
Second, a method for constructing color texture features is proposed, which is
based on the combination of the model-based intermediate representation (MBIR)
with self-organizing maps (SOM). Third, the dimension of feature space is
reduced by the combination of PCA and linear discriminant analysis (LDA).
Finally, Bayesian is used as the classifier on the dataset including 180, 240
and 90 grade I, grade II and grade III images respectively. This method has
the best identification on the grade III with a sensitivity equalling 98.9$\%$
and a specificity equalling 98.7$\%$, besides, the overall classification
accuracy is 85.5$\%$.

The co-occurrence matrix provides a statistical approach, which is used to
characterize the spatial dependencies of gray levels of an image[176].
In[116], co-occurrence matrix method is used to extract homogeneity feature.
In [117], energy metric is extracted for determining the locations of
follicle. In [141], a co-occurrence matrix is used to quantify the texture
features. In [119], there are two texture features are extracted. One is the
output of a grayscale image after a median filter, another is the energy
obtained using a co-occurrence matrix. In [108], co-occurrence extracts five
color texture features including homogeneity, energy, contrast, correlation
and entropy.

In[102], a method for automatically identifying CB cells and non-CB cells is
proposed. The step of cell identification includes color space conversion of
RGB to $L^{*}a^{*}b^{*}$, Otsu thresholding, morphology operations, extraction
of area and perimeter of the cell. Then, the principal component analysis
(PCA) is applied to extract color texture features in three color spaces
including RGB, HSI and $L^{*}a^{*}b^{*}$. Next, a quadratic discriminant
analysis (QDA) is used for classification through 10-fold cross validation.
The H$\&$E stained dataset includes 218 CB and 218 non-CB images, the dataset
is divided with an 80$\%$-20$\%$ ratio. Finally, a classification accuracy
equalling 82.56$\%$ is obtained. In [103], the same method and dataset are
used for automatically detecting CB and non-CB images of FL. In the experiment
result, the classification accuracy, sensitivity and specificity are
82.56$\%$, 86.67$\%$ and 86.96$\%$ respectively.

In[105], a method similar to[102] is applied to recognize CB cells from non-CB
cells in FL images. In this method, color texture features extracted by using
PCA are used in the QDA classifier and variance is used to quantify color
texture features.

In[93], a total of 173 extracted texture features include Zernike features,
Daubechies, wavelet features, Gabor features, skeleton features and Haralick
features. SVM Recursive Feature Elimination (SVM-RFE) is applied to eliminate
the irrelevant features. In[157], the texture features extracted from each
block include texture co-occur, texture Tamura, texture wavelet, texture Gabor
and local binary patterns (LBPs). In [129], the extracted texture features
include entropy, homogeneity, mean and contrast.

Figure 33: Quantized images produced by uniform gray-level, color quantization
and SOFM. The figure corresponds to Fig. 3 in [99].

In[139], the _Gray-Level Run Length Matrix_ (GLRLM) is applied to derive
higher order statistics to represent the texture of images. The features
include the features of a wide spectrum and texture. In [153], the Gray-Level
Run Length (GLRL) algorithm is applied for calculating the Gray-Level Non-
uniformity (GLN) of nuclei, GLN is used for quantifying the texture
abnormalities. The mean and skewness of histogram are extracted by calculating
the grayscale histogram of nuclei. In [64], skewness and kurtosis are
extracted. Skewness is used to measure the symmetry of local signals, kurtosis
is used to measure the signal intensity under a normal distribution.

In[49], an analysis is conducted, which is applied for cells and tissues. In
this work, different texture features are extracted from different color
spaces and are used in classification, for the purpose of determining the
feature set which can be used in different classification tasks. In order to
determine the feature set, the existing method of grayscale of color images is
generalized. The generalized method is used in calculating matrices of gray-
level co-occurrence, gray-level difference and gray-level run-length. SVM is
applied for classification using the IICBU-2008 dataset to compare the feature
sets. The experiment is conducted in five color spaces, and the result in HSV
is the best.

In[128], texture features including contrast, correlation, energy and entropy
are extracted by Gray-Level Co-occurrence Matrices (GLCM).

In[94], the first-order statistical features of texture including standard
deviation and average intensity are extracted is used for $k$-means
clustering.

In[88], texture features extracted from blocks include texture co-occur,
texture wavelet, texture Tamura, texture Gabor and LBPs.

In[126], an LFT algorithm is developed to speed up the process of extracting
texture features, which is based on image shifting and image integral and is
applied on MDC color space.

In[107], the texture features extracted using co-occurrence histogram includes
mean, standard deviation, correlation, energy, contrast, entropy and
homogeneity. The texture feature extracted using normalized co-occurrence
matrix include mean, entropy and homogeneity.

In [154], Log-Gabor filter is used to extract texture features which include
entropy, contrast, homogeneity and spatial frequency. In [128], the mean and
standard deviation are extracted through Gabor filters.

SIFT is an algorithm for describing local characteristics of an image in CV,
it extracts features through finding angles, location and the grayscale at the
extreme points. Features extracted through SIFT have characteristics that are
scale and rotation invariant. In [121], a method for automatically detecting
lymphocytes is proposed. Scale-Invariant Feature Transform (SIFT) is used in
the step of feature extraction. Fig. 34 shows the key points retrieved by
SIFT.

Figure 34: The key points retrieved by SIFT (color online). (a) Key points
without morphological refining. (b) Key points after morphological refining.
The figure corresponds to Fig. 4 in [121].

####  5.1.3 Shape Feature Extraction

In[174], the contour of boundary of the touching cells is extracted by the
followed steps: First, L2E robust estimation is applied to produce a rough
estimation for the boundaries. Second, a robust gradient vector flow extracts
the contour from the background.

In[116, 177], the size of a cell, eccentricity are extracted for feature
construction. In [102], the area and perimeter of cells are extracted. In
[114], area and nuclear to cytoplasm ratio are extracted in the feature
vector. In [105], the largest area is extracted through Otsu thresholding,
opening, closing, labeling and area classification. In [108], the
morphological features extracted include the length of major axes, length of
minor axes and area of ellipses. In [172], two features are extracted from
segmented images, which include curvature and center of the nuclei. In [154],
the extracted shape features include area, bounding box, perimeter, convex
area, solidity, major axes and minor axes. In [122], six features are
extracted including perimeter, area, aspect-ratio, circularity and two
measurements quantifying the shape of nuclei. In [124], a total of 28
geometrical and morphological features are extracted, which include the
percentage of region of the tumor over the region of whole tissue, the area
ratio, mean and the longest axis. In [125], the extracted geometrical and
morphological features include max, mean, skewness, variance and kurtosis,
etc.

In[153], individual nuclei and their size construct the first feature through
connected component labeling. Then, the perimeter of nuclei is extracted.
Next, aspect ratio and ellipse residual are extracted after estimating the
best fitting ellipse through the Orthogonal Distance Regression (ODR).

###  5.2 Statistical Feature Extraction

In[116], skewness, energy, mean intensity, kurtosis and entropy are extracted
for obtaining the intensity characteristics.

###  5.3 Graph-based Feature Extraction

In[121], Voronoi diagram, Delaunay triangulation and minimum spanning tree are
constructed for extracting the arrangement information of lymphocyte nuclei
because lymphocyte detection is not enough to describe the abnormalities
lymphatic infiltration. Voronoi diagram is constructed using the segmented
objects to produce a tessellation for an image. Delaunay triangulation is
constructed through connecting the centroid of the adjacent diagram of the
Voronoi diagram. Suppose there is a connected graph $G$, $G$ may have multiple
spanning trees, and these spanning trees have all the vertices of $G$. When
traversing all the vertices of these spanning trees, the spanning tree with
the least cost is the minimum spanning tree.

In[115], a method combining manifold learning with graph-based features, which
is used to identify lymphocytic infiltration in histopathology images of
breast cancer. Graph-based features extracted in the method include Voronoi
diagram, Delaunay triangulation, minimum spanning tree, nuclear features and
Varma-zisserman texton-based features. Manifold learning is used for feature
dimensionality reduction. The dataset includes 22, 10 and 9 cases of high,
medium and low lymphocytic infiltration. SVM is used to classification through
3-fold cross-validation and an accuracy equalling 89.50$\%$ $\pm$ 6.22 $\%$ is
obtained.

In[101], a method for detecting and grading lymphocytic infiltration of breast
cancer is proposed. a method for detecting and grading lymphocytic
infiltration of breast cancer is proposed. First, the lymphocytes are detected
by region growing algorithm and Markov random field algorithm. Second, 50
features are extracted from three graphs including the Voronoi diagram,
Delaunay triangulation and minimum spanning tree (as shown in Fig. 35). The
three graphs are constructed with the center of single detected lymphocytes as
vertices. Next, the high-dimensional feature vector is reduced through graph
embedding. Finally, the method is tested on a dataset including 41 H$\%$E
stained images. SVM is applied to classification through more than 100 3-fold
cross validation and a classification accuracy that is higher than 90$\%$ is
obtained.

In[108], a method is proposed for grading FL images, which is based on a
multi-scale feature analysis. The cell graph is used to describe the
structural tissue of cell and represents the histological image of undirected
and unweighted graphs. The graph nodes are composed of cytological components
and the graph edges consist of approximate adjacencies. The features extracted
from cell graphs are used for classification. Three classifiers including SVM,
Bayesian and $k$-nearest-neighbors ($k$NN) are selected for classification and
the dataset includes 510 FL images. The result shows that when the cell graph
features are combined with MBIR and low-level features, the accuracy is
higher.

Figure 35: (a) and (e) stand for high and low levels of lymphocytic
infiltration. (b) and (f) stand for corresponding Voronoi diagrams. (c) and
(g) stand for corresponding Delaunay triangulations. (d) and (h) stand for
corresponding minimum spanning trees. The figure corresponds to Fig. 7 in
[101].

In[120], a method for automatically detecting cancer in H$\&$E stained images
of the gastric lymph nodes is proposed. In this method, three features are
combined, which include Higher-order Local Auto-Correlation (HLAC) feature,
wavelet feature, Delaunay feature. These features are connected into one
vector after being calculating in images, and then SVM is used as a classifier
to detect cancer. The dataset includes 19 training images and 381 testing
images. In addition, each image is divided into 120 patches. Finally, the best
results of sensitivity and specificity are 95.7$\%$ and 82.1$\%$.

###  5.4 Deep Learning Feature Extraction

In addition to the traditional methods of low-level feature extraction
described above, some advanced deep learning methods have been widely used in
LHIA in recent years. There is no need to manually extract features in deep
neural networks, and better features can be automatically extracted. A neural
network extracts visual features which can be overlooked by human visual
inspection. The convolutions in the neural network can filter a large number
of features contained in an image. For example, in the feature extraction of
edge detection, convolutions can help determine the number of nuclei with
abnormal shapes and the density of nuclei neighborhoods.

In[44], CNN replaces traditional methods for feature extraction. Traditional
methods rely on hand-made quantitative feature extractors for feature
extraction, while CNN can automatically extract image features. Fig. 36 shows
a CNN model. C1, C2, C3, and C4 are convolutional layers for feature
extraction through continuously extracting higher-level features from the
image patch. In [160], the CNNs through 12 initial feature filters obtain a
network containing 258,327 parameters. The parameters.

Figure 36: The figure corresponds to Figure 1 in [44].

In[178], Faster R-CNN is used in classifying lymphoma. The input image is
transformed into a tensor, then the tensor is sent into a CNN which is pre-
trained or initialized to generate a feature map. Then, the region proposal
network is used to calculate the feature to obtain a region proposal
containing the target. A new tensor is obtained by using the region of
interest pooling, and the tensor is put into the fully connected layers for
classification.

In[135], a new method based on CNN is proposed, which is combined with
multiple instances, domain adversarial and multi-scale. The method
automatically detects tumor-specific features. In the CNN network, the feature
extractor is obtained by training a single-scale DA-MIL network, then the
multi-scale DA-MIL network is trained through plugging the feature extractor
into the CNN network.

In[179], an ensemble model is proposed, which is based on handcrafted fractal
features and deep learning. 300 fractal features are obtained through features
extraction. These features compose an image after being reshaped into a
10*10*3 matrix, and the image is the input to the CNN.

In[180], the convolutional layer of CNN extracts features, the pooling layer
reduces dimension. A hierarchical feature extraction module is generated by
stacking the convolutional and pooling layer. The feature map is reshaped into
a 3D matrix and is flattened to form a vector for feature extraction.

###  5.5 Other Features Extraction

In[177], a method of automatic non-rigid registration is proposed, which is
used in different staining histopathology images. This method extracts
matching high level features on the down-sampled images of original images.
Fig. 37 shows the result of feature extraction. Then, the match candidates are
formed by matching features between the base images and float images. The
dataset in this experiment includes 5 pairs of FL slides, one is stained with
CD3 and another is stained with H$\&$E in each pair. The statistical result
demonstrates that the accuracy of this method is equivalent to that of manual
registration.

Figure 37: The result of high-level feature extraction. The figure corresponds
to Fig. 4 in [177].

In[104], a two-stage method is used to classify three types of lymphoma: CLL,
FL and MCL. In the outer level, Fourier, Chebyshev and wavelets, Chebyshev of
Fourier and wavelets of Fourier are calculated. The set of global features is
calculated in the inner level, then the calculated features are fused into a
feature vector.

In[130], a method is proposed, which is developed for automatically detecting
malignant cells in microscopic images of FL. The cell nuclei are first
detected after preprocessing and segmentation, next the expectation
maximization algorithm is used to split touching cells. Then, the candidate
CBs are selected by their size, shape and intensity histogram. Then, in the
step of feature extraction, the image consisting of cell and its surroundings
is used as a feature vector for classifying CBs and non-CBs. The data set
including 70 CBs images and 110 non-CBs images is divided into training set
and validation set at 80$\%$ to 20$\%$. A LDA classifier is applied for
classification, and average 82.58$\%$ of malignant cells are detected.

###  5.6 Summary

This section has described feature extraction methods, including traditional
and deep learning-based methods. In traditional methods, the extracted
features are generally color features, texture features and shape features,
and the commonly used traditional methods include the conversion of color
space, GLCM, co-occurrence matrix, LBP, Gabor filter, SIFT and so on. However,
advanced algorithms are researched and applied in LHIA with the development of
MV. Deep learning is now being studied by more and more people, especially in
the field of MV, and it has also shown good performance. With the advancement
of deep learning technology, feature extraction is no longer limited to manual
extraction because deep neural networks can automatically extract better
features.

Table 11: Feature Extraction Methods. Reference | Year | Team | Details
---|---|---|---
[148] | 2007 | M Zorman | R component of the RGB image
[99] | 2008 | O Sertel |  | Color Texture Analysis Using SOFM,
---
the gray level co-occurrence features
[115] | 2008 | A Basavanhally | Graph-Based Architectural Features
[174] | 2008 | L Yang | contour
[116] | 2008 | O Sertel | color, texture, morphological features
[177] | 2009 | L Cooper | matching high level features
[101] | 2009 | AN Basavanhally | Graph-based architectural
[102] | 2009 | K Belkacem-Boussaid | geometrical features vector and texture color features
[100] | 2009 | O Sertel | morphological and topological features, color, texture
[103] | 2010 | K Belkacem-Boussaid | Geometric and color texture features
[114] | 2010 | O Sertel | $L^{*}u^{*}v^{*}$
[93] | 2010 | J Cheng |  | 49 Zernike features, 30 Daubechies 4 wavelet features,
---
60 Gabor features, 5 skeleton features,
13 Haralick features and 16 morphological features
[104] | 2010 | NV Orlov | Transform-based global features
[139] | 2010 | O Sertel | gray-level run length matrix (GLRLM)
[117] | 2010 | S Samsi | texture and color
[105] | 2010 | K Belkacem-Boussaid | Morphometric and color texture features
[94] | 2010 | H Fatakdawala | texture
[88] | 2010 | T Meng | color and texture feature, C-RSPM model
[118] | 2010 | J Han | color and texture feature
[126] | 2011 | H Kong | color texture feature
[149] | 2011 | M Zorman | R component of the RGB image
[141] | 2012 | M Oger |  | Each pixel is represented by a feature
---
vector of color and texture information.
[107] | 2012 | HC Akakin | color, texture
[119] | 2012 | S Samsi | color, gray level feature, texture
[108] | 2012 | B Oztan |  | the combination of cell-graph based features
---
with the MBIR and color-textural features
[172] | 2013 | B Arora | no of centers, length of curvature
[154] | 2013 | BS Sandhya | morphological and texture features
[157] | 2013 | T Meng | A set of 505 visual features
[129] | 2013 | E Acar | 72 color and texture features
[120] | 2014 | T Ishikawa | HLAC feature, Wavelet feature, Delaunay feature
[130] | 2014 | E Mchail |  | The whole image of the cell with its
---
surroundings are used as a feature vector
Table 12: Continue: Feature Extraction Methods. Reference | Year | Team | Details
---|---|---|---
[153] | 2014 | E Mchail | morphological and textural features
[121] | 2014 | YL Kuo | Scale-invariant feature transform
[122] | 2015 | MD Zarella | color-based feature, shape-based feature
[123] | 2015 | MFA Fauzi | 64-dimensional color histogram features
[49] | 2015 | C Di Ruberto | GLCM features, GLRLM features, GLDM features
[140] | 2016 | J Chen | Extension of FCN
[181] | 2016 | Y Song |  | the local DSIFT features and patch-level features that are
---
learned using a deep belief network (DBN) model
[124] | 2016 | D Wang | geometrical and morphological features
[125] | 2016 | R Chen | geometrical and morphological features
[64] | 2016 | P Shi |  | Color intensities, mean and standard deviation
---
value of pixels, local texture features
[134] | 2016 | N Codella | CNN
[44] | 2016 | G Litjens | CNN
[150] | 2017 | K Xiao | CNN
[90] | 2017 | GF Roberto | percolation theory
[167] | 2019 | N Brancati | Supervised Encoder FusionNet
[156] | 2019 | UV Somaratne | A model using transfer learning with fine-tuning
[182] | 2020 | D Li | GOTDP-MP-CNNs
[178] | 2020 | B Sheng | faster R-CNN
[160] | 2020 | JS Mohlman | CNNs
[135] | 2020 | N Hashimoto | Multi-scale domain-adversarial multiple-instance CNN
[128] | 2020 | F Bianconi | 13 image descriptors
[45] | 2020 | I Kandel | CNN
[161] | 2020 | C Syrykh | BNN
[183] | 2020 | A Nugaliyadde | RCNN
[184] | 2020 | D Candelero | Evolutionary Algorithms
[112] | 2021 | AS Martins | A Hermite polynomial algorithm
[138] | 2021 | TA Azevedo Tosta |  | Statistical and Haralick’s features extracted
---
from wavelet and ranklet transforms
[179] | 2021 | GF Roberto | An ensemble model based on handcrafted fractal features and deep learning
[180] | 2021 | N Dif | Transfer learning from synthetic labels

##  6 Classification Methods

In this section, we introduce the classification tasks in LHIA. After feature
extraction, different classifiers are selected to classify the
histopathological images. In LHIA, common classification tasks include staging
and grading malignant lymphomas (such as FL), classification of several
malignant lymphomas (such as CLL, MCL and FL) and classification of CB and
non-CB. The methods of classification include traditional ML methods and deep
learning-based methods.

###  6.1 Traditional Machine Learning based Classification Methods

In[101], SVM is applied to discriminate high and low lymphocytic infiltration
samples. In [107], SVM is used to classify FL and neuroblastoma, and the
radial basis function (RBF) is selected in SVM. In [108, 153], SVM is used to
grade FL. In [154], SVM is used to classify CBs and non-CBs. The steps of
preprocessing and feature extraction are described in Sec. 3 and Sec. 5.
Finally, an accuracy of 84.53$\%$ is obtained in 70 training datasets and 30
test datasets. In [49], SVM is used for classification and validates the
texture features extracted from the HSV color space are superior to the
features of other color spaces. In [64, 185], SVM is used to classify three
lymphomas (CLL, FL and MCL). In [136], SVM is used to classify FL and MCL.

In[129], a framework for grading FL is developed. In the framework, the image
is represented by a third-order tensor including images, features and scales.
The basic patterns of each mode can be extracted from these tensor models. A
total of 500 FL images are analyzed through SVM classifier and the accuracy
can reach up to 90$\%$.

In[99, 100, 108], Bayesian classifier is selected to grade FL. In [175], a
Bayesian classifier is applied to classify CBs in FL tissue samples. In[100,
108, 123], $k$NN classifier is selected to classify FL into three histological
grades. In[102, 103, 105], QDA is selected to classify CBs and non-CBs.

Weighted-Neighbor Distance (WND), which relies on the distance of sample to
class. The training process of the Naive Bayes Network (BBN) classifier is to
estimate the class prior probability based on the training set, and estimate
the conditional probability for each attribute. RBF is a real-valued function
whose value depends only on the distance from the origin. In[104], a method is
proposed to classify CLL, FL and MCL. WND, BBN and RBF are used as classifiers
after feature extraction. In the training set, there are 57 images for each
type of lymphoma. In the test set, CLL, FL and MCL include 56, 82 and 65
images respectively. Finally, an average classification rate equalling 99$\%$
is obtained by WND.

In[88], a framework is proposed for improving the classification accuracy,
which is based on the Collateral Representative Subspace Projection Modeling
(C-RSPM) classifier. Each classifier of C-RSPM uses the known-class data
consisting of different types in training dataset [186]. In this framework,
the original image is divided into 25 blocks, and one C-RSPM is constructed in
each block. A multimodal fusion algorithm is used to determine the final class
label after classification. IICBU-2008 dataset is divided into training set
and test set at a ratio of 2 to 1. IICBU-2008 dataset is divided into the
training set and test set at a ratio of 2 to 1. In the experiment, an average
classification accuracy of 92.7$\%$ is obtained, while the highest average
classification accuracy previously reported using the same data set is 85$\%$.
In [157], the steps of preprocessing and feature extraction are similar to
[88], and described in Sec. 3.6 and Sec. 5. The C-RSPM classifier is used for
classification through three-fold cross-validation. The IICBU-2008 dataset is
used in the experiment and the highest classification accuracy of 96.81$\%$ is
obtained compared with other classifiers including SVM, BayesNet and BBN, etc.

In[148, 149], symbol-based ML methods are used to classify FL. In[144], a
minimum distance classifier is used to classify pixels, which is a supervised
approach. In[91], a random forest classifier is used to classify three NHL
into CLL, FL and MCL.

Table 13: Traditional Classification Methods. Reference | Year | Team | Details
---|---|---|---
[99] | 2008 | O Sertel | Bayesian classifier
[101] | 2009 | AN Basavanhally | SVM
[102] | 2009 | K Belkacem-Boussaid | QDA
[100] | 2009 | O Sertel | Bayesian and $k$NN
[103] | 2010 | K Belkacem-Boussaid | QDA
[104] | 2010 | NV Orlov | WND, RBF and BBN
[105] | 2010 | K Belkacem-Boussaid | QDA
[88] | 2010 | T Meng | C-RSPM
[144] | 2012 | A Schmitz | minimum distance classifier
[107] | 2012 | HC Akakin | SVM
[108] | 2012 | B Oztan | SVM, Bayesian, $k$NN
[154] | 2013 | BS Sandhya | SVM
[129] | 2013 | E Acar | SVM
[153] | 2014 | E Mchail | SVM
[123] | 2015 | MFA Fauzi |  $k$NN
[49] | 2015 | C Di Ruberto | SVM
[181] | 2016 | Y Song | SVM
[64] | 2016 | P Shi | SVM
[136] | 2017 | TAA Tosta | SVM
[185] | 2017 | Y Song | SVM
[175] | 2017 | K Dimitropoulos | Bayesian
[91] | 2018 | MG Ribeiro | Random forest
[137] | 2018 | TAA Tosta | SVM

###  6.2 Deep Learning based Classification Methods

In[172], a method is proposed to grade FL. The step of segmentation is
described in Sec. 4.2, two features are fed a classifier which is composed of
a hidden layer with 10 neurons. In[162], a deep neural network is used to
classify lymphomas including DCBCL, FL and reactive lymphoid hyperplasia. The
proposed network includes 11 layers containing four convolutional and two
fully connected layers (as shown in Fig. 38). 388 sections are prepared into
WSIs which are divided into patches with different magnifications ($\times$5,
$\times$20 and $\times$40). Finally, the deep neural network wins pathologists
with an accuracy of 97$\%$ which demonstrates the method is potential in
diagnosing malignant lymphoma.

Figure 38: The architecture of the deep neural network. The figure corresponds
to Fig. 4 in [162].

In[158], a deep learning model based on CNN is developed to classify normal
and tumor slides in images of Camelyon16. The proposed deep learning model has
12 convolutional layers and max pooling, and the size of inputs image is
64$\times$64$\times$3\. The architecture of the model is shown in Fig. 39. The
masks of tumor region are generated through the model, then a random forest
classifier is utilized for classification. Finally, an AUC of 0.94 is
obtained, which surpasses the result of the winner of the Camelyon16
Challenge.

Figure 39: The architecture of the proposed deep learning model. The figure
corresponds to Fig. 3 in [158].

In[187], a CNN model is applied to classify four lymphomas including benign
lymph node, DLBCL, Burkitt lymphoma and small lymphocytic lymphoma. In[143], a
new method is proposed for four-classification in dataset Camelyon17. The
method combines two networks including Cycle-Consistent Generative Adversarial
Network (CycleGAN) and a densely connected deep neural network.

In[111], a classification framework is developed, which is based on the method
of multi-space image reconstruction and technology of transfer learning. The
classification results can be more accurate through the auxiliary information
obtained by the method of multi-space reconstruction. The VGG16 is applied to
reconstruct images and then the long short-term memory (LSTM) is used for the
selection and refinement of features. The softmax classifier is utilized in
the classification task. IICBU-2008 lymphoma dataset is applied in the
framework, and each image is divided into 336 patches with 64$\times$64
pixels. Finally, an ACC of 98.94$\%$ is obtained, which is higher than other
results of[188, 88, 134].

In[89], a classification model is developed. The first step is the conversion
of color space (as described in Sec. 3.1), the original images are converted
into blue ratio and $L^{*}a^{*}b$ spaces. Then, texture and statistical
features are extracted in the patches of blue ratio images and color features
are extracted from $L^{*}a^{*}b$ color space. These features are processed to
obtain hand-crafted representations and then a random forest classifier is
applied for classification. Next, the images are cropped and used as input of
a google inception net, and the softmax classifier is used in classification.
The IICBU-2008 dataset is applied in the model and an overall ACC equalling
0.991 and an AUC equalling 0.9978 are obtained. The results show the model is
an efficient classification method of NHL images.

In[167], a residual CNN is used in the classification of lymphoma subtypes.
Residual CNN is compared with other deep neural networks including U-net and
ResNet. The IICBU-2008 dataset is used in the method and an ACC of 97.67$\%$
is obtained. The result is higher than the results of U-net and ResNet, this
shows that the method is efficient.

In[182], an transfer learning platform with 17 CNNs is designed to improve the
diagnosis of DLBCL. The dataset is collected from three hospitals: 500 DLBCL
images and 505 non-DLBCL images are from hospital A, 1467 square images of
DLBCL and 1656 square images of non-DLBCL are from hospital B, 204 DLBCL
images and 198 non-DLBCL images are from hospital C. 80$\%$ of the dataset are
used as the training set, 10$\%$ are used as the validation set and the last
10$\%$ are used as the testing set. Finally, a diagnostic rate close to
100$\%$ is obtained and the result shows that the platform has the efficiency
of clinical diagnosis.

In[178], faster Region-based CNNs (R-CNNs) are used to classify different
images of lymphomas. In the process of faster R-CNN, the input image is first
represented as a tensor, then the tensor is used as the input of a pre-trained
CNN for the purpose of generating a feature map. Next, the region proposal
network is used to compute the feature for obtaining the region proposal
network. The region network may contain the target. The dataset includes
images of lymphocytes, lymphoma cells and blasts and is divided into 60$\%$
training set, 20$\%$ validation set and 20$\%$ test set. Finally, the result
of detection rate higher than 96$\%$ is obtained which shows the method has a
potential in the diagnosis of lymphoma.

In[160], a CNN is applied to classify Burkitt lymphoma and DLBCL. The
architecture of the CNN is shown in Fig. 40, there are 78 convolutional layers
that are divided into three equal modules. A total of 10818 images of BL and
DLBCL are collected in the experiment, an AUC of 0.92 and 94$\%$ cases are
correctly classified. The results show that the proposed CNN is a promising
tool for classifying BL and DLBCL.

Figure 40: The architecture of the proposed CNN. The figure corresponds to
Figure 2 in [160].

In[45], a new CNN is proposed to classify histopathology images. The CNN model
is comprised of 15 convolutional and two fully connected layers. The PCam
dataset is used for testing the CNN and an AUC of 95.46$\%$ is obtained.

In[135], a novel CNN is proposed to classify the malignant lymphomas subtype.
The CNN combines some frameworks including multiple-instance, multi-scale and
domain adversarial. 196 clinical samples of malignant lymphoma are used in the
method, and the result of ACC higher than 0.871 wins other methods. The result
shows that the method is efficient in classifying malignant lymphomas.

Table 14: Deep Learning Methods for Classification. Reference | Year | Team | Details
---|---|---|---
[172] | 2013 | B Arora | A hidden layer with 10 neurons
[158] | 2017 | MF Jamaluddin | Deep learning model based on LeNet
[143] | 2018 | T Wollmann | CycleGAN and desely connected deep neural network
[187] | 2019 | H El Achi | CNN
[111] | 2019 | H Zhu | VGG16
[89] | 2019 | J Bai | google inception net
[167] | 2019 | N Brancati | Residual CNN
[182] | 2019 | D Li | Transfer learning platform with 17 CNNs
[178] | 2020 | B Sheng | Faster R-CNN
[45] | 2020 | I Kandel | CNN
[135] | 2020 | N Hashimoto | CNN

###  6.3 Summary

From the reviewed works, SVM and KNN are the commonly used traditional
classification methods in LHIA. Deep learning classification methods are the
mainstream methods and achieve good results, and the CNN network is the most
commonly used deep learning method.

##  7 Detection Methods

In this section, we introduce the detection tasks in LHIA. The detection tasks
of LHIA include the detection of malignant cells, CB, follicles and
lymphocytic infiltration. In many cases, the detection tasks are carried out
through classification methods.

###  7.1 Traditional Machine Learning based Detection Methods

In[115], SVM is used to identify lymphocytic infiltration in breast cancer. In
[120], a method for automatically detecting cancer in the gastric lymph node
is proposed (as described in Sec. 5.3). SVM is used as a classifier for
classification. In [122], SVM successfully predicts the status of lymph node
metastasis in breast carcinoma, 101 patients participate in the experiment and
a correct prediction rate equalling 88.4$\%$ is obtained.

In[121], a method is developed, which is applied for automatically detecting
lymphocytic infiltration in H$\&$E stained images of breast cancer. The steps
of preprocessing and feature extraction are described in Sec. 3.6 and Sec.
5.1. The dataset includes 47 slides (training set includes 10 slides,
validation set includes 37 slides), 40 slides are randomly selected for
testing. SVM is applied for classification, and the highest accuracy of
93.23$\%$ is obtained.

In[93], a framework is proposed to identify lymphocytes and CBs. SVM-RFE is
selected for classification after the steps of segmentation, feature
extraction and feature selection. SVM-RFE uses SVM as a classifier and
recursively eliminates features that are irrelevant. Two datasets are used,
the first dataset includes 10 H$\&$E stained breast cancer images, the second
dataset includes 5 H$\&$E stained FL images. Ten trials are performed through
five-fold cross-validation in the experiment, lymphocytes are identified in
the first dataset and CBs are identified in the second dataset. Finally, a
series of results (Dice = 0.73, Overlap (equals to JAC) = 0.57, SEN = 0.57,
SPE = 1, PPV = 1, HD = 4.58, MAD = 0.77) are obtained in the first dataset,
and the results of SEN = 0.38, FPR = 82.85 are obtained in the second dataset.
The results show that the framework has a good performance in identifying
lymphocytes and CBs. Fig. 41 shows the lymphocytes identified in breast cancer
images. Fig. 42 shows CBs identified in FL images.

Figure 41: First row: manually annotations. Second row: lymphocytes identified
by the framework. The figure corresponds to Fig. 2 in [93].

Figure 42: First row: manually annotations. Second row: The CBs identified by
the framework. The figure corresponds to Fig. 3 in [93].

In[130], a method for automatically detecting malignant cells in FL images is
proposed (as described in Sec. 5.5). The LDA classifier is used for
classification and an average of 82.58$\%$ of malignant cells are detected.
In[101], lymphocytes are detected by an algorithm that combines region growing
and Markov random field (MRF).

Table 15: Traditional Detection Methods. Reference | Year | Team | Details
---|---|---|---
[115] | 2008 | A Basavanhally | SVM
[101] | 2009 | AN Basavanhally |  | An algorithm which combines
---
region growing and MRF.
[93] | 2010 | J Cheng | SVM-RFE
[120] | 2014 | T Ishikawa | SVM
[130] | 2014 | E Mchail | LDA
[121] | 2014 | YL Kuo | SVM
[122] | 2015 | MD Zarella | SVM

###  7.2 Deep Learning based Detection Methods

In[140], a deep neural network model is designed for detecting lymphocytes in
H$\&$E stained images. The model is an extension of FCN (as shown in Fig. 43)
and includes two parts: encoder component and decoder component. In the FCN,
each encoder block (red boxes) uses a residual learning function to process
images and transforms images into upper scale, then the highest level
abstraction is extracted by the bridge block, the decoder block restores the
extracted abstraction. In the reference, an extension of FCN is developed: (1)
the information is decoded in the bridge block. (2) Rich features are retained
in decoder blocks. (3) Pooling layers are replaced by convolution layers. (4)
Shortcut link is set in each encoder block. (5) Dropout layers are set in
blocks of encoder and decoder. Two datasets are used in the experiment, the
first dataset includes 100 fields of view (FOV), the second dataset includes
7335 image patches. Fig. 44 shows an example of detection result.

Figure 43: The whole architecture of the network. The figure corresponds to
Fig. 1 in [140].

Figure 44: Left: Original image. Middle: A probability map. Right: The
visualization result. The color column shows the confidence score of
detection. The figure corresponds to Fig. 3 in [140].

In[124], a framework for detecting cancer metastasis in images of Camelyon16
is proposed. In the method, a deep convolutional neural network is trained
with millions of training patches to classify tumor and normal patches. Then,
tumor probability heatmaps are constructed for the localization task and the
classification task based on the slide. Fig. 45 shows the Visualization result
of tumor detection. The GoogLeNet is selected as the deep neural network in
the classification module. GoogLeNet has 27 layers and more than 6 million
parameters, with characteristics of a fast speed and stability. Finally, an
AUC equalling 0.925 and a tumor localization score equalling 0.733 are
obtained. The framework wins the tasks in the challenge of Camelyon16.

Figure 45: Visualization result of tumor detection. The figure corresponds to
Figure. 3 in [124].

In[125], a deep CNN is trained with millions of patches from the Camelyon16
dataset. The steps of preprocessing and feature extraction are described in
Sec. 3 and Sec. 5. AlexNet is selected in the classification module to
discriminate tumor and normal tissue tiles. AlexNet has five convolutional
layers, three fully-connected layers, 60 million parameters. Fig. 46 shows the
architecture of the AlexNet layer. Finally, a sensitivity equalling 0.96, a
specificity equalling 0.89 and an AUC equalling 0.90 are obtained. The
evaluation result shows the framework is efficient.

Figure 46: The architecture of AlexNet layer. The figure corresponds to Fig. 2
in [125].

In[150], a method for identifying and classifying cancer in the dataset of
Camelyon17 is developed. ResNeXt architecture with an inputs size of
224$\times$224$\times$3 is selected to initially detect breast cancer
metastases of lymph nodes through a tumor probability map. ResNeXt is a deep
CNN with 101 layers and has a characteristic of stability and high efficiency.
Next, an ensemble learning approach is applied for two classification tasks
including 5 types of pN-stage of lymph node metastases. Finally, an average
kappa score of 99.5$\%$ is obtained in the three-class classification task
through five fold cross validation, an average kappa score of 89.57$\%$ is
obtained in the two-class classification task through five fold cross
validation.

In[189], a framework is proposed to automatically detect and localize tumors
of small pixels with 100$\times$100 in microscopy images with a size of
100,000$\times$100,000. A CNN architecture called InceptionV3 is utilized to
carry out the detection task in the Camelyon16 dataset. InceptionV3 can
process more and richer space features and increase feature diversity. In the
experiment, 92.4$\%$ of tumors are detected. The result surpasses the result
of 82$\%$ detected by the previous method and the result of 73$\%$ detected
manually. Besides, an AUC of more than 97$\%$ is obtained in the Camelyon16
dataset. These results show that the proposed method can reduce the FNR of
metastasis detection.

In[151], a deep learning framework called neural conditional random field
(NCRF) is developed to detect metastasis in lymph nodes. Each WSI is divided
into small patches, and the spatial correlations of neighboring patches are
considered by NCRF through a fully connected CRF. The CRF is incorporated on
the top of the feature extraction module of CNN. Fig. 47 shows the
architecture of NCRF. Finally, the framework is applied on the Camelyon16
dataset and an average FROC score equalling 0.8069 is obtained. The result is
higher than the previous best result of 0.8074 reported in [124].

Figure 47: The architecture of NCRF. The figure corresponds to Figure 1 in
[151].

In[156], a deep learning model is proposed to identify FL. The model improves
the identification through transfer learning with fine-tuning. The model
increases the accuracy from 58$\%$ to 100$\%$ in a private dataset.

Table 16: Deep Learning Methods for Detection. Reference | Year | Team | Details
---|---|---|---
[140] | 2016 | J Chen | Extension of FCN
[124] | 2016 | D Wang | GoogLeNet
[125] | 2016 | R Chen | AlexNet
[150] | 2017 | K Xiao | ResNeXt
[189] | 2017 | Y Liu | InceptionV3
[156] | 2019 | UV Somaratne | Transfer learning

###  7.3 Summary

The reviewed works show that SVM and $k$nn are the commonly used traditional
detection methods of LHIA. Deep learning-based detection methods are the main-
stream methods and achieve good results, and CNN is the most commonly used
deep learning method.

##  8 Methodology Analysis

This section selects some more suitable methods from each task for LHIA
analysing.

###  8.1 Analysis of Image Preprocessing in LHIA

In image preprocessing of LHIA, colour-based techniques are widely used. In
addition to these commonly used methods, other methods can also preprocess the
image well for subsequent experiments. In [94], the EM-based method is useful
for analyzing other biomedical images. In [88, 157], the method of dividing an
image into blocks is suitable for relatively large images in order to reduce
the space complexity of feature extraction and extract different local
features. In [106, 145, 146], the region-based method can remove isolated
tissue areas in the image, so the image with many noises in the background can
be preprocessed with this method. In [165, 166, 167, 156, 96, 45, 168, 163,
135], data augmentation methods are used to avoid the issue of biases and
overfitting, Data enhancement methods are used to avoid bias and overfitting
problems, but in addition to these, this method can also solve the problem of
too few images in the dataset.

###  8.2 Analysis of Image Segmentation in LHIA

In the process of image segmentation in LHIA, the threshold-based and
$k$-means clustering methods are widely used. In [171], optimal adaptive
thresholding is used to minimize under- and over-segmented nuclei. The
sensitivity of 100$\%$ and maximum precisions of 100$\%$ show the method is
useful in segmenting histopathology images. However, the method of global and
local threshold only obtains a precision of 91.79$\%$ in [170]. Adaptive
thresholding can also obtain a good result in segmenting liver cysts [190].
The threshold segmentation method has the advantages of simple calculation,
high calculation efficiency, fast speed, and easy implementation. It has been
widely used in applications where calculation efficiency is important. It has
a better segmentation effect on images with large contrast between the target