∎

11institutetext:
11email<EMAIL_ADDRESS>33institutetext:
1Information Technology University, Lahore, Pakistan
2University of North Carolina at Charlotte, USA
3Chughtai Institute of Pathology, Lahore, Pakistan
4Ittefaq hospital, Lahore, Pakistan

# A Dataset and Benchmark for Malaria Life-Cycle Classification in Thin Blood
Smear Images

Qazi Ammar Arshad 1 Mohsen Ali1 Saeed-ul Hassan1 Chen Chen2 Ayisha Imran 3
Ghulam Rasul 4 Waqas Sultani 1

(Received: date / Accepted: date)

###### Abstract

Malaria microscopy, microscopic examination of stained blood slides to detect
parasite Plasmodium, is considered to be a gold-standard for detecting life-
threatening disease malaria. Detecting the plasmodium parasite requires a
skilled examiner and may take up to 10 to 15 minutes to completely go through
the whole slide. Due to a lack of skilled medical professionals in the
underdeveloped or resource deficient regions, many cases go misdiagnosed,
which result in unavoidable medical complications. We propose to complement
the medical professionals by creating a deep learning-based method to
automatically detect (localize) the plasmodium parasites in the photograph of
stained film. To handle the unbalanced nature of the dataset, we adopt a two-
stage approach. Where the first stage is trained to detect blood cells and
classify them into just healthy or infected. The second stage is trained to
classify each detected cell further into the malaria life-cycle stage. To
facilitate the research in machine learning-based malaria microscopy, we
introduce a new large scale microscopic image malaria dataset. Thirty-eight
thousand cells are tagged from the 345 microscopic images of different Giemsa-
stained slides of blood samples. Extensive experimentation is performed using
different Convolutional Neural Networks on this dataset. Our experiments and
analysis reveal that the two-stage approach works better than the one-stage
approach for malaria detection. To ensure the usability of our approach, we
have also developed a mobile app that will be used by local hospitals for
investigation and educational purposes. The dataset, its annotations, and
implementation codes will be released upon publication of the paper.

††journal: Neural Computing and Applications

##  1 INTRODUCTION

Out of 400 species of the Anopheles mosquito only 60 species carry plasmodium,
a parasite that causes malaria. Of that, only a female mosquito’s bite can
transfer the parasite to humans. According to the World Health Organization,
only in 2018, 228 million cases of malaria occurred worldwide and there were
405,000 deaths globally due to malaria world2019world .. Malaria-causing
mosquito, Anopheles, is found in tropical and subtropical regions including
Africa, Latin, America, and Asia; and adversely affects many resource-deprived
regions. African continent shares $90\%$ of the deaths caused by malaria, with
$68\%$ of the Ethiopian population and $97\%$ of the Democratic Republic of
the Congo lives in areas at risk of malaria. alemu2017performance ;
mukadi2013external .

Although rapid testing kits are becoming more common, the standard method of
diagnosis of malaria is a microscopic analysis of the stained blood films or
slides abbas2018machine ; devi2018hybrid ; mukadi2013external . Being a low-
cost and simple technique, it’s a vital tool for detecting both malarial
parasites and their density. Treatment in the early-stage being vital for
avoiding medical complications rely heavily on early diagnosis. Examination of
slides is taxing and difficult since a small portion of the slide is visible
through microscope and parasite itself is of small size and available in low-
density world2015methods ”, resulting in 10-15 minutes for a thorough
examination. Adding to the time constraint, low-quality equipment or not
sufficiently skilled medical personnel malera2011research  further adversely
affect the quality of malaria microscopy analysis, leading to inaccurate
results ashraf2012developing ; wongsrichanalai2007review . For the successful
recovery of a patient, it is vital to diagnose and treat malarial infection
early. Named as among diseases of the poor, malaria mostly affects the poorest
communities of the world, where preventive measures are not affordable and
medical treatment is not readily available.

We propose a method to supplement the medical professionals in increasing the
accuracy and decreasing the time required for the examination. Once the slides
are prepared, the images could be taken from either the microscopic camera or
using the mobile phone camera through the microscope’s eye-piece. Malaria
cells will be detected, counted, and classified by our algorithm. The output
of our algorithm shown by overlaying results on the image can help medical
professionals to analyze decisions made by the algorithm. With the help of our
such technology even a less trained expert, not surprisingly common in
developing countries ashraf2012developing ; wongsrichanalai2007review , can
detect malaria parasites in the field prepared blood slides.

In this paper, we present a two-stage approach for malaria detection and
malaria life-cycle-stage classification. During the first stage, given the
image captured by a microscopic camera or mobile phone, microscopic cells are
efficiently segmented using morphological operation and watershed algorithm.
This is followed by the extraction of deep convolution features for malaria
versus non-malaria classification. Once the malaria cells are detected, in the
second stage, we employ another deep classifier for the malaria stage
classifications.

Currently, no large public microscopic image dataset annotating malaria is
available, especially one from the developing country. We have collected
samples from a local hospital in Lahore, Pakistan, and got them annotated from
the expert. Our dataset contains P.vivax malaria species in four different
life cycle stages including Ring, Schizont, Trophozoite, Gametocyte. Finally,
to make our approach to be used in practical scenarios, we have also developed
a mobile app that makes our approach user friendly. Extensive experiments are
conducted to handle the bias introduced due to the class imbalance in the
dataset. The experimental results indicate the efficacy of our approach.

##  2 Related Work

The automatic analysis of medical images, i.e., the images containing
microscopic cells AppliedSoftComputingCells ; AppliedSoftComputingCells2 ;
abbas2018machine ; devi2018hybrid ; khashman2012investigation , MRI
AppliedSoftComputingMR ; AppliedSoftComputingMR2 , CT-scans
AppliedSoftComputingCT ; AppliedSoftComputingCT2  etc., can have huge impact
on accuracy and efficiency of diagnostics. Due to this, several recent works
have devised automatic methods for medical image analysis; including the
images containing the malaria parasites.

Devi et al. devi2018hybrid  introduced a hybrid classifier that is a
combination of three different classifiers (SVM, kNN, and ANN) and used it for
the life-cycle stage classification of malaria cells. Their results show that
their proposed hybrid classifier performs better than the individual
classifier. Gloria et al. diaz2009semi  adopted a two-stage approach and used
histogram features for the classification of malaria cells. The first stage is
used to classify healthy and malaria cells, while the second stage is used to
identify the in-between malaria stages (ring, trophozoite, gametocyte, and
schizont). Abbas et al. abbas2018machine  proposed a color segmentation
through an adaptive algorithm of Gaussian mixture model for the classification
of healthy and malaria-infected cells in thin blood smear and estimated the
level of parasitemia. Khashman .khashman2012investigation  used convolutional
neural networks for the classification of blood cells for microscopic images.
Their neural network model classified the blood cells into three classes i.e.,
red blood cells, white blood cells, and platelets. Kiskin et al
.kiskin2020bioacoustic  developed an algorithm to detect mosquitoes from
acoustic data and showed that audio recording classification through CNN
employing wavelet transformations achieve better results as compared to the
classifiers that use handcrafted features.

Di Ruberto et al. di2002analysis  introduced automatic detection and
classification of Giemsa stained blood cells. They employed a morphological
operation for cell segmentation and enhanced cell roundness by disk base
structuring elements. Tek et al. tek2006malaria  used weighted k-nearest
neighbor to classify Giemsa stained blood cells. Rao et al. rao2002automatic
used mathematical morphological operation on Giemsa stained blood slides to
analyze the different life cycle stages of malaria species. Sio et al.
sio2007malariacount  introduced a software named MalariaCount that counts the
number of infected blood cells. They used binary morphology to classify
between malaria vs healthy cells. Their experimental results on P. falciparum-
infected blood slides demonstrated the automatic calculation of the value of
parasitemia. However, their approach does not address the classification of
different plasmodium species and the life cycle stage of specific malaria
species. Tek et al. tek2010parasite  presented a framework that used a color
normalization technique to improve the segmentation of blood cells. After that
they extracted cells from these slides, classify them as infected or healthy
and also identify the species and life cycle stage of plasmodium. Kumarasamy
et al. kumarasamy2011robust  proposed a unique solution for malaria species
and stage classification. A radial basis Support Vector Machine (SVM) was
trained on binary morphology and texture features to identify the infection
stage of the parasite. Linder et al. linder2014malaria  used SVM for malaria-
infected cell identification in Giemsa stained thin blood smear. Parasite
candidate regions were selected based on their color and size. After features
extraction (SIFT, local binary pattern, and local contrast), an SVM is trained
on these features for parasite detection. Bhowmick et al. bhowmick2012computer
proposed to use the watershed algorithm for cell segmentation on scanning
electron microscopic images captured at 2000X magnification. After that, they
extracted different geometric features to train a multi-layer neural network
for the classification task. Fatima et al. fatima2020automatic  used adaptive
threshold and mathematical morphological algorithms for malaria parasite
detection in thin blood films microscopic images. Similarly, Molina et al.
molina2020sequential  designed a sequential classification model in which they
have combined three different classification modules. The first module used
SVM and the second and third modules used Linear Discriminant Analysis (LDA)
for classification tasks.

Due to the resurgence of deep convolutional neural networks, recent years have
witnessed significant progress in different areas of computer vision,
including malaria detection Bibin et al. bibin2017malaria  introduced a deep
learning-based binary classifier for the classification of infected vs non-
infected cells using color and texture features and a deep belief network
(DBN). A hybrid model for malaria-infected blood cell classification was
introduced by Devi et al. devi2018malaria . They employed a combination of
three classifiers (SVM, k-NN, and Naive Bayes) and also introduced an optimal
feature-set which is a combination of some existing and new features. Authors
in vijayalakshmi2019deep  introduced a novel deep model in which they replaced
the last layer of the VGG network with SVM (named as VGG-SVM) and used it for
the classification of plasmodium falciparum-infected blood cells.

Similar to malaria detection in a thin blood smear, several approaches have
been proposed for automatic malaria detection in a thick blood smear. Mehanian
et al. mehanian2017computer  used Convolutional Neural Network (CNN) for the
detection of the malaria parasite in the thick blood smear with sufficient
accuracy. Salamah et al. salamah2019robust  purpose thick blood smear
segmentation technique using intensity slicing and morphological operation for
malaria parasite detection of microscopic images. The proposed method is
robust to noise, artifacts, and intensive variation in blood slides. Yang et
al. yang2019deep  introduced a new publicly available thick blood smear
dataset containing 1819 from 150 patients. Classification of malaria parasites
is performed in two stages. The first stage uses intensity-based Iterative
Global Minimum Screening (IGMS) to generate the parasite candidates and in the
second stage, customized CNN is used to classify the candidate region as
parasite or background.

In contrast to the above approach, we presented a new indigenously collected
dataset containing microscopic images of malaria slides using a thin smear and
put forward a two-stage approach for accurate cell classification. Also, we
provide a mobile app and an educational use-case to make our approach user-
friendly.

Figure 1: The flow diagram of the proposed malaria detection approach.
Processes start with an image captured with a mobile phone/microscopic camera
followed by the pre-processing binary mask generation. After segmenting the
cells using the watershed algorithm, individual cells are fed into our binary
classification network which separates healthy and infected cells. Finally,
the infected cells are fed into another multi-class classification model for
life cycle-stage detection.

##  3 Proposed Malaria Detection Method

Our goal is to detect and localize malaria-infected cells in images that are
captured using either a mobile phone camera or through a camera attached with
a digital microscope. Our proposed approach to tackle this challenging problem
is based on five observations: (1) since the purpose is to facilitate doctor
on time, the proposed method should be efficient and must have high
classification accuracy, (2) in practical scenarios, sometimes doctors just
want to know the presence or absence of malaria, while in some other critical
situations, they are interested to find out actual malaria stage, therefore
two-stage approach which serves both purposes would be preferable, (3) to make
the medical professional comfortable with the method, the approach should be
trained and tested on the locally acquire microscopic data, (4) a user-
friendly interface to employ and interpret the output of the algorithm, (5)
finally, to make new medical practitioners more comfortable with computer
vision-based diagnostics, the approach should help them in their educational
understanding of the problem. To make our paper self-contained and be
reproducible, we describe details of our approach in the following sections.
Specifically, we discuss cell segmentation and the details of the two-stage
approach for malaria-infected cell detection. Finally, we discuss the
experimental results, our mobile app, and the application of our approach for
education purposes.

###  3.1 Cropping the area of interest

Using the mobile phone camera to perform microscopic analysis introduces
various artifacts. One of them is the dark region surrounding the region of
interest, observed when viewing the slide through the mobile camera Fig. 2.a.
Due to the static focal length in most mobile phones, this could not be
removed by camera settings. These darker shaded regions represent the
background and could affect the performance of the segmentation method. Using
the basic image processing techniques, these darker regions are removed. The
first image is converted to the grayscale image and contrast is enhanced by
applying the histogram equalization. The resultant image is converted into the
binary image by a threshold. The threshold is picked using the Otsu method.
The original image and crop darker region according to the largest contour as
shown in Fig 2.a and Fig 2.b respectively. Secondly, we count the ratio of
darker to non-darker pixels in both row and column and crop the image from the
locations where this ratio goes below a certain threshold. Fig 2.c shows the
result of the final image.

Figure 2: Process of removing unwanted darker regions images captured from
microscope through mobile phone: a) shows the unwanted darker region around
the original image captured with a mobile phone mounted over an optical
microscope, b) shows the image after the first pre-processing step in which
image is cropped by finding the largest on the original image, c) represents
the image after applying automatic thresholding method of preprocessing to
completely remove the darker region from the image.  Figure 3: a) shows the
original image captured with a mobile phone/microscopic camera, (b) represents
histogram equalization is applied to the gray image to enhance contrast, (c)
shows a binary mask of the image after applying a set of morphological
operations and, (d) demonstrate labels generated using the watershed
algorithm. Finally, (e) shows segmentation results indicating red blood cells
in a thin blood smear image.

###  3.2 Cell Localization

An important step for the malaria cell analysis is to identify and localize
the cells in the microscopic image, which is predicting the bounding box
around all of the cells in the image. Microscopic cell localization is
performed employing the combination of mathematical morphological operations
and watershed algorithm. Specifically we perform following steps to localize
(Fig. 3).

  1. 1.

Binarization: We first convert the input image into grayscale and apply
histogram equalization on the image to enhance the contrast of blood cells
from the background. After that, we divide our input image into sub-parts and
then apply otsu thresholding otsu1979threshold  to get a binary mask of these
sub-images. In some cases due to non-uniform staining of peripheral blood
slides, light illumination may not be the same in the whole image, therefore,
this division helps in getting different thresholds for foreground and
background in the different regions of the same image. After this, these sub-
images are combined to get a binary mask.

  2. 2.

Noise Removal: To enhance the cell structure, we applied two mathematical
morphological operations. First, we apply an opening operation to remove small
artifacts from the binary image. Second, erosion operation is performed to
separate the clustered cells from each other. Figure. 3.c shows the binary
mask generated after otsu thresholding and morphological operations.

  3. 3.

Segmentation: The watershed algorithm roerdink2000watershed  is applied on
binary masks to generate unique labels for each cell. Finally, we estimate
contour on these labels to get the individual cell localization (Figure. 3.d
and Figure. 3.e). A few cell localization results are shown in Figure 6.

###  3.3 Cell’s infection-stage Recognition

Each localized cell is sent to the classification module, which labels each
detected cell according to the stage of infection. Two strategies are adopted
for the classification module, in the first one, we train a single-stage
classification to predict all the labels including whether the cell is healthy
or not. This strategy, although reasonable, is adversely affected by the
imbalanced distribution of healthy and non-healthy samples of the cells. In
the second one, a two-stage setup is introduced to handle this issue. Below we
detail both designs.

Single-stage Classification (SSC): A multi-label classification network is
trained to classify the cells extracted from the localization step. The
network consist of a Convolutional Neural Network-based feature extractor is
employed to extract the features, which are input to fully-connected layers.
The output layers infer the probability of the cell belonging to four malarial
life-stage classes i.e., ring, trophozoite, schizont, gametocyte, and the
healthy class. To discover the convolution neural network architecture that
performs best for our problem, we have used VGG16, VGG19 simonyan2014very ,
ResNet50v2 he2016deep , DenseNet169, DenseNet201 huang2017densely  and the
architecture proposed in Rajaraman et al.rajaraman2018pre . All these networks
are first trained on imageNet dataset deng2009imagenet  and then fine-tuned on
our dataset.

Normally microscopic slides dataset contains a large number of healthy cells
and very few infected cells. Due to high-class imbalance, the learning process
is biased towards predicting the health cells correctly rather than
differentiating between the stages. To counter the effects of class imbalance,
we design a cascade structure consisting of two stages, as described below.

Two-stage Classification (TSC): To address the limitations of SSC, we propose
a two-stage class classification approach. The first stage performs binary
classification which classifies cells as healthy and malarial infected. After
that, the cells that are indicated as malaria are further fed into a multi-
class classification network to identify the life cycle stage of infected
cells. To handle the situation, where health cells miss-classified as
infected, the second stage also predicts healthy as one of the possible
labels. Figure. 1 shows the architecture of our two-stage method that is a
combination of two different networks.

Similar to SSC, different CNN models, VGG16, VGG19, ResNet50v2, DenseNet169
and DenseNet201 and the architecture proposed by Rajaraman et
al.rajaraman2018pre  are used in binary classification (first stage). For the
training of the second stage, a subset of training data is generated that
contains all malaria classes and randomly sampled healthy cells, such that the
training data is balanced. From the many different versions of the first
stage, we select the best binary classifier to classify the cells into malaria
and then use the second stage to further classify the malaria cells into their
different types. Our experiments demonstrate that the two-stage approach
outperforms the one-stage approach.

Malaria Dataset | Malaria Species |  | Stage
---
Classification
| Classification
---
Labels
| Number of
---
annotated cells
| BBX
---
Localization
| Slide
---
Images
Segmented-Malaria rajaraman2018pre  | P.f | No | Binary | $27,558$ | No | No
MaMic Image Database linder2014malaria  | P.f | No | Binary | $16,991$ | No | Yes
P.vivax (malaria)BBBC041v1  | P.v | Yes | MultiClass | $80,000$ | Yes | Yes
malaria-655boray2010  | P.f, P.v ,P.m, P.o | Yes | Multiclass | 4363 | No | Yes
MP-IDBloddo2018mp  | P.f, P.v ,P.m, P.o | Yes | Multiclass | 840 | No | Yes
IML-Malaria Dataset | P.v | Yes | MultiClass | $38,449$ | Yes | Yes
Table 1: A comparison of different malaria datasets. Binary classification
labeled datasets only describe cells as healthy or malaria while multi-class
datasets provide labels of different life cycle stages of plasmodium. whereas
P.f represents Plasmodium falciparum, P.v represents Plasmodium vivax, P.m
represents Plasmodium malariae and P.o represents Plasmodium ovale.

##  4 Datasets for Automatic Detection of Malaria-parasite

Machine learning models in general and deep learning in specific are highly
dependent upon good quality annotated datasets of reasonable size. We found
currently available datasets lacking and hence collected a large one to help
facilitate the research in this important direction. Before describing our
malaria dataset, we brieﬂy review the existing microscopic blood smear dataset
for malaria detection.

Segmented-Malaria rajaraman2018pre : This dataset contains $27,558$ individual
cell images of P. falciparum which are manually annotated by trained experts.
This dataset contains an equal number of healthy and infected cells where the
images of thin blood films stained with Giemsa are captured with a mobile
phone mounted on a microscope. The dataset contains individual images of blood
cells which limits its usage in the real-world application where more focus is
on malarial cell detection and localization.

MaMic Image Database linder2014malaria : This dataset contains images of blood
slides that are used to identify P. falciparum. Images are captured with a
microscopic camera that scans the whole slide and captured around 549 images
of a single slide. Every single image has 1280 $\times$ 1024 pixels
resolution. After that, these images are stitched together and compressed into
ECW (Enhanced Compressed Wavelet) format. These ECW file formats are uploaded
to an image management platform server WebMicroscope
111http://demo.webmicroscope.net/SlideCollections that provide a whole-slide
view. For analysis, these images can be download as 1500 $\times$ 1500 pixels
tiles in PNG format. The annotation task of blood cells is done by a trained
expert that labels the cells as certain parasites or uncertain parasites. This
dataset is relatively small consisting of only $16,991$ blood cells in total.

P.vivax (malaria) BBBC041v1 : This dataset consists of 1364 images of thin
blood smear stained with Gamesa. The dataset contains around $80,000$ blood
cells. The annotation contains cell-level bounding boxes annotations of the
healthy and infected cell. Also, the dataset contains annotations of the cells
at different life cycle stages including ring, trophozoite, schizont, and
gametocyte

Malaria-655 boray2010 : This dataset contains 655 images of nine different
thin blood slides that are prepared separately from each other at different
time. To prepare their dataset, the blood smear is fixed on the slide with
methanol and stained with Giemsa stain. This dataset includes images of four
different species of malaria including P.falciparum, P.vivax, P.ovale, and
P.malariae. SP200 research microscope with a mounted Canon A60 camera is used
for capturing images. Images are captured at a 100x objective lens. The images
in this dataset are of 1600 $\times$ 1200 pixels resolution. Data is annotated
by the trained experts and the annotation contains 669 parasite images and
3431 healthy cell images. This dataset, however, has only a dot at the center
of the cell to represent its location instead of the bounding box. This
information is not enough to train the object detection algorithms to extract
the cells for further processing.

MP-IDB loddo2018mp : This dataset contains four different genus of malaria
parasites including P.vivax, P.falciparum, P.ovale and P. malariae. Each genus
in the dataset has four distinct life cycle stages of plasmodium. Dataset is
acquired at 100x magnification with Leica DM2000 optical laboratory microscope
coupled with a built-in camera. It consists of only 229 images with a
2592$\times$1944 pixels resolution that represents four genus of plasmodium.
Each image has at least one stage of the malaria parasite in it. It has
$48,000$ blood cells and images are labeled by expert pathologists. They
provide image-level labels that indicate the presence of a malaria parasite
and the name of the image contains the information of the life cycle stage of
the parasite in a specific image.

Limitations of existing dataset: Each of the above-mentioned datasets has its
limitations: Segmented-Malaria rajaraman2018pre  dataset contains individual
images of blood cell without image level labels or information. MaMic image
database linder2014malaria  dataset contains a small number of blood cells. P.
vivax (malaria) BBBC041v1  is collected from a developed world and does not
necessarily represents our indigenous hematology, Malaria-655 boray2010  lacks
bounding box level annotations and similarly MP-IDB loddo2018mp  only provides
image-level labels. Table 1 demonstrates the detailed comparisons between
existing and the proposed dataset.

Our Dataset–IML-Malaria:

Figure 4: This figure shows different stage of malaria including ring,
trophozoite, schizont and gametocyte in IML-Malaria dataset. The top row shows
healthy cells. Please see Section 1.1 for the details of different malaria
stages. Figure 5: Examples of images in IML-Malaria dataset. Slide images look
different to each others, due to the effect of different staining conditions.
The malaria stages shown as a) Trophozoite and Ring stage b) Schizont stage c)
Ring and Gametocyte stage. Blue circles in (d) show different regions
containing artifacts in the blood slide.

We collected blood samples from malaria-infected people living in the province
of Punjab, Pakistan. Pakistan being an agricultural country, having warm and
humid weather, and a large irrigation network is among regions with a high
risk of malarial infection kakar2010malaria . Like many developing countries,
malaria control efforts have been hampered due to a lack of expensive
laboratory instruments, and, more importantly, skilled technicians and doctors
specifically in the remote areas qureshi2019occurrence . Every year, on
average, 500,000 confirmed cases of malarial infection and 50,000 deaths are
caused by it are reported in Pakistan khan2019malaria . The reasons are
agriculture practices, monsoon season rains and vast irrigation network
kakar2010malaria . Another reason for malaria infection in Pakistan is
frequent floods qureshi2019occurrence . Two major species causing malaria in
Pakistan are P.vivax and P.falciparum. The extensive research done by Qureshi
et al. qureshi2019occurrence  shows that 81.3% cases were caused by P. vivax,
14.7% by P. falciparum, and the remaining 4% cases are caused by a mixed
infection in Pakistan. Pakistan is facing many challenges in malaria control
due to misdiagnosis, lack of expensive laboratory instruments, and, more
importantly, skilled people specifically in remote areas qureshi2019occurrence
.

We have collected a new malaria dataset that is captured with an XSZ-107
series microscope. Globally 49% cases of malaria qureshi2019occurrence ;
alias2014spatial  are reported due to P.vivax, therefore, we concentrate on
collecting images of microscopic slides infected by genus of plasmodium. Blood
smears were prepared by spreading thin blood film on the slide and were left
to dry. After fixing the blood smear through methanol, the slide is stained
using Giemsa solution for 15 - 20 minutes. Finally, the cover-slips are
permanently fixed on the slide to keep them safe. Our thin-film blood slides
are prepared by trained experts in labs. Images are captured with the help of
a camera mounted on a microscope at 100x 222The actual magnification is 100
$\times$ 10 (eye-piece) objective magnification. Images are annotated and
captured by an expert hematologist, at a local institute. The dataset contains
345 images consisting of $111$ blood cells on average, both healthy and ones
infected by the parasite. We annotate the infected cells into classes per
life-cycle stage, that is, Ring, Trophozoite, Schizont, and Gametocyte as
shown in Figure. 4. In addition to this bounding box for each cell is also
annotated. It must be noted that trophozoite and schizont are very rare
loddo2018mp  and so they are present in very low numbers in our data
collection as well. Comparison between different datasets are shown in Table
1. Figure 5 some examples of images in our dataset.

##  5 Experiments and Results

We evaluate our proposed approach on two levels of microscopic analysis, the
first one is cell localization and another one is stage classification.

###  5.1 Cell Localization

As our CNN are trained on individual cells so we first need to localize blood
cells for classification. As mentioned in section 3.2, we apply the watershed
segmentation algorithm with a combination of morphological operations to
localize cells in the blood slide image. The cell is considered localized
(true positive) if the intersection over union (IOU) between predicted and
ground truth cell bounding box is greater than 50% otherwise the predicted box
is considered false positive. Similarly, the cells which are not detected by
the algorithm are considered as a false negative. Table.2 shows the results of
cell localization on our dataset. From the table, it can be seen that 34,367
blood cells are accurately detected with 91.14% $F_{1}$-Score.

Figure 6: Result of cell localization on IML-Malaria dataset. The blue and red boxes show the ground truth predicted cell bounding boxes respectively. Dataset | TP | FP | P | R |  $F_{1}$-score
---|---|---|---|---|---
| IML-
---
Malaria
34367 | 4082 | 0.94 | 0.89 | 0.91
Table 2: Localization results on IML-Malaria dataset. TP, FP, P and R
represents true positive, false positive, precision and recall respectively.

###  5.2 Malarial life-cycle stage Classification

Plasmodium parasites have different life-cycle phases including ring,
trophozoite, schizont, and gametocyte stage. The life cycle stage helps in
measuring the degree of the parasite inside the patient’s blood. Our IML-
Malaria dataset is similarly labeled, including the label healthy for the
cells without infection. We randomly divide the dataset into training (70%),
testing (20%), and validation (10%) set. All the base-networks used, were pre-
trained on imageNet deng2009imagenet  dataset before plugged into our pipeline
and fine-tuned on the malaria microscopy task. In our experiment we use the
same setting for all the networks (Figure. 1), i.e., using the convolution
layer of these state of the art networks and then adding two fully connected
layers at the end of each network. Accuracy of trained single SSC model and
TSC model over the testing dataset are given in Table.3 and Table.5. To
compute the average accuracy, we first compute class-wise accuracy and then
divide the sum of these accuracies by the number of classes. Below we discuss
the performance of SSC and TSC separately.

Single Stage Classification (SSC) Results : Results of SSC with different
backbone networks are compared in Table.3. CNN proposed by rajaraman2018pre
shows the average accuracy of 41%, whereas DenseNet201 outperforms with an
average accuracy of 74%. Although the average accuracy appears reasonable, the
confusion matrix, Figure. 7, indicates that the results are skewed by the
healthy cells.

| Avg. Accuracy |  $F_{1}$-score
---|---|---
Rajaraman et al. rajaraman2018pre  | 46.1% | 51.26%
VGG-16 | 55.35% | 52.40%
VGG-19 | 70.44% | 76.48%
ResNet50v2 | 72.86% | 76.44%
DenseNet169 | 73.91% | 77.62%
DenseNet201 | 74.56% | 75.52%
Table 3: Single Stage multi-class classification results on IML-Malaria
dataset. Among different CNN models used, DenseNet201 has the highest
classification accuracy. Figure 7: Single stage multi-class classification
confusion matrix of DenseNet201 that outperforms in single stage setting on
test set of IML-Malaria dataset.

Two-Stage Classification Both average accuracy and $F_{1}$ score improves
(Table.5) when two-stage model is applied over the test data. As indicated by
the confusion matrix, Figure 8, the prediction accuracy improves for the
Gametocyte and Trophozoit, with a small improvement in detection of the Ring
stage. However, Schizont does not show much improvement. This could be
associated with a very small number of Schizont samples in the training and
testing data.

We perform extensive experiments to investigate how different design decisions
affect the overall accuracy of the model. First-stage (binary stage), is
trained using the different backbones and tested on the validation set. It was
found that ResNet50v2 has higher accuracy than other backbone models. Keeping
ResNet50v2 as the backbone for the first stage, we trained our second stage
with different backbone networks. Table 4 shows the accuracies of binary
classifications and the model with ResNet50v2 backbone shows the best accuracy
on the test data too.

Table.5 shows the result of two-stage classification on our malaria dataset.
The combination of ResNet50v2 at stage one and stage two shows the top average
accuracy of 79.61%. Table.6 shows the comparison of single-stage and two-stage
classification. From Table.6, it can be seen that the accuracy of each network
is improved in two-stage settings. Looking at the results in the context of
Table 4, one can see the benefit of using the first stage to balance out the
samples coming to the second stage for evaluations.

Figure 8: Two-stage multi-class classification confusion matrix which use ResNet50v2 at both stages. | Avg. Accuracy |  $F_{1}$-score
---|---|---
Rajaraman et al. rajaraman2018pre  | 84.93% | 76.27%
VGG-16 | 83.66% | 75.23%
VGG-19 | 94.79% | 90.12%
ResNet50v2 | 95.63% | 89.91%
DenseNet121 | 94.77% | 88.60%
DenseNet169 | 95.61% | 89.16%
DenseNet201 | 95.18% | 89.45%
Table 4: Classification results of stage-1 for different CNN models on IML-Malaria dataset. ResNet50v2 shows the best accuracy for binary classification. | Avg. Accuracy |  $F_{1}$-score
---|---|---
Rajaraman et al. rajaraman2018pre  | 59.03% | 60.59%
VGG-16 | 79.04% | 80.72%
VGG-19 | 71.1% | 76.05%
ResNet50v2 | 79.61% | 82.04%
DenseNet121 | 78.11% | 79.40%
DenseNet169 | 76.35% | 79.08%
DenseNet201 | 77.24% | 79.27%
Table 5: Two-stage multi-class classification results on IML-Malaria dataset. ResNet50v2 is used in the first stage for binary classification and then all CNN models are used for multi-class classification. Combination of ResNet50v2 at both stages shows the highest accuracy. |  | Avg. Accuracy
---
Single Stage
| Avg. Accuracy
---
Two-Stage
Rajaraman et al. rajaraman2018pre  | 46.1% | 59.03%
VGG-16 | 55.35% | 79.04%
VGG-19 | 70.44% | 71.1%
ResNet50v2 | 72.86% | 79.61%
DenseNet169 | 73.91% | 76.35%
DenseNet201 | 74.56% | 77.24%
Table 6: Comparison of single stage and two-stage classification: For all CNN
models, two-stage classification outperforms one-stage classification
accuracies. Figure 9:  Snapshot of our mobile application through one can
efficiently count and localize total and malaria infected cell in an image
captured through a mobile camera mounted over the microscope.

##  6 Mobile Application

To make our solution accessible and usable in practical scenarios, we have
developed a mobile application. The mobile app provides a user-friendly
interface to capture images from a microscope using a mobile camera and
achieve the total cell counts, number of malaria cells, and visualization of
each malaria cells on the fly. Note that all processing related to CNN is done
on the server and the mobile app is just used as an interface. To use the App
in the field, the user just needs to connect his mobile to the internet.

Figure 10: Blood slide image stitching that give a global view of the whole
slide. Red circles shows the malaria infected blood cells.

The details of the mobile app and malaria detection process through it is as
follows: The mobile app is developed in Swift-5 using the Xcode IDE.
Storyboard is used for app design and Alamofire API alamofire  is employed for
http networking. Server APIs are built using the Django REST framework django
that connects mobile apps with python server. Images are captured through a
smartphone attached to a conventional light microscope. Then, the mobile app
sends the image to our server that contains segmentation and classification
code for cell counting, detection of malaria-infected blood cells, and
classification of the malaria life cycle stage. All of these results are shown
to users on mobile phones. This type of application will help many field
technicians to save their time and efforts mostly in the regions where malaria
is endemic. Our application provides a complete and end-to-end solution for
detecting malaria with conventional light microscopes. Figure. 9 shows a brief
overview of our mobile application.

##  7 Educational Use-case

Constructing an interface for easy acquisition of the microscopic images
provides us an opportunity to represent a complete slide as one. Manual slide
examination required mechanical movement and focus adjustment repeatedly which
is a very hectic task. We implement a stitching mechanism, generating a global
slide view as shown in Figure 10.Such representation is not only helpful for
the practitioners but can be vital in educating the medical students and
retraining the professionals. Training the medical practitioners and students
on such recent technology will make them comfortable in using the technology
to augment their expertise. This module could be extended to microscopic
analysis other than malaria too.

##  8 Conclusion

In this paper, we provide a machine learning-based mechanism for automatic
malaria microscopy. Malaria results in four hundred and five thousand
fatalities every year from two hundred and twenty-eight million infected, many
from the poor regions of the world. Machine learning-based intervention is
necessary to counter the low-quality equipment and less-trained staff being
used for diagnosis through blood slides and increase the speed of diagnosis
since timely diagnosis is vital for a speedy recovery. We have collected a new
large dataset of malaria microscopic images which help training deep learning-
based research on this problem. A Convolutional Neural Network-based, two-
stage pipeline is proposed for accurate malaria cell stage recognition. The
cascaded nature of our architecture helps counter the effects of the
imbalanced nature of the dataset. To keep the computational cost low, instead
of the deep learning-based method, a watershed algorithm is used with the
morphological operations to localize the cells. Extensive experiments were
performed to measure performance. The mean accuracy of the selected model was
$79.61\%$ accuracy and $82.04\%$. To make our approach to be easily used by
medical practitioners, a new user-friendly mobile-based application is built
which helps in detecting and localizing malaria cells in real-time. Our
experimental results show that the two-stage approach performs better than the
one-stage approach for the malaria life cycle stage classification. We are in
the process of employing the mobile app on a large scale to verify its
efficacy.
Declaration of Competing Interest: We wish to confirm that there are no known
conflicts of interest associated with this publication and there has been no
significant financial support for this work that could have influenced its
outcome. We confirm that the manuscript has been read and approved by all
named authors and that there are no other persons who satisfied the criteria
for authorship but are not listed. We further confirm that the order of
authors listed in the manuscript has been approved by all of us. We confirm
that we have given due consideration to the protection of intellectual
property associated with this work and that there are no impediments to
publication, including the timing of publication, with respect to intellectual
property. In so doing we confirm that we have followed the regulations of our
institutions concerning intellectual property. We understand that the
Corresponding Author is the sole contact for the Editorial process (including
Editorial Manager and direct communications with the office). He/she is
responsible for communicating with the other authors about progress,
submissions of revisions, and final approval of proofs. We confirm that we
have provided a current, correct email address which is accessible by the
Corresponding Author.
Acknowledgement The project is partially supported by an unrestricted gift
award from Facebook, USA. The opinions, findings, and conclusions or
recommendations expressed in this publication are those of the author(s) and
do not necessarily reflect those of Facebook.

## References

  * (1) Abbas, N., Saba, T., Mohamad, D., Rehman, A., Almazyad, A.S., Al-Ghamdi, J.S.: Machine aided malaria parasitemia detection in giemsa-stained thin blood smears.  Neural Computing and Applications 29(3), 803–818 (2018)
  * (2) Alamofire.  https://github.com/Alamofire/Alamofire
  * (3) Alemu, M., Tadesse, D., Hailu, T., Mulu, W., Derbie, A., Hailu, T., Abera, B.: Performance of laboratory professionals working on malaria microscopy in tigray, north ethiopia.  Journal of parasitology research 2017 (2017)
  * (4) Alias, H., Surin, J., Mahmud, R., Shafie, A., Zin, J.M., Nor, M.M., Ibrahim, A.S., Rundi, C.: Spatial distribution of malaria in peninsular malaysia from 2000 to 2009.  Parasites & vectors 7(1), 186 (2014)
  * (5) Ashraf, S., Kao, A., Hugo, C., Christophel, E.M., Fatunmbi, B., Luchavez, J., Lilley, K., Bell, D.: Developing standards for malaria microscopy: external competency assessment for malaria microscopists in the asia-pacific.  Malaria journal 11(1), 352 (2012)
  * (6) Image set BBBC041v1.  https://data.broadinstitute.org/bbbc/BBBC041/
  * (7) Bhowmick, S., Das, D.K., Maiti, A.K., Chakraborty, C.: Computer-aided diagnosis of thalassemia using scanning electron microscopic images of peripheral blood: a morphological approach.  Journal of Medical Imaging and Health Informatics 2(3), 215–221 (2012)
  * (8) Bibin, D., Nair, M.S., Punitha, P.: Malaria parasite detection from peripheral blood smear images using deep belief networks.  IEEE Access 5, 9099–9108 (2017)
  * (9) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database.  In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee (2009)
  * (10) Devi, S.S., Laskar, R.H., Sheikh, S.A.: Hybrid classifier based life cycle stages analysis for malaria-infected erythrocyte using thin blood smear images.  Neural Computing and Applications 29(8), 217–235 (2018)
  * (11) Devi, S.S., Roy, A., Singha, J., Sheikh, S.A., Laskar, R.H.: Malaria infected erythrocyte classification based on a hybrid classifier using microscopic images of thin blood smear.  Multimedia Tools and Applications 77(1), 631–660 (2018)
  * (12) Di Ruberto, C., Dempster, A., Khan, S., Jarra, B.: Analysis of infected blood cell images using morphological operators.  Image and vision computing 20(2), 133–146 (2002)
  * (13) malERA Consultative Group on Diagnoses, Diagnostics: A research agenda for malaria eradication: diagnoses and diagnostics.  PLoS medicine 8(1), e1000396 (2011)
  * (14) Díaz, G., González, F.A., Romero, E.: A semi-automatic method for quantification and classification of erythrocytes infected with malaria parasites in microscopic images.  Journal of Biomedical Informatics 42(2), 296–307 (2009)
  * (15) Django REST framework.  https://www.django-rest-framework.org
  * (16) Fatima, T., Farid, M.S.: Automatic detection of plasmodium parasites from microscopic blood images.  Journal of Parasitic Diseases 44(1), 69–78 (2020)
  * (17) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)
  * (18) Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708 (2017)
  * (19) Kakar, Q., Khan, M., Bile, K.: Malaria control in pakistan: new tools at hand but challenging epidemiological realities.  EMHJ-Eastern Mediterranean Health Journal, 16 (Supp.), 54-60, 2010 (2010)
  * (20) Khan, W., Rahman, A.U., Shafiq, S., Ihsan, H., Khan, K.: Malaria prevalence in malakand district, the north western region of pakistan.  JPMA 69(946) (2019)
  * (21) Khashman, A.: Investigation of different neural models for blood cell type identification.  Neural Computing and Applications 21(6), 1177–1183 (2012)
  * (22) Kiskin, I., Zilli, D., Li, Y., Sinka, M., Willis, K., Roberts, S.: Bioacoustic detection with wavelet-conditioned convolutional neural networks.  Neural Computing and Applications 32(4), 915–927 (2020)
  * (23) Konar, D., Bhattacharyya, S., Gandhi, T.K., Panigrahi, B.K.: A quantum-inspired self-supervised network model for automatic segmentation of brain mr images.  Applied Soft Computing (2020)
  * (24) Kumarasamy, S.K., Ong, S., Tan, K.S.: Robust contour reconstruction of red blood cells and parasites in the automated identification of the stages of malarial infection.  Machine Vision and Applications 22(3), 461–469 (2011)
  * (25) Linder, N., Turkki, R., Walliander, M., Mårtensson, A., Diwan, V., Rahtu, E., Pietikäinen, M., Lundin, M., Lundin, J.: A malaria diagnostic tool based on computer vision screening and visualization of plasmodium falciparum candidate areas in digitized blood smears.  PLoS One 9(8) (2014)
  * (26) Loddo, A., Di Ruberto, C., Kocher, M., Prod’Hom, G.: Mp-idb: the malaria parasite image database for image processing and analysis.  In: Sipaim–Miccai Biomedical Workshop, pp. 57–65. Springer (2018)
  * (27) Lu, Y., Qin, X., Fan, H., Lai, T., Li, Z.: Wbc-net: A white blood cell segmentation network based on unet++ and resnet.  Applied Soft Computing (2021)
  * (28) Mehanian, C., Jaiswal, M., Delahunt, C., Thompson, C., Horning, M., Hu, L., Ostbye, T., McGuire, S., Mehanian, M., Champlin, C., et al.: Computer-automated malaria diagnosis and quantitation using convolutional neural networks.  In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 116–125 (2017)
  * (29) Mittal, M., Goyal, L.M., Kaur, S., Kaur, I., Verma, A., Jude Hemanth, D.: Deep learning based enhanced tumor segmentation approach for mr brain images.  Applied Soft Computing (2019)
  * (30) Molina, A., Alférez, S., Boldú, L., Acevedo, A., Rodellar, J., Merino, A.: Sequential classification system for recognition of malaria infection using peripheral blood cell images.  Journal of Clinical Pathology (2020)
  * (31) Mukadi, P., Gillet, P., Lukuka, A., Atua, B., Sheshe, N., Kanza, A., Mayunda, J.B., Mongita, B., Senga, R., Ngoyi, J., et al.: External quality assessment of giemsa-stained blood film microscopy for the diagnosis of malaria and sleeping sickness in the democratic republic of the congo.  Bulletin of the World Health Organization 91, 441–448 (2013)
  * (32) Organization, W.H., et al.: Methods for surveillance of antimalarial drug efficacy. 2009.  Geneva, Switzerland (2015)
  * (33) Organization, W.H., et al.: World malaria report 2019 (2019)
  * (34) Otsu, N.: A threshold selection method from gray-level histograms.  IEEE transactions on systems, man, and cybernetics 9(1), 62–66 (1979)
  * (35) Qureshi, N.A., Fatima, H., Afzal, M., Khattak, A.A., Nawaz, M.A.: Occurrence and seasonal variation of human plasmodium infection in punjab province, pakistan.  BMC infectious diseases 19(1), 935 (2019)
  * (36) Rajaraman, S., Antani, S.K., Poostchi, M., Silamut, K., Hossain, M.A., Maude, R.J., Jaeger, S., Thoma, G.R.: Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images.  PeerJ 6, e4568 (2018)
  * (37) Rao, K.M.a., Dempster, A., Jarra, B., Khan, S.: Automatic scanning of malaria infected blood slide images using mathematical morphology (2002)
  * (38) Roerdink, J.B., Meijster, A.: The watershed transform: Definitions, algorithms and parallelization strategies.  Fundamenta informaticae 41(1, 2), 187–228 (2000)
  * (39) Salamah, U., Sarno, R., Arifin, A.Z., Nugroho, A.S., Rozi, I.E., Asih, P.B.S.: A robust segmentation for malaria parasite detection of thick blood smear microscopic images.  Int. J. Adv. Sci. Eng. Inf. Technol. 9(4), 1450–1459 (2019)
  * (40) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * (41) Sio, S.W., Sun, W., Kumar, S., Bin, W.Z., Tan, S.S., Ong, S.H., Kikuchi, H., Oshima, Y., Tan, K.S.: Malariacount: an image analysis-based program for the accurate determination of parasitemia.  Journal of microbiological methods 68(1), 11–18 (2007)
  * (42) Tek, F.B., Dempster, A.G., Kale, I.: Malaria parasite detection in peripheral blood images.  BMVA (2006)
  * (43) Tek, F.B., Dempster, A.G., Kale, I.: Images of thin blood smears with bounding boxes around malaria parasites (malaria-655).  Computer Vision and Image Understanding (2010).  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2719653/
  * (44) Tek, F.B., Dempster, A.G., Kale, I.: Parasite detection and identification for automated thin blood film malaria diagnosis.  Computer vision and image understanding 114(1), 21–32 (2010)
  * (45) Toğaçar, M., Ergen, B., Cömert, Z.: Classification of white blood cells using deep features obtained from convolutional neural network models based on the combination of feature selection methods.  Applied Soft Computing (2020)
  * (46) Vijayalakshmi, A., et al.: Deep learning approach to detect malaria from microscopic images.  Multimedia Tools and Applications pp. 1–21 (2019)
  * (47) Wang, B., Jin, S., Yan, Q., Xu, H., Luo, C., Wei, L., Zhao, W., Hou, X., Ma, W., Xu, Z., Zheng, Z., Sun, W., Lan, L., Zhang, W., Mu, X., Shi, C., Wang, Z., Lee, J., Jin, Z., Lin, M., Jin, H., Zhang, L., Guo, J., Zhao, B., Ren, Z., Wang, S., Xu, W., Wang, X., Wang, J., You, Z., Dong, J.: Ai-assisted ct imaging analysis for covid-19 screening: Building and deploying a medical ai system.  Applied Soft Computing (2021)
  * (48) Wongsrichanalai, C., Barcus, M.J., Muth, S., Sutamihardja, A., Wernsdorfer, W.H.: A review of malaria diagnostic tools: microscopy and rapid diagnostic test (rdt).  The American journal of tropical medicine and hygiene 77(6_Suppl), 119–127 (2007)
  * (49) Yang, F., Poostchi, M., Yu, H., Zhou, Z., Silamut, K., Yu, J., Maude, R.J., Jaeger, S., Antani, S.: Deep learning for smartphone-based malaria parasite detection in thick blood smears.  IEEE Journal of Biomedical and Health Informatics 24(5), 1427–1438 (2019)
  * (50) Zhou, T., Lu, H., Yang, Z., Qiu, S., Huo, B., Dong, Y.: The ensemble deep learning model for novel covid-19 on ct images.  Applied Soft Computing (2021)