# The Lagrangian remainder of Taylor’s series, distinguishes
$\mathcal{O}(f(x))$ time complexities to polynomials or not

Nikolaos P. Bakas, Elias Kosmatopoulos, Mihalis Nicolaou, , Savvas A.
Chatzichristofis Computation-based Science and Technology Research Center, The
Cyprus Institute, 20 Konstantinou Kavafi Street, 2121, Aglantzia Nicosia,
Cyprus. e-mail<EMAIL_ADDRESS>University of Thrace, Electrical
and Computer engineering. e-mail<EMAIL_ADDRESS>Science and Technology Research Center, The Cyprus Institute, 20 Konstantinou
Kavafi Street, 2121, Aglantzia Nicosia, Cyprus. e-mail:
<EMAIL_ADDRESS>Systems Lab & Department of Computer Science,
Neapolis University Pafos, 2 Danais Avenue, 8042 Pafos, Cyprus. e-mail:
<EMAIL_ADDRESS>
## Abstract

The purpose of this letter is to investigate the time complexity consequences
of the truncated Taylor series, known as Taylor Polynomials [1, 2, 3]. In
particular, it is demonstrated that the examination of the $\mathbf{P=NP}$
equality, is associated with the determination of whether the $n^{th}$
derivative of a particular solution is bounded or not. Accordingly, in some
cases, this is not true, and hence in general.

##  1 Univariate complexity

######  Definition 1.

Let the given problem is a known analytic function $f$ of one variable
$x\in\mathbf{Z}^{+}$. Initially, the authors consider one-dimensional $x$, and
later they generalize the results. Respectively, the time complexity of the
given problem, according to the literature [4], may be written in the generic
form of:

$\mathcal{O}(f(x)).$

The execution time is usually calculated by some elementary algebraic
operations of integers or real numbers [5], thus, this assumption is
considered to be adequate and valid. Accordingly, the Taylor series expansion
of $f$ at $x+x_{0}$ may be written by

$f(x)=f({{x}_{0}})+\frac{{f}^{\prime}({{x}_{0}})}{1!}(x-{{x}_{0}})+\frac{{f}^{\prime\prime}({{x}_{0}})}{2!}{{(x-{{x}_{0}})}^{2}}+\cdots+\frac{{{f}^{(n)}}({{x}_{0}})}{n!}{{(x-{{x}_{0}})}^{n}}+\cdots,$

with infinite terms, that is

$f(x)={{T}_{n}}(x)+{{R}_{n}}(x),$ (1)

where ${{T}_{n}}(x)$ is the Taylor polynomial of order $n$, and ${{R}_{n}}(x)$
the remainder of the $n^{th}$ degree Taylor polynomial.

If the function $f$ is analytic, it has $n+1$ derivatives for each point in
the interval $\left|x-{{x}_{0}}\right|\leq r$. Moreover, if the derivatives
for each point are in the interval
$\mid{{f}^{\left(n+1\right)}}\left(x\right)\mid\leq M$, then, the Lagrangian
form [6, 7] of the remainder may be expressed by:

$|{{R}_{n}}(x)|\leq\frac{M}{(n+1)!}|x-{{x}_{0}}{{|}^{n+1}}\forall
x:|x-{{x}_{0}}|\leq r.$ (2)

This is subsequent of the explicit form of the remainder, stating that there
exist a $\xi$ among $x$, and $x_{0}$, such that

${{R}_{n}}(x)\leq\frac{f^{(n)}(\xi)}{(n+1)!}(x-{{x}_{0}}{{)}^{n+1}}.$

######  Theorem 1.

If $\mid{{f}^{\left(n+1\right)}}\left(x\right)\mid\leq M$, the algorithm with
$\mathcal{O}(f(x))$ complexity, runs in polynomial time.

###### Proof.

Equations 1 and 2 obtain that:

$f(x)={{T}_{n}}(x)+{{P}_{n+1}}(x),$

thus,

$\mathcal{O}(f(x))=x^{n+1}.$

which apparently is a polynomial, and hence of class $\mathbf{P}$. ∎

######  Lemma 1.

If $\mid{{f}^{\left(n+1\right)}}\left(x\right)\mid>M$, $f$ cannot be expressed
as polynomials.

###### Proof.

By utilizing Borel’s theorem [8], stating that any formal series
$\sum_{n=0}^{\infty}a_{n}\left(x-x_{0}\right)^{n}$ is the Taylor series of a
$C^{\infty}$-smooth function defined in an open neighborhood of $x_{0}$, it is
derived that if $f^{(n)}$ is not bounded, $f$ cannot be written as a power
series, and hence as a polynomial, thus the problem is not in $\mathbf{P}$. In
other words, if the problem was in $\mathbf{P}$, it could be written as a
polynomial, and this would be the Taylor series, which is absurd as no $n$
exist such that the $f^{(n)}$ is limited by a $M$. ∎

##  2 Multiple dimensions

The given problem might have execution time depending on two or more variables
$\mathbf{x}=\\{x_{1},x_{2},\dots,x_{m}\\}\in\mathbf{Z}^{m+}$, hence is is a
function $f$ of a variable $\mathbf{x}$, that is

$\mathcal{O}(f(\mathbf{x})).$

The above mentioned procedure may be applied to define if the complexity of
the algorithm is polynomial, for each dimension $i\in\\{1,2,\dots,m\\}$,
concerning the others as constants. If for one dimension
$\mathcal{O}(f({x_{i}}))$ is not a polynomial, then
$\mathcal{O}(f(\mathbf{x}))$ is not.

##  3 Examples

The presented analysis of computational time with Taylor series constitutes a
basis for the investigation of whether a given complexity of
$\mathcal{O}(\cdot)$ is a polynomial or not.

###  3.1 Example of $\left|f^{(n)}(x)\right|>M$

####  3.1.1 $f(x)=e^{x}$

In this simple case, one can easily observe that $f^{(n)}(x)=e^{x}\forall n$,
which is not bounded by any $M$ for all $x$.

$r=\lim_{n\rightarrow\infty}\left|{\frac{a_{n}}{a_{n+1}}}\right|=\lim_{n\to\infty}\frac{\frac{x^{n}}{n!}}{\frac{x^{n+1}}{{n+1}!}}=\lim_{n\to\infty}\frac{n+1}{x}=\infty,$

and the series converges everywhere.

####  3.1.2 $f(x)=2^{x}$

A variety of algorithms execute in times calculated by a number, raised to the
$x^{th}$ power, for example the matrix chain multiplication via brute-force
search is $2^{x}$, a well as the lower bounds for the $\mathbf{AC}^{0}$
problem [9]. In this case:

$\frac{df}{dx}=2^{x}log(2),$ $\frac{d^{2}f}{dx^{2}}=2^{x}log^{2}(2),$

and hence

$\frac{d^{n}f}{dx^{n}}=2^{x}log^{n}(2),$

which is not bounded by any $M$ for all $x$.

Apparently,

${{2}^{x}}=1+\frac{x\log(2)}{1!}+\frac{{{x}^{2}}{{\left(\log(2)\right)}^{2}}}{2!}+\cdots+\frac{{{x}^{n}}{{\left(\log(2)\right)}^{n}}}{n!}$

$r=\underset{n\to\infty}{\mathop{\lim\inf}}\,\frac{1}{\sqrt[n]{|{{a}_{n}}|}}=\underset{n\to\infty}{\mathop{\lim\inf}}\,\frac{\sqrt[n]{n!}}{x\left(\log(2)\right)}=\infty,$

or equivalently,

$r=\underset{n\to\infty}{\mathop{\lim}}\,\left|\frac{{{a}_{n}}}{{{a}_{n+1}}}\right|=\frac{\frac{{{x}^{n}}{{\left(\log(2)\right)}^{n}}}{n!}}{\frac{{{x}^{n+1}}{{\left(\log(2)\right)}^{n+1}}}{(n+1)!}}=\lim_{n\to\infty}\frac{(n+1)}{x\log(2)}=\infty,$

and the series converges everywhere.

####  3.1.3 $f(x)=2^{x^{1/2}}$

E.g. the monotone circuits computing [10]. This is an example which though
seems exponential, it is not verified if it indeed in not a polynomial (in
contrast with 3.2.2).

$\frac{d}{dx}2^{x^{1/2}}=\frac{2^{x^{1/2}-1}log(2)}{x^{1/2}},$

and hence,

$\underset{x\to\infty}{\lim}\frac{d}{dx}2^{x^{1/2}}=\infty.$

####  3.1.4 $f(x)=x^{\log(x)}$

Any monotone formula for the computation of a monotone function in NP must
have size at least $\Omega(x^{\log(x)})$ [11, 12].

$\frac{d}{dx}{x^{\log_{2}(x)}}=\frac{2x^{\frac{log(x)}{log(2)}-1}log(x)}{log(2)},$

where $log$ is the natural logarithm, and hence,

$\underset{x\to\infty}{\lim}\frac{d}{dx}x^{\log(x)}=\infty.$

Accordingly,

$\frac{\partial^{n}x^{\log_{2}(x)}}{\partial
x^{n}}=(-1)^{n}x^{\log_{2}(x)-n}\left(-\log_{2}(x)\right)_{n}\text{/;}n\in\mathbb{Z}\land
n\geq 0\land x\log(2)\neq\log(x)$

and $(\cdot)_{n}$ is the Pochhammer Symbol, with

$(\xi)_{n}=(\Gamma(\xi+n))/(\Gamma(\xi))=(\xi+1)\dots(\xi+n-1).$

Thus $\left(-\log_{2}(x)\right)_{n}$, never vanishes for $x>1$. Additionally,

$\lim_{x\to\infty}\,x^{\log(x)/\log(2)-n}=\infty,$

an hence the $n^{th}$ derivative of $f$ is not bounded. The alliterating sign
$(-1)^{n}$ results in $+/-\infty$ for the limit of
$\frac{\partial^{n}x^{\log_{2}(x)}}{\partial x^{n}}$.

###  3.2 Examples of $\left|f^{(n)}(x)\right|\leq M$

####  3.2.1 $f(x)=xlog(x)$

This is a commonly resultant time (e.g. comparison sort).

$\frac{df}{dx}=log(x)+x\frac{1}{x},$ $\frac{d^{2}f}{dx^{2}}=\frac{1}{x},$

which is $\leq M,\forall x>1$, even for the second derivative ($n=2$).

In this case, $x\in\mathbf{Z}^{+}$, and hence, the radius of convergence is
required to be $>0$. For $x>\frac{1}{2}$:

$log(x)=\sum_{n=1}^{\infty}\frac{\left(\frac{x-1}{x}\right)^{n}}{n},$

and hence

$xlog(x)=\sum_{n=1}^{\infty}x\frac{\left(\frac{x-1}{x}\right)^{n}}{n},$

thus, it is obtained that:

$r=\lim_{n\to\infty}|\frac{a_{n}}{a_{n+1}}|=\lim_{n\to\infty}\frac{x\frac{\left(\frac{x-1}{x}\right)^{n}}{n}}{x\frac{\left(\frac{x-1}{x}\right)^{n+1}}{n+1}}=\lim_{n\to\infty}\frac{n+1}{n\frac{x-1}{x}}=\frac{x}{x-1}>1\forall
x\in\mathbf{Z}^{+}.$

####  3.2.2 $f(x)=2^{\log_{2}(x)}$

$\frac{d2^{\log_{2}(x)}}{dx}=1.$

####  3.2.3 $f(x)=2^{\log_{2}\left(\log_{2}(x)\right)}$

Results to:

$\frac{d2^{\log_{2}\left(\log_{2}(x)\right)}}{dx}=\frac{1}{x\log(2)},f^{(2)}(x)=-\frac{1}{x^{2}\log(2)},$

$f^{(3)}(x)=\frac{2}{x^{3}\log(2)},f^{(4)}(x)=-\frac{6}{x^{4}\log(2)},f^{(5)}(x)=\frac{24}{x^{5}\log(2)},$

thus

$\frac{d^{n}2^{\log_{2}\left(\log_{2}(x)\right)}}{dx^{n}}=(-1)^{n-1}\frac{(n-1)!}{x^{n}\log(2)}$

and hence,

$\lim_{x\to\infty}\sup f^{(n)}(x)=\lim_{x\to\infty}\inf f^{(n)}(x)=0;\forall
n\geq 1.$

##  4 Conclusion

P versus NP problem is to determine whether every problem whose solution can
be verified in polynomial time, can be also solved in polynomial time [13]. In
this letter, the authors highlighted that for a given complexity of
$\mathcal{O}(f(x))$ for the solution of a problem, under certain criteria for
a $n^{th}$ derivative of $f$, $f^{(n)}(x)$, this problem cannot be considered
as a polynomial. We do not consider non-analytic, non differentiable problems,
which anyway do not affect the obtain conclusion. Furthermore, some specific
$\mathbf{NP}$ problems, have known exponential lower bounds, e.g. [14, 15, 10,
11, 12]. In some of these cases, the $n^{th}$ derivative of $f$ is not
bounded. Hence, in such cases, their solution can be quickly verified, in
polynomial time, but on the other hand, by utilizing the aforementioned
rationale, the lower bound of their solution cannot be expressed with a
polynomial function of computational time, and hence cannot belong to class
$\mathbf{P}$. Accordingly, the authors derive that in these cases (and hence
in general), the proposed rationale, apart from distinguishing polynomial or
not algorithms, based on existing lower bounds, might answer the question if

$\mathbf{P}\neq\mathbf{NP}.$

## References

  * [1] N. P. Bakas, “Taylor polynomials in high arithmetic precision as universal approximators,” _arXiv preprint arXiv:1909.13565_ , 2019. [Online]. Available: https://arxiv.org/abs/1909.13565
  * [2] E. S. Katsoprinakis and V. N. Nestoridis, “Partial sums of Taylor series on a circle,” _Annales de l’institut Fourier_ , 2011.
  * [3] V. Nestoridis, “Universal Taylor series,” _Annales de l’institut Fourier_ , 2011.
  * [4] M. Sipser _et al._ , _Introduction to the Theory of Computation_. Thomson Course Technology Boston, 2006, vol. 2.
  * [5] D. S. Johnson and M. R. Garey, _Computers and intractability: A guide to the theory of NP-completeness_. WH Freeman, 1979.
  * [6] M. Kline, _Calculus: an intuitive and physical approach_. Courier Corporation, 1998.
  * [7] T. M. Apostol, “Calculus. 1967,” _Jon Wiley & Sons_, 1967.
  * [8] R. Narasimhan, _Analysis on real and complex manifolds_. Elsevier, 1985, vol. 35.
  * [9] R. Impagliazzo, R. Paturi, and F. Zane, “Which problems have strongly exponential complexity?” _Journal of Computer and System Sciences_ , vol. 63, no. 4, pp. 512–530, 2001.
  * [10] N. Alon and R. B. Boppana, “The monotone circuit complexity of boolean functions,” _Combinatorica_ , vol. 7, no. 1, pp. 1–22, 1987.
  * [11] A. A. Razborov, “Applications of matrix methods to the theory of lower bounds in computational complexity,” _Combinatorica_ , vol. 10, no. 1, pp. 81–93, 1990.
  * [12] R. Robere, T. Pitassi, B. Rossman, and S. A. Cook, “Exponential lower bounds for monotone span programs,” in _2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS)_. IEEE, 2016, pp. 406–415.
  * [13] S. Cook, “The p versus np problem,” _The millennium prize problems_ , pp. 87–104, 2006.
  * [14] E. Ukkonen, “Exponential lower bounds for some np-complete problems in a restricted linear decision tree model,” _BIT Numerical Mathematics_ , vol. 23, no. 2, pp. 181–192, 1983.
  * [15] S. Kikot, R. Kontchakov, V. Podolskii, and M. Zakharyaschev, “Exponential lower bounds and separation for query rewriting,” in _International Colloquium on Automata, Languages, and Programming_. Springer, 2012, pp. 263–274.


# Dermo-DOCTOR: A framework for concurrent skin lesion detection and
recognition using a deep convolutional neural network with end-to-end dual
encoders

Md. Kamrul Hasan11footnotemark: 1 Department of Electrical and Electronic
Engineering, Khulna University of Engineering & Technology, Khulna-9203,
Bangladesh  Department of Biomedical Engineering, Khulna University of
Engineering & Technology, Khulna-9203, Bangladesh  Department of Computer
Science and Engineering, Khulna University of Engineering & Technology,
Khulna-9203, Bangladesh<EMAIL_ADDRESS>Shidhartho Roy
<EMAIL_ADDRESS>Chayan Mondal<EMAIL_ADDRESS>Md. Ashraful Alam
<EMAIL_ADDRESS>Md. Toufick E Elahi<EMAIL_ADDRESS>Aishwariya Dutta<EMAIL_ADDRESS>S. M. Taslim Uddin Raju
<EMAIL_ADDRESS>Md. Tasnim Jawad<EMAIL_ADDRESS>Mohiuddin Ahmad<EMAIL_ADDRESS>
###### Abstract

### Background and Objective

Automated skin lesion analysis for simultaneous detection and recognition is
still challenging for inter-class homogeneity and intra-class heterogeneity,
leading to low generic capability of a Single Convolutional Neural Network
(CNN) with limited datasets.

### Methods

This article proposes an end-to-end deep CNN-based framework for simultaneous
detection and recognition of the skin lesions, named Dermo-DOCTOR, consisting
of two encoders. The feature maps from two encoders are fused channel-wise,
called Fused Feature Map (FFM). The FFM is utilized for decoding in the
detection sub-network, concatenating each stage of two encoders’ outputs with
corresponding decoder layers to retrieve the lost spatial information due to
pooling in the encoders. For the recognition sub-network, the outputs of three
fully connected layers, utilizing feature maps of two encoders and FFM, are
aggregated to obtain a final lesion class. We train and evaluate the proposed
Dermo-Doctor utilizing two publicly available benchmark datasets, such as
ISIC-$2016$ and ISIC-$2017$.

### Results

The achieved segmentation results exhibit mean intersection over unions of
$85.0\,\%$ and $80.0\,\%$ respectively for ISIC-$2016$ and ISIC-$2017$ test
datasets. The proposed Dermo-DOCTOR also demonstrates praiseworthy success in
lesion recognition, providing the areas under the receiver operating
characteristic curves of $0.98$ and $0.91$ respectively for those two
datasets. The experimental results show that the proposed Dermo-DOCTOR
outperforms the alternative methods mentioned in the literature, designed for
skin lesion detection and recognition.

### Conclusion

As the Dermo-DOCTOR provides better-results on two different test datasets,
even with limited training data, it can be an auspicious computer-aided
assistive tool for dermatologists.

###### keywords:

Malignant melanoma , Skin lesion detection and recognition , Convolutional
neural networks , Dual encoder networks , ISIC skin lesion datasets.

††journal: Biomedical Signal Processing and Controllabel4label4footnotetext:
Department of EEE, KUET, Khulna-9203, Bangladesh.

##  1 Introduction

###  1.1 Problem Presentation

Cancer is an abnormal and uncontrolled growth of dividing cells, damaging
different body cells and contributing to the world’s second-leading cause of
death [76]. Although melanomas constitute less than $5.0\,\%$ of all skin
cancers, they make up about $75.0\,\%$ of deaths related to skin cancer in the
United States (US) alone [18]. Age-standardized melanoma rates of the top $20$
countries [9] is presented in Fig. 1.

(a)  Figure 1: A world heat map of the age-standardized rates per $1.0$
million population of the top $20$ countries with the highest skin melanoma
rates in $2018$ [9], displaying highly skin cancer affected regions.

According to the world health organization, $466914$ new cases of skin cancer
will be diagnosed in $2040$ ($54.27\,\%$ men and $45.73\,\%$ women), and
$105904$ ($58.14\,\%$ men and $41.86\,\%$ women) will die. The five-year
survival rate of melanoma, the deadliest variety of early detection, is as
high as $99.0\,\%$, but delayed diagnosis leads significantly to a colorectal
survival rate decrease of $23.0\,\%$ [84]. However, reliable early
identification is highly imperative as the five-year survival rate will be
increased by $90.0\,\%$ approximately [23]. Dermatologists generally examine
images via naked-eye through visual examination, requiring a high level of
expertise and focus. The manual inspection by dermatologists is often very
tiresome, time-consuming, subjective, and fault-prone. The precision of skin
lesions’ diagnosis by the dermatologists suffers from inter-class homogeneity
and intra-class heterogeneity. Moreover, the ratio of dermatologists per $1.0$
million population in the US, South Australia, and Europe are respectively
$34.0$, $26.0$, and $59.34$, which are very low compared to the required
numbers [15, 16, 26]. However, an automated Computer-aided Screening (CAS)
system has become popular among dermatologists to alleviate the above
limitations, reduce the working burden of dermatologists, and accelerate
diagnosis rates [50]. Such CAS systems essentially consist of several integral
parts, where the segmentation for Region of Interest (ROI) extraction and the
classification for lesion recognition. However, image-based automated CAS
systems are highly challenging for the following hurdles:

  * •

Wide range of intra-class variance in colors, textures, edges, and shapes and
homogeneity in inter-classes.

  * •

Sometimes, low contrasts and unclear boundaries (edges) in the malignant and
other class images.

  * •

Lesion ROI frequently shares similar visual characteristics and subtle
distinctions due to lighting, perspective, and spatial information within an
image.

  * •

The appearance of different artifacts, such as natural (hairs, veins) or
synthetic (air bubbles, ruler lines, color balance charts, marker signs,
paint, ink color, artificial objects, etc.), LED lighting, darker border
(microscopic effects), and non-uniform vignetting, as depicted in Fig. 2.

  * •

Lesion ROI only covers a small proportion of local, subtle grain, and global
context information.

  * •

Unavailability of a large number of manually annotated images, which is the
core requirement of the supervised learning systems.

(a)  Figure 2: An example of the challenging dermoscopic images in ISIC
dataset [13, 29] with different artifacts [35].

###  1.2 Recent Methods

The state-of-the-art methods for skin lesion segmentation and recognition are
reviewed and described in the following two subsections.

####  1.2.1 Methods for Lesion Segmentation

A multi-stage Fully Convolutional Network (mFCN) with parallel integration was
introduced by Bi et al. [8]. During the training process, mFCN learned from
both the training data and the coarse results obtained from the previous
(m-1)FCN stage. The summation of the earlier results with the current result
had two benefits: boost the training data and optimize lesion boundary
learning. Navarro et al. [62] proposed a superpixels adaptation-based
segmentation approach to get tight-to-boundaries of the skin lesions. The
authors applied the Scale-Invariant Feature Transform (SIFT) [54] and Gaussian
distribution to detect the feature points and place these points to the
initial centers. Finally, they applied the simple linear iterative clustering
technique to these points for generating the final lesion masks. An encoder-
decoder network was built by Sarker et al. [71], called SLSDeep. The encoder
in SLSDeep was dilated residual network-based design, while the decoder had a
pyramid pooling network. Additionally, they proposed a combined negative log-
likelihood and endpoint error-based cross-entropy loss function. Jahanifar et
al. [45] developed an improved saliency detection supervised method for the
lesion segmentation, which was designed based on the discriminative regional
feature integration. They used a thresholding algorithm for generating a new
pseudo background region. Goyal et al. [27] designed an automatic ensemble of
DL methods, such as DeeplabV3+ [19] and Mask R-Convolutional Neural Network
(R-CNN) [38], for generating the precise lesion boundaries. They combined the
result of two models in three ways: a combination of both masks, picking the
larger segmented area from the output of both methods, and picking a smaller
area from those outputs. Finally, the authors discovered that the first method
outperforms the other ensembling methods. Al-Masni et al. [2] proposed a Full
Resolution Convolutional Network (FrCN) for the skin lesion segmentation,
which learns full resolution features from each pixel of an input image. A
segmentation method is realized by Hawas et al. [36] for the neutrosophic
graph cut algorithm. The initial clusters were obtained using Histogram-based
Clustering Estimation (HBCE) with the corresponding centroids. The genetic
algorithm was applied to optimize the HBCE for getting the optimal threshold.
Then, the Neutrosophic C-means (NCM) mapped the lesions into a Neutrosophic
Set (NS) domain. Finally, for the lesion segmentation, the graph cut algorithm
was a cost function. Amin et al. [5] segmented the lesion in two steps. In the
first step, the authors performed preprocessing to resize the images to
$240\times 240\times 3$ and convert the RGB into $Lab$ to select the luminance
channel. Finally, in the second step, biorthogonal $2D$ wavelet transform and
OTSU algorithm were applied for the lesion segmentation. Xie et al. [86]
produced a deep CNN, called mutual bootstrapping deep CNN (MB-DCNN). MB-DCNN
has three networks, such as a coarse segmentation network (coarse-SN), a mask-
guided classification network (mask-CN), and an enhanced segmentation network
(enhanced-SN). Coarse-SN was used to roughly segment the lesion and feed the
label region to mask-CN to boost the classification task. Al Nazi and Abir [4]
compared the performance of two variations of the UNet [70] model for the
lesion segmentation, such as UNet without spatial dropout and UNet with
spatial dropout. In the end, the authors showed that augmentation and dropout,
as regularization methods, with UNet, had less prone to overfitting and
provided better-segmented lesion masks. Pour and Seker [67] offered a
segmentation model based on CNN with CIELAB color space and transformed domain
feature extraction. The authors initially implemented a scratch model inspired
by UNet and FCN, then gradually improved the model by injecting features from
the transformed domain and adding the input image color model CIElab. They
succeeded in coping with the constraints that included small data set, removal
of artifacts, excessive data increase, and contrast stretching. They also
confirmed that the CNN model’s performance with a domain transfer feature is
better than the CNNs with a deep layer network.

####  1.2.2 Methods for Lesion Recognition

Many image analysis-based methods have already been proposed and developed by
the researchers for dermoscopic image recognition, where the algorithms
generally depend on the detection and extraction of low-level handcrafted
features, such as colors, shapes, textures, and etc. Cheng et al. [11]
extracted different features from the first‐order histogram probability such
as area, roundness (thinness), mean, standard deviation, skew, energy, and
entropy. Finally, the authors applied different classifiers, namely quadratic
discriminant analysis and Multilayer Perceptron (MLP), on the selected
features by the Principal Components Analysis (PCA) [55]. Different visual
cues, such as ABCD (Asymmetry, Border, Color, and Differential structures)
rule of dermoscopy [61], texture (neighboring gray-level dependence matrix,
angular second-moment, and kinetics of skin lesions) were extracted by
Maglogiannis and Doukas [56]. The authors also selected the features using the
sequential backward floating selection, PCA, and generalized sequential
feature selection algorithms. Finally, they employed MLP and Support Vector
Machine (SVM) [21] for the lesion classification. Oliveira et al. [64]
computed different local features employing the bag-of-features [25] approach
and texture features, where the authors selected the subset of features using
a heuristic search approach. In the end, they applied SVM, Bayesian network,
Decision Tree (DT) [1], and Artificial Neural Network (ANN) as classifiers.
Hameed et al. [31] developed an intellectual Multi-Class Multi-Level (MCML)
classification algorithm employing two approaches, such as traditional machine
learning and deep learning. In the former method, they applied preprocessing,
segmentation, extraction of features, and classification. As a preprocessing,
they removed the hair, black frames, and circle. Finally, the authors
classified the texture and color features employing the ANN. Mporas et al.
[60] applied a median filter followed by bottom-hat filtering to detect
natural hair or similar to hair artifacts. They segmented the ROIs using the
active contour model on the grayscale image. Finally, they extracted different
color-based features for classification using the MLP and other Machine
Learning (ML) algorithms.

However, as described earlier, the lesion classification algorithms are very
complex as they essentially rely on the handcrafted features extraction
method, requiring prior knowledge [20] and lots of parameter tuning. Extensive
feature engineering is the key to achieving better-performance from them,
which is often impossible due to the presence of different artifacts in the
dermoscopic images (see in Fig. 2). The development of various CNN-based
classifiers has achieved a remarkable result on the ImageNet dataset [14].
Nowadays, in many computer vision problems, the contribution of both CNNs and
DL techniques are undeniable [28]. CNN is an excellent feature extractor,
which necessarily alleviates the manual feature engineering as in the
algorithms mentioned above, therefore applying it to recognize medical images
[87]. Mahbod et al. [57] presented an ensemble-based model for CNNs that
combines inter-and intra-architecture network fusion. The authors applied the
fine-tuning of pre-trained VGGNet, AlexNet [48], and two types of ResNet.
Finally, the average prediction probability classification vectors from
different sets were fused to provide the final prediction. Brinker et al. [10]
exercised ResNet-$50$ with transfer learning [82]. For the optimization of the
model, they adopted three techniques. Firstly, they exclusively trained the
adapted last layer, then fine-tuned all layers’ parameters, and finally, a
sudden increment of the learning rate at specific time steps during fine-
tuning. Zhang et al. [91] presented an Attention Residual Learning (ARL) CNN
model for the skin lesion recognition, which was composed of multiple ARL
blocks, a global average pooling, and a classification layer. Each ARL block
employed residual learning and novel attention learning mechanisms to improve
its capability for discriminative representation. The authors proposed the
attention learning mechanism, which aimed to utilize the intrinsic self-
attention ability of DCNNs, i.e., using the feature maps learned from a high
layer to generate a low-layer attention map instead of applying extra
learnable layers. An integrated framework for skin lesion boundary detection
as well as for skin lesions classification was described by Al-Masni et al.
[3]. Firstly, a deep learning method, named FrCN, was used for the lesion
boundary extraction. Then, geometric augmentation and transfer learning were
integrated with four CNN networks, such as Inception-V3 [79], ResNet-$50$,
Inception-ResNet-V2, and DenseNet-201 [42] for the lesion classification. They
also showed that segmented lesions improve lesion classification results.
Yilmaz and Trocan [88] implemented three deep CNN models named AlexNet,
GoogLeNet, and ResNet-$50$. They compared classification performance as well
as time complexity of the implemented models. For data augmentation, a style-
based Generative Adversarial Network (GAN) architecture was proposed by Qin et
al. [68]. In the end, the authors applied ResNet-$50$, with transfer learning,
for the lesion classification. Khan et al. [47] proposed a model for the
lesion classification, which included the localization of lesion ROI via
faster region-based CNN, feature extraction, and feature selection by
iteration-controlled Newton-Raphson method. The ABC-based method was first
used for contrast stretching and then used for lesion segmentation.
DenseNet-$201$, via transfer learning, was used to extract deep-level
features, and those features were classified using an MLP. Gessert et al. [24]
ensembled different DL methods, such as EfficientNets [80], SENet [41], and
ResNeXt, by a selection strategy. They used multi-resolution input by multi-
crop evaluation and two different cropping strategies. The encoding of
metadata as a feature vector was concatenated with the dense (fully connected)
neural network. Valle et al. [83] optimized the hyperparameter of two deep CNN
models, ResNet-101-V2, and Inception-V4 employing transfer learning with data
augmentation. They select the best performing classifier using the ANOVA test
[73]. Finally, the authors concluded that the transfer learning and ensembling
model is a better choice for lesion classification.

###  1.3 Our Contribution

The above-discussions on the automatic skin lesion diagnosis methods confirm
that the deep CNN approaches are commonly applied nowadays than the different
systems relying on handcrafted features. The former approaches provide good
reproducibility of results and boost diagnostic procedures’ speed while being
end-to-end methods. However, the CNN-based skin lesion analysis methods suffer
from data scarcity to evade overfitting. The ensembling of different CNN
architectures can mitigate those CNN’s limitations, as proven by Harangi [32].
In many articles [24, 27, 30, 32, 51, 58, 66, 75], the authors first trained
different CNN models independently and then aggregated their outputs for
developing ensembling models. Such an ensembling is tedious and time-
consuming, leading to massive time and resources for training and testing.
However, to eradicate those limitations, an end-to-end ensemble approach for
skin lesion analysis without compromising state-of-the-art outcomes is highly
essential. With the aforementioned thing in mind, this article aims to provide
the following contributions:

  * •

Develop an end-to-end ensembling model with dual encoders in our Dermo-DOCTOR
framework, concatenating two different feature maps from those two encoders to
broaden the lesion’s depth information. Such a proposed network with two
different encoders with the same input is likely to learn more discriminating
features with limited training samples.

  * •

Incorporate segmented lesion ROIs for the recognition as ROIs enable the
classifier to learn the abstract region and detailed structural description
while avoiding surrounding healthy regions.

  * •

Apply geometry- and intensity-based image augmentations and transfer learning
to alleviate overfitting; the class rebalancing techniques to protect the
classifier from being biased towards any particular class with more samples.

  * •

Develop and compare two other networks for detection (UNet and FCN8s) and
recognition (ResNet-$50$ and Xception [12]) under the same experimental
settings.

  * •

Demonstrate state-of-the-art lesion detection and recognition results, to our
best knowledge, on two IEEE International Symposium on Biomedical Imaging
(ISBI) datasets, such as ISIC-$2016$ and ISIC-$2017$, having a different
number of classes.

  * •

Implement a possible application of our Dermo-DOCTOR, deploying its trained
weights, which runs in a web browser (see in YouTube222Dermo-DOCTOR App:
https://bit.ly/Dermo-DOCTOR).

The rest of the paper is structured accordingly. We explain the design of the
Dermo-DOCTOR framework and datasets in section 2. The results and discussions
of the extensive experiments, with the proper interpretation, are reported in
section 3. Finally, we conclude the article in section 4.

##  2 Materials and Methods

This section manifests the materials and methods, describing the proposed
Dermo-DOCTOR pipeline in Section 2.1. We explain the utilized datasets,
integral preprocessing, and the proposed network in Subsections 2.1.1, 2.1.2,
and 2.1.3, respectively. Sections 2.2 and 2.3 respectively describe our web
application and network’s training protocol.

###  2.1 Proposed Framework

The overall Dermo-DOCTOR framework is illustrated in Fig. 3.

(a)  Figure 3: The proposed pipeline for concurrent detection and recognition
systems, where the preprocessing has incorporated with the proposed network to
build a precise diagnostic system. An input $I_{1}$ or $I_{2}$ generates two
outputs $O_{1}$ and $O_{2}$, where $O_{1}$ and $O_{2}$ respectively denote the
segmentation and recognition results.

We utilize two different input types (either $I_{1}$ or $I_{2}$), where
$I_{1}$ or $I_{2}$ is a binary or a multi-class categorization task. An input,
either $I_{1}$ or $I_{2}$, generates two different outputs, such as
segmentation ($O_{1}$) and recognition ($O_{2}$). The outputs $O_{1}$ and
$O_{2}$ are then processed to provide lesion detection and recognition
results. We process the predicted lesion masks to generate the bounding box
around the lesion, naming lesion detection. However, different crucial
integral parts of the Dermo-DOCTOR are explained in the following subsections.

####  2.1.1 Datasets

Two different datasets, such as ISIC-2016 [29] and ISIC-2017 [13], are used to
validate our proposed pipeline, whose class-wise distributions are presented
in Fig. 4.

(a)  (b)

Figure 4: The distributions of the utilized ISIC datasets, where (a) is for
ISIC-2016 and (b) is for ISIC-2017. The validation set for the ISIC-2016 is
not available publicly (see in the left figure).

The ISIC-2016 contains a binary class, aiming to classify as either Nevus
(Nev) or Melanoma (Mel), explicating that class samples are imbalanced
($4.2:1$ for $Nev:Mel$). On the other hand, the ISIC-2017 is a multi-class
categorization task, intending to classify as either Nevus (Nev), or
Seborrheic Keratosis (SK), or Melanoma (Mel). The distribution of ISIC-2017
also tells that class samples are highly imbalanced, where $Nev:SK:Mel$ in the
training set is $11.8:2.8:1$. The training samples’ imbalanced distribution
makes the classifier biased towards the particular class with more samples,
mitigated in the proposed pipeline by adopting two techniques (see in
subsection 2.1.2).

####  2.1.2 Preprocessing

We have applied class rebalancing and different image augmentations (both
geometry- and intensity-based) as a preprocessing, which are concisely
explained as follows:

Rebalancing. The class imbalance is a common phenomenon in the medical imaging
domain as manually annotated images are very complex and arduous to achieve
[32]. Such a class imbalance can be partially overcome using two commonly used
approaches, such as the data-level method and algorithmic level method [37].
We have combined additional images to the underrepresented class from the ISIC
archive [44] and weighted the loss function. For weighing the loss function,
we apply $W_{i}=N_{i}/N$, where $W_{i}$, $N$, and $N_{i}$ are the weight for
$i^{th}$ class, the total sample numbers, and the sample numbers in the
$i^{th}$ class, respectively.

Augmentation. One of the crucial challenges in the medical imaging domain is
coping with the small datasets, such as in the ISIC datasets [32]. However, we
have applied different augmentations based on geometric transformations, such
as rotation, flipping, shifting, zooming, and image processing functions, such
as gamma, logarithmic, sigmoid corrections, and stretching, or shrinking the
intensity levels.

However, the input $I_{1}$ or $I_{2}$ produces the lesion recognition output
by applying two preprocessing types: the $P_{1}:$ only segmentation and the
$P_{2}:$ rebalancing and augmentation with segmentation.

####  2.1.3 Proposed Network

Nowadays, CNN-based methods outperform the radiologists with high values of
balanced accuracy as proven in [46, 69]. Nevertheless, they trained the models
with an enormous number of annotated images. However, CNNs may be obliquely
limited when employed with highly variable and distinctive image datasets with
limited samples and having inter-class homogeneity and intra-class
heterogeneity, as in dermoscopic ISIC datasets [13, 29]. Ensembling the
network is likely to alleviate data scarcity limitation for the CNN training
[17, 32, 49, 59, 72]. In this context, we propose a CNN-based end-to-end
ensemble network for simultaneous lesion detection and recognition, consisting
of two encoders, a decoder, and three Fully-connected Layers (FCLs), as shown
in Fig. 5. The different parts of the proposed dual encoder network are
explained in the following paragraphs.

(a)  Figure 5: The proposed network for the Dermo-DOCTOR application, where
the first encoder-decoder sub-network is applied for the lesion segmentation,
while the second sub-network is employed for the lesion recognition. The
segmented lesion masks are used for ROI extraction for further classification
and detection utilizing the bounding boxes around the lesions.

Encoder-1. The first encoder ($f^{en-1}$) in the proposed network is presented
in Fig. 6.

(a)  Figure 6: The encoder-1 of the proposed network, where the Conv and Iden
blocks are stacked on top of each other. The notation ($n\times$) under the
Iden block denotes the number of repetitions ($n$-times).

It includes Identity (Iden) and Convolutional (Conv) blocks, applying the skip
connections [40] in both blocks. There are two main advantages of such a skip
connection. Firstly, the lack of regularization of the new layers does not
affect their performance, and secondly, the new layers are not nil even when
they are regulated. In encoder-1, an input convolution is adopted before Iden
and Conv blocks, followed by a max-pooling. By stacking these blocks on top of
each other, encoder-1 has been designed for getting a lesion feature map (see
in Fig. 6). The output feature map of the encoder-$1$ is defined as
$X_{en-1}=f^{en-1}(I_{in})$, where $X_{en-1}\in\mathcal{R}^{B\times H\times
W\times D}$, and $B$, $H$, $W$, $D$, and $I_{in}$ respectively denote the
batch size, height, width, depth (channel), and input batch of images. The
encoder-$1$ is divided into five sub-blocks ($E_{1n}$ and $n=1,2,...,5$),
while the input image resolutions in each sub-block are down-sampled in half
of the input resolutions. In the segmentation sub-network, each sub-block’s
outputs ($E_{1n}$ and $n=1,2,...,5)$ will be used as an input for the skip
connections to regain the lost spatial information due to pooling in the
encoders.

Encoder-2. Within the encoder-2 ($f^{en-2}$), three block components are
employed, such as entry flow, middle flow, and exit flow [12]. Fig. 7 depicts
the constructional details of the encoder-2.

(a)  Figure 7: The encoder-2 of the proposed network, where depth-wise
separable convolutions [12] were employed instead of traditional convolutions
to make it lightweight for real-time applications.

The batch of input images firstly passes through the input flow, then the
central flow, repeated eight times ($8\times$), and finally through the exit
flow. All flows employ Depth-wise Separable Convolution (DwSC) [12] and
residual connections. The former has been used to create a lightweight
network, while the latter has the advantages discussed earlier in encoder-1.
The output feature map of the encoder-2 is defined as
$X_{en-2}=f^{en-2}(I_{in})$, where $X_{en-2}\in\mathcal{R}^{B\times H\times
W\times D}$, and $B$, $H$, $W$, $D$, and $I_{in}$ respectively denotes batch
size, height, width, depth (channel), and input batch of images. The encoder-2
is also divided into five sub-blocks ($E_{2n}$ and $n=1,2,...,5$), in which
input resolutions are also down-sampled into half the resolutions of each sub-
block. Every sub-blocks ($E_{2n}$ and $n=1,2,...,5$) are then used as the skip
connections, when it is decoded in the detection sub-network.

Detection Sub-Network. The decoder semantically projects the salient features
of lower resolution from the encoders onto the pixel space having a higher
resolution to achieve a semantic lesion pixel label [22, 53, 70]. The reduced
feature maps (to attain spatial invariance) from the encoder often cause a
loss in spatial resolution, bringing zigzag edge information, coarseness,
checkerboard artifacts, and over- and under-segmentation in the segmented
masks [33, 35, 53, 63, 70]. Although there are many approaches to alleviate
these problems in the segmentation [2, 6, 33, 35, 53, 63, 70], there is still
room for performance improvement. In our detection sub-network, the obtained
outputs from two encoders are concatenated channel-wise for enlarging the
depth representation of the feature map, which is named as a Fused Feature Map
(FFM), where $FFM\in\mathcal{R}^{B\times H\times W\times 2D}$. We have applied
skip connections, inspired by the UNet, to tackle the subsampling limitations.
The $FFM\in\mathcal{R}^{B\times H\times W\times 2D}$ is an input to the
decoder of our detection sub-network (see in Fig. 8).

(a)  Figure 8: The decoder of the proposed detection sub-network for
reconstructing a segmentation mask with the input resolution from the
encoders’ low-resolution features.

Unlike the earlier networks, we skip the features from two different encoders
to recover the lost spatial information (see in Fig. 8). The channel-
concatenation in each stage of decoder is presented as
$[E_{1n}+\\!\\!\\!\\!+\,E_{2n}+\\!\\!\\!\\!+\,D_{n}]$, where $E_{1n}$,
$E_{2n}$, $D_{n}$, and $+\\!\\!\\!\\!+\,$ respectively denote skipped feature
maps from encoder-1 and encoder-2, decoder feature map (at $n^{ht}$ stage),
and channel concatenation. $E_{1n}$, $E_{2n}$, and $D_{n}$ are the same scaled
feature maps and $n=4,3,...,1$. Such a dual encoder skipped feature has
enhanced depth information, which is likely to improve the segmentation
accuracy by better retrieving the lost spatial information. Besides, we employ
batch normalization [43] to overcome the internal covariate shift in the
training phase. We also compact our network’s design, employing a DwSC [12] in
place of standard convolution. We decrease the parameters by a factor of
$(1/N+1/K^{2})$ for each convolution in our Dermo-DOCTOR, where $N$ and $K$
respectively indicate the filter number and kernel size [34].

Recognition Sub-Network. Different feature maps from the encoder-1, encoder-2,
and FFM are classified into desired categories applying the FCLs. We employ a
Global Average Pooling (GAP) layer [52] before the FCL for vectorizing the 2D
feature maps into a single long continuous linear vector, as it improves
generalization and prevents overfitting [52]. Additionally, each FCL layer is
followed by a dropout layer [78] as a regulariser, where we randomly set
$50.0\,\%$ neurons of the FC layer to zero during the training. However, the
two different feature maps of two different encoders are utilized to recognize
the Lesion Type (LT) separately using the two FCLs, termed as LT-1 and LT-2
(see in Fig. 5). Besides, the FFM also generates LT-3, as explicated in Fig.
5. Finally, the output probability ($O_{j=1,2}$) is the average of the LT-1,
LT-2, and LT-3. The output ($O_{j=1,2}$) lies in $N$-dimensional space, where
$O_{1}\in\mathcal{R}^{N=2}$ and $O_{2}\in\mathcal{R}^{N=3}$ respectively for
the inputs $I_{1}$ or $I_{2}$ by applying the proposed preprocessing (either
$P_{1}$ or $P_{2}$). It is noteworthy that the output lesion class
($O_{j=1,2}$) is obtained from the end-to-end training.

###  2.2 Designing of Web Application

The proposed web application named Dermo-DOCTOR, for the end-users, is
depicted in Fig. 9.

(a)  Figure 9: The dermo-DOCTOR prototype, where the user can select or drag a
dermoscopic image (png, jpg, bmp, or jpeg) as an input. The desired number of
output classes also can be selected in the input panel. The processing and
output panels are dedicated to select the process types and display the
recognized class with the probabilities.

We utilize browser-supported languages such as Hypertext Markup Language
(HTML), Cascading Style Sheets (CSS), and Javascript, etc. for developing the
Dermo-DOCTOR application. A python web framework package, called Flask, is
used for developing an application by deploying our proposed CNN-based
detection and recognition models and their trained weights. We apply HTML and
CSS to design a graphical user interface with three panels, such as an input
panel, a processing panel, and an output panel (see in Fig. 9). The user can
select the query image (drag-and-drop or direct upload) and the number of
query classes (both binary and multi-class) in the input panel. Then, the user
can also start the process or reset the selections in the processing panel.
The return results from the host machine are displayed in the output panel.

###  2.3 Training Protocol

The encoder kernels are initialized with the pre-trained ImageNet weights,
whereas the decoder kernels are initialized with the he normal distribution
[39]. The Aspect Ratio (AS) distribution tells that most of the images in
ISIC-2016 and ISIC-2017 datasets have an AS of $3:4$. Therefore, we resize all
the images to $192\times 256$ pixels using the nearest-neighbor interpolation
for the detection. Again, the AS distribution of both datasets’ extracted
lesion ROIs reveals that most of the ROIs have an AS of $1:1$. Hence, we again
resize the lesion ROIs to $192\times 192$ pixels using a nearest-neighbor
interpolation for the recognition. Additionally, we have standardized and
rescaled the training images to $[0\,1]$ for both detection and recognition.
We employ Eq. 1 as a loss function and intersection over union as a metric for
training the detection sub-network of the proposed Dermo-DOCTOR.

$L\,(y,\hat{y})=1-\frac{\displaystyle\sum_{i=1}^{N}y_{i}\times\hat{y_{i}}}{\displaystyle\sum_{i=1}^{N}y_{i}+\displaystyle\sum_{i=1}^{N}\hat{y_{i}}-\sum_{i=1}^{N}y_{i}\times\hat{y_{i}}}-\frac{1}{N}\sum_{i=1}^{N}[y_{i}\log\hat{y_{i}}+(1-y_{i})\log(1-\hat{y_{i}})],$
(1)

where $y$ and $\hat{y}$, $N$ respectively denote the true and predicted label,
the total pixel numbers. In Eq. 1, $\log\hat{y_{i}}$ and $\log(1-\hat{y_{i}})$
are the estimation of log-likelihood of pixel being lesion or not,
respectively. The product of $y$ and $\hat{y}$ in Eq. 1 is the estimation of
similarity (intersection) between true and predicted lesion masks. We employ
categorical cross-entropy as a loss function and accuracy as a metric for
training the recognition sub-network.

##  3 Results and Discussion

This section bestows different lesion detection results and subsequent
recognition in Subsections 3.1 and 3.2, respectively. The segmented lesion
masks are utilized for ROI extraction to classify and detect the bounding
boxes around the lesions.

###  3.1 Results for Detection

Firstly, we exhibit the quantitative and qualitative segmentation results,
applying the proposed Dermo-DOCTOR and two other well-known networks: the UNet
and the FCN8s. Secondly, we compare our outcomes with several state-of-the-art
results utilizing the same datasets. To quantify the segmentation correctness,
we use mean Recall (mRc), mean Specificity (mSp), and mean Intersection over
Union (mIoU), which are defined in Eq. 2.

$\begin{split}mRc=\frac{1}{M\times
N}\displaystyle\sum_{i=1}^{N}\displaystyle\sum_{j=1}^{M}\frac{TP_{ij}}{TP_{ij}+FN_{ij}},\\\
mSp=\frac{1}{M\times
N}\displaystyle\sum_{i=1}^{N}\displaystyle\sum_{j=1}^{M}\frac{TN_{ij}}{TN_{ij}+FP_{ij}},\\\
mIoU=\frac{1}{M\times
N}\displaystyle\sum_{i=1}^{N}\displaystyle\sum_{j=1}^{M}\frac{TP_{ij}}{TP_{ij}+FN_{ij}+FP_{ij}},\end{split}$
(2)

where $M$ and $N$ denote the pixel and sample numbers, whereas $TP$, $TN$,
$FN$, and $FP$ indicate true positive (lesion as a lesion), true negative
(background as a background), false negative (lesion as a background), and
false positive (background as a lesion), respectively.

Table 1 confers the segmentation results of three methods on two separate
datasets: the ISIC-2016 and the ISIC-2017.

Table 1: Quantitative segmentation results on ISIC-$2016$ and ISIC-$2017$ test datasets using the Dermo-DOCTOR, UNet, and FCN8s. The winner metrics for the ISIC-2016 are presented in bold font, whereas they are underlined for the ISIC-2017. [HTML]C0C0C0 [HTML]C0C0C0 |  [HTML]C0C0C0 |  [HTML]C0C0C0Performance metrics
---|---|---
[HTML]C0C0C0 [HTML]C0C0C0  Testing datasets | [HTML]C0C0C0  Models | mRc | mSp | mIoU
| Dermo-DOCTOR | $\mathbf{0.92\pm 0.13}$ | $0.97\pm 0.07$ | $\mathbf{0.85\pm 0.12}$
| UNet | $0.86\pm 0.13$ | $\mathbf{0.98\pm 0.04}$ | $0.83\pm 0.13$
ISIC-$2016$ | FCN8s | $0.84\pm 0.15$ | $0.98\pm 0.05$ | $0.80\pm 0.14$
| Dermo-DOCTOR | $\underline{0.86\pm 0.17}$ | $\underline{0.97\pm 0.07}$ | $\underline{0.80\pm 0.17}$
| UNet | $0.83\pm 0.21$ | $0.97\pm 0.07$ | $0.76\pm 0.21$
ISIC-$2017$ | FCN8s | $0.86\pm 0.18$ | $0.93\pm 0.10$ | $0.71\pm 0.17$

The mean overlapping between the actual and predicted masks are as high as
$85.0\,\%$ and $80.0\,\%$ for the ISIC-2016 and ISIC-2017. The Dermo-DOCTOR
mIoU beats nearby UNet by the margins of $2.0\,\%$ and $4.0\,\%$ accordingly
for the ISIC-2016 and ISIC-2017 datasets. The mean type-I and type-II errors
for ISIC-2016 of the Dermo-DOCTOR are $3.0\,\%$ and $8.0\,\%$, respectively,
outperforming the UNet by a margin of $6.0\,\%$ concerning the type-II error,
while defeated by a margin of $1.0\,\%$ for type-I error. However, $1.0\,\%$
error in mSp is quite acceptable in segmentation as the lesion edge
information will be preserved with less over-segmentation. Similarly, for
ISIC-2017, the Dermo-DOCTOR exceeds UNet by a border of $3.0\,\%$ for type-II
error, while they have similar type-I errors. For both the datasets, the FCN8s
is a defeating method as it produces low metrics than the other two methods
(see in Table 1). In contrast, the proposed Dermo-DOCTOR provides better-
segmented masks, leading to better lesion detection and recognition.

Fig. 10 depicts several qualitative results from the proposed Dermo-DOCTOR,
UNet, and FCN8s.

Figure 10: Qualitative segmentation results using the FCN8s, UNet, and Dermo-
DOCTOR networks. Green, Red, and Yellow colors indicate the TP, FN, and FP
regions, respectively. The top-right IoUs are given for quantitative
evaluation.

The segmented lesions from FCN8s suffer from checkerboard artifact, providing
zigzag lesion boundaries (see in Fig. 10). The possible cause of such poor
results is that the low-resolution features across the high-resolution feature
map are not constant due to the non-divisible upscaling factor, generating
checkerboard artifacts in the lesion masks [33, 35]. However, the lesion masks
from the UNet and Dermo-DOCTOR are blessed with smoother lesion boundaries,
which are the image-based classifiers’ salient features. The quantitative and
qualitative results of the UNet point that it undergoes from the additional
false-negative region, leading to under-segmentation. However, the proposed
Dermo-DOCTOR achieves lower false-positive and false-negative regions relative
to the other implementations. As in the proposed Dermo-DOCTOR, the dual
encoders can learn more salient features compared to the single encoder in
UNet and FCN8s, especially when training with a small dataset. Again, Fig. 11
illustrates the segmentation results of several challenging images on the
ISIC-2016 and ISIC-2017 datasets.

Figure 11: Qualitative segmentation results on two different test datasets
employing our Dermo-DOCTOR, where the TP, FN, and FP regions are respectively
denoted by the Green, Red, and Yellow colors. The top-right IoU exhibits
quantitative validation.

The lesion segmentation results in Fig. 11 describe that the segmented lesion
masks are precise even the query images contain different artifacts (see in
Fig. 2), and even the ROIs are tiny in size. Although the extracted ROIs have
few false-positive regions (yellow color) with negligible false-negative
regions (red color), it is not surprising for further lesion classification,
as they preserve the lesion boundaries.

Table 2 exhibits lesion segmentation results from the proposed Dermo-DOCTOR
and other state-of-the-art methods, trained and tested on the same ISIC
datasets.

Table 2: Comparative lesion segmentation results for the proposed Dermo-DOCTOR and other state-of-the-art methods on both the ISIC-2016 and ISIC-2017 test datasets. [HTML]C0C0C0 [HTML]C0C0C0 |  [HTML]C0C0C0ISIC-2016 test dataset |  [HTML]C0C0C0ISIC-2017 test dataset
---|---|---
[HTML]C0C0C0 [HTML]C0C0C0 Segmentation Methods | mIoU | mRc | mSp | mIoU | mRc | mSp
FCN ensemble [90] | $0.84$ | $0.91$ | $0.96$ | - | - | -
Fusion Structure [81] | $0.85$ | $0.92$ | $0.96$ | - | - | -
DCL-PSI [7] | $0.85$ | $\mathbf{0.93}$ | $0.96$ | $0.72$ | $0.80$ | $0.94$
DSNet [35] | - | - | - | $0.77$ | $\mathbf{0.87}$ | $0.95$
HRFB [85] | $0.85$ | $0.87$ | $0.96$ | $0.78$ | $\mathbf{0.87}$ | $0.96$
iFCN [65] | - | - | - | $0.78$ | $0.85$ | $\mathbf{0.98}$
Proposed Dermo-DOCTOR (2020) | $\mathbf{0.85}$ | $\underline{0.92}$ | $\mathbf{0.97}$ | $\mathbf{0.80}$ | $\underline{0.86}$ | $\underline{0.97}$
DCL-PSI: Deep Class-specific Learning with Probability based Step-wise
Integration
HRFB: High-Resolution Feature Block
iFCN: improved Fully Convolutional Network

The Dermo-DOCTOR generates the best segmentation results for three out of the
six cases while performs as second-best for the remaining three cases. Our
Dermo-DOCTOR outperforms the DCL-PSI [7] for ISIC-2016 by a margin of
$1.0\,\%$ for mSp, whereas it has been lost by $1.0\,\%$ concerning mRc of
DCL-PSI, while mIoU is constant for both methods. The current method (HRFB) of
Xie et al. [85] has $5.0\,\%$ and $1.0\,\%$ less mRc and mSp, respectively,
for ISIC-2016, while the mIoU is the same as our proposed method. It reveals
that the proposed Dermo-DOCTOR has $5.0\,\%$ and $1.0\,\%$ less type-II and
type-I errors, respectively, in the segmented lesion masks than HRFB. The
Dermo-DOCTOR also defeats the other two methods, the FCN ensemble [90] and
Fusion Structure [81], for all metrics and mSp for the ISIC-2016 dataset.
Moreover, the methods in Table 2 for the ISIC-2017 dataset have been defeated
by the proposed Dermo-DOCTOR for mIoU, where it beats the nearby HRFB and iFCN
[65] by a margin of $2.0\,\%$. Regarding the mRc and mSp, the Dermo-DOCTOR has
been lost by $1.0\,\%$ by the methods in [35, 65, 85] (see in Table 2). Still,
the Dermo-DOCTOR performs better as it has $2.0\,\%$ more mIoU than the nearby
methods [65, 85] for the ISIC-2107 dataset.

However, the above-discussions quantitatively and qualitatively demonstrate
that the proposed Dermo-DOCTOR yields the most reliable skin lesion’s ROIs,
which proves its supremacy for lesion segmentation. Consequently, we will
further utilize the proposed masks for lesion ROI extraction to perform the
lesion recognition in the next subsection.

###  3.2 Results for Recognition

This subsection exhibits the quantitative and qualitative results for lesion
recognition, applying the proposed Dermo-DOCTOR and two other implemented
well-known networks: the ResNet-$50$ and the Xception. In the end, we compare
our results with several state-of-the-art results for those datasets. We
utilize recall, precision, and F1-score to quantify the recognition
efficiency, where they respectively quantify the type-II error, the positive
predictive values, and the harmonic mean of recall and precision for revealing
the trade-off between them. Additionally, we also estimate the ROC curve and
its corresponding AUC value to evaluate any randomly elected query image’s
prediction probability.

Table 3 gives the lesion recognition results, showing the outcomes for three
networks and two datasets.

Table 3: The recognition results from numerous comprehensive experiments on the ISIC-$2016$ and ISIC-$2017$ test datasets on three separate networks, highlighting the weighted average metrics’ highest values. [HTML]C0C0C0 [HTML]C0C0C0 |  [HTML]C0C0C0 |  [HTML]C0C0C0ResNet-$50$ |  [HTML]C0C0C0Xception |  [HTML]C0C0C0Dermo-DOCTOR
---|---|---|---|---
[HTML]C0C0C0 [HTML]C0C0C0 |  [HTML]C0C0C0 |  [HTML]C0C0C0Preprocessing |  [HTML]C0C0C0Preprocessing |  [HTML]C0C0C0Preprocessing
[HTML]C0C0C0 [HTML]C0C0C0Test Dataset | [HTML]C0C0C0Class-wise Metrics | $P_{1}$ | $P_{2}$ | $P_{1}$ | $P_{2}$ | $P_{1}$ | $P_{2}$
|  | Nev |  $0.94$ | 0.87 |  $0.97$ | $0.91$ |  $0.96$ | $0.90$
|  | Mel |  $0.45$ | $0.93$ |  $0.25$ | $0.92$ |  0.37 | $0.95$
| Recall | W. Avg. |  $0.84$ | $0.88$ |  $0.83$ | $0.91$ |  $0.84$ | $\mathbf{0.91}$
|  | Nev |  $0.87$ | $0.98$ |  $0.84$ | $0.98$ |  $0.86$ | $0.99$
|  | Mel |  $0.64$ | $0.63$ |  $0.70$ | $0.71$ |  $0.68$ | $0.70$
| Precision | W. Avg. |  $0.83$ | $0.91$ |  $0.81$ | $0.93$ |  $0.83$ | $\mathbf{0.93}$
|  | Nev |  $0.90$ | $0.92$ |  $0.90$ | $0.94$ |  0.91 | $0.94$
|  | Mel |  $0.53$ | $0.75$ |  $0.37$ | $0.80$ |  $0.48$ | $0.80$
ISIC-2016 | F1-score | W. Avg. |  $0.83$ | $0.89$ |  $0.80$ | $0.91$ |  $0.82$ | $\mathbf{0.91}$
|  | Nev |  $0.80$ | $0.80$ |  $0.86$ | $0.80$ |  $0.87$ | $0.82$
|  | SK |  $0.59$ | $0.80$ |  $0.66$ | $0.83$ |  $0.66$ | $0.82$
|  | Mel |  $0.45$ | $0.59$ |  $0.42$ | $0.53$ |  $0.50$ | $0.62$
| Recall | W. Avg. |  $0.70$ | $0.76$ |  $0.74$ | $0.75$ |  $0.77$ | $\mathbf{0.78}$
|  | Nev |  $0.80$ | $0.88$ |  $0.83$ | $0.88$ |  $0.84$ | $0.89$
|  | SK |  $0.64$ | $0.59$ |  $0.61$ | $0.52$ |  $0.65$ | $0.65$
|  | Mel |  $0.43$ | $0.56$ |  $0.52$ | $0.62$ |  $0.58$ | $0.57$
| Precision | W. Avg. |  $0.70$ | $0.78$ |  $0.74$ | $0.77$ |  $0.76$ | $\mathbf{0.79}$
|  | Nev |  $0.80$ | 0.84 |  0.85 | $0.84$ |  $0.85$ | $0.85$
|  | SK |  $0.61$ | $0.68$ |  $0.63$ | $0.64$ |  $0.65$ | $0.73$
|  | Mel |  $0.44$ | $0.57$ |  $0.46$ | $0.57$ |  $0.53$ | $0.59$
ISIC-2017 | F1-score | W. Avg. |  $0.70$ | $0.76$ |  $0.74$ | $0.76$ |  $0.76$ | $\mathbf{0.78}$
$P_{1}$: Segmentation; $P_{2}$: Segmentation+Rebalancing+Augmentation; W.
Avg.: Weighted Average

The weighted average of recalls for ResNet-$50$, Xception, and Dermo-DOCTOR
have been respectively improved by the margins of $4.0\,\%$, $8.0\,\%$, and
$7.0\,\%$ for ISIC-2016, when we employ the preprocessing $P_{2}$ instead of
baseline $P_{1}$ (see in Table 3). The highest possible recall ($0.91$) for
ISIC-2016 is received from the proposed Dermo-DOCTOR classifier, applying the
proposed preprocessing $P_{2}$ on the segmented masks from the proposed Dermo-
DOCTOR segmentor. The rebalancing and augmentation, along with the
segmentation, in preprocessing $P_{2}$ reduces the FN-rates ($63.0\,\%$ to
$5.0\,\%$) of Mel class. It also decreases the FP-rates ($32.0\,\%$ to
$30.0\,\%$) for the same class while applying our proposed Dermo-DOCTOR. Such
reductions, in FN- and FP-rates, are praiseworthy of our proposed
preprocessing $P_{2}$ and Dermo-DOCTOR comparing the baseline classifiers
(ResNet-$50$ and Xception) and preprocessing $P_{1}$. Likewise, the weighted
average of precision and F1-score on ISIC-2016 are also respectively improved
by $10.0\,\%$ and $9.0\,\%$ for the preprocessing $P_{2}$. Remarkably, the
harmonic mean of recall and precision for the Mel class of ISIC-2016 has
significantly strengthened by a margin of $32.0\,\%$ with preprocessing
($P_{2}$) and Dermo-DOCTOR. However, comparing all the experiments on the
ISIC-2016, the proposed $P_{2}$ and Dermo-DOCTOR are the best preprocessing
and classifier having a type-II error of $9.0\,\%$ and positive predictive
value of $93.0\,\%$. Table 3 also notes that the weighted average of recall,
precision, and F1-score have the respective maximum values of $0.78$, $0.79$,
and $0.78$, for the preprocessing $P_{2}$ and proposed Dermo-DOCTOR on the
ISIC-2017. The recall of SK and Mel classes in the ISIC-2017 dataset, applying
the proposed Dermo-DOCTOR and preprocessing $P_{2}$, have respectively updated
by the margins of $16.0\,\%$ and $12.0\,\%$. In comparison, it decreases by
$1.0\,\%$ for the Nev class, which is acceptable in the medical diagnostic
system (as the positive class is significantly improved). A margin of
$5.0\,\%$ has raised the Nev class’s positive predictive value, whereas it is
closer or equal for the other two classes for the proposed Dermo-DOCTOR with
$P_{2}$. However, comparing all the experiments on the ISIC-2017, the proposed
$P_{2}$ and Dermo-DOCTOR are the best preprocessing and classifier, with a
type-II error of $22.0\,\%$ and positive predictive value of $79.0\,\%$.

The experimental lesion categorization results on ISIC-2016 and ISIC-2017
datasets demonstrate that the two-classes’ (ISIC-2016) recognition performance
is better than the three-classes (ISIC-2017). The addition of SK class in
ISIC-2017 reduces the weighted average recall, precision, and F1-score by the
margins of $13.0\,\%$, $14.0\,\%$, and $13.0\,\%$, respectively. The higher
similarity of SK with Nev and Mel classes is the possibility of such reduced
ISIC-2017 test results. Noticeably, more classes tend to bring complications
to the classifiers, especially when the training has fewer examples and inter-
class similarities. The joining of different heterogeneous samples to each
class can enhance the categorization results in ISIC-2017.

The further analysis of different classifiers and preprocessing have presented
in the ROC curves in Fig. 12, showing the highest possible AUC as $0.98$ and
$0.91$ respectively for ISIC-2016 and ISIC-2017.

(a)  (b)

Figure 12: The ROC curves ((a) for ISIC-2016 and (b) for ISIC-2017) of skin
lesion recognition, where we have plotted the ROC for the proposed Dermo-
DOCTOR and implemented ResNet-$50$ and Xception with two different
preprocessing ($P_{1}$ and $P_{2}$).

The rebalancing and augmentation employment with segmentation ($P_{2}$)
heightens AUC for both ISIC-$2016$ and ISIC-$2017$ datasets of all
classifiers. The Dermo-DOCTOR with $P_{2}$ has beaten the baseline Xception
and ResNet-$50$ respectively by $3.0\,\%$ and $3.0\,\%$ in terms of AUC for
the ISIC-2016 dataset. Whereas it surpasses them by the margins of $3.0\,\%$
and $4.0\,\%$ for AUC for the ISIC-2017 dataset. However, the earlier
discussions for the lesion recognition on ISIC-$2016$ and ISIC-$2017$ test
datasets expose the supremacy of the proposed Dermo-DOCTOR and preprocessing
$P_{2}$.

The detailed class-wise performances of the lesion recognition by the proposed
Dermo-DOCTOR and preprocessing $P_{2}$ are exhibited in Table 4.

Table 4: The confusion matrix for the ISIC-$2016$ with 379 samples (left) and ISIC-$2016$ with 600 samples (right) test datasets, employing the proposed Dermo-DOCTOR and preprocessing $P_{2}$ |  |  | Actual
---|---|---
|  | Nev | Mel
| Nev |  [HTML]C0C0C0  | 273
---
$89.80\,\%$
| 4
---
$5.33\,\%$
Predicted | Mel |  | 31
---
$10.20\,\%$
[HTML]C0C0C0  | 71
---
$94.67\,\%$
|  |  | Actual
---|---|---
|  | Nev | SK | Mel
| Nev |  [HTML]C0C0C0  | 321
---
$81.68\,\%$
| 10
---
$11.11\,\%$
| 28
---
$23.93\,\%$
| SK |  | 23
---
$5.85\,\%$
[HTML]C0C0C0  | 74
---
$82.22\,\%$
| 17
---
$14.53\,\%$
Predicted | Mel |  | 49
---
$12.47\,\%$
| 6
---
$6.67\,\%$
[HTML]C0C0C0  | 72
---
$61.54\,\%$

The ISIC-2016’s confusion matrix in Table 4 (left) shows that among 304-Nev
samples correctly recognized samples are $273\,(89.80\,\%)$, whereas barely
$31\,(10.20\,\%)$-Nev samples are recognized as Mel (as FP). It also reveals
that among 75-Mel samples, rightly recognized samples are $71\,(94.67\,\%)$,
whereas only $4\,(5.33\,\%)$-Mel samples are improperly recognized as Nev (as
FN). Again, the ISIC-2017’s confusion matrix (see in Table 4 (right))
demonstrates that $81.68\,\%$-Nev samples are correctly recognized as Nev
class, while $18.32\,\%$-Nev samples are wrongly recognized to other classes
as FP ($5.85\,\%$ as SK and $12.47\,\%$ as Mel). Similarly, $17.78\,\%$ and
$38.46\,\%$ samples of the SK- and Mel-classes belong to FP and FN,
respectively. Although the $38.46\,\%$ of the positive samples (Mel) are
improperly recognized, it is still better than the baseline $58.0\,\%$ errors
in the baseline (Xception with preprocessing $P_{1}$). Fig. 13 bestows
qualitative results from the proposed Dermo-DOCTOR classifier and
preprocessing $P_{2}$ for the lesion recognition into different, either two
classes or three classes.

Figure 13: Example of several qualitative classification results of the
challenging images of the ISIC-2016 and ISIC-2017 test datasets using the
Dermo-DOCTOR, where the recognition has accomplished using the segmented ROIs
(green color) from the Dermo-DOCTOR.

For concurrent detection and recognition, we utilize the segmented masks and
categorized class for contouring the lesions (green color) and label
annotation on the image to help the dermatologists for further assessment (see
in Fig. 13). More concurrent results for all the test images are available on
YouTube (ISIC-$2016$333ISIC-$2016$ (Detection & Recognition):
https://bit.ly/Dermo-DOCTOR_ISIC_16 and ISIC-$2017$444ISIC-$2017$ (Detection &
Recognition): https://bit.ly/Dermo-DOCTOR_ISIC_17). However, the results in
Fig. 13 illustrate a few challenging images, where we show some wrongly
recognized images. Those qualitative results depict that the detection and
recognition are precise even the query test images contain different artifacts
(see in Fig. 2). Although the Dermo-DOCTOR incorrectly predicts some images,
they visually seem like a predicted class.

Table 5 describes the comparison of the results of our Dermo-DOCTOR and other
methods, which were trained and tested on the same ISIC datasets. The proposed
Dermo-DOCTOR produces the best recognition for two out of the six cases while
performing second-best with the winning methods on the other four cases (see
in Table 5).

Table 5: The state-of-the-art comparison with proposed Dermo-DOCTOR, which had trained, validated, and tested on the ISIC-$2016$ and ISIC-$2017$ datasets. [HTML]C0C0C0 [HTML]C0C0C0 |  [HTML]C0C0C0ISIC-2016 test dataset |  [HTML]C0C0C0ISIC-2017 test dataset
---|---|---
[HTML]C0C0C0 [HTML]C0C0C0Classification Methods | Recall | Precision | AUC | Recall | Precision | AUC
ResNet-$50$ [10] | $0.56$ | $0.71$ | $0.85$ | - | - | -
GR [74] | - | - | - | $0.15$ | - | $0.91$
ARLCNN [91] | - | - | - | $0.77$ | - | $\mathbf{0.92}$
IR [3] | $0.82$ | - | $0.77$ | $0.76$ | - | -
FPRPN [77] | $0.99$ | $\underline{0.82}$ | $0.81$ | $0.98$ | $\mathbf{0.82}$ | $0.79$
MFA [89] | $0.60$ | $0.69$ | $\underline{0.86}$ | - | $0.72$ | $0.90$
Proposed Dermo-DOCTOR (2020) | $\underline{0.91}$ | $0.93$ | $0.98$ | $\underline{0.78}$ | $\underline{0.79}$ | $\underline{0.91}$
GR: Gabor Wavelet-based CNN [74]
ARLCNN: Attention Residual Learning CNN (ResNet-$14$ & ResNet-$50$) [92]
IR: Inception-ResNet-V$2$ (ISIC-$2016$), ResNet-$50$ (ISIC-$2017$,ISIC-$2018$)
[3]
FPRPN: Feature Pyramid Network (FPN) and Region Proposal Network (RPN) [77]
MFA: Multi-network based feature aggregation [89]

Comparison of ISIC-2016. The proposed network produces the best results for
the AUC by beating the state-of-the-art of Yu et al. [89] with a $12.0\,\%$
border. Concerning the type-II errors, Dermo-DOCTOR is behind the state-of-
the-art (FPRPN) of Song et al. [77] by a border of $8.0\,\%$, but the Dermo-
DOCTOR outperforms it by a $11.0\,\%$ border for the positive predictive
value. However, Dermo-DOCTOR beats the FPRPN by a $1.5\,\%$ perimeter in terms
of balanced accuracy. Whereas Dermo-DOCTOR also exceeds the FPRPN by a
$17.0\,\%$ margin for AUC.

Comparison of ISIC-2017. The Dermo-DOCTOR serves as the second-best results
concerning all the metrics, where it beats the state-of-the-art FPRPN [77]
with a margin of $12.0\,\%$ for AUC, although FPRPN has defeated it for
balanced accuracy. The proposed Dermo-DOCTOR has defeated ARLCNN [91] by a
$1.0\,\%$ border concerning the recall, but ARLCNN wins by a $1.0\,\%$ margin
in AUC. However, the proposed Dermo-DOCTOR produces the second-highest results
for recall by beating the third-best ARLCNN [91] with a margin of $1.0\,\%$.
For precision, it performs so by vanquishing third-best MFA [89] with a border
of $7.0\,\%$, and for AUC by beating third-best MFA with a margin of
$1.0\,\%$.

###  3.3 Applications

Fig. 14 illustrates the prototype of the developed web application deploying
our Dermo-DOCTOR, which runs in a web browser at http://127.0.0.1:5000/ by
accessing the CNN environments of the local machine.

(a)  Figure 14: Our designed web application to detect and recognize skin
lesions simultaneously. Dermatologists can utilize this application by
selecting or dragging the image as an input. The app will confer the lesion
detection and recognition results for either binary class or multi-classes.

The app takes a dermoscopic image (png, jpg, bmp, or jpeg) as an input,
displaying the user’s diagnosis result as soon as the back-end model analyzes
a given image. The recognized class with its probability is displayed in the
output panel, highlighting the lesion ROI by a green color bounding box.
Therefore, it helps the dermatologists focus on that detected area for cross-
checking the predicted class. A real-time utilization of the Dermo-DOCTOR has
been uploaded to YouTube555Dermo-DOCTOR App: https://bit.ly/Dermo-DOCTOR_App,
which confers less time-latency of getting results. The reason for that is the
app sends images to the host and receives the host’s results through the
internet and the time for prediction in the host (higher traffic in the host
will increase the latency). However, the dedicated machine with GPU can
alleviate this time-latency limitation. We have tested the app on our local
machine and could not make it public due to resource limitations.

##  4 Conclusion

Despite the present colossal challenges due to high visual and intra-class
variability, inter-class similarity, and the presence of different artifacts,
automated skin lesion detection and recognition are incredibly crucial.
However, this article proposed and developed an automated CNN-based lesion
detection and recognition network, integrating ROI extraction by segmentation,
image augmentations, and class rebalancing. Our experimental results
demonstrated that the proposed Dermo-DOCTOR could detect and recognize the
lesion more accurately as we concatenated features from two different
encoders. Such a concatenation provides more prominent and discriminating
feature maps of the skin lesions comparing a single encoder. The segmented
lesions rather than the whole images can provide more salient and
representative features from the CNNs, leading to improved lesion recognition.
Moreover, the rebalanced class distribution attained better performance of the
recognition as compared to the imbalanced distribution. Additionally, the
augmentation led the CNN-based classifier to be more generic as CNNs can learn
from diverse training samples. Thus, it achieved state-of-the-art performance
to detect and recognize the lesions from two different test datasets, such as
ISIC-$2016$ and ISIC-$2017$. We will further explore and investigate the
effects of improved segmentation and weighting of the underrepresented classes
in the future. The deployment of our framework to a web application precisely
detected and recognized the lesions concurrently. The developed web
application will be improved, making it more user-friendly for dermatologists
and deploying it to the google cloud platform for clinical applications.

## Author Contributions

M. K. Hasan: Conceptualization, Methodology, Software, Formal analysis,
Investigation, Writing- Review & Editing, Supervision; S. Roy: Validation,
Data Curation, Writing- Original Draft; C. Mondal: Validation, Data Curation,
Writing- Original Draft; M. A. Alam: Conceptualization, Data Curation; M. T.
E. Elahi: Validation, Writing- Original Draft; A. Dutta: Validation, Writing-
Original Draft; S. M. T. U. Raju: Software; M. T. Jawad: Validation, Writing-
Review & Editing; M. Ahmad: Supervision.

## Acknowledgements

None. No funding to declare.

## Conflict of Interest

All authors have no conflict of interest to publish this research.

## References

  * Acharya et al. [2012] Acharya, U.R., Molinari, F., Sree, S.V., Chattopadhyay, S., Ng, K.H., Suri, J.S., 2012\.  Automated diagnosis of epileptic eeg using entropies.  Biomedical Signal Processing and Control 7, 401–408.
  * Al-Masni et al. [2018] Al-Masni, M.A., Al-antari, M.A., Choi, M.T., Han, S.M., Kim, T.S., 2018.  Skin lesion segmentation in dermoscopy images via deep full resolution convolutional networks.  Computer methods and programs in biomedicine 162, 221–231.
  * Al-Masni et al. [2020] Al-Masni, M.A., Kim, D.H., Kim, T.S., 2020.  Multiple skin lesions diagnostics via integrated deep convolutional networks for segmentation and classification.  Computer Methods and Programs in Biomedicine 190, 105351.
  * Al Nazi and Abir [2020] Al Nazi, Z., Abir, T.A., 2020\.  Automatic skin lesion segmentation and melanoma detection: Transfer learning approach with u-net and dcnn-svm, in: Proceedings of International Joint Conference on Computational Intelligence, Springer. pp. 371–381.
  * Amin et al. [2020] Amin, J., Sharif, A., Gul, N., Anjum, M.A., Nisar, M.W., Azam, F., Bukhari, S.A.C., 2020\.  Integrated design of deep features fusion for localization and classification of skin cancer.  Pattern Recognition Letters 131, 63–70.
  * Badrinarayanan et al. [2017] Badrinarayanan, V., Kendall, A., Cipolla, R., 2017.  SegNet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence 39, 2481–2495.
  * Bi et al. [2019] Bi, L., Kim, J., Ahn, E., Kumar, A., Feng, D., Fulham, M., 2019.  Step-wise integration of deep class-specific learning for dermoscopic image segmentation.  Pattern recognition 85, 78–89.
  * Bi et al. [2017] Bi, L., Kim, J., Ahn, E., Kumar, A., Fulham, M., Feng, D., 2017.  Dermoscopic image segmentation via multistage fully convolutional networks.  IEEE Transactions on Biomedical Engineering 64, 2065–2074.
  * Bray et al. [2018] Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R.L., Torre, L.A., Jemal, A., 2018\.  Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries.  CA: a cancer journal for clinicians 68, 394–424.
  * Brinker et al. [2019] Brinker, T.J., Hekler, A., Enk, A.H., von Kalle, C., 2019\.  Enhanced classifier training to improve precision of a convolutional neural network to identify images of skin lesions.  PloS one 14.
  * Cheng et al. [2008] Cheng, Y., Swamisai, R., Umbaugh, S.E., Moss, R.H., Stoecker, W.V., Teegala, S., Srinivasan, S.K., 2008.  Skin lesion classification using relative color features.  Skin Research and Technology 14, 53–64.
  * Chollet [2017] Chollet, F., 2017.  Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258.
  * Codella et al. [2018] Codella, N.C., Gutman, D., Celebi, M.E., Helba, B., Marchetti, M.A., Dusza, S.W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al., 2018\.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic), in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), IEEE. pp. 168–172.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009\.  Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee. pp. 248–255.
  * Dennis Schmid [2018] Dennis Schmid, 2018.  _Number of dermatologists in selected European countries in 2015_.  https://tinyurl.com/y59xoc7n [Accessed: 4 Sept 2020].
  * Department of Health (2017) [Commonwealth of Australia] Department of Health (Commonwealth of Australia), 2017\.  _Dermatology 2016 Factsheet_.  https://tinyurl.com/y3z39b9r [Accessed: 15 Jun 2020].
  * Dolz et al. [2020] Dolz, J., Desrosiers, C., Wang, L., Yuan, J., Shen, D., Ayed, I.B., 2020\.  Deep cnn ensembles and suggestive annotations for infant brain mri segmentation.  Computerized Medical Imaging and Graphics 79, 101660.
  * Estava et al. [2017] Estava, A., Kuprel, B., Novoa, R., et al., 2017.  Dermatologist level classification of skin cancer with deep neural networks [j].  Nature 542, 115\.
  * Everingham et al. [2010] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A., 2010.  The pascal visual object classes (voc) challenge.  International journal of computer vision 88, 303–338.
  * Fujisawa et al. [2019] Fujisawa, Y., Inoue, S., Nakamura, Y., 2019.  The possibility of deep learning-based, computer-aided skin tumor classifiers.  Frontiers in Medicine 6, 191\.
  * Furey et al. [2000] Furey, T.S., Cristianini, N., Duffy, N., Bednarski, D.W., Schummer, M., Haussler, D., 2000\.  Support vector machine classification and validation of cancer tissue samples using microarray expression data.  Bioinformatics 16, 906–914.
  * Garcia-Garcia et al. [2018] Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J., 2018.  A survey on deep learning techniques for image and video semantic segmentation.  Applied Soft Computing 70, 41–65.
  * Ge et al. [2017] Ge, Z., Demyanov, S., Chakravorty, R., Bowling, A., Garnavi, R., 2017.  Skin disease recognition using deep saliency features and multimodal learning of dermoscopy and clinical images, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 250–258.
  * Gessert et al. [2020] Gessert, N., Nielsen, M., Shaikh, M., Werner, R., Schlaefer, A., 2020.  Skin lesion classification using ensembles of multi-resolution efficientnets with meta data.  MethodsX , 100864.
  * Ghadiyaram and Bovik [2017] Ghadiyaram, D., Bovik, A.C., 2017\.  Perceptual quality prediction on authentically distorted images using a bag of features approach.  Journal of vision 17, 32–32.
  * Glazer and Rigel [2017] Glazer, A.M., Rigel, D.S., 2017\.  Analysis of trends in geographic distribution of us dermatology workforce density.  JAMA dermatology 153, 472–473.
  * Goyal et al. [2019] Goyal, M., Oakley, A., Bansal, P., Dancey, D., Yap, M.H., 2019.  Skin lesion segmentation in dermoscopic images with ensemble deep learning methods.  IEEE Access .
  * Guo et al. [2016] Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., Lew, M.S., 2016.  Deep learning for visual understanding: A review.  Neurocomputing 187, 27–48.
  * Gutman et al. [2016] Gutman, D., Codella, N.C., Celebi, E., Helba, B., Marchetti, M., Mishra, N., Halpern, A., 2016.  Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic).  arXiv:1605.01397 .
  * Ha et al. [2020] Ha, Q., Liu, B., Liu, F., 2020\.  Identifying melanoma images using efficientnet ensemble: Winning solution to the siim-isic melanoma classification challenge.  arXiv preprint arXiv:2010.05351 .
  * Hameed et al. [2020] Hameed, N., Shabut, A.M., Ghosh, M.K., Hossain, M., 2020\.  Multi-class multi-level classification algorithm for skin lesions classification using machine learning techniques.  Expert Systems with Applications 141, 112961.
  * Harangi [2018] Harangi, B., 2018.  Skin lesion classification with ensembles of deep convolutional neural networks.  Journal of biomedical informatics 86, 25–32.
  * Hasan et al. [2021a] Hasan, M.K., Alam, M.A., Elahi, M.T.E., Roy, S., Martí, R., 2021a.  Drnet: Segmentation and localization of optic disc and fovea from diabetic retinopathy image.  Artificial Intelligence in Medicine 111, 102001.
  * Hasan et al. [2021b] Hasan, M.K., Calvet, L., Rabbani, N., Bartoli, A., 2021b.  Detection, segmentation, and 3d pose estimation of surgical tools using convolutional neural networks and algebraic geometry.  Medical Image Analysis , 101994.
  * Hasan et al. [2020] Hasan, M.K., Dahal, L., Samarakoon, P.N., Tushar, F.I., Martí, R., 2020.  Dsnet: Automatic dermoscopic skin lesion segmentation.  Computers in Biology and Medicine , 103738\.
  * Hawas et al. [2020] Hawas, A.R., Guo, Y., Du, C., Polat, K., Ashour, A.S., 2020\.  Oce-ngc: A neutrosophic graph cut algorithm using optimized clustering estimation algorithm for dermoscopic skin lesion segmentation.  Applied Soft Computing 86, 105931\.
  * He and Garcia [2009] He, H., Garcia, E.A., 2009.  Learning from imbalanced data.  IEEE Transactions on knowledge and data engineering 21, 1263–1284.
  * He et al. [2017] He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017\.  Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969.
  * He et al. [2015] He, K., Zhang, X., Ren, S., Sun, J., 2015.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of the IEEE international conference on computer vision, pp. 1026–1034.
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Hu et al. [2018] Hu, J., Shen, L., Sun, G., 2018.  Squeeze-and-excitation networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141.
  * Huang et al. [2017] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017.  Densely connected convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708.
  * Ioffe and Szegedyet [2015] Ioffe, S., Szegedyet, C., 2015\.  Batch Normalization: accelerating deep network training by reducing internal covariate shift.  arXiv:1502.03167 .
  * ISIC [2018] ISIC, 2018.  _ISIC Archive_.  https://tinyurl.com/11hgg83u [Accessed: 09 May 2020].
  * Jahanifar et al. [2018] Jahanifar, M., Tajeddin, N.Z., Asl, B.M., Gooya, A., 2018\.  Supervised saliency map driven segmentation of lesions in dermoscopic images.  IEEE journal of biomedical and health informatics 23, 509–518.
  * Kermany et al. [2018] Kermany, D.S., Goldbaum, M., Cai, W., Valentim, C.C., Liang, H., Baxter, S.L., McKeown, A., Yang, G., Wu, X., Yan, F., et al., 2018\.  Identifying medical diagnoses and treatable diseases by image-based deep learning.  Cell 172, 1122–1131.
  * Khan et al. [2020] Khan, M.A., Sharif, M., Akram, T., Bukhari, S.A.C., Nayak, R.S., 2020.  Developed newton-raphson based deep features selection framework for skin lesion recognition.  Pattern Recognition Letters 129, 293–303.
  * Krizhevsky et al. [2012] Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  Imagenet classification with deep convolutional neural networks, in: Advances in neural information processing systems, pp. 1097–1105.
  * Kumar et al. [2016] Kumar, A., Kim, J., Lyndon, D., Fulham, M., Feng, D., 2016.  An ensemble of fine-tuned convolutional neural networks for medical image classification.  IEEE journal of biomedical and health informatics 21, 31–40.
  * Lattoofi et al. [2019] Lattoofi, N.F., Al-sharuee, I.F., Kamil, M.Y., Obaid, A.H., Mahidi, A.A., Omar, A.A., et al., 2019\.  Melanoma skin cancer detection based on abcd rule, in: 2019 First International Conference of Computer and Applied Sciences (CAS), IEEE. pp. 154–157.
  * Lee et al. [2018] Lee, Y.C., Jung, S.H., Won, H.H., 2018.  Wonderm: Skin lesion classification with fine-tuned neural networks.  arXiv preprint arXiv:1808.03426 .
  * Lin et al. [2013] Lin, M., Chen, Q., Yan, S., 2013.  Network in network.  arXiv:1312.4400 .
  * Long et al. [2015] Long, J., Shelhamer, E., Darrell, T., 2015.  Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440.
  * Lowe [2004] Lowe, D.G., 2004.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision 60, 91–110.
  * Maćkiewicz and Ratajczak [1993] Maćkiewicz, A., Ratajczak, W., 1993\.  Principal components analysis (pca).  Computers & Geosciences 19, 303–342.
  * Maglogiannis and Doukas [2009] Maglogiannis, I., Doukas, C.N., 2009\.  Overview of advanced computer vision systems for skin lesions characterization.  IEEE transactions on information technology in biomedicine 13, 721–733.
  * Mahbod et al. [2019] Mahbod, A., Schaefer, G., Ellinger, I., Ecker, R., Pitiot, A., Wang, C., 2019\.  Fusing fine-tuned deep features for skin lesion classification.  Computerized Medical Imaging and Graphics 71, 19–29.
  * Mahbod et al. [2020] Mahbod, A., Schaefer, G., Wang, C., Dorffner, G., Ecker, R., Ellinger, I., 2020\.  Transfer learning using a multi-scale and multi-network ensemble for skin lesion classification.  Computer methods and programs in biomedicine 193, 105475.
  * Moitra and Mandal [2020] Moitra, D., Mandal, R.K., 2020\.  Prediction of non-small cell lung cancer histology by a deep ensemble of convolutional and bidirectional recurrent neural network.  Journal of Digital Imaging , 1–8.
  * Mporas et al. [2020] Mporas, I., Perikos, I., Paraskevas, M., 2020.  Color models for skin lesion classification from dermatoscopic images, in: Advances in Integrations of Intelligent Methods. Springer, pp. 85–98.
  * Nachbar et al. [1994] Nachbar, F., Stolz, W., Merkle, T., Cognetta, A.B., Vogt, T., Landthaler, M., Bilek, P., Braun-Falco, O., Plewig, G., 1994.  The abcd rule of dermatoscopy: high prospective value in the diagnosis of doubtful melanocytic skin lesions.  Journal of the American Academy of Dermatology 30, 551–559.
  * Navarro et al. [2018] Navarro, F., Escudero-Viñolo, M., Bescós, J., 2018.  Accurate segmentation and registration of skin lesion images to evaluate lesion change.  IEEE journal of biomedical and health informatics 23, 501–508.
  * Odena et al. [2016] Odena, A., Dumoulin, V., Olah, C., 2016.  Deconvolution and checkerboard artifacts.  Distill 1, e3.
  * Oliveira et al. [2018] Oliveira, R.B., Papa, J.P., Pereira, A.S., Tavares, J.M.R., 2018\.  Computational methods for pigmented skin lesion classification in images: review and future trends.  Neural Computing and Applications 29, 613–636.
  * Öztürk and Özkaya [2020] Öztürk, Ş., Özkaya, U., 2020\.  Skin lesion segmentation with improved convolutional neural network.  Journal of digital imaging .
  * Pacheco et al. [2019] Pacheco, A.G., Ali, A.R., Trappenberg, T., 2019.  Skin cancer detection based on deep learning and entropy to detect outlier samples.  arXiv preprint arXiv:1909.04525 .
  * Pour and Seker [2020] Pour, M.P., Seker, H., 2020\.  Transform domain representation-driven convolutional neural networks for skin lesion segmentation.  Expert Systems with Applications 144, 113129.
  * Qin et al. [2020] Qin, Z., Liu, Z., Zhu, P., Xue, Y., 2020.  A gan-based image synthesis method for skin lesion classification.  Computer Methods and Programs in Biomedicine , 105568\.
  * Rajpurkar et al. [2017] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al., 2017\.  Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning.  arXiv:1711.05225 .
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer. pp. 234–241.
  * Sarker et al. [2018] Sarker, M.M.K., Rashwan, H.A., Akram, F., Banu, S.F., Saleh, A., Singh, V.K., Chowdhury, F.U., Abdulwahab, S., Romani, S., Radeva, P., et al., 2018\.  Slsdeep: Skin lesion segmentation based on dilated residual and pyramid pooling networks, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 21–29.
  * Savelli et al. [2020] Savelli, B., Bria, A., Molinara, M., Marrocco, C., Tortorella, F., 2020.  A multi-context cnn ensemble for small lesion detection.  Artificial Intelligence in Medicine 103, 101749.
  * Scheffe [1999] Scheffe, H., 1999.  The analysis of variance. volume 72.  John Wiley & Sons.
  * Serte and Demirel [2019] Serte, S., Demirel, H., 2019\.  Gabor wavelet-based deep learning for skin lesion classification.  Computers in biology and medicine 113, 103423.
  * Shahin et al. [2018] Shahin, A.H., Kamal, A., Elattar, M.A., 2018.  Deep ensemble learning for skin lesion classification from dermoscopic images, in: 2018 9th Cairo International Biomedical Engineering Conference (CIBEC), IEEE. pp. 150–153.
  * Siegel et al. [2020] Siegel, R.L., Miller, K.D., Jemal, A., 2020.  Cancer statistics, 2020.  CA: A Cancer Journal for Clinicians 70, 7–30.
  * Song et al. [2020] Song, L., Lin, J.P., Wang, Z.J., Wang, H., 2020.  An end-to-end multi-task deep learning framework for skin lesion analysis.  IEEE Journal of Biomedical and Health Informatics .
  * Srivastava et al. [2014] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research 15, 1929–1958.
  * Szegedy et al. [2016] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016.  Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826.
  * Tan and Le [2019] Tan, M., Le, Q.V., 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946 .
  * Tang et al. [2019] Tang, Y., Yang, F., Yuan, S., et al., 2019.  A multi-stage framework with context information fusion structure for skin lesion segmentation, in: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), IEEE. pp. 1407–1410.
  * Torrey and Shavlik [2010] Torrey, L., Shavlik, J., 2010\.  Transfer learning, in: Handbook of research on machine learning applications and trends: algorithms, methods, and techniques. IGI Global, pp. 242–264.
  * Valle et al. [2020] Valle, E., Fornaciali, M., Menegola, A., Tavares, J., Bittencourt, F.V., Li, L.T., Avila, S., 2020.  Data, depth, and design: Learning reliable models for skin lesion analysis.  Neurocomputing 383, 303–313.
  * Venugopal and Stoffel [2019] Venugopal, A., Stoffel, E.M., 2019\.  Colorectal cancer in young adults.  Current treatment options in gastroenterology 17, 89–98.
  * Xie et al. [2020a] Xie, F., Yang, J., Liu, J., Jiang, Z., Zheng, Y., Wang, Y., 2020a.  Skin lesion segmentation using high-resolution convolutional neural network.  Computer Methods and Programs in Biomedicine 186, 105241.
  * Xie et al. [2020b] Xie, Y., Zhang, J., Xia, Y., Shen, C., 2020b.  A mutual bootstrapping model for automated skin lesion segmentation and classification.  IEEE Transactions on Medical Imaging .
  * Yadav and Jadhav [2019] Yadav, S.S., Jadhav, S.M., 2019\.  Deep convolutional neural network based medical image classification for disease diagnosis.  Journal of Big Data 6, 113\.
  * Yilmaz and Trocan [2020] Yilmaz, E., Trocan, M., 2020\.  Benign and malignant skin lesion classification comparison for three deep-learning architectures, in: Asian Conference on Intelligent Information and Database Systems, Springer. pp. 514–524.
  * Yu et al. [2020] Yu, Z., Jiang, F., Zhou, F., He, X., Ni, D., Chen, S., Wang, T., Lei, B., 2020.  Convolutional descriptors aggregation via cross-net for skin lesion recognition.  Applied Soft Computing , 106281.
  * Yuan [2017] Yuan, Y., 2017.  Automatic skin lesion segmentation with fully convolutional-deconvolutional networks.  arXiv preprint arXiv:1703.05165 .
  * Zhang et al. [2019] Zhang, J., Xie, Y., Xia, Y., Shen, C., 2019.  Attention residual learning for skin lesion classification.  IEEE transactions on medical imaging 38, 2092–2103.
  * Zhang et al. [2020] Zhang, N., Cai, Y.X., Wang, Y.Y., Tian, Y.T., Wang, X.L., Badami, B., 2020\.  Skin cancer diagnosis based on optimized convolutional neural network.  Artificial Intelligence in Medicine 102, 101756.