# CIFF-Net: Contextual Image Feature Fusion for Melanoma Diagnosis

Md Awsafur Rahman§, , Bishmoy Paul§, , Tanvir Mahmud, , and Shaikh Anowarul
Fattah M. A. Rahman, B. Paul, T. Mahmud, and S. A. Fattah are with the
Department of Electrical and Electronic Engineering, Bangladesh University of
Engineering and Technology, Dhaka-1000, Bangladesh, e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and
fattah@eee.buet.ac.bd).

###### Abstract

Melanoma is considered to be the deadliest variant of skin cancer causing
around 75% of total skin cancer deaths. To diagnose Melanoma, clinicians
assess and compare multiple skin lesions of the same patient concurrently to
gather contextual information regarding the patterns, and abnormality of the
skin. So far this concurrent multi-image comparative method has not been
explored by existing deep learning-based schemes. In this paper, based on
contextual image feature fusion (CIFF), a deep neural network (CIFF-Net) is
proposed, which integrates patient-level contextual information into the
traditional approaches for improved Melanoma diagnosis by concurrent multi-
image comparative method. The proposed multi-kernel self attention (MKSA)
module offers better generalization of the extracted features by introducing
multi-kernel operations in the self attention mechanisms. To utilize both self
attention and contextual feature-wise attention, an attention guided module
named contextual feature fusion (CFF) is proposed that integrates extracted
features from different contextual images into a single feature vector.
Finally, in comparative contextual feature fusion (CCFF) module, primary and
contextual features are compared concurrently to generate comparative
features. Significant improvement in performance has been achieved on the
ISIC-2020 dataset over the traditional approaches that validate the
effectiveness of the proposed contextual learning scheme.

###### Index Terms:

Melanoma Diagnosis, Skin lesion recognition, Clinical-inspired, Dermoscopy
images, Deep learning

§§footnotetext: Equal contribution

##  I Introduction

Skin cancer is the most common cancer globally, with Melanoma being the most
deadly form responsible for an overwhelming majority of skin cancer deaths
[1]. It is a major public health problem, with an estimated 106,110 cases
expected to be diagnosed in US in 2021 [2]. Although the mortality is
significant when detected early, Melanoma survival exceeds 95% [3]. Finding
Melanoma at an early stage is crucial; early detection can vastly increase the
chances of cure [4].

Existing approaches in melanoma detection have achieved expert-level
performance in controlled studies examining single images using deep learning
algorithms but they lack incorporation of patient-level information [3, 5, 6,
7, 8, 9, 10]. Recent studies have demonstrated the ability of AI-driven
decisions to match, if not outperform, clinicians’ decisions in the diagnosis
of individual skin lesion images in controlled reader studies.However, the
reader study did not accurately reflect clinical scenarios where clinicians
have access to examine all lesions on a patient [3].

Figure 1: Example of clinical context. An atypical lesion found on a patient
with many atypical lesions is less suspicious for malignancy as opposed to an
atypical lesion that is an outlier on the patient. Figure 2: An overview of
proposed scheme. Primary image is passed to the Feature Extractor and MKSA
module to produce Primary Feature vector. Whereas all contextual images are
processed similarly, but with a different network, to produce intermediate
features which are then used by the CFF and CCFF blocks to produce contextual
and comparative features, respectively. All extracted features are used by
Classifier to produce prediction with joint optimization using weighted loss.

In practice, dermatologists base their judgment holistically on multiple
lesions on the same patient [11, 3]. This patient-level contextual information
is frequently used by clinicians to diagnose Melanoma and is especially useful
in ruling out false positives in patients with many atypical nevi. Clinicians
frequently assess skin lesions for biopsy by assessing them in context with
the rest of the lesions on a given patient’s body [3]. As demonstrated in Fig.
1, a lesion with malignancy-predictive features among many similar lesions is
thought not to be as dangerous as an odd lesion on a patient whose other
lesions are more benign-looking. The latter is known in dermatology as the
ugly duckling sign and is frequently used to diagnose Melanoma [11, 12],
especially in patients with multiple melanocytic lesions.

In this paper, a novel method is proposed (similar to clinicians’ method for
identifying Melanoma) to compare primary image with contextual images for
Melanoma detection while addressing above mentioned issues of existing
approaches. The major contributions of the proposed method are summarized
below:

  1. 1.

Patient-level contextual information is incorporated which mimics expert
clinicians’ strategy to identify Melanoma images that have not been explored
by existing approaches so far.

  2. 2.

To aid the feature extraction process of traditional deep learning models in
Melanoma classification, a multi-kernel self attention module (MKSA) is
proposed. This module employs attention on different scales to specifically
enhance important features.

  3. 3.

To selectively concentrate on a few relevant contextual features while
ignoring unnecessary ones, a contextual feature fusion (CFF) module is
proposed. This module extracts necessary features from patient-level
contextual information for primary image classification using an attention
mechanism.

  4. 4.

To mimic the comparing strategy of clinicians, a comparative contextual
feature fusion (CCFF) module is proposed. It compares primary and contextual
images pairwise to generate comparative features for further processing.

  5. 5.

To optimize the proposed method for both primary and contextual images
compound loss function was introduced. It takes ground truth and prediction
for both primary and contextual images and outputs the cost for both primary
and contextual images.

##  II Related Work

###  II-A Hand-crafted methods

At an earlier stage, many manually crafted features were proposed for skin
lesion classification, including texture [13, 14, 15]. However, hand-crafted
features are often not robust and have poor discriminative ability. They also
require high computation due to high dimensions and struggle to capture deeper
semantic information. As such, they are no longer competitive against CNN-
based methods.

###  II-B CNN-based methods

CNN-based approaches showed dominating performance in recent years. [16]
introduces a two-stream deep neural network information fusion framework with
a fusion-based contrast enhancement module and a feature selection framework
module for generating more discriminative features. [17] proposed MB-DCNN
model for simultaneous skin lesion segmentation and classification. Here
different networks mutually transfer knowledge to facilitate each other in the
process of accurately locating and classifying skin lesions. L-CNN [18]
utilizes two feature extraction modules, a feature discrimination network, and
model fusion strategy to improve recognition performance. In [19], the authors
employed a methodology wherein deep features were extracted from multiple
augmented versions of an image using ResNets. Subsequently, these features
were aggregated using Fisher Vector (FV) encoding, and an SVM kernel was used
to finally classify melanoma.

###  II-C Clinically inspired methods

The Taxonomies model [20] introduces a hierarchical structure and an attention
module to mimic the decision-making processes of doctors. However, it only
uses one image as input, ignoring the practice of using multiple inputs, which
is commonly used by dermatologists. Despite their successes, these methods are
limited by their failure to incorporate medical expertise, thereby hindering
the interpretability of their results. In [12] this problem is approached as
outlier detection by variational autoencoder (VAE). Their approach utilizes
reconstruction loss and distance of feature vectors to detect outliers. Even
though their approach shows promising results, their model can only be trained
on a single patient, and thus predicting for a new patient requires the model
to be retrained on that patient’s relevant data. More recently, the clinically
inspired method (CI-Net) [21] proposed a three-step process including zooming,
observation, and comparison, along with a distinguishing strategy to
simultaneously learn from two images and determine if they belong to the same
class. However, this method only compares different features within a single
image, while in practice clinicians compare multiple images to diagnose
melanoma. Additionally, the distinguish strategy employed serves the purpose
of contrastive learning rather than the ”ugly duckling” method used by expert
clinicians. Therefore, although it has achieved promising results, this method
does not adequately incorporate patient-level contextual information.

##  III Methodology

In Melanoma classification, additional lesion images of a patient can provide
important context in providing a decision. For instance, in Fig. 1, even
though all the lesions of patient IP_0139601 seem severe, in truth they are
benign as they all are similar. Whereas, in patient IP_3397861, even though
the 2nd lesion sample(1st row, 2nd column) appears less severe, it is actually
malignant as it is different from other samples. From Fig. 1 it is evident
that images with additional context can be a vital asset for improved
diagnosis of Melanoma as opposed to the traditional method of inferring from a
single image.

The objective of the proposed scheme is to aid the conventional Melanoma
classification process by incorporating information from other images from the
same patient. In the experiment, the original image to classify is referred to
as primary image and the images that provide additional context to the primary
image classification are referred to as contextual images. An overview of
proposed method is described in Fig. 2.

Figure 3: Multi-Kernel Self Attention (MKSA) Module. It consists of 3 parallel
Self-Attention(SA) modules with different scales, whose outputs are merged and
finally added to a Global Attention Pooled input. $\alpha$ is the learnable
weight for addition. Figure 4: Self Attention(SA) Module. It has variable
kernel shape, which allows for processing on multiple scales.

###  III-A Problem Formulation

Let us consider the set of images and their corresponding ground truths are
represented by $\mathbf{A}$, and $\mathbf{B}$, respectively, such that
$A_{i}\in R^{h\times w\times c}$ and $B_{i}\in\\{0,1\\}$. Here,
$i\in\\{1,2,3,\dots N\\}$ and $(h,w,c)$ denote height, width, and channel of a
single image from the set of $N$ images.

To incorporate a set of contextual images on the diagnosis of a particular
primary image, different sets of images can be formed where each set contains
a primary image with several contextual images from a particular patient.
Hence, a set of images, $X$, with $m$ contextual images, and its corresponding
ground truth, $Y$, can be represented as

$\displaystyle X_{i}=\\{P_{i},C_{1i},C_{2i},\dots,C_{mi}\\}$ (1)
$\displaystyle Y_{i}=\\{y^{p}_{i},y^{c}_{1i},\dots,y^{c}_{mi}\\}$ (2)
$\displaystyle P,C\in\mathbf{A};\
y^{p},y^{c}\in\mathbf{B};X_{i}\in\mathbf{X},Y_{i}\in\mathbf{Y}$
$\displaystyle\forall i\in\\{1,\dots,N\\}$

Here, $P_{i}$ denotes primary lesion of the $i_{th}$ set, and $C_{ji}$
represents the $j_{th}$ contextual image of that particular set from the total
$N$ lesions. Whereas, $y^{p},\ y^{c}$ denote the ground truth of a primary and
contextual lesion, respectively.

The objective is to extract optimum primary ($F_{primary}$), contextual
feature ($F_{contextual}$), and comparative features ($F_{comparative}$) from
each set of lesions $X$ for the precise diagnosis, and hence

$\displaystyle F_{primary}^{i}=\mathcal{F}(P_{i})$ (3) $\displaystyle
F_{contextual}^{i}=\mathcal{F}(C_{1i},\dots,C_{mi})$ (4) $\displaystyle
F_{comparative}^{i}=\mathcal{F}(P_{i},C_{1i},\dots,C_{mi})$ (5)
$\displaystyle\forall i\in\\{1,\dots,N\\}$

Hence, $F_{primary}$ represents only the primary lesion, $F_{contextual}$
considers all of the contextual lesions on a particular set, and
$F_{comparative}$ considers each contextual lesion in comparison with the
primary lesion of a particular set.

Afterward, the extracted contextual ($F_{contextual}$) and comparative
features ($F_{comparative}$) are supposed to be integrated with the primary
lesion feature ($F_{primary}$) for the diagnosis of the primary lesion,
$y_{pred}^{p}$. Whereas, a set of diagnoses, ($y^{c}_{pred}$) is also provided
for the contextual lesions that separately consider features from each
contextual lesion. These can be represented as

$\displaystyle
y^{p}_{pred,i}=\mathcal{FC}(F_{primary}^{i},F_{contextual}^{i},F_{comparative}^{i})$
(6) $\displaystyle y^{c}_{pred,ij}=\mathcal{FC}(C_{ij});\ \forall
j\in\\{1,\dots,m\\}$ (7) $\displaystyle
Y^{pred}_{i}=\\{y^{p}_{pred,i},y^{c}_{pred,1i},\dots,y^{c}_{pred,mi}\\}$ (8)
$\displaystyle\forall i\in\\{1,\dots,N\\}$

Where $\mathcal{FC}$ denotes fully connected classifier layers. Finally, the
objective function, $\mathcal{L}_{obj}$, can be defined as

$\displaystyle
L_{obj}=\arg\min_{\theta}\mathcal{L}(\theta,\mathbf{X,Y,Y^{pred}})$ (9)

where $\theta$ denotes the set of network parameters, . Hence, the network
parameter should be optimized for the precise diagnosis of the primary and
contextual lesions.

###  III-B Proposed Multi Kernel Self Attention (MKSA) Module

Conventional methods of feature extraction from images rely on CNN network as
feature extractor to generate 2D feature maps which are flattened and pooled
differently across the channels to generate the final feature vector. But this
method fails to extract inter-area or inter-pixel relations between two non-
adjacent areas in the feature maps. The proposed Multi-kernel Self Attention
(MKSA) module shown in Fig. 3 can generate potent feature vectors by examining
inter-area relations between generated feature maps.

The MKSA module comprises parallel branches of Self Attentions (SA). The self
attention kernels shown in Fig. 4 apply an attention mask to the input feature
map that prioritizes useful information. These attention maps are Global
Average Pooled to produce 1D feature vectors. Then feature vectors are
concatenated, and a fully connected layer is used to reduce their dimension.
Then original vector is added to processed vector in a residual manner with
learnable weight.

Conventional self attention blocks, such as introduced in [22] only examine
the inter-pixel relations whereas in the proposed form of self attention
kernel(SA Kernel in Fig. 4), the inter-pixel relations as well as the inter-
area relations. The MKSA module consists of parallel branches of SA block with
varying kernel sizes (e.g: 1x1, 2x2, and 4x4 kernel size) to extract features
on multiple field-of-vision.

In this experiment, conventional CNN networks such as VGG[23], Resnet[24] and
Densenets[25], EfficientNet[26] are used and since EfficientNets yield the
highest baseline scores, they are prioritized in subsequent ablation studies.

Figure 5: Comparative Contextual Feature Fusion (CCFF) Module (on the left)
comprises of Contextual Feature Fusion (CFF) Module (on the right) and BFF
Module. Figure 6: Binary Feature Fusion (BFF) Module comprises cascaded Binary
Fusion Unit (BFU) Modules. The cascaded BFU modules allow for added depth to
process the features.

###  III-C Proposed Contextual Feature Fusion (CFF) Module

Contextual feature fusion (CFF) module shown in Fig. 7 is proposed to generate
a contextual feature vector by passing the contextual image vectors to the CFF
module. As input, the CFF module takes a group of feature vectors which are
processed individually by a Feature Attention(FA) submodule. This submodule
consists of dense layers that process the inter-feature relations of the input
and assign a probability mask to them. This probability mask is applied to the
input feature vector. So in the resultant output feature mask, the significant
features are prioritized. The processed feature vectors generated from
multiple contextual images are then concatenated.

The subsequent submodule focuses on assigning a probability mask in each of
the group images based on the information available from the processed vectors
and expressing them in a smaller vector. This submodule takes a weighted sum
of each of the features in the processed vectors by multiplying the
concatenated matrix with a weight vector with the same length as the feature
vector. The softmax of this weighted sum assigns a probability of importance
across the features in the feature vectors. The final output is a weighted sum
of the same features of multiple image feature vectors.

###  III-D Proposed Comparative Contextual Feature Fusion (CCFF) Module

Siamese Neural Networks[27] introduced the idea of comparing image feature
vectors generated from the same model to find similarities or dissimilarities
between them. When the image feature vectors are generated from the same
feature extractor model, the l1-distance between the two feature vectors can
represent the feature similarities between the two images.

But utilizing this idea to suit the case of comparing the group images with
the single image poses a problem: the same feature extractor is not being used
to generate feature vectors for the single image (primary image) which is
being classified and the group of images that are providing contextual
information (contextual images). Since utilizing the contextual feature
vectors to aid the primary image classification and utilizing the primary
feature vector to classify the primary image are separate tasks, the feature
extractors assigned to them would create separate features. Finding any
relation between two generated feature vectors would require additional
processing of the feature vectors using information from both features. The
Comparative and Contextual Feature Fusion(CCFF) module is proposed to examine
the features to generate a representation of the similarity or dissimilarity
between the two different types of feature vectors. The CCFF module is
comprised of two submodules: Binary Feature Fusion(BFF) and the Contextual
Feature Fusion Module(CFF) discussed above.

The Binary Feature Fusion(BFF) submodule (shown in Fig. 6) takes two feature
vectors and learns to compare them for extracting underlying relations. This
is done by concatenating the two vectors and propagating them through a
densely connected layer. This layer finally generates a comparative feature
vector from these two feature vectors.

To model more complex features with comparative information, a cascaded
version of this submodule is proposed and the original submodule is extended
to the module in Fig. 5. In this extended submodule, the original comparative
feature vector is concatenated with the two original inputs separately. The
two resultant vectors are passed through densely connected layers
respectively. So the comparative feature vector and the original input feature
vector are jointly processed to create better inputs for the subsequent layers
of the cascaded submodule and the generated features are deeper. At the final
cascaded layer, the processed inputs generate a single comparative feature
vector by the above-mentioned method of concatenation and subsequent
propagation into a fully connected layer. Each unit of the cascaded submodule
is named Binary Feature Unit(BFU). The experiments performed displayed that
the cascaded submodule performs better in extracting deep features compared to
a single BFU unit, so the cascaded submodule is used as part of the proposed
solution.

For each of the contextual images present, its feature vector and the primary
image feature vector are utilized as inputs in the proposed BFF submodule and
the resultant feature vectors are passed into the proposed CFF submodule,
which finally generates a single feature vector containing the information
processed by the proposed submodules.

Figure 7: Contextual Feature Fusion(CFF) Module. It takes multiple feature
vectors as input and generates a single feature vector as output. TABLE I:
Comparison of Performances of Different Modules with Different Backbones

Modules | Contextual Image | DenseNet121 | EfficientNetB3 | EfficientNetB5
---|---|---|---|---
Spec. | Sens. |  | AUC
---
(cv)
| AUC
---
(lb)
Spec. | Sens. |  | AUC
---
(cv)
| AUC
---
(lb)
Spec. | Sens. |  | AUC
---
(cv)
| AUC
---
(lb)
Baseline | No | 0.820 | 0.192 | 0.87 | 0.867 | 0.916 | 0.234 | 0.924 | 0.914 | 0.943 | 0.254 | 0.932 | 0.928
Baseline + MKSA | No | 0.827 | 0.220 | 0.882 | 0.874 | 0.920 | 0.260 | 0.925 | 0.916 | 0.947 | 0.283 | 0.938 | 0.925
Baseline + CFF | Yes | 0.830 | 0.255 | 0.891 | 0.883 | 0.933 | 0.308 | 0.925 | 0.917 | 0.951 | 0.330 | 0.941 | 0.930
| Baseline + CCFF
---
(BFF w 1 BFU only)
Yes | 0.831 | 0.267 | 0.893 | 0.883 | 0.921 | 0.310 | 0.923 | 0.912 | 0.941 | 0.339 | 0.940 | 0.929
Baseline + CCFF | Yes | 0.833 | 0.274 | 0.898 | 0.887 | 0.928 | 0.315 | 0.926 | 0.918 | 0.956 | 0.342 | 0.944 | 0.933
| Baseline + CFF +
---
CCFF
Yes | 0.842 | 0.311 | 0.902 | 0.892 | 0.935 | 0.352 | 0.927 | 0.920 | 0.961 | 0.361 | 0.948 | 0.938
| Baseline + MKSA +
---
CCF
Yes | 0.840 | 0.323 | 0.910 | 0.895 | 0.946 | 0.367 | 0.929 | 0.924 | 0.956 | 0.379 | 0.950 | 0.940
| Baseline + MKSA +
---
CCFF
Yes | 0.841 | 0.342 | 0.913 | 0.899 | 0.950 | 0.380 | 0.931 | 0.925 | 0.964 | 0.395 | 0.952 | 0.942
| Baseline + CFF +
---
MKSA + CCFF
Yes | 0.850 | 0.350 | 0.915 | 0.907 | 0.954 | 0.390 | 0.932 | 0.927 | 0.968 | 0.401 | 0.957 | 0.945

TABLE II: Comparison with Different Existing Approaches

Method Score | AUC | Acc. | Sens. | Spec.
---|---|---|---|---
Proposed | 0.957 | 0.983 | 0.401 | 0.968
CI-Net [21] | 0.920 | 0.955 | 0.321 | 0.914
EfficientNet [26] | 0.911 | 0.942 | 0.203 | 0.980
Taxonomies [20] | 0.907 | 0.950 | 0.282 | 0.931
SE-Net [28] | 0.908 | 0.944 | 0.135 | 0.898
L-CNN [18] | 0.904 | 0.951 | 0.273 | 0.913
MB-DCNN [17] | 0.904 | 0.944 | 0.178 | 0.942
CBAM [29] | 0.896 | 0.938 | 0.153 | 0.926
DenseNet [25] | 0.889 | 0.936 | 0.316 | 0.903
ResNet [24] | 0.884 | 0.940 | 0.254 | 0.897

###  III-E Classifier Network

For the final classification from feature vectors, fully connected layers are
used. The primary classifier uses a non-linear combination of primary,
contextual, and comparative features to classify while the contextual
classifier only uses information from contextual features. They accumulate
high-level convolutional features to find the best non-linear combination to
classify images.

###  III-F Proposed Loss Function

The primary optimization goal of the proposed model is to generate accurate
predictions made by the primary classifier. However, allowing the contextual
classifier to generate predictions for the group of images used as contextual
information and optimizing those predictions too is a secondary task, since
optimizing this would aid the ability of the contextual feature extractor and
classifier to extract meaningful information from the group images.

For a single group of $N$ images, where $(N-1)$ images are supplementary and a
single image is primary, the losses are calculated as:

$\displaystyle\mathcal{L}_{contextual}=\frac{1}{N-1}\sum_{i=1}^{N-1}\mathcal{L}(y_{context_{i}},\hat{y}_{context_{i}})$
(10)
$\displaystyle\mathcal{L}_{primary}=\mathcal{L}(y_{primary},\hat{y}_{primary})$
(11)
$\displaystyle\mathcal{L}_{total}=\alpha*\mathcal{L}_{primary}+(1-\alpha)*\mathcal{L}_{contextual}$
(12)

Here $\mathcal{L}$ is the loss function and $\alpha$ can be adjusted to
selectively prioritize contextual feature quality and primary prediction
result. For our experiments, we used $\alpha=0.8$. The Total Loss is finally
summed over the batches and used to calculate gradients using backpropagation.
For our experiments, we used Binary Cross-entropy as both the primary loss
function and contextual loss function.

###  III-G Proposed Multi-Phase Optimization Algorithm

For the experiments, different conventional image feature extraction models
aided with the MKSA module are used as Feature Extractors. In the first phase,
the Feature Extractor is trained conventionally as a binary classifier with
all the images without any patient-level contextual information so it utilizes
information from only the primary image. Additionally, the feature extractors
are initialized with pre-trained Imagenet weights, which creates a better
starting point for the optimization algorithm. After the optimization, the
Feature Extractor becomes capable of generating feature maps from the primary
image. However, as this phase lacks patient-level contextual information, this
phase can’t reach the optimal result.

In the second phase, the training is done with the scheme shown in Fig. 2.
This scheme utilizes all the contextual images along with primary images. To
preserve the Feature Extractor’s learning, its weights are kept frozen in this
phase. The output of feature extractors and MKSA module is propagated to the
CFF and CCFF modules, where they create potent features which aid in the
classification of the primary image. A secondary task of predicting the
contextual image is also included to aid the optimization process.

In the third phase, to introduce further optimization all modules are fine-
tuned keeping their weights trainable. In this phase, all modules learn
simultaneously with one another which is essential for optimal performance.
Hence, this scheme is capable of exploiting contextual patient-level
information along with single-image information which results in optimal
performance.

##  IV Results and Discussions

###  IV-A Dataset Description

We extensively conduct various experiments on ISIC 2020 Challenge Dataset [3].
This dataset is based on the larger ISIC archive that contains the largest
publicly available collection of quality-controlled dermoscopic images of skin
lesions generated by the international skin imaging collaboration (ISIC). All
images from this dataset include some information corresponding to the
diagnosis and are labeled by some medical research institutes. The dataset
represents $2,056$ patients ($20.8$% with at least one Melanoma) from three
continents, consisting of $33,126$ dermoscopic images. Among those images,
$584$ (1.8%) images are of histopathologically confirmed Melanoma compared
with benign Melanoma mimickers. Each patient has a number of images ranging
from $2$ to $115$. Among all the patients, $1143$ patients have images more
than $10$ having $237$ Melanoma cases. Notably, experiments are conducted on
ISIC20 [3] dataset due to the lack of alternative datasets with comparable
contextual information, as far as our current knowledge extends

###  IV-B Experimental Setup

Different hyper-parameters of the proposed network are chosen through
extensive experimentation for better performance. Adam optimizer is employed
for the optimization of the network during the training of all three phases.
In the first phase, the model is trained with a learning rate of
$\gamma_{1}=10^{-3}$ along with ReduceLROnPlateau [30] scheduler. In the
second phase, the learning rate is reduced to $\gamma_{2}=10^{-4}$ for proper
learning of CFF, CCFF, and MKSA modules and in the third phase, the learning
rate is further reduced to $\gamma_{3}=10^{-5}$ for the preservation of
feature extractor’s weights and proper fine-tuning. For experimentation,
$2\times$ NVIDIA RTX $3090$ GPU is used. The network is trained for $100$
epochs with batch size 32. For proper validation, a five-fold cross-validation
scheme is carried out. To avoid data leakage data split is kept identical for
all three phases. It is ensured that the images kept for the training stage do
not use in the validation stage. To prevent early over-fitting, simple
augmentations namely, Random-Flip, Random-Rotation, Random-Shift, and Random-
Zoom are used.

Figure 8: Visual comparison of the proposed method with a traditional single
image-based method. Each primary image has ground truth (top left) and
prediction (top right) displayed in blue and black, respectively, with and
without contextual images

###  IV-C Comparison with existing SOTA approaches

The proposed solution outperforms existing SOTA single model approaches on
both natural image classification and skin lesion classification, as shown in
Table II. The experiments are conducted with a fixed 5-fold cross-validation
scheme across all models. The model performance is further checked on Kaggle
leaderboard of the ISIC20 dataset, where it achieved a single model AUC score
of 0.9450 on unseen test data. It is mention-worthy that

###  IV-D Ablation Study

####  IV-D1 Effect of different backbone as feature extractor

The baseline experiments are conducted using conventional CNN models, where
EfficientNets[26] and DenseNets[25] outperformed other models. The ablation
studies on the highest-performing models are detailed in Table I. From the
table, it can be observed that the proposed modules perform well across
architectures with varying depths. The highest performance is observed on
EfficientNetB5, which our proposed models augment even further.

####  IV-D2 Effect of the contextual feature

Only a single image (no contextual images) is used in the baseline model which
mimics the traditional deep learning-based approaches. The performance of
different modules with and without contextual images is provided in Table I.
The inclusion of only Contextual Feature (CF) and Comparative Contextual
Feature(CCF) results in improvements of 2.8% and 3.2%, respectively, while
including both provides an improvement of 4.5% for the VGG16 backbone.
Furthermore, the inclusion of MKSA modules with only CFF and CCFF modules
provides around 3.9% and 3.7% improvements while including the MKSA module
with both CFF and CCFF modules results in 5% improvement over baseline with
VGG16 backbone. Hence, contextual features (CFF & CCFF modules) are
contributing to considerable improvements over the baseline performance.
Similar improvements can also be noticeable for other backbones as well.

####  IV-D3 Effect of the Multi Kernel Self Attention (MKSA) Module

In Table I, at the first stage (without contextual information), the
introduction of this block to the backbone improved the AUC score by 1.175% on
average across the architectures used. When used in conjunction with the CCF
block, this block increased the AUC score too, which can be observed from the
table. The table also demonstrates that this module boosted the AUC score when
used with the entirety of the proposed solutions.

####  IV-D4 Effect of the number of contextual images

In practice, it is often not possible to have exactly 10 contextual images for
each patient, and Clinicians are often required to base their decisions on the
images available, which would vary from patient to patient. To adapt to this
case, the model is required to be able to make a decision based on a random
number of images per patient.

To model this unique problem in the training method, a new augmentation is
introduced. In this method, contextual images are randomly replaced with
images that are black throughout, and their corresponding labels are set to 0.
The model is then trained on these images as input, where the model is
optimized to perform even when there is no relevant information in some of the
input images. This is done to make the model more robust to the availability
of contextual images. The model performance is found to be robust with
changing the number of available contextual images. Even with up to 50% of the
contextual images blacked out randomly, the AUC score is 0.891, and with only
1 image randomly blacked out, the score increases to 0.90.

####  IV-D5 Effect of the proposed Multi-phase optimization algorithm

Initially, the module consisting of the baseline feature extractor with an
MKSA block is trained on the dataset without any contextual module with a
learning rate of $\gamma_{1}$ = 0.01, which scored slightly better than the
baseline performance. Then the same module weights are used to initialize the
primary and contextual feature extractors that are used with the subsequent
modules. In the second phase of the experiment, the feature extractor weights
are frozen and only the CFF and CCFF submodules are trained with a learning
rate of $\gamma_{2}$ = 0.0075. This boosted the score by 2% on average. And in
the final phase, the feature extractor weights are unfrozen and trained with a
learning rate of $\gamma_{3}$ = 0.006. On average, this phase yielded another
1.8% boost.

###  IV-E Qualitative analysis of the effect of contextual information

To analyze the improvement of the proposed solution over the baseline,
statistical results such as the AUC score have been used as the determining
metric. The effect of contextual information on the model’s decision is
summarized in Fig. 8. As this problem is a ranking prediction problem, a
sample having more confidence means that sample is more likely to be Melanoma.
In Fig. 8 each row is associated with one patient where the first column
indicates the primary image and the rest of the column indicates contextual
images. On each image there are $3$ labels; on the top left, ground truth is
shown, and on the top right the model prediction is shown with (blue) and
without (black) contextual images respectively, separated by a vertical line.
From Fig. 8 it can be conceived that identifying Melanoma using only primary
images is very difficult. Hence, a model without contextual images fails to
provide a confident and accurate prediction. For primary images (1st column)
it is apparent that the proposed method provides a confident and accurate
prediction for Melanoma whereas the traditional method provides a doubtful and
less accurate prediction. As the proposed method utilizes comparative features
along with contextual features, it has been able to provide a more improved
decision upon observing the patient skin condition from the multiple
contextual images hence resulting in a more confident and accurate state.

##  V Conclusion

In this paper, an automated scheme is proposed to incorporate patient-level
contextual information for accurate diagnosis of Melanoma diseases that
provides exceptional performances with substantial improvement of AUC score
over the existing approaches. The proposed method exploits single image-based
features along with contextual and comparative features to assess Melanoma. It
computes contextual features using a multi-kernel-based self attention
mechanism and comparative features using additional pairwise comparison. The
proposed scheme to identify Melanoma imitates clinicians’ diagnosis strategy
which can be employed for a clinical explanation. The experimental results
indicate that introducing this scheme contributed to increased performance.
Therefore, the proposed scheme can be easily employed for identifying Melanoma
that can be an effective alternative to other state-of-the-art approaches.

## References

  * [1] A. J. Miller and M. C. Mihm Jr, “Melanoma,” _New England Journal of Medicine_ , vol. 355, no. 1, pp. 51–65, 2006.
  * [2] “Cancer facts and figures 2021,” https://www.cancer.org/cancer/melanoma-skin-cancer/about/key-statistics.html, accessed: 2021-01-05.
  * [3] V. Rotemberg, N. Kurtansky, B. Betz-Stablein _et al._ , “A patient-centric dataset of images and metadata for identifying melanomas using clinical context,” _Scientific Data_ , vol. 8, no. 1, pp. 1–8, 2021.
  * [4] F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, “Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: A Cancer Journal for Clinicians_ , vol. 68, no. 6, pp. 394–424, 2018.
  * [5] J. Xingguang, W. Yuan, Z. Luo, Z. Yu _et al._ , “Deep neural network for melanoma classification in dermoscopic images,” in _2021 IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE)_. IEEE, 2021, pp. 666–669.
  * [6] M. Adewunmi, “Enhanced melanoma classifier with VGG16-CNN,” _ScienceOpen Posters_ , 2021.
  * [7] J. R. H. Lee, M. Pavlova, M. Famouri, and A. Wong, “Cancernet-SCa: Tailored deep neural network designs for detection of skin cancer from dermoscopy images,” _arXiv preprint arXiv:2011.10702_ , 2020.
  * [8] A. G. Pacheco and R. A. Krohling, “An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 25, no. 9, pp. 3554–3563, 2021.
  * [9] R. Rastghalam, H. Danyali, M. S. Helfroush, M. E. Celebi, and M. Mokhtari, “Skin melanoma detection in microscopic images using hmm-based asymmetric analysis and expectation maximization,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 25, no. 9, pp. 3486–3497, 2021.
  * [10] L. Song, J. Lin, Z. J. Wang, and H. Wang, “An end-to-end multi-task deep learning framework for skin lesion analysis,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 24, no. 10, pp. 2912–2921, 2020.
  * [11] C. Gaudy-Marqueste, Y. Wazaefi, Y. Bruneu, R. Triller _et al._ , “Ugly duckling sign as a major factor of efficiency in melanoma detection,” _JAMA Dermatology_ , vol. 153, no. 4, pp. 279–284, 2017.
  * [12] M. Mohseni, J. Yap, W. Yolland, A. Koochek, and S. Atkins, “Can self-training identify suspicious ugly duckling lesions?” in _Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1829–1836.
  * [13] R. J. Stanley, W. V. Stoecker, and R. H. Moss, “A relative color approach to color discrimination for malignant melanoma detection in dermoscopy images,” _Skin Research and Technology_ , vol. 13, no. 1, pp. 62–72, 2007.
  * [14] L. Ballerini, R. B. Fisher, B. Aldridge, and J. Rees, “A color and texture based hierarchical k-nn approach to the classification of non-melanoma skin lesions,” in _Color medical image analysis_. Springer, 2013, pp. 63–86.
  * [15] H. Ganster, P. Pinz, R. Rohrer, E. Wildling, M. Binder, and H. Kittler, “Automated melanoma recognition,” _IEEE transactions on medical imaging_ , vol. 20, no. 3, pp. 233–239, 2001.
  * [16] M. Attique Khan, M. Sharif, T. Akram, S. Kadry, and C.-H. Hsu, “A two-stream deep neural network-based intelligent system for complex skin cancer types classification,” _International Journal of Intelligent Systems_ , 2021.
  * [17] Y. Xie, J. Zhang, Y. Xia, and C. Shen, “A mutual bootstrapping model for automated skin lesion segmentation and classification,” _IEEE transactions on medical imaging_ , vol. 39, no. 7, pp. 2482–2493, 2020.
  * [18] L. Wei, K. Ding, and H. Hu, “Automatic skin cancer detection in dermoscopy images based on ensemble lightweight deep learning network,” _IEEE Access_ , vol. 8, pp. 99 633–99 647, 2020.
  * [19] Z. Yu, X. Jiang, F. Zhou, J. Qin, D. Ni, S. Chen, B. Lei, and T. Wang, “Melanoma recognition in dermoscopy images via aggregated deep convolutional features,” _IEEE Transactions on Biomedical Engineering_ , vol. 66, no. 4, pp. 1006–1016, 2018.
  * [20] C. Barata, M. E. Celebi, and J. S. Marques, “Explainable skin lesion diagnosis using taxonomies,” _Pattern Recognition_ , vol. 110, p. 107413, 2021.
  * [21] Z. Liu, R. Xiong, and T. Jiang, “Ci-net: Clinical-inspired network for automated skin lesion recognition,” _IEEE Transactions on Medical Imaging_ , 2022.
  * [22] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [23] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [25] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4700–4708.
  * [26] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [27] G. Koch, R. Zemel, R. Salakhutdinov _et al._ , “Siamese neural networks for one-shot image recognition,” in _ICML Deep Learning Workshop_ , vol. 2. Lille, 2015.
  * [28] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [29] S. Woo, J. Park, J. Lee, and I. S. Kweon, “Cbam: convolutional block attention module. in proceedings of the european conference on computer vision (eccv): 3-19,” 2018.
  * [30] C. A. Lu, “Traffic scene anomaly detection,” Ph.D. dissertation, UTAR, 2017.