# CosSIF: Cosine similarity-based image filtering to overcome low inter-class
variation in synthetic medical image datasets

Mominul Islam1,∗, Hasib Zunair2 and Nabeel Mohammed1
1North South University, Dhaka, Bangladesh
2Concordia University, Montreal, QC, Canada
∗Corresponding Author: Mominul Islam, E-mail<EMAIL_ADDRESS>
###### Abstract

Crafting effective deep learning models for medical image analysis is a
complex task, particularly in cases where the medical image dataset lacks
significant inter-class variation. This challenge is further aggravated when
employing such datasets to generate synthetic images using generative
adversarial networks (GANs), as the output of GANs heavily relies on the input
data. In this research, we propose a novel filtering algorithm called Cosine
Similarity-based Image Filtering (CosSIF). We leverage CosSIF to develop two
distinct filtering methods: Filtering Before GAN Training (FBGT) and Filtering
After GAN Training (FAGT). FBGT involves the removal of real images that
exhibit similarities to images of other classes before utilizing them as the
training dataset for a GAN. On the other hand, FAGT focuses on eliminating
synthetic images with less discriminative features compared to real images
used for training the GAN. Experimental results reveal that employing either
the FAGT or FBGT method with modern transformer and convolutional-based
networks leads to substantial performance gains in various evaluation metrics.
FAGT implementation on the ISIC-2016 dataset surpasses the baseline method in
terms of sensitivity by 1.59% and AUC by 1.88%. Furthermore, for the HAM10000
dataset, applying FABT outperforms the baseline approach in terms of recall by
13.75%, and with the sole implementation of FAGT, achieves a maximum accuracy
of 94.44%.

Keywords: Medical Image Classification; Generative Adversarial Networks;
Cosine Similarity; Swin-Transformer; ViT; ConvNeXt.

##  1 Introduction

Medical image analysis is a critical component of modern healthcare, enabling
accurate diagnosis, effective treatment, and continuous monitoring of various
diseases [1]. The advent of deep learning has created a new horizon in this
field, delivering significant improvements in the early detection and
classification of diseases. Numerous studies have highlighted the efficacy of
deep learning in medical imaging [2, 3], leading to its widespread adoption in
multiple medical domains, including radiology, dermatology, and ophthalmology
[4, 5, 6]. In radiology, deep learning models have surpassed the diagnostic
accuracy of radiologists in detecting breast cancer in mammography images [7].
Similarly, in dermatology, deep learning has demonstrated outstanding
performance in identifying skin cancer from dermoscopy images [8]. In
ophthalmology, deep learning models have been used to diagnose diabetic
retinopathy and age-related macular degeneration from retinal images [9, 10].

Figure 1: The illustration depicts the pipeline of our research, which
involves identifying the minority class from the dataset, oversampling through
GAN and transformation techniques, the adoption of our proposed FBGT and FAGT
methods to mitigate low inter-class variation by leveraging our novel CosSIF
algorithm, and ultimately training classifiers using the augmented dataset. In
the case of multiple minority classes, the pipeline is repeated until the
classifier training stage. It is recommended to view the illustration in
color.

One of the major challenges in developing deep learning models for medical
image analysis is the limited availability of datasets. This issue is
particularly significant in classification tasks, where obtaining a balanced
dataset with high inter-class variation is difficult [11]. Inter-class
variation refers to the differences in appearance between different classes of
images [12]. The scarcity of a balanced dataset arises when obtaining a
sufficient number of images from certain classes is challenging, resulting in
an imbalanced distribution of classes. For example, Zech et al. [13] reported
that training a deep learning model to detect pneumonia in chest radiographs
was significantly impacted by the imbalanced nature of the dataset. The
authors noted that obtaining a balanced dataset with a sufficient number of
images of the positive class was difficult due to the low prevalence of
pneumonia in the population. Therefore, low inter-class variation and class
imbalance in medical image datasets significantly undermine the applicability
of deep learning techniques in medical imaging.

The use of generative adversarial networks (GANs) has increasingly gained
popularity in recent years to address class imbalances in datasets [14]. GANs
are generative models that can produce synthetic images that closely resemble
real images [15]. Several studies have reported success using GANs to address
the class imbalance in various domains, including medical image analysis [16].
The conventional approach for training a GAN involves utilizing every image of
the minority class in a dataset. [17]. However, when dealing with images that
have low inter-class variation, this method can be problematic. This is
because the GAN may generate synthetic images that visually resemble images
from other classes, resulting in a dataset that is technically balanced but
presents challenges for a neural network attempting to distinguish differences
between classes. The is largely attributed to the lack of diversity in the
synthetic images that makes it difficult for a network to learn discriminative
features required for accurate classification.

In response to the challenge of GANs producing visually similar images with
less discriminative features when trained on datasets with low inter-class
variation, we propose a novel filtering algorithm called Cosine Similarity-
based Image Filtering (CosSIF). We utilize CosSIF to introduce two filtering
methods: Filtering Before GAN Training (FBGT) and Filtering After GAN Training
(FAGT). The CosSIF algorithm is employed to determine the similarity between
two sets of images. For instance, in a dataset consisting of two classes, A
and B, CosSIF calculates the similarity of each image from class A with all
the images in class B. The resulting similarity scores generated by CosSIF are
then used by FBGT or FAGT to filter out the most similar or dissimilar images.
FBGT involves removing real images from the minority class that exhibit visual
resemblance to images from other classes before incorporating them into the
training dataset of a GAN. This ensures that the GAN does not learn certain
features that contribute to generating visually similar images. However,
implementing FBGT requires retraining the GAN with the filtered images. In
contrast, FAGT operates on a pre-trained GAN, where similarity calculations
are conducted between the synthetic images generated by the GAN and the real
images used for training the GAN. The architecture of our proposed algorithm
and filtering methods is illustrated in Fig. 1. To evaluate the effectiveness
of our approaches, we perform experiments using modern transformers such as
Vision Transformer (ViT) and Swin Transformer, as well as convolutional-based
networks like ConvNeXt. The key contributions of our work in this paper can be
summarized as follows:

  * •

We propose CosSIF, an image similarity calculation algorithm with cosine
similarity as its backbone, capable of identifying visually similar images of
a specific class to images of another class/classes in a dataset.

  * •

We propose two filtering methods, FBGT and FAGT, to regulate GANs synthetic
image generation capabilities in an effort to reduce low inter-class
variability in medical image datasets.

  * •

We propose a reproducible train-test split for the HAM10000 dataset, which can
facilitate the comparison of our proposed methods with future experiments
conducted by others.

  * •

We experimentally demonstrate that the utilization of FAGT on the ISIC-2016
dataset surpasses the baseline method, MelaNet [18], in terms of sensitivity
by 1.59% and AUC by 1.88%. Furthermore, the utilization of FAGT and FBGT
exceeds the baseline method, IRv2+SA [19], in terms of recall by 13.72% and
13.75%, respectively.

The remaining sections of this paper are organized as follows: In Section 2,
we discuss related studies on class imbalance and low inter-class variation in
medical image classification. Moreover, we explore the usage of cosine
similarity in computer vision. In Section 3, we present a comprehensive
overview of our proposed CosSIF algorithm, as well as the FBGT and FAGT
filtering methods. Furthermore, this section provides detailed descriptions of
our selected GAN architecture and gives a brief overview of our chosen
transformer and convolutional-based network models. In Section 4, we give an
overview of the utilized datasets, present the selected configurations for
classifier and GAN training. Subsequently, we perform experiments by employing
our proposed algorithm and filtering methods. In Section 5, we conduct an
ablation study of our experiments and compare the performance of our trained
classifiers against strong baseline methods. Finally, Section 6 presents the
conclusions and a discussion on possibilities for future work.

##  2 Related Work

Several studies have delved into a multitude of strategies to address class
imbalance in medical image datasets. These approaches encompass oversampling
techniques that involve either transformations or the implementation of
generative adversarial networks (GANs) [20]. For instance, Zunair and Hamza
[18] employed CycleGAN, a GAN model consisting of dual-generator and
discriminator modules, to effectively increase the representation of the
minority class in the dataset [21]. On the other hand, researchers such as
Datta et al. [19] and Lan et al. [22] opted for alternative transformation
methods, adjusting image rotation and focus to diversify the dataset without
resorting to GANs. While these research papers used different methods to
address the class imbalance issue, they did not propose any solutions to
handle the low inter-class variation in the generated synthetic datasets. In
this paper, we aim to address this issue by eliminating images that contain
limited distinguishable features during the oversampling process.

Identifying visually similar images begins with mathematically calculating the
similarity between two images, preferably in a higher dimension. Multiple
formulas exist for this task, including Mean Square Error (MSE), Cosine
Similarity, and Euclidean Distance. In the field of computer vision, the use
of cosine similarity is fairly prevalent for calculating the similarity
between images. Ilham et al. [23] used cosine similarity to compare the vector
representation of a query image with the vector representations of all the
images in the database. Similarly, Kaur et al. [24] proposed a content-based
image retrieval system (CBIR) to assist dermatologists in diagnosing skin
diseases. The system utilized various techniques such as feature extraction,
similarity matching, and cosine similarity to retrieve the most similar images
to the query image. Tao et al. [25] introduced Saliency-Guided Constrained
Clustering approach with cosine similarity (SGC3) for image cosegmentation.
This method employs cosine similarity to calculate the feature similarity
between data points and its cluster centroid. Given the extensive utilization
of cosine similarity in similarity calculations, as demonstrated by the
mentioned authors, we also choose to employ it as the foundation of our
similarity calculation algorithm. Similar to these authors, we transform the
two input images into vectors and calculate their cosine similarity. However,
as we need to calculate the cosine similarities for thousands of images, it
becomes necessary to reduce the pixel dimensions of the input images.
Therefore, we optimize our algorithm by reducing the number of pixels to
64x64. This adjustment significantly improves the computation time of our
algorithm. Moreover, we utilize cosine similarity to compute the cosine
distance, ensuring that the calculated distances are non-negative values.

##  3 Proposed Method

This section describes the main components of our proposed algorithm and
methods. It begins with a comprehensive overview of the CosSIF algorithm,
followed by a detailed explanation of the FBGT and FAGT methods, along with a
comparison between them. Next, the architecture of the GAN is described, along
with a hybrid augmentation process designed specifically to suit the outlined
GAN architecture. Finally, the section concisely discusses the transformers
and convolutional-based network models used for training classifiers.

###  3.1 CosSIF

The details of the Cosine Similarity-based Image Filtering (CosSIF) algorithm
are divided into several parts: class selection, image rescaling, similarity
calculation, optimization, backbone, algorithm, filtering, and adaptability.

#### Class Selection

The CosSIF algorithm begins by selecting a class from a dataset, referred to
as the target class. This target class acts as the anchor for similarity
calculations, while the remaining classes are considered secondary classes.
The target class is denoted as $\textbf{{T}}^{[c]}$, and a secondary class is
denoted as $\textbf{{S}}^{[c]}$, where $c$ represents the class name. The
total number of images in the target class is represented by $p$, and the
total number of images in the secondary class is represented by $q$. Equations
1 and 2 represent all the images within the $\textbf{{T}}^{[c]}$ and
$\textbf{{S}}^{[c]}$ classes, respectively.

$\textbf{{T}}^{[c]}=\\{t^{[c]}_{1},t^{[c]}_{2},\ldots,t^{[c]}_{p}\\}$ (1)

$\textbf{{S}}^{[c]}=\\{s^{[c]}_{1},s^{[c]}_{2},\ldots,s^{[c]}_{q}\\}$ (2)

In the case of multiple secondary classes, they are represented as a set $X$.
Equation 3 represents all the classes within the set $X$, where $m$ denotes
the total number of secondary classes.

$X=\\{{\textbf{{S}}^{[c_{1}]}},{\textbf{{S}}^{[c_{2}]}},\ldots,{\textbf{{S}}^{[c_{m}]}}\\}$
(3)

#### Image Rescaling

Upon selecting the target and secondary classes, all images belonging to these
classes are resized to a smaller size, typically 64x64 pixels by default. This
resizing step enables faster similarity calculations between images. To
achieve even faster computation, the image size can be further reduced.
Conversely, if a more detailed pixel-based computation is desired, the size
can be increased beyond the default 64x64 pixels, albeit with an increase in
computation time.

Figure 2: The illustration portrays the step-by-step process of the CosSIF
algorithm. It commences by selecting the target class ${T^{[c_{2}]}}$ and a
set $X$ comprising secondary classes. Subsequently, the images within the
selected classes undergo resizing to 64x64 pixels. Following this, similarity
scores are calculated for each image in ${T^{[c_{2}]}}$ by comparing them to
all images in $X$. For each image in ${T^{[c_{2}]}}$, a record $\eta$ is
created to store individual similarity scores $I$ and their corresponding
image identifiers. The record $\eta$ is then sorted in descending order, with
the first entry representing the maximum similarity score $I_{max}$. Once the
similarity calculation for all images in ${T^{[c_{2}]}}$ has been completed,
the resulting set of records $R$ is obtained. Finally, $R$ is sorted in
ascending order based on the maximum similarity score $I_{max}$, thereby
concluding the similarity calculation process.

#### Similarity Calculation

Following the image rescaling process, the algorithm proceeds to perform the
similarity calculation. It starts by selecting an image from the target class
$\textbf{{T}}^{[c]}$ and calculates its similarity score, denoted as $I$, with
all the other images in the secondary class $\textbf{{S}}^{[c]}$. If there are
multiple secondary classes, the similarity measure is computed for all images
across the entire set $X$.

During the similarity calculation, a record denoted as $\eta$ is maintained
for each image in $\textbf{{T}}^{[c]}$, storing all the computed similarity
scores along with the corresponding image identifiers (image and class names).
The record $\eta$ is then sorted in descending order based on the individual
similarity scores $I$. Therefore, for any image in $\textbf{{T}}^{[c]}$, the
first entry in the record $\eta$ contains the maximum similarity score,
denoted as $I_{max}$, along with its associated image identifiers.

The algorithm iterates through all images in the target class
$\textbf{{T}}^{[c]}$ and records their similarities and corresponding image
identifiers. Once the iteration process is complete, we obtain a set $R$ of
records, as defined in Equation 4. It is important to note that the total
number of records, denoted by $z$, in $R$, is equal to the total number of
images, denoted by $p$, in $\textbf{{T}}^{[c]}$.

$R=\\{\eta_{1},\eta_{2},\ldots,\eta_{z}\\}$ (4)

Finally, the set $R$ of records is sorted in ascending order based on the
individual maximum similarity scores $I_{\text{max}}$. Figure 2 provides a
detailed illustration of the CosSIF algorithm.

#### Optimization

To tackle the issue of the record size growing excessively large as the number
of images in $\textbf{{S}}^{[c]}$ or $X$ increases, an optimization technique
is introduced in the similarity calculation module. Rather than recording the
similarity for each image in $\textbf{{T}}^{[c]}$ with every other image in
$\textbf{{S}}^{[c]}$ or $X$, only a limited range of images with the highest
similarity scores are recorded. However, the similarity calculation is still
performed for all images in $\textbf{{S}}^{[c]}$ or $X$. This approach
effectively reduces the size of the record and addresses the scalability
concern.

#### Backbone

The CosSIF algorithm analyzes images and computes their level of similarity.
It utilizes cosine similarity to determine the degree of similarity between
two images, as well as cosine distance to measure the positive distance
between them.

The cosine similarity measures the cosine of the angle between two vectors.
Let’s assume that $\mathbf{u}$ and $\mathbf{v}$ are two arbitrary vectors. The
cosine similarity between the vectors is defined by Equation 5, where
$\mathbf{u}\cdot\mathbf{v}$ represents the dot product of $\mathbf{u}$ and
$\mathbf{v}$, and $\|\mathbf{u}\|$x$\|\mathbf{v}\|$ denotes the product of
their magnitudes. For a more detailed visual representation, refer to Figure 3
and 4.

$\begin{split}\text{cosine similarity
$(\mathbf{u},\mathbf{v})$}&=cos(\theta)=\frac{\mathbf{u}\cdot\mathbf{v}}{\mathbf{\|u\|}\text{x}\mathbf{\|v\|}}\\\
\end{split}$ (5)

Figure 3: The illustration depicts the process of calculating the cosine
similarity between two images. It begins by dividing a color image into its
three RGB (red, green, blue) layers, where each layer is represented as a
square matrix. Each layer contains different pixel values, which are then
normalized by dividing each pixel by 255. Next, all layers are flattened into
a vector. Considering that each image has a resolution of 64x64 pixels and
consists of 3 layers, the resulting vector dimension becomes 1x12288. This
procedure is repeated for both Image 1 and Image 2, resulting in two vectors,
$\mathbf{u}$ and $\mathbf{v}$, respectively. The cosine similarity between
these two vectors is then calculated. Figure 4: The graph depicts two
arbitrary vectors $\mathbf{u}$ and $\mathbf{v}$, representing two images,
where the cosine of the angle $\theta$ between these vectors represents the
cosine similarity value.

While cosine similarity measures the similarity between two vectors based on
their angles, it can occasionally yield negative values. This poses a
challenge when comparing it with other computed similarity scores. To overcome
this issue, the cosine distance is calculated by subtracting the cosine
similarity from 1. This calculation ensures that the similarity score is a
non-negative value. The cosine distance is defined the Equation 6.

cosine distance = 1 - cosine similarity (6)

#### Algorithm

The detailed procedure of the CosSIF algorithm is presented in Algorithm 1.
The input consists of the target class $\textbf{{T}}^{[c]}$ and either a
secondary class $\textbf{{S}}^{[c]}$ or a set $X$ of secondary classes. The
output is a set $R$ of records, sorted in ascending order, which holds the
computed similarity scores for each image in $\textbf{{T}}^{[c]}$ compared to
all other images in $\textbf{{S}}^{[c]}$ or $X$.

Algorithm 1 CosSIF

0: Target class
$\textbf{{T}}^{[c]}=\\{t^{[c]}_{1},t^{[c]}_{2},\ldots,t^{[c]}_{p}\\}$,
secondary class
$\textbf{{S}}^{[c]}=\\{s^{[c]}_{1},s^{[c]}_{2},\ldots,s^{[c]}_{q}\\}$ or a set
$X$ of secondary classes.

0: Set $R$ of records, sorted in ascending order.

1: $R=\\{\\}$

2: Resize all images to 64x64.

3: for $t^{[c]}_{1}$ in $\textbf{{T}}^{[c]}$ do

4: $\eta_{1}=\\{\\}$

5: if (secondary class != $X$) then

6: $X$ = $\\{\textbf{{S}}^{[c]}\\}$

7: else

8: for $\textbf{{S}}^{[c]}$ in $X$ do

9: for $s^{[c]}_{1}$ in $\textbf{{S}}^{[c]}$ do

10: Calculate cosine similarity of $t^{[c]}_{1}$ and $s^{[c]}_{1}$. 5

11: Calculate cosine distance. 6

12: Similarity score $I$ = cosine distance

13: Save record $\eta_{1}=\\{t^{[c]}_{1},\\{s^{[c]}_{1},I\\}\\}$

14: end for

15: end for

16:
$\eta_{1}=\\{t^{[c]}_{1},\\{s^{[c]}_{1},I\\}\\},\\{s^{[c]}_{2},I\\},\ldots\\}$

17: Sort $\eta_{1}$ in descending order by similarity scores.

18: $\eta_{1}=\\{t^{[c]}_{1},\\{s^{[c]}_{max},I_{max}\\}\\},\ldots\\}$

19: end if

20: Append $\eta_{1}$ to set $R$.

21: end for

22: Sort $R$ in ascending order by $I_{max}$.

23: Set $R=\\{\eta_{1},\eta_{2},\ldots,\eta_{z}\\}$

#### Filtering

Once the similarity calculation is completed, the filtering process is
initiated. To recap, the similarity calculation generates a set $R$ of
records, as shown in Equation 4, which is subsequently sorted in ascending
order. For each record $\eta$ in $R$, $I_{max}$ represents the maximum
similarity score, and the associated image identifiers are used to identify
the image in $\textbf{{T}}^{[c]}$ that achieves this similarity score with an
image in $\textbf{{S}}^{[c]}$ or $X$.

Since the set $R$ is sorted in ascending order by $I_{max}$, the first record,
$\eta_{1}$, has the lowest $I_{max}$, while the last record, $\eta_{z}$, has
the highest. Therefore, the process of filtering out the most similar images
in begins with the $\eta_{z}$. From there, the filtering process gradually
moves up the list of $\eta$ in $R$, with each subsequent image having a lower
$I_{max}$ than the previous one. Conversely, to filter out the most dissimilar
images, the filtering process starts from the $\eta_{1}$. In this case, the
filtering process gradually moves down the list of $\eta$ in $R$, with each
subsequent image having a higher similarity score than the previous one.

In summary, the CosSIF algorithm enables the filtering of images based on
their cosine similarity, identifying the most similar or dissimilar ones.
However, the algorithm doesn’t directly perform the filtering task; instead,
it generates the essential information needed for filtering. The actual
filtering is accomplished using our proposed FBGT and FAGT methods.

#### Adaptability

The CosSIF algorithm has been designed with future reusability in mind,
allowing it to be applied to various tasks. To facilitate this adaptability,
we have incorporated certain features that may not be immediately useful but
could prove valuable in future works. One such feature relates to the
similarity calculation process, where we provide the option to restrict the
range of saved records.

In the current implementation of our research, we have chosen to limit this
range to only 1. This means that for each image in a selected target class,
there is at most one image from the secondary class that is most similar. This
suffices for our purposes, as the generated set $R$ of sorted records obtained
from the CosSIF algorithm already gives us the most similar or dissimilar
images that can be filtered from the target class.

However, let’s consider a scenario where it is necessary to know all possible
similarities that each image in the target class shares with other images in
the secondary class. In such cases, we can easily modify and reuse the CosSIF
algorithm by adjusting the similarity range. By increasing the range, we can
obtain the desired results and retrieve all the relevant similarities for each
image.

Therefore, the flexibility of the CosSIF algorithm allows it to be applied to
a variety of tasks, and with slight modifications, it can accommodate
different requirements in future applications.

###  3.2 FBGT

The Filtering Before GAN Training (FBGT) aims to eliminate real images from
the minority class that display resemblances to images from other classes
before employing them as the training dataset for a GAN. FBGT method commence
by selecting the target and secondary classes. Here, the target class
$\textbf{{T}}^{[c]}$ represents the minority class within a given dataset, and
the remaining classes are collectively referred to as a set $X$ of secondary
classes. Then, the CosSIF algorithm is employed to calculate the similarity
scores for each image in $\textbf{{T}}^{[c]}$ with all other images in $X$.
CosSIF generates a set $R$ of records which is shorted in ascending order.

Following the completion of the similarity calculation and the generation of
the set $R$ of records, the subsequent step in FBGT focuses on filtering
images from the target class $\textbf{{T}}^{[c]}$. In this step, the number of
images to be filtered from $\textbf{{T}}^{[c]}$ is determined by a
hyperparameter denoted as $\mathcal{\alpha}$, where $0<\alpha<1$. The value of
$\mathcal{\alpha}$ is calculated using the following formula:

$\alpha=\frac{100-\text{\% of images to be removed}}{100}$ (7)

The formula for calculating the number of filtered images, $f$, is given by:

$f=\left\lceil p\times\alpha\right\rceil$ (8)

where, the symbol $\left\lceil\ \right\rceil$ represents the ceiling function,
which rounds up the result of the multiplication to the nearest integer. The
value of $f$ represents the threshold point, which indicates the number of
images that have been filtered from $\textbf{{T}}^{[c]}$. The newly filtered
target class, $\textbf{{T}}^{[c]}_{filtered}$, composed of $f$ images is given
by:

${\textbf{{T}}^{[c]}_{filtered}=\\{t^{[c]}_{1},t^{[c]}_{2},\ldots,t^{[c]}_{f}\\}}$
(9)

$\textbf{{T}}^{[c]}_{filtered}$ is the output of FBGT method. It contains the
newly filtered images that are going to be used for oversampling.

###  3.3 FAGT

The Filtering After GAN Training (FAGT) method calculates similarities between
the synthetic images generated by a GAN and real images of the class on which
the GAN was trained. In this method, the target class $\textbf{{T}}^{[c]}$
consists of images that are synthetically generated via a trained GAN, while
the secondary class $\textbf{{S}}^{[c]}$ composed of real images, serves as
the training dataset for that GAN. It is important to note that in FAGT, there
is no possibility of having a set of secondary classes. Following the
selection of $\textbf{{T}}^{[c]}$ and $\textbf{{S}}^{[c]}$, the FAGT method
utilizes the CosSIF algorithm, leading to the generation of a set R of
records.

In the FAGT method, the process of filtering images from the target class
$\textbf{{T}}^{[c]}$ becomes more a bit more complex, compared to FBGT. Unlike
FBGT, where the number of images in the filtered target class $f$ is not user-
defined but rather calculated using Equations 7 and 8, in FAGT, the value of
$f$ is determined by the user. This value represents both the number of images
in the filtered target class $\textbf{{T}}^{[c]}_{filtered}$ and the number of
images required for oversampling. In FAGT, $f$ is considered a constant value.
To control the output of the filtering process, the hyperparameter $\alpha$ is
used to calculate the value of $p$, which denotes the number of synthetic
images generated by a GAN. The formula for calculating $p$ is given by:

$p=\left\lceil\frac{f}{\alpha}\right\rceil$ (10)

While $p$ is fixed in the FBGT method, it is variable in the FAGT method. This
is because the quality of the synthetic images produced by a GAN can vary,
leading to changes in the value of $p$. If a GAN produces synthetic images
with fewer discriminative features compared to the real images, more images
need to be filtered out from a larger set of images, resulting in an increased
value of $p$. Conversely, if a GAN is capable of generating images with
similar discriminative features compared to real images, then the value of $p$
decreases. This implies that fewer images need to be filtered out from a
smaller set of synthetic images. Thus, in the FAGT method, $p$ behaves more
like a hyperparameter.

Figure 5 depicts this dependency in two setups, namely $\Psi_{1}$ and
$\Psi_{2}$. In the first setup, $\Psi_{1}$, it is assumed that the GAN
produces more random images with significant deviations from the real images,
thereby necessitating a higher filtering requirement. Conversely, the second
setup, $\Psi_{2}$, assumes that the GAN generates synthetic images that
closely resemble the real images, resulting in a decreased need for filtering.

Figure 5: A visual representation of the correlation between the total number
of images, $p$, in the target class $\textbf{{T}}^{[c]}$, and the total number
of images, $f$, in the filtered target class $\textbf{{T}}^{[c]}_{filtered}$,
employing the FAGT method, demonstrates that the extent of necessary filtering
is influenced by the GAN’s capacity to generate synthetic images that closely
resemble the real images used to train the GAN.

###  3.4 Binary vs Multiclass Classification

In both the FBGT and FAGT methods, it is possible to eliminate similar and
dissimilar images from a calculated set $R$ of records. In binary
classification, the removal of similar images from one class and dissimilar
images from another class enhances the distinction between the two classes,
resulting in improved filtering outcomes. However, in multiclass
classification, it is crucial to avoid removing dissimilar images, as
eliminating these images in relation to all other classes can lead to the loss
of images that possess distinct features essential for accurate
classification. Therefore, for multiclass classification, it is recommended to
eliminate only the similar images.

###  3.5 FBGT vs FAGT

The FBGT and FAGT methods are two approaches that produces more robust
oversampled datasets that reduce the issue of low inter-class variation.
However, there are differences between these two methods that need to be
considered.

The FBGT method requires retraining the GAN with the newly filtered dataset,
which can be a time-consuming process. Therefore, it may not be practical to
use this method when dealing with multiclass classification problems that
require oversampling for several classes. In contrast, the FAGT method can be
applied to a pre-trained GAN, making it faster than the FBGT as it does not
require retraining. However, the FAGT method requires filtering more images
since $p$ is a variable for this method, resulting in longer computation time
for filtering compared to the FBGT.

In the FBGT method, both the target class and secondary class consist of real
images. This implies that the set R of records obtained after the similarity
calculation is universal and can be utilized later by anyone to filter out
images. However, in the FAGT method, the target class is composed of synthetic
images randomly generated by a trained GAN, which necessitates the
recalculation of similarity each time the method is employed. As a result, the
FBGT method is more efficient than the FAGT method when it comes to filtering
images.

Furthermore, it is essential to address a potential question regarding the
FAGT method. Although similarities between real images of a specific class and
real images from other classes are not directly calculated, the effectiveness
of the method lies in the context of medical image datasets. Typically, all
classes in such datasets consist of similar types of images (e.g., skin
lesions, CT scans) but at different stages. When the GAN generates images that
do not distinctly resemble the real images used during its training, these
synthetic images may end up closely resembling images from other classes. As a
result, the removal of synthetic images that deviate from the real images
serves as a multipurpose filtering method. On one hand, it contributes to
generating images with greater discriminative features, while on the other
hand, it effectively addresses the issue of low inter-class variation.

###  3.6 GAN Architecture

The FBGT and FAGT methods are independent of GAN architecture, meaning that
the filtering process remains constant regardless of any selected GAN
framework. However, the choice of GAN architecture is often determined by the
total image size in the dataset. Typically, training a GAN requires a large
number of images. However, in medical image analysis, the minority class often
consists of an extremely low volume of images, posing a challenge for the GAN
to converge during training. Consequently, we tend to choose a GAN
architecture that performs well with a small set of images.

In this paper, we use StyleGAN2-ADA as our GAN architecture. StyleGAN2-ADA
builds upon the improvements of StyleGAN2, with the key enhancement being the
addition of adaptive data augmentation (ADA) techniques. [26]. This method
allows for the generation of high-quality, diverse images even when the
dataset is small or imbalanced, making it a valuable tool in medical image
analysis and other applications where training data may be limited.
StyleGAN2-ADA consists of two main parts: the generator and the discriminator.
The generator’s objective is to create realistic images, while the
discriminator’s goal is to differentiate between real and generated images.

#### Generator

The generator $G$ consists of several key components: the mapping network, the
synthesis network, and the Adaptive Instance Normalization (AdaIN) layers. The
mapping network $h$ takes a latent code $z$ and maps it to a style vector $w$:

$w=h(z)$ (11)

The synthesis network $g$ takes the style vector $w$ and a noise tensor $n$,
and generates an image $x$:

$x=g(w,n)$ (12)

Adaptive instance normalization (AdaIN) [27] is used to modulate the feature
maps in the synthesis network with the style vector $w$. Given a feature map
$F$ and the style vector $w$, AdaIN produces a styled feature map
$F^{\prime}$:

$F^{\prime}=\text{AdaIN}(F,w)$ (13)

#### Discriminator

The discriminator $D$ is a convolutional neural network that classifies
whether an input image $x$ is real or generated. It takes an image $x$ as
input and outputs a scalar probability value $y$:

$y=D(x)$ (14)

#### Adaptive Discriminator Augmentation

In StyleGAN2-ADA, the discriminator is trained on both the real images and
their augmented counterparts. The augmentation function $V$ takes an image $x$
and an augmentation parameter $\mu$ to produce an augmented image
$x^{\prime}$:

$x^{\prime}=V(x,\mu)$ (15)

#### Loss

Both the generator and discriminator losses are based on the binary cross-
entropy loss function. The generator seeks to minimize its loss, which
represents the difference between the discriminator’s output on generated
images and the target output. In essence, the generator aims to maximize the
probability of the discriminator classifying the generated images as real:

$\displaystyle L_{G}=-\mathbb{E}_{z\sim y(z)}[\log D(g(h(z),n))]$ (16)

Here, $z$ is a random latent code sampled from the prior distribution $y(z)$,
$h$ is the mapping network, $g$ is the synthesis network, $n$ is the noise
tensor, and $D$ is the discriminator. $\mathbb{E}$ is expectation, which
represents the average value of the expression inside the brackets.

The discriminator aims to minimize its loss, which consists of two parts: the
difference between the discriminator’s output on real images and the target
output, and the difference between the discriminator’s output on generated
images and the target output:

$\displaystyle L_{D}$ $\displaystyle=-\mathbb{E}_{x\sim
y_{\text{data}}(x)}[\log D(x)]$ $\displaystyle\quad-\mathbb{E}_{z\sim
y(z)}[\log(1-D(g(h(z),n)))]$ (17)

Here, $x$ is an image sampled from the true data distribution
$y_{\text{data}}(x)$, and the other variables have the same meaning as in the
generator loss.

###  3.7 Hybrid Augmentation

Typically, oversampling via a GAN involves merging the real images with
generated synthetic images after the GAN training. However, in some datasets,
certain classes have an extremely low volume of training images. This makes
the learning process of a GAN really difficult, even when using StyleGAN2-ADA.

The architecture of StyleGAN2-ADA includes a component known as adaptive
discriminator augmentation (ADA), which is vital for training with a small
number of images. During training, this component takes an input image $x$ and
produces an augmented image $x^{\prime}$, as expressed in Equation 15. To
increase the variability of $x^{\prime}$, it’s important for the training
dataset to contain sufficient variation. Therefore, we perform a minor
oversampling of the minority classes by applying various transformations to
the images before using them as a training dataset for StyleGAN2-ADA. These
transformations include adjusting the focus, rotating the images, shifting
their positions, and flipping them horizontally or vertically. The
oversampling via transformations improves the variability of $x^{\prime}$
during the process, which enhances the quality of the synthetically generated
images produced by StyleGAN2-ADA.

###  3.8 Model Architectures

To assess the efficacy of our FBGT and FAGT methods, we employ pre-trained
transformer and convolutional-based models. We train these models using the
oversampled dataset, incorporating the FBGT and FAGT methods in some instances
and excluding them in others for comparison purposes. For our experiments, we
utilize pre-trained Swin Transformer [28], Vision Transformer (ViT) [29], and
ConvNeXt [30] models. We fine-tune these models by adapting their output
layers to accommodate the classes within our dataset.

The Swin Transformer, proposed by Liu et al. [28], is a hierarchical
transformer model specifically designed for computer vision tasks. It
introduces a local representation to capture both local and global context,
using a shifted window-based self-attention mechanism and a hierarchical
architecture. The Vision Transformer (ViT), introduced by Dosovitskiy et al.
[29], applies the transformer architecture to computer vision tasks by
dividing input images into patches and processing them as tokens with
positional encodings. It has shown excellent performance on large-scale
datasets but is known to be data-hungry. ConvNeXt, a model introduced by Liu
et al. [30], demonstrates the potential of pure ConvNets by modernizing a
standard ResNet to compete with the performance of Vision Transformers.

##  4 Experiments

In this section, extensive experiments are carried out to assess the
performance of the proposed FBGT and FAGT methods, comparing them with strong
baseline methods. The source code required to reproduce the experimental
results is available at: https://github.com/mominul-ssv/cossif

###  4.1 Datasets

The effectiveness of FBGT and FAGT methods is analyzed using two datasets:
ISIC-2016 and HAM10000. The ISIC-2016 dataset is employed to test both methods
for binary classification, while the HAM10000 dataset is used for multiclass
classification. Both datasets exhibit significant class imbalance with low
inter-class variation, making them ideal choices for testing the filtering
methods.

#### ISIC-2016

The ISIC-2016 Task 3 dataset contains 900 training and 379 testing dermoscopic
images for skin lesion analysis and melanoma classification. With 173
malignant and 737 benign lesions in the training set, and 75 malignant and 304
benign lesions in the testing set, the dataset exhibits class imbalance [31].
The images of malignant and benign lesions exhibit similar appearances,
leading to low inter-class variation within the dataset, as observed in Figure
6 (a). Figure 7 depicts the number of images in each class along with the
train-test split.

Figure 6: The visual depictions of images from various classes indicate low
inter-class variation both in the ISIC-2016 dataset, as shown in (a), and the
HAM10000 dataset, as shown in (b).

#### HAM10000

The HAM10000 dataset consists of 10,015 clinical images of skin lesions,
sourced from various locations worldwide and annotated by dermatologists. The
dataset includes seven classes of skin lesions: actinic keratoses and
intraepithelial carcinoma (akiec) with 327 images, basal cell carcinoma (bcc)
with 514 images, benign keratosis-like lesions (bkl) with 1,099 images,
dermatofibroma (df) with 115 images, melanoma (mel) with 1,113 images,
melanocytic nevi (nv) with 6,705 images, and vascular lesions (vasc) with 142
images [32]. This distribution results in a highly unbalanced dataset.
Moreover, the images also exhibit low inter-class variation, as can be seen in
Figure 6 (b).

Figure 7: The bar plot displays the train-test split and the number of samples
in each class of the ISIC-2016 dataset, where substantial class imbalance is
present in both the training and testing datasets.

The HAM10000 dataset does not include a predefined train-test split. This is
particularly problematic as with no pre-defined split, it is difficult to
compare the performance of our work with already existing state-of-the-art
approaches. Some research papers focus on demonstrating high accuracy, which
can sometimes involve manipulating the associated test data. Moreover, most
work does not provide a reproducible train-test split, leading to difficulties
in verifying the reported results.

Therefore, to address this issue, we employ a reproducible train-test split in
our work. We partition the dataset into 9,187 images for training and 828 for
testing. This partitioning process entails removing duplicates from the test
set, which are composed of identical images with slight visual augmentations.
Consequently, the training set contains these augmented images, while the test
set is devoid of different augmentations of the same images. To perform the
split, we use the scikit-learn train-test split library [33], providing a
random state of 42 as an input parameter. This approach ensures consistency in
the images within the training and testing sets for future experiments. Figure
8 depicts the quantity of images in each class along with the distribution of
the train-test split.

Figure 8: The bar plot provides a visual representation of the train-test
split and the logarithmic scale depiction of the sample distribution across
different classes within the HAM10000 dataset, revealing a significant class
imbalance in both the training and testing datasets.

###  4.2 Preprocessing

In our experiment, we resize images from both datasets to a resolution of
256x256 pixels to ensure that the minority classes meet the criteria for being
utilized as training datasets for the GAN. We opt to oversample both benign
and malignant classes in the ISIC-2016 dataset and oversample all classes,
except for melanocytic nevi (nv), in the HAM10000 dataset, as it already has a
sufficient number of real images available and is the majority class.

###  4.3 Dataset Filtering

The dataset filtering process involves the utilization of the FBGT and FAGT
methods on both the ISIC-2016 and HAM10000 datasets. Furthermore, as outlined
in the methodology, we apply hybrid augmentation techniques to both datasets.
This involves oversampling images through transformations, while
simultaneously training GANs to generate synthetic images. We conduct a total
of three primary experiments in our study. Experiment I utilizes the FBGT
method, Experiment II employs the FAGT method, and Experiment III does not
employ either the FBGT or FAGT methods.

#### Experiment I

In Experiment I, we apply the FBGT method to both benign and malignant classes
of the ISIC-2016 dataset and the akiec, bcc, bkl, df, mel, and vasc classes of
the HAM10000 dataset. This involves conducting similarity calculations using
CosSIF and filtering real images from the GAN training dataset that exhibit
the highest similarity scores with images from other classes. We then perform
minor oversampling via transformation with the newly filtered images
associated with each class, followed by conducting GAN training for all the
selected classes using the associated filtered GAN training datasets. Finally,
we employ the trained GANs to generate synthetic images for each selected
class, consequently resolving the issues of low inter-class variation. For a
visual representation of this process, please refer to Fig. 1.

In the FBGT method, we have a hyperparameter called $\alpha$ that determines
the number of images to be filtered from the real images. Instead of randomly
selecting a number for filtering, we consider three specific values for
$\alpha$. The variation in the number of filtered images for different
$\alpha$ values when using the FBGT method can be observed in Table 2 and 1
for the HAM10000 and ISIC-2016 datasets, respectively.

Table 1: The variation in the number of filtered images, denoted as $f$, from
the total number of real/synthetic images, denoted as $p$, for three different
values of $\alpha$ when implementing the FBGT and FAGT methods on the benign
and malignant classes of the ISIC-2016 dataset.

|  | Classes
---|---|---
|  | Benign | Malignant
| $\alpha$ | Total $p$ | Filtered $f$ | Total $p$ | Filtered $f$
|  $\alpha=$ 0.80 |  | 582 |  | 139
|  $\alpha=$ 0.85 | 727 | 618 | 173 | 148
FBGT |  $\alpha=$ 0.90 |  | 655 |  | 156
|  $\alpha=$ 0.75 | 1441 |  | 1530 |
|  $\alpha=$ 0.80 | 1351 | 1081 | 1435 | 1148
FAGT |  $\alpha=$ 0.85 | 1271 |  | 1350 |

Table 2: The variation in the number of filtered images, denoted as $f$, from
the total number of real/synthetic images, denoted as $p$, for three different
values of $\alpha$ when implementing the FBGT or FAGT methods on the HAM10000
dataset.

|  | Classes
---|---|---
|  | akiec | bcc | bkl | df | mel | vasc
| $\alpha$ | Total $p$ | Filtered $f$ | Total $p$ | Filtered $f$ | Total $p$ | Filtered $f$ | Total $p$ | Filtered $f$ | Total $p$ | Filtered $f$ | Total $p$ | Filtered $f$
|  $\alpha=$ 0.80 |  | 244 |  | 391 |  | 827 |  | 88 |  | 864 |  | 106
|  $\alpha=$ 0.85 | 304 | 259 | 488 | 415 | 1033 | 879 | 109 | 93 | 1079 | 918 | 132 | 113
FBGT |  $\alpha=$ 0.90 |  | 274 |  | 440 |  | 930 |  | 99 |  | 972 |  | 119
|  $\alpha=$ 0.75 | 5624 |  | 5368 |  | 3241 |  | 7201 |  | 3142 |  | 7181 |
|  $\alpha=$ 0.80 | 5272 | 4218 | 5032 | 4026 | 3038 | 2431 | 6751 | 5401 | 2946 | 2357 | 6732 | 5386
FAGT |  $\alpha=$ 0.85 | 4962 |  | 4736 |  | 2860 |  | 6354 |  | 2772 |  | 6336 |

#### Experiment II

In Experiment II, we apply the FAGT method to pre-trained GANs that are
individually trained using the real images from the benign and malignant
classes of ISIC-2016, as well as the akiec, bcc, bkl, df, mel, and vasc
classes of the HAM10000 dataset. By utilizing these pre-trained GANs, we
generate synthetic images for each class. Subsequently, we perform similarity
calculations to compare the generated synthetic images with the real training
images. Based on the similarity results obtained from the calculations, we
selectively remove synthetic images with lower discriminative features
associated with the selected class. This process ensures that the filtered
synthetic images closely resemble features of the real images used to train
the GAN.

Like the FBGT method, the FAGT method also utilizes the hyperparameter known
as $\alpha$ to determine the number of images to be filtered from the
synthetic images. Similarly, we consider three specific values for $\alpha$,
and the variations in the number of images can be observed in Table 2 and 1
for the HAM10000 and ISIC-2016 datasets, respectively. To visualize the
internal processes of the FAGT method, please refer to Fig. 1.

#### Experiment III

In Experiment III, neither the FBGT nor FAGT methods are employed. Instead, we
utilize the same pre-trained GAN as used in the FAGT method, but without
implementing the similarity calculation and filtering process. This experiment
is referred to as No-Filtering. Unlike the other experiments, No-Filtering
does not involve any hyperparameter tuning. This experiment is solely
conducted to analyze the efficacy of the FBGT and FAGT methods.

###  4.4 Dataset Augmentation

Dataset augmentation is performed after the completion of dataset filtering.
To achieve the final dataset augmentation for each class, we combine the real
images with a batch of oversampled images obtained through transformations, as
well as the synthetic images generated by GANs. During Experiment I,
Experiment II, and Experiment III, the output is the final augmented dataset.
The final augmented datase, as shown in Figure 9, consists of 2000 images for
both the benign and malignant classes, effectively addressing the class
imbalance in the ISIC-2016 dataset. Similarly, in Figure 10, the final
augmented HAM10000 dataset is displayed, with the akiec, bcc, bkl, df, mel,
nv, and vasc classes each containing 6042 images. We chose this number as it
matches the number of images in the overrepresented nv class, thus resolving
the class imbalance present in the HAM10000 dataset by oversampling the
remaining classes to this range.

Figure 9: The pie charts show the composition of the augmented datasets,
indicating the contributions of real, transformed, and synthetic images for
the benign and malignant classes of the ISIC-2016 dataset. Figure 10: The pie
charts visually represent the composition of the augmented dataset, showcasing
the contributions of real, transformed, and synthetic images for all seven
classes of the HAM10000 dataset.

Although the final augmented ISIC-2016 dataset contains 2000 images for each
class and the HAM10000 dataset contains 6042 images for each class, it is
important to note that these numbers are predetermined at the beginning of our
experiments. There is a relationship in the FAGT method between the number of
generated synthetic images and the number of filtered images. In this case,
the number of filtered images is fixed and determined based on the size of the
final augmented dataset. For example, in the ISIC-2016 dataset, we need to
filter 1081 synthetic images from the benign class. This number is not
randomly generated but derived from the size of the final augmented dataset.
Therefore, if the final augmentation consists of 2000 images, with 727 real
images and 192 images oversampled through transformations, the required number
of synthetic images is calculated as (2000 - (727 + 192) = 1081). Thus, 1081
is the constant number used to control the output of the FAGT method. If the
GAN generates more random images that don’t resemble real images of the benign
class, a larger pool of synthetic images must be generated, from which 1081
images are filtered. Conversely, if the GAN generates images that closely
resemble real images of the benign class, a smaller pool of generated images
is sufficient for filtering 1081 images.

###  4.5 GAN Configuration

As mentioned in the methodology section, we employ StyleGAN2-ADA as our GAN
architecture. Each selected class for oversampling is trained using the same
StyleGAN2-ADA configuration. In this configuration, we employ 400 kimg, which
is equivalent to 400 epochs. Typically, StyleGAN2-ADA necessitates a larger
number of epochs to generate realistic-looking synthetic images. However, for
our experiment, we set kimg equal to 400 to accommodate our hardware
constraints. Despite this limitation, we still acquire satisfactory synthetic
images that fulfill our needs for analyzing the efficacy of our filtering
methods.

###  4.6 Training Classifiers

As outlined in the methodology section, we utilize the Swin Transformer, ViT,
and ConvNeXt models for training our classifiers. Specifically, we use the
pre-trained versions of these models and fine-tune them using our final
augmented datasets. To facilitate training, we resize the images in the final
augmented datasets to 224x224 pixels. For optimization during training, we use
AdamW, which is an algorithm designed for training deep learning models that
extends the Adam optimizer to include weight decay regularization. We use a
learning rate of $5e^{-5}$ during training for all these models.

###  4.7 Evaluation Metrics

In classification tasks, it is essential to select appropriate evaluation
metrics to accurately assess model performance. This section discusses various
evaluation metrics, including recall, F1-score, sensitivity, accuracy, and
AUC, which are employed to analyze the performance of modern transformer and
convolutional-based network models and evaluate the efficacy of the FBGT and
FAGT methods.

#### Recall

Recall, also known as sensitivity, measures the proportion of actual positive
instances that are correctly predicted as positive. For binary classification,
recall can be defined as:

$Recall=\frac{TP}{TP+FN}$ (18)

where $TP$ represents the number of true positives and $FN$ represents the
number of false negatives. For multiclass classification, macro-average recall
is utilized. It is computed by calculating the recall for each class
individually, treating each distinct class as a positive class and the
remaining classes as negative classes. Then, the average of these recall
values is taken. The formula for macro-average recall is given by:

$Recall_{macro}=\frac{1}{k}\sum_{i=1}^{k}\frac{TP_{i}}{TP_{i}+FN_{i}}$ (19)

where $k$ represents the number of classes.

#### F1-score

The F1-score, which is the harmonic mean of precision and recall, provides a
balance between the two metrics and is especially useful when dealing with
imbalanced datasets or when both false positives and false negatives are of
concern. In multiclass classification, we use the macro-average F1-score,
which is calculated by computing the F1-score for each class individually and
then taking the average of these values. The formula for macro-average
F1-score is given by:

$F1\text{-}score_{macro}=\frac{1}{k}\sum_{i=1}^{k}2\cdot\frac{Precision_{i}\cdot
Recall_{i}}{Precision_{i}+Recall_{i}}$ (20)

where $Precision=\frac{TP}{TP+FP}$. By using the macro-average F1-score, we
can obtain an overall performance measure of the multiclass classification
model, while taking into account the performance of each individual class.

#### Accuracy

Accuracy measures the proportion of correctly predicted instances over the
total number of instances. For binary classification, accuracy can be defined
as:

$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (21)

For multiclass classification, accuracy can be calculated as:

$Accuracy=\frac{\sum_{i=1}^{k}TP_{i}}{\sum_{i=1}^{k}(TP_{i}+TN_{i}+FP_{i}+FN_{i})}$
(22)

#### AUC

The AUC (Area Under the Curve) is a popular evaluation metric for binary
classification models, and it measures the model’s ability to distinguish
between positive and negative classes. The AUC is the area under the receiver
operating characteristic (ROC) curve, which is a plot of the true positive
rate (TPR) against the false positive rate (FPR) at different classification
thresholds. The formula for AUC for binary classification can be written as:

$AUC_{binary}=\int_{0}^{1}TPR(FPR^{-1}(t))dt$ (23)

where $TPR$ is the true positive rate, $FPR^{-1}$ is the inverse of the false
positive rate, and $t$ is the threshold value. For multiclass classification,
the AUC is generally calculated using the one-vs-rest (OVR) approach. In the
OVR approach, we treat each class as the positive class and the remaining
classes as the negative class, and we compute the AUC for each class
separately. Then, we take the average of these AUC values to obtain the
overall AUC score. The formula for AUC for multiclass classification using the
OVR approach can be written as:

$AUC_{multiclass}=\frac{1}{k}\sum_{i=1}^{k}AUC_{i}$ (24)

where $AUC_{i}$ represents the AUC value for the $i$-th class obtained using
the OVR approach.

###  4.8 Baselines

Our baseline for the ISIC-2016 dataset is the results achieved by the MelaNet
model, designed by Zunair and Hamza [18]. However, due to an inconsistency in
the reported sensitivity result in their paper, we recomputed this metric
using the publicly available pre-trained MelaNet model. For the HAM10000
dataset, we use the results of the IRv2+SA model as our baseline, as reported
by Datta et al. [19]. Both baselines demonstrate state-of-the-art
classification performance.

###  4.9 Experimental Setup

The experiments were conducted in two different setups with different hardware
configurations. Setup 1, which employed a Linux server with 2-core Intel(R)
Xeon(R) CPU @ 2.20GHz, 13 GB RAM, and 1x NVIDIA P100 16GB GPU was utilized for
training StyleGAN2-ADA for different minority classes and for training
classifiers for the HAM10000 dataset. Setup 2, which used a Windows machine
with 6-core AMD Ryzen 5 5600H CPU @ 3.30GHz, 16 GB RAM, and 1x NVIDIA RTX 3060
6GB GPU, was utilized for implementing the FBGT and FAGT methods and for
training classifiers for the ISIC-2016 dataset.

##  5 Results

This section presents a thorough performance analysis of the FBGT and FAGT
methods on each variation of the final augmented dataset. The analysis is
conducted in two parts. Firstly, we investigate the difference in performance
while utilizing the FBGT and FAGT methods against No-Filtering. Secondly, we
compare our best models with state-of-the-art baseline models.

The experimental results on the ISIC-2016 dataset are presented in Table 3(a),
using metrics such as sensitivity, FN, and AUC for evaluation. Similarly,
Table 3(b) presents the experimental results on the HAM10000 dataset,
employing evaluation metrics like recall, F1-score, accuracy, and AUC. Since
the classifier training is performed with the final augmented datasets, and
there are three distributions of datasets for three distinct values of
$\alpha$ for both FBGT and FAGT methods, we experimentally conduct
hyperparameter tuning on the Swin Transformer. We select the best-performing
Swin Transformer model by analyzing its performance on the test set and choose
the corresponding $\alpha$ value and its associated augmented dataset to train
the ViT and ConnNeXt models.

###  5.1 Ablation Study

From the analysis presented in Table 3(a) and 3(b), it is evident that our
trained models exhibit improved performance on most evaluation metrics when
utilizing either the FBGT or FAGT methods, as opposed to No-Filtering, on both
the ISIC-2016 and HAM10000 datasets. These performance gains are observed
across different distributions of augmented training datasets, each
corresponding to a different $\alpha$ value. The following provides a detailed
analysis of the observed performance gains.

Table 3: Performance analysis of the fine-tuned Swin Transformer, ViT, and
ConvNeXt models with the application of FBGT and FAGT methods for different
$\alpha$ values, compared to No-Filtering, on the ISIC-2016 dataset in Table
(a), and the HAM10000 dataset in Table (b).

|  | % |  | %
---|---|---|---|---
|  |  |  | Positive Class
Model | Filtering Method | Sensitivity | FN | AUC
| FBGT ($\alpha=$ 0.80) | 66.67 | 25 | 81.36
|  ($\alpha=$ 0.85) | 57.33 | 32 | 83.13
|  ($\alpha=$ 0.90) | 62.67 | 28 | 82.81
| FAGT ($\alpha=$ 0.75) | 62.67 | 28 | 83.40
|  ($\alpha=$ 0.80) | 58.67 | 31 | 83.45
|  ($\alpha=$ 0.85) | 58.67 | 31 | 83.17
Swin-Transformer | No-Filtering | 57.33 | 32 | 80.68
| FBGT ($\alpha=$ 0.80) | 60.00 | 30 | 83.91
| FAGT ($\alpha=$ 0.75) | 72.00 | 21 | 82.47
ViT | No-Filtering | 53.33 | 35 | 82.94
| FBGT ($\alpha=$ 0.80) | 58.67 | 31 | 79.38
| FAGT ($\alpha=$ 0.75) | 57.33 | 32 | 80.01
ConvNeXt | No-Filtering | 56.00 | 33 | 75.46

(a) Dataset: ISIC-2016

|  | %
---|---|---
|  | Macro Average |  | Average
Model | Filtering Method | Recall | F1-score | Accuracy | AUC
| FBGT ($\alpha=$ 0.80) | 81.82 | 83.94 | 94.08 | 98.03
|  ($\alpha=$ 0.85) | 80.60 | 81.61 | 93.48 | 96.33
|  ($\alpha=$ 0.90) | 79.12 | 82.40 | 93.84 | 97.44
| FAGT ($\alpha=$ 0.75) | 81.90 | 80.84 | 93.36 | 96.95
|  ($\alpha=$ 0.80) | 79.30 | 81.55 | 93.72 | 97.17
|  ($\alpha=0.85$) | 82.48 | 81.90 | 94.08 | 97.23
Swin-Transformer | No-Filtering | 78.13 | 79.36 | 93.24 | 97.15
| FBGT ($\alpha=$ 0.80) | 83.34 | 81.85 | 92.39 | 97.67
| FAGT ($\alpha=$ 0.85) | 85.94 | 82.77 | 93.00 | 98.27
ViT | No-Filtering | 83.19 | 81.12 | 91.79 | 96.95
| FBGT ($\alpha=$ 0.80) | 81.38 | 81.58 | 93.60 | 97.65
| FAGT ($\alpha=$ 0.85) | 81.80 | 84.06 | 94.44 | 97.54
ConvNeXt | No-Filtering | 79.86 | 79.98 | 92.63 | 97.32

(b) Dataset: HAM10000

#### ISIC-2016 Dataset

Table 3(a) provides a detailed analysis of the performance of various models
trained on the ISIC-2016 dataset. It is evident that all trained models, when
utilizing either the FBGT or FAGT methods, exhibit performance improvements
over the No-Filtering approach, which does not incorporate either of the
filtering methods.

Figure 11: The graph provides a visual comparison of the performance of the
trained models in terms of sensitivity and FN count while utilizing the FAGT
and FBGT methods with optimal $\alpha$ values on ISIC-2016 dataset.

Regarding individual model performance with the FBGT method, the Swin
Transformer model with a hyperparameter value of $\alpha=0.80$ achieves the
best performance, with a false negative (FN) count of 25 and a sensitivity of
66.67%. Similarly, when applying the FAGT method, the model with $\alpha=0.75$
demonstrates the best performance, resulting in a FN count of 28 and a
sensitivity of 62.67%. These optimal $\alpha$ values are also applied to the
ViT and ConvNeXt models.

For the ViT model, applying the FAGT method with $\alpha=0.75$ leads to
significant improvements, surpassing all other trained models with a
sensitivity of 72.00% and a FN count of 21. While the utilization of the FBGT
method enhances the performance of the ViT model, the improvement is not as
significant. Similarly, the ConvNeXt model shows performance improvements with
the FBGT and FAGT methods, although the gains are not as substantial as those
observed in the other models. Figure 11 provides a visual representation of
the comparison between the performance of the trained models utilizing the
FAGT and FBGT methods on ISIC-2016 dataset.

#### HAM10000 Dataset

Table 3(b) presents the performance analysis of the models trained on the
HAM10000 dataset. It is apparent that all trained models, irrespective of
their utilization of the FBGT or FAGT methods, surpass the performance of the
No-Filtering approach, which does not incorporate any filtering methods.

The table clearly shows that all trained models, regardless of whether they
used the FBGT or FAGT methods, outperform the No-Filtering approach, which
does not utilize any filtering methods. It is apparent that all trained
models, irrespective of their utilization of the FBGT or FAGT methods, surpass
the performance of the No-Filtering approach, which does not incorporate any
filtering methods.

Figure 12: The graph provides a visual comparison of the performance of the
trained models in terms of recall, F1-score and accuracy, while utilizing the
FAGT and FBGT methods with optimal $\alpha$ values on the HAM10000 dataset.
Table 4: Comparison of our best-performing models against the baselines and
other approaches by researchers on the ISIC-2016 dataset is presented in Table
(a), and the HAM10000 dataset is presented in Table (b).

| Positive Class |  |
---|---|---|---
Method | AUC (%) | Sensitivity (%) | FN
Yu et al. [34] (without seg.) | 78.20 | 42.70 | -
Yu et al. [34] (with seg.) | 78.30 | 54.70 | -
Gutman et al. [31] | 80.40 | 50.70 | -
MelaNet [18] (Baseline) | 81.18 | 70.67 | 22
ViT+FAGT ($\alpha=0.75$) | 82.47 | 72.00 | 21

(a) Dataset: ISIC-2016

|  | Macro Avg.
---|---|---
Method | Accuracy (%) | Recall (%)
Khan et al. [35] | 89.80 | $-$
Onur Sevli [36] | 91.51 | $-$
Chaturvedi et al. [37] | 93.20 | $-$
Afza et al. [38] | 93.40 | $-$
IRv2+SA [19] (Baseline) | 93.48 | 71.93
Swin T.+FBGT ($\alpha=0.80$) | 94.04 | 81.82
ConvNeXt+FAGT ($\alpha=0.85$) | 94.44 | 81.80

(b) Dataset: HAM10000

When it comes to individual model performance, utilizing the FBGT method with
$\alpha=0.80$, the Swin Transformer model achieves the best results, with a
recall of 81.82% and an F1-score of 83.94%. Similarly, when the FAGT method is
employed with an $\alpha$ value of 0.85, the best performance is observed,
resulting in a recall of 82.48% and an F1-score of 81.90%. Similar to the
findings in Table 3(a), the $\alpha$ values that yield the best performance
for the Swin-Transformer model are also applied to the ViT and ConvNeXt
models.

In the case of the ViT model, utilizing the FAGT method with $\alpha=0.85$
achieves the highest recorded recall of 85.94% and the highest average AUC of
98.27%. As for the ConvNeXt model, applying the FAGT method with $\alpha=0.85$
results in an accuracy of 94.44%, an F1-score of 84.06%, and a recall of
81.80%. This is by far the best-performing model when we consider F1-score and
accuracy as our evaluation metrics. The use of the FBGT method does improve
the performance of the ViT and ConvNeXt models as well. However, this
improvement isn’t as significant as using the FAGT method. For a visual
depiction of the performance comparison between the trained models utilizing
the FAGT and FBGT methods on HAM10000 dataset, please refer to Figure 12.

###  5.2 Comparison Against Baselines

In this section, we compare our best-performing models against strong baseline
models. As mentioned before, we choose the baseline model MelaNet[18] for both
the ISIC-2016 dataset and the baseline IRv2+SA [19] HAM10000 datasets.
Furthermore, we compare our models with other strong models proposed by
various researchers. The overall comparisons for the ISIC-2016 and HAM10000
datasets can be visualized in Tables 4(a) and 4(b).

#### ISIC-2016 Dataset

Our best-performing model trained on the ISIC-2016 dataset is the ViT model,
utilizing the FAGT method with $\alpha=0.75$. This model surpasses the
baseline method, MelaNet, by 1.59% in sensitivity and 1.88% in AUC. Our
baseline remained unmatched for pure classification without any segmentation
until the introduction of our proposed method, which outperforms it in every
evaluation category. In Table 4(a), a detailed comparison is presented,
showcasing the performance of our best-performing model against the baseline
and other approaches proposed in different studies on the ISIC-2016 dataset.

#### HAM10000 Dataset

The ConvNeXt model, utilizing the FAGT method with $\alpha=0.85$, is our best-
performing model trained on the HAM10000 dataset. It outperforms the baseline
method, IRv2+SA, by 13.72% in recall and 1.03% in accuracy. Similarly, the
Swin Transformer model, employing the FABT method with an $\alpha=0.80$,
surpasses the baseline IRv2+SA by 13.75% in recall and 0.60% in accuracy.
Furthermore, our model utilizes 42,294 training images, whereas the baseline
IRv2+SA uses 51,699 images, making our approach more sample-efficient. Table
4(b) provides a performance comparison of our best-performing models, against
the baseline and other approaches proposed in different studies on the
HAM10000 dataset.

(a)

(b)

(c)

(d)

Figure 13: The subfigures showcase the four variations of the ISIC-2016
dataset using 2D UMAP visualizations, offering a comparative view of the
dataset’s distribution and clustering patterns for each variation.

(a)

(b)

(c)

(d)

Figure 14: The subfigures present UMAP visualizations of the four different
variations of the HAM10000 dataset, enabling a convenient comparison of the
distribution and clustering patterns across each variation.

###  5.3 Feature Visualization

To visualize the data distribution, we utilize the Uniform Manifold
Approximation and Projection for Dimension Reduction (UMAP) technique [39]. We
analyze four different variations of the ISIC-2016 dataset, which are
presented in the subfigures of Figure 13, and four variations of the HAM10000
dataset, which are shown in the subfigures of Figure 14.

Variation 1, depicted in subfigures 13(a) and 14(a), represents the original
dataset consisting of real images. Variation 2, displayed in subfigures 13(b)
and 14(b), comprises unfiltered oversampled datasets that include real,
transformed, and GAN-generated synthetic images. This variation corresponds to
the No-Filtering experiment. In variations 3 and 4, illustrated in subfigures
13(c), 13(d), 14(c), and 14(d), we have filtered oversampled datasets that
also incorporate real, transformed, and GAN-generated synthetic images. These
variations utilize the FBGT and FAGT methods, and the composition of the
datasets is determined by the hyperparameter $\alpha$. The selection of the
dataset composition is based on the highest performance observed on the Swin
Transformer classifier.

The subfigures in Fig. 13 display the two-dimensional (2D) UMAP embeddings of
the ISIC-2016 dataset. In subfigure 13(a), there are 727 malignant and 173
benign lesions in the distribution. Conversely, subfigures 13(b), 13(c), and
13(d) depict oversampled datasets with 2000 malignant and 2000 benign lesions.
It is noticeable in subfigure 13(b) that the data distribution for both benign
and malignant classes overlaps. However, in subfigures 13(c) and 13(d), the
distribution appears to separate into two distinct groups, indicating the
effectiveness of the FBGT and FAGT methods compared to the No-Filtering
approach.

Similarly, the subfigures in Fig. 14 present the three-dimensional (3D) UMAP
embeddings of the HAM10000 dataset. In subfigure 14(a), the distribution
consists of 9186 skin lesions. Conversely, subfigures 14(b), 14(c), and 14(d)
depict oversampled datasets, each showing a portion of the 42,294 skin
lesions. When observing the 3D representations, it becomes apparent that in
subfigures 14(c) and 14(d), the data points for each class become more
distinguishable in a three-dimensional space. In contrast, the unfiltered 3D
representation in subfigure 14(b) exhibits sparse data points, making it
challenging for the classifier to identify specific regions for each class. As
a result, the classification task becomes more difficult.

The formation of clustered regions, as opposed to a single concentrated area,
further substantiates the effectiveness of the FBGT and FAGT methods,
addressing the issue of low inter-class variation within a dataset. By
utilizing these techniques, we can substantially refine the classification
process, ultimately leading to more accurate and dependable outcomes.

##  6 Conclusion

This paper introduces Cosine Similarity-based Image Filtering (CosSIF), a
robust dataset filtering algorithm. We utilize CosSIF to create two filtering
approaches: FBGT and FAGT. These methods rely on cosine similarity as the main
metric for similarity calculation and aim to reduce the volume of GAN-
generated synthetic images from the minority class that resemble similarity to
images from the majority class. Our experimental results demonstrate that
models trained on datasets processed with either the FBGT or FAGT methods show
improved performance compared to models without these filtering methods.
Through comprehensive experiments, we demonstrate that the proposed FAGT
method, when applied to the ISIC-2016 dataset and trained with ViT model using
a tuned $\alpha$ value of 0.75, improves sensitivity by 1.59% and AUC by 1.88%
compared to the baseline MelaNet. When applying the FAGT and FBGT methods to
the HAM10000 dataset, trained with ConvNeXt and Swin Transformer models,
respectively, using tuned $\alpha$ values of 0.80 and 0.85, we observe
significant improvements in recall. Specifically, the FAGT method achieves a
recall improvement of 13.72% over the baseline IRv2+SA, with an accuracy of
94.44%, while the FBGT method achieves a recall improvement of 13.75% over the
same baseline, with an accuracy of 94.04%. For future research, our aim is to
enhance the similarity calculation algorithm by incorporating a feature
extraction and feature-based similarity calculation module. Additionally, we
aim to apply our algorithm and filtering methods to various medical domains,
including X-rays, CT scans, and MRI images. Furthermore, we plan to utilize
the proposed CosSIF algorithm to develop a downsampling technique suitable for
all image classification tasks.

## Compliance with Ethical Standards

#### Funding Statement

The authors did not receive any external financial support for this research.
The study was entirely self-financed, with all related costs being borne
exclusively by the authors of the paper.

## References

  * [1] P. Bir and V. E. Balas, “A review on medical image analysis with convolutional neural networks,” in 2020 IEEE International Conference on Computing, Power and Communication Technologies (GUCON), pp. 870–876, IEEE, 2020.
  * [2] A. A. Adegun, S. Viriri, and R. O. Ogundokun, “Deep learning approach for medical image analysis,” Computational Intelligence and Neuroscience, vol. 2021, 2021.
  * [3] M. I. Razzak, S. Naz, and A. Zaib, “Deep learning for medical image processing: Overview, challenges and the future,” Classification in BioApps, pp. 323–350, 2018.
  * [4] M. P. McBee, O. A. Awan, A. T. Colucci, C. W. Ghobadi, N. Kadom, A. P. Kansagra, S. Tridandapani, and W. F. Auffermann, “Deep learning in radiology,” Academic radiology, vol. 25, no. 11, pp. 1472–1480, 2018.
  * [5] A. R. Lopez, X. Giro-i Nieto, J. Burdick, and O. Marques, “Skin lesion classification from dermoscopic images using deep learning techniques,” in 2017 13th IASTED international conference on biomedical engineering (BioMed), pp. 49–54, IEEE, 2017.
  * [6] Z. Wang, P. A. Keane, M. Chiang, C. Y. Cheung, T. Y. Wong, and D. S. W. Ting, “Artificial intelligence and deep learning in ophthalmology,” in Artificial Intelligence in Medicine, pp. 1519–1552, Springer, 2022.
  * [7] S. M. McKinney, M. Sieniek, V. Godbole, J. Godwin, N. Antropova, H. Ashrafian, T. Back, M. Chesus, G. S. Corrado, A. Darzi, et al., “International evaluation of an ai system for breast cancer screening,” Nature, vol. 577, no. 7788, pp. 89–94, 2020.
  * [8] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun, “Dermatologist-level classification of skin cancer with deep neural networks,” nature, vol. 542, no. 7639, pp. 115–118, 2017.
  * [9] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, et al., “Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs,” Jama, vol. 316, no. 22, pp. 2402–2410, 2016.
  * [10] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” Medical image analysis, vol. 42, pp. 60–88, 2017.
  * [11] T. Huynh, A. Nibali, and Z. He, “Semi-supervised learning for medical image classification using imbalanced training data,” Computer Methods and Programs in Biomedicine, p. 106628, 2022.
  * [12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [13] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann, “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study,” PLoS medicine, vol. 15, no. 11, p. e1002683, 2018.
  * [14] V. Sampath, I. Maurtua, J. J. Aguilar Martin, and A. Gutierrez, “A survey on generative adversarial networks for imbalance problems in computer vision tasks,” Journal of big Data, vol. 8, pp. 1–59, 2021.
  * [15] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196, 2017.
  * [16] S. Kazeminia, C. Baur, A. Kuijper, B. van Ginneken, N. Navab, S. Albarqouni, and A. Mukhopadhyay, “Gans for medical image analysis,” Artificial Intelligence in Medicine, vol. 109, p. 101938, 2020.
  * [17] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” IEEE signal processing magazine, vol. 35, no. 1, pp. 53–65, 2018.
  * [18] H. Zunair and A. B. Hamza, “Melanoma detection using adversarial training and deep transfer learning,” Physics in Medicine & Biology, vol. 65, no. 13, p. 135005, 2020.
  * [19] S. K. Datta, M. A. Shaikh, S. N. Srihari, and M. Gao, “Soft attention improves skin cancer classification performance,” in Interpretability of Machine Intelligence in Medical Image Computing, and Topological Data Analysis and Its Applications for Medical Data, pp. 13–23, Springer, 2021.
  * [20] F. Garcea, A. Serra, F. Lamberti, and L. Morra, “Data augmentation for medical imaging: A systematic literature review,” Computers in Biology and Medicine, p. 106391, 2022.
  * [21] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
  * [22] Z. Lan, S. Cai, X. He, and X. Wen, “Fixcaps: An improved capsules network for diagnosis of skin cancer,” IEEE Access, vol. 10, pp. 76261–76267, 2022\.
  * [23] A. A. Ilham, I. Nurtanio, et al., “Image search optimization with web scraping, text processing and cosine similarity algorithms,” in 2020 IEEE International Conference on Communication, Networks and Satellite (Comnetsat), pp. 346–350, IEEE, 2020.
  * [24] S. Kaur and D. Aggarwal, “Image content based retrieval system using cosine similarity for skin disease images,” Adv. Comput. Sci. Int. J, vol. 2, pp. 89–95, 2013.
  * [25] Z. Tao, H. Liu, H. Fu, and Y. Fu, “Image cosegmentation via saliency-guided constrained clustering with cosine similarity,” in Proceedings of the AAAI conference on artificial intelligence, vol. 31, 2017.
  * [26] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, “Training generative adversarial networks with limited data,” Advances in neural information processing systems, vol. 33, pp. 12104–12114, 2020.
  * [27] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110–8119, 2020.
  * [28] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021.
  * [29] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  * [30] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986, 2022.
  * [31] D. Gutman, N. C. Codella, E. Celebi, B. Helba, M. Marchetti, N. Mishra, and A. Halpern, “Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (isbi) 2016, hosted by the international skin imaging collaboration (isic),” arXiv preprint arXiv:1605.01397, 2016.
  * [32] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” Scientific data, vol. 5, no. 1, pp. 1–9, 2018.
  * [33] “sklearn.model_selection.train_test_split — scikit-learn 1.2.2 documentation.” https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html.  (Accessed on 03/26/2023).
  * [34] L. Yu, H. Chen, Q. Dou, J. Qin, and P.-A. Heng, “Automated melanoma recognition in dermoscopy images via very deep residual networks,” IEEE transactions on medical imaging, vol. 36, no. 4, pp. 994–1004, 2016.
  * [35] M. A. Khan, M. Y. Javed, M. Sharif, T. Saba, and A. Rehman, “Multi-model deep neural network based features extraction and optimal selection approach for skin lesion classification,” in 2019 international conference on computer and information sciences (ICCIS), pp. 1–7, IEEE, 2019.
  * [36] O. Sevli, “A deep convolutional neural network-based pigmented skin lesion classification application and experts evaluation,” Neural Computing and Applications, vol. 33, no. 18, pp. 12039–12050, 2021.
  * [37] S. S. Chaturvedi, J. V. Tembhurne, and T. Diwan, “A multi-class skin cancer classification using deep convolutional neural networks,” Multimedia Tools and Applications, vol. 79, no. 39-40, pp. 28477–28498, 2020.
  * [38] F. Afza, M. Sharif, M. A. Khan, U. Tariq, H.-S. Yong, and J. Cha, “Multiclass skin lesion classification using hybrid deep features selection and extreme learning machine,” Sensors, vol. 22, no. 3, p. 799, 2022.
  * [39] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018.