# Probabilistic Diagnostic Tests for Degradation Problems in Supervised
Learning

Gustavo A. Valencia-Zapata  Network for Computational Nanotechnology, Purdue
University, West Lafayette IN, US  School of Electrical and Computer
Engineering, Purdue University, West Lafayette IN, US  Corresponding author:
Gustavo A. Valencia-Zapata<EMAIL_ADDRESS>Carolina Gonzalez-Canas
School of Industrial Engineering, Purdue University, West Lafayette IN, US
Michael G. Zentner  HUBzero Research Group, Information Technology at Purdue,
West Lafayette IN, US  Okan Ersoy  School of Electrical and Computer
Engineering, Purdue University, West Lafayette IN, US  Gerhard Klimeck
Network for Computational Nanotechnology, Purdue University, West Lafayette
IN, US  School of Electrical and Computer Engineering, Purdue University, West
Lafayette IN, US

###### Abstract

Several studies point out different causes of performance degradation in
supervised machine learning. Problems such as class imbalance, overlapping,
small-disjuncts, noisy labels, and sparseness limit accuracy in classification
algorithms. Even though a number of approaches either in the form of a
methodology or an algorithm try to minimize performance degradation, they have
been isolated efforts with limited scope. Most of these approaches focus on
remediation of one among many problems, with experimental results coming from
few datasets and classification algorithms, insufficient measures of
prediction power, and lack of statistical validation for testing the real
benefit of the proposed approach. This paper consists of two main parts: In
the first part, a novel probabilistic diagnostic model based on identifying
signs and symptoms of each problem is presented. Thereby, early and correct
diagnosis of these problems is to be achieved in order to select not only the
most convenient remediation treatment but also unbiased performance metrics.
Secondly, the behavior and performance of several supervised algorithms are
studied when training sets have such problems. Therefore, prediction of
success for treatments can be estimated across classifiers.

_K_ eywords Class imbalance $\cdot$ Overlapping $\cdot$ Small-disjuncts
$\cdot$ Noisy labels $\cdot$ Sparseness $\cdot$ Imbalanced data $\cdot$
Gaussian Mixture Models $\cdot$ Separation index

##  1 Introduction

A diagnosis is an investigation or analysis of causes or nature of a
condition, situation, or problem. For example, in Health Care, the diagnostic
process gathers information related to a patient health problem in order to
plan the best path or treatment. Recent retrospective studies of adult
healthcare and post-mortem examinations show a diagnostic error leading to up
to 17% of hospital adverse events and approximately 10% of patient deaths [1].
Therefore, several efforts in Health Care focus on improving diagnosis such as
reducing uncertainty, early diagnosis, and second opinion [2], [3]. Using the
analogy of "Medical Diagnosis", it is plausible to argue that the concept of
"Diagnostic" as human cognitive function can be incorporated in the Artificial
Intelligence domain. This research investigates whether the implementation of
diagnostic tests aids to characterize the most common problems of
classification tasks in order to select the most convenient treatment for
avoiding classification degradation. Thereby, the questions now arise:

  * •

What are the most representative problems of classification? Does exist any
correlation, causality or association between them?

  * •

Is it possible to measure a direct impact of each problem to the classifier
degradation?

  * •

Is there any process or technique to diagnose training datasets and subsequent
remediation (treatments) selection before the training stage

Mainly there are 5 degradation problems related to training datasets that
limit the learning process: class imbalance, sparseness, small-disjuncts,
overlapping, and noise labels. The class imbalance is one of the most
recurrent problems and has received big interest in the artificial
intelligence field in recent years [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. A
dataset is imbalanced when its class distribution or classification categories
do not have similar proportions in terms of the number of instances. This is a
problem because classification algorithms are not designed to handle
unbalanced datasets and tend to favor the major class over minor ones [14]. A
second important issue related to class imbalance is the performance
measurement. The most used performance metric for classification algorithms is
the area under the ROC curve (AUC) [15], which may provide an overly
optimistic performance evaluation and wrong performance evaluation for
unbalance cases [16]. Several questions have arisen about the class imbalance
problem; for its proper functioning: what is the optimal class distribution
for training a classification algorithm? What are treatments for handling
class imbalance? Is the class imbalance only responsible for classifier
degradation?

Several approaches [17, 18, 19], have tackled these questions applying
resampling (oversampling and undersampling) to reach the balance between
classes modifying the distribution of the dataset. More sophisticated
techniques change that distribution by generating synthetic data as a balance
strategy, not only for reducing overfitting and improving performance in
classification tasks, but also for handling the sparseness in the feature
space [13]. On the other hand, cost-sensitive learning treat the class
imbalance problem arguing that each type of error has its own costs [20, 21].
Therefore, instead of changing the distribution of the whole dataset, cost-
sensitive based models assume that misclassification cost for the observed
dataset is known and use a "cost matrix" for describing it. Different
approaches suggest that class imbalance is not completely responsible for
degradation and argue that the phenomena of small-disjuncts contributes to
overfitting and misclassification [12, 22, 23, 24, 25, 26]. Small-disjuncts
are subclasses of few instances inside the minor class. Other studies state
that the "overlapping" regions between classes [9, 27] and "noisy labels" due
mislabeling [28] have strong negative effect over the classification algorithm
performance. Different approaches based on ensemble methods [28, 6, 11, 29,
30, 31] combine multiple learning algorithms and strategies for handling class
imbalance, noisy labels detection [28], and improve performance.

This research presents a comprehensive empirical study of degradation problems
in supervised learning trough datasets from real-world domains. The paper is
organized as follows: Section 2 describes the importance of diagnostic methods
in machine learning, research aims and approach. Section 3 describes the five
major degradation problems and remediation techniques for handling degradation
in supervised learning algorithms. Section 4 presents the implementation of a
probabilistic diagnostic model for classifier degradation, which supports the
framework proposed in this research. Section 5 presents experimental results
for several datasets across classification algorithms and techniques for
handling degradation problems. Finally, Section 6 presents general conclusions
and recommendations for future work.

##  2 Lack of diagnosis in supervised learning

The motivation of the present work is to reduce degradation of classification
algorithms caused by inherent problems associated with datasets. Although
there exist methodologies or techniques that try to mitigate degradation, most
of those efforts are blind to the characteristics of the datasets under study.

The diagnostic process in Health Care has several components [1] such as
stages related to information gathering, diagnostic testing, clinical history,
etc. This research uses similar components applied to the artificial
intelligence domain, specifically associated with supervised learning. In
other words, the process starts with the "dataset" or target, which
experiences a problem. After the dataset engages with the process, information
is gathered from several sources (characteristics, testing, records, best
practices, and so on) for building a diagnosis. Consecutively, an explanation
of the problems is built. Finally, a treatment is defined based on the
diagnosis, and its outcomes are available as inputs to the diagnostic process.
Two fundamental principles are adopted from Health Care:

  1. 1.

Correct diagnosis avoids inappropriate treatments [1]. Even though a number of
approaches (treatments) either in the form of a methodology or a technique try
to minimize performance degradation in supervised learning, they have been
isolated efforts without the input from a diagnostic process.

  2. 2.

Treatment procedures according to the needs and characteristics of individual
patients [32, 33]. Treatment effectiveness for classifier degradation does not
exclusively depend on the degradation problem (class imbalance, sparseness,
small-disjuncts, overlapping, and noise labels), but also on characteristics
associated with the training set relevant for treatment selection and reaching
optimal results.

##  3 Degradation problems for supervised learning

Data complexity can be associated with issues such as class imbalance,
overlapping, sparseness, and others [34]. This section presents five
degradation problems for supervised learning and their most well-known
remediation techniques. The damage caused by those degradation problems can
range from hampering the training stage to dramatically reduce classification
algorithm performance.

###  3.1 Class imbalance and Sparseness

Imbalanced data is present in many fields and represents a continuous
challenge for Data Mining and Artificial Intelligence tasks. For instance, in
medical diagnosis, some diseases are infrequent or hard to detect because of
limited number of cases. Fraud detection is a well-known case of class
imbalance, in which the number of fraud transactions is much smaller than the
legitimate ones. As previously mentioned, part of misclassification in
supervised learning tasks may be attributed to the effect of class imbalance
and the fact that most of the classifiers are not designed to address this
effect. Class imbalance occurs when one or more classes from the training set
have few instances as compared to other classes. Usually, the level of class
imbalance is measured using the Imbalance Ratio (IR), which indicates the
relationship between two classes that expresses how much bigger one is than
the other. For example, in a binary classification task (dataset with two
classes), the class with few observations or minority class is called the
"Positive class". On the other hand, the majority class is called the
"Negative class" and the IR is estimated using the ratio between the total
numbers of observations for each class. Generally, the class imbalance problem
has been addressed with two main approaches: methods based on resampling
(oversampling and undersampling) and synthetic data techniques, and cost-
sensitive methods. In the case of the latter approach, it argues that errors
do not have the same cost. The present research focuses on resampling and
synthetic data techniques, because cost-sensitive techniques assume the prior
knowledge about the cost of misclassification for each class, which is a
condition unknown in most cases and defined by the nature and domain of the
dataset. The most representative resampling methods are oversampling and
undersampling [35, 36]. These two techniques seek to reach the balance between
classes modifying the distribution of the dataset. The first replicates
randomly some instances from the Positive class. The main disadvantage of
oversampling is overfitting [13, 36] and the low performance for unseen
testing instances at the training stage [22]. In contrast, undersampling
subtracts a random sample of a specified size from the observations of the
Negative class, leading losing important information and limiting the learning
process for such as class. C. Drummond et al. [37] tested oversampling and
undersampling techniques over class imbalance datasets using the decision tree
learner C4.5, showing better performance for classification tasks when
undersampling technique was selected. However, more modern strategies such as
Synthetic Minority Oversampling Technique (SMOTE) [13], Borderline-SMOTE [14],
Adaptative Synthetic Sampling Approach for Imbalanced Learning (ADASYN) [38],
Density-base SMOTE (DBSMOTE) [39], and BalanceCascade and EasyEnsemble [11]
combine oversampling, undersampling, and algorithms in order to produce
synthetic entries, reach balance, avoid overfitting, and obtain better
performance.

Several studies argue that the imbalance issue in data should be understood as
the level of sparseness and the low degree of instances for some classes [13,
14, 40]. Therefore, synthetic data techniques such as SMOTE [13] and its
extensions not only generate new synthetic entries for balancing, but also
builds larger and better defined decision regions for the Positive class by
filling holes in the space.

###  3.2 Small-disjuncts

Holte et al. [22] introduced the term of small-disjuncts, which would be the
major cause of learning problems in the Positive class [22, 41]. The small-
disjuncts could be interpreted as a direct consequence of the "Sparseness". In
simple words, small-disjuncts are "sub-clusters" or subclasses of few
instances inside the Positive class. Generally, the small-disjunct concept can
be generalized as the imbalance problem between-class and within-class, the
between-class case refering to the imbalance between the Negative class and
the Positive class, which is usually estimated by the IR. On the other hand,
the within-class refers to how observations are distributed within each class,
since a class could be composed of several subclasses with imbalance in the
number of instances [23, 24, 25, 26]. This research extends the concept of
small-disjuncts to the Negative class. Therefore, using the new definition of
small-disjuncts: both classes can have subclasses of few instances that are
difficult to handle during the training stage for classification algorithms.

###  3.3 Overlapping and noisy labels

Datasets can have ambiguous regions, which contain observations from two or
more classes with similar probability. This problem is known as overlapping.
Diverse studies claim that classifier degradation is not only affected by the
class imbalance problem or small-disjuncts, but perhaps even more so, in terms
of the degree of overlapping [9, 27, 42]. Garcia et al. [9] conducted several
experiments over two-dimensional artificial datasets with two classes
separated by a line orthogonal to one of the axis and six different
classifiers. They found that changing the IR of the overlapping region is more
beneficial than changing the overall IR in the dataset. Although the
simplicity of the data is far from real datasets, the results point out the
degree of overlap as an important factor in the process of building
classifiers. Most of the treatments for handling Noisy Labels focus on
cleaning the overlap regions by using the k-Nearest Neighbor Rule as a
component in their procedure. Labeling error is a recurrent problem in
supervised learning datasets. This problem describes instances with wrong
label assignment due to different causes (i.e. data-entry error, subjectivity,
and lack of information for the labeling process). Quinlan [43, 44, 45]
conducted several experiments in order to measure the effect of class noise
over decision tree performance, finding two patterns: class noise is more
harmful than feature (characteristics) noise, and cleaning noisy observations
from training sets produces classifiers with better accuracy. Condensed
Nearest Neighbors algorithm (CNN) [46] was the first attempt of filtering
noisy observations or building a better subset from the training set using
Nearest Neighbor Rule (NN Rule) [47, 48]. Several subsequent proposals based
on CNN concepts have been discussed, such as Edited Nearest Neighbors (ENN)
[49], Neighborhood Cleaning Rule NCL [50], Tomek [51], Mutual Nearest
Neighborhood (MNN) [52], and One-Side Selection (OSS) [36]. For instance,
Tomek algorithm extends that approach calling ENN for increasing the number of
neighbors and focus on removing noisy observations from the Negative class
close to the borderline with the Positive class and returning a better subset
for the training stage. On the other hand, Brodley et al. [53] used Ensemble
filters for detecting noisy observations. In this approach, the filter is
composed of a set of classifiers called base-level detectors, which receive
the training set and detect the misclassified cases. Finally, the filter
removes observations based on majority voting or consensus and generate a
training subset.

###  3.4 Treatments and side effects

As previously stated, there are numerous treatments in the form of methodology
or technique that seeks to mitigate negative effects of the degradation
problems. However, the effectiveness of such treatments may be reduced by lack
of diagnosis. The following is the list of techniques implemented in Section 5
related to experimental framework.

  1. 1)

Resampling (Random). This technique combines oversampling and undersampling to
obtain a balanced dataset. Naive oversampling replicates randomly some
instances from the Positive class (minority). On the other hand, naive
undersampling removes randomly some instances from the Negative class
(majority).

  2. 2)

Synthetic Minority Oversampling Technique (SMOTE) [13]. This algorithm
generates synthetic instances of the Positive class (minority) based on the
feature space similarities, using the k-Nearest Neighbor algorithm. Besides,
the Negative class (majority) instances are under-sampled in order to reach
the balance between classes.

  3. 3)

Borderline-SMOTE [14]. This technique is an extension of SMOTE, which focuses
on generating synthetic Positive instances in the neighborhood of the borders.

  4. 4)

Adaptative Synthetic Sampling Approach for Imbalanced Learning (ADASYN)[38].
This algorithm is an extension of SMOTE, which generates synthetic instances
using a density distribution based on k-Nearest Neighbors algorithm. Thus,
this technique automatically decides the number of synthetic instances that
must be generated for each real Positive instance.

  5. 5)

Density-base SMOTE (DBSMOTE) [39]. This technique initially clusters the
Positive class using DBSCAN algorithm [54] and generates synthetic instances
along a shortest path from each Positive instance to a pseudo-centroid of the
cluster.

  6. 6)

Edited Nearest Neighbors (ENN) [49]. It removes any instance whose class label
differs from the class of at least half of its k-Nearest Neighbors. Therefore,
both classes may be under-sampled.

  7. 7)

Neighborhood Cleaning Rule (NCL) [50]. This algorithm is a modification of
ENN, which only excludes instances from the Negative class (majority) in two
steps: first, it removes Negatives instances which are misclassified by their
3-Nearest Neighbors. Then, the neighbors of each Positive instance are found
and the ones belonging to the Negative class are removed.

  8. 8)

One-Side Selection (OSS) [36]. This technique combines Tomek links [51] and
CNN [46]. First, Tomek links focus on cleaning overlapping between classes by
removing Negative instances using 1-Nearest Neighbor. Then, CNN removes
instances from both classes that are not correctly classified by the 1-Nearest
Neighbor Rule.

This research categorizes the above treatments based on the degradation
problems in order to remedy and possible side effects. Table 1 summarizes such
categorization. The symbol (+) refers to the problem that the treatment
intends to mitigate. Contrarily, the symbol ($-$) refers to negative side
effects due to the treatment. For instance, Random combines undersampling and
oversampling to obtain a balanced dataset; however, the first one removes
instances that may be important and the second one produces overfitting. A
second very concrete example is OSS, which solves the problems of overlapping
and noisy labels by removing instances. However, it reduces the amount of
data, produces holes in the space, and may increase the class imbalance. On
the contrary, SMOTE and its extensions not only regenerate synthetic instances
for reaching balance between classes, but also fill the space of the Positive
class. Therefore, those techniques reduce the sparseness effect and build
better decision regions.

Table 1: Treatments: Theoretical benefits and side effects. The symbol (+) refers to the problem that the treatment intends to remediate. Contrarily, the symbol ($-$) refers to negative side effects due to the treatment.  | Class | Sparseness | Small | Overlapping | Noisy | Amount | Overfitting
---|---|---|---|---|---|---|---
Treatment | Imbalance |  | disjuncts |  | labels | of data |
Random | (+) |  |  |  |  | ($-$) | ($-$)
SMOTE | (+) | (+) |  | ($-$) | ($-$) | ($-$) |
B-SMOTE | (+) |  |  |  |  | ($-$) |
DBSMOTE | (+) | (+) | (+) |  |  | ($-$) |
ADASYN | (+) | (+) |  |  |  | ($-$) |
ENN | ($-$) | ($-$) |  | (+) | (+) | ($-$) |
NCL | (+) | ($-$) |  | (+) | (+) | ($-$) |
OSS | ($-$) | ($-$) |  | (+) | (+) | ($-$) |

##  4 Probabilistic diagnostic tests

After the main degradation problems and their remediation treatments have been
mentioned, this section presents a model called "Probabilistic Diagnostic
Model for Handling Classifier Degradation", which allows to determine the
level of degradation problems present in classification datasets. References
of previous dataset diagnosis efforts point to basic description of the
dataset such as missing values, duplicate instances, and feature statistics
such as frequency, percentage, mean, and standard deviation. Nevertheless,
there is no explicitly reference related to tools, techniques, methodologies
or any other approach for diagnosis of degradation problems on supervised
learning.

###  4.1 Diagnostic components

The diagnostic model consists of five main components, which have a direct
connection with the five degradation problems discussed in Section 3. These
components are connected through a workflow where each component provides
information to the next one during the diagnostic process. Figure 1
illustrates the sequence of the diagnostic model, where dataset (training set)
is the input to the flow and the diagnostic report is the output. Subclasses
detection is the first component, which is conducted for each class. Then, the
process runs several "diagnostic tests" that measure the criticality levels of
the degradation problems. Finally, the output of the process is a diagnostic
report.

  1. 1.

Subclasses detection: Previous proposals have formulated strategies of
segmentation for the training set [4, 24, 25, 26, 23, 55] by using algorithms
such as Principal Direction Devise Partition (PDDP) [56] and K-means [57, 58,
54]. Those strategies focus on finding subclasses (clusters) for the positive
class or even for the complete training set regardless of the membership
class. Nevertheless, the major drawbacks of those approaches are the lack of
parameters associated with the subclasses description and need to provide the
number of subclasses (clusters) in the training set a priori. The first
component of the diagnostic model runs a segmentation process of partitioning
each class into subclasses, implementing Gaussian Mixture Models (GMM) [59,
60] and the EM algorithm [61], which is used to estimate the mixture
parameters. The advantage of this approach is to obtain statistical parameters
that describe each subclass for subsequent analysis. However, there is an
inevitable, indeed necessary, question at the time of modeling a mixture of
probabilistic distributions or any segmentation technique: how many clusters
(subclasses) should be included in the model? A second question is related to
the covariance matrix model selection for the GMM. For this research, the
total number of subclasses per class and the covariance matrix model are
determined by seeking to balance the increase in likelihood and the complexity
of the mixture model, introducing a penalty term for each parameter. Then, the
selection of the best model is addressed by the Bayesian Information Criterion
(BIC) [62, 63], which is described with more details in Appendix A. Once the
number of subclasses per class is defined, it is possible not only moving to
the next component of the diagnostic tool, but also knowing more detail about
the complexity of the dataset in terms of small-disjuncts. Greater number of
subclasses is associated with higher difficulty levels of classification [40,
26]. Moreover, it should be taken into consideration that not all detected
subclasses are small-disjuncts; such category usually describes subclasses
with few instances.

  2. 2.

Imbalance ratio (IR): This component not only estimates the IR between
classes, but also between subclasses. This information enables to focus any
resampling or synthetic data process over specific subclasses in future
remediation treatments. The IR is estimated as many times as the combination
of detected subclasses taken 2 at a time without repetition.

  3. 3.

Degree of overlap: the third component measures the degree of overlap based on
the separation index (J*) proposed by Qiu et al. [64, 65]. This index measures
the magnitude of the gap between pairs of subclasses. It has a value between
$-$1 and +1, where negative values indicate subclasses overlap, zero means
subclasses are touching, and positive values indicate subclasses are
separated. Experiments based on Multivariate Gaussian distributions (MGD) [54]
suggest that values larger than 0,21 indicate subclasses are well-separated.
The separation index is estimated as many times as the combination of
subclasses taken 2 at a time without repetition. Further information about
estimation process of the separation index estimation can be found in Appendix
A. One recurrent challenge of classification algorithms is to build a
mathematical function sufficiently flexible and able to discriminate instances
even with high levels of overlapping between classes. Therefore, some
techniques focus on solving overlapping issues before the training stage, for
instance, generating Positive instances in the overlap regions by reinforcing
borderlines [14] or removing noisy instances from the Negative class [50, 51,
36]. The feasibility, advisability, and parametrization of those techniques
could be elements easy influenced by the separation index.

  4. 4.

Noise level: this component detects noisy instances using a 3-Nearest
Neighbors Rule and estimates the noise ratio for the training dataset, the
Positive class, and the Negative class. Since separation indices between
subclasses are known, this component is able to identity noisy instances
between well-separated subclasses. In other words, this component detects
Noisy Labels unrelated to the overlapping issue, and estimates a ratio related
to the noise due overlap issues.

  5. 5.

Dispersion test: the fifth component implements two procedures: the first one
estimates the degree of sparseness for each subclass by calculating the
average and standard deviation of the distances (Euclidian, Mahalanobis, and
Manhattan) to the subclass median. The second procedure implements the
Anderson’s test [66, 67, 68] (PERMDISP2) for multivariate homogeneity of
groups dispersion, which is a multivariate analogue of Levene’s test for
homogeneity of variances [69]. Anderson’s test seeks to validate if the
dispersion of two or more subclasses is equal (similar sparseness).

Figure 1: Diagnostic model process. The training set is the input to the flow
and the diagnostic report is the output.

###  4.2 Case in point: Artificial Domain

This subsection presents a systematic example of the procedures and outputs of
the diagnostic model. The purpose of this example is to provide a full
understanding of the diagnostic model components and outputs. The case in
point considers a two-dimensional artificial dataset for classification, which
has been generated on the basis of Gaussian distributions and three controlled
parameters: size of the subclasses, degree of overlapping, and distributional
parameters (mean vector and covariance matrix).

  1. 1.

Output 1: Training set basic description. The first output is related to the
basic characteristic of the dataset under analysis. In the supervised learning
case, such dataset is the training set. Table 3 shows the first output, which
is a list of basic characteristics and values for the artificial training set.
The Negative class with 404 instances represents 82.4% of the data. The
Positive class with 86 instances represents 17.6% of the data. Therefore, this
training set is said to have an Imbalance Ratio of 4.7:1 or IR= 4.7. In other
words, there is one instance in the Positive class per 4.7 instances in the
Negative class.

Table 2: Output 1: Basic description for the artificial training set. Dimensions | 2
---|---
Duplicate Instances | NA
Total instances | 490
Instances Positive class | 86
Instances Negative class | 404
IR | 4.7

Table 3: Output 2: Subclass detection for the artificial training set.  Positive class |
---|---
Instances | 86
Covariance model | VEE
Num. Subclasses | 2
Negative class |
Instances | 404
Covariance model | VEI
Num. Subclasses | 3

Figure 2: BIC plots for covariance matrix selection fitted to the artificial
training set. a) The top three best models for the Positive class are: VEE
(ellipsoidal, equal shape and orientation), VEI (diagonal, varying volume,
equal shape), and VEV (ellipsoidal, equal shape). There are two subclasses for
the Positive class. b) The top three best models for the Negative class are:
VEI (diagonal, varying volume, equal shape), VEV (ellipsoidal, equal shape),
and VII (spherical, unequal volume). There are three subclasses for the
Negative class

  2. 2.

Output 2: subclasses detection. Table 3 shows the next output related to the
fist component of the diagnostic tool, which uses GMM for sub-subclass
detection. For the Positive class, two subclasses were detected using the
covariance matrix model VEE (ellipsoidal, equal shape and orientation). On the
other hand, the Negative class has three subclasses detected by the covariance
matrix model VEI (diagonal, varying volume, equal shape). Figure 2 illustrates
the best number of subclasses and top three covariance matrix models based on
the BIC for the artificial dataset. In Figure 2 (a), all covariance matrix
models are in agreement with two subclasses for the Positive class, where the
best covariance model is VEE. In Figure 2 (b), two out of three covariance
matrix models are in agreement with three subclasses for the Negative class,
and VEI is the best covariance model. Figure 3 makes a visual representation
of the segmentation results through the GMM over the artificial dataset. The
overall result is five subclasses, two for the Positive class and three for
the Negative class. Further information about covariance matrix models
implemented in GMM are given in Appendix A.

  3. 3.

Output 3: IR and Overlap matrix (IRO). Figure 5 shows the third output related
to the IR and Overlap (IRO) matrix for the artificial training set. This is a
square matrix, whose size is determined by the number of subclasses. The upper
triangular part shows the IR between subclasses. For instance, the IR between
N-01 (Negative subclass 01) and N-03 (Negative subclass 03) is 3. That means
there is one instance in N-03 per 3 instances in N-01. The diagonal of the
matrix contains the number of instances per subclass. For example, P-01
(Positive subclass 01) is the smaller subclass with 36 instances in the
artificial training set. Finally, the lower triangular part of the matrix
shows the separation index J* between subclasses. For instance, J* = 0.29
between N-02 and P-02, which makes sense since both subclasses are far from
each other according to Figure 3.

Figure 3: Detecting subclasses by using GMM. a) The original training set. b)
Training set with five subclasses.
N = Negative subclasses and P = Positive subclasses.

Figure 4: Output 3: IRO matrix for the artificial training set. The upper
triangular part shows the IR between subclasses, the diagonal contains the
number of instances per subclass, and the lower triangular part shows
separation indexes between subclasses.

Figure 5: Output 4: Noise matrix for the artificial training set. This matrix
shows the amount of noise between subclasses.

  4. 4.

Output 4: Noise level. The noise component has several tables as outputs.
Table 5 summarizes the number of noisy and valid instances per class for the
artificial training set. Additionally, Table 5 breakdowns the number of noisy
instances per subclass and includes the noise location (last two columns). The
column called "Noise overlap" refers to noise between subclasses with a
separation index less than 0.2 (close subclasses). Then, this noise could be
due to the overlap effect instead of mislabeled issues. On the other hand, the
column called "Noise Label" refers to noise between subclasses with a
separation index larger or equal to 0.2 (well-separated subclasses).
Therefore, noisy instances are likely associated with mislabeled issues. For
the artificial training set, all noisy instances are associated with the
overlap effect. Under the noise location concept, the diagnostic tool proposes
three new metrics, which describes noise proportions for training sets. The
first metric is the Noise Ratio (NR), which is the quantitative relation
between valid and all noisy instances. For instance, the artificial training
set has a NR = 17.14. That means there is one noise instance per 17.14 valid
instances. The second metric is the Noise Overlap Ratio (NOR), that considers
noisy instances associated with close subclasses. Finally, the Noise Label
Ratio (NLR) considers noise related to well-separated subclasses. In the
example, all noisy instances are associated with close subclasses. Therefore,
NR = NOR, and NLR does not apply. Figure 5 shows the Noise matrix. This is a
square matrix, whose size is determined by the number of subclasses. Each
column of the matrix represents the instances in an estimated subclass based
on the 3-Nearest Neighbors Rule while each row represents the instances in an
actual subclass (original membership). Thus, the diagonal of the matrix
contains the number of valid instances per subclass (instances that kept the
same subclass label after the noise analysis). For instance, reading the first
row: 176 instances from N-01 did not change their membership, one instance
moved to N-03 (it is not considered noise because N-01 and N-03 belong to the
same Negative class), 10 noisy instances from P-01, and 2 noisy instances from
P-02.

Table 4: Output 4: Noisy and valid instances per class for the artificial training set. Class | Noise | Valid
---|---|---
Negative | 15 | 389
Positive | 12 | 74

Table 5: Output 4: The number of noisy instances per subclass including the noise location for the artificial training set.  | Instances | Valid | Noise | Noise | Noise
---|---|---|---|---|---
Subclass |  |  |  | overlap | label
N-01 | 189 | 177 | 12 | 12 | 0
N-02 | 152 | 152 | 0 | 0 | 0
N-03 | 63 | 60 | 2 | 3 | 0
P-01 | 36 | 26 | 10 | 10 | 0
P-02 | 50 | 48 | 2 | 2 | 0

  5. 5.

Output 5: Dispersion test. This component has two outputs. The first one
presents the dispersion per subclass by estimating the average and standard
deviation of the distances to the subclass median. For instance, Table 7 shows
such dispersion per class. Specifically, P-01 and N-01 are the subclasses with
larger dispersion values. Besides the dispersion table, this first output
includes dispersion plots per class. Figure 6 presents such dispersions for
the artificial training set, where visual sparseness not only coincides with
the values from Table 7, but also with that observed in Figure 3. In case of
detecting two or more subclasses within a class (refer subclass detection
component), a second output shows the results of the Anderson’s multivariate
test for homogeneity of variance. The goal is to test the null hypothesis that
assumes variance is equal across subclasses. A p-value less than 0.05
indicates a violation of the assumption. Therefore, same resampling or
synthetic data treatments strategy cannot necessarily be generalized to all
subclasses. For example, Table 7 makes evident the latter statement for the
artificial dataset.

Table 6: Output 5: Subclass dispersion for the artificial training set. | Mean | Std
---|---|---
N-01 | 0.96 | 0.46
N-02 | 0.63 | 0.32
N-03 | 0.65 | 0.33
P-01 | 1.05 | 0.85
P-02 | 0.57 | 0.41

Table 7: Output 5: Anderson’s multivariate test for homogeneity of variance for the artificial training set. Df (Degree of freedom), Sum Sq (Sum of squares), Mean Sq (Mean squared), F (The F-test), N.Perm (Number of permutations), and p-value (Probability value). | Df | Sum.Sq | Mean.Sq | F | N.Perm | p-value
---|---|---|---|---|---|---
Negative |  |  |  |  |  |
Subclass | 1 | 5 | 49 | 12.3 | 999 | 0.002
Residuals | 84 | 34 | 0.4 |  |  |
Positive |  |  |  |  |  |
Subclass | 2 | 11 | 5.3 | 34 | 999 | 0.001
Residuals | 401 | 63 | 0.2 |  |  |

Figure 6: Output 5: Subclass dispersion for artificial training set. a)
Dispersion for Negative subclasses. b) Dispersion for Positive subclasses. The
dotted lines represent the median as a reference point. For each subclass
plotted on the X axis, the distance of each observation to the median, is
plotted in the Y axis. Dark dots represent the mean of the distances to the
median and the error bars the standard deviation.

##  5 Experimental methodology

Previous studies suggested that the class imbalance is not an issue by itself,
but performance degradation of classifiers is also related to other problems
such as overlapping, noisy labels, and small-disjuncts [9, 12, 40, 25, 27, 36,
70]. Although these papers indicate a relationship between such problems, the
true correlation and causality between them is not yet well-established.
Furthermore, common limitations of previous studies are based on lack of
statistical validation, few datasets and classifiers, and the frequent use of
simple or artificially generated datasets. Therefore, their conclusions may
hold only for those limited scenarios. The experimental framework implemented
in the present research is composed of three stages:

  1. 1.

DDP: The first stage seeks to build "Dataset Degradation Profiles" (DDP) for
datasets based on the diagnostic algorithm output. This makes possible to know
deeper the relationship between degradation problems and their critical
levels.

  2. 2.

Classifiers performance: The second stage builds and evaluates several
classification algorithms in order to assess the level of agreement between
performance metrics, and tests classification algorithms robustness in
presence of degradation problems.

  3. 3.

Treatments comparison: Finally, the last stage compares several techniques for
handling degradation problems by making use of the DDP results. This stage
mainly seeks to make evident that the success of classification tasks depends
on the combination of several factors related to dataset diagnosis, treatment,
and classification algorithm.

The Diagnostic model and treatments (techniques for handling degradation
problems) were studied empirically by using 49 datasets from real-world
domains. Table 14 describes main characteristics of that collection, which
varies in their size (instances), number of features (Feat.), and class
imbalance. First 35 datasets are from the UCI machine learning repository
[71], 7 datasets, identified with ‘$\ast$’, from `R` packages, 6 datasets from
different repositories, and one new dataset, the "Simulated dataset",
generated by the author. Those multiclass datasets marked with ‘$\dagger$’
were binarized using the original dataset description. Finally, datasets
identified with ‘$\bullet$’ are subsets of the original datasets. Experiments
from second and third stages are based on 14 classification algorithms, whose
parameter values were set based on two criteria: authors recommendations and
model tuning using resampling. In the latter case, the `caret` (Classification
And Regression Training) package [72, 73, 74, 75] was used, which is available
in `R` open source software. Package `caret` offers a framework for building
machine learning models. For instance, it includes several functions for data
splitting, pre-processing, feature selection, variable importance estimation,
model tuning using resampling. This research used that latter component, which
allowed to choose the optimal model across parameters for classification
algorithms. Finally, the experimental results are obtained based on 10-fold
cross-validation. In summary, first stage results come from the 49 datasets
diagnostic analysis. Second stage experimental results stem from every
combination of the 49 datasets and 14 classifiers (674 trained models); in
short, all classifiers in this stage were trained by using original raw
datasets without the intervention of any remediation technique for handling
degradation problems. Finally, last stage experimental results derive from
every combination of the 49 datasets, 14 classifiers, and 9 treatments (5995
trained models). Table 12 shows a brief description of the classifiers
implemented, which lists functions, packages and tuning parameters for 14
classification algorithms. The list of treatments for degradation problems and
set parameters are shown in Table 13.

##  6 Experimental Results

As previously mentioned, different studies suggest class imbalance is not
completely responsible for classifiers degradation and argue that the problem
of small-disjuncts is even more severe [12, 22, 23, 24, 25, 26]. However, the
small-disjunct concept itself can be interpreted as the imbalance problem
between-class and within-class. Furthermore, one of the reasons why small-
disjuncts show higher misclassification than large disjuncts is due to class
imbalance [76, 77]. On the other hand, other studies state that
misclassification is not solely caused by class imbalance, but also associated
with the degree of overlapping among classes [9, 12, 27]. Keeping the above in
mind, the diagnostic tool integrates those elements by building an output
called "IR and Overlap matrix", which creates a "diagnosis" or profile of the
target dataset.

###  6.1 Dataset profiling

In the following, a comparative example is presented using the Ozone and the
Wine datasets (Refer Table 14) to illustrate the benefits derived from the
diagnostic model. Initially, the Ozone dataset looks more complex since it has
higher dimensionality (number of features) and class imbalance (IR). On the
other hand, the amount of data may be the major issue for the Wine dataset.
Figure 7 compares the IR and Overlap matrix for those datasets. The diagonal
of each matrix contains the number of instances per subclass (white), which
allows for identifying small-disjuncts in both classes. The upper triangular
part (blue) shows the IR between subclasses. Finally, the lower triangular
part (red) of the matrix shows the separation index (degree of overlapping)
between subclasses. The intensity of the colors identifies how critical the
problem based on each metric is. It is important to mention that overlap
between subclasses from the same class is not considered a problem. Overall,
Wine has more subclasses (disjuncts), less amount of data, higher IR between
subclasses, and lack of overlap. In contrast, the most relevant problem of the
Ozone dataset is related to several regions with high overlap and some
subclasses with few instances. Moreover, the segment chart below of each IRO
matrix shows the performance of 14 classifiers for each dataset by using the
G-mean performance metric. There is an evident low performance for most of the
classifiers in the Ozone dataset case, suggesting a need to implement
remediation treatments. Furthermore, even though the Wine dataset has more
small-disjuncts and higher IR between subclasses, this dataset does not
require any remediation treatment. Finally, the Wine dataset may be considered
as not a challenging case for classification task given lack of overlapping
between subclasses from different classes.

In general, the aim of the DDP (Dataset Degradation Profile) is to provide
useful information for selecting or designing a tailored treatment. Dataset
profiles are categorized according to levels of criticality as follows:

  * •

If IR is equal to or less than 10, then level is "Low", otherwise level is
"High".

  * •

If the number of Disjuncts (subclasses) is equal to or less than 10, then
level is "Low", otherwise level is "High".

  * •

If NOR (Noise Overlap Ratio) is equal or less than 0.1, then level is "Low",
otherwise level is "High".

Figure 7: IRO matrix comparison and performance for multiple classifiers. The
diagonal contains the number of instances per subclass (white), which allows
for identifying small-disjuncts in both classes. The upper triangular part
(blue) shows the IR between subclasses. Finally, the lower triangular part
(red) of the matrix shows the separation index (degree of overlapping) between
subclasses. The intensity of the colors identifies how critical the problem
based on each metric is. The segment chart below of each IRO matrix shows the
performance of 14 classifiers for each dataset by using the G-mean performance
metric.

Figure 9 shows a contingency table that summarize the results of the DDP
process for the 49 datasets. The table describes the relationship between
three degradation problems (IR, Disjuncts, and Overlap) in terms of
frequencies of their levels of criticality (Low and High). The level "Unknown"
from the Overlap problem refers to datasets for which it was not possible to
find an optimal projection for estimating the separation index.

Figure 8: Contingency table of dataset profiles. Cross levels show the number
of datasets for each criticality combination.

Figure 9: Performance metrics comparison by using SVM-G. The AUC and the
G-mean are consistent with ${Acc}^{-}$ and ${Acc}^{+}$ for the Wine dataset.
However, the AUC is an optimistic metric for the Ozone dataset (85%) despite
its low ${Acc}^{+}$ (0%). On the other hand, the G-mean is more fair metric
based on the levels of errors.

###  6.2 Performance Metrics Selection

Performance metrics are used to evaluate how well classification algorithms
conduct the discrimination task between classes. Several studies point out
limitations of the global accuracy as a metric of performance in class
imbalance situation [78, 79, 80, 81, 82, 83]. In a nutshell, the global
accuracy metric is heavily biased to favor the Negative class and may hide
poor performance for the Positive class. Furthermore, Positive class is
usually the class of interest in real world applications. Typically, the aim
is to learn in detail the Positive class distribution in order to recognize
its members as accurately as possible. For example, fraud detection is a well-
known case in point, where the number of fraud transactions is much smaller
than the legitimate ones and the goal is to detect such a transactions
immediately in order to avoid financial losses. In conclusion, the main goal
of classifiers is not only improving the accuracy for the Positive class, but
also avoiding decreasing the accuracy for the Negative class during the
process. Continuing with the same fraud example, financial institutions want
to increase the detection of the number of "True Positive" (TP) cases (real
fraud transactions) and avoid increasing the number of "False Positive" (FP)
cases (legitime transactions classified as fraud) because the latter means
higher administrative costs related to responding to the false fraud alerts.
Practitioners have opted to use the Area under the Curve AUC of the Receiver
Operating Characteristic ROC [15] as the metric of performance for supervised
learning, since it is considered more appropriate for imbalance learning [84,
85, 86]. However, recent studies have shown that the AUC may provide an
optimistic performance evaluation [16, 87] and draw wrong conclusions for
imbalance datasets. Thereby, the first crucial contribution of this study in
this context is to clarify how convenient it is to use the AUC as performance
metric for datasets in the presence of degradation problems. If the AUC metric
is shown to be optimistic and misleading, it will be not only inconvenient for
classifiers performance evaluation, but also for testing the effectiveness of
the remediation treatments for degradation problems such as SMOTE, OSS, ENN,
and so on. For instance, Figure 9 compares four performance metrics (AUC,
${Acc}^{-}$, ${Acc}^{+}$, and G-mean) associated with a Support Vector Machine
with Gaussian Kernel (SVM-G) implementation for the Ozone and the Wine
datasets.

As previously mentioned, the Wine dataset is not a challenging case for
classification; Thereby, these metrics agree in terms of the SVM-G.
Nevertheless, the Ozone dataset shows discrepancy between some metrics. Even
though the AUC is above 80% and the accuracy of the Negative class
(${Acc}^{-}$) is 100%, the accuracy of the Positive class (${Acc}^{+}$) is 0%.
This could imply that the AUC is an inaccurate metric in relation to certain
cases. For example, the low ${Acc}^{+}$ may be associated with high level of
global IR (13.76) in the Ozone dataset. In summary, there is one instance in
the Positive class per 13.76 instances in the Negative class. Moreover, the
separation index shows overlap problems between subclasses (IRO matrix from
Figure 6.2). On the other hand, the G-mean is more conservative and intuitive
metric because shows the balance between the accuracy from both classes. Since
the ${Acc}^{+}$ for the Ozone dataset is zero, the G-mean punishes the
classifier performance because it is unable to classify one of the classes
correctly. Technically, the G-mean (also called G-measure) is the geometric
mean of the ${Acc}^{+}$ and the ${Acc}^{-}$, which returns a value between 0
and 1. Technical details of the evaluation metrics implemented in the present
research such as AUC, (F1-score), G-mean, sensitivity, specificity, and
precision can be found in Appendix B. This example makes evident limitations
of the AUC as performance metric for complex datasets. Since the ROC graphs
are drawn using True Positive Rates (TPR) and False Positive Rates (FPR), the
relative imbalance between classes directly affects the FPR, which tends to
zero and generates misleading estimates of the AUC measure.

In general, most classifiers are not designed with the assumption of dealing
with class imbalance [14], Figure 9 is a clear example of this by using
${Acc}^{+}$ and the G-mean metrics, which are performance measures more
conservative than the AUC. The SVM-G showed a low performance in term of
${Acc}^{+}$ for the Ozone dataset, which is not surprising due to its high
imbalance ratio. Figure 11 clearly shows the disagreement between AUC and
G-mean metrics for the Ozone dataset across 14 classification algorithms,
where the AUC for all cases always overpass the G-mean. Another important
finding related to the Wine dataset is that for well-separated subclasses the
performance metric did not decrease even with high levels of imbalance ratio
between subclasses. Thereby, the present experimental stage has two main
purposes: the first one seeks to generalize the improper use of the AUC as
performance metric for datasets under degradation problems. The second one
refers to establishing solid causality with statistical evidence between
degradation problems and classifiers’ performances by using real-world
datasets.

Figure 11 compares the distribution of the AUC and G-mean metrics for the 49
datasets across the 14 classification algorithms (674 trained models). The AUC
boxplot shows a lower variability and its values are closer to the high levels
of prediction power. Moreover, AUC outliers below the lower quartile show
better performance than the outliers from the G-mean. Finally, the measures of
central tendency such as the median and the mean have larger values in the AUC
than the G-mean distribution. Furthermore, this figure shows the result of the
paired samples Wilcoxon test [88, 89, 90], which is a non-parametric version
of the paired t-test. The null hypothesis states that the AUC has values of
prediction power equal to or lower than the G-mean values. In contrast, the
alternative hypothesis claims that the AUC has values of prediction power
higher than the G-mean values. As the p-value turns out to be 5.06e-88, and is
less than the 0.05 significance level, it is possible to reject the null
hypothesis. Therefore, for the experimental design defined in the present
research, the Wilcoxon for paired samples test confirms the risk and
inconvenience of using the AUC. A fair question could be raised as to the AUC
limitations being the same through levels of criticality of the degradation
problems.

Figure 10: Performance metric comparison for the Ozone dataset across 14
classification algorithms. The AUC for all classifiers get higher performance
values (more optimistic) than the G-mean.

Figure 11: Boxplots for AUC and G-mean metrics. It compares the performance
distributions of the 49 datasets across the 14 classification algorithms. The
AUC shows more optimistic prediction power than the G-mean. Moreover, the
paired samples Wilcoxon test rejects the hypothesis that states the AUC has
values of prediction power equal to or lower than the G-mean.

Figure 12 answers the above question by comparing the scatter between AUC and
G-mean for levels of criticality of each degradation problem. The ideal
scenario would be a high linear positive correlation between values for both
performance metrics, even for the low and high values. In a nutshell, if the
dots do not lie close to the 45-degree line, discrepancy between metrics
exists. It is evident that the AUC limitations persist through the different
levels of criticality; however, for high levels of IR, the AUC is more
optimistic due to the low values of the FPR. For the next degradation problem,
the dispersion of the disjunct levels does not show significant difference.
Finally, the overlap problem shows an interesting behavior through its levels.
High levels of overlap make harder to learn the distribution for both classes.
Then, the FPR is increased and the TPR is reduced, which decreases the AUC
values. On the other hand, low levels of overlap avoid decreasing the "True
Negative" value and follow a similar dispersion due to the high level of IR.
Similarly, the paired samples Wilcoxon test was conducted for each level of
criticality. Since all p-values turns out to be less than the 0.05
significance level, it is possible to reject the null hypothesis. Therefore,
for the experimental design defined in the present research, the Wilcoxon for
paired samples test confirms the risk and inconvenience of using the AUC even
for different levels of criticality.

Figure 12: AUC and G-mean comparison across criticality levels of degradation
problems. It compares the scatter between AUC and G-mean by levels of
criticality of degradation problems for the 49 datasets across the 14
classification algorithms. In a nutshell, if the dots do not lie close to the
45-degree line, discrepancy between metrics exists. Moreover, the paired
samples Wilcoxon test rejects the hypothesis that states the AUC has values of
prediction power equal to or lower than the G-mean values.

###  6.3 Classifiers robustness in the presence of degradation problems

Once AUC limitations in the presence of degradation problems are evidenced,
the next aim of this research is to test classification algorithms′ robustness
in the presence of degradation problems. The following results correspond to
the same experimental design defined in the previous section. Basically, this
section seeks to validate three recurrent statements in the literature by
incorporating a considerable real-world dataset, several classification
algorithms, unbiased performance metrics, and statistical validation. The
three statement to validate are:

  * S1:

The more class imbalance, the bigger the classifier performance degradation.

  * S2:

The greater the number of disjuncts, the bigger the classifier performance
degradation.

  * S3:

The larger the overlap regions, the bigger the classifier performance
degradation.

Table 8 illustrates the results of an alternative version of the Wilcoxon
signed-rank test, which is used to compare two related samples. In this
example, results are derived with the 49 datasets and SVM-G classifier by
making use of three performance metrics (G-mean, F1-score, and AUC).
Basically, two tests were implemented in order to validate if statistical
differences between the performance metric distribution across the levels of
criticality for each degradation problem exist. In other words, these tests
assess the truthfulness of the three above statements. Test A compares if the
performance metric distributions between levels are equal, and Test B compares
if the performance metric distribution of the datasets with high levels of
criticality have better prediction power than the performance metric
distribution of the datasets with low levels of criticality. Both tests have a
0.05 significance level. The p-values shown in the table are used to reject or
not reject the null hypothesis. However, an easier notation combines results
of both statistical tests. Symbol "$\checkmark$" means existing statistical
evidence to accept the statement. On the other hand, symbol "$\times$" implies
that there is no statistical evidence to accept the statement. Therefore, it
is not possible to reject that the performance metric distributions across the
levels are different.

Table 8: p-values of the Wilcoxon signed-rank tests for degradation problems by using a SVM-G classifier. Test A compares if the performance metric distributions between levels are equal and Test B compares if the performance metric distribution of the datasets with high levels of criticality have better prediction power than the performance metric distribution of the datasets with low levels of the criticality. | Statements for degradation problems
---|---
Metric |  S1 (IR) |  S2 (Disjuncts) |  S3 (Overlap)
G-mean |  |  |  |  |  |
Test A | 0.00812 | $\checkmark$ | 0.86476 | $\times$ | 0.07202 | $\checkmark$
Test B | 0.00406 | 0.57548 | 0.03601
F1-score |  |  |  |  |  |
Test A | 0.03484 | $\checkmark$ | 0.82594 | $\times$ | 0.06566 | $\checkmark$
Test B | 0.01742 | 0.41297 | 0.03283
AUC |  |  |  |  |  |
Test A | 0.15913 | $\times$ | 0.984 | $\times$ | 0.01767 | $\checkmark$
Test B | 0.07956 | 0.51598 | 0.00883

Analyzing the outputs from Table 8, it is possible to state the following
conclusions for the classifier SVM-G under the experimental framework defined
in this research:

  1. 1.

The more the class imbalance, the bigger the classifier performance
degradation for metrics G-mean and F1-score. On the other hand, AUC metric is
not affected by the imbalance problem since it is an optimistic and misleading
performance metric.

  2. 2.

It is not true that the greater the number of disjuncts is, the bigger the
classifier performance degradation occurs; however, disjuncts detection
procedures allow to estimate the imbalance ratio within classes [26, 76].

  3. 3.

It is true that the larger the overlap regions, the bigger classifier
performance degradation.

Table 9 shows the same statistical analysis for the complete set of
classifiers by using the G-mean as performance metric, which is a more
conservative and intuitive metric than the AUC because it shows the balance
between the accuracy from both classes. The column "Tests" validates the
statements about level of criticality (Test A and Test B) and column "% Diff"
shows the difference in percentage between the means of the levels of
criticality. Some important findings are the followings:

  * •

Ten out of fourteen classifiers present G-mean degradation in the presence of
high levels of IR.

  * •

Thirteen of fourteen classifiers show G-mean degradation in the presence of
high levels of overlap.

  * •

The G-mean for all classifiers is not affected by high levels of disjuncts.

  * •

QDA classifier does not show statistical difference for the G-mean
distribution between high and low levels of IR, disjuncts and overlap.

Table 9: Wilcoxon test for degradation problems across classifiers by using the G-mean performance metric. The column "Tests" validates the statements about level of criticality and column "% Diff." shows the difference in percentage between the means of criticality levels. | Statements for degradation problems
---|---
|  S1 (IR) |  S2 (Disjuncts) |  S3 (Overlap)
Classifier | Tests | %Diff. | Tests | %Diff. | Tests | %Diff.
NB | $\times$ | 0.07 | $\times$ | 0.06 | $\checkmark$ | 0.05
LOGREG | $\checkmark$ | 0.15 | $\times$ | 0.03 | $\checkmark$ | 0.17
LDA | $\times$ | 0.13 | $\times$ | 0.03 | $\checkmark$ | 0.17
QDA | $\times$ | 0.12 | $\times$ | -0.01 | $\times$ | 0.02
C5.0 | $\checkmark$ | 0.17 | $\times$ | -0.05 | $\checkmark$ | 0.12
CART | $\checkmark$ | 0.21 | $\times$ | -0.08 | $\checkmark$ | 0.14
k-NN | $\checkmark$ | 0.22 | $\times$ | -0.03 | $\checkmark$ | 0.10
MLN | $\times$ | 0.16 | $\times$ | 0.00 | $\checkmark$ | 0.14
SVM-G | $\checkmark$ | 0.29 | $\times$ | -0.01 | $\checkmark$ | 0.09
SVM-P | $\checkmark$ | 0.27 | $\times$ | 0.05 | $\checkmark$ | 0.18
RF | $\checkmark$ | 0.23 | $\times$ | 0.01 | $\checkmark$ | 0.08
SGB | $\checkmark$ | 0.18 | $\times$ | 0.01 | $\checkmark$ | 0.09
ADABOOTS | $\checkmark$ | 0.23 | $\times$ | 0.01 | $\checkmark$ | 0.08
RBFN | $\checkmark$ | 0.24 | $\times$ | -0.02 | $\checkmark$ | 0.08

###  6.4 Statistical identification of the best treatment

To demonstrate that it is not possible to find one treatment that is always
the best in remediation for all degradation problems and classifiers, a
ranking score was computed over three performance metrics (AUC, F1-score, and
G-mean) to each treatment results across the 5995 trained models, which were
built over valid combination of 49 datasets, 14 classifiers, and 9 treatments
(techniques for handling degradation problems). Therefore, the ranking score
for the best performing treatment is 9 and the worst performing treatment is
1. Then, the nonparametric Friedman test [91, 92, 93] and multiple comparison
of treatments were applied to the ranking results. This test not only allows
to detect differences between the treatments results across multiple
experimental related samples, but also implements the post hoc Friedman tests
(multiple comparisons between treatments) by using the criterion Fisher′s
Least Significant Differences (LSD) [89, 90, 94], which groups the treatments
according to similarities by using the Compact Letter Display method [95]. The
null hypothesis for the Friedman test is that there are no differences between
treatment results across multiple experimental attempts. If the null
hypothesis is rejected (p-value less that the 0.05 significance level), it can
be concluded that at least two of the treatments are significantly different
from each other. Then, a multiple comparisons analysis can be executed in
order to know which of the treatments are significantly different from which
other treatments. In particular, the multiple comparisons results are
presented by listing the treatments in order of decreasing average score and
grouping the treatments that are not significantly different. Thus, the
procedure creates groups (identified by letters) as a union of different
treatments that can contain one or more members, and the members of these
groups are the nine different treatment in this research (Raw, Random, SMOTE,
B-SMOTE, DBSMOTE, ADASYN, ENN, NCL, and OSS). Treatments with the same
letter(s) are not statistically different. On the contrary, treatments that
are statistically different get different letters. Treatments can have more
than one letter to reflect overlap between the groups of treatments. Table 10
shows the Friedman test results for 49 datasets, 14 classifiers, and 9
treatments by using the G-mean performance metric. According to the p-value
column, all values turns out to be less than the 0.05 significance level.
Therefore, it is possible to reject the null hypothesis and accept that there
are statistical differences between the treatments across classifiers. For
instance, the NB (Naive Bayes) classifier shows better performance by using
SMOTE treatment (letter "a"). On the contrary, B-SMOTE, DBSMOTE, NCL, and OSS
treatments grouped by the letter "c" are the less convenient alternatives for
the NB classifier; even worse than the Raw data (original data). These results
call into question those in previous research on degradation problems, which
present successful techniques across few classifiers and datasets. More
importantly, this analysis supports one of the hypotheses of the present
research that states it is not possible to find one treatment that is the best
in remediation for all degradation problems, datasets or classifiers. The
selection of the "best treatment" or even the most convenient classification
algorithm depends on the available information, knowledge associated with the
target dataset, and strengths and weaknesses of classification algorithms. For
this reason, the diagnostic test for degradation problems has a direct
relevance to the treatment selection in order to obtain successful
classification outcomes and to avoid degradation.

Table 10: Friedman test results for all datasets by using the G-mean metric. According to the p-value column, all values turns out to be less than the 0.05 significance level. Then, there are statistical differences between treatments across classifiers. | Treatment |
---|---|---
Classifier |  Raw |  Random |  SMOTE |  B-SMOTE |  DBSMOTE |  ADASYN |  ENN |  NCL |  OSS | p-values
NB | bc | ab | a | c | c | bc | abc | c | c | $2.0\mathrm{e}{-02}$
LOGREG | e | a | ab | bc | ab | bc | cd | d | de | $2.2\mathrm{e}{-10}$
LDA | e | a | bc | bc | cd | ab | e | de | e | $1.0\mathrm{e}{-10}$
QDA | d | ab | abc | cd | bcd | a | abcd | ab | abcd | $2.2\mathrm{e}{-02}$
C5.0 | d | ab | ab | a | abcd | a | cd | bcd | abc | $4.6\mathrm{e}{-04}$
CART | d | a | ab | ab | ab | a | cd | ab | bc | $8.1\mathrm{e}{-05}$
k-NN | c | a | ab | bc | ab | a | c | ab | ab | $4.4\mathrm{e}{-06}$
MLN | c | ab | ab | ab | bc | a | bc | ab | bc | $2.1\mathrm{e}{-02}$
SVM-G | e | a | a | ab | bc | a | d | bc | c | $0.0\mathrm{e}{-00}$
SVM-P | e | ab | a | bc | ab | ab | de | bc | cd | $3.5\mathrm{e}{-09}$
RF | d | b | a | b | b | a | cd | bc | bc | $1.3\mathrm{e}{-08}$
SGB | e | a | ab | ab | bc | ab | de | bcd | cd | $2.2\mathrm{e}{-07}$
ADABOOTS | d | bc | ab | abc | ab | a | cd | bc | abc | $6.3\mathrm{e}{-06}$
RBFN | e | cd | a | cd | bc | ab | de | bc | c | $1.1\mathrm{e}{-07}$

As previously mentioned in the experimental methodology, one of the goals is
to compares several treatments for degradation problems by making use of the
DDP results over 49 datasets. The new hypothesis states that success of
classification tasks depends on the combination of several factors such as
DDP, treatment, and classification algorithm. Based on the information from
Figure 9, two dataset profiles are selected for validating the hypothesis:

  * •

Profile A: datasets with high levels of class imbalance and levels of
disjuncts. 8 datasets have this profile: Seismic, Wilt, Abalone, Blocks,
nanoHUB, Fraud, Mammo, and SDSS.

  * •

Profile B: datasets with high levels of overlap and low levels of class
imbalance. 16 datasets have this profile: Blood, College, Glass, ILPD,
Ionosphere, MDRR, Phoneme, Pima1, Pima2, Ringnorm, Satimage, Sonar, Spam,
Sports, Vertebral, and Weather.

Similarly to the results for the complete datasets, the Friedman test for
these two profiles showed statistical differences between the treatments
across the 14 classifiers by using the mean of the G-mean ranks. On the
contrary, if this test uses the mean of the AUC ranks; then, there is no
statistical differences between the treatments across several classifiers for
both profiles. This situation not only makes evident the drawback of the AUC
mentioned in the sections before, but also its limitations as performance
metric for comparing remediation treatments for degradation problems. Table 11
shows the heat maps of the mean rank for the G-mean metric for both dataset
profiles across treatments and classifiers. High mean rank values tend to
increase the steepness of the color gradient to green. On the the other hand,
low mean rank values tend to decrease the steepness of the color gradient to
red. The purpose of these figures is not to identify the best classification
algorithm, but the best treatment in terms of the DDP and classifier. It is
evident Profiles A and B show different response patterns.

In general, training sets treated with remediation techniques showed better
classification performance than Raw training sets. Some significant findings
are presented below:

  * •

Data cleaning techniques (ENN, NCL, and OSS) showed low performance for
datasets with Profile A.

  * •

In general, Random, SMOTE, and ADASYN show good performance across classifiers
for datasets with Profile A.

  * •

There is no consistent treatment across classifiers for datasets with Profile
B. Therefore, the best classification performance for datasets with high
overlap and low class imbalance is obtained by selecting carefully the
combination between treatment and classifier.

  * •

For NB (Naive Bayes) classifier the best remediation techniques for datasets
with Profile B are ENN and Raw.

  * •

For k-NN (k-Nearest Neighbors) classifier the best remediation techniques for
datasets with Profile B are OSS and NCL .

  * •

BDSMOTE technique takes top places associated with Profile B for LOGREG
(Logistic Regression), CART (Classification and Regression Trees), and SVM-P
(Support Vector Machines with Gaussian Kernel).

  * •

Raw (original datasets) showed low performance for SVM-G (Support Vector
Machines with Gaussian Kernel) and RBFN (Radial Basis Function Network).

Table 11: Mean ranks of the G-mean metric based on Dataset Degradation Profile
(DDP) across 49 classifiers and 9 treatments. High mean rank values tend to
increase the steepness of the color gradient to green. On the the other hand,
low mean rank values tend to decrease the steepness of the color gradient to
red.

(a) Profile A: high class imbalance and high disjuncts. | Treatment
---|---
Classifier |  Raw |  Random |  SMOTE |  B-SMOTE |  DBSMOT |  ADASYN |  ENN |  NCL |  OSS
NB |  4 .37  |  6 .00  |  6 .87  |  7 .37  |  5 .12  |  6 .37  |  4 .37  |  3 .37  |  3 .12
LOGREG |  2 .00  |  8 .00  |  7 .87  |  6 .00  |  6 .37  |  6 .37  |  2 .62  |  4 .12  |  3 .62
LDA |  3 .50  |  8 .12  |  6 .75  |  6 .12  |  6 .37  |  6 .87  |  3 .62  |  3 .37  |  3 .87
QDA |  4 .00  |  7 .25  |  6 .37  |  5 .62  |  3 .50  |  7 .12  |  3 .37  |  5 .00  |  5 .00
C5.0 |  3 .12  |  6 .75  |  7 .25  |  5 .25  |  5 .50  |  7 .50  |  2 .87  |  4 .00  |  3 .62
CART |  3 .50  |  6 .87  |  8 .25  |  4 .75  |  4 .62  |  8 .00  |  3 .12  |  4 .75  |  3 .25
k-NN |  2 .62  |  7 .75  |  7 .25  |  5 .37  |  5 .75  |  7 .00  |  3 .00  |  4 .62  |  3 .50
MLN |  2 .87  |  5 .75  |  6 .37  |  6 .37  |  4 .62  |  7 .75  |  3 .25  |  4 .12  |  4 .62
SVM-G |  1 .87  |  8 .12  |  7 .87  |  6 .37  |  4 .87  |  7 .75  |  2 .25  |  4 .00  |  3 .37
SVM-P |  2 .50  |  8 .12  |  7 .75  |  7 .62  |  6 .12  |  6 .25  |  2 .87  |  3 .87  |  3 .50
RF |  3 .00  |  5 .37  |  8 .00  |  3 .87  |  6 .62  |  7 .75  |  3 .62  |  4 .62  |  3 .62
SGB |  2 .37  |  7 .25  |  7 .87  |  6 .00  |  4 .87  |  7 .62  |  2 .12  |  4 .37  |  3 .12
ADABOOTS |  2 .62  |  3 .62  |  8 .00  |  5 .37  |  6 .75  |  8 .12  |  4 .25  |  3 .87  |  3 .37
RBFN |  1 .87  |  5 .87  |  8 .00  |  5 .50  |  6 .50  |  7 .00  |  3 .00  |  5 .00  |  3 .50

(b) Profile B: high overlap and low class imbalance. | Treatment
---|---
Classifier |  Raw |  Random |  SMOTE |  B-SMOTE |  DBSMOT |  ADASYN |  ENN |  NCL |  OSS
NB |  6 .50  |  6 .31  |  5 .18  |  4 .31  |  3 .75  |  5 .25  |  7 .18  |  5 .50  |  5 .06
LOGREG |  3 .68  |  6 .25  |  6 .00  |  5 .75  |  6 .31  |  5 .12  |  5 .18  |  3 .81  |  4 .68
LDA |  3 .86  |  7 .20  |  6 .00  |  5 .93  |  5 .06  |  5 .86  |  3 .73  |  4 .33  |  4 .86
QDA |  4 .58  |  5 .58  |  4 .83  |  3 .16  |  6 .00  |  5 .66  |  6 .08  |  6 .75  |  6 .16
C5.0 |  4 .25  |  5 .25  |  5 .93  |  6 .81  |  5 .43  |  5 .00  |  5 .43  |  4 .56  |  5 .50
CART |  4 .87  |  6 .87  |  4 .93  |  4 .25  |  6 .62  |  4 .43  |  5 .18  |  5 .87  |  6 .25
k-NN |  3 .68  |  5 .50  |  4 .68  |  4 .37  |  5 .62  |  5 .18  |  4 .25  |  6 .25  |  6 .81
MLN |  4 .50  |  5 .87  |  5 .31  |  5 .62  |  5 .25  |  5 .50  |  4 .87  |  5 .31  |  4 .93
SVM-G |  2 .37  |  7 .00  |  6 .31  |  6 .00  |  5 .75  |  6 .81  |  3 .43  |  6 .12  |  4 .75
SVM-P |  3 .31  |  5 .81  |  5 .93  |  4 .93  |  6 .50  |  5 .93  |  4 .18  |  6 .00  |  5 .06
RF |  4 .12  |  6 .25  |  6 .93  |  6 .00  |  4 .50  |  7 .00  |  3 .93  |  5 .12  |  5 .00
SGB |  4 .18  |  6 .87  |  6 .00  |  6 .68  |  5 .87  |  4 .68  |  4 .50  |  5 .31  |  5 .43
ADABOOTS |  3 .68  |  5 .81  |  6 .12  |  5 .93  |  5 .62  |  6 .31  |  5 .18  |  5 .00  |  5 .18
RBFN |  3 .06  |  5 .18  |  7 .12  |  4 .31  |  5 .50  |  6 .43  |  3 .93  |  5 .56  |  5 .31

##  7 Conclusions and Future Work

The performance of supervised learning algorithms is hindered by the presence
of degradation problems and lack of diagnosis for training sets before
building classifiers. This research describes how the introduction of
diagnostic methods help minimize degradation in classification performance.

In this research, an experimental framework was proposed to describe the
complete diagnostic method, where experimental results were derived from
statistical analysis over every combination of the 49 datasets, 14
classifiers, and 9 treatments (techniques for handling degradation problems).
Experimental results in this study allowed to implement statistical validation
procedure related to performance degradation in supervised learning. 1) The
more the class imbalance, the bigger the classifier performance degradation
for metrics G-mean and F1-score. On the other hand, AUC is not affected by the
imbalance problem since it is an optimistic and misleading performance metric.
2) It is not true that the greater number of disjuncts, the bigger the
classifier performance degradation; however, disjuncts detection allows
knowing the imbalance ratio within classes. 3) It is true that the larger the
overlap regions, the bigger the classifier performance degradation.

In general, the SMOTE treatment takes top places in terms of increasing the
classification performance, even overpassing its most recent extensions such
as Borderline-SMOTE and DBSMOTE. However, SMOTE based techniques ignore the
problem of small-disjuncts, more specifically subclass imbalance (imbalance
within classes).

Even though data cleaning methods are designed to handling overlap problems,
experiments over datasets with Profile B (high overlap and low class
imbalance) showed poor classification performance. In general, results for the
three data cleaning techniques used in this research (ENN, NCL, and OSS) were
overpassed by sampling techniques. This can be explained by the wrong
generalization of the concept of "Noisy Labels" or "mislabeling". In other
words, instances located in overlap regions not necessarily have wrong labels.
Thereby, these strategies of removing or label reallocation increase the
problem of sparseness and reduce the classification performance. This latter
result is a generalization of the Quinlan [43] results applied to decision
trees.

Results from datasets with high levels of class imbalance and disjuncts
support that classification performance is significantly improved by
increasing the number of synthetic Positive instances. In contrast, cleaning
techniques such as ENN, NCL, and OSS showed adverse effects on classification
performance.

Future work is related to building a technique based on diagnostic tests and
DDP, which has its primary focus of reversing the degradation effects due to
class imbalance between-classes and within-classes (small-disjuncts),
overlapping and noisy labels problems.

Table 12: Classification algorithms, tuning parameters, and packages in R open source software. Abbreviation names: NB (Naive Bayes), LOGREG (Logistic Regression), LDA (Linear Discriminant Analysis), QDA (Quadratic Discriminant Analysis), C5.0, CART (Classification and Regression Trees), k-NN (k-Nearest Neighbors), MLN (Multilayer Neural Network), SVM-G (Support Vector Machines with Gaussian Kernel), SVM-P (Support Vector Machines with Polynomial Kernel), RF (Random Forest), SGB (Stochastic Gradient Boosting), ADABOOST (AdaBoost Classification Trees), and RBFN (Radial Basis Function Network). Classifier | Function | Package | Tuning parameters
---|---|---|---
NB | nb | klaR | fL (Laplace Correction)
usekernel (Distribution Type)
adjust (Bandwidth Adjustment)
LOGREG | glm | R Stats | NA
LDA | lda | MASS | NA
QDA | qda | MASS | NA
C5.0 | C5.0 | C5.0, plyr | trials = c(1,5,10,15,20)
winnow = c(T,F), model = tree
CART | rpart | rpart | method = clas, cp = 0, xval = 10
k-NN | knn | class | k (Num. Neighbors)
MLN | nnet | nnet | size (Num. Hidden Units)
decay (Weight Decay)
bag (Baggaing)
SVM-G | ksvm | kernlab | type = C-svc, kernel= rbfdot
kpar = automatic, C=1
SVM-P | ksvm | kernlab | type = C-svc, kernel= polydot
kpar = automatic, C=1
RF | rf | randomForest | mtry (Num. Randomly Selected Predictors)
SGB | gbm | gbm, plyr | n.trees (Num. Boosting Iterations)
interaction.depth (Max Tree Depth)
shrinkage (Shrinkage)
n.minobsinnode (Min. Terminal Node Size)
ADABOOST | adaboost | fastAdaboost | nIter (Num. Trees), method
RBFN | rbfDDA | RSNNS | negativeThreshold
(Activation Limit for Conflicting Classes)

Table 13: Treatments for handling degradation problems and packages in R open source software. Abbreviation names: Raw (original set), Random (resampling), SMOTE (Synthetic Minority Oversampling Technique), B-SMOTE (Borderline-SMOTE), DBSMOTE (Density-base SMOTE), ADASYN (Adaptative Synthetic Sampling Approach for Imbalanced Learning), ENN (Edited Nearest Neighbors), NCL (Neighborhood Cleaning Rule NCL), and OSS (One-Side Selection). Name | Function | Package | Tuning parameters
---|---|---|---
Raw | - | - | -
Random | - | - | Over (Oversampling parameter)
Under (Undersampling parameter)
SMOTE | SMOTE | DMwR | perc.over, perc.under
k = 3
|  |  | Over (Oversampling parameter)
|  |  | Under (Undersampling parameter)
B-SMOTE | BLSMOTE | smotefamily | K = 3 (Num. Neighbors sampling process)
C = 3 (Num. Neighbors safe-level process)
dupSize (Num. Synthetic Positive instances)
method = ‘type2’
DBSMOTE | DBSMOTE | smotefamily | dupSize (Num. Synthetic Positive instances)
ADASYN | ADASYN | smotefamily | K = 3 (Num. Neighbors sampling process)
ENN | ubENN | unbalanced | K = 3 ( Neighbors)
NCL | ubNCL | unbalanced | K = 3 (Num. Neighbors)
OSS | ubOSS | unbalanced | -

Table 14 describes main characteristics of the dataset collection, which
varies in their size, number of features (Feat.), and class imbalance. First
35 datasets are from the UCI machine learning repository [71], 7 datasets,
identified with ‘$\ast$’, from `R` packages, 6 datasets from different
repositories, and one new dataset, the "Simulated dataset", generated by the
author. Those multiclass datasets marked with ‘$\dagger$’ were binarized using
the original dataset description. Finally, datasets identified with
‘$\bullet$’ are subsets of the original datasets. Abbreviation names: id
(dataset identification), Name (dataset name), Feat. (num. of features), Obs.
(num. of observations), Mis. (num. or missing values), Dup. (num. of
duplicated observations), IR trn (IR training dataset), IR test (IR testing
dataset), %Train (percentage training), %Test (percentage testing), %Over
(percentage oversampling), %Under (percentage undersampling), IR$\star$ (new
IR trn after treatment).

Table 14: Description of datasets id | Name | Feat. | Obs. | Mis. | Dup. | IR trn | IR test | %Train | %Test | %Over | %Under | IR$\star$
---|---|---|---|---|---|---|---|---|---|---|---|---
2 | EEG | 14 | 14980 | 0 | 0 | 1.23 | 1.23 | 70.51 | 29.49 | 20 | 600 | 1.00
3 | Blocks$\dagger$ | 10 | 5473 | 0 | 45 | 45.44 | 49.63 | 70.15 | 29.85 | 2000 | 105 | 1.00
4 | ILPD$\dagger$ | 9 | 583 | 4 | 8 | 2.50 | 2.45 | 70.40 | 29.60 | 100 | 200 | 1.00
5 | Glass$\dagger$ | 9 | 214 | 0 | 1 | 1.88 | 2.60 | 66.20 | 33.80 | 100 | 180 | 1.11
6 | QSAR | 41 | 1055 | 0 | 1 | 2.11 | 1.71 | 68.41 | 31.59 | 110 | 190 | 1.00
7 | Ozone | 72 | 2534 | 687 | 0 | 13.76 | 12.67 | 71.14 | 28.86 | 1200 | 108 | 1.00
8 | Occupancy | 5 | 17895 | 0 | 867 | 3.21 | 3.76 | 42.73 | 57.27 | 200 | 150 | 1.00
9 | Vertebral | 6 | 310 | 0 | 0 | 2.45 | 1.53 | 69.03 | 30.97 | 200 | 120 | 1.00
10 | Haberman | 3 | 306 | 0 | 7 | 3.02 | 2.13 | 68.56 | 31.44 | 200 | 150 | 1.00
11 | Spam | 57 | 4601 | 0 | 238 | 1.51 | 1.53 | 69.20 | 30.80 | 50 | 300 | 1.00
12 | Gamma | 10 | 19020 | 0 | 60 | 1.86 | 1.87 | 70.46 | 29.54 | 100 | 185 | 1.00
13 | Blood | 4 | 748 | 0 | 131 | 2.56 | 3.48 | 62.24 | 37.76 | 200 | 125 | 1.20
14 | Seismic | 11 | 2584 | 0 | 2 | 13.50 | 16.23 | 71.30 | 28.70 | 1000 | 110 | 1.00
15 | Wilt | 5 | 4839 | 0 | 14 | 57.45 | 1.67 | 89.64 | 10.36 | 2000 | 105 | 1.00
16 | Abalone$\dagger$ | 7 | 4177 | 0 | 0 | 19.89 | 24.46 | 69.52 | 30.48 | 1000 | 110 | 1.00
17 | Audit | 8 | 776 | 1 | 26 | 1.98 | 1.57 | 68.49 | 31.51 | 100 | 198 | 1.00
18 | Wireless$\dagger$ | 7 | 2000 | 0 | 0 | 2.94 | 3.16 | 70.45 | 29.55 | 100 | 200 | 1.00
19 | User$\dagger$ | 5 | 403 | 0 | 0 | 9.75 | 4.58 | 64.02 | 35.98 | 900 | 105 | 1.07
20 | Shuttle$\dagger\bullet$ | 9 | 17500 | 0 | 0 | 4.00 | 3.84 | 71.43 | 28.57 | 300 | 133 | 1.00
21 | Image$\dagger$ | 18 | 2310 | 0 | 120 | 1.27 | 1.43 | 69.41 | 30.59 | 30 | 420 | 1.00
22 | Happiness | 6 | 143 | 0 | 9 | 1.08 | 0.72 | 58.96 | 41.04 | 20 | 530 | 1.22
23 | Skin | 3 | 12500 | 0 | 3723 | 2.38 | 4.00 | 57.97 | 42.03 | 100 | 200 | 1.00
24 | Seeds | 7 | 210 | 0 | 0 | 1.82 | 2.45 | 67.14 | 32.86 | 100 | 182 | 1.11
25 | Musk1 | 166 | 476 | 0 | 0 | 1.40 | 1.08 | 71.22 | 28.78 | 50 | 280 | 1.06
26 | CTG$\dagger$ | 20 | 2129 | 3 | 8 | 3.58 | 3.40 | 70.30 | 29.70 | 200 | 170 | 1.13
27 | Sonar | 60 | 208 | 0 | 0 | 1.18 | 1.06 | 69.23 | 30.77 | 20 | 590 | 1.00
28 | Forest$\dagger$ | 27 | 523 | 0 | 0 | 4.35 | 6.07 | 37.86 | 62.14 | 400 | 109 | 1.16
29 | HTRU2 | 8 | 17898 | 0 | 0 | 9.90 | 9.97 | 70.45 | 29.55 | 800 | 110 | 1.00
30 | Adult$\oplus$ | 6 | 26281 | 0 | 24 | 3.13 | 3.23 | 37.99 | 62.01 | 200 | 150 | 1.00
31 | Sports | 59 | 1000 | 0 | 3 | 1.86 | 1.52 | 67.60 | 32.40 | 100 | 185 | 1.08
32 | Banknote | 4 | 1372 | 0 | 11 | 1.21 | 1.28 | 69.29 | 30.71 | 20 | 600 | 1.00
33 | Electrical | 13 | 10000 | 0 | 0 | 1.78 | 1.73 | 70.43 | 29.57 | 100 | 177 | 1.13
34 | Wine$\dagger$ | 13 | 178 | 0 | 0 | 2.94 | 2.20 | 73.03 | 26.97 | 200 | 146 | 1.00
35 | Breast | 9 | 699 | 0 | 153 | 1.03 | 0.49 | 66.79 | 33.21 | 10 | 1034 | 1.14
36 | nanoHUB | 18 | 14110 | 0 | 142 | 39.63 | 38.02 | 70.39 | 29.61 | 2000 | 105 | 1.00
37 | Simulated | 2 | 700 | 0 | 0 | 4.70 | 5.18 | 70.00 | 30.00 | 300 | 130 | 1.00
38 | Weather$\ast$ | 17 | 266 | 0 | 0 | 4.52 | 4.55 | 68.64 | 31.36 | 400 | 113 | 1.11
39 | Pima1$\ast$ | 7 | 532 | 0 | 0 | 2.11 | 1.76 | 71.99 | 28.01 | 100 | 200 | 1.00
40 | Ionosphere$\ast$ | 32 | 351 | 0 | 1 | 1.76 | 1.96 | 79.71 | 20.29 | 76 | 233 | 1.00
41 | Pima2$\ast$ | 8 | 768 | 0 | 0 | 2.01 | 1.58 | 69.40 | 30.60 | 200 | 150 | 1.00
42 | Ringnorm | 20 | 7400 | 0 | 0 | 1.02 | 1.02 | 70.31 | 29.69 | 2 | 5100 | 1.00
43 | College$\ast$ | 17 | 777 | 0 | 0 | 3.05 | 2.03 | 69.24 | 30.76 | 200 | 152 | 1.00
44 | Iris$\ast\dagger$ | 4 | 150 | 0 | 0 | 1.74 | 2.56 | 62.00 | 38.00 | 76 | 228 | 1.00
45 | MDRR$\ast$ | 342 | 528 | 0 | 2 | 1.36 | 1.16 | 71.67 | 28.33 | 40 | 339 | 1.00
46 | Mammo | 6 | 11183 | 0 | 2343 | 28.56 | 46.58 | 62.86 | 37.14 | 1000 | 110 | 1.00
47 | Phoneme | 5 | 5404 | 0 | 32 | 2.39 | 2.48 | 70.16 | 29.84 | 100 | 200 | 1.00
48 | Fraud$\bullet$ | 30 | 10492 | 0 | 13 | 21.94 | 18.69 | 70.49 | 29.51 | 1000 | 110 | 1.00
49 | SDSS$\dagger\bullet$ | 15 | 10000 | 0 | 0 | 11.28 | 9.74 | 69.39 | 30.61 | 1000 | 110 | 1.00

1

2

3

Input : $S$ | is the training set, $X=\\{x_{1},x_{2},...,x_{i+1},x_{i+2},...x_{M}\\}$ are the instances of $S$
---|---
$Label$ | is the class membership vector of $X_{i}$. Positive ($P$) or Negative ($N$)
$k$ | is the number of k-Nearest Neighbors
$C.min$ | is the minimum num. of instances per component (subclass)
$G.max$ | is the maximum num. of mixture components for which the BIC is calculated
$sp.th$ | is the separation index threshold
$dist$ | is the distance measure to be use. 1 = Mahalanobis, 2 = Manhattan, and 3 = Euclidean

4

Output : $S_{1}$ | if $C.min$ is not covered, $S_{1}=$ Subclass membership for outlier analysis
---|---
$S_{2}$ | if $C.min$ is covered, $S_{2}=S_{1}+$ Diagnostic report

5

6Diagnosis$(S,k,C.min,G.max,sp.th,dist)$

7

/* Find subclasses by Gaussian Mixture Models */

8 foreach  _Class $\in$ $S$_ do

9 Best Model = $\operatorname*{argmax}_{x\in Class}$
BIC$\left(\mathbf{X};\theta\right)$, see Eq. (5)

10 Let $S_{1}=\\{$Subclass membership $\cup\ S\\}$;

11 $P_{g}=\\{p_{1},...,p_{K_{P}}\\}$, set of subclasses from P; $K_{P}=$ Num.
of subclasses in $P$;

12 $N_{j}=\\{n_{1},...,n_{K_{N}}\\}$, set of subclasses from N; $K_{N}=$ Num.
of subclasses in $N$;

13 $L_{P}=Length(P_{g})$, vector with the length of each subclass P;

14 $L_{N}=Length(N_{j})$, vector with the length of each subclass N;

15

16if  _any $(L_{p},L_{N})$ < C.min_ then

17 Return $S_{1}$;

18

19else

/* Calculate IR between subclasses */

20 $Upper=$ IR$(Sets)$, upper triangular matrix;

21 $Diag=$ $L_{P}$ $\cup$ $L_{N}$, diagonal of the triangular matrix;

22

/* Find the optimal projection between subclasses */

23 if  _Projection is TRUE_ then

24 $Lower=J^{*}(S_{1})$, lower triangular matrix, see Eq. (7);

25

26 else

27 $Lower=$ IR$(S_{1})$, lower triangular matrix, see Eq. (7);

28

29 IRO $=\\{Upper\cup Lower\cup Diag\\}$, Imbalance Ratio and Overlap matrix;

/* Noisy label analysis */

30

31 foreach  _$X_{i}\in S$_ do

32 Find class.neighbors, class with largest num. of instances among the
k-Nearest Neighbors;

33 if  _$Label_{i}=class.neighbors$_ then

34 $x_{i}$ has the correct label;

35

36 else

37 $x_{i}$ is noise;

38 if  _$x_{i}$ is located between two subclasses with $J^{*}\geq sp.th$_ then

39 $x_{i}\in$ NL, Noisy Label set;

40

41 else

42 $x_{i}\in$ NO, Noise Overlap set;

43

44

45

/* Dispersion analysis: DA */

46 foreach  _Class $\in S$_ do

47 Estimate $mean$ and $std$ to the $median$ per subclass by using $dist$;

48 Execute the test for homogeneity of multivariate dispersion

49 Diagnostic report = IRO, NL, NO, and DA;

50 Let $S_{2}=S_{1}+$ Diagnostic report;

51 Return $S_{2}$;

52

Algorithm 1 Diagnostic tests

##  Appendix A Gaussian Mixture Models and Separation Index

Gaussian Mixture Models (GMM): the Multivariate Gaussian distribution can be
defined over a $D$-dimensional vector $x$ of contiguous variables, which is
given by

$\mathcal{N}\left(x\middle|\mu,\Sigma\right)=\frac{1}{\left(2\pi\right)^{\frac{D}{2}}}\frac{1}{\left|\Sigma\right|^{\frac{1}{2}}}\exp{\left\\{-\frac{1}{2}\left(x-\mu\right)^{T}\Sigma^{-1}\left(x-\mu\right)\right\\}}$
(1)

where $\mu$ is the mean vector of dimension $D$, $\Sigma$ is the covariance
matrix of dimension $D$$\times$$D$, and $|\Sigma|$ denotes the determinant of
$\Sigma$. Then, GMM can be written as a linear superposition of $K$
Multivariate Gaussian Distributions [59, 60] in the form

$p\left(x|\pi,\mu,\Sigma\right)=\sum_{k=1}^{K}\pi_{k}\mathcal{N}\left(x\middle|\mu_{k},\Sigma_{k}\right)$
(2)

Where $\pi_{k}=\ \left\\{\pi_{1},\ldots\pi_{K}\right\\}$ is the mixing
coefficient vector (probability vector of membership) and
$\sum_{k=1}^{K}{\pi_{k}=1}$ in order to be valid probabilities. Moreover,
$\mu_{k}=\ \left\\{\mu_{1},\ldots\mu_{K}\right\\}$ and $\Sigma_{k}=\
\left\\{\Sigma_{1},\ldots\Sigma_{K}\right\\}$.

For a given training set $\mathbf{X}=\ \left\\{x_{1},\ \ldots,\
x_{N}\right\\}$, $\hat{\mathcal{L}}$ is the Maximum Likelihood Estimator (LME)
and its log-likelihood function is expressed by `Equation` 4 , which estimates
the probability of membership of $K$ components. The EM algorithm is used to
maximize this function and estimate the mixture parameters [61].

$\hat{\mathcal{L}}{\left(\mathbf{X};\theta\right)}=p\left(\mathbf{X}|\pi,\mu,\mathrm{\Sigma}\right)\
$ (3)

$log{p\left(\mathbf{X}|\pi,\mu,\mathrm{\Sigma}\right)}{=\sum_{n=1}^{N}log{\left\\{\sum_{k=1}^{K}{\pi_{k}\mathcal{N}\left(x\middle|\mu_{k},\mathrm{\Sigma}_{k}\right)}\right\\}}}\
$ (4)

Determining the optimal number of components $K$ continues to be a problem for
empirical cluster techniques such as K-means. However, probabilistic
approaches based on mixture models can solve this problem using a penalty
function. If the number of components $K$ is increased in the `Equation` 4, it
results in an increase in the dimensionality of the model, causing a
monotonous increase in its likelihood. This is particularly a problem because
it is not useful to obtain as many components as instances. Therefore, the
best GMM is the one that maximizes the Bayesian Information Criterion (BIC)
[62, 63], which seeks to balance the increase in likelihood and the complexity
of the model by introducing a penalty term for each parameter. BIC is defined
by

$BIC{\left(\mathbf{X};\theta\right)}=log{p\left(\mathbf{X}|\pi,\mu,\mathrm{\Sigma}\right)}-\frac{1}{2}\eta\left(\theta\right)log{\left(N\right)}\
$ (5)

where $\eta(\theta)$ is the number of free parameters in the model that
represents the complexity of the model. GMM build ellipsoidal subclasses
(clusters), centered at the means $\mu_{k}$ and with covariance
$\mathrm{\Sigma}_{k}$. The covariance matrix can be determined by geometric
characteristics. For instance, Banfield et al. [96] develop a covariance
matrix model in terms of its eigenvalues decomposition in the term form

$\mathrm{\Sigma}_{k}=\lambda_{k}D_{k}{A_{k}D}_{k}^{T}\ $ (6)

where $D_{k}$ is the orthogonal matrix of eigenvectors and defines the
orientation, $A_{k}$ is a diagonal matrix whose elements are proportional to
the eigenvalues of $\mathrm{\Sigma}_{k}$ and determines the shape of the
density contours, and $\lambda_{k}$ is a scalar that defines the volume of the
corresponding ellipsoid. This covariance model is the generalization of
several earlier proposals based on GMM. Table 15 is taken for the `mclust`
package [97, 98] in `R` open source software, which shows names, models, and
geometric interpretation of the covariances model implemented for this
research.

Table 15: Parametrization of the covariance matrix for Gaussian models. Source [97] Model | $\mathrm{\Sigma}_{k}$ | Distribution | Volume | Shape | Orientation
---|---|---|---|---|---
EII | $\lambda I$ | Spherical | Equal | Equal | $-$
VII | $\lambda_{k}I$ | Spherical | Variable | Equal | $-$
EEI | $\lambda A$ | Diagonal | Equal | Equal | Coordinate axes
VEI | $\lambda_{k}A$ | Diagonal | Variable | Equal | Coordinate axes
EVI | $\lambda A_{k}$ | Diagonal | Equal | Variable | Coordinate axes
VVI | $\lambda_{k}A_{k}$ | Diagonal | Variable | Variable | Coordinate axes
EEE | $\lambda DAD^{T}$ | Ellipsoidal | Equal | Equal | Equal
EVE | $\lambda DA_{k}D^{T}$ | Ellipsoidal | Equal | Variable | Equal
VEE | $\lambda_{k}DAD^{T}$ | Ellipsoidal | Variable | Equal | Equal
VVE | $\lambda_{k}DA_{k}D^{T}$ | Ellipsoidal | Variable | Variable | Equal
EEV | $\lambda D_{k}{AD}_{k}^{T}$ | Ellipsoidal | Equal | Equal | Variable
VEV | $\lambda_{k}D_{k}{AD}_{k}^{T}$ | Ellipsoidal | Variable | Equal | Variable
EVV | $\lambda D_{k}{A_{k}D}_{k}^{T}$ | Ellipsoidal | Equal | Variable | Variable
VVV | $\lambda_{k}D_{k}{A_{k}D}_{k}^{T}$ | Ellipsoidal | Variable | Variable | Variable

Separation index: this index ($J^{\ast}$) was proposed by Qiu et al. [64, 65].
This index measures the magnitude of the gap between pairs of subclasses. It
has a value between $-$1 and +1, where negative values indicate subclasses are
overlapped, zero means subclasses are touching, and positive values indicate
subclasses are separated. Initially, it is necessary to find the optimal
projection in one-dimension space in which two subclasses have the maximum
separation. The initial projection is selected between two possible methods:
$\left(\mathrm{\Sigma}_{1}+\mathrm{\Sigma}_{2}\right)^{-1}\left(\mu_{2}-\mu_{1}\right)$
and $\left(\mu_{2}-\mu_{1}\right)$. Then, the Newton-Raphson method is used to
search the optimal projection. Therefore, it requires that both covariances
matrices from subclasses must be positive definite. This assumption is covered
because all subclasses follow a multivariate Gaussian distribution. In a
nutshell, the covariance matrix must be a symmetric positive definite matrix
in order to the `Equation` 7 makes sense.

Let $J_{12}^{\ast}$ be the optimal separation index between subclasses 1 and
2. Then, $J_{12}^{\ast}=J_{12}(a^{\ast})$ , where $a^{\ast}$ is the optimal
projection direction witch maximizes $J_{12}$. Finally, the Separation index
$J^{\ast}$ is estimated by using `Equation` 7.

${J(a}^{\ast})=\frac{a^{T}\left(\mu_{2}-\mu_{1}\right)-q_{\frac{\alpha}{2}}\left(\sqrt{a^{T}\mathrm{\Sigma}_{1}a}+\sqrt{a^{T}\mathrm{\Sigma}_{2}a}\right)}{a^{T}\left(\mu_{2}-\mu_{1}\right)+q_{\frac{\alpha}{2}}\left(\sqrt{a^{T}\mathrm{\Sigma}_{1}a}+\sqrt{a^{T}\mathrm{\Sigma}_{2}a}\right)}$
(7)

where $\mu_{1}$, $\mu_{2}$, $\mathrm{\Sigma}_{1}$, and $\mathrm{\Sigma}_{2}$
are the probabilistic parameters of the two subclasses, $\alpha\ \in(0,05)$ is
a tuning parameter indicating the percentage of data in the extremes to
downweigh, $q_{\alpha/2}=\ z_{\alpha/2}$, where $z_{\alpha/2}$ is the upper
$\alpha/2$ quantile of the univariate standard normal distribution.

##  Appendix B Performance Metrics

The confusion matrix is used to describe the performance of a classification
algorithm on a set of test data for which the true classes are known. Table 16
shows the confusion matrix elements for a binary classification problem.
Traditionally, the overall accuracy ($Acc$) is the most important metric to
evaluate the performance of a classifier, which can be calculated using
`Equation` 8.

Table 16: Confusion matrix for a binary classification task |  | Prediction
---|---|---
|  | Positive (P) | Negative (N)
Observations | Positive (P) | TP | FN
Negative (N) | FP | TN

$Acc=\frac{TP+TN}{TP+TN+FP+FN}\ $ (8)

$Sensitivity={Acc}^{+}=\frac{TP}{TP+FN}$ (9)

$Precision=\ \frac{TP}{TP+FP}$ (10)

$Specificity={Acc}^{-}=\frac{TN}{TN+FP}$ (11)

Sometimes the overall accuracy can be misleading [81] and models with a lower
overall accuracy could have better predictive power. For instance, in case of
imbalance data a classifier can reach high accuracy levels for the Negative
class; however, the accuracy for the Positive class is low. Therefore, the
overall accuracy has a bias toward the Negative class in imbalance data. To
solve this situation, more specific metrics are used such as sensitivity
(recall, true positive rate, or ${Acc}^{+}$), specificity (true negative rate
or ${Acc}^{-}$) and precision (positive predictive value).

The Receiver Operating Characteristic (ROC) curve is a well-accepted technique
for summarizing classifier performance [15]. The area under the ROC curve
(AUC) is a measure of how well a classifier can distinguish between two
classes [84, 99].

$AUC=\frac{1}{P\times N}\sum_{i=1}^{P}\sum_{j=1}^{N}1_{p_{i}>p_{j}}$ (12)

Where $i$ runs over all Positive instances with true class 1, and $j$ runs
over all Negative instances with true class 0; $p_{i}$ and $p_{j}$ denote the
probability score assigned by the classifier to instance $i$ and $j$,
respectively. 1 is the indicator function: it outputs 1 if, and only if, the
condition $(p_{i}>p_{j})$ is satisfied. In simple words, the ROC curve is
generated by plotting the "True Positive Rate" (TPR) against the "False
Positive Rate" (FPR). However, This AUC metric may provide an overly
optimistic performance evaluation [16]. Thereby, performance evaluation
metrics such as Precision-Recall curves, F1-score (F-measure), the geometric
mean (G-mean or G-measure), and the correlation coefficient $\phi$ are
considerate more convenient in the class imbalance context.

$TPR=\ \frac{TP}{TP+FN}$ (13)

$FPR=\ \frac{FP}{FP+TN}\ $ (14)

In Statistics, F1-score (also F-score or F-measure) uses the Precision and
Sensitivity providing a single accuracy measurement for a classifier [82].
Formally, it is the harmonic average of the Precision and Recall, which return
a value between 0 and 1. Where 1 indicates perfect Precision and Sensitivity.

$F1-score=\ \ 2\ \times\ \frac{Precision\ \times\
Sensitivity}{Precision+Sensitivity}$ (15)

The Geometric mean score (G-mean or G-measure) is the geometric mean of
Specificity and Sensitivity, which return a value between 0 and 1 [37, 100].

$G-mean=\sqrt{Sensitivity\ \times\ Specificity}=\sqrt{{Acc}^{+}\times\
{Acc}^{-}}$ (16)

In Statistics, the phi coefficient (also phi correlation coefficient or mean
square contingency coefficient) measure of the degree of association between
two binary variables, which are considered positively associated if most of
the data falls along the diagonal cells of the confusion matrix. Phi
coefficient return a value between $-$1 and 1, where values between 0.7 to 1
indicate strong association.

$\phi=\frac{TP\ \times\ TN-FP\ \times\
FN}{\sqrt{\left(TP+FP\right)\times\left(FN+TN\right)\times\left(TP+FN\right)\times\left(FP+TN\right)}}\
$ (17)

## References

  * [1] Committee on Diagnostic Error in Health Care.  Improving Diagnosis in Health Care.  2015\.
  * [2] Brett J. Bordini, Alyssa Stephany, and Robert Kliegman.  Overcoming Diagnostic Errors in Medical Practice.  Journal of Pediatrics, 2017.
  * [3] Mark L. Graber.  The incidence of diagnostic error in medicine, 2013.
  * [4] Georgios Douzas, Fernando Bacao, and Felix Last.  Improving imbalanced learning through a heuristic oversampling method based on k-means and SMOTE.  Information Sciences, 465:1–20, 2018.
  * [5] Alberto Fernández, Salvador García, Francisco Herrera, and Nitesh V. Chawla.  SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year Anniversary, 2018.
  * [6] Wei Lu, Zhe Li, and Jinghui Chu.  Adaptive Ensemble Undersampling-Boost: A novel learning framework for imbalanced data.  Journal of Systems and Software, 132:272–282, 2017.
  * [7] Ai X., Wu J., Sheng V.S., Zhao P., and Cui Z.  Immune centroids oversampling method for binary classification.  Computational intelligence and neuroscience, 2015.
  * [8] Cigdem Beyan and Robert Fisher.  Classifying imbalanced data sets using similarity based hierarchical decomposition.  Pattern Recognition, 2015.
  * [9] Vicente García, Jose Sánchez, and Ramon Mollineda.  An Empirical Study of the Behavior of Classifiers on Imbalanced and Overlapped Data Sets.  In Luis Rueda, Domingo Mery, and Josef Kittler, editors, Progress in Pattern Recognition, Image Analysis and Applications, pages 397–406, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
  * [10] Taghi M. Khoshgoftaar, Chris Seiffert, Jason Van Hulse, Amri Napolitano, and Andres Folleco.  Learning with limited minority class data.  In Proceedings - 6th International Conference on Machine Learning and Applications, ICMLA 2007, pages 348–353, 2007.
  * [11] Xu Ying Liu, Jianxin Wu, and Zhi Hua Zhou.  Exploratory under-sampling for class-imbalance learning.  In Proceedings - IEEE International Conference on Data Mining, ICDM, pages 958–962, 2006.
  * [12] Rc Prati, G Batista, and Mc Monard.  Learning with class skews and small disjuncts.  Lecture notes in computer science, 3171:296–306, 2004.
  * [13] Nitesh V. Chawla et. al.  SMOTE.  Journal of Artificial Intelligence Research, 16:321–357, 2002.
  * [14] Nitesh V Chawla, Nathalie Japkowicz, and Aleksander Kołcz.  Editorial: Special Issue on Learning from Imbalanced Data Sets.  ACM SIGKDD Explorations Newsletter, 2004.
  * [15] J A Swets.  Measuring the accuracy of diagnostic systems.  Science (New York, N.Y.), 240(4857):1285–1293, 1988.
  * [16] Jesse Davis and Mark Goadrich.  The relationship between Precision-Recall and ROC curves.  In Proceedings of the 23rd international conference on Machine learning - ICML ’06, pages 233–240, 2006.
  * [17] Nathalie Japkowicz.  The Class Imbalance Problem: Significance and Strategies.  Proceedings of the 2000 International Conference on Artificial Intelligence, pages 111—-117, 2000.
  * [18] Gary M. Weiss and Foster Provost.  Learning when training data are costly: The effect of class distribution on tree induction.  Journal of Artificial Intelligence Research, 19:315–354, 2003.
  * [19] Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz.  A multiple resampling method for learning from imbalanced data sets.  Computational Intelligence, 2004.
  * [20] Pedro Domingos.  MetaCost: A General Method for Making Classifiers Cost-Sensitive.  Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, 1999.
  * [21] Charles Elkan.  The foundations of cost-sensitive learning.  In IJCAI International Joint Conference on Artificial Intelligence, pages 973–978, 2001.
  * [22] Robert C Holte, L Acker, and BW Porter.  Concept Learning and the Problem of Small Disjuncts.  IJCAI International Joint Conference on Artificial Intelligence, pages 813–818, 1989.
  * [23] Nathalie Japkowicz.  Concept-Learning in the Presence of Between-Class and Within-Class Imbalances.  In Eleni Stroulia and Stan Matwin, editors, Advances in Artificial Intelligence: 14th Biennial Conference of the Canadian Society for Computational Studies of Intelligence, AI 2001 Ottawa, Canada, June 7–9, 2001 Proceedings, pages 67–77. Springer Berlin Heidelberg, Berlin, Heidelberg, 2001.
  * [24] Adam S Nickerson, Nathalie Japkowicz, and Evangelos Milios.  Using Unsupervised Learning to Guide Resampling in Imbalanced Data Sets.  In Proceedings of the Eighth International Workshop on AI and Statitsics, page 5, 2001.
  * [25] Taeho Jo and Nathalie Japkowicz.  Class imbalances versus small disjuncts.  ACM SIGKDD Explorations Newsletter, 6(1):40, 2004.
  * [26] Gustavo A. Valencia-Zapata, Daniel Mejia, Gerhard Klimeck, Michael G. Zentner, and Okan Ersoy.  A Statistical Approach to Increase Classification Accuracy in Supervised Learning Algorithms.  The IPSI BgD Transactions on Internet Research, 13(2):27–32, 2017\.
  * [27] R.C. Prati, G.E. Batista, and M.C. Monard.  Class imbalances versus class overlapping: an analysis of a learning system behavior.  MICAI 2004: Advances in Artificial Intelligence, 2004.
  * [28] Carla E. Brodley and Mark A. Friedl.  Identifying Mislabeled Training Data.  Journal of Artificial Intelligence Research, 1999.
  * [29] David Opitz and Richard Maclin.  Popular Ensemble Methods: An Empirical Study.  Journal of Artificial Intelligence Research, 1999.
  * [30] Robi Polikar.  Ensemble based systems in decision making, 2006.
  * [31] Lior Rokach.  Ensemble-based classifiers.  Artificial Intelligence Review, 2010.
  * [32] Larry E. Beutler, Kathleen Someah, Satoko Kimpara, and Kimberley Miller.  Selecting the most appropriate treatment for each patient.  International Journal of Clinical and Health Psychology, 16(1):99–108, 2016.
  * [33] J. Larry Jameson and Dan L. Longo.  Precision Medicine-Personalized, Problematic, and Promising.  Obstetrical & Gynecological Survey, 2015.
  * [34] Alberto Fernández, Salvador García, Mikel Galar, Ronaldo C. Prati, Bartosz Krawczyk, and Francisco Herrera.  Learning from Imbalanced Data Sets.  2018\.
  * [35] L. Breiman, J. Friedman, R. Olshen, and C. Stone.  Classification and regression trees Regression trees.  Wadsworth and Brooks, Monterey, CA, 1984.
  * [36] Mirosolav Kubat and Stan Matwin.  Addressing the Curse of Imbalanced Training Sets: One Sided Selection.  In Proceedings of the Fourteenth International Conference on Machine Learning, pages 179–186, 1997.
  * [37] Chris Drummond and R.C. Holte.  C4.5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling.  In Proceedings of the International Conference on Machine Learning (ICML 2003). Workshop on Learning from Imbalanced Data Sets II, pages 1–8, Washington, DC, 2003.
  * [38] Haibo He, Yang Bai, Edwardo A. Garcia, and Shutao Li.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning.  In Proceedings of the International Joint Conference on Neural Networks, pages 1322–1328, 2008.
  * [39] Chumphol Bunkhumpornpat, Krung Sinapiromsaran, and Chidchanok Lursinsap.  DBSMOTE: Density-based synthetic minority over-sampling technique.  Applied Intelligence, 2012.
  * [40] Nathalie Japkowicz and Shaju Stephen.  The class imbalance problem: A systematic study.  Intelligent Data Analysis, 2002.
  * [41] Gary M Weiss and Haym Hirsh.  A Quantitative Study of Small Disjuncts.  In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, pages 665–670. AAAI Press, 2000.
  * [42] Gustavo E. A. P. A. Batista, Ronaldo C. Prati, and Maria Carolina Monard.  A study of the behavior of several methods for balancing machine learning training data.  ACM SIGKDD Explorations Newsletter, 2004.
  * [43] J. R. Quinlan.  Induction of Decision Trees.  Expert Systems, 1986.
  * [44] J R Quinlan.  Learning from noisy data.  In Proceedings of the Second International Machine Learning Workshop, pages 58–64. University of Illinois at Urbana-Champaign, 1983.
  * [45] J R Quinlan.  The Effect of Noise on Concept Learning.  In Machine Learning, An Artificial Intelligence Approach Volume II, chapter 6, pages 149–166. Morgan Kaufmann, 1986.
  * [46] P E Hart.  The condensed nearest neighbor rule.  IEEE Transactions on Information Theory, 14:515–516, 1968.
  * [47] T. M. Cover and P. E. Hart.  Nearest Neighbor Pattern Classification.  IEEE Transactions on Information Theory, 1967.
  * [48] Thomas M. Cover.  Estimation by the Nearest Neighbor Rule.  IEEE Transactions on Information Theory, pages 50–55, 1968.
  * [49] Dennis L. Wilson.  Asymptotic Properties of Nearest Neighbor Rules Using Edited Data.  IEEE Transactions on Systems, Man and Cybernetics, 1972.
  * [50] Jorma Laurikkala.  Improving Identification of Difficult Small Classes by Balancing Class Distribution.  In Proceedings of the 8th Conference on AI in Medicine in Europe: Artificial Intelligence Medicine, AIME ’01, pages 63–66, Berlin, Heidelberg, 2001. Springer-Verlag.
  * [51] I Tomek.  An Experiment with the Edited Nearest-Neighbor Rule.  Systems, Man and Cybernetics, IEEE Transactions on, SMC-6(6):448–452, 1976.
  * [52] K. Chidananda Gowda and G. Krishna.  The Condensed Nearest Neighbor Rule Using the Concept of Mutual Nearest Neighborhood.  IEEE Transactions on Information Theory, 1979.
  * [53] Carla E Brodley and Mark A Friedl.  Identifying and Eliminating Mislabeled Training Instances.  Computer Engineering, 1996.
  * [54] X Ester, M., Kriegel, H. P., Sander, J., & Xu.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise.  Kdd, 1996.
  * [55] X. Ai, J. Wu, Z. Cui, X. Xian, and Y. Yao.  Enrich the data density of cluster for imbalanced learning using immune representatives.  In 2016 IEEE International Conference on the Science of Electrical Engineering, ICSEE 2016, pages 1–5, 2017.
  * [56] Daniel Boley.  Principal direction divisive partitioning.  Data Mining and Knowledge Discovery, 1998.
  * [57] James MacQueen.  Some Methods for classification and Analysis of Multivariate Observations.  In 5th Berkeley Symposium on Mathematical Statistics and Probability 1967, pages 281–297, 1967.
  * [58] Stuart P. Lloyd.  Least Squares Quantization in PCM.  IEEE Transactions on Information Theory, 1982.
  * [59] K. Pearson.  Contributions to the Mathematical Theory of Evolution.  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2006.
  * [60] D M Titterington, A F M Smith, and U E Makov.  Statistical Analysis of Finite Mixture Distributions.  Wiley, New York, 1985.
  * [61] D B Rubin A. P. Dempster N. M. Laird, A P Dempster, N M Laird, and D B Rubin.  Maximum Likelihood from Incomplete Data via the EM Algorithm.  Journal of the Royal Statistical Society. Series B (Methodological), 1977.
  * [62] Gideon Schwarz.  Estimating the dimension of a model.  The Annals of Statistics, 6(2):461–464, 1978.
  * [63] Chris Fraley and Adrian Raftery.  How Many Clusters ? Which Clustering Method ? Answers Via Model-Based Cluster Analysis.  The Computer Journal, 41(8):578–588, 1998.
  * [64] Weiliang Qiu and Harry Joe.  Separation index and partial membership for clustering.  Computational Statistics and Data Analysis, 2006.
  * [65] Weiliang Qiu and Harry Joe.  Generation of random clusters with specified degree of separation, 2006\.
  * [66] Marti J. Anderson.  Distance-based tests for homogeneity of multivariate dispersions.  Biometrics, 2006.
  * [67] Marti J. Anderson, Kari E. Ellingsen, and Brian H. McArdle.  Multivariate dispersion as a measure of beta diversity.  Ecology Letters, 2006.
  * [68] Marti. J Anderson.  PERMDISP: a FORTRAN computer program for permutational analysis of multivariate dispersions (for any two-factor ANOVA design) using permutation tests.  Technical report, Department of Statistics, University of Auckland, New Zealand., 2004.
  * [69] Morton B. Brown and Alan B. Forsythe.  Robust tests for the equality of variances.  Journal of the American Statistical Association, 1974.
  * [70] Gustavo E A P A Batista, Ronaldo C Prati, and Maria C Monard.  Balancing Strategies and Class Overlapping.  In Advances in Intelligent Data Analysis VI: 6th International Symposium on Intelligent Data Analysis, IDA 2005, Madrid, Spain, September 8-10, 2005. Proceedings, pages 24–35. 2005.
  * [71] Dheeru Dua and Casey Graff.  {UCI} Machine Learning Repository, 2017.
  * [72] Max Kuhn.  caret Package.  Journal of statistical software, 2008.
  * [73] Max Kuhn.  Building predictive models in R using the caret package.  Journal of Statistical Software, 2008.
  * [74] Max Kuhn.  A Short Introduction to the caret Package.  R Foundation for Statistical Computing, 2017.
  * [75] Max Kuhn.  caret Package: Classification and Regression Training.  https://Cran.R-Project.Org/, 2017.
  * [76] G.M. Weiss.  The Effect of Small Disjuncts and Class Distrubution on Decision Tree Learning.  Phd thesis, Rutgers University, 2003.
  * [77] Nathalie Japkowicz.  Class imbalances: are we focusing on the right issue.  In ICML Workshop on Learning from Imbalanced Data Sets, page 63, 2003.
  * [78] Gary M Weiss.  Mining with Rarity: A Unifying Framework.  SIGKDD Explorations Newsletter, 2004.
  * [79] Marcus Maloof.  Learning When Data Sets are Imbalanced and When Costs are Unequal and Unknown.  Analysis, 2003.
  * [80] Foster Provost and Tom Fawcett.  Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions.  KDD, 1997.
  * [81] F. Provost, T. Fawcett, and R. Kohavi.  The case against accuracy estimation for comparing induction algorithms.  In 15th International Conference on Machine Learning, pages 445–453, 1998.
  * [82] Haibo He and Edwardo A. Garcia.  Learning from imbalanced data.  IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284, 2009.
  * [83] GM Weiss and F Provost.  The effect of class distribution on classifier learning: an empirical study.  Technical report, Rutgers University, 2001.
  * [84] Andrew P. Bradley.  The use of the area under the ROC curve in the evaluation of machine learning algorithms.  Pattern Recognition, 30(7):1145–1159, 1997.
  * [85] Tom Fawcett.  ROC Graphs: Notes and Practical Considerations for Researchers.  HP Labs Tech Report HPL-2003-4, 2004.
  * [86] Tom Fawcett.  An introduction to ROC analysis.  Pattern Recognition Letters, 27(8):861–874, 2006.
  * [87] Takaya Saito and Marc Rehmsmeier.  The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets.  PLoS ONE, 2015.
  * [88] H. B. Mann and D. R. Whitney.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other.  The Annals of Mathematical Statistics, 1947.
  * [89] Myles Hollander and Douglas Wolfe.  Nonparametric Statistical Methods, 2nd Edition.  1999\.
  * [90] W. J. Conover.  Practical Nonparametric Statistics. New York, 3 edition, 1999.
  * [91] Milton Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance.  Journal of the American Statistical Association, 32(200):675–701, 1937.
  * [92] Milton Friedman.  A Correction: The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance.  Journal of the American Statistical Association, 34(205), 1939.
  * [93] Milton Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings.  The Annals of Mathematical Statistics, 11(1):86–92, 1940.
  * [94] R. A. Fisher.  The design of experiments.  Oliver and Boyd, London, 1935.
  * [95] Hans Peter Piepho.  An algorithm for a letter-based representation of all-pairwise comparisons.  Journal of Computational and Graphical Statistics, 2004.
  * [96] Jeffrey D. Banfield and Adrian E. Raftery.  Model-Based Gaussian and Non-Gaussian Clustering.  Biometrics, (49):803–821, 1993.
  * [97] Luca Scrucca, Michael Fop, T Brendan Murphy, and Adrian E Raftery.  mclust 5: Clustering, Classification and Density Estimation Using Gaussian Finite Mixture Models.  The R journal, 2016.
  * [98] C Fraley and AE Raftery.  MCLUST: Software for model-based cluster and discriminant analysis.  Technical report, University of Washington, Seattle, 1998.
  * [99] R O Duda, P E Hart, and D G Stork.  Pattern Classification.  2000\.
  * [100] Nathalie Japkowicz.  Assessment Metrics for Imbalanced Learning.  In Imbalanced Learning: Foundations, Algorithms, and Applications, pages 187–206. 2013.