For several years till date, the major issues in terms of solving for classification problems are the issues of Imbalanced data. Because majority of the machine learning algorithms by default assumes all data are balanced, the algorithms do not take into consideration the distribution of the data sample class. The results tend to be unsatisfactory and skewed towards the majority sample class distribution. This implies that the consequences as a result of using a model built using an Imbalanced data without handling for the Imbalance in the data could be misleading both in practice and theory. Most researchers have focused on the application of Synthetic Minority Oversampling Technique (SMOTE) and Adaptive Synthetic (ADASYN) Sampling Approach in handling data Imbalance independently in their works and have failed to better explain the algorithms behind these techniques with computed examples. This paper focuses on both synthetic oversampling techniques and manually computes synthetic data points to enhance easy comprehension of the algorithms. We analyze the application of these synthetic oversampling techniques on binary classification problems with different Imbalanced ratios and sample sizes.

§ INTRODUCTION

When it comes to solving classification problems using machine learning algorithms, the issue of Imbalanced data has become pervasive and ubiquitous. As shown in Figure <ref>, a data is said to be Imbalanced if the sample of one class is in a higher number than the other class. The class with the highest number in the data is referred to as the majority sample class and the class with the least number in the data is minority sample class. While we may assume that machine learning classifiers should be able to classify any data without any bias towards a certain sample class, this is not the case when the machine learning classifier is fed with an Imbalanced data. By default, these classifiers assume that there are even sample class distributions in the data and so tend to produce unsatisfactory results when the sample class distributions are Imbalanced. These results tend to be biased toward the majority class distribution. One could understand why the significance of this area of research continues to grow because in most cases, the focus of interest lies in the minority sample class. And if the machine learning is favoring the majority sample class, that implies we are getting the opposite result of what we expect to get. In most cases and especially in real world applications, Imbalanced data are born 'naturally". That means they are inherently in the data. Examples of the naturally inherent Imbalanced data include but not limited to people with a certain chronic condition vs. those without, people who commit bank fraud vs those who do not, employees who churn vs those who stay. The effect of modeling or predicting using an Imbalanced data can cause a greater damage than good. Take for example, in a financial institution where there is critical need to identify fraud transaction which are mostly rare. Any errors in detecting a fraudulent transaction causes a major financial blow to the company. The same applies to the medical diagnosis field where certain health conditions are rare and therefore physicians or doctors cannot afford to incorrectly diagnose a patient. The effect of such incorrect diagnosis could be extremely dangerous to the patient.Handling Imbalanced classification problems can be approached in two ways - either through allocating different costs to training examples or re-sample the original data, either by over-sampling the minority class or by under-sampling the majority class [1]. In this paper, the focus will be on re-sampling of the original data set using synthetic oversampling techniques, SMOTE (Synthetic Minority Oversampling Technique), and its extension ADASYN (Adaptive Synthetic) technique. Section 2 gives an overview of reporting measures; Accuracy, Recall, Precision, F1-Score and Area Under the  Receiver Operator Characteristic (ROC) Curve (AUC). Section 3 presents the details of our oversampling techniques, SMOTE and ADASYN. Section 4 presents data analysis results comparing the two oversampling techniques to each other using some machine learning classifiers. Section 5 concludes the paper and offer recommendations for future work.

A graphical representation of an Imbalanced Binary Data.

§ REPORTING MEASURES

In evaluating the performance of a classifier, we use the confusion matrix. A confusion matrix has 4 quadrants and each of the observation data points in a classification problem  belongs to one of the possible four outcomes as shown in Figure <ref>. As shown in Figure <ref>, the True Positive (TP) quadrant shows the samples that are positive and have been classified as Positive, the False Positive (FP) quadrant shows the samples that are negative but have been classified as positive, the True Negative (TN) quadrant shows samples that are negative and have been classified as negative and the False Negative (FN) quadrant shows samples that are positive but have classified as negative.  A simple computation involving two or more of the quadrant: True Positive , False Positive, True Negative and False Negative results in the models reporting measures. These include but not limited to Accuracy, Precision, Recall/Sensitivity, F1-score, AUC score, G-mean, Discriminant Power among other reporting measures. Since this study focused on binary-class problems, we  consider model's accuracy, precision, sensitivity/recall, F1-score and Area Under the Curve (AUC) score as our performance measures on the models.

The Confusion Matrix

It is also important to know how the reporting measures are calculated and most importantly what they mean.

* Accuracy is the most popular evaluation metric in the data community. But accuracy is a great reporting measure  when there is a balanced data set between the sample classes. And because most classification problems especially real world problems are naturally Imbalanced, relying on this measure could be misleading to solving the problem. It is therefore recommended that we consider other measurements when evaluating model performance. Accuracy can be calculated mathematically as:

\begin{equation}
Accuracy=\frac{TP + TN}{TP+ TN + FP + FN}
\end{equation}

* Precision is the ratio between True Positives observations and all Positive observations. That is, of all the instances the classifier labeled as positive, what fraction were actually correct? Also called Positive Predictive Value (PPV), it can be described as a measure of a classifiers exactness. Mathematically,

\begin{equation}
Precision = \frac{TP}{TP + FP}
\end{equation}

* Recall/ Sensitivity is a measure of a classifiers completeness. Recall can be described as the ration of the number of positive predictions and the number of positive class values, that is, the ability of the classifier to find all positive instances. Mathematically,

\begin{equation}
Recall = \frac{TP}{TP + FN}
\end{equation}

* F1-score is normally considered as the model's accuracy that takes into consideration Precision and Recall. And this is particularly very useful and not misleading compared to accuracy especially when dealing with asymmetric or imbalance data set. F1-Score takes into account the weighted harmonic mean of both the classifiers exactness (Precision) and completeness (Recall/Sensitivity). The higher the Precision and Recall, the higher F1-score will be. This measure optimizes Precision and Recall into a single value. Mathematically,

\begin{equation}
F1-Score = \frac{2*(Recall * Precision)}{Recall + Precision}
\end{equation}

* AUC (Area Under the Receiver Operator Characteristic (ROC) Curve) Score
By plotting the True Positive Rate against the False Positive Rate at various threshold levels, we create the ROC graph. AUC is simply the area under the ROC curve which shows the  likelihood that a classifier would rank a positive observation randomly chosen higher than a negative observation randomly chosen. It is best understood to be a measure of separability - the ability of the classifier to distinguish between the sample class distribution. The closer the AUC score is to 1, the better the classifier is able to distinguish between the minority and majority sample class. When faced with an Imbalanced data, this is a better measure to report than accuracy. AUC score measure can be computed as;

\begin{equation}
AUC = \frac{1 + TP - FP}{2}
\end{equation}

§ OVERSAMPLING TECHNIQUE

For many years, there has been an extended research on the use of oversampling techniques in solving class imbalance problems. The extended research on this technique can be attributed to its ability to retain the original data set while preventing the loss of important information. Some of these researches on oversampling technique were developed by notable researchers such as (Douzas & Bação, 2018), (Last, Douzas, & Bacao, 2017), (Nekooeimhr & Lai-Yuen, 2016), (Li, Fong, Wong, Mohagmmed, & Fiaidhi , 2016), (Sun, Song , Zhu, Xu, & Zhou, 2015), (Menardi & Torelli, 2014) and (Bowyer, Hall, Kegelmeyer, & Chawla, 2002). This technique creates a balanced data set by generating new samples to be added to the minority sample class. Oversampling can be done either through random oversampling where the data set is balanced through replicating the existing minority sample class or through synthetic oversampling where the data set is balanced through creating new synthetic minority samples by linear interpolation. The focus of this study will be on the application of synthetic oversampling in handling Imbalanced data because this method unlike random oversampling avoids over fitting, therefore improving the generalization ability of the classifier.

§.§ 3a. Synthetic Minority Oversampling Technique (SMOTE)

This approach, inspired by a technique that proved successful in handwritten character recognition (Ha & Bunke, 1997),  Synthetic Minority Oversampling Technique (SMOTE) was first pioneered by Chawla in 2002 [1]. In SMOTE algorithm, minority class is over sampled by generating synthetic examples rather than by oversampling with replacement for simple random oversampling. To avoid the issue of over fitting when increasing minority sample class, SMOTE creates synthetic data points by working within the current feature space. New synthetic data points  are extracted from interpolation, so the original data set still has significance. SMOTE interpolates values using a K - nearest neighboring technique for each minority class instance and generates attribute values for new data instances [6].A new synthetic data point is created for each minority sample data by taking the difference between the a minority sample class feature vector and the nearest neighbor belonging to the same sample class and multiplying it by a random number between 0 and 1 and then adding the results back to the minority sample class feature vector. This creates a random line segment between every pair of existing features. This results in a new instance generated  within the data set [2]. The cycle is replicated for the remaining minority sample data [7].One such downside stems from the fact that SMOTE arbitrarily tries to over-sample a minority instance with a uniform likelihood. Although this helps the approach to efficiently counter imbalances between the classes, the problems of disparity within the class and small disjoints are overlooked. Input areas that report multiple minority populations are highly likely to be further inflated, while sparsely populated minority areas are likely to remain sparse (Prati et al., 2004). This increases the complexity of the problem and lowers the learning classifier efficiency. Another downside is that SMOTE will intensify the noise present in the data further. This is likely to happen when a noisy minority sample, which is situated between instances of the majority class, and its nearest minority neighbor interpolates linearly. The approach is susceptible to noise generation, as it does not differentiate between overlapping class regions and so-called protected areas (Bunkhumpornpat et al., 2009). This impedes the classifier's ability to define the problems boundaries [3].

A Schematic Diagram on how to create a Synthetic data using the SMOTE Technique.

An Imbalanced class distribution (LEFT) as shown in Figure <ref> contains more blue colors than orange colors. Using SMOTE, the algorithm finds the K-nearest neighbour of a data point in the minority class (orange colors) and creates some synthetic data points on the lines joining the primary point and the neighbors as shown in Figure <ref> (middle). These new neighbors synthetic data points generated share similar characteristics of the other minority data points. Theses synthetic data points now help balance the original class distribution (right), which improves the model's generalization ability. The use of SMOTE for class distribution balancing ensures no loss of data, alleviates over fitting and the process is very easy to implement. However, SMOTE should be used with extra care when dealing with higher dimensional space. Researchers and practitioners have widely embraced SMOTE, perhaps due to its versatility and added benefit with regard to random oversampling [3]. The table below has been adopted and modified from the original research by (Chawla et al ...2002) to simplify the processes of the SMOTE algorithm.

Let $x_1$, $x_2$,...,$x_n$ be the minority class feature vectors in the n dimensional space of X

Let N be the number of synthetic instances to generate

Let K be the number of nearest neighbour

Synthetic set of artificial instances

1. For i in range (N) do,

2. Select randomly a minority class feature vector $x_i$

3. From $x_i$'s K-nearest minority class neighbors, randomly select a neighbor $\hat x_{i}$

4.  diff = $\hat x_{i}$ - $x_i$

$\delta = $random number between 0 and 1$\\
5.  $new Sample$ = $ $x_i$ + diff * $\delta   $

6.  $Synthetic \Longleftarrow new Sample$

To better appreciate how SMOTE works, this paper created an Imbalanced data with 10 data points - three(3) minority class and seven (7) majority class as shown in  Table 1. We then used this Imbalance data to create two (2) minority synthetic data points by using the SMOTE algorithm steps as shown in the table above. For the purpose of illustration, we set K (number of nearest neighbors) to be 2.

Example of Imbalanced Data

No     No     No     Yes    Yes     Yes     Yes     Yes     Yes     Yes

5     4    5     2   1   3   4   4   5   5

3     3     2    6   4   2.5   3   4.5   5   6

Let N = 2

Let K = 2

Synthetic  set of artificial instances

1. For i in range (N=1) do,

2. Select (4 3)

3. Randomly select a neighbor (5 3)

4.  diff = (5 3) - (4 3) = (1 0)

$\delta = 0.5\\
5.  $new Sample$ =  $(4 3) + [(1 0) * 0.5] $ \\
6.  Synthetic $1$ \Longleftarrow $(4.5 3)$ \\
\\~\\
$1. for i in range (N=2) do,$\\
2. $Select (5 2)$\\
3. $Randomly select a neighbor (5 3)$\\
4.  $diff = (5 3) - (5 2) = (0 1)$ \\
%\delta = 0.5\\
5.  $new Sample$ =  $(5 2) + [(0 1) * 0.5] $ \\
6.  Synthetic $2$ \Longleftarrow $(5 2.5)

end for

Example for SMOTE

As shown in Figure <ref> below, we have manually calculated 2 synthetic data points (4.5 3) and (5 2.5) by using the steps in the SMOTE algorithm. This increased our minority class from three (3) to five (5), therefore improving the imbalance ratio significantly. We then plot the new synthetic data, (4.5 3) and (5 2.5) together with the original data set as shown in Figure <ref>.

A schematic diagram of the Binary Data before and after the application of SMOTE algorithm.

§.§ 3b. Adaptive Synthetic (ADASYN) Sampling Approach

Adaptive Synthetic sampling approach is an extension or improvement of the SMOTE algorithm. The concept of Adaptive Synthetic (ADASYN) sampling approach for
imbalanced learning  was first introduced by He
(2008). ADASYN tries to generate more synthetic instances
on the region with less positive instances than one with more
positive instances to increase the recognition of positive.
This algorithm uses the number of negative neighbors in K-nearest neighbors of each positive instance to form a distribution function. The distribution function determines how many synthetic instances are generated from that positive instance [8]. For harder to learn minority samples, ADASYN generates more synthetic data points/observations as compared to minority sample class that are easier to learn. Ultimately, ADASYN is a pseudo-probabilistic algorithm in the sense that a fixed number instances is generated for each minority instance based on a weighted distribution of its neighbors [4]. Using SMOTE, each minority sample class has equal chance to be selected in the creation of the  synthetic data samples but ADASYN uses the density distribution as a criterion to automatically evaluate the number of synthetic data samples to be produced for each example of minority sample class data [5].Therefore, ADASYN approach improves data distribution learning by reducing the bias generated by the class disparity and moving the classification decision boundary to the challenging examples (He et al, 2008). Because ADASYN is very sensitive to outliers, it is advisable to deal with outliers during data preprocessing before applying ADASYN procedure. Compared to SMOTE, ADASYN put more focus on the minority samples that are difficult to learn by generating more synthetic data points for these difficult and hard to learn minority class samples [3].

Schematic diagram of ADASYN algorithm application.

As shown in Figure <ref>, the original data set has some minority class examples that are difficult for an algorithm to learn. ADASYN synthetically generates new minority samples using a density distribution based on the number of majority sample class neighbors. A minority sample class surrounded with more nearest neighbors from the majority sample class is considered hard-to-train. Therefore the algorithm will give priority to the hard-to-learn minority sample class by generating more synthetic points/observations than the less hard-to-learn minority sample class. The table below shows the pseudo algorithm for ADASYN and has been adopted from (He et al 2008) and modified.

Let m be the number of minority samples

Let n be the number of majority samples

Let $\beta $ be the ratio of the balance level of the synthetic samples. NB: $ \beta $$\in$ $(0,1]$

Let $x_i$ for i=1,2,3...m be the minority class feature vectors in the n dimensional space of X

Let G be the number of synthetic instances to generate

Let $g_i$ for i=1,2,3...m be the number of synthetic data generated for each $x_i$

Let K be the Number of Nearest Neighbour

Let  $\delta $$\in$ [0,1]

Synthetic set of artificial instances

$G = \beta \times (n-m) $

For i in range (m),

Find K for every $x_i$

Calculate    $r_i$ = $f_k$/K , where  $f_k$   is the number of feature vectors in the K nearest neighbors belonging to the majority class

Calculate    $\hat r_{i}$ =   $r_i$ / $\sum_{i=1}^{m} r_{i}$, so that $(\sum_{i=1} \hat r_{i} = 1)$

Calculate   $g_i$ =  $\hat r_{i} \times G $

For i in range ($g_i$) and for j in range ($m$), do

From $x_i$'s K-nearest minority class neighbors, randomly select a neighbor $\hat x_{i_j}$

diff = $\hat x_{i_j}$ - $x_i$

$New Sample_{i_j}$ = $x_i$ + diff * $\delta \\
$Synthetic $\Longleftarrow  New Sample_{i_j}$

We computed new synthetic minority data points using the ADASYN algorithm using the same data points used in SMOTE application. We created three (3) minority synthetic data points by using the ADASYN algorithm steps as shown in the table above. For the purpose of illustration, we set K to be 2.

Let m = 3, n = 7, K = 2, $\beta $ = 0.75 &  $\delta$ = 0.5

1. $G = (7-3) * 0.75 = 3 $

2.   ** There is only 1 majority class in each of the distinct neighbourhood

i.     $r_i$ = $1/2$ for $i=1$ to $3$

ii. $\sum_{i=1}^{3} r_{i}$=1/2 + 1/2 + 1/2 = 3/2

3. $\hat r_{i}$ = 1/2 * 2/3 = 1/3 for $i=1$ to $3$

4. $g_i$ = $1/3 * 3 = 1$ for $i=1$ to $3$

5. From (4 3) 2-nearest minority class neighbors, randomly select a neighbor (5 3)

i.  diff = (5 3) - (4 3)

ii.$New Sample_{i_j}$ = (4 3) + [(1 0) * 0.5]

iii. Synthetic $\Longleftarrow  $(4.5 3)$\\
\\
$ 6. From (5 3) 2-nearest minority class neighbors, randomly select a neighbor (5 2)$\\
i.  $diff$ = $(5 2)$ - $(5 3)$\\
ii.$$New Sample_{i_j}$ = (5 3) + [(0 -1) * 0.5]

iii. Synthetic $\Longleftarrow  $(5 2.5)$\\
\\
$ 7. From (5 2) 2-nearest minority class neighbors, randomly select a neighbor (4 3)$\\
i.  $diff$ = $(4 3)$ - $(5 2)$ \\
ii.$$New Sample_{i_j}$ = (5 2)+ [(-1 1) * 0.5]

iii. Synthetic $\Longleftarrow  $(4.5 2.5)

end for

Example for ADASYN

We manually calculated 3 synthetic data points (4.5 3), (5 2.5) and (4.5 2.5) by using the steps in the ADASYN algorithm and data points from Table 1. This increased our minority class from three (3) to six (6), therefore improving the Imbalance ratio significantly. We plot the new synthetic data points,  (4.5 3), (5 2.5) and (4.5 2.5) together with the original data set as shown in Figure <ref>.

A schematic diagram of the Binary Data before and after the application of ADASYN algorithm.

§ DATA SET SUMMARY & PRE-PROCESSING

Using real world data sets obtained from Kaggle and UCI Machine Learning Repository , we test learning capabilities of the SMOTE and ADASYN algorithm on two-class imbalance data. Table 2 shows a brief description of the data sets used in this paper.

Description of data sets

Data Set     Attributes     Sample Size     Minority     Majority

Blood Transfusion Service Center    5     748     178     570

Pima Indians Diabetes    8    768     268     500

IBM HR Analytics Employee Attrition     35    1470     237    1233

§.§ Blood Transfusion Service Center Data

This data set originates from the Blood Transfusion Service Center in Hsin-Chu City in Taiwan. The objective of this data set is to use certain measurements to predict whether a person will donate blood. The data set has 5 variables or features with 748 data examples. There are 178 minority class and 570 majority class. This data set is a two-class Imbalanced data set and hence no modification was needed.

§.§ Pima Diabetes Data

This data set is owned by the National Institute of Diabetes and Digestive and Kidney Diseases. There are 768 data examples with 8 attributes. This data set is a two class Imbalanced data set with a binary outcome, 0 (not diabetic) and 1 (diabetic) with the aim to predict using certain measurements whether a person is diabetic or not.

§.§ IBM HR Analytics Employee Attrition Data

This fictitious data set was created by IBM data scientists for the purpose of uncovering the factors that lead to employee attrition. There are 1470 data sample with 237 minority sample class and 1233 majority sample class.

All of the three different data set were checked for missing values, fortunately there were none for all of the data sets. Certain features were converted from categorical values to numerical values to boost performance measures. The outcome variables for IBM data set were re-coded from a nominal binary outcome to a numerical binary outcome. For the purpose of been able to interpret the output and to avoid information loss, we did not perform any dimensional reduction technique especially to the IBM HR Analytics Employee Attrition data set. Again, some of the classifiers used in this paper inherently deal with dimension reduction while learning the data. All of our data sets were standardized to provide a common ground for all the classifiers even though some classifiers can still perform well whether the data is standardized or not. The data sets were split into 80% train data and 20% test data. The classifiers first learned using the train data and the test data was used to measure the classifiers learning abilities.This paper made use of 4 classification classifiers to test the learning abilities of the algorithms on all of the data sets. These classifiers included Logistic Regression, Support Vector Machine (SVM), Random Forest and XGBoost.

§ EXPERIMENTAL RESULTS

Combined with SMOTE or ADASYN, 4 classifiers; Logistic Regression, Support Vector Machine (SVM), Random Forest and XGBoost were used as the learning models in our experiment. Without applying SMOTE or ADASYN algorithms, we created baseline models using the original data sets and provided the reporting measures using each of the classifiers mentioned. The reported measures were used to illustrate the performances of these learning techniques and we compared the learning ability of the classifiers before and after performing SMOTE and ADASYN. The following tables below contain the experimental results. The best classifier for each of the three applications ( baseline, SMOTE and ADASYN) and the optimal reported measures in terms of F1-Score and AUC Score are highlighted with bold text. F1-Score and the AUC Score are given much importance because these measures are less leading and provide a more accurate evaluation of the model performance especially when the data is Imbalanced compared to accuracy.The baseline models for all of the three data sets reported high accuracy greater than 70% when applied to any of the 4 classifiers used in this paper. Although the baseline models reported high accuracy, the harmonic mean which is the F1-Score were relatively low. The reported F1-Score for the Pima Diabetes Data (Table <ref>) were much higher than the same score for the Blood Transfusion Service Center Data (Table <ref>) and the IBM HR Analytics Employee Attrition Data (Table <ref>) for the baseline models. Again, the AUC Score compared to to accuracy were also much lower for the baseline models. AUC Score for the Pima Diabetes Data were higher than the other two data sets. And this could be as a result of the differences in data Imbalance among other issues such as sample size, and type of classifier used. While the minority sample class in the Pima Diabetes Data accounted for 35% of the data, IBM HR Analytics Attrition Data and the Blood Transfusion Service Center Data minority sample class accounted for 24% and 16% respectively in the data.There were significant improvement in model performances after handling for the data Imbalance through the application of SMOTE and ADASYN. As shown in Table <ref>, <ref>, <ref>, all of the reported measures showed higher scores compared with the base models for each of the classifier combination. The improvement in the scores confirmed that indeed machine learning classifiers are less sensitive to sample class distribution in a data. Most importantly , the ability of the classifiers to distinguish between majority and minority sample class was improved in all of the data sets after the application of SMOTE and ADASYN

Blood Transfusion Service Center Data

6|c|Original Data Set

Classifier     Accuracy   Precision   Recall   F1-score   AUC

Logistic Regression     0.73       0.57    0.10   0.20    0.54

SVM      0.74     0.67     0.10      0.20   0.54

Random Forest      0.71    0.45     0.32    0.37     0.59

XGBoost    0.75    0.58   0.37   0.45   0.63

Logistic Regression      0.75       0.74    0.79   0.77    0.75

SVM      0.76     0.79     0.74      0.76   0.76

Random Forest      0.80    0.82     0.78    0.80     0.80

XGBoost    0.80    0.81   0.81   0.81    0.80

Logistic Regression      0.70       0.75    0.71   0.73    0.70

SVM      0.66     0.72     0.63      0.67   0.66

Random Forest      0.74    0.78     0.75    0.77     0.74

XGBoost    0.72    0.75   0.74   0.75   0.71

Pima Diabetes Data

6|c|Original data set

Classifier     Accuracy   Precision   Recall   F1-score   AUC

Logistic Regression      0.82       0.76    0.62   0.68   0.77

SVM      0.79     0.70     0.55     0.62     0.73

Random Forest      0.81     0.71     0.64      0.67     0.76

XGBoost    0.82    0.70   0.70   0.70    0.79

Logistic Regression      0.81       0.78    0.82   0.80   0.81

SVM      0.85     0.83     0.87     0.85     0.86

Random Forest      0.85     0.83     0.87     0.85     0.86

XGBoost    0.85    0.82   0.88    0.85   0.85

Logistic Regression      0.72       0.73   0.71   0.72   0.72

SVM      0.82    0.79     0.87      0.83   0.82

Random Forest      0.85     0.82     0.90     0.86     0.85

XGBoost    0.82   0.78   0.89    0.83   0.82

IBM HR Analytics Employee Attrition Data

6|c|Original Data Set

Classifier     Accuracy   Precision   Recall   F1-score   AUC

Logistic Regression      0.89       0.83    0.41   0.55   0.70

SVM      0.86   0.90    0.20     0.31   0.59

Random Forest      0.84     0.71    0.10    0.18     0.55

XGBoost    0.87    0.75   0.31    0.43   0.64

Logistic Regression      0.87       0.90    0.85   0.88   0.87

SVM      0.88    0.93     0.84      0.88   0.88

Random Forest      0.91     0.93     0.90     0.91     0.91

XGBoost    0.89    0.92   0.86    0.89   0.89

Logistic Regression      0.87       0.89    0.84   0.86    0.87

SVM      0.89     0.91     0.86     0.88   0.88

Random Forest      0.91    0.92     0.90 0.91    0.91

XGBoost    0.89    0.91   0.86   0.88   0.88

Figure <ref>, <ref> & <ref> shows the ROC graph for the baseline models and the models after the application of the synthetic oversampling techniques. As seen in Figure <ref>, <ref> & <ref>, there were improvement in the classifiers ability to distinguish between the minority and majority classes after applying SMOTE or ADASYN to the Imbalanced data. Random Forest and XGBoost when combined with SMOTE or ADASYN performed better than Logistic Regression and Support Vector Machine. As shown in Figure <ref>, the combination of SMOTE and the classifiers outperformed ADASYN. Random Forest or XGBoost when combined with SMOTE yielded an AUC score of 0.80. That means that 80% of the time the model is able to distinguish between a blood donor from a non-blood donor. With the Pima Diabetes Data, a combination of the SMOTE technique and the Random Forest classifier (Figure <ref>) gave the optimal AUC Score of 0.86 compared with the baseline model of 0.76. This implies that the model is able to distinguish between a patient who is diabetic and a patient who is not 86% of the time as compared to 76% for the baseline model. Random Forest classifier on the IBM HR Analytics Employee Attrition Data  reported an AUC score of 91% after applying both SMOTE and ADASYN (Figure <ref>).In general, we observed in this paper that, the reporting measures scores were better after dealing with data Imbalance using SMOTE or ADASYN. However, choosing the best combination of sampling technique and classifier are essential to handling the problem of Imbalanced data in the best possible way.

Blood Transfusion Service Center Data

Pima Diabetes Data

IBM HR Analytics Employee Attrition Data

§ CONCLUSION

In this paper, we focused on handling Imbalanced data  with particular focus on two re-sampling methodologies - SMOTE and ADASYN. We also focused on explaining the algorithms behind this techniques with computed examples. We discussed that, compared to SMOTE, ADASYN put more focus on the minority samples that are difficult to learn by generating more synthetic data points for these difficult and hard to learn minority sample class. We also discussed that while SMOTE considers a  uniform weight in generating new synthetic data for all minority points, ADASYN considers a density distribution in deciding the number of synthetic samples to be generated for a particular minority data point. We also from the experimental study realized how the reporting measure. accuracy could be misleading and proposed like many other researchers to rely on other metrics in conjunction with accuracy when reporting. While both SMOTE and ADASYN when applied to the Imbalanced data, resulted in an improvement in the reporting measures, SMOTE generally reported higher reporting measure scores than ADASYN. However, there are not enough evidence to generalize based on this paper that, SMOTE performs better than ADASYN in handling Imbalanced data. Factors not limited to the type of classifier, parameter tuning and the type of Imbalanced data can definitely affect the performances of SMOTE and ADASYN.For future research, we will focus on how the combination of the oversampling techniques- SMOTE or ADASYN and classifiers perform on a test data that has different Imbalance sample class ratios from the train data set.

[1]
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip
Smote: synthetic minority over-sampling technique.
In Journal Of Artificial Intelligence Research, Volume 16, pages 321-357, 2002.

[2]
S. Deepa and V. Bharathi.
Textural Feature Extraction and
Classification of Mammogram Images using CCCM and
In IOSR Journal of Computer Engineering (IOSR-JCE),vol. 10, no. 6, pp. 07-13, June 2013.

[3]
Felix Last,Georgios Douzas,Fernando Bacao
Oversampling for Imbalanced Learning Based on K-Means and SMOTE.
Information Sciences. arXiv:1711.00837:465 (2018) 1-20

[4]
Haibo He, Yang Bai, E. A. Garcia and Shutao Li.
ADASYN: Adaptive synthetic sampling approach for imbalanced learning.
2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong., pp. 1322-1328,2008.

[5]
Huang, Peng Jun.
Classification of Imbalanced Data Using
Synthetic Over-Sampling Techniques.
University of California, Los Angeles, ProQuest Dissertations Publishing.,1590106, 2015.

[6]
S. Maheshwari, J. Agrawal, and S. Sharma.
New approach for classification of highly imbalanced datasets using evolutionary algorithms.
Int. J.
Sci. Eng. Res.,  vol. 2, no. 7, pp. 1–5, 2011.

[7]
K. P. Satyasree and J. Murthy.
An exhaustive literature review on class
imbalance problem.
Int. J. Emerg. Trends Technol. Comput. Sci.,  vol. 2,
pp. 109–118, May 2013.

[8]
Wacharasak Siriseriwan, Krung Sinapiromsaran.
Adaptive neighbor synthetic minority oversampling technique
under 1NN outcast handling.
Songklanakarin J. Sci. Technol., 39 (5), 565-576, Sep - Oct. 2017.


Supposed that a $p$-order adaptive aPC needs to be modeled for the stochastic
model $Y=f\left(\bm{\xi}\right)$ with the random input variable
$\bm{\xi}=\left\\{{{\xi}_{k}}\left|k=1,2,\cdots,d\right.\right\\}$. Therefore,
the number $Q$ of order combinations of all single variables ${\xi}_{k}$ is

$Q={{\left(p+1\right)}^{d}}.$ (52)

For $q=1,2,\cdots,Q$, the $k\text{th}$ index $\tilde{s}_{q}^{k}$ of the
multivariate index set $\tilde{s}_{q}$ is determined by

$\tilde{s}_{q}^{k}=\left\\{\begin{aligned}
&\left(q-1\right)\bmod\left(p+1\right)\qquad\qquad\quad k=d\\\
&\left\lfloor\frac{q-1}{{{\left(p+1\right)}^{d-k}}}\right\rfloor\bmod\left(p+1\right)\qquad
k=1,2,\cdots,(d-1),\\\ \end{aligned}\right.$ (53)

where $x\bmod y$ returns the remainder after division of $x$ by $y$, and
$\left\lfloor x\right\rfloor$ rounds $x$ to the nearest integer less than or
equal to $x$. Thus, the multivariate index set $\tilde{s}_{q}$ can be
obtained, i.e.

${{\tilde{s}}_{q}}=\left\\{\tilde{s}_{q}^{1},\tilde{s}_{q}^{2},\cdots,\tilde{s}_{q}^{k},\cdots,\tilde{s}_{q}^{d}\right\\}.$
(54)

Refer to Eq.(7), the sum of multivariate index $\tilde{s}_{q}^{k}$ in
$\tilde{s}_{q}$ should less than or equal to $p$, i.e.

$\sum\limits_{k=1}^{d}{\tilde{s}_{q}^{k}}\leq p.$ (55)

Therefore, only $M$ multivariate index sets meet the above conditions.
Supposed that $M$ multivariate index sets are
$\left\\{{{s}_{i}}|i=1,2,\cdots,M\right\\}$, where
${{s}_{i}}=\left\\{s_{i}^{k}|k=1,2,\cdots,d\right\\}$.

## References

  * Lee et al. [2020] Lee, T., Bilionis, I., Tepole, A.B..  Propagation of uncertainty in the mechanical and biological response of growing tissues using multi-fidelity gaussian process regression.  Computer Methods in Applied Mechanics and Engineering 2020;359:112724.  doi:10.1016/j.cma.2019.112724.
  * Liu et al. [2021] Liu, X., Yu, X., Tong, J., Wang, X., Wang, X..  Mixed uncertainty analysis for dynamic reliability of mechanical structures considering residual strength.  Reliability Engineering and System Safety 2021;209:107472.  doi:10.1016/j.ress.2021.107472.
  * Yao et al. [2011] Yao, W., Chen, X., Luo, W., van Tooren, M., Guo, J..  Review of uncertainty-based multidisciplinary design optimization methods for aerospace vehicles.  Progress in Aerospace Sciences 2011;47(6):450–479.  doi:10.1016/j.paerosci.2011.05.001.
  * Yao et al. [2013] Yao, W., Chen, X., Huang, Y., Gurdal, Z., van Tooren, M..  Sequential optimization and mixed uncertainty analysis method for reliability-based optimization.  AIAA Journal 2013;51(9):2266–2277.  doi:10.2514/1.J052327.
  * Sun and You [2021] Sun, L., You, F..  Machine learning and data-driven techniques for the control of smart power generation systems: An uncertainty handling perspective.  Engineering 2021;7(9):1239–1247.  doi:10.1016/j.eng.2021.04.020.
  * Xu et al. [2020] Xu, J., Zhang, Y., Dang, C..  A novel hybrid cubature formula with pearson system for efficient moment-based uncertainty propagation analysis.  Mechanical Systems and Signal Processing 2020;140.  doi:10.1016/j.ymssp.2020.106661.
  * Zhang and Xu [2021] Zhang, Y., Xu, J..  Efficient reliability analysis with a cda-based dimension-reduction model and polynomial chaos expansion.  Computer Methods in Applied Mechanics and Engineering 2021;373.  doi:10.1016/j.cma.2020.113467.
  * Hong et al. [2021] Hong, Z., Jiang, X., Feng, Y., Tian, Q., Tan, J..  Reliability topology optimization of collaborative design for complex products under uncertainties based on the tlbo algorithm.  Engineering 2021;doi:10.1016/j.eng.2021.06.027.
  * Zheng et al. [2019] Zheng, X., Yao, W., Xu, Y., Chen, X..  Improved compression inference algorithm for reliability analysis of complex multistate satellite system based on multilevel bayesian network.  Reliability Engineering and System Safety 2019;189:123–142.  doi:10.1016/j.ress.2019.04.011.
  * Zheng et al. [2020] Zheng, X., Yao, W., Xu, Y., Chen, X..  Algorithms for bayesian network modeling and reliability inference of complex multistate systems: Part i – independent systems.  Reliability Engineering and System Safety 2020;202.  doi:10.1016/j.ress.2020.107011.
  * Fan et al. [2018] Fan, Y., Huang, G., Zhang, Y., Li, Y..  Uncertainty quantification for multivariate eco-hydrological risk in the xiangxi river within the three gorges reservoir area in china.  Engineering 2018;4(5):617–626.  doi:10.1016/j.eng.2018.06.006.
  * Rubinstein and Kroese [2016] Rubinstein, R.Y., Kroese, D.P..  Simulation and the Monte Carlo Method.  Wiley Series in Probability and Statistics; 2016.  doi:10.1002/9781118631980.
  * Xiu and Karniadakis [2003] Xiu, D.B., Karniadakis, G.E..  Modeling uncertainty in flow simulations via generalized polynomial chaos.  Journal of Computational Physics 2003;187(1):137–167.  doi:10.1016/s0021-9991(03)00092-5.
  * Yao et al. [2019] Yao, W., Tang, G., Wang, N., Chen, X..  An improved reliability analysis approach based on combined form and beta-spherical importance sampling in critical region.  Structural and Multidisciplinary Optimization 2019;60(1):35–58.  doi:10.1007/s00158-019-02193-y.
  * Oladyshkin and Nowak [2012] Oladyshkin, S., Nowak, W..  Data-driven uncertainty quantification using the arbitrary polynomial chaos expansion.  Reliability Engineering and System Safety 2012;106:179–190.  doi:10.1016/j.ress.2012.05.002.
  * Wan et al. [2020] Wan, H., Ren, W., Todd, M.D..  Arbitrary polynomial chaos expansion method for uncertainty quantification and global sensitivity analysis in structural dynamics.  Mechanical Systems and Signal Processing 2020;142.  doi:10.1016/j.ymssp.2020.106732.
  * Tripathy and Bilionis [2018] Tripathy, R.K., Bilionis, I..  Deep uq: Learning deep neural network surrogate models for high dimensional uncertainty quantification.  Journal of Computational Physics 2018;375:565–588.  doi:10.1016/j.jcp.2018.08.036.
  * Zhang et al. [2019] Zhang, D., Lu, L., Guo, L., Karniadakis, G.E..  Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems.  Journal of Computational Physics 2019;397:108850.  doi:10.1016/j.jcp.2019.07.048.
  * Wong Felix [1985] Wong Felix, S..  Slope reliability and response surface method.  Journal of Geotechnical Engineering 1985;111(1):32–53.  doi:10.1061/(ASCE)0733-9410(1985)111:1(32).
  * Roussouly et al. [2013] Roussouly, N., Petitjean, F., Salaun, M..  A new adaptive response surface method for reliability analysis.  Probabilistic Engineering Mechanics 2013;32:103–115.  doi:10.1016/j.probengmech.2012.10.001.
  * Kaymaz [2005] Kaymaz, I..  Application of kriging method to structural reliability problems.  Structural Safety 2005;27(2):133–151.  doi:10.1016/j.strusafe.2004.09.001.
  * Echard et al. [2011] Echard, B., Gayton, N., Lemaire, M..  Ak-mcs: An active learning reliability method combining kriging and monte carlo simulation.  Structural Safety 2011;33(2):145–154.  doi:10.1016/j.strusafe.2011.01.002.
  * Li et al. [2006] Li, H.s., Lü, Z.z., Yue, Z.f..  Support vector machine for structural reliability analysis.  Applied Mathematics and Mechanics 2006;27(10):1295–1303.  doi:10.1007/s10483-006-1001-z.
  * Bourinet et al. [2011] Bourinet, J.M., Deheeger, F., Lemaire, M..  Assessing small failure probabilities by combined subset simulation and support vector machines.  Structural Safety 2011;33(6):343–353.  doi:10.1016/j.strusafe.2011.06.001.
  * Wiener [1938] Wiener, N..  The homogeneous chaos.  American Journal of Mathematics 1938;6.
  * Ghanem and Spanos [1993] Ghanem, R., Spanos, P.D..  A stochastic galerkin expansion for nonlinear random vibration analysis.  Probabilistic Engineering Mathematics 1993;8(3-4):255–264.  doi:10.1016/0266-8920(93)90019-R.
  * Matthies and Keese [2005] Matthies, H.G., Keese, A..  Galerkin methods for linear and nonlinear elliptic stochastic partial differential equations.  Computer Methods in Applied Mechanics and Engineering 2005;194(12-16):1295–1331.  doi:10.1016/j.cma.2004.05.027.
  * Isukapalli et al. [1998] Isukapalli, S.S., Roy, A., Georgopoulos, P.G..  Stochastic response surface methods (srsms) for uncertainty propagation: application to environmental and biological systems.  Risk analysis : an official publication of the Society for Risk Analysis 1998;18(3):351–363.  doi:10.1111/j.1539-6924.1998.tb01301.x.
  * Li and Zhang [2007] Li, H., Zhang, D..  Probabilistic collocation method for flow in porous media: Comparisons with other stochastic methods.  Water Resources Research 2007;43(9):n/a–n/a.  doi:10.1029/2006WR005673.
  * Shi et al. [2009] Shi, L., Yang, J., Zhang, D., Li, H..  Probabilistic collocation method for unconfined flow in heterogeneous media.  Journal of Hydrology 2009;365(1-2):4–10.  doi:10.1016/j.jhydrol.2008.11.012.
  * Pan and Dias [2017] Pan, Q., Dias, D..  Sliced inverse regression-based sparse polynomial chaos expansions for reliability analysis in high dimensions.  Reliability Engineering and System Safety 2017;167:484–493.  doi:10.1016/j.ress.2017.06.026.
  * Blatman and Sudret [2010] Blatman, G., Sudret, B..  An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis.  Probabilistic Engineering Mechanics 2010;25(2):183–197.  doi:10.1016/j.probengmech.2009.10.003.
  * Blatman and Sudret [2011] Blatman, G., Sudret, B..  Adaptive sparse polynomial chaos expansion based on least angle regression.  Journal of Computational Physics 2011;230(6):2345–2367.  doi:10.1016/j.jcp.2010.12.021.
  * Schmidhuber [2015] Schmidhuber, J..  Deep learning in neural networks: An overview.  Neural Networks 2015;61:85–117.  doi:10.1016/j.neunet.2014.09.003.
  * Goodfellow et al. [2016] Goodfellow, I., Bengio, Y., Courville, A..  Deep Learning.  The MIT Press; 2016.
  * Hornik et al. [1989] Hornik, K., Stinchcombe, M., White, H..  Multilayer feedforward networks are universal approximators.  Neural Networks 1989;2(5):359–366.  doi:10.1016/0893-6080(89)90020-8.
  * Cybenko [1989] Cybenko, G..  Approximation by superpositions of a sigmoidal function.  Mathematics of Control, Signals and Systems 1989;2:303–314.
  * Shahane et al. [2019] Shahane, S., Aluru, N.R., Vanka, S.P..  Uncertainty quantification in three dimensional natural convection using polynomial chaos expansion and deep neural networks.  International Journal of Heat and Mass Transfer 2019;139:613–631.  doi:10.1016/j.ijheatmasstransfer.2019.05.014.
  * Czél et al. [2014] Czél, B., Woodbury, K.A., Gróf, G..  Simultaneous estimation of temperature-dependent volumetric heat capacity and thermal conductivity functions via neural networks.  International Journal of Heat and Mass Transfer 2014;68:1–13.  doi:10.1016/j.ijheatmasstransfer.2013.09.010.
  * Lye et al. [2021] Lye, K.O., Mishra, S., Ray, D., Chandrashekar, P..  Iterative surrogate model optimization (ismo): An active learning algorithm for pde constrained optimization with deep neural networks.  Computer Methods in Applied Mechanics and Engineering 2021;374.  doi:10.1016/j.cma.2020.113575.
  * Zhang et al. [2021] Zhang, X., Xie, F., Ji, T., Zhu, Z., Zheng, Y..  Multi-fidelity deep neural network surrogate model for aerodynamic shape optimization.  Computer Methods in Applied Mechanics and Engineering 2021;373.  doi:10.1016/j.cma.2020.113485.
  * Raissi et al. [2019] Raissi, M., Perdikaris, P., Karniadakis, G.E..  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational Physics 2019;378:686–707.  doi:10.1016/j.jcp.2018.10.045.
  * Zheng et al. [2022] Zheng, X., Yao, w., Gong, Z., Zhang, Y., Zhang, x..  Physics-informed deep reversible regression model for temperature field reconstruction of heat-source systems 2022;URL: arXiv:2202.06860v1.
  * Gong et al. [2021] Gong, Z., Zhou, W., Zhang, J., Peng, W., Yao, W..  Physics-informed deep reversible regression model for temperature field reconstruction of heat-source systems 2021;URL: arXiv:2106.11929v2.
  * Xu et al. [2015] Xu, B., Wang, N., Chen, T., Li, M..  Empirical evaluation of rectified activations in convolution network.  ArXiv Preprint 2015;arXiv: 1505.00853v2.
  * Kingma and Ba [2015] Kingma, D.P., Ba, J..  Adam: A method for stochastic optimization.  In: International Conference on Learning Representations. 2015,.
  * Duchi et al. [2011] Duchi, J., Hazan, E., Singer, Y..  Adaptive subgradient methods for online learning and stochastic optimization.  Journal of Machine Learning Research 2011;12:2121–2159.
  * Ruder [2017] Ruder, S..  An overview of gradient descent optimization algorithms.  ArXiv preprint 2017;arXiv: 1609.04747v2.
  * Mircea [2002] Mircea, G..  Stochastic Calculus: Applications in Science and Engineering.  Boston: Birkhauser; 2002.
  * Meecham and Siegel [1964] Meecham, W.C., Siegel, A..  Wiener-hermite expansion in model turbulence at large reynolds numbers.  Physics of Fluids 1964;7(8):1178–1190.
  * Siegel [2004] Siegel, A..  Wiener‐hermite expansion in model turbulence in the late decay stage.  Journal of Mathematical Physics 2004;6(5):707–721.
  * Choi et al. [2004] Choi, S.K., Grandhi, R.V., Canfield, R.A..  Structural reliability under non-gaussian stochastic behavior.  Computers and Structures 2004;82(13-14):1113–1121.  doi:10.1016/j.compstruc.2004.03.015.
  * Baydin et al. [2015] Baydin, A.G., Pearlmutter, B.A., Radul, A.A..  Automatic differentiation in machine learning: a survey.  ArXiv preprint 2015;arXiv: 1502.05767.
  * Parzen [1962] Parzen, E..  On estimation of a probability density function and mode.  Annals of Mathematical Statistics 1962;33(3):1065–1076.  doi:10.1214/aoms/1177704472.
  * Chen [2017] Chen, Y.C..  A tutorial on kernel density estimation and recent advances.  Biostatistics and Epidemiology 2017;1(1):161–187.  doi:10.1080/24709360.2017.1396742.
  * Nair and Hinton [2010] Nair, V., Hinton, G..  Rectified linear units improve restricted boltzmann machines vinod nair.  In: Proceedings of the 27th International Conference on Machine Learning. 2010, p. 1–8.
  * Sun et al. [2014] Sun, Y., Wang, X., Tang, X..  Deeply learned face representations are sparse, selective, and robust.  ArXiv preprint 2014;arXiv: 1412.1265.
  * Hendrycks and Gimpel [2016] Hendrycks, D., Gimpel, K..  Gaussian error linear units (gelus).  arXiv preprint arXiv:160608415 2016;.
  * Pedregosa et al. [2011] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 2011;12:2825–2830.
  * Singh et al. [2007] Singh, P.K., Jain, P.K., Jain, S.C..  A genetic algorithm-based solution to optimal tolerance synthesis of mechanical assemblies with alternative manufacturing processes: focus on complex tolerancing problems.  International Journal of Production Research 2007;42(24):5185–5215.  doi:10.1080/00207540410001733931.
  * Lee and Chen [2009] Lee, S.H., Chen, W..  A comparative study of uncertainty propagation methods for black-box-type problems.  Structural and Multidisciplinary Optimization 2009;37(3):239–253.  doi:10.1007/s00158-008-0234-7.
  * Lee [2019] Lee, S..  Reliability based design optimization using response surface augmented moment method.  Journal of Mechanical Science and Technology 2019;33(4):1751–1759.  doi:10.1007/s12206-019-0327-9.
  * Fan et al. [2016] Fan, W., Wei, J., Ang, A.H.S., Li, Z..  Adaptive estimation of statistical moments of the responses of random systems.  Probabilistic Engineering Mechanics 2016;43:50–67.  doi:10.1016/j.probengmech.2015.10.005.
  * Rackwitz [2001] Rackwitz, R..  Reliability analysis—a review and some perspectives.  Structural Safety 2001;23(4):365–395.  doi:10.1016/S0167-4730(02)00009-7.
  * Meng et al. [2020] Meng, X., Liu, J., Cao, L., Yu, Z., Yang, D..  A general frame for uncertainty propagation under multimodally distributed random variables.  Computer Methods in Applied Mechanics and Engineering 2020;367.  doi:10.1016/j.cma.2020.113109.