A comparative study on machine learning models combining with outlier
detection and balanced sampling methods for credit scoring

Hongyi Qiana (qianhongyi@buaa.edu.cn), Shen Zhangb
(zhangshen95726@buaa.edu.cn), Baohui Wangb (wangbh@buaa.edu.cn), Lei Pengc
(penglei@chamc.com.cn), Songfeng Gaoc (gaosongfeng@chamc.com.cn), You Songb,a
<EMAIL_ADDRESS>
a School of Computer Science and Engineering, Beihang University, Beijing
100191, PR China
b School of Software, Beihang University, Beijing 100191, PR China
c HuaRong RongTong (Beijing) Technology Co., Ltd, Beijing 100033, PR China

Corresponding Author:
You Song
School of Computer Science and Engineering, Beihang University, Beijing
100191, PR China
School of Software, Beihang University, Beijing 100191, PR China
Email<EMAIL_ADDRESS>
###### Abstract

Peer-to-peer (P2P) lending platforms have grown rapidly over the past decade
as the network infrastructure has improved and the demand for personal lending
has grown. Such platforms allow users to create peer-to-peer lending
relationships without the help of traditional financial institutions.
Assessing the borrowers’ credit is crucial to reduce the default rate and
benign development of P2P platforms. Building a personal credit scoring
machine learning model can effectively predict whether users will repay loans
on the P2P platform. And the handling of data outliers and sample imbalance
problems can affect the final effect of machine learning models. There have
been some studies on balanced sampling methods, but the effect of outlier
detection methods and their combination with balanced sampling methods on the
effectiveness of machine learning models has not been fully studied. In this
paper, the influence of using different outlier detection methods and balanced
sampling methods on commonly used machine learning models is investigated.
Experiments on 44,487 Lending Club samples show that proper outlier detection
can improve the effectiveness of the machine learning model, and the balanced
sampling method only has a good effect on a few machine learning models, such
as MLP.

###### keywords:

Credit scoring , Comparative study , Outlier detection , Balanced sampling ,
Machine learning

††journal: Expert Systems with Applications

##  1 Introduction

Personal credit is one of the main businesses of digital financial services,
and peer-to-peer (P2P) lending is the main battlefield of personal credit
business. As intermediary institutions, P2P lending platforms connect
borrowers and lenders, saving the cumbersome process of traditional financial
institutions and reducing transaction costs [Zhao et al., 2017]. This type of
lending usually involves smaller amounts but comes with a higher risk of
default. Since the outbreak of COVID-19 in late 2019, the global P2P lending
market has had to face more and more default loans problems, which puts higher
demand on lenders to review personal loan applications.

To reduce the loss of loan default, some scholars proposed methods of
predicting the default probability of personal loans, which is also called
credit scoring [Crook et al., 2007, Dastile et al., 2020, Lessmann et al.,
2015]. Specifically, credit scoring comprehensively examines the various
indicators of the borrower and conducts a comprehensive assessment of their
ability to fulfill economic commitments, which is usually modeled as a binary
problem. For example, Lending Club is the largest online loan marketplace and
borrowers can easily access lower interest rate loans through a fast online
interface. When building a credit scoring model using the Lending Club
dataset, the borrowers’ default status is represented by the “loan_status”
indicator. The “Fully Paid” status represents that the applicant has fully
paid the loan which is good, while the “Charged Off” status represents that
the applicant has not paid the installments in due time for a long period of
time which is bad.

Table 1: Previous experiences on lending club.

Year |  Study | Sample size |  Methods |  Best result with…
---|---|---|---|---
2015 |  [Malekipirbazari & Aksakalli, 2015] | 68,000 |  LR, RF, KNN, SVM |  RF
2017 |  [Kim & Cho, 2017] | 332,844 |  DT, Label propagation, TSVM |  DT+Label propagation+TSVM
2017 |  [Xia et al., 2017] | 49,795 |  LR, RF, CSXGBoost |  CSXGBoost
2018 |  [Ma et al., 2018] | 480,018 |  XGBoost, LightGBM |  LightGBM
2018 |  [Namvar et al., 2018] | 66,376 |  LR, LDA, RF |  LR
2018 |  [Xia et al., 2018] | 2,642 |  LR, GPC, SVM, DT, Bag-SVM, Bag-GPC, RF, XGBoost, Bstacking |  Bstacking
2018 |  [He et al., 2018] | 95,633 |  LR, RF, LDA, SVM, DT, KNN, GBDT, AdaBoost, XGBoost |  XGBoost
2019 |  [Ruyu et al., 2019] | 2,022,302 |  LR, Naive Bayes, DT, SVM |  DT
2019 |  [Bastani et al., 2019] | 96,200 |  RF, SVM, GBDT, WDP |  WDP
2020 |  [Melo Junior et al., 2020] | 23,677 |  LR, RF, BRF, MLP, SVM, SVM, XGBoost, KNU, RMkNN |  BRF+KNU+RMkNN
2020 |  [Niu et al., 2020] | 366,466 |  LR, RF, DT, AdaBoost, Bagging, XGBoost, DTE-ROS, DTE-SMOTE, DTE-RUS, DTE-UnderBagging, DTE-SBD, DTE-SBC, REMDD |  REMDD
2020 |  [Xia et al., 2020] | 64,139 |  LR, RF, DT, MLP, GBDT, XGBoost, CatBoost |  CatBoost
2020 |  [Song et al., 2020] | 70,860 |  LR, RF, DT, MLP, GBDT, MV–ACME, AdaBoost, DM–ACME |  DM–ACME
2020 |  [Teply & Polena, 2020] | 212,280 |  LR , MLP, LDA, SVM, RF, B-Net, Naive Bayes, DT, KNN |  LR
2021 |  [Liu et al., 2021] | 2,642 |  LR, RF, DT, SVM, AdaBoost, AugBoost, GBDT, RP, MLP, Mg-GBDT |  Mg-GBDT
2021 |  [Lee et al., 2021] | 44,092 |  SVM, RF, XGBoost, MLP, GCN |  GCN
2021 |  [Engelmann & Lessmann, 2021] | 42,506 |  LR, RF, GBDT, KNN, DT |  LR
2021 |  [Moscato et al., 2021] | 877,956 |  LR, RF, MLP |  RF

  * 1.

TSVM is the transductive SVM; CSXGBoost is the cost-sensitive extreme gradient
boosting model; GPC is the Gaussian process classifier; Bag-SVM is the bagging
+ SVM model; Bag-GPC is the bagging + GPC model; Bstacking is the bagging +
stacking model; WDP is the wide and deep learning; DTE is the decision tree
ensemble model; SBD is a combination of SMOTE and bagging with differentiated
sampling rates; SBC is an under-sampling based on clustering; REMDD is a
resampling ensemble model based on data distribution; DM–ACME is a distance-
to-model and adaptive clustering-based multi-view ensemble learning method;
MV–ACME is DM-ACME with ensemble strategies composed of hard probability and
majority voting; BRF is the balanced random forest; KNU is the k-Nearest
Oracles-Union; RMkNN is the reduced minority kNN; B-Net is the bayesian
network; RP is the random projection; Mg-GBDT is the multi-grained augmented
gradient boosting decision trees. AugBoost if the unsupervised feature
augmentation Boosting. GCN is the graph convolutional network.

As one of the most commonly used datasets in the field of credit scoring, many
scholars have done their researches on the Lending club dataset. Some of these
results are shown in table 1. It should be noted that although the table shows
the optimal results of these studies, they are not completely equivalent in
comparison due to different starting conditions. However, an observable trend
is that ensemble learning methods and hybrid models often achieve better
results than single classifiers [Liu et al., 2021, Xia et al., 2018].
Recently, some studies have applied deep learning methods to reflect nonlinear
relationships between borrower’s attributes and default risk. [Bastani et al.,
2019] and [Lee et al., 2021] applied wide and deep learning (WDP)-based and
graph convolutional network (GCN)-based credit default prediction model to P2P
lending respectively.

Similar to most personal credit scoring datasets, the Lending Club dataset
faces the class imbalance problem. The actual default of customers is only a
minority, which will affect the construction of credit scoring model. Many
balanced sampling methods have been applied to solve the problem [Moscato et
al., 2021, Namvar et al., 2018]. Commonly used methods include simple random
sampling methods such as random undersampling (RUS) and random oversampling
(ROS), and more complex oversampling methods such as synthetic minority
oversampling technique (SMOTE). Recently, researchers are also making other
attempts to reduce the impact of sample imbalance while preserving the
information of the majority class. [Melo Junior et al., 2020] used a novel
approach defining the local regions of a dynamic selection technique to deal
with the imbalanced nature of the credit scoring dataset. [Engelmann &
Lessmann, 2021] applied a conditional Wasserstein GAN to construct synthetic
data, which achieves better results than traditional sampling methods.

Besides the problem of the unbalanced dataset, when using real-world data to
build a credit scoring model, outliers also affect the robustness of the
model. And the outlier detection algorithm can detect noise or abnormal values
and reduce their impact on the model. There have been some attempts at outlier
detection studies on other datasets of credit scoring [Wei et al., 2019, Xia,
2019, Zhang et al., 2021]. However, the existing Lending Club researches only
focus on the balanced sampling method, while the research on outlier detection
method is still in a blank state.

This article aims to build a framework for credit scoring that covering a
variety of outlier detection and balanced sampling methods and models, find
the applicable conditions of different methods, and find their optimal
combination. To the best of our knowledge, this is the first comparative study
that combines outlier detection, imbalanced data processing, and machine
learning models on the Lending club dataset.

The rest of the paper is organized as follows. Section 2 introduces common
outlier detection method, balanced sampling method, and machine learning
models and their application in the field of credit scoring. Section 3
presents the proposed credit scoring framework. Section 4 shows the
experimental settings, including dataset description and evaluation criteria.
The experimental results are analyzed in Section 5. Section 6 presents the
conclusion and discusses the direction of future work.

##  2 Related work

The proposed framework includes three technologies: outlier detection method,
balanced sampling method, and machine learning model. This section will review
the representative work in each area.

###  2.1 Outlier detection method

Real-world datasets usually contain a small number of outliers, because
natural or man-made reasons such as small probability events occur or input
errors in the data acquisition system, and these outliers will affect the
performance of the machine learning model. In response to this problem,
scholars have proposed a series of outlier detection algorithms to identify
the outliers in the datasets. Both local outliers factor (LOF) [Breunig et
al., 2000] and isolation forest (IForest) [Liu et al., 2008] are the classical
outlier detection methods. LOF judges whether an instance is an outlier by
comparing the density of each point and its neighboring points. A low-density
point is more likely to be recognized as an outlier. However, [Liu et al.,
2008] pointed out that density is not always a good measure of outlier
detection, because the density of a group of outliers is often very high,
while the density of the edges of inliers points is low. IForest is ensembled
by multiple decision trees, divides the hyperplane, and calculates the number
of hyperplanes that isolate a sample. The number of hyperplanes that isolate
an outlier in the low-density space is less than inliers. [He et al., 2003]
improved LOF and proposed a cluster-based LOF outlier detection method
(CBLOF). For the latest outlier detection method, [Li et al., 2020] proposed a
copula-based outlier detection method (COPOD) inspired by copulas for modeling
multivariate data distribution. COPOD first constructs an empirical copula and
then uses it to predict tail probabilities of each given data point to
determine its level of extremeness, which makes it an ideal choice for high
dimensional datasets.

Outlier detection algorithms have also been applied in credit scoring. [Zhang
et al., 2020] proposed a bagging-based local outlier factor algorithm to
identify the outliers and subsequently boost them back into the training set
to form the outlier-adapted training set to enhance the outlier adaptability
of base classifiers. In another of their studies, [Zhang et al., 2021]
proposed a new voting-based outlier detection method to enhance the classic
outlier detection algorithms by integrating the outlier scores through the
weighted voting mechanism, and boost the outlier scores into the training set
to form an outlier-adapted training set. Outlier detection is also used to
reject inference in credit scoring, employing the rejected data can mitigate
the sample bias caused by building the model only with the accepted
applicants. For example, [Xia, 2019] applies IForest to find “good” applicants
whose loan applications are rejected due to accidental factors.

###  2.2 Balanced sampling method

Real-world datasets often face the class imbalance problem, the default
instances in personal credit scoring datasets usually are a minority class.
When the ratio of the majority sample to the minority sample exceeds 3:1, the
prediction results of the classifiers will be skewed towards the majority
samples. Using the balanced sampling method, a dataset with balanced positive
and negative samples can be obtained to eliminate the bias in the training of
the classifier.

The existing balanced sampling methods can be divided into three categories:

  * 1.

Under-sampling method: selecting samples from the original set to make the
number of positive and negative instances equal. For example, random
undersampling (RUS) is a fast and easy way to balance the data by randomly
selecting a subset of data for the targeted classes. [Wilson, 1972] proposed a
method called Edited nearest neighbor (ENN) which applies a nearest-neighbors
algorithm and edits the dataset by removing samples that do not agree enough
with their neighborhood. In addition, instance hardness threshold (IHT) [Smith
et al., 2014] s a specific algorithm in which a classifier is trained on the
data and the samples with lower probabilities are removed.

  * 2.

Over-sampling method: generating new samples in the classes which are under-
represented. Random oversampling (ROS) is the most naive strategy to generate
new samples by randomly sampling with the replacement of the currently
available samples. Apart from the ROS, there are two popular methods to over-
sample minority classes: the Synthetic Minority Oversampling Technique (SMOTE)
[Chawla et al., 2002] and the Adaptive Synthetic (ADASYN) [He et al., 2008]
sampling method. SMOTE and ADASYN generate new samples by interpolation, and
the samples used to interpolate new synthetic samples differ. Specifically,
ADASYN focuses on generating samples next to the original samples which are
wrongly classified using a k-Nearest Neighbors classifier while the basic
implementation of SMOTE will not make any distinction between easy and hard
samples to be classified using the nearest neighbors rule.

  * 3.

Hybrid method: combining with over-sampling and under-sampling method. SMOTE
can generate noisy samples by interpolating new points between marginal
outliers and inliers. This issue can be solved by cleaning the space resulting
from over-sampling. In this regard, SMOTETomek [Batista et al., 2004] and
SMOTEENN [Batista et al., 2003] added Tomek’s link [Tomek, 1976] and ENN to
the pipeline after applying SMOTE over-sampling to obtain a cleaner space
respectively.

The balanced sampling method is widely used in the credit scoring field.
[Moscato et al., 2021] and [Namvar et al., 2018] compared the performance of
various balanced sampling methods on the Lending Club dataset. In their
research, RUS achieved the best results among all sampling methods. However,
neither of them added the no sampling method to the experiments as a
comparison. In addition to the usual methods, there have been other attempts.
[Engelmann & Lessmann, 2021] proposed an approach based on a conditional
Wasserstein GAN that can effectively model tabular datasets with numerical and
categorical variables and pays special attention to the downstream
classification task through an auxiliary classifier loss.

###  2.3 Machine learning model

Machine learning is dedicated to the study of how to use the experience to
improve the performance of the system itself by computing data and is an
effective method to solve complex real-world problems. During the years of its
development, many machine learning models have emerged. For linear models,
logistic regression (LR) is the most commonly used algorithm and is often used
as a benchmark model for model performance comparison [Ala’Raj & Abbod, 2016].
For datasets with fewer data dimensions (e.g. Australia, Germany, Japan, etc.
credit scoring datasets), it can effectively mine the interrelationships
between credit data variables [Abelian & Castellano, 2017, Crook et al., 2007,
Zhang et al., 2019]. However, its effectiveness is limited when dealing with
high dimension, sparse data. Another commonly used credit scoring benchmark
method is the multilayer perceptron (MLP) [Garcia et al., 2019, Hájek, 2011,
Moscato et al., 2021]. MLP is composed of an input layer, hidden layers, and
an output layer, and a backpropagation algorithm is used to learn model
parameters. MLP is susceptible to outliers and imbalance sample problems.

With the development of machine learning, ensemble learning models have
gradually become a popular research direction. Unlike traditional learning
methods for training one classifier, ensemble learning methods usually consist
of multiple base classifiers. [Hansen & Salamon, 1990] and [Schapire, 1990]
pointed out that the ensemble of a set of classifiers often predicts more
accurately than the best individual classifier. The representative ideas of
ensemble learning are Bagging [Breiman, 1996] and Boosting [Schapire, 1999].
Random forest (RF) [Breiman, 2001] is one of the representatives of Bagging,
which is an ensemble by multiple decision trees, and each decision tree acts
as a weak learner. Gradient boosting decision tree (GBDT) is one of the
popular boosting algorithms [Friedman, 2001]. GBDT uses an additive model
which is a linear combination of basis functions, and continuously reduces the
residuals generated during the training process.

The ensemble tree models are usually more robust to outliers and sample
imbalance problems. In credit scoring, [Malekipirbazari & Aksakalli, 2015]
proposed an RF-based classification method for predicting borrower status.
Compared with other algorithms such as LR, SVM, and KNN, RF obtains the best
results. Recently, more and more state-of-art solutions are built based on
GBDT. [Liu et al., 2021] proposed a step-wise multi-grained augmented gradient
boosting decision trees (mg-GBDT) for credit scoring. This method adopts
multi-granularity scanning for feature enhancement, which enriches the input
features of GBDT.

##  3 The proposed methodology

In this paper, a systematic study combining multiple outlier detection,
balanced sampling methods, and machine learning models is proposed. The
experimental process is shown in Figure 1. The process can be divided into
four steps: data preprocessing, outlier detection, balanced sampling method,
model training, and testing. Data preprocessing includes missing value filling
and standardization. Then, samples containing outliers in the dataset were
removed by an outlier detection algorithm. After that, the balanced sampling
method is used to balance the number of positive and negative samples.
Finally, the machine learning model is trained and tested on the processed
dataset. The entire process is discussed in detail below.

Figure 1: Flowchart of the proposed methodology.

Step 1. Data pre-processing

Filling missing values and standardization.

Some numeric variables in the original dataset have missing values, which need
to be preprocessed. These missing values are filled in by the mean of that
indicator across all samples. In addition, since different variables usually
have different value ranges, the value range of each indicator needs to be
standardized to ensure the validity of the credit scoring model. In this
paper, each variable is normalized according to formula 1:

$x^{{}^{\prime}}=\frac{x-u}{s}$ (1)

where u is the mean of the samples, and s is the standard deviation of the
samples. Specifically, the scikit-learn111https://scikit-learn.org library is
used to fill in missing values and normalize the data.

Step 2. Outlier detection

Calculate the anomaly degree of each sample and remove samples with high
anomaly degrees from the training data according to the set threshold.

There are always some outlier samples in real datasets, which can mislead the
training of machine learning models, especially linear models such as LR. In
this paper, four outlier detection algorithms are used to deal with outliers,
namely LOF, CBLOF, IForest, and COPOD. Specifically, the algorithms imported
from pyod222https://github.com/yzhao062/pyod library are used to evaluate the
degree of outliers of the samples, and default parameters are used for each
outlier detection algorithm. The samples with the top 0.5% of the outlier
scores obtained by the algorithms were removed from the training data.

Step 3. Balanced sampling method

Construct a positive and negative sample-balanced dataset using the balanced
sampling method.

Credit scoring datasets often face the problem of sample imbalance, which can
make the classification model heavily biased. For example, MLP calculates the
loss of each sample and then back-propagates the updated weights, and the
sample imbalance can easily lead to the MLP model whose weight coefficients
”serve” the large-class samples and have less ability to discriminate among
the small-class samples.

Therefore, the balanced sampling method is used in this paper to build
positive and negative sample balanced datasets, as shown in table 2. A total
of eight sampling methods in three categories are used to solve the sample
imbalance problem, among which RUS, IHT, and ENN belong to under-sampling
methods, ROS, SMOTE, and ADASYN belong to over-sampling methods, SMOTEENN and
SMOTETomek belong to hybrid methods. Specifically, the imbalanced-
learn333https://imbalanced-learn.org/ library is used for the positive and
negative sample balancing process of the samples, and default parameters are
used for each algorithm.

Table 2: Sampling methods. Sampling type | Sampling methodology | Definition
---|---|---
Under-sampling method | RUS | Random Under Sampler
| IHT | Instance Hardness Threshold
| ENN | Edited Nearest Neighbours
Over-sampling method | ROS | Random Over Sampler
| SMOTE | Synthetic Minority Oversampling Technique
| ADASYN | Adaptive Synthetic
Hybrid method | SMOTEENN | SMOTE + ENN
| SMOTETomek | SMOTE + Tomek Links

Step 4. Model training and testing

Train the model and compare the predicted result with the real value.

Experiments were conducted using a variety of machine learning models,
including LR, MLP, RF, and GBDT. Specifically, the machine learning model is
imported from the scikit-learn library, and the 5-fold cross-validation
training process is used. Each time 80% of the data is selected as the
training set, and the remaining 20% is used as the test set. The
hyperparameters of each model are shown in table 3.

Table 3: Parameters of different models. Classifiers | Parameters | Values
---|---|---
LR | penalty | l2
| tol | 1e-4
| solver | lbfgs
MLP | hidden_layer_sizes | (100, )
| activation | relu
| learning_rate_init | 0.01
| early_stopping | TRUE
RF | n_estimators | 100
| max_depth | 4
GBDT | n_estimators | 100
| learning_rate | 0.1
| max_depth | 4

##  4 Experimental setup

This section describes the experimental setup in detail, consisting of two
parts, dataset description, and evaluation criteria.

###  4.1 Dataset description

The dataset used in this paper is from Lending
Club444https://www.lendingclub.com/ in Q2 2017. Lending Club is a US P2P
lending company, headquartered in San Francisco, California. It was the first
P2P lender to offer loan trading on a secondary market and is the world’s
largest P2P lending platform. The original dataset contains a total of 105,451
samples and 151 features, with “loan_status” as the target variable. As shown
in table 4, “loan_status” has 7 states, “Current”, “Fully Paid”, “Charged
Off”, “Late (31-120 days)”, “In Grace Period”, “Late (16-30 days)”, and
“Default”. Referring to the practice in previous papers [Moscato et al., 2021,
Namvar et al., 2018], only the samples with “loan_status” status as “Charged
off” and “Fully Paid” are taken as positive and negative samples respectively.
This results in an unbalanced dataset containing 44,487 samples with a
positive sample ratio of 23.80%.

Table 4: Dataset characterization. Loan status | Amount
---|---
Current | 58201
Fully Paid | 33901
Charged Off | 10586
Late (31-120 days) | 1782
In Grace Period | 650
Late (16-30 days) | 326
Default | 5
Total | 105451

Variables with a high proportion of missing values in the original features
were not used. Of the remaining variables, only those that would be available
to an investor before deciding to fund the loan were chosen. As a result,
except the target variable “loan_status”, 24 variables are actually used. The
descriptions of each field were shown in table 5.

Table 5: Indicator descriptions. LoanStatNew | Description
---|---
addr_state | The state provided by the borrower in the loan application.
annual_inc | The self-reported annual income provided by the borrower during registration.
application_type | Indicates whether the loan is an individual application or a joint application with two co-borrowers.
dti | Borrower’s total monthly debt payments divided by the borrower’s self-reported monthly income.
earliest_cr_line | The month the borrower’s earliest reported credit line was opened.
emp_length | Employment length in years.
fico_range_high | The upper boundary range the borrower’s FICO at loan origination belongs to.
fico_range_low | The lower boundary range the borrower’s FICO at loan origination belongs to.
home_ownership | The home ownership status provided by the borrower during registration.
initial_list_status | The initial listing status of the loan.
installment | The monthly payment owed by the borrower if the loan originates.
int_rate | Interest Rate on the loan.
loan_amnt | The listed amount of the loan applied for by the borrower.
loan_status | Current status of the loan.
mort_acc | Number of mortgage accounts.
open_acc | The number of open credit lines in the borrower’s credit file.
pub_rec | Number of derogatory public records.
pub_rec_bankruptcies | Number of public record bankruptcies.
purpose | A category provided by the borrower for the loan request.
revol_bal | Total credit revolving balance.
revol_util | Revolving line utilization rate, or the amount of credit relative to all available revolving credit.
sub_grade | LC assigned loan subgrade.
term | The number of payments on the loan. Values are in months and can be either 36 or 60.
total_acc | The total number of credit lines currently in the borrower’s credit file.
verification_status | Indicates if income was verified by LC, not verified, or if the income source was verified.

Some variables are preprocessed before input to the model:

  * 1.

Models such as LR cannot directly deal with class-type variables. The category
variables “sub_grade”, “home_ownership”, “verification_status”, “purpose”,
“addr_state”, “initial_list_status”, and “application_type” are coded as one-
hot;

  * 2.

The original credit score “fico_score” provides the values “low” and “high”,
which are replaced by their average value;

  * 3.

Log transformations are performed on exponential numerical variables such as
“annual_inc” and “revol_bal”;

After the above processing, the actual number of variables input to the model
is 123. Table 6 shows the statistical characteristics of the numerical
variables.

Table 6: Indicators statistics. variable name | count | mean | std | min | 25% | 50% | 75% | max
---|---|---|---|---|---|---|---|---
loan_amnt | 44487 | 14147.24 | 9343.50 | 1000.00 | 7000.00 | 12000.00 | 20000.00 | 40000.00
term | 44487 | 41.59 | 10.15 | 36.00 | 36.00 | 36.00 | 36.00 | 60.00
int_rate | 44487 | 13.90 | 5.58 | 5.32 | 10.42 | 12.74 | 16.02 | 30.99
installment | 44487 | 436.24 | 287.29 | 30.12 | 225.65 | 356.78 | 583.89 | 1719.83
emp_length | 41614 | 5.88 | 3.76 | 0.00 | 2.00 | 6.00 | 10.00 | 10.00
dti | 44456 | 18.68 | 13.78 | 0.00 | 11.95 | 17.87 | 24.34 | 999.00
earliest_cr_line | 44487 | 2000.65 | 7.59 | 1957.00 | 1997.00 | 2002.00 | 2006.00 | 2014.00
open_acc | 44487 | 11.85 | 5.74 | 0.00 | 8.00 | 11.00 | 15.00 | 88.00
pub_rec | 44487 | 0.28 | 0.66 | 0.00 | 0.00 | 0.00 | 0.00 | 22.00
revol_util | 44455 | 46.75 | 24.53 | 0.00 | 27.70 | 45.90 | 65.10 | 138.90
total_acc | 44487 | 24.48 | 12.11 | 2.00 | 16.00 | 23.00 | 31.00 | 146.00
mort_acc | 44487 | 1.51 | 1.80 | 0.00 | 0.00 | 1.00 | 2.00 | 18.00
pub_rec_bankruptcies | 44487 | 0.16 | 0.41 | 0.00 | 0.00 | 0.00 | 0.00 | 6.00
log_annual_inc | 44487 | 4.83 | 0.27 | 0.00 | 4.68 | 4.83 | 4.98 | 6.75
fico_score | 44487 | 699.73 | 33.49 | 662.00 | 672.00 | 692.00 | 717.00 | 847.50
log_revol_bal | 44487 | 3.96 | 0.55 | 0.00 | 3.73 | 4.02 | 4.27 | 6.02

###  4.2 Evaluation criteria

Comparing the effectiveness of models requires the use of appropriate
evaluation criteria. In the field of credit scoring, overall accuracy (ACC) is
one of the most commonly used indicators, which is defined as the ratio of the
number of correctly classified samples to the total number of samples. In
addition, TNR, TPR, FPR, and G-mean were also popular evaluation indicators.
These indicators can be calculated according to formulas 2 \- 6. The
probability threshold for classifying the test set is set to exactly the
extent that its positive and negative sample ratios remain consistent with the
original dataset.

$ACC=\frac{TP+FN}{TP+FP+FN+TN}$ (2) $TNR=\frac{TN}{TN+FP}$ (3)
$TPR=\frac{TP}{TP+FN}$ (4) $FPR=\frac{FP}{FP+TN}$ (5) $G-mean=\sqrt{TNR*TPR}$
(6)

The confusion matrix consisting of TP, TN, FP, and FN in these formulas is
shown in table 7, where TP and TN represent the numbers of correctly
classified normal and default users, respectively. FP and FN denote the
numbers of misclassified normal and default users, respectively.

Table 7: The confusion matrix. Actual Class | Prediction Results |
---|---|---
|  | Positive class (Default)
---
| Negative class (Normal)
---
| Positive class (Default)
---
TP | FN
| Negative class (Normal)
---
FP | TN

However, the above metrics are strongly influenced by the probability
threshold, and the comprehensive performance of the classifier can be well
evaluated by using the area under the curve (AUC). AUC is an evaluation metric
based on the receiver operating characteristic (ROC) curve, which is equal to
the area under the ROC curve and well reflects the trade-off between the true
positive rate and the false positive rate of the prediction model under
different probability thresholds. The AUC will be used as the main evaluation
metric.

##  5 Experimental results and analysis

The performance of classifiers was evaluated according to different outlier
detection and sampling strategies

###  5.1 The original results of each model

Table 8 shows the classification results of LR. In general, the difference
brought by using different outlier detection and balanced sampling methods is
not significant, the difference between the first and the bottom of the AUC
ranking is only 0.0064. Among the outlier detection and balanced sampling
methods, the most effective are LOF and None (AUC 0.7079), and the most
ineffective are None and ADASYN (AUC 0.7015). The most effective and least
effective criteria are the maximum and minimum AUC that can be achieved using
the strategy, as well as for other models. However, under any other criteria
like G-mean, the best strategy to match LR is IForest and SMOTE.

Table 8: Classification results (LR). Model | reject_strategy | sampling_strategy | ACC | AUC | G-mean | TPR | TNR | FPR
---|---|---|---|---|---|---|---|---
LR | None | None | 0.7298 | 0.7071 | 0.5893 | 0.4203 | 0.8264 | 0.1736
LR | None | RUS | 0.7294 | 0.7066 | 0.5887 | 0.4196 | 0.8262 | 0.1738
LR | None | IHT | 0.7279 | 0.7031 | 0.5861 | 0.4163 | 0.8252 | 0.1748
LR | None | ENN | 0.7282 | 0.7040 | 0.5867 | 0.4170 | 0.8254 | 0.1746
LR | None | ROS | 0.7291 | 0.7072 | 0.5882 | 0.4189 | 0.8260 | 0.1740
LR | None | SMOTE | 0.7292 | 0.7040 | 0.5883 | 0.4190 | 0.8260 | 0.1740
LR | None | ADASYN | 0.7271 | 0.7015 | 0.5848 | 0.4147 | 0.8247 | 0.1753
LR | None | SMOTEENN | 0.7272 | 0.7015 | 0.5849 | 0.4148 | 0.8247 | 0.1753
LR | None | SMOTETomek | 0.7296 | 0.7041 | 0.5891 | 0.4200 | 0.8263 | 0.1737
LR | COPOD | None | 0.7297 | 0.7077 | 0.5893 | 0.4202 | 0.8264 | 0.1736
LR | COPOD | RUS | 0.7300 | 0.7071 | 0.5897 | 0.4208 | 0.8265 | 0.1735
LR | COPOD | IHT | 0.7279 | 0.7026 | 0.5862 | 0.4164 | 0.8252 | 0.1748
LR | COPOD | ENN | 0.7278 | 0.7050 | 0.5860 | 0.4162 | 0.8251 | 0.1749
LR | COPOD | ROS | 0.7294 | 0.7077 | 0.5887 | 0.4195 | 0.8261 | 0.1739
LR | COPOD | SMOTE | 0.7300 | 0.7042 | 0.5896 | 0.4207 | 0.8265 | 0.1735
LR | COPOD | ADASYN | 0.7272 | 0.7013 | 0.5849 | 0.4148 | 0.8247 | 0.1753
LR | COPOD | SMOTEENN | 0.7274 | 0.7025 | 0.5853 | 0.4154 | 0.8249 | 0.1751
LR | COPOD | SMOTETomek | 0.7296 | 0.7042 | 0.5890 | 0.4198 | 0.8263 | 0.1738
LR | IForest | None | 0.7303 | 0.7074 | 0.5902 | 0.4214 | 0.8267 | 0.1733
LR | IForest | RUS | 0.7283 | 0.7067 | 0.5867 | 0.4171 | 0.8254 | 0.1746
LR | IForest | IHT | 0.7280 | 0.7030 | 0.5863 | 0.4165 | 0.8252 | 0.1748
LR | IForest | ENN | 0.7282 | 0.7047 | 0.5867 | 0.4170 | 0.8254 | 0.1746
LR | IForest | ROS | 0.7293 | 0.7074 | 0.5886 | 0.4194 | 0.8261 | 0.1739
LR | IForest | SMOTE | 0.7306 | 0.7046 | 0.5907 | 0.4220 | 0.8269 | 0.1731
LR | IForest | ADASYN | 0.7278 | 0.7017 | 0.5859 | 0.4161 | 0.8251 | 0.1749
LR | IForest | SMOTEENN | 0.7273 | 0.7027 | 0.5851 | 0.4151 | 0.8248 | 0.1752
LR | IForest | SMOTETomek | 0.7302 | 0.7046 | 0.5901 | 0.4213 | 0.8267 | 0.1733
LR | CBLOF | None | 0.7298 | 0.7073 | 0.5894 | 0.4204 | 0.8264 | 0.1736
LR | CBLOF | RUS | 0.7287 | 0.7069 | 0.5876 | 0.4181 | 0.8257 | 0.1743
LR | CBLOF | IHT | 0.7288 | 0.7028 | 0.5876 | 0.4182 | 0.8257 | 0.1743
LR | CBLOF | ENN | 0.7283 | 0.7050 | 0.5867 | 0.4171 | 0.8254 | 0.1746
LR | CBLOF | ROS | 0.7296 | 0.7072 | 0.5890 | 0.4199 | 0.8263 | 0.1737
LR | CBLOF | SMOTE | 0.7288 | 0.7043 | 0.5877 | 0.4183 | 0.8258 | 0.1742
LR | CBLOF | ADASYN | 0.7265 | 0.7016 | 0.5838 | 0.4135 | 0.8243 | 0.1757
LR | CBLOF | SMOTEENN | 0.7276 | 0.7027 | 0.5856 | 0.4158 | 0.8250 | 0.1750
LR | CBLOF | SMOTETomek | 0.7287 | 0.7043 | 0.5875 | 0.4180 | 0.8257 | 0.1743
LR | LOF | None | 0.7299 | 0.7079 | 0.5895 | 0.4205 | 0.8265 | 0.1735
LR | LOF | RUS | 0.7298 | 0.7064 | 0.5894 | 0.4204 | 0.8264 | 0.1736
LR | LOF | IHT | 0.7269 | 0.7023 | 0.5844 | 0.4143 | 0.8245 | 0.1755
LR | LOF | ENN | 0.7281 | 0.7051 | 0.5864 | 0.4167 | 0.8253 | 0.1747
LR | LOF | ROS | 0.7303 | 0.7076 | 0.5903 | 0.4214 | 0.8268 | 0.1732
LR | LOF | SMOTE | 0.7298 | 0.7039 | 0.5894 | 0.4204 | 0.8264 | 0.1736
LR | LOF | ADASYN | 0.7272 | 0.7017 | 0.5850 | 0.4149 | 0.8247 | 0.1753
LR | LOF | SMOTEENN | 0.7277 | 0.7024 | 0.5858 | 0.4160 | 0.8250 | 0.1750
LR | LOF | SMOTETomek | 0.7296 | 0.7039 | 0.5890 | 0.4198 | 0.8263 | 0.1738

Table 9 shows the classification results of MLP, the use of different outlier
detection and balanced sampling methods brought the greatest differences, with
the difference between the first and the last ranked AUC being 0.0879. This
suggests that the MLP is very sensitive to data outliers and sample
imbalances. Among the outlier detection and balanced sampling methods, the
most effective were LOF and IHT (AUC 0.7041), slightly below the LR. And the
least effective were None and ADASYN (AUC 0.6162), which is the same as LR.
Under any other criteria like G-mean, the best strategy to match MLP is CBLOF
and IHT.

Table 9: Classification results (MLP). Model | reject_strategy | sampling_strategy | ACC | AUC | G-mean | TPR | TNR | FPR
---|---|---|---|---|---|---|---|---
MLP | None | None | 0.7230 | 0.6953 | 0.5778 | 0.4061 | 0.8220 | 0.1780
MLP | None | RUS | 0.7225 | 0.6936 | 0.5768 | 0.4049 | 0.8216 | 0.1784
MLP | None | IHT | 0.7282 | 0.7014 | 0.5866 | 0.4169 | 0.8253 | 0.1747
MLP | None | ENN | 0.7131 | 0.6730 | 0.5606 | 0.3854 | 0.8155 | 0.1845
MLP | None | ROS | 0.6938 | 0.6267 | 0.5260 | 0.3447 | 0.8028 | 0.1972
MLP | None | SMOTE | 0.6908 | 0.6201 | 0.5204 | 0.3383 | 0.8008 | 0.1992
MLP | None | ADASYN | 0.6880 | 0.6162 | 0.5154 | 0.3325 | 0.7990 | 0.2010
MLP | None | SMOTEENN | 0.7075 | 0.6553 | 0.5506 | 0.3735 | 0.8118 | 0.1882
MLP | None | SMOTETomek | 0.6928 | 0.6245 | 0.5242 | 0.3426 | 0.8021 | 0.1979
MLP | COPOD | None | 0.7246 | 0.6979 | 0.5804 | 0.4094 | 0.8230 | 0.1770
MLP | COPOD | RUS | 0.7240 | 0.6944 | 0.5794 | 0.4081 | 0.8226 | 0.1774
MLP | COPOD | IHT | 0.7278 | 0.7029 | 0.5860 | 0.4162 | 0.8251 | 0.1749
MLP | COPOD | ENN | 0.7157 | 0.6774 | 0.5650 | 0.3907 | 0.8172 | 0.1828
MLP | COPOD | ROS | 0.6904 | 0.6257 | 0.5198 | 0.3376 | 0.8006 | 0.1994
MLP | COPOD | SMOTE | 0.6901 | 0.6176 | 0.5192 | 0.3369 | 0.8004 | 0.1996
MLP | COPOD | ADASYN | 0.6913 | 0.6227 | 0.5216 | 0.3395 | 0.8012 | 0.1988
MLP | COPOD | SMOTEENN | 0.7077 | 0.6532 | 0.5509 | 0.3738 | 0.8119 | 0.1881
MLP | COPOD | SMOTETomek | 0.6886 | 0.6240 | 0.5166 | 0.3339 | 0.7994 | 0.2006
MLP | IForest | None | 0.7212 | 0.6917 | 0.5746 | 0.4023 | 0.8208 | 0.1792
MLP | IForest | RUS | 0.7229 | 0.6960 | 0.5775 | 0.4058 | 0.8219 | 0.1781
MLP | IForest | IHT | 0.7271 | 0.7028 | 0.5847 | 0.4146 | 0.8246 | 0.1754
MLP | IForest | ENN | 0.7157 | 0.6775 | 0.5650 | 0.3907 | 0.8172 | 0.1828
MLP | IForest | ROS | 0.6904 | 0.6273 | 0.5199 | 0.3376 | 0.8006 | 0.1994
MLP | IForest | SMOTE | 0.6936 | 0.6281 | 0.5257 | 0.3444 | 0.8027 | 0.1973
MLP | IForest | ADASYN | 0.6917 | 0.6242 | 0.5221 | 0.3403 | 0.8014 | 0.1986
MLP | IForest | SMOTEENN | 0.7055 | 0.6537 | 0.5471 | 0.3693 | 0.8105 | 0.1895
MLP | IForest | SMOTETomek | 0.6945 | 0.6285 | 0.5273 | 0.3462 | 0.8032 | 0.1968
MLP | CBLOF | None | 0.7256 | 0.6980 | 0.5822 | 0.4115 | 0.8237 | 0.1763
MLP | CBLOF | RUS | 0.7223 | 0.6952 | 0.5765 | 0.4046 | 0.8215 | 0.1785
MLP | CBLOF | IHT | 0.7293 | 0.7035 | 0.5886 | 0.4194 | 0.8261 | 0.1739
MLP | CBLOF | ENN | 0.7114 | 0.6655 | 0.5574 | 0.3816 | 0.8143 | 0.1857
MLP | CBLOF | ROS | 0.6916 | 0.6302 | 0.5220 | 0.3401 | 0.8014 | 0.1986
MLP | CBLOF | SMOTE | 0.6939 | 0.6280 | 0.5261 | 0.3448 | 0.8028 | 0.1972
MLP | CBLOF | ADASYN | 0.6869 | 0.6184 | 0.5133 | 0.3302 | 0.7983 | 0.2017
MLP | CBLOF | SMOTEENN | 0.7072 | 0.6566 | 0.5500 | 0.3728 | 0.8116 | 0.1884
MLP | CBLOF | SMOTETomek | 0.6918 | 0.6235 | 0.5224 | 0.3406 | 0.8015 | 0.1985
MLP | LOF | None | 0.7266 | 0.6969 | 0.5839 | 0.4136 | 0.8243 | 0.1757
MLP | LOF | RUS | 0.7238 | 0.6945 | 0.5792 | 0.4078 | 0.8225 | 0.1775
MLP | LOF | IHT | 0.7284 | 0.7041 | 0.5870 | 0.4174 | 0.8255 | 0.1745
MLP | LOF | ENN | 0.7081 | 0.6552 | 0.5516 | 0.3747 | 0.8122 | 0.1878
MLP | LOF | ROS | 0.6955 | 0.6311 | 0.5291 | 0.3482 | 0.8039 | 0.1961
MLP | LOF | SMOTE | 0.6921 | 0.6253 | 0.5228 | 0.3410 | 0.8017 | 0.1983
MLP | LOF | ADASYN | 0.6895 | 0.6223 | 0.5182 | 0.3358 | 0.8000 | 0.2000
MLP | LOF | SMOTEENN | 0.7082 | 0.6596 | 0.5518 | 0.3750 | 0.8122 | 0.1878
MLP | LOF | SMOTETomek | 0.6922 | 0.6247 | 0.5232 | 0.3414 | 0.8018 | 0.1982

The classification results of RF are shown in table 10, the difference between
the top and bottom AUC ranking is 0.0207, the fluctuation is between LR and
MLP. Among the outlier detection and balanced sampling methods, the most
effective were IForest and None (AUC 0.6985), below the upper limit of what
can be achieved by LR and MLP. And the least effective were IForest and ADASYN
(AUC 0.6778), both the best and worst RF models use IForest. Among the
sampling methods, ADASYN still gets the worst results.

Table 10: Classification results (RF). Model | reject_strategy | sampling_strategy | ACC | AUC | G-mean | TPR | TNR | FPR
---|---|---|---|---|---|---|---|---
RF | None | None | 0.7242 | 0.6977 | 0.5797 | 0.4085 | 0.8227 | 0.1773
RF | None | RUS | 0.7247 | 0.6981 | 0.5806 | 0.4096 | 0.8231 | 0.1769
RF | None | IHT | 0.7184 | 0.6920 | 0.5697 | 0.3964 | 0.8189 | 0.1811
RF | None | ENN | 0.7229 | 0.6943 | 0.5774 | 0.4058 | 0.8219 | 0.1781
RF | None | ROS | 0.7221 | 0.6956 | 0.5761 | 0.4041 | 0.8213 | 0.1787
RF | None | SMOTE | 0.7161 | 0.6819 | 0.5657 | 0.3916 | 0.8174 | 0.1826
RF | None | ADASYN | 0.7149 | 0.6778 | 0.5637 | 0.3891 | 0.8167 | 0.1833
RF | None | SMOTEENN | 0.7153 | 0.6856 | 0.5642 | 0.3898 | 0.8169 | 0.1831
RF | None | SMOTETomek | 0.7171 | 0.6827 | 0.5675 | 0.3937 | 0.8181 | 0.1819
RF | COPOD | None | 0.7246 | 0.6976 | 0.5805 | 0.4094 | 0.8230 | 0.1770
RF | COPOD | RUS | 0.7237 | 0.6971 | 0.5789 | 0.4076 | 0.8224 | 0.1776
RF | COPOD | IHT | 0.7188 | 0.6916 | 0.5704 | 0.3973 | 0.8192 | 0.1808
RF | COPOD | ENN | 0.7222 | 0.6947 | 0.5764 | 0.4044 | 0.8214 | 0.1786
RF | COPOD | ROS | 0.7226 | 0.6969 | 0.5771 | 0.4053 | 0.8217 | 0.1783
RF | COPOD | SMOTE | 0.7176 | 0.6833 | 0.5683 | 0.3947 | 0.8184 | 0.1816
RF | COPOD | ADASYN | 0.7145 | 0.6780 | 0.5629 | 0.3881 | 0.8163 | 0.1837
RF | COPOD | SMOTEENN | 0.7166 | 0.6855 | 0.5665 | 0.3925 | 0.8177 | 0.1823
RF | COPOD | SMOTETomek | 0.7163 | 0.6827 | 0.5661 | 0.3920 | 0.8175 | 0.1825
RF | IForest | None | 0.7248 | 0.6985 | 0.5808 | 0.4098 | 0.8231 | 0.1769
RF | IForest | RUS | 0.7231 | 0.6980 | 0.5779 | 0.4063 | 0.8220 | 0.1780
RF | IForest | IHT | 0.7199 | 0.6920 | 0.5723 | 0.3995 | 0.8199 | 0.1801
RF | IForest | ENN | 0.7224 | 0.6942 | 0.5766 | 0.4047 | 0.8215 | 0.1785
RF | IForest | ROS | 0.7217 | 0.6968 | 0.5754 | 0.4033 | 0.8211 | 0.1789
RF | IForest | SMOTE | 0.7154 | 0.6817 | 0.5645 | 0.3901 | 0.8170 | 0.1830
RF | IForest | ADASYN | 0.7142 | 0.6778 | 0.5624 | 0.3875 | 0.8162 | 0.1838
RF | IForest | SMOTEENN | 0.7162 | 0.6861 | 0.5659 | 0.3918 | 0.8175 | 0.1825
RF | IForest | SMOTETomek | 0.7156 | 0.6822 | 0.5649 | 0.3906 | 0.8171 | 0.1829
RF | CBLOF | None | 0.7247 | 0.6972 | 0.5806 | 0.4095 | 0.8230 | 0.1770
RF | CBLOF | RUS | 0.7236 | 0.6967 | 0.5788 | 0.4074 | 0.8224 | 0.1776
RF | CBLOF | IHT | 0.7177 | 0.6909 | 0.5686 | 0.3950 | 0.8185 | 0.1815
RF | CBLOF | ENN | 0.7221 | 0.6936 | 0.5761 | 0.4042 | 0.8214 | 0.1786
RF | CBLOF | ROS | 0.7214 | 0.6951 | 0.5749 | 0.4026 | 0.8209 | 0.1791
RF | CBLOF | SMOTE | 0.7144 | 0.6826 | 0.5628 | 0.3880 | 0.8163 | 0.1837
RF | CBLOF | ADASYN | 0.7141 | 0.6782 | 0.5623 | 0.3874 | 0.8161 | 0.1839
RF | CBLOF | SMOTEENN | 0.7155 | 0.6848 | 0.5646 | 0.3903 | 0.8170 | 0.1830
RF | CBLOF | SMOTETomek | 0.7156 | 0.6827 | 0.5648 | 0.3905 | 0.8171 | 0.1829
RF | LOF | None | 0.7237 | 0.6970 | 0.5788 | 0.4075 | 0.8224 | 0.1776
RF | LOF | RUS | 0.7238 | 0.6970 | 0.5791 | 0.4077 | 0.8225 | 0.1775
RF | LOF | IHT | 0.7182 | 0.6925 | 0.5694 | 0.3959 | 0.8188 | 0.1812
RF | LOF | ENN | 0.7212 | 0.6940 | 0.5747 | 0.4024 | 0.8208 | 0.1792
RF | LOF | ROS | 0.7229 | 0.6961 | 0.5775 | 0.4058 | 0.8219 | 0.1781
RF | LOF | SMOTE | 0.7159 | 0.6828 | 0.5654 | 0.3912 | 0.8173 | 0.1827
RF | LOF | ADASYN | 0.7135 | 0.6781 | 0.5611 | 0.3860 | 0.8157 | 0.1843
RF | LOF | SMOTEENN | 0.7162 | 0.6853 | 0.5659 | 0.3918 | 0.8175 | 0.1825
RF | LOF | SMOTETomek | 0.7158 | 0.6823 | 0.5653 | 0.3910 | 0.8173 | 0.1827

The classification results of GBDT is shown in table 11. The difference
between the top and bottom AUC ranking is 0.0116, it’s closer to the RF which
is the same as the tree model. Among the outlier detection and balanced
sampling methods, the most effective are LOF and None (AUC 0.7107), which is
the same as LR and is the best of the four models. The least effective are
CBLOF and ADASYN (AUC 0.6991), the worst effects of the four models all use
ADASYN.

Table 11: Classification results (GBDT). Model | reject_strategy | sampling_strategy | ACC | AUC | G-mean | TPR | TNR | FPR
---|---|---|---|---|---|---|---|---
GBDT | None | None | 0.7319 | 0.7102 | 0.5929 | 0.4247 | 0.8278 | 0.1722
GBDT | None | RUS | 0.7294 | 0.7087 | 0.5887 | 0.4195 | 0.8261 | 0.1739
GBDT | None | IHT | 0.7283 | 0.7062 | 0.5869 | 0.4173 | 0.8255 | 0.1745
GBDT | None | ENN | 0.7292 | 0.7079 | 0.5883 | 0.4191 | 0.8260 | 0.1740
GBDT | None | ROS | 0.7303 | 0.7085 | 0.5903 | 0.4214 | 0.8268 | 0.1732
GBDT | None | SMOTE | 0.7246 | 0.7013 | 0.5804 | 0.4094 | 0.8230 | 0.1770
GBDT | None | ADASYN | 0.7237 | 0.7005 | 0.5789 | 0.4076 | 0.8224 | 0.1776
GBDT | None | SMOTEENN | 0.7268 | 0.7010 | 0.5843 | 0.4141 | 0.8245 | 0.1755
GBDT | None | SMOTETomek | 0.7240 | 0.7015 | 0.5795 | 0.4082 | 0.8226 | 0.1774
GBDT | COPOD | None | 0.7312 | 0.7105 | 0.5918 | 0.4233 | 0.8273 | 0.1727
GBDT | COPOD | RUS | 0.7289 | 0.7076 | 0.5879 | 0.4185 | 0.8258 | 0.1742
GBDT | COPOD | IHT | 0.7291 | 0.7056 | 0.5882 | 0.4189 | 0.8260 | 0.1740
GBDT | COPOD | ENN | 0.7306 | 0.7078 | 0.5908 | 0.4221 | 0.8270 | 0.1730
GBDT | COPOD | ROS | 0.7327 | 0.7097 | 0.5942 | 0.4264 | 0.8283 | 0.1717
GBDT | COPOD | SMOTE | 0.7247 | 0.7002 | 0.5805 | 0.4095 | 0.8230 | 0.1770
GBDT | COPOD | ADASYN | 0.7237 | 0.6996 | 0.5789 | 0.4076 | 0.8224 | 0.1776
GBDT | COPOD | SMOTEENN | 0.7272 | 0.7024 | 0.5849 | 0.4148 | 0.8247 | 0.1753
GBDT | COPOD | SMOTETomek | 0.7243 | 0.7004 | 0.5799 | 0.4087 | 0.8228 | 0.1772
GBDT | IForest | None | 0.7315 | 0.7106 | 0.5923 | 0.4239 | 0.8275 | 0.1725
GBDT | IForest | RUS | 0.7301 | 0.7096 | 0.5899 | 0.4210 | 0.8266 | 0.1734
GBDT | IForest | IHT | 0.7284 | 0.7053 | 0.5870 | 0.4174 | 0.8255 | 0.1745
GBDT | IForest | ENN | 0.7300 | 0.7078 | 0.5897 | 0.4208 | 0.8265 | 0.1735
GBDT | IForest | ROS | 0.7300 | 0.7094 | 0.5897 | 0.4208 | 0.8265 | 0.1735
GBDT | IForest | SMOTE | 0.7244 | 0.7017 | 0.5801 | 0.4090 | 0.8229 | 0.1771
GBDT | IForest | ADASYN | 0.7238 | 0.6996 | 0.5791 | 0.4077 | 0.8225 | 0.1775
GBDT | IForest | SMOTEENN | 0.7261 | 0.7017 | 0.5830 | 0.4126 | 0.8240 | 0.1760
GBDT | IForest | SMOTETomek | 0.7235 | 0.7008 | 0.5785 | 0.4071 | 0.8223 | 0.1777
GBDT | CBLOF | None | 0.7309 | 0.7104 | 0.5912 | 0.4226 | 0.8271 | 0.1729
GBDT | CBLOF | RUS | 0.7295 | 0.7088 | 0.5888 | 0.4197 | 0.8262 | 0.1738
GBDT | CBLOF | IHT | 0.7299 | 0.7055 | 0.5895 | 0.4205 | 0.8265 | 0.1735
GBDT | CBLOF | ENN | 0.7293 | 0.7077 | 0.5885 | 0.4193 | 0.8261 | 0.1739
GBDT | CBLOF | ROS | 0.7311 | 0.7090 | 0.5916 | 0.4231 | 0.8273 | 0.1727
GBDT | CBLOF | SMOTE | 0.7243 | 0.7017 | 0.5799 | 0.4088 | 0.8228 | 0.1772
GBDT | CBLOF | ADASYN | 0.7237 | 0.6991 | 0.5789 | 0.4076 | 0.8224 | 0.1776
GBDT | CBLOF | SMOTEENN | 0.7271 | 0.7026 | 0.5847 | 0.4146 | 0.8246 | 0.1754
GBDT | CBLOF | SMOTETomek | 0.7245 | 0.7018 | 0.5802 | 0.4092 | 0.8229 | 0.1771
GBDT | LOF | None | 0.7310 | 0.7107 | 0.5913 | 0.4228 | 0.8272 | 0.1728
GBDT | LOF | RUS | 0.7300 | 0.7081 | 0.5897 | 0.4208 | 0.8265 | 0.1735
GBDT | LOF | IHT | 0.7280 | 0.7057 | 0.5863 | 0.4166 | 0.8252 | 0.1748
GBDT | LOF | ENN | 0.7305 | 0.7085 | 0.5906 | 0.4219 | 0.8269 | 0.1731
GBDT | LOF | ROS | 0.7304 | 0.7099 | 0.5903 | 0.4215 | 0.8268 | 0.1732
GBDT | LOF | SMOTE | 0.7247 | 0.7010 | 0.5806 | 0.4096 | 0.8231 | 0.1769
GBDT | LOF | ADASYN | 0.7237 | 0.7001 | 0.5788 | 0.4075 | 0.8224 | 0.1776
GBDT | LOF | SMOTEENN | 0.7265 | 0.7017 | 0.5836 | 0.4133 | 0.8242 | 0.1758
GBDT | LOF | SMOTETomek | 0.7247 | 0.7011 | 0.5806 | 0.4096 | 0.8231 | 0.1769

###  5.2 Comprehensive comparison of machine learning models

Table 12 synthesizes the prediction results of the four models. GBDT obtains
the best results whether comparing the case without outlier detection and
balanced sampling method (AUC 0.7102), comparing the average values of all
same-model methods (9*5=45 types, AUC 0.7051), or comparing the best values
that the models can obtain (AUC 0.7107).

Among the best strategies for each model, LOF outlier detection and no sample
balance methods each take three seats. IForest is more effective for RF, and
IHT downsampling methods are more effective for MLP. RF performs better than
MLP when outlier detection and balanced sampling methods are not used.
However, combining LOF and IHT, the AUC of the best MLP model even surpassed
the best RF model.

Table 12: Comprehensive comparison of the machine learning models.

Mode | Model | Reject strategy | Sampling strategy | ACC | AUC | G-mean | TPR | TNR | FPR
---|---|---|---|---|---|---|---|---|---
None | GBDT | None | None | 0.7319 | 0.7102 | 0.5929 | 0.4247 | 0.8278 | 0.1722
LR | None | None | 0.7298 | 0.7071 | 0.5893 | 0.4203 | 0.8264 | 0.1736
RF | None | None | 0.7242 | 0.6977 | 0.5797 | 0.4085 | 0.8227 | 0.1773
MLP | None | None | 0.7230 | 0.6953 | 0.5778 | 0.4061 | 0.8220 | 0.1780
Mean | GBDT | All mean | All mean | 0.7277 | 0.7051 | 0.5858 | 0.4160 | 0.8251 | 0.1749
LR | All mean | All mean | 0.7287 | 0.7046 | 0.5875 | 0.4181 | 0.8257 | 0.1743
RF | All mean | All mean | 0.7192 | 0.6895 | 0.5712 | 0.3982 | 0.8195 | 0.1805
MLP | All mean | All mean | 0.7068 | 0.6574 | 0.5490 | 0.3721 | 0.8113 | 0.1887
Best | GBDT | LOF | None | 0.7310 | 0.7107 | 0.5913 | 0.4228 | 0.8272 | 0.1728
LR | LOF | None | 0.7299 | 0.7079 | 0.5895 | 0.4205 | 0.8265 | 0.1735
RF | IForest | None | 0.7248 | 0.6985 | 0.5808 | 0.4098 | 0.8231 | 0.1769
MLP | LOF | IHT | 0.7284 | 0.7041 | 0.5870 | 0.4174 | 0.8255 | 0.1745

  * 1.

The mode “None” is the case comparing the models’ classification results
without outlier detection and balanced sampling methods. The mode “Mean” is
the case comparing the average classification results of each model using
every outlier detection and balanced sampling methods. The mode “Best” is the
case comparing the best classification results that the models can achieved.

###  5.3 Comprehensive comparison of the outlier detection methods

Table 13 synthesizes the results of the different outlier detection methods.
In general, as long as the outlier detection methods are used, the effect will
be better than without it. After taking the average of the experimental
results of the same outlier detection algorithms (9*4=36 types), the ranking
from highest to lowest is IForest, COPOD, CBLOF, LOF, None. The average
classification results using IForest is the best, but in most cases, LOF can
help the model get the best results according to table 12.

Table 13: Comprehensive comparison of the outlier detection methods.

Reject strategy | ACC | AUC | G-mean | TPR | TNR | FPR | Rank
---|---|---|---|---|---|---|---
IForest | 0.7207 | 0.6896 | 0.5734 | 0.4011 | 0.8204 | 0.1796 | 1.5
COPOD | 0.7208 | 0.6891 | 0.5737 | 0.4014 | 0.8205 | 0.1795 | 3
CBLOF | 0.7204 | 0.6891 | 0.5731 | 0.4007 | 0.8203 | 0.1797 | 3
LOF | 0.7207 | 0.6891 | 0.5735 | 0.4012 | 0.8204 | 0.1796 | 3.25
None | 0.7206 | 0.6888 | 0.5733 | 0.4010 | 0.8204 | 0.1796 | 4.25

  * 1.

The “Rank” is calculated as the mean of the ranking (AUC) of outlier detection
methods under the same model and the same balanced sampling method.

###  5.4 Comprehensive comparison of the balanced sampling methods

Table 14 synthesizes the results of the different balanced sampling methods.
After taking the mean of the experimental results of the same balanced
sampling methods (5*4=20 types), it is found that the effect of not using a
balanced sampling method is the best, simple RUS and ROS are ranked 2, 3.
While complex balanced sampling methods are less effective, and the commonly
used SMOTE algorithm is ranked only the second to last. The ADASYN algorithm,
which achieved the worst results among the four types of models, ranked first
from the bottom. Except for the effect of IHT on MLP, for other models, the
balanced sampling method will reduce the classification results. The ranking
is similar to two relevant studies [Moscato et al., 2021, Namvar et al., 2018]
which using the Lending Club data too. In their research, RUS also achieved
the best results among all sampling methods. However, neither of them added
the no sampling method to the experiments as a comparison.

Table 14: Comprehensive comparison of the different balanced sampling methods

Sampling strategy | ACC | AUC | G-mean | TPR | TNR | FPR | Rank
---|---|---|---|---|---|---|---
None | 0.7274 | 0.7029 | 0.5853 | 0.4154 | 0.8249 | 0.1751 | 1.25
RUS | 0.7264 | 0.7018 | 0.5836 | 0.4133 | 0.8242 | 0.1758 | 2.75
ROS | 0.7187 | 0.6853 | 0.5699 | 0.3971 | 0.8191 | 0.1809 | 3.25
ENN | 0.7233 | 0.6941 | 0.5781 | 0.4066 | 0.8221 | 0.1779 | 4
IHT | 0.7259 | 0.7008 | 0.5826 | 0.4121 | 0.8238 | 0.1762 | 4.5
SMOTEENN | 0.7193 | 0.6864 | 0.5713 | 0.3984 | 0.8195 | 0.1805 | 6.25
SMOTETomek | 0.7155 | 0.6782 | 0.5643 | 0.3902 | 0.8170 | 0.1830 | 6.75
SMOTE | 0.7155 | 0.6779 | 0.5644 | 0.3904 | 0.8171 | 0.1829 | 7.25
ADASYN | 0.7137 | 0.6750 | 0.5611 | 0.3864 | 0.8158 | 0.1842 | 9

  * 1.

The “Rank” is calculated as the mean of the ranking (AUC) of balanced sampling
methods in the case of the same model and the same outlier detection method.

##  6 Conclusions

Credit scoring is crucial for P2P lending platforms, outliers and sample
imbalance make the construction of credit scoring models challenging. There
have been many studies using balanced sampling methods to deal with the sample
imbalance problem [Moscato et al., 2021, Namvar et al., 2018, Niu et al.,
2020], but few studies have incorporated outlier detection methods into the
modeling process.

This paper provides a systematic study for the application of outlier
detection and balanced sampling methods in the field of credit scoring. The
experiments use real P2P lending platform data and cover a variety of outlier
detection, balanced sampling methods, and machine learning models. The
experimental results show that the outlier detection algorithm can enhance the
robustness of the model. In addition, a suitable balanced sampling method
brings a large improvement to MLP but is relatively less useful to LR, RF, and
GBDT.

In future plans, firstly, more advanced models, such as ensemble strategies or
deep learning models, will be studied. Secondly, feature selection is also an
important part of the construction of the credit scoring model, which will be
considered to be integrated into the existing framework.

## Acknowledgments

This work was supported by HuaRong RongTong (Beijing) Technology Co., Ltd. We
acknowledge HuaRong RongTong (Beijing) for providing us with high-performance
machines for computation. We also acknowledge the anonymous reviewers for
proposing detailed modification advice to help us improve the quality of this
manuscript.

## References

  * Abelian & Castellano [2017] Abelian, J., & Castellano, J. G. (2017).  A comparative study on base classifiers in ensemble methods for credit scoring.  Expert Systems with Applications, 73, 1–10. doi:10.1016/j.eswa.2016.12.020.
  * Ala’Raj & Abbod [2016] Ala’Raj, M., & Abbod, M. F. (2016).  Classifiers consensus system approach for credit scoring.  Knowledge-Based Systems, 104, 89–105. URL: http://dx.doi.org/10.1016/j.knosys.2016.04.013. doi:10.1016/j.knosys.2016.04.013.
  * Bastani et al. [2019] Bastani, K., Asgari, E., & Namavari, H. (2019).  Wide and deep learning for peer-to-peer lending.  Expert Systems with Applications, 134, 209–224. URL: https://doi.org/10.1016/j.eswa.2019.05.042. doi:10.1016/j.eswa.2019.05.042. arXiv:1810.03466.
  * Batista et al. [2003] Batista, G. E. A. P. A., Bazzan, A. L. C., & Monard, M. C. (2003).  Balancing Training Data for Automated Annotation of Keywords: a Case Study.  In S. Lifschitz, N. F. A. Jr., G. J. P. Jr., & R. Linden (Eds.), II Brazilian Workshop on Bioinformatics, December 3-5, 2003, Macaé, RJ, Brazil (pp. 10--18).
  * Batista et al. [2004] Batista, G. E. A. P. A., Prati, R. C., & Monard, M. C. (2004).  A study of the behavior of several methods for balancing machine learning training data.  Sigkdd Explorations, 6, 20--29.
  * Breiman [1996] Breiman, L. (1996).  Bagging predictors.  Machine Learning, 24, 123--140. URL: https://doi.org/10.1007/BF00058655. doi:10.1007/bf00058655.
  * Breiman [2001] Breiman, L. (2001).  Random forests.  Machine Learning, 45, 5--32. URL: https://doi.org/10.1023/A:1010933404324. doi:10.1023/A:1010933404324.
  * Breunig et al. [2000] Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000).  LOF: Identifying Density-Based Local Outliers.  SIGMOD Rec., 29, 93--104. URL: https://doi.org/10.1145/335191.335388. doi:10.1145/335191.335388.
  * Chawla et al. [2002] Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002).  SMOTE: synthetic minority over-sampling technique.  Journal of Artificial Intelligence Research, 16, 321--357.
  * Crook et al. [2007] Crook, J. N., Edelman, D. B., & Thomas, L. C. (2007).  Recent developments in consumer credit risk assessment.  European Journal of Operational Research, 183, 1447--1465. doi:10.1016/j.ejor.2006.09.100.
  * Dastile et al. [2020] Dastile, X., Celik, T., & Potsane, M. (2020).  Statistical and machine learning models in credit scoring: A systematic literature survey.  Applied Soft Computing Journal, 91, 106263. URL: https://doi.org/10.1016/j.asoc.2020.106263. doi:10.1016/j.asoc.2020.106263.
  * Engelmann & Lessmann [2021] Engelmann, J., & Lessmann, S. (2021).  Conditional Wasserstein GAN-based oversampling of tabular data for imbalanced learning.  Expert Systems with Applications, 174. doi:10.1016/j.eswa.2021.114582. arXiv:2008.09202.
  * Friedman [2001] Friedman, J. H. (2001).  Greedy function approximation: A gradient boosting machine.  Annals of Statistics, 29, 1189--1232. URL: http://www.jstor.org/stable/2699986. doi:10.1214/aos/1013203451.
  * Garcia et al. [2019] Garcia, V., Marqus, A. I., & Snchez, J. S. (2019).  Exploring the synergetic effects of sample types on the performance of ensembles for credit risk and corporate bankruptcy prediction.  Inf. Fusion, 47, 88--101. URL: https://doi.org/10.1016/j.inffus.2018.07.004. doi:10.1016/j.inffus.2018.07.004.
  * Hájek [2011] Hájek, P. (2011).  Municipal credit rating modelling by neural networks.  Decision Support Systems, 51, 108--118. doi:10.1016/j.dss.2010.11.033.
  * Hansen & Salamon [1990] Hansen, L. K., & Salamon, P. (1990).  Neural network ensembles.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 993--1001.
  * He et al. [2008] He, H., Bai, Y., Garcia, E. A., & Li, S. (2008).  ADASYN: Adaptive synthetic sampling approach for imbalanced learning.  In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) (pp. 1322--1328).
  * He et al. [2018] He, H., Zhang, W., & Zhang, S. (2018).  A novel ensemble method for credit scoring: Adaption of different imbalance ratios.  Expert Systems with Applications, 98, 105--117. doi:10.1016/j.eswa.2018.01.012.
  * He et al. [2003] He, Z., Xu, X., & Deng, S. (2003).  Discovering cluster-based local outliers.  Pattern Recognition Letters, 24, 1641--1650. URL: https://www.sciencedirect.com/science/article/pii/S0167865503000035. doi:https://doi.org/10.1016/S0167-8655(03)00003-5.
  * Kim & Cho [2017] Kim, A., & Cho, S.-B. (2017).  Dempster-Shafer Fusion of Semi-supervised Learning Methods for Predicting Defaults in Social Lending.  In D. Liu, S. Xie, Y. Li, D. Zhao, & E.-S. M. El-Alfy (Eds.), Neural Information Processing - 24th International Conference, ICONIP 2017, Guangzhou, China, November 14-18, 2017, Proceedings, Part II (pp. 854--862).  Springer volume 10635 of Lecture Notes in Computer Science.  URL: https://doi.org/10.1007/978-3-319-70096-0%5C_87. doi:10.1007/978-3-319-70096-0\\_87.
  * Lee et al. [2021] Lee, J. W., Lee, W. K., & Sohn, S. Y. (2021).  Graph convolutional network-based credit default prediction utilizing three types of virtual distances among borrowers.  Expert Systems with Applications, 168, 114411. URL: https://doi.org/10.1016/j.eswa.2020.114411. doi:10.1016/j.eswa.2020.114411.
  * Lessmann et al. [2015] Lessmann, S., Baesens, B., Seow, H. V., & Thomas, L. C. (2015).  Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research.  European Journal of Operational Research, 247, 124--136. doi:10.1016/j.ejor.2015.05.030.
  * Li et al. [2020] Li, Z., Zhao, Y., Botta, N., Ionescu, C., & Hu, X. (2020).  COPOD: Copula-Based Outlier Detection.  arXiv e-prints, (p. arXiv:2009.09463). arXiv:2009.09463.
  * Liu et al. [2008] Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008).  Isolation Forest.  In 2008 Eighth IEEE International Conference on Data Mining (pp. 413--422).
  * Liu et al. [2021] Liu, W., Fan, H., & Xia, M. (2021).  Step-wise multi-grained augmented gradient boosting decision trees for credit scoring.  Engineering Applications of Artificial Intelligence, 97, 104036. URL: https://doi.org/10.1016/j.engappai.2020.104036. doi:10.1016/j.engappai.2020.104036.
  * Ma et al. [2018] Ma, X., Sha, J., Wang, D., Yu, Y., Yang, Q., & Niu, X. (2018).  Study on a prediction of P2P network loan default based on the machine learning LightGBM and XGboost algorithms according to different high dimensional data cleaning.  Electronic Commerce Research and Applications, 31, 24--39. URL: https://doi.org/10.1016/j.elerap.2018.08.002. doi:10.1016/j.elerap.2018.08.002.
  * Malekipirbazari & Aksakalli [2015] Malekipirbazari, M., & Aksakalli, V. (2015).  Risk assessment in social lending via random forests.  Expert Systems with Applications, 42, 4621--4631. URL: http://dx.doi.org/10.1016/j.eswa.2015.02.001. doi:10.1016/j.eswa.2015.02.001.
  * Melo Junior et al. [2020] Melo Junior, L., Nardini, F. M., Renso, C., Trani, R., & Macedo, J. A. (2020).  A novel approach to define the local region of dynamic selection techniques in imbalanced credit scoring problems.  Expert Systems with Applications, 152, 113351. URL: http://www.sciencedirect.com/science/article/pii/S0957417420301767. doi:https://doi.org/10.1016/j.eswa.2020.113351.
  * Moscato et al. [2021] Moscato, V., Picariello, A., & Sperlí, G. (2021).  A benchmark of machine learning approaches for credit score prediction.  Expert Systems with Applications, 165. doi:10.1016/j.eswa.2020.113986.
  * Namvar et al. [2018] Namvar, A., Siami, M., Rabhi, F., & Naderpour, M. (2018).  Credit risk prediction in an imbalanced social lending environment.  International Journal of Computational Intelligence Systems, 11, 925--935. URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85060499389&doi=10.2991%2Fijcis.11.1.70&partnerID=40&md5=6d87be8d6f8cfadb7155f79dbea149f3. doi:10.2991/ijcis.11.1.70.
  * Niu et al. [2020] Niu, K., Zhang, Z., Liu, Y., & Li, R. (2020).  Resampling ensemble model based on data distribution for imbalanced credit risk evaluation in P2P lending.  Information Sciences, 536, 120--134. URL: https://doi.org/10.1016/j.ins.2020.05.040. doi:10.1016/j.ins.2020.05.040.
  * Ruyu et al. [2019] Ruyu, B., Mo, H., & Haifeng, L. (2019).  A Comparison of Credit Rating Classification Models Based on Spark- Evidence from Lending-club.  Procedia Computer Science, 162, 811--818. URL: https://doi.org/10.1016/j.procs.2019.12.054. doi:10.1016/j.procs.2019.12.054.
  * Schapire [1990] Schapire, R. E. (1990).  The Strength of Weak Learnability.  Mach. Learn., 5, 197--227. URL: https://doi.org/10.1007/BF00116037. doi:10.1007/BF00116037.
  * Schapire [1999] Schapire, R. E. (1999).  A Brief Introduction to Boosting.  In T. Dean (Ed.), Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, IJCAI 99, Stockholm, Sweden, July 31 \- August 6, 1999. 2 Volumes, 1450 pages (pp. 1401--1406).  Morgan Kaufmann.  URL: http://ijcai.org/Proceedings/99-2/Papers/103.pdf.
  * Smith et al. [2014] Smith, M. R., Martinez, T., & Giraud-Carrier, C. (2014).  An instance level analysis of data complexity.  Machine Learning, 95, 225--256.
  * Song et al. [2020] Song, Y., Wang, Y., Ye, X., Wang, D., Yin, Y., & Wang, Y. (2020).  Multi-view ensemble learning based on distance-to-model and adaptive clustering for imbalanced credit risk assessment in P2P lending.  Information Sciences, 525, 182--204. doi:10.1016/j.ins.2020.03.027.
  * Teply & Polena [2020] Teply, P., & Polena, M. (2020).  Best classification algorithms in peer-to-peer lending.  North American Journal of Economics and Finance, 51, 100904. URL: https://doi.org/10.1016/j.najef.2019.01.001. doi:10.1016/j.najef.2019.01.001.
  * Tomek [1976] Tomek, I. (1976).  Two modifications of cnn.  IEEE Transactions on Systems, Man, and Cybernetics, SMC-6, 769--772. doi:10.1109/TSMC.1976.4309452.
  * Wei et al. [2019] Wei, S., Yang, D., Zhang, W., & Zhang, S. (2019).  A Novel Noise-Adapted Two-Layer Ensemble Model for Credit Scoring Based on Backflow Learning.  IEEE Access, 7, 99217--99230. URL: https://doi.org/10.1109/ACCESS.2019.2930332. doi:10.1109/ACCESS.2019.2930332.
  * Wilson [1972] Wilson, D. L. (1972).  Asymptotic Properties of Nearest Neighbor Rules Using Edited Data.  systems man and cybernetics, 2, 408--421.
  * Xia [2019] Xia, Y. (2019).  A Novel Reject Inference Model Using Outlier Detection and Gradient Boosting Technique in Peer-to-Peer Lending.  IEEE Access, 7, 92893--92907. doi:10.1109/ACCESS.2019.2927602.
  * Xia et al. [2020] Xia, Y., He, L., Li, Y., Liu, N., & Ding, Y. (2020).  Predicting loan default in peer-to-peer lending using narrative data.  Journal of Forecasting, 39, 260--280. doi:10.1002/for.2625.
  * Xia et al. [2018] Xia, Y., Liu, C., Da, B., & Xie, F. (2018).  A novel heterogeneous ensemble credit scoring model based on bstacking approach.  Expert Systems with Applications, 93, 182--199. doi:10.1016/j.eswa.2017.10.022.
  * Xia et al. [2017] Xia, Y., Liu, C., & Liu, N. (2017).  Cost-sensitive boosted tree for loan evaluation in peer-to-peer lending.  Electronic Commerce Research and Applications, 24, 30--49. URL: http://dx.doi.org/10.1016/j.elerap.2017.06.004. doi:10.1016/j.elerap.2017.06.004.
  * Zhang et al. [2019] Zhang, W., He, H., & Zhang, S. (2019).  A novel multi-stage hybrid model with enhanced multi-population niche genetic algorithm: An application in credit scoring.  Expert Systems with Applications, 121, 221--232. doi:10.1016/j.eswa.2018.12.020.
  * Zhang et al. [2021] Zhang, W., Yang, D., & Zhang, S. (2021).  A new hybrid ensemble model with voting-based outlier detection and balanced sampling for credit scoring.  Expert Systems with Applications, 174. doi:10.1016/j.eswa.2021.114744.
  * Zhang et al. [2020] Zhang, W., Yang, D., Zhang, S., Ablanedo-Rosas, J. H., Wu, X., & Lou, Y. (2020).  A novel multi-stage ensemble model with enhanced outlier adaptation for credit scoring.  Expert Systems with Applications, 165. doi:10.1016/j.eswa.2020.113872.
  * Zhao et al. [2017] Zhao, H., Ge, Y., Liu, Q., Wang, G., Chen, E., & Zhang, H. (2017).  P2P Lending Survey: Platforms, Recent Advances and Prospects.  ACM Trans. Intell. Syst. Technol., 8. URL: https://doi.org/10.1145/3078848. doi:10.1145/3078848.