# Forecasting Probability of Default for Consumer Loan Management with
Gaussian Mixture Models

Hamidreza Arian111E-Mail<EMAIL_ADDRESS>
Sharif University
of Technology  Seyed Mohammad Sina Seyfi222E-Mail<EMAIL_ADDRESS>
Sharif University
of Technology  Azin Sharifi333E-Mail<EMAIL_ADDRESS>
Sharif University
of Technology

( October 2020)

###### Abstract

Credit scoring is an essential tool used by global financial institutions and
credit lenders for financial decision making. In this paper, we introduce a
new method based on Gaussian Mixture Model (GMM) to forecast the probability
of default for individual loan applicants. Clustering similar customers with
each other, our model associates a probability of being healthy to each group.
In addition, our GMM-based model probabilistically associates individual
samples to clusters, and then estimates the probability of default for each
individual based on how it relates to GMM clusters. We provide applications
for risk managers and decision makers in banks and non-bank financial
institutions to maximize profit and mitigate the expected loss by giving loans
to those who have a probability of default below a decision threshold. Our
model has a number of advantages. First, it gives a probabilistic view of
credit standing for each individual applicant instead of a binary
classification and therefore provides more information for financial decision
makers. Second, the expected loss on the train set calculated by our GMM-based
default probabilities is very close to the actual loss, and third, our
approach is computationally efficient.

Keywords: Credit Scoring, Machine Learning, Probability of Default, Gaussian
Mixture Models, Financial Decision Making

JEL classification: C61, G11.

##  1 Introduction

In the face of the recent challenges, it has been a crucial issue for bankers
and credit lenders to find models to predict borrower’s bankruptcy with high
precision (Altman et al. (2014), Veganzones and Séverin (2018)). As a result,
credit scoring systems have evolved as an essential part of bankruptcy
prediction and credit risk management solutions. Financial institutions,
especially banks, are widely using credit scoring models to allocate credit to
good applicants, and to differentiate between good and bad borrowers. Applying
credit scoring models will reduce credit transaction costs and the potential
risk of a bad loan loss, and results in improvements in credit lending
decisions. The main objective of credit risk analysis is to use personal and
financial information to forecast default probability of individual customers,
and to reduce the credit risk associated with issuing loan.

Within the bankruptcy prediction studies that have been published, there is a
wide range of classification methods used, including Linear Discriminant
Analysis (LDA), Logistic Regression (LR) (Dimitras et al. (1996), Ohlson
(1980)), and more recently, nonlinear techniques such as Support Vector
Machines (SVMs) and Neural Networks (NNs) (Lee et al. (1996), Huang et al.
(2007), and Mancisidor et al. (2020)). Recently, advanced machine learning and
data-driven approaches have gained popularity in offering state-of-the-art
solutions for predicting bankruptcy. In addition, in machine learning
applications in the corporate sector, the relationship between company
features and credit decision-making is learned from the data rather than
modeling the company’s financial characteristics (Atiya (2001)). Most of the
work in credit scoring has been focused on building models to provide a binary
prediction on whether a customer will default on their loan or not. Such
binary classification routines can be based on various types of supervised
leaning algorithms (Hand et al. (2001), Zhang and Härdle (2010), Min and Lee
(2005), Salcedo-Sanz et al. (2005), and Wu et al. (2007)).

Hand and Henley (1997) review several statistical classification models for
credit scoring. They emphasize that the credit scoring problem does not have a
unique best solution. As a result of such surveys, numerous researchers have
focused on providing hybrid and combined models for classification challenges
present in building credit scoring solutions and some propose to combine
domain knowledge with classification methods (Sinha and Zhao (2008)). Some
researchers implement a scorecard model using fewer classifiers while some
other employ multiple classifiers and combine them with ensemble techniques
(Ala’raj and Abbod (2016) and Xiao et al. (2020)). To find an efficient
ensemble classifier, Hsieh and Hung (2010), presents the class-wise
classification. In order to investigate the performance of a combination of
multiple classifiers with respect to a single classifier, Tsai and Wu (2008)
compare single and multiple classifiers for the bankruptcy prediction. They
conclude multiple classifiers do not outperform the single ones in all cases.
In contrary, Wang et al. (2011) show that multiple classifiers could
outperform single classification algorithms. Their results suggest that three
ensemble methods, namely Bagging, Boosting, and Stacking, can improve
individual learners, like LR, Decision Trees, Artificial Neural Networks, and
SVMs. In addition, Zhang et al. (2010) introduce vertical bagging decision
trees model, which provides a combination of classifiers. SVMs are also
capable of being a hybrid component with data mining techniques, where
clustering and classification stages are performed through SVM (Chen et al.
(2012)). As another interesting research, Clustered Support Vector Machine
(CSVM) for credit scoring has been reported as an efficient and flexible
classifier method compared to other nonlinear SVM approaches (Harris (2015)).

Despite the broad range of binary approaches used, there are techniques
capable of estimating the probability of whether or not an individual would
default on their loan (Yeh and Lien (2009), Capotorti and Barbanera (2012),
Gönen et al. (2012), Kruppa et al. (2013), Antunes et al. (2017), and Feng et
al. (2018)). Default probabilities provide more comprehensive consumer
creditworthiness data than a binary or multi-category scoring. As most credit
scoring articles are concerned with classification aspects of the subject,
further study is needed to update scorecards with information on probability
of default and forecasting profitability of the credit lending business itself
(Fang and Chen (2019)). Either case, for binary classification or calculating
associated default probabilities, companies need to obtain individual
applicants’ information from commercial credit companies that gather customer
information and use a mathematical model to infer a measure of propensity to
default. By categorizing credit card users based on their usage into two
groups of transactors and revolvers, So et al. (2014) provide two separate
scores for profit maximization. As an alternative approach to default
probability estimation, Serrano-Cinca and Gutiérrez-Nieto (2016) estimate the
profitability by internal rate of return resulting in superior business
profits. The purpose of this study is to introduce one of the most promising
among recently developed statistical techniques – the Gaussian Mixture models
– for default prediction and profit maximization.

In this paper, a new classification approach for predicting default
probabilities is introduced. Our method is based on Gaussian Mixture model
(GMM) clustering. GMMs are one of the most widely used unsupervised mixture
models which are popular for their use in clustering applications (Bishop
(2006)). Using GMMs, we estimate a probability of default for each individual
customer. All customers will be placed in different clusters with respect to
their feature similarities. Upon clustering data by GMM, the ratio of good and
bad customers on each cluster can suggest a probability of bankruptcy for
members of that cluster. By using an appropriate decision threshold, according
to risk tolerance of the loan issuer, all loan applicants can be classified in
two groups. After the model is trained, given a new test sample, our model
first calculates a probability that the new sample belongs to each cluster,
and then uses the default probabilities of clusters to estimate how likely
this new sample is to default. Training data is used for finding the optimal
number of clusters, and the parameters of each cluster is estimated using
Expectation-Maximization (EM) algorithm. According to these parameters, any
sample belongs to a cluster with a given probability. As an interesting result
from a risk managerial perspective, we show that our approach can predict
Expected Loss (EL) of bad loans very well, even when the data is significantly
imbalanced. Finally, we propose an approach for calculating a minimum
probability decision threshold for a credit lender to use along with GMM-based
default probabilities for individual customers for profit maximization.

In what follows, section 2 provides the mathematical background of our
approach for predicting default probabilities, where we discuss the Gaussian
Mixture clustering, and how probabilities can be associated to these clusters.
In section 3, we show how GMM probabilities can be used to forecast EL for
consumer loan portfolios and discuss the application of our model for profit
maximization of credit institutions. Section 4 provides information on the
data we have used for our analysis and numerical results with regards to
default probabilities and EL. Finally, section 5 concludes the paper.

##  2 Methodology

In this section, we introduce our probabilistic approach for estimating
default probability of consumer loan holders. Our approach is based on
Gaussian Mixture Model which is a popular unsupervised clustering technique
but its applications in credit scoring has not been fully explored yet.

###  2.1 Gaussian Mixture Models

Gaussian Mixture models (GMM) are probability distribution functions defined
as weighted summation of a finite set of normal distributions

$p(\mathbf{x}|\boldsymbol{\theta})=\sum_{i=1}^{N_{c}}\omega_{i}\phi\left(\mathbf{x}|\boldsymbol{\mu_{i}},\boldsymbol{\Sigma}_{i}\right),$
(1)

with the restriction of $\sum_{i=1}^{N_{c}}w_{i}=1$. In our credit scoring
application, $\mathbf{x}$ is the customer’s features data, $N_{c}$ is assumed
to be the number of clusters,
$\boldsymbol{\omega}=\\{\omega_{i}\\}_{i=1}^{N_{c}}$ is the set of cluster
weights, and $\boldsymbol{\mu}=\\{\boldsymbol{\mu_{i}}\\}_{i=1}^{N_{c}}$ and
$\boldsymbol{\Sigma}=\\{\boldsymbol{\Sigma_{i}}\\}_{i=1}^{N_{c}}$ are the
means and covariance matrices of clusters, respectively. $\phi$ also indicates
a normal distribution density

${\phi\left(\mathbf{x}|\boldsymbol{\mu_{i}},\boldsymbol{\Sigma_{i}}\right)=\frac{1}{(2\pi)^{D/2}\left|\boldsymbol{\Sigma_{i}}\right|^{1/2}}}{\exp\left\\{-\frac{1}{2}\left(\mathbf{x}-\boldsymbol{\mu_{i}}\right)^{\prime}\boldsymbol{\Sigma_{i}}^{-1}\left(\mathbf{x}-\boldsymbol{\mu_{i}}\right)\right\\}}.$
(2)

Within the GMM framework, each cluster is uniquely defined with its
parameters: ${\omega_{i},\boldsymbol{\mu_{i}},\boldsymbol{\Sigma_{i}}}$.

###  2.2 Probabilistic Clustering with Gaussian Mixture Model

GMM is capable of dividing data points into clusters with respect to their
statistical features. The clusters are represented by their means, covariances
and their weights. Unlike k-means clustering, GMM provides a probabilistic
association between samples and clusters, rather than a binary classification
(Bishop (2006)). For estimating this probabilistic association, a latent
variable $\mathbf{z}=(z_{1},z_{2},\allowbreak\cdots,z_{N_{c}})$ is considered
such that $z_{i}\in\\{0,1\\}$ and $\sum_{i}{z_{i}}=1$, whereby
$p(\mathbf{x}|z_{k}=1)=\phi(\mathbf{x}|\boldsymbol{\mu_{k}},\boldsymbol{\Sigma_{k}})$.
The joint distribution can be calculated by
$p(\mathbf{x},\mathbf{z})=p(\mathbf{z})p(\mathbf{x}|\mathbf{z})$. GMM
clustering is an unsupervised approach in which the clusters are not labeled.
For credit scoring applications, our algorithm labels each GMM cluster
according to its members and their associated probabilities. For calculating
the probability of each point belonging to each cluster, we define the GMM-
based probabilities as

$p(\mathbf{x}\in
C_{j}|\mathbf{\theta})=\frac{\omega_{j}\phi(\mathbf{x}|\mathbf{\mu_{j}},\mathbf{\Sigma_{j}})}{\sum_{i=1}^{N_{c}}\omega_{i}\phi\left(\mathbf{x}|\boldsymbol{\mu_{i}},\boldsymbol{\Sigma}_{i}\right)},$
(3)

where $C_{j}$ represents the $j$th Gaussian cluster.

###  2.3 Parameter estimation

Using training data set, we calibrate GMM parameter set
$\boldsymbol{\theta}=\\{\boldsymbol{\omega},\boldsymbol{\mu},\boldsymbol{\Sigma}\\}$
to maximize the log-likelihood function $\sum_{i=1}^{n}\log
p(\mathbf{x_{i}}|\mathbf{\theta})$. Since in the case of GMM parameter
estimation, maximum likelihood does not have a closed form solution, we use
the Expectation-Maximization (EM) approach for estimating
$\boldsymbol{\theta}$. For a faster convergence of EM algorithm, we use
k-means initialization to find the first set of parameters. Then parameters
will be updated in each step of the EM algorithm according to

$\displaystyle\boldsymbol{{\mu}_{k}^{\text{ new }}}$
$\displaystyle=\frac{1}{n_{k}}\sum_{n=1}^{N}r\left(z_{nk}\right)\mathbf{{x}_{n}},\quad{\omega}_{k}^{\text{
new }}=\frac{\sum_{n=1}^{N}r\left(z_{nk}\right)}{N},$ (4)
$\displaystyle\mathbf{{\Sigma}_{k}^{\text{ new }}}$
$\displaystyle=\frac{1}{n_{k}}\sum_{n=1}^{N}r\left(z_{nk}\right)\left(\mathbf{{x}_{n}}-\boldsymbol{{\mu}_{k}^{\text{
new }}}\right)\left(\mathbf{{x}_{n}}-\boldsymbol{{\mu}_{k}^{\text{ new
}}}\right)^{\mathrm{T}},$

where $r(z_{nk})$ is the responsibility function

$r\left(z_{nk}\right)=\frac{\omega_{k}\phi\left(\mathbf{{x}_{n}}|\boldsymbol{\mu_{k}},\boldsymbol{\Sigma_{k}}\right)}{\sum_{j=1}^{N_{c}}\omega_{j}\phi\left(\mathbf{{x}_{n}}|\mathbf{\boldsymbol{\mu_{j}}},\boldsymbol{\Sigma_{j}}\right)}.$
(5)

When the log-likelihood function reaches the maximum number of iterations or
when more iterations do not improve the corresponding log-likelihood function
significantly, the algorithm terminates.

###  2.4 Probability of default for each cluster

In our model, each individual credit applicant can be associated to a given
cluster with the probability given in equation 3. Considering $n$ customers,
with the target of $\mathbf{y}=\\{y_{1},\cdots,y_{n}\\}$, with $y_{i}$ being
either 0 (bad client) or 1 (good client), we define the pay-back probability
of each cluster by

$p(y_{C_{j}}=1|\mathbf{\theta})=\frac{\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})y_{i}}{\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})},$ (6)

where $p(y_{C_{j}}=1|\mathbf{\theta})$ is the probability of being healthy in
the Gaussian cluster $C_{j}$ given $\mathbf{\theta}$. The probability of
default for each cluster can be calculated from pay-back probability by
$p(y_{C_{j}}=0|\mathbf{\theta})=1-p(y_{C_{j}}=1|\mathbf{\theta})$ as

$p(y_{C_{j}}=0|\mathbf{\theta})=\frac{\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})(1-y_{i})}{\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})}.$ (7)

###  2.5 Probability of default for individual applicants based on the
clusters probabilities

For calculating the probability of default for loan applicants in the test
set, we use cluster-wise pay-back probabilities from equation 6 across all
clusters to estimate the pay-back probability,
$p(y_{\text{new}}=1|\mathbf{\theta})$, for a new applicant from the test set

$p(y_{\text{new}}=1|\mathbf{\theta})=\frac{\sum_{j=1}^{N_{c}}p(y_{C_{j}}=1|\mathbf{\theta})p(\mathbf{x}_{\text{new}}\in
C_{j}|\mathbf{\theta})}{\sum_{j=1}^{N_{c}}p(\mathbf{x}_{\text{new}}\in
C_{j}|\mathbf{\theta})},$ (8)

where $\mathbf{x_{\text{new}}}$ is the feature vector of the customer in the
test set and $y_{\text{new}}$ represents its corresponding target. Similarly,
the default probability of $\mathbf{x_{\text{new}}}$ is calculated by
$p(y_{\text{new}}=0|\mathbf{\theta})=1-p(y_{\text{new}}=1|\mathbf{\theta})$ as

$p(y_{\text{new}}=0|\mathbf{\theta})=\frac{\sum_{j=1}^{N_{c}}p(y_{C_{j}}=0|\mathbf{\theta})p(\mathbf{x}_{\text{new}}\in
C_{j}|\mathbf{\theta})}{\sum_{j=1}^{N_{c}}p(\mathbf{x}_{\text{new}}\in
C_{j}|\mathbf{\theta})},$ (9)

Equation 9 uses the probabilistic information of all constructed clusters to
compute the default probability of a new customer. Note that in both equations
(8) and (9), the denominator is equal to one and both equations can be
simplified. However, we have kept the summation in the denominator for
clarification purposes. The relationship between cluster-wise probabilities
and new data features is displayed in Figure 1.

Figure 1: This graph shows the cluster-wise default probabilities,
$p(y_{C_{j}}=0|\mathbf{\theta})$ , as well as the probabilities that a new
sample, like $\mathbf{x}_{\text{new}}$, belongs to each cluster,
$p(\mathbf{x}_{\text{new}}\in C_{j}|\mathbf{\theta})$. The default probability
calculated for $\mathbf{x}_{\text{new}}$ is the weighted average of the
cluster-wise default probabilities weighted by the probability that
$\mathbf{x}_{\text{new}}$ belongs to $C_{j}$, namely
$\sum_{j=1}^{N_{c}}p(y_{C_{j}}=0|\mathbf{\theta})p(\mathbf{x}_{\text{new}}\in
C_{j}|\mathbf{\theta})$ (see equation (9) and note that
$\sum_{j=1}^{N_{c}}p(\mathbf{x}_{\text{new}}\in C_{j}|\mathbf{\theta})=1$).
This is worthwhile to mention that clusters are probabilistically overlapping
as each sample belongs to all clusters with a given probability. The direction
in which each cluster is spanned may also be different from others as each has
its own covariance matrix.

##  3 Applications to Risk Management

In this section, we provide applications of our probability modeling technique
for risk management of retail loans, including calculation of expected loss
and a decision threshold for the probability of default.

###  3.1 Expected loss for loan issuer

Expected loss (EL) is the summation of all possible losses that a loan issuer
may bear due to default of borrowers. EL depends not only on probability of
default, but also on the recovery rate (RR) and the Exposure at Default (EAD).
Mathematically, the expected loss calculated by a predictive model is

$\text{EL}_{\text{model}}=\sum_{i}\text{PD}_{i}\times(1-\text{R}_{i})\times\text{EAD}_{i},$
(10)

and similarly, the actual expected loss from the data is

$\text{EL}_{\text{actual}}=\sum_{i}{y_{i}\times(1-\text{R})\times\text{EAD}},$
(11)

where $R_{i}$ and $\text{EAD}_{i}$ are the RR and EAD for customer $i$. To
assess the performance of our model for predicting loss due to bad loans, we
define the relative EL error by

$\epsilon_{\text{EL}}=\frac{\text{EL}_{\text{model}}-\text{EL}_{\text{actual}}}{\text{EL}_{\text{actual}}},$
(12)

provided that the denominator is not zero. Here we will show that our
terminology above for defining default probabilities based on GMM clusters is
self-consistent and provides zero error for estimation of EL on the train set.

######  Proposition.

On a consumer loan portfolio, suppose the probability of default is given by
equation (9) and each customer has a constant loan amount and recovery rate.
Then the GMM-based model provides perfect precision in predicting the expected
loss on the train set, i.e., $\epsilon_{EL}=0$.

###### Proof.

Let us assume that constant capitals, $\text{EAD}_{i}=\text{EAD}$, and
recovery rates, $\text{R}_{i}=\text{R}$, are associated to all customers. We
will show that for the train set, EL calculated by equation (11) perfectly
matches the EL calculated by labels $\mathbf{y}=\\{y_{1},\cdots,y_{n}\\}$ in
the train set with $y_{i}$ being either 0 or 1. To see this mathematically,
note that substituting $\text{PD}_{i}$ with $p(y_{i}=0|\theta)$ from equation
(8) to equation (11), gives the model EL as

$\text{EL}_{\text{model}}=\sum_{i=1}^{n}{\sum_{j=1}^{N_{c}}p(y_{C_{j}}=0|\mathbf{\theta})p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})}\times(1-\text{R})\times\text{EAD}.$ (13)

then by substitution from equation (7), and by assuming probability
$p(y_{C_{j}}=0|\mathbf{\theta})$ is associated to all members of cluster
$C_{j}$, we have

$\displaystyle\text{EL}_{\text{model}}=$
$\displaystyle\sum_{i=1}^{n}{\sum_{j=1}^{N_{c}}\frac{\sum_{i=1}^{n}\Big{(}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})(1-y_{i})\Big{)}}{\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})}\times(1-\text{R})\times\text{EAD}$ (14)
$\displaystyle=$
$\displaystyle\sum_{j=1}^{N_{c}}\Big{(}{\frac{\sum_{i=1}^{n}\Big{(}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})(1-y_{i})\Big{)}}{\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})}}\sum_{i=1}^{n}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})\Big{)}\times(1-\text{R})\times\text{EAD}$
$\displaystyle=$
$\displaystyle\sum_{j=1}^{N_{c}}{{\sum_{i=1}^{n}\Big{(}p(\mathbf{x}_{i}\in
C_{j}|\mathbf{\theta})(1-y_{i})\Big{)}}}\times(1-\text{R})\times\text{EAD}=\text{
EL}_{\text{actual}}$

and therefore, $\epsilon_{\text{EL}}=0$ on the train data. ∎

This is not a coincidence that when R and EAD are assumed constant, this error
for the train set is $\epsilon_{\text{EL}}=0$. The reason, as shown
mathematically above, is due to the fact that GMM-based default probabilities
are calculated for each cluster according to equation (7), in such a way that
it represents how many of the individual samples are labeled as bad borrowers.
We will show a numerical example of the proposition above in section 4.4.
Since the assumption of equal EADs is not realistic, we also provide numerical
examples when EADs follow a normal distribution with a given mean and standard
deviation.

###  3.2 Profit-maximizing based on credit scoring

As lenders aim to maximize their profit from providing credit to their
customers, modern scorecard solutions are considering profit maximization into
their credit scoring architecture (Kozodoi et al. (2019)). Here in this
section, we discuss how our GMM-based credit scoring model can assist issuers
to maximize their profit. As maximizing the profit is equivalent to minimizing
the loss, we analyze both the loss and the income for the lender.

Suppose that the score of a consumer, with credit feature $\mathbf{x_{i}}$, is
$s(\mathbf{x_{i}})$. Let $M_{0}$ be the capital lent to the borrower. We
suppose $M$ is the amount the borrower is obligated to pay to the lender, and
without loss of generality and for simplicity purposes, we assume that the
interest rate’s effect on the loan amount is embedded in $M$. If the customer
defaults, the amount $MR_{i}$ is paid to the lender, where $R_{i}$ is the
recovery rate for $i^{th}$ consumer. Let $p(y=1|\mathbf{x_{i}})$ be the
conditional probability of being good given the feature set $\mathbf{x_{i}}$.
Therefore, at time $T$, when the loan must be paid back, the customer $i$ pays
the amount $\mathcal{I}(\mathbf{x_{i}})$ to the lender

$\mathcal{I}(\mathbf{x_{i}})=\left\\{\begin{array}[]{ll}0,&\text{if rejecting
$\mathbf{x_{i}}$},\\\ M,&\text{if accepting $\mathbf{x_{i}}$ and be good
customer},\\\ MR_{i},&\text{if accepting $\mathbf{x_{i}}$ and be bad
customer}.\end{array}\right.$ (15)

The mathematical expectation of the amount paid to the lender for giving loan
to the customer $\mathbf{x_{i}}$ is

$\displaystyle\mathbb{E}(\mathcal{I}(\mathbf{x_{i}})|\mathbf{x_{i}})$
$\displaystyle=Mp(y=1|\mathbf{x_{i}})$ (16)
$\displaystyle+MR_{i}(1-p(y=1|\mathbf{x_{i}})).$

The net profit is therefore
$\mathbb{E}(\mathcal{I}(\mathbf{x_{i}})|\mathbf{x_{i}})-M_{0}$ and the total
income can be calculated by

$\displaystyle\sum_{i=1}^{n}\mathbb{E}(\mathcal{I}(\mathbf{x_{i}})|\mathbf{x_{i}})$
$\displaystyle=\sum_{i=1}^{n}[Mp(y=1|\mathbf{x_{i}})$ (17)
$\displaystyle+MR_{i}(1-p(y=1|\mathbf{x_{i}}))]$
$\displaystyle=M(\sum_{i=1}^{n}[p(y=1|\mathbf{x_{i}})(1-R_{i})+R_{i}]).$

By assuming the recovery rates are constant, $R_{i}=R$ for $i=1,\dots,n$, we
can further simplify the total income as

$\sum_{i=1}^{n}\mathbb{E}(\mathcal{I}(\mathbf{x_{i}})|\mathbf{x_{i}})=nMR+M(1-R)\sum_{i=1}^{n}p(y=1|\mathbf{x_{i}}).$
(18)

Now let us define a minimum decision threshold $p_{\text{min}}$ for the lowest
probability such that only customers with $p(y=1|x_{i})\geq p_{\text{min}}$
are given loans. The expected income is

$\displaystyle\sum_{i=1}^{n}\mathbb{E}(\mathcal{I}(\mathbf{x_{i}})|\mathbf{x_{i}})$
$\displaystyle=mMR+M(1-R)(\sum_{i=1}^{m}p(y=1|\mathbf{x_{i}}))$ (19)
$\displaystyle\geq mMR+M(1-R)\sum_{i=1}^{m}p_{\text{min}}$
$\displaystyle=mM\left(R+p_{\text{min}}(1-R)\right),$

where $m$ is the number of customers with pay back probabilities higher than
$p_{\text{min}}$. The above equation guarantees a lower bound for the lender’s
income by selecting $p_{\text{min}}$.

The amount of loss for the customer $\mathbf{x_{i}}$,
$\mathcal{L}(\mathbf{x_{i}})$, is

$\mathcal{L}(\mathbf{x_{i}})=\left\\{\begin{array}[]{ll}0,&\text{if rejecting
$\mathbf{x_{i}}$},\\\ 0,&\text{if accepting $\mathbf{x_{i}}$ and be good
customer},\\\ M(1-R_{i}),&\text{if accepting $\mathbf{x_{i}}$ and be bad
customer},\end{array}\right.$ (20)

and the total expected loss for the lender from giving loan to $n$ customers,
$\mathscr{L}(n)$, is therefore

$\displaystyle\mathscr{L}(n)=$
$\displaystyle\sum_{i=1}^{n}\mathbb{E}(\mathcal{L}(\mathbf{x_{i}}))=\sum_{i=1}^{n}M(1-R_{i})(1-p(y=1|\mathbf{x_{i}}))$
(21) $\displaystyle=$ $\displaystyle
nM(1-R)-M(1-R)\sum_{i=1}^{n}p(y=1|\mathbf{x_{i}})),$

where in the last equation, we have assumed the recovery rates are constant
across all clients, $R_{i}=R$. Equation (21) provides a linear relationship
between the lender’s loss and the default probabilities. Obviously, the total
loss is a function of $n$, the number of customers applying for a loan.

By choosing the decision threshold probability, $p_{\text{min}}$, we can
consequently calculate an upper bound for the total expected loss as

$\displaystyle\mathscr{L}(n)$
$\displaystyle=mM(1-R)-M(1-R)\sum_{i=1}^{m}p(y=1|\mathbf{x_{i}}))$ (22)
$\displaystyle\leq mM(1-R)-mM(1-R)p_{\text{min}}$
$\displaystyle=mM(1-R)(1-p_{\text{min}}).$

Clearly, $m=m_{p_{\text{min}}}$ is a function of decision threshold
$p_{\text{min}}$ and is related to the upper bound of the total expected loss,
$\mathscr{L}(n)^{*}$, by the non-linear equation

$\mathscr{L}(n)^{*}=mM(1-R)(1-p_{\text{min}}).$ (23)

This non-linear function can be solved numerically to give $p_{\text{min}}$ as
a function of total expected loss

$p_{\text{min}}=\mathcal{F}(\frac{\mathscr{L}(n)^{*}}{M(1-R)}),$ (24)

Therefore, $p_{\text{min}}$ can be uniquely determined by inputs
$\mathscr{L}(n)^{*}$, $M$ and $R$. The non-linear function $\mathcal{F}$
depends on the relationship between the number of clients to whom loans are
issued, $m$, and the decision threshold, $p_{\text{min}}$. We estimate this
functional relationship by the train set data and then apply it to the test
set.

The average amount of loss for all applicants has the upper bound,
$\mathscr{L}^{\prime}=\frac{\mathscr{L}(n)^{*}}{n}$ given by

$\mathscr{L}^{\prime}=\frac{m}{n}M(1-R)(1-p_{\text{min}}).$ (25)

This equation shows the amount of loss per customer which directly depends on
the ratio of clients given a loan to all clients, $\frac{m}{n}$.

##  4 Empirical Analysis

In this section, we provide our empirical analysis on a number of credit data
sets to forecast the probability of default for each customer based on the
methodology in chapter 2. We then use our model to classify clients, calculate
the expected loss and the upper bounds for total loss based on estimated
probabilities discussed in chapter 3.

###  4.1 Data

We use a diversified group of five data sets from the UCI Repository of
Machine Learning Database including German Credit Data, Australian Credit
Approval and Japanese Credit Screening, Bank Marketing Data Set
(Telemarketing) used by Moro et al. (2014), default of credit card clients
Data Set in Taiwan used by Yeh and Lien (2009) (Dua and Graff (2017)), as well
as a large data set from the U.S. Small Business Administration (SBA) used by
Li et al. (2018) in some groups for our model.

The German Credit Data consists of 1000 customers. The Australian Credit
Approval is composed of 307 good customers and 383 bad customers, and Japanese
Credit Screening has 297 and 364 good and bad customers, respectively. The
number of instances and features in Taiwanese data set is 30000 and 24
feature. For the telemarketing data-set, we have 45211 instances and 17
attributes. In this data set, the default is one of the variables which has
been used as the target in our model. The SBA data set has 27 features and
more than 300000 of instances are included in the our study.

####  4.1.1 Data Preprocessing

We transform categorical features to numerical features before inputting into
the GMM. We use dummy variables for converting the categorical variables to
numerical features. We also use a one-hot-encoder for German data set to
convert the dummies to binary variables, increasing the number of features to
62. The German data is imbalanced, having 30% unhealthy applicants against 70%
healthy ones. In order to balance this data set, we apply a Synthetic Minority
Oversampling Technique (SMOTE). This method is also applied to other data sets
except for the Australian and Japanese data sets, because they are fairly
balanced. Note that SMOTE is only applied to the training sets. Chawla et al.
(2002) propose this method for handling the unbalanced data sets, suggesting
to draw a line between samples which are same in label, and randomly
generating new fake samples on that line. Therefore, if $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$ are two such samples with the same label, then the new
generated sample is

$\mathbf{x_{\text{generated}}}=(1-\alpha)\mathbf{x_{1}}+\alpha\mathbf{x_{2}}$
(26)

where $\alpha$ is a sample drawn from a uniform distribution
$\mathcal{U}[0,1]$. By randomly selecting $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$ and generating new samples enough times, the data set will be
balanced.

We split the data set to train and test sets with the ratio of $\frac{2}{3}$
and $\frac{1}{3}$. For data sets from SBA, because the number of instances are
fairly high, we use the ratio of 50 % for training and testing set. Using
SMOTE, the number of healthy and unhealthy instances in the training set
becomes equal. In Australian and Japanese data sets, there is no need to use
SMOTE since they are fairly balanced.

###  4.2 Estimating the number of clusters

To find the number of clusters, $N_{c}$, we use two criteria which are Akaike
Information Criterion (AIC) and Bayesian Information Criterion (BIC)

$\displaystyle AIC=$ $\displaystyle 2k-2\ln{\hat{L}},$ (27) $\displaystyle
BIC=$ $\displaystyle\ln(n)k-2\ln{\hat{L}},$

where

$\hat{L}=p(\mathbf{x}|\mathbf{\theta},N_{c})$ (28)

is the maximized value of the likelihood function of the Gaussian Mixture
model with $N_{c}$ components, $n$ is the number of the train set samples, $k$
is the number of model parameters, and $\mathbf{x}$ is a train sample. The AIC
and BIC methods give the optimal number of GMM components when they reach
their minimum level. Figure 2 shows the AIC and BIC values for different
$N_{c}$s, proposing $N_{c}=9$ for German, Japan and Australian sets,
$N_{c}=80$ for Taiwanese data, $N_{c}=55$ for telemarketing data, and
$N_{c}=28$ for SBA case is used. We also used 90, 145, and 150 number of
components for different sub-groups of SBA.

The probability of default for individuals in both train and test sets is
shown in figure 3. Our model, as figure 3 shows, is capable of distinguishing
good and bad applicants. As the train set of the German data set is
imbalanced, it is clear from panels (a) and (b) of the figure that the model
is not estimating the default probability of such samples correctly. However,
as illustrated in panels (c), (d), (e) and (f), in Japanese and Australian
data sets which are fairly balanced, the estimated probabilities of default
for the train and test sets are along the same lines.

###  4.3 Measuring probability of default

After creating $N_{c}$ clusters using the information criteria approach from
last section, the results for the German, Australian and Japanese data sets
are shown in table 1. The GMM weights represent the relative importance of
each cluster. In the forth row of each sub-table, we assign a probability
$p(\mathbf{x_{i}}\in C_{j}|\boldsymbol{\theta})$ to a given sample with
feature $\mathbf{x_{i}}$, indicating how likely $\mathbf{x_{i}}$ belongs to
the cluster $C_{j}$. The ratio associated to each cluster showing the number
of good to bad customers in that specific cluster is related to the cluster-
based probability, computed by equation 3.

For the sake of simplicity, in order to classify customers, we set a decision
threshold $D=\frac{1}{2}$. Each customer with $p(x|\theta)\geq D$ will be
labeled as $1$, and otherwise as $0$. Therefore, our probabilistic model can
also work as a binary classifier. We evaluate our model by computing the
confusion matrix and accuracy measures of precision, recall, $F1$, and the
ROC-AUC scores. The confusion matrix shows True Positives (TP), and True
Negatives (TN), in the first row, and False Positives (FP) and False Negatives
(FN) in the second row, respectively.

For a comparative analysis of our GMM-based approach, we use two other
benchmarks; Support Vector Machine (SVM), and Logistic regression (LR). In our
comparison, we use an SVM with radial basis kernel and a regularization
parameter $C=1$. Model accuracy, as the ratio of the number of true
predictions to the number of samples, are shown in table 3. Looking at the
table, it is clear that GMM, when compared to other two benchmarks, provides a
reasonable level of accuracy on both train and test sets.

The SVM approach is not able to perform as well as GMM due to the fact that it
can not separate unstructured data without normalization while GMM is a more
flexible approach which is capable of exploring directional behavior of the
data with its embedded covariance structure. In other words, in contrary to
the GMM, the SVM cannot handle the non-normalized data. Since we used the same
structure of data for all models, it affects the SVM’s accuracy. The recall,
precision, and the $F_{1}$ score for the above classifiers are shown in table
3.

###  4.4 Expected loss

The expected loss will be computed under two different conditions: first when
the loan amounts are constant for all customers, and second when the loan
amounts follow a normal distribution. For the first case, we suppose that the
recovery rate for all customers is $0.5$, and the constant amount of loan is
assumed to be $$1000$. For loans with inconstant amounts, we assume that the
retail portfolio of loans has individual loan amounts which follow a normal
distribution with a certain mean and standard deviation. More precisely, we
assume that each customer’s loan amount is independently drawn form a normal
distribution with a mean of $$1000$ and a standard deviation of $$100$. In
each case, we calculate the expected loss using our GMM-based probabilistic
model, and compare it with the actual loss for both train and test sets. As
discussed in section 3.1, in the train set when the loan amount is constant,
the actual loss and the expected loss are the same. Due to some reasonable
errors in our GMM-based probability calculation, the actual and expected loss
amounts differ on the test set. The frequency of customers’ actual loss and
predicted loss for train and test sets, with fixed and random amounts of loans
are shown in figure 8. In each panel in figure 8, the upper left quarter shows
the predicted loss with $M=1000$, and the lower left quarter demonstrates the
bar chart of actual losses with the same amount of $M$. In the upper right
quarter, the bar chart of predicted loss when $M$ is random is presented. The
bar chart of actual loss for a random $M$ is represented in the lower left
quarter. It is clear that the predicted loss for those defaulting, is less
than the actual loss, while the predicted loss for customers not defaulting is
more than $0$. GMM can properly forecast default probabilities and models the
distribution of the loss. When compared with the real outcomes of 0 or 1 in
the train set, our GMM-based model has a zero error on the train set, and
provides acceptable estimations of the expected loss on the test set.

For calculating the probability with SVM, we use the method proposed by Platt
et al. (1999), converting the SVM’s scores to probabilities using the Sigmoid
function. We use the same loan amounts and recovery rates as for the GMM-based
model to find the expected loss for SVM and LR. The actual loss and expected
loss for each model and the differences are shown in table 2. When the amount
of loan is constant, the error of predicted loss for GMM is significantly
lower than that predicted by SVM and LR, showing that the GMM is working well
with biased data in comparison to SVM and LR. When the amount of loan is
random, the GMM’s error is less than SVM and LR’s in the train set. In the
German test set, the GMM’s prediction error is significantly less than other
models. For Japanese, Australian, Taiwanese test sets the GMM’s error is also
less than SVM and LR. For the SBA datasets, however, the other models are
working better than GMM.

###  4.5 Managing the loss

In this section, we provide numerical results of our decision threshold
selection technique and show how it can be used by a credit lender to manage
the amount of the loss due to bad loans. The main objective here is to find a
decision threshold $p_{\text{min}}$ for managerial decision making to give
loans to customer with feature $\mathbf{x_{i}}$ when
$p(y=1|\mathbf{\theta})\geq p_{\text{min}}$. As shown in equation 23, the
amount of $\mathscr{L}(n)^{*}$ depends on the number of customers, loan
amounts, recovery rate and the decision threshold. In figure 9, we investigate
the relationship between $\frac{m}{n}(1-p_{\text{min}})$ and $p_{\text{min}}$,
where $m$ is the number of customers having $p(y=1|\mathbf{x_{i}})\geq
p_{\text{min}}$. We also calculate the average loss per customer as
$\mathscr{L}(n)^{\prime}=\frac{\mathscr{L}(n)^{*}}{n}$.

By assuming $M(1-R)$ is constant, $\mathscr{L}(n)^{\prime}$ is linearly
proportional to $\frac{m}{n}(1-p_{\text{min}})$. Given this value for the
train set, we can apply it for the test set and calculate the loss. Figure 9
provides a comparison on train and test sets. As shown,
$\frac{m}{n}(1-p_{\text{min}})$ for train and test sets in Australia and Japan
data sets are very close to each other, and provide an accurate prediction of
loss for test set. In the German data set, however, since the data is
imbalanced, we observe a lag between two lines of train and test sets. Giving
loans based on $p_{\text{min}}$, the actual average loss
$\mathscr{L}(n)^{\prime}$ for both train and test sets are shown in figure 10.
The accuracy of the predicted losses depends on the relationships estimated in
figure 9.

We calculate an upper bound for the loss according to equation 22, for both
train and test sets. The results are shown in figure 11. As this figure shows,
the actual loss is below the upper bound we estimated in the train set (panels
a, b and c). In the test sets (panels d, e and f), the actual loss is under
the upper bound, for decision thresholds below $p_{\text{min}}=0.8$ for
Australian and Japanese data sets, and $p_{\text{min}}=0.7$ for German data
set.

##  5 Conclusion and Future Research

We proposed a probabilistic unsupervised learning approach for predicting
default probability of consumer credit borrowers using GMM. Contrary to many
other supervised learning approaches, we have used a flexible and efficient
unsupervised learning technique. We used Gaussian Mixture clustering to group
similar samples and then associated a cluster-wise probability to each group.
From the default probability defined on the cluster level, we could derive
default probabilities for individual samples.

Our GMM-based probabilistic model has interesting applications in risk
management of retail loan portfolios. Thanks to the probabilistic nature of
our approach, we could give the user the flexibility to choose a decision
threshold as a lower probability bound for decision makers to use for credit
allocations to maximize the profit of their credit business. In addition, we
showed that using our GMM-based default probabilities for calculating the
expected loss due to bad loans, we can forecast the loss amount very well in
both train and test sets.

As there has been little work on the applications of Gaussian Mixture models
in the credit scoring literature, for future research, we think that like
other previously published supervised techniques such as neural networks,
support vector machines and decision trees, it might be interesting to see the
benefits of using hybrid GMM-based models and how their probabilistic nature
benefits the literature. Another avenue of research could pursue the
applications of GMM-based techniques in other classification problems in
business such as customer churn prediction.

## References

  * Ala’raj and Abbod [2016] Maher Ala’raj and Maysam F Abbod.  Classifiers consensus system approach for credit scoring.  _Knowledge-Based Systems_ , 104:89–105, 2016.
  * Altman et al. [2014] Edward I Altman, Malgorzata Iwanicz-Drozdowska, Erkki K Laitinen, and Arto Suvas.  distressed firm and bankruptcy prediction in an international context: A review and empirical analysis of altman’s z-score model.  _Available at SSRN 2536340_ , 2014.
  * Antunes et al. [2017] Francisco Antunes, Bernardete Ribeiro, and Francisco Pereira.  Probabilistic modeling and visualization for bankruptcy prediction.  _Applied Soft Computing_ , 60:831–843, 2017.
  * Atiya [2001] Amir F Atiya.  Bankruptcy prediction for credit risk using neural networks: A survey and new results.  _IEEE Transactions on neural networks_ , 12(4):929–935, 2001.
  * Bishop [2006] Christopher M Bishop.  _Pattern recognition and machine learning_.  springer, 2006.
  * Capotorti and Barbanera [2012] Andrea Capotorti and Eva Barbanera.  Credit scoring analysis using a fuzzy probabilistic rough set model.  _Computational Statistics & Data Analysis_, 56(4):981–994, 2012.
  * Chawla et al. [2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16:321–357, 2002.
  * Chen et al. [2012] Weimin Chen, Guocheng Xiang, Youjin Liu, and Kexi Wang.  Credit risk evaluation by hybrid data mining technique.  _Systems Engineering Procedia_ , 3:194–200, 2012.
  * Dimitras et al. [1996] Augustinos I Dimitras, Stelios H Zanakis, and Constantin Zopounidis.  A survey of business failures with an emphasis on prediction methods and industrial applications.  _European Journal of Operational Research_ , 90(3):487–513, 1996.
  * Dua and Graff [2017] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.  URL http://archive.ics.uci.edu/ml.
  * Fang and Chen [2019] Fang Fang and Yuanyuan Chen.  A new approach for credit scoring by directly maximizing the kolmogorov–smirnov statistic.  _Computational Statistics & Data Analysis_, 133:180–194, 2019.
  * Feng et al. [2018] Xiaodong Feng, Zhi Xiao, Bo Zhong, Jing Qiu, and Yuanxiang Dong.  Dynamic ensemble classification for credit scoring using soft probability.  _Applied Soft Computing_ , 65:139–151, 2018.
  * Gönen et al. [2012] Gülefşan Bozkurt Gönen, Mehmet Gönen, and Fikret Gürgen.  Probabilistic and discriminative group-wise feature selection methods for credit risk analysis.  _Expert Systems with Applications_ , 39(14):11709–11717, 2012.
  * Hand and Henley [1997] David J Hand and William E Henley.  Statistical classification methods in consumer credit scoring: a review.  _Journal of the Royal Statistical Society: Series A (Statistics in Society)_ , 160(3):523–541, 1997.
  * Hand et al. [2001] David J Hand, Hua Gui Li, and Niall M Adams.  Supervised classification with structured class definitions.  _Computational statistics & data analysis_, 36(2):209–225, 2001.
  * Harris [2015] Terry Harris.  Credit scoring using the clustered support vector machine.  _Expert Systems with Applications_ , 42(2):741–750, 2015.
  * Hsieh and Hung [2010] Nan-Chen Hsieh and Lun-Ping Hung.  A data driven ensemble classifier for credit scoring analysis.  _Expert systems with Applications_ , 37(1):534–545, 2010.
  * Huang et al. [2007] Cheng-Lung Huang, Mu-Chen Chen, and Chieh-Jen Wang.  Credit scoring with a data mining approach based on support vector machines.  _Expert systems with applications_ , 33(4):847–856, 2007.
  * Kozodoi et al. [2019] Nikita Kozodoi, Stefan Lessmann, Konstantinos Papakonstantinou, Yiannis Gatsoulis, and Bart Baesens.  A multi-objective approach for profit-driven feature selection in credit scoring.  _Decision Support Systems_ , 120:106–117, 2019.
  * Kruppa et al. [2013] Jochen Kruppa, Alexandra Schwarz, Gerhard Arminger, and Andreas Ziegler.  Consumer credit risk: Individual probability estimates using machine learning.  _Expert Systems with Applications_ , 40(13):5125–5131, 2013.
  * Lee et al. [1996] Kun Chang Lee, Ingoo Han, and Youngsig Kwon.  Hybrid neural network models for bankruptcy predictions.  _Decision Support Systems_ , 18(1):63–72, 1996\.
  * Li et al. [2018] Min Li, Amy Mickel, and Stanley Taylor.  “should this loan be approved or denied?”: A large dataset with class assignment guidelines.  _Journal of Statistics Education_ , 26(1):55–66, 2018.
  * Mancisidor et al. [2020] Rogelio A Mancisidor, Michael Kampffmeyer, Kjersti Aas, and Robert Jenssen.  Deep generative models for reject inference in credit scoring.  _Knowledge-Based Systems_ , page 105758, 2020.
  * Min and Lee [2005] Jae H Min and Young-Chan Lee.  Bankruptcy prediction using support vector machine with optimal choice of kernel function parameters.  _Expert systems with applications_ , 28(4):603–614, 2005.
  * Moro et al. [2014] Sérgio Moro, Paulo Cortez, and Paulo Rita.  A data-driven approach to predict the success of bank telemarketing.  _Decision Support Systems_ , 62:22–31, 2014.
  * Ohlson [1980] James A Ohlson.  Financial ratios and the probabilistic prediction of bankruptcy.  _Journal of accounting research_ , pages 109–131, 1980.
  * Platt et al. [1999] John Platt et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 10(3):61–74, 1999.
  * Salcedo-Sanz et al. [2005] Sancho Salcedo-Sanz, José-Luis Fernández-Villacañas, Maria Jesus Segovia-Vargas, and Carlos Bousono-Calzon.  Genetic programming for the prediction of insolvency in non-life insurance companies.  _Computers & Operations Research_, 32(4):749–765, 2005.
  * Serrano-Cinca and Gutiérrez-Nieto [2016] Carlos Serrano-Cinca and Begoña Gutiérrez-Nieto.  The use of profit scoring as an alternative to credit scoring systems in peer-to-peer (p2p) lending.  _Decision Support Systems_ , 89:113–122, 2016.
  * Sinha and Zhao [2008] Atish P Sinha and Huimin Zhao.  Incorporating domain knowledge into data mining classifiers: An application in indirect lending.  _Decision Support Systems_ , 46(1):287–299, 2008\.
  * So et al. [2014] Mee Chi So, Lyn C Thomas, Hsin-Vonn Seow, and Christophe Mues.  Using a transactor/revolver scorecard to make credit and pricing decisions.  _Decision Support Systems_ , 59:143–151, 2014.
  * Tsai and Wu [2008] Chih-Fong Tsai and Jhen-Wei Wu.  Using neural network ensembles for bankruptcy prediction and credit scoring.  _Expert systems with applications_ , 34(4):2639–2649, 2008.
  * Veganzones and Séverin [2018] David Veganzones and Eric Séverin.  An investigation of bankruptcy prediction in imbalanced datasets.  _Decision Support Systems_ , 112:111–124, 2018.
  * Wang et al. [2011] Gang Wang, Jinxing Hao, Jian Ma, and Hongbing Jiang.  A comparative assessment of ensemble learning for credit scoring.  _Expert systems with applications_ , 38(1):223–230, 2011.
  * Wu et al. [2007] Chih-Hung Wu, Gwo-Hshiung Tzeng, Yeong-Jia Goo, and Wen-Chang Fang.  A real-valued genetic algorithm to optimize the parameters of support vector machine for predicting bankruptcy.  _Expert systems with applications_ , 32(2):397–408, 2007.
  * Xiao et al. [2020] Jin Xiao, Xu Zhou, Yu Zhong, Ling Xie, Xin Gu, and Dunhu Liu.  Cost-sensitive semi-supervised selective ensemble model for customer credit scoring.  _Knowledge-Based Systems_ , 189:105118, 2020.
  * Yeh and Lien [2009] I-Cheng Yeh and Che-hui Lien.  The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients.  _Expert Systems with Applications_ , 36(2):2473–2480, 2009.
  * Zhang et al. [2010] Defu Zhang, Xiyue Zhou, Stephen CH Leung, and Jiemin Zheng.  Vertical bagging decision trees model for credit scoring.  _Expert Systems with Applications_ , 37(12):7838–7843, 2010.
  * Zhang and Härdle [2010] Junni L Zhang and Wolfgang K Härdle.  The bayesian additive classification tree applied to credit risk modelling.  _Computational Statistics & Data Analysis_, 54(5):1197–1205, 2010.

## Appendix

##  Appendix A Numberical Results on German, Japenese and Australian Data Sets

(a) Germany

(b) Japan

(c) Australia

(d) Telemarketing

(e) Taiwanese

(f) SBAcase

(g) SBA 1

(h) SBA 2

(i) SBA3

Figure 2: AIC and BIC measures for optimal number of components (horizontal axis: the number of Gaussian components, vertical axis: the information criteria). Table 1: Clusters’ features in the German, Japanese, and Australian Data sets German Data Set | Clusters
---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Train | Good Customers | 31 | 20 | 2 | 2 | 76 | 9 | 20 | 31 | 267
Bad Customers | 20 | 26 | 202 | 9 | 26 | 18 | 16 | 23 | 118
Ratio | 1.55 | 0.77 | 0.01 | 0.22 | 2.92 | 0.50 | 1.25 | 1.35 | 2.26
Probability | 0.61 | 0.43 | 0.01 | 0.18 | 0.75 | 0.33 | 0.56 | 0.57 | 0.69
Weight | 0.06 | 0.05 | 0.22 | 0.01 | 0.11 | 0.03 | 0.04 | 0.06 | 0.42
Test | Good Customers | 2 | 0 | 1 | 0 | 72 | 0 | 0 | 6 | 161
Bad Customers | 1 | 0 | 0 | 0 | 39 | 0 | 0 | 0 | 52
Ratio | 2.00 | – | – | – | 1.85 | – | – | – | 3.10
Japanese Data Set | Clusters
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Train | Good Customers | 11 | 1 | 147 | 3 | 1 | 18 | 7 | 2 | 16
Bad Customers | 40 | 0 | 24 | 0 | 0 | 42 | 0 | 0 | 128
Ratio | 0.28 | – | 6.13 | – | – | 0.43 | – | – | 0.13
Probability | 0.21 | 1.00 | 0.86 | 1.00 | 1.00 | 0.30 | 1.00 | 1.00 | 0.11
Weight | 0.12 | 0.00 | 0.39 | 0.01 | 0.00 | 0.14 | 0.02 | 0.00 | 0.33
Test | Good Customers | 3 | 0 | 72 | 0 | 0 | 7 | 0 | 0 | 9
Bad Customers | 24 | 0 | 16 | 0 | 0 | 18 | 0 | 0 | 72
Ratio | 0.13 | – | 4.50 | – | – | 0.39 | – | – | 0.13
Australian Data Set | Clusters
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
Train | Good Customers | 40 | 2 | 2 | 4 | 10 | 19 | 2 | 7 | 120
Bad Customers | 118 | 0 | 0 | 0 | 4 | 114 | 0 | 7 | 11
Ratio | 0.34 | – | – | – | 2.50 | 0.17 | – | 1.00 | 10.91
Probability | 0.25 | 1.00 | 1.00 | 1.00 | 0.71 | 0.14 | 1.00 | 0.50 | 0.92
Weight | 0.34 | 0.00 | 0.00 | 0.01 | 0.03 | 0.29 | 0.00 | 0.03 | 0.28
Test | Good Customers | 20 | 0 | 0 | 0 | 2 | 13 | 0 | 0 | 66
Bad Customers | 54 | 0 | 0 | 0 | 1 | 62 | 0 | 0 | 12
Ratio | 0.37 | – | – | – | 2.00 | 0.21 | – | – | 5.50

(a) Telemarketing Train set

(b) Telemarketing Test Set

(c) Taiwanese Train set

(d) Taiwanese Test Set

(e) SBA-case Train set

(f) SBA-case Test Set

Figure 3: The probability of default for individuals Telemarketing, Taiwanese
and SBA-case data sets.

(a) Germany Train set

(b) Germany Test Set

(c) Australia Train set

(d) Australia Test Set

(e) Japan Train set

(f) Japan Test Set

Figure 4: The probability of default for individuals German, Australian and
Japanese data sets.

(a) SBA (first 100 k selected group) Train set

(b) SBA (first 100 k selected group) Test Set

(c) SBA (second 100 k selected group) Train set

(d) SBA (second 100 k selected group) Test Set

(e) SBA (third 100 k selected group) Train set

(f) SBA (third 100 k selected group) Test Set

Figure 5: The probability of default for individuals German, Australian and
Japanese data sets.

(a) German train data

(b) German test data

(c) Japanese train data

(d) Japanese test data

(e) Australian train data

(f) Australian test data

Figure 6: The amount of actual loss versus predicted loss.

(a) Telemarketing train data

(b) Telemarketing test data

(c) SBA train data

(d) SBA test data

(e) Taiwanese train data

(f) Taiwanese test data

Figure 7: The amount of actual loss versus predicted loss.

(a) SBAk1 train data

(b) SBAk1 test data

(c) SBAk2 train data

(d) SBAk2 test data

(e) SBAk2 train data

(f) SBAk3 test data

Figure 8: The amount of actual loss versus predicted loss. Table 2: Actual and
predicted loss for different models and their errors

$M=1000$ fixed (Train Set) |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
| Total |  Actual |  Predicted Loss | Relative |  Predicted Loss | Relative |  Predicted Loss | Relative
| Capital | Loss | (GMM) | Error | (SVM) | Error | (LR) | Error
Germany | 916,000 | 229,000 | 229,000 | 0.00% | 228,252 | 0.08% | 226,144 | 0.31%
Japan | 440,000 | 117,000 | 117,000 | 0.00% | 115,716 | 0.29% | 112,961 | 0.92%
Australia | 460,000 | 127,000 | 127,000 | 0.00% | 125,412 | 0.35% | 129,779 | 0.60%
Taiwan | 31,200,000 | 7,800,000 | 7,800,000 | 0.00% | 7,817,018 | 0.05% | 7,805,297 | 0.02%
SBA | 1,868,000 | 467,000 | 467,000 | 0.00% | 458,198 | 0.47% | 467,000 | 0.00%
Telemarketing | 43,470,000 | 10,867,500 | 10,867,500 | 0.00% | 10,822,793 | 0.10% | 10,869,983 | 0.01%
SBA 50k1 | 82,446,000 | 20,611,500 | 20,611,500 | 0.00% | 16,390,771 | 5.12% | 20,618,452 | 0.01%
SBA 50k2 | 82,526,000 | 20,631,500 | 20,631,500 | 0.00% | 12,538,894 | 9.81% | 20,637,919 | 0.01%
SBA 50k3 | 82,438,000 | 20,609,500 | 20,609,500 | 0.00% | 15,174,912 | 6.59% | 20,615,872 | 0.01%
$M=1000$ fixed (Train Set) |  |  |  |  |  |  |  |
| Total |  Actual |  Predicted Loss | Relative |  Predicted Loss | Relative |  Predicted Loss | Relative
| Capital | Loss | (GMM) | Error | (SVM) | Error | (LR) | Error
Germany | 334,000 | 46,000 | 49,144 | 0.94% | 83,527 | 11.24% | 55,510 | 2.85%
Japan | 221,000 | 65,000 | 61,346 | 1.65% | 57,749 | 3.28% | 57,762 | 3.28%
Australia | 230,000 | 64,500 | 63,364 | 0.49% | 62,967 | 0.67% | 66,799 | 1.00%
Taiwan | 9,999,000 | 1,117,500 | 2,008,574 | 8.91% | 2,345,955 | 12.29% | 2,378,428 | 12.61%
SBA | 699,000 | 112,000 | 135,868 | 3.41% | 122,963 | 1.57% | 111,860 | 0.02%
Telemarketing | 13,727,000 | 1,437,500 | 3,103,969 | 12.14% | 3,296,921 | 13.55% | 3,122,712 | 12.28%
SBA 50k1 | 50,000,000 | 4,466,500 | 10,868,920 | 12.80% | 10,013,849 | 11.09% | 10,332,796 | 11.73%
SBA 50k2 | 50,000,000 | 4,400,000 | 10,847,613 | 12.90% | 7,649,808 | 6.50% | 10,528,968 | 12.26%
SBA 50k3 | 50,000,000 | 4,397,000 | 11,006,260 | 13.22% | 9,285,423 | 9.78% | 10,434,015 | 12.07%
$M\sim\mathcal{N}(1000,100)$ (Train Set) |  |  |  |  |  |  |  |
| Total |  Actual |  Predicted Loss | Relative |  Predicted Loss | Relative |  Predicted Loss | Relative
| Capital | Loss | (GMM) | Error | (SVM) | Error | (LR) | Error
| (Approixmate) |  |  |  |  |  |  |
Germany | 916,000 | 228,345 | 228,366 | 0.00% | 227,496 | 0.09% | 225,407 | 0.32%
Japan | 440,000 | 117,453 | 117,382 | 0.02% | 116,099 | 0.31% | 113,370 | 0.93%
Australia | 460,000 | 127,285 | 127,314 | 0.01% | 125,865 | 0.31% | 130,140 | 0.62%
Taiwan | 31,200,000 | 7,786,223 | 7,786,399 | 0.00% | 7,817,018 | 0.10% | 7,805,297 | 0.06%
SBA | 1,868,000 | 467,176 | 466,700 | 0.03% | 458,409 | 0.47% | 467,322 | 0.01%
Telemarketing | 43,470,000 | 10,869,859 | 10,869,292 | 0.00% | 10,822,516 | 0.11% | 10,870,556 | 0.00%
SBA 50k1 | 82,446,000 | 20,635,076 | 20,625,158 | 0.01% | 16,395,034 | 5.14% | 20,627,681 | 0.01%
SBA 50k2 | 82,526,000 | 20,629,775 | 20,628,724 | 0.00% | 12,539,906 | 9.80% | 20,633,195 | 0.00%
SBA 50k3 | 82,438,000 | 20,607,390 | 20,602,555 | 0.01% | 14,967,292 | 6.84% | 20,252,249 | 0.43%
$M\sim\mathcal{N}(1000,100)$ (Test Set) |  |  |  |  |  |  |  |
| Total |  Actual |  Predicted Loss | Relative |  Predicted Loss | Relative |  Predicted Loss | Relative
| Capital | Loss | (GMM) | Error | (SVM) | Error | (LR) | Error
| (Approixmate) |  |  |  |  |  |  |
Germany | 334,000 | 46,214 | 49,084 | 0.86% | 83,452 | 11.15% | 55,629 | 2.82%
Japan | 221,000 | 65,086 | 61,994 | 1.40% | 58,370 | 3.04% | 58,217 | 3.11%
Australia | 230,000 | 64,196 | 63,187 | 0.44% | 62,669 | 0.66% | 66,643 | 1.06%
Taiwan | 9,999,000 | 1,119,712 | 2,011,281 | 8.92% | 2,348,869 | 12.29% | 2,380,581 | 12.61%
SBA | 699,000 | 112,197 | 135,952 | 3.40% | 123,251 | 1.58% | 112,040 | 0.02%
Telemarketing | 13,727,000 | 59,779 | 154,197 | 0.69% | 3,302,660 | 23.62% | 3,127,802 | 22.35%
SBA 50k1 | 50,000,000 | 4,464,741 | 10,869,494 | 12.81% | 10,011,965 | 11.09% | 10,330,776 | 11.73%
SBA 50k2 | 50,000,000 | 4,403,849 | 10,851,358 | 12.90% | 7,656,332 | 6.50% | 10,537,883 | 12.27%
SBA 50k3 | 50,000,000 | 4,392,787 | 10,997,210 | 13.21% | 9,280,042 | 9.77% | 10,424,647 | 12.06%

(a) Australia

(b) Japan

(c) Germany

(d) Telemarketing

(e) Taiwan

(f) SBA-case

(g) SBA (first 100 k)

(h) SBA (second 100 k)

(i) SBA (third 100 k)

Figure 9: The relationship between decision threshold and
$\frac{m}{n}(1-p_{\text{min}})$

(a) Australia

(b) Japan

(c) Germany

(d) Telemarketing

(e) Japan

(f) Germany

(g) SBA (first 100 k selected)

(h) SBA (second 100 k selected)

(i) SBA (third 100 k selected)

Figure 10: The relationship between loss and $p_{\text{min}}$

(a) Australia

(b) Japan

(c) Germany

(d) Australia

(e) Japan

(f) Germany

Figure 11: The upper bound for the loss function using different decision
thresholds (panels a, b and c represent the train set, while panels d, e and f
show the test set results)

(a) Telemarketing

(b) Taiwan

(c) SBA-case

(d) Telemarketing

(e) Taiwan

(f) SBA-case

Figure 12: The upper bound for the loss function using different decision
thresholds (panels a, b and c represent the train set, while panels d, e and f
show the test set results)

(a) SBA 1

(b) SBA 2

(c) SBA 3

(d) SBA 1

(e) SBA 2

(f) SBA 3

Figure 13: The upper bound for the loss function using different decision
thresholds (panels a, b and c represent the train set, while panels d, e and f
show the test set results) Table 3: Comparison of GMM’s performance with SVM
and LR through different accuracy measurements

Train | Accuracy | Precision score | recall score | F1 score | ROC AUC score
---|---|---|---|---|---
GMM | SVM | LR | GMM | SVM | LR | GMM | SVM | LR | GMM | SVM | LR | GMM | SVM | LR
Germany | 0.74 | 0.55 | 0.82 | 0.68 | 0.53 | 0.82 | 0.93 | 0.83 | 0.82 | 0.78 | 0.65 | 0.82 | 0.74 | 0.56 | 0.89
Japan | 0.84 | 0.66 | 0.83 | 0.87 | 0.82 | 0.82 | 0.78 | 0.34 | 0.80 | 0.82 | 0.48 | 0.81 | 0.84 | 0.69 | 0.90
Australia | 0.82 | 0.67 | 0.85 | 0.87 | 0.80 | 0.87 | 0.71 | 0.34 | 0.77 | 0.78 | 0.48 | 0.82 | 0.81 | 0.70 | 0.92
Taiwan | 0.73 | 0.63 | 0.62 | 0.72 | 0.69 | 0.66 | 0.76 | 0.49 | 0.50 | 0.74 | 0.57 | 0.57 | 0.79 | 0.69 | 0.67
SBA | 0.85 | 0.88 | 0.99 | 0.98 | 0.97 | 0.99 | 0.71 | 0.78 | 0.99 | 0.82 | 0.86 | 0.99 | 0.94 | 0.98 | 1.00
Telemarketing | 0.65 | 0.60 | 0.68 | 0.65 | 0.68 | 0.68 | 0.64 | 0.37 | 0.67 | 0.65 | 0.48 | 0.68 | 0.71 | 0.65 | 0.74
SBA 50k1 | 0.68 | 0.44 | 0.78 | 0.72 | 0.46 | 0.82 | 0.59 | 0.78 | 0.73 | 0.65 | 0.58 | 0.77 | 0.75 | 0.44 | 0.82
SBA 50k2 | 0.69 | 0.47 | 0.78 | 0.72 | 0.49 | 0.82 | 0.62 | 0.92 | 0.73 | 0.67 | 0.64 | 0.77 | 0.76 | 0.52 | 0.82
SBA 50k3 | 0.68 | 0.46 | 0.78 | 0.70 | 0.47 | 0.82 | 0.62 | 0.84 | 0.72 | 0.66 | 0.61 | 0.77 | 0.75 | 0.41 | 0.82
Test | Accuracy | Precision score | recall score | F1 score | ROC AUC score
GMM | SVM | LR | GMM | SVM | LR | GMM | SVM | LR | GMM | SVM | LR | GMM | SVM | LR
Germany | 0.72 | 0.67 | 0.74 | 0.72 | 0.76 | 0.82 | 1.00 | 0.80 | 0.83 | 0.84 | 0.78 | 0.82 | 0.50 | 0.59 | 0.77
Japan | 0.84 | 0.69 | 0.84 | 0.82 | 0.74 | 0.80 | 0.79 | 0.37 | 0.81 | 0.80 | 0.50 | 0.81 | 0.83 | 0.73 | 0.87
Australia | 0.80 | 0.66 | 0.82 | 0.84 | 0.77 | 0.85 | 0.67 | 0.33 | 0.71 | 0.75 | 0.46 | 0.77 | 0.79 | 0.70 | 0.88
Taiwan | 0.74 | 0.55 | 0.56 | 0.86 | 0.88 | 0.87 | 0.79 | 0.49 | 0.50 | 0.82 | 0.63 | 0.64 | 0.73 | 0.68 | 0.67
SBA | 0.80 | 0.85 | 0.99 | 0.99 | 0.99 | 0.99 | 0.71 | 0.79 | 1.00 | 0.83 | 0.88 | 0.99 | 0.94 | 0.99 | 1.00
Telemarketing | 0.62 | 0.47 | 0.67 | 0.85 | 0.90 | 0.89 | 0.63 | 0.37 | 0.66 | 0.73 | 0.52 | 0.76 | 0.65 | 0.67 | 0.73
SBA 50k1 | 0.62 | 0.66 | 0.75 | 0.91 | 0.80 | 0.95 | 0.60 | 0.78 | 0.73 | 0.72 | 0.79 | 0.83 | 0.73 | 0.44 | 0.81
SBA 50k2 | 0.63 | 0.76 | 0.75 | 0.91 | 0.82 | 0.95 | 0.62 | 0.92 | 0.73 | 0.74 | 0.86 | 0.82 | 0.74 | 0.52 | 0.82
SBA 50k3 | 0.64 | 0.70 | 0.74 | 0.91 | 0.81 | 0.95 | 0.62 | 0.84 | 0.72 | 0.74 | 0.82 | 0.82 | 0.73 | 0.42 | 0.82