# Cost-Sensitive Stacking: an Empirical Evaluation

Natalie Lawrance ID (✉) Email<EMAIL_ADDRESS>Department of Business
Technology and Operations, Vrije Universiteit Brussel (VUB), Brussels, Belgium
Marie-Anne Guerry ID Department of Business Technology and Operations, Vrije
Universiteit Brussel (VUB), Brussels, Belgium  George Petrides ID Department
of Mathematics and Statistics, University of Cyprus, Nicosia, Cyprus

###### Abstract

Many real-world classification problems are cost-sensitive in nature, such
that the misclassification costs vary between data instances. Cost-sensitive
learning adapts classification algorithms to account for differences in
misclassification costs. Stacking is an ensemble method that uses predictions
from several classifiers as the training data for another classifier, which in
turn makes the final classification decision.

While a large body of empirical work exists where stacking is applied in
various domains, very few of these works take the misclassification costs into
account. In fact, there is no consensus in the literature as to what cost-
sensitive stacking is. In this paper we perform extensive experiments with the
aim of establishing what the appropriate setup for a cost-sensitive stacking
ensemble is. Our experiments, conducted on twelve datasets from a number of
application domains, using real, instance-dependent misclassification costs,
show that for best performance, both levels of stacking require cost-sensitive
classification decision.

##### Keywords

Cost-sensitive learning, classification, ensemble learning, stacked
generalization, stacking, blending

##  1 Introduction

Cost-sensitive learning is relevant in many real-world classification
problems, where different misclassification errors incur different costs. A
prominent example is the field of medicine, where misdiagnosing an ill patient
for a healthy one (a false negative) entails delayed treatment and potentially
life-threatening consequences, while an error in the opposite direction (a
false positive) would incur unnecessary medical examination costs and stress
for the patient. Cost-sensitive classifiers can account for the differences in
costs not only between different classes, but also between data instances,
making instance-dependent cost-sensitive classification decisions.

Many cost-sensitive classifiers employ ensemble methods, which combine
predictions from several classifiers to obtain better generalisation
performance. Superiority of ensembles over individual classifiers is very well
known and has been extensively studied ([8, 37]). Most cost-sensitive
classification ensembles are homogeneous in nature, meaning their components
are instantiated using the same learning algorithm.

_Stacked generalization_ or _stacking_ [31] is a well known and widely applied
heterogeneous ensemble, where the predictions of classifiers produced by
different learning algorithms (the _base-learners_) are used as training
inputs to another learning algorithm (the _meta-learner_) to produce a _meta
classifier_ , which makes the final classification decision. In the
literature, the _base-_ and _meta-_ levels of stacking are also referred to
_level-0_ and _level-1_.

Homogeneous cost-sensitive ensembles such as cost-sensitive boosting and
bagging are widely studied and have been shown very successful [25]. Examples
of cost-sensitive stacking, on the other hand, are scarce and unsystematic,
representing for the most part applications to single domains, where the
classifiers are trained on synthetic, class-dependent costs and are evaluated
with cost-insensitive performance metrics. For a discussion on the importance
of real costs for a proper evaluation see the work by [25]. In fact, there is
currently no consensus as to how a cost-sensitive stacking ensemble is to be
composed and at what stage (level-0 or level-1) cost-sensitive decision-making
should be used. This can be clearly seen in Table 1, which gives an overview
of existing cost-sensitive stacking literature. Stacking is typically made
cost-sensitive simply through the application of a cost-sensitive classifier
either at level-0 (CS-CiS), level-1 (CiS-CS) or at both levels of the the
ensemble (CS-CS), resulting in a total of three possible stacking setups. To
the best of our knowledge, no comparison of all three setups on multiple
domains with appropriate evaluation exists in the literature. Previous related
work used arbitrary artificial costs in model training and evaluated cost-
sensitive models using performance metrics that are either cost-invariant or
that focus on the performance of only the positive class.

In this work we aim to fill this gap by providing a thorough comparison of
various cost-sensitive stacking ensembles on multiple domains using real,
instance-dependent costs and performance metrics appropriate for cost-
sensitive problems.

###  1.1 Our contributions

  * •

The main contribution of this work is a rigorous empirical comparison of
different setups of cost-sensitive stacking ensembles over multiple domains.
We evaluate using appropriate performance metrics and attempt to establish
best practice.

  * •

Secondly, we introduce a novel cost-sensitive classifier combination method,
inspired by MEC-voting and stacking, which we call _MEC-weighted-stacking_.

  * •

Finally, we present a list of publicly available datasets with clearly defined
instance-dependent misclassification costs. The costs are based either on the
literature, or are defined by us based on both the literature and expert
knowledge of the data providers. We also define instance-dependent costs for a
well known ‘credit-g’ dataset from the UCI Machine learning repository, for
which only class-dependent costs were available to date.

###  1.2 Outline

The remainder of the paper is structured as follows. Section 2 presents an
overview of the relevant literature. MEC-weighted stacking is introduced in
Section 3. Our hypotheses to be tested, the experimental setup and the
datasets used in the study are discussed in Section 4. Section 5 details the
results of our extensive experiments, while the main outcomes and limitations
are discussed in Section 6. Section 7 concludes the paper.

##  2 Related work

While stacking has been widely used in machine learning applications (the
interested reader is invited to peruse the survey on stacking literature by
[27]), few works are dedicated to the study of cost-sensitive stacking.

We identified in the literature three different cost-sensitive stacking
setups: CiS-CS, CS-CiS or CS-CS, where the ensemble was made cost-sensitive
simply through the application of a cost-sensitive classifier either at
level-0, level-1 or at both levels of the ensemble. In most cases, the method
used to make the classification cost-sensitive is the direct cost-sensitive
decision as introduced by [35], also called DMECC [25].

One of the first papers to discuss stacking in a cost-sensitive context was
[6]. The authors propose cost-insensitive level-0 and cost-sensitive level-1
stacking setup (_CiS-CS setup_), which was compared to a number of different
classifier combinations schemes on 16 classification problems. The
misclassification costs they used were artificially generated by randomly and
uniformally sampling costs from on the interval $[1,10]$. Several other
studies followed adopting the same CiS-CS stacking setup, however none of the
studies explicitly reasoned or justified this choice.

Table 1: Summary of cost-sensitive stacking literature

Publication Stacking Level-0 Level-1 Real Costs CS setup algorithm algorithm
costs type evaluation [6] CiS-CS DT, KNN, NB LR c $\checkmark$ [19] CS-CiS DT,
KNN, NB MT c [5] CS-CS DT LR $\checkmark$ i $\checkmark$ [34] CiS-CS DT, KNN,
NB LR c [7] CiS-CS ExT, GBDT, LDA, LR, RF LR c [33] CiS-CS DT, KNN, RF, SVM
DT, KNN, NB, SVM c [13] CiS-CS, CS-CiS, CS-CS DT, NB, KNN, SVM LR, ExT c
$\checkmark$ _this paper_ CiS-CS, CS-CiS, CS-CS DT, KNN, LR, SVM Adab, DT,
KNN, LR, RF, SVM $\checkmark$ i $\checkmark$ _Costs type:_ c: class-dependent,
i: instance-dependent _Algorithms:_ Adab: Adaboost, DT:decision tree, ExT:
extremely randomised trees, GBDT: gradient boosted trees, KNN: k-nearest
neighbour, LDA: linear discriminant, LR: logistic regression, MT: Meta
Decision Trees, NB: naive bayes, RF: random forest, SVM: support vector
machines

Several more papers demonstrated similar examples of multiple-domain studies
of CiS-CS stacking with arbitrary costs ( [7, 33, 34]). These mainly differ in
the type and the number of algorithms that are employed in the ensemble. We
note that all of them used cost-insensitive metrics for classifier evaluation.

[19] considers a stacking setup, where level-0 classifiers were cost-sensitive
while level-1 was cost-insensitive (_CS-CiS setup_). The misclassification
costs were assumed to be equal to the inverse of the class priors. This
approach is very commonly adopted in the absence of information about real
misclassification costs. It is, however, not appropriate, see [25] for a
discussion. The resulting stacking classifier was compared to known ensemble
methods using classification accuracy, a metric that by design assumes equal
misclassification costs.

Most examples of stacking use different learning algorithms in level-0,
however in his original work Wolpert suggested that this must not be the case
and the technique can also be applied when a single algorithm is considered.
[5] propose a cost-sensitive variant of _bag-stacking_ , a method originally
proposed by [29], using bagged cost-sensitive decision trees in level-0 and
using cost-sensitive logistic regression in level-1, thus implicitly proposing
a _CS-CS_ stacking setup. To the best of our knowledge, this study is the only
example where real instance-dependent costs were used in model training.
Models were evaluated using a cost-sensitive metric called the savings score,
proposed in [2].

The only study to date that considers all three different cost-sensitive
stacking setups is one by [13] on the application domain of software defect
prediction. The misclassification costs were selected based on a literature
however the authors emphasised that they treated costs as one of the
hyperparameters of the classifier, which, we must note, is incorrect, as was
previously discussed in [15]. The experiments are run on 15 datasets using the
same class-dependent cost matrix on all. Balanced error-based metrics were
used for evaluation together with cost-based evaluation metrics.

Identifying real misclassification costs is a complex task, which for many
applications may prove too difficult to define and compute. Most studies
resort to artificially generated misclassification costs (see [25] for a
discussion on why this is inappropriate) and error-based evaluation metrics
are typically employed to assess generalisation performance of cost-sensitive
stacking. Examples of metrics used include the AUC, the arithmetic or
geometric mean of class-specific accuracies, the F-measure, and the Matthew’s
correlation coefficient (MCC). All of these metrics assume equal
misclassification costs, and the F-measure does not incorporate the
performance on the negative class, so using these metrics is not compatible
with cost-sensitive learning [17].

One of the challenges of stacking is the choice of the learning algorithms for
the ensemble. Earlier studies proposed to use linear regression to combine
level-0 inputs [30], however Wolpert does not impose any particular
restrictions on which algorithm to use in level-1, and he believed that his
famous ‘No Free Lunch Theorem’ [32] applies to the meta-learner as well. For
the overview of which learning algorithms were used in cost-sensitive stacking
ensembles to date we refer our reader to the summary Table 1.

##  3 MEC-weighted stacked generalization

In the typical supervised classification framework, a learning algorithm $A$
is presented with a set $\mathcal{S}$ of data instances
$(\textbf{x}_{i},y_{i})$, each describing some object $i$. We call
$\textbf{x}_{i}$ a feature vector, and $y_{i}$ the class label of that object,
drawn from a finite, discrete set of classes $\\{1,\ldots,K\\}$. In this paper
we will consider the binary classification problem, where $y_{i}\in\\{0,1\\}$.
The learning algorithm $A$, given $\mathcal{S}$ as input, after a process
called training, produces a classifier $C$, whose task is to predict the
correict class label $\hat{y}_{C}(\textbf{x}_{j})\in\\{0,1\\}$ for a
previously unseen feature vector $\textbf{x}_{j}$.

Training any number $L$ of learning algorithms on the same set of data
instances $\mathcal{S}$, we obtain a set of classifiers
$\mathcal{C}=\\{C_{1}\ldots C_{L}\\}$, and for each feature vector
$\textbf{x}_{i}$ the corresponding set of predictions
$\hat{\mathcal{Y}}(\textbf{x}_{i})=\\{\hat{y}_{C_{1}}(\textbf{x}_{i}),\ldots,\hat{y}_{C_{L}}(\textbf{x}_{i})\\}$.
$C$ is called _an ensemble of classifiers_ if the predictions from
$\hat{\mathcal{Y}}(\textbf{x}_{i})$ are combined, in some way, into a single
prediction of the class label for the data instance $\textbf{x}_{i}$.

Stacking differs from other classifier ensembles in that the predictions from
the set $\hat{\mathcal{Y}}(\textbf{x}_{i})$ are combined with the original
class label $y_{i}$ to form the set
$\mathcal{S}_{meta}=\\{(\hat{y}_{C_{1}}(\textbf{x}_{i}),\ldots,\hat{y}_{C_{L}}(\textbf{x}_{i})),y_{i}\\}$
of meta level data instances subsequently used in another round of algorithm
training to produce a new classifier, which is used to obtain the final
predictions.

The novel method we propose in this paper is inspired by the cost-sensitive
weights for model votes paradigm described in [25], and consequently called
_MEC-weighted stacking_. To each classifier $C$, we can assign a weight
$w_{C}$ based on that classifier’s cost-performance on the validation set:
$w_{C}=f(\epsilon)$, where $\epsilon$ is the sum of the misclassification
costs of all data instances incorrectly classified by $C$ on a validation set
and $f(\epsilon)$ is a transformation function, which for example can take one
of the following forms: $f(\epsilon)=\text{ln}((1-\epsilon)/\epsilon)$,
$f(\epsilon)=1-\epsilon$, $f(\epsilon)=\exp((1-\epsilon)/\epsilon)$, and
$f(\epsilon)=((1-\epsilon)/\epsilon)^{2}$.

The general stacking procedure is thus modified with the additional step of
collecting the MEC-weights for each of the predictions from the set
$\hat{\mathcal{Y}}(\textbf{x}_{i})$, yielding the weighted set of predictions
$\hat{\mathcal{Y}}_{MEC}(\textbf{x}_{i})=\\{(w_{C_{1}}\hat{y}_{C_{1}}(\textbf{x}_{i}),\ldots,w_{C_{L}}\hat{y}_{C_{L}}(\textbf{x}_{i})),y_{i}\\}$,
which is used in meta classifier training instead of
$\hat{\mathcal{Y}}(\textbf{x}_{i})$.

##  4 Experimental setup

###  4.1 Data

Table 2: Characteristics of the datasets used in our experiments

Application Dataset alias # instances # Attr % positives Instance-dependent
domain costs source 1 Bankruptcy bankruptcy (private) 404999 221 3.31 this
publication 2 Churn churn_kgl (Kaggle*) 7043 21 26.54 [25] 3 Churn churn_AB
[3] 9410 45 4.83 [3] 4 Credit risk credit_kgl (Kaggle*) 112915 15 11.70 [2] 5
Credit risk credit_de_uci [12] 1000 20 30.00 this publication 6 Credit risk
credit_kdd09 [28] 38938 39 19.89 [2] 7 Credit risk credit_ro_vub [24] 18918 24
16.95 [24] 8 Direct marketing dm_pt_uci [12, 22] 45211 17 11.27 [4] 9 Direct
marketing dm_kdd98 [12] 95412 (train) 479 5.08 [25] 96367 (test) 5.06 10 Fraud
detection fraud_ulb_kgl [21] 284807 31 0.17 [25] 11 Fraud detection
fraud_ieee_kgl (Kaggle*) 590540 432 3.50 [25] 12 HR analytics absenteeism_be
(private) 36853 (train) 71 14.50 [20] 35884 (test) 10.76 * Kaggle:
https://www.kaggle.com/

In this study we use a collection of 10 publicly available datasets and 2
private datasets, for which misclassification costs have either already been
defined or will be defined here. This collection of datasets represents a
number of application domains: _credit scoring_ , _customer churn prediction_
, _direct marketing_ , _credit card fraud detection_ , and _HR analytics_.

###  4.2 Misclassification costs

Table 2 presents the references both to the datasets and to relevant
publications where the instance-dependent misclassification costs for a given
domain were introduced. Most of the datasets are large, the number of
instances ranging between 1000 and almost 600000. The number of input features
ranges from 15 to 479. All of these datasets demonstrate a large degree of
class imbalance, where the percentage of positives reaches at most 30%, and in
dataset _fraud_ulb_kgl_ less than 1%.

In this work we propose instance-dependent costs for these two datasets, for
which no costs were previously defined.

The German credit dataset is well known and is referred to as _credit_de_uci_
in Table 2. Only class-dependent costs were available for this data set, where
the prediction task is to identify customers that will default on their loan.
We define instance-dependent costs using the conceptual framework proposed by
[2]. For any data instance $i$, the cost of a false negative $C_{FN}^{i}$ is
defined as loss given default and constitutes 75% of the credit line, while
the cost of a false positive $C_{FP}^{i}$ is the loss of the potential profit
from rejecting a good customer, plus the sum of the average expected loss and
the average expected profit estimated on the training sample. We define
profits as simply the interests earned on the credit line in the current year.
The profits are calculated using historic interest rates for the year 2000 in
Germany, which we apply randomly and uniformly to the whole sample.

The bankruptcy dataset was provided by the credit risk department of a
European utilities-provider, who was interested in predicting the risk of
corporate bankruptcy for new customers. With minor modifications, it readily
transfers to the same credit risk model described above. Here the credit line
is equivalent 90 days of utilities usage by the customer, which, in case of
default, the provider loses in full, so $C_{FP}^{i}$ equals the credit amount.
The profit margins were provided to us and are calculated per customer based
on the assumption of a 12-month contract. Thus, the $C_{FP}^{i}$ then equals
the annual profit margins for the potentially good customer plus the expected
average loss and expected average profit calculated on the given sample.

####  4.2.1 Data preprocessing

We take care to employ the same preprocessing steps for each of the datasets
in the sample, as recommended by the works that first published them.

In addition to that, we apply the following preprocessing steps to all
datasets. All numeric variables are rescaled using the quantile statistics,
which are robust to outliers. Missing values of numeric variables are imputed
with sample median, and of categorical variables are encoded as a separate
category. All categorical variables are transformed using weight-of-evidence
coding [1].

####  4.2.2 Data partitioning

The classifier performance estimates are obtained by means of repeated
stratified k-fold cross-validation. The $5\times 2$ cross-validation suggested
by [10] is used to train and evaluate stacking ensembles. This resampling is
repeated 5 times using different random seeds, and the results are averaged
across folds and across iterations. Large datasets with more than 100000
observations, to keep training times manageable, were split into five disjoint
subsets, uniformly at random.

We note that two datasets in our sample are provided with a separate test set,
used to evaluate model performance. In this case, for fairness of comparison,
we perform the split into folds on each of the training and test datasets
using the same seed, we then proceed using the training partition of the
training set and the test partition of the test set. The training partition of
the test set remains unused in evaluation. When training and test data sets
contain the same observations at different time periods (e.g. in bankruptcy
prediction) we ensure that training and test datasets are disjoint and do not
contain overlapping data instances.

###  4.3 Learning algorithms

The choice of the algorithms for the base- and meta-level of stacking remains
one of the challenges of stacked generalization. To the best of our knowledge,
no study exists that demonstrates the necessity to use a specific algorithm
combination in either base- or meta-level of stacking. The main requirement
for the base classifiers of any ensemble is that they are sufficiently
accurate (meaning they predict better than a random guess) and sufficiently
diverse (meaning their errors are uncorrelated) [11]. In a heterogeneous
ensemble, where the decisions of different learning algorithms are combined,
the number of base-learners need not be large [26]. All algorithms below have
previously been described and discussed in detail in a number of machine
learning textbooks, for example [16], so we refrain from repeating these
descriptions here.

####  4.3.1 Base-learners

The base learners in our experiments are four well known classification
algorithms, which are: CART Decision Tree (DT), K-Nearest Neighbors (KNN),
Support Vector Machines (SVM) and Logistic Regression (LR). Unlike [7] and
[33] before us, we choose not to use ensembles such as Random Forest or
Extremely Randomised Trees in the base level of stacking. The reasons for this
are two-fold. Firstly, ensembles in general, and stacking in particular are
typically built on weak base-learners, which these very powerful models, which
are themselves ensembles, certainly are not. Secondly these methods are based
on decision trees and their errors will be correlated with DT. In our choice
we also considered the recommendations of [8], one of the largest empirical
studies known to date comparing algorithm performance on 121 datasets. Their
results on binary problems (55 UCI datasets) demonstrate that Random Forest,
SVM, Bagging and Decision Trees have the highest probability of obtaining more
than 95% of accuracy, while classifiers of the Naive Bayes (NB) family are not
competitive in comparison. We therefore do not include NB in our experiments,
unlike some previous studies in cost-sensitive stacking.

####  4.3.2 Meta-learners

The choice of the meta-learner constitutes a challenge as well, as was called
’the black art’ by the original author of stacked generalization [31]. To keep
the scale of our experiments manageable and to allow for statistical
comparison between stacking and base classifiers, we use the same four
algorithms that were used in the level-0 of stacking. In addition to that we
also use two homogeneous ensemble methods that, according to [8], perform well
on most problems, namely Adaboost (Ada) and Random Forest (RF).

####  4.3.3 Cost-sensitive learners

While many variants of cost-sensitive learning algorithms exist that can
incorporate the misclassification costs during classifier training [25], in
this study we are not interested in comparing cost-sensitive learning
algorithms, but in ways of combining cost-sensitive and cost-insensitive
learners in a single ensemble. For our purposes it is important that the two
classifiers we compare are different in all but one thing, that is the
composition of the ensemble. We therefore choose to turn known cost-
insensitive classifiers cost sensitive by applying a cost-sensitive threshold
adjustment method called DMECC [25]. In this method, each data instance is
classified according to its individual cost-sensitive decision threshold,
which is based on the ratio of misclassification costs of that particular data
instance. The threshold is calculated as follows:
$T_{cs}^{i}=\frac{C_{FP}^{i}-C_{TN}^{i}}{{C_{FP}^{i}-C_{TN}^{i}+C_{FN}^{i}-C_{TP}^{i}}}\enspace$,
where $C_{TN}^{i}$ and $C_{TP}^{i}$ refer to the costs of correct
classification, and $C_{FN}^{i}$ and $C_{FP}^{i}$ refer to the
misclassification costs of the positive and negative data instances
respectively. A given record is classified as positive when its estimated
probability of being positive exceeds its individual cost-sensitive threshold
$T_{cs}^{i}$ [35].

Since some learning algorithms (such as DT or SVM) are known not to produce
reliable probability estimates, we applied isotonic calibration [36] to all
base-learners.

####  4.3.4 Cost-sensitive stacking

To the best of our knowledge no definition exists of what constitutes cost-
sensitive stacking. Based on the insights from the literature earlier
discussed in Section 2, we see three main possibilities of introducing cost-
sensitivity into the ensemble structure.

  1. 1.

Level-0 classifiers are cost-sensitive, level-1 classifiers are cost-
insensitive.

  2. 2.

Level-0 classifiers are cost-insensitive, level-1 classifiers are cost-
sensitive.

  3. 3.

Both level-0 and level-1 classifiers are cost-sensitive.

We consider 4 functional forms for the MEC-weights as introduced in Section 3,
which resulted in a total of 15 stacking setups to be compared. The complete
list of ensemble compositions is presented in Table 3.

Table 3: The complete list of stacking setups compared in our study.

Stacking setup Level-0 Level-1 Level-1 alias algorithm type input weights
$f(\epsilon)$ algorithm type 1 type-1 CS 1 CiS 2 type-1_exp CS
$\exp((1-\epsilon)/\epsilon)$ CiS 3 type-1_ln CS $\ln((1-\epsilon)/\epsilon)$
CiS 4 type-1_sq CS $((1-\epsilon)/\epsilon)^{2}$ CiS 5 type-1_acc CS
$1-\epsilon$ CiS 6 type-2 CiS 1 CS 7 type-2_exp CiS
$\exp((1-\epsilon)/\epsilon)$ CS 8 type-2_ln CiS $\ln((1-\epsilon)/\epsilon)$
CS 9 type-2_sq CiS $((1-\epsilon)/\epsilon)^{2}$ CS 10 type-2_acc CiS
$1-\epsilon$ CS 11 type-3 CS 1 CS 12 type-3_exp CS
$\exp((1-\epsilon)/\epsilon)$ CS 13 type-3_ln CS $\ln((1-\epsilon)/\epsilon)$
CS 14 type-3_sq CS $((1-\epsilon)/\epsilon)^{2}$ CS 15 type-3_acc CS
$1-\epsilon$ CS

MEC-weighted stacking renders the level-1 classifier cost sensitive through
manipulation of the training data in a cost-sensitive way by applying MEC-
weights to the training data of the level-1 classifier. We consider it an
alternative to obtaining an ensemble where both training levels are cost-
sensitive, which is the third stacking setup stated above.

####  4.3.5 Software used

All of our experiments were performed using the Python programming language
(version 3.8). Cost-insensitive algorithm implementations came from the
_scikit-learn_ (version 1.1.1) Python library [23], while the cost-sensitive
implementations are our own.

###  4.4 Evaluation

####  4.4.1 Evaluation metrics

Contrary to previous studies in cost-sensitive stacking, we would like to
emphasise the importance of using appropriate evaluation metrics for cost-
sensitive classifiers. Most authors use traditional evaluation metrics such as
ROC_AUC, Precision or F1 metric. ROC_AUC is known to be cost-invariant, since
it is a measure that aggregates classifier performance over all possible
class-dependent thresholds, thus implicitly averaging performance over
multiple class-dependent costs, which is not appropriate. Other error-based
metrics typically assume equal class-dependent costs, which, again, is not
appropriate, when instance-dependent costs are known at estimation time. Cost-
sensitive learning aims to adapt the classification decision of a learning
algorithm to the differences between misclassification costs assigned to each
of the classes. It is therefore important that the evaluation metrics used to
assess the performance of cost-sensitive classifiers is also adapted to
account for the difference in misclassification costs. The typical evaluation
metric used in cost-sensitive literature is the total misclassification cost
[14], that simply adds up the errors weighted with their individual
misclassification costs, as defined on the test set. Another option is to
normalise the total misclassification cost over some budget constraint, which
will depend on the application domain. A more general way to do this is to use
the savings score proposed in [2], where the total misclassification costs are
normalised with the cost of either misclassifying all positives as negatives,
or misclassifying all negatives as positives, which ever is smallest. This
gives a metric on the interval between 0 and 1, facilitating comparison across
different datasets, when necessary.

Since the majority of comparisons in our study is performed based on average
ranks, it requires no commensurability of the evaluation metrics, so the
models are ranked according to their total misclassification costs, which
allows for more precise outcome.

####  4.4.2 Multiple classifier comparison

In order to compare multiple classifiers on multiple datasets, we use the
standard approach of the combination of the Friedman omnibus test and post-hoc
Nemenyi test [18]. The Friedman test is conducted under the null-hypothesis
that all algorithms in comparison are equivalent in performance. If this null-
hypothesis is rejected, the post-hoc test can be performed to identify pairs
of classifiers whose performance is significantly different, which is measured
using the critical difference statistic, and can be visualised using the
critical differences diagrams [9]. The non-parametric tests, such as the
Friedman test, are preferred in case where the number of datasets in
comparison is less than 30, which is the number of datasets necessary to
satisfy the normality assumptions of parametric statistical tests, such as
ANOVA [9]. The post-hoc test is known to be of low power, not rejecting the
null even if the null was rejected for the Friedman test. In this case, we
additionally apply Wilcoxon signed-ranks test, as appropriate, which is used
for pairwise comparisons of classifiers on multiple datasets. This test ranks
differences in performances of a given pair of classifiers, under the null
hypothesis that the median difference in ranks is zero. It therefore allows
establishing whether the observed differences in performance between two
classifiers are significant. It is considered more powerful than its
parametric equivalent, the paired t-test when the assumptions of the latter
cannot be guaranteed. It is also considered more powerful than the Sign test,
which counts the number of wins, losses and ties [9].

##  5 Experimental results

The purpose of our experiments is twofold. Firstly, we would like to compare
the performance of the different cost-sensitive stacking setups in order to
determine which of them results in the lowest cost-loss and can be recommended
to practitioners. Secondly, we aim to empirically evaluate MEC-weighted
stacking, which is a new cost-sensitive stacking method we earlier described
in Section 3.

Despite our best efforts, not all classifiers trained successfully on all 12
datasets. In particular, we were unable to collect results for the MEC-
weighted stacking where the weights were defined by the logarithmic function
on the credit scoring problem _credit_ro_vub_ , and MEC-weighted stacking with
exponential weights were missing on the fraud detection dataset
_fraud_ulb_kgl_. The full results for all 15 stacking setups are thus
available on 10 datasets, instead of 12. Unweighted stacking results, however,
are available on all 12 datasets, which we briefly discuss, for completeness.

###  5.1 Finding the best cost-senstive stacking setup

####  5.1.1 Overall comparison

Figure 1: Comparing all classifiers by average rank across 10 datasets. Lower
numbers correspond to better rank.

We begin with an overall comparison, where all classifiers are evaluated and
ranked on each of the 10 datasets, and for each of them an average rank is
computed across all datasets. Figure 1 presents the average ranks for all
stacking classifiers, where the vertical axis shows the stacking setup and the
horizontal axis shows the corresponding level-1 algorithm. The comparison
consists of a total of 90 classifiers (6 algorithms and 15 stacking setups).
For brevity, we adopt the aliases for each of the stacking setups earlier
presented in Table 3.

We notice immediately that the ranking demonstrates clusters with stacking
ensembles of type-3 ranking the best, while type-1 ensembles rank the worst.
We note that models built with KNN and SVM algorithms tend to rank lower than
decision tree based models or logistic regression. However, the general
picture of type-3 stacking ranking the best and type-1 ranking the worst
remains unchanged for KNN and SVM, although the differences in ranks between
the three groups are smaller than for other algorithms.

Whether these differences in ranks are statistically significant will be
discussed in the following subsection, where we demonstrate the outcomes of
statistical tests that compare the performance of various stacking classifiers
across multiple domains.

####  5.1.2 Comparing unweighted stacking setups on 12 datasets

Table 4: The outcome of the Friedman multiple hypothesis testing.

Test statistic ($\chi_{(k-1)}$) Ada DT KNN LR RF SVM Unweighted ($k=3,n=12$)
6.5 32.69** 32.59** 32.79** 32.69** 32.59** 32.59** All ($k=15,n=10$) 3.94
132.44** 132.09** 132.35** 132.29** 132.09** 132.09** ** significant at 0.01
level $k$: number of stacking setups in comparison, $n$: number of datasets in
comparison

(a) Adaboost

(b) Decision Tree

(c) K-Nearest Neighbors

(d) Logistic Regression

(e) Random Forest

(f) Support-Vector Machine

Figure 2: Pairwise comparison of the three unweighted stacking setups on 12
datasets using Nemenyi test at 0.05 significance level.

We begin by testing the null hypothesis that the three unweighted stacking
setups show no difference in performance. The comparison is performed for each
of the six classification algorithms used as level-1 learners. The null
hypothesis of the Friedman test was rejected for all 6 comparisons, and the
test statistics are presented in row 1 of Table 4.

We proceed with the post-hoc Nemenyi test to evaluate the alternative
hypothesis that the performance of three stackings setups is not equal. Figure
2 presents the results of the post-hoc tests at the 0.05 significance level.
We find that type-3 stacking ranks best and is significantly different from
both type-2 and type-1 for all algorithms except SVM, where the difference is
only significant for the comparison between type-3 and type-1, but no
conclusions can be made regarding the differences between ensembles of type-3
and type-2. Similarly, no conclusions can be made regarding the differences in
rank between type-2 and type-1 stacking ensembles.

Since the outcome of the post-hoc tests are ambiguous in the case of SVM, we
also perform the Wilcoxon rank sum test under the null hypothesis that the
median of the paired differences is zero. For the comparison between type-3
and type-2 unweighted stacking the null is rejected at 0.05 level.

We conclude from these tests that type-3 stacking performs significantly
better than the other two stacking setups.

####  5.1.3 Comparing all cost-sensitive stacking setups on 10 datasets

(a) Adaboost

(b) Decision Tree

(c) K-Nearest Neighbors

(d) Logistic Regression

(e) Random Forest

(f) Support-Vector Machine

Figure 3: Comparing all stacking setups on 10 datasets using Nemenyi post-hoc
test at 0.05 significance level.

We proceed to compare all 15 stacking classifiers on 10 datasets. The outcome
of the Friedman rank sum test can be found in row 2 of Table 4. The null
hypothesis of the Friedman test is rejected for every meta-learner at the 1%
significance level, so we conclude that the performance of all 15 models is
not equal and proceed with the post-hoc test. Figure 3 shows the outcome of
the Nemenyi test at 0.05 significance level.

These are for the most part consistent with what we observed in Figure 1,
where the classifiers tend to cluster by stacking setup, type-3 being the
leader, type-2 the second-best and type-1 ranking worst. Similar to what we
observed above with unweighted stacking, we can reject the null that type-3
stacking and its MEC-weighted variants are equal in performance to type-1
stacking and variants. This holds for all algorithms except KNN and SVM. For
stacking ensembles with KNN in level-1 _type-3_ and _type-3_acc_ classifiers
are not significantly different from _type-1_exp_ and _type-1_sq_ , while for
SVM no significant differences were detected between _type-3_exp_ and
_type-3_sq_ and other type-1 ensembles.

Since Nemenyi post hoc test is not powerful enough to establish whether the
differences between the three stacking setups are statistically significant,
additional testing is required. From the outcomes of the post hoc test we
observed that type-3 stacking generally tends to rank highest, and is
therefore of most interest to us. We therefore perform the Wilcoxon rank sum
test for all combinations of pairwise comparisons of stacking algorithms of
type-3 vs type-1 and of type-3 vs type-2 under the null hypothesis that the
median of the rank differences between the two groups is equal to zero. The
complete tables with the obtained test statistics and p-values can be found in
the Appendix. We find that the null could be confidently rejected for all
comparisons between type-3 and type-1 stacking ensembles, we refer the reader
to the Table 7 in the Appendix for details.

As for the comparison of stacking type-3 with type-2, the only algorithm where
the null could not be rejected was SVM. We found that the differences between
all type-3 MEC-weighted stacking variants and type-2 stacking ensembles were
not significant. However, type-3 unweighted stacking was significantly
different from all type-2 stacking variants, see Table A.2 for details.

We can therefore recommend type-3 stacking, where both levels of stacking are
cost-sensitive, as the winner.

###  5.2 Evaluating MEC-weighted stacking

Table 5:  Pairwise comparison of unweighted and MEC-weighted stacking using
Wilcoxon rank sum test. Statistically significant values are marked with
boldface (significance 0.05) and italics (significance 0.1).

Level-1 Unweighted stacking type-1 vs algorithm type-1_acc type-1_exp
type-1_ln type-1_sq Ada 18.0 (0.3) 18.0 (0.3) 18.0 (0.3) 18.0 (0.3) DT 23.0
(0.63) 23.0 (0.63) 23.0 (0.63) 23.0 (0.63) KNN 27.5 (1.0) 21.0 (0.56) 17.0
(0.32) 26.5 (0.92) LR 27.0 (0.96) 14.0 (0.15) 22.0 (0.56) 10.5 (0.08) RF 23.0
(0.63) 23.0 (0.63) 23.0 (0.63) 23.0 (0.63) SVM 18.0 (0.3) 27.0 (0.96) 22.0
(0.56) 27.0 (0.96) Unweighted stacking type-2 vs type-2_acc type-2_exp
type-2_ln type-2_sq Ada 22.0 (0.56) 23.0 (0.63) 23.0 (0.63) 23.0 (0.63) DT
27.5 (1.0) 27.5 (1.0) 26.5 (0.92) 18.5 (0.35) KNN 25.0 (0.8) 21.0 (0.5) 25.0
(0.8) 21.0 (0.5) LR 10.5 (0.08) 20.5 (0.47) 24.5 (0.76) 16.5 (0.26) RF 27.5
(1.0) 22.5 (0.61) 17.5 (0.3) 26.5 (0.92) SVM 23.5 (0.68) 19.5 (0.41) 18.5
(0.36) 10.5 (0.08) Unweighted stacking type-3 vs type-3_acc type-3_exp
type-3_ln type-3_sq Ada 13.0 (0.16) 19.0 (0.43) 19.0 (0.43) 19.0 (0.43) DT 2.0
(0.01) 11.0 (0.11) 0.0 (0.0) 10.0 (0.08) KNN 24.0 (0.77) 17.0 (0.32) 24.0
(0.77) 16.0 (0.28) LR 13.0 (0.16) 19.0 (0.43) 14.0 (0.19) 24.0 (0.77) RF 4.0
(0.01) 8.0 (0.05) 3.0 (0.01) 9.0 (0.06) SVM 10.0 (0.08) 0.0 (0.0) 9.0 (0.06)
7.0 (0.04)

Our next research question is whether within the same setup, MEC-weighted
stacking offers any improvement over the unweighted stacking. To determine
whether there is a statistically significant difference in performance between
the MEC-weighted stacking models and their unweighted counterparts we perform
pairwise comparison using Wilcoxon rank sum test. The test statistics and
corresponding p-values from the 72 comparisons are reported in Table 5. Values
that are significant at 5% level were highlighted with boldface text, weakly
significant values at 10% level were highlighted with italics. As previously,
the results are reported per learning algorithm used as the level-1
classifier.

We find almost no significant differences in performance between unweighted
and weighted stacking for setups of type-1 and type-2 with rare exceptions.
Surprisingly, only one comparison of stacking type-1, where the level-1
classifier is cost-insensitive shows a significant difference in performance.
Namely, the stacking setup _type-1_sq_ learned with Logistic Regression in
level-1. Referring back to the average rankings reported in Figure 1 it
happens to be the best performing Logistic Regression stacking of type-1, so
in this instance MEC-weighted stacking is significantly better than its
counterpart with equally weighted meta-inputs. It is, however, an exception,
and we must conclude that introducing cost-sensitivity through MEC-weights
into level-1 of stacking has no positive impact on type-1 stacking
performance.

Similar conclusions can be drawn for stacking of type-2, where base
classifiers are cost-insensitive but the meta-classifier is made cost-
sensitive using DMECC. Here we observe only two statistically significant test
outcomes, both of which have lower average ranks than the unweighted stacking
of type-2.

For the setup of type-3 where both level-0 and level-1 of stacking are made
cost-sensitive using DMECC, the null could not be rejected for AdaBoost,
Logistic Regression and KNN. Most of the MEC-weighted ensembles built with
Decision Tree, Random Forest and SVM were significantly different from
unweighted stacking of type-3. Looking at the differences in the average
ranks, however we note that unweighted stacking of type-3 ranks noticeably
better than any of the MEC-weighted models.

We must therefore conclude that MEC-weighted stacking does not offer a
statistically significant improvement over conventional stacking.

###  5.3 Comparing cost-sensitive stacking with single cost-sensitive models

Finally, one might ask whether the effort involved in training the level-1
classifier is worth it. To answer this we compare the best stacking classifier
with the corresponding single classifier. Having previously determined the
best stacking setup, where DMECC was applied in both levels, and no MEC-
weights are applied, we will omit other classifiers from this analysis. We
average classifier performance across cross-validation folds using the savings
metric (see Section 4.4) for commensurability. We also rank the resulting
selection of classifiers by savings and average ranks across 12 datasets. The
results are reported in Table 6, where the winning classifier (per algorithm)
is marked with boldface font, the best performer per dataset is marked with
italics.

We note that cost-sensitive stacking always achieves positive savings, meaning
its total misclassification costs are lower than the predetermined budget.
Stacking has higher average savings on all algorithms except KNN and Random
Forest. In terms of average ranks, stacking wins for all level-1 algorithms
except KNN, which, we note, is one of the worst ranking algorithms in our
study.

Table 6: Comparing single classifiers with type-3 unweighted stacking. Savings
score is reported for each classifier, higher is better. Best model per
dataset is marked with italics. The winning classifier is marked with boldface
letters.

Ada DT KNN LR RF SVM Dataset Single Stacking Single Stacking Single Stacking
Single Stacking Single Stacking Single Stacking absenteeism_be_1 0.188 0.225
0.219 0.242 0.199 0.224 0.188 0.23 0.172 _0.243_ 0.188 0.223 bankruptcy 0.03
0.123 0.112 0.123 0.105 0.058 -0.043 _0.126_ 0.31 0.123 -0.024 0.024 churn_AB
_0.171_ 0.081 0.052 0.087 0.07 0.019 0.062 0.082 0.1 0.086 0.033 0.04
churn_kgl _0.311_ 0.295 0.0 0.297 0.242 0.031 0.302 0.295 0.273 0.296 0.225
0.066 credit_de_uci 0.399 0.391 0.293 0.387 0.337 0.351 0.396 0.388 _0.424_
0.386 0.405 0.354 credit_kdd09 _0.318_ 0.303 0.277 0.302 0.287 0.276 0.312
0.303 0.313 0.302 0.289 0.279 credit_kgl _0.511_ 0.411 0.156 0.411 0.408 0.202
-0.053 0.411 0.499 0.411 0.378 0.175 credit_ro_vub 1.793 1.796 1.787 1.795
1.786 1.785 1.727 1.793 1.762 _1.797_ 1.773 1.786 dm_kdd98_train 0.108 0.143
0.033 _0.147_ 0.035 0.061 0.122 0.119 0.059 _0.147_ 0.036 0.044 dm_pt_uci
_0.568_ 0.558 0.537 0.557 0.551 0.511 0.562 0.558 0.556 0.557 0.551 0.529
fraud_ieee_kgl 0.109 0.494 0.444 0.505 0.477 0.438 0.372 0.495 _0.584_ 0.506
0.407 0.444 fraud_ulb_kgl -0.062 0.714 0.625 0.701 0.679 0.706 0.75 0.72
_0.762_ 0.728 -0.124 0.688 Avg Savings 0.37 0.461 0.378 0.463 0.431 0.389
0.391 0.46 0.484 0.465 0.345 0.388 Avg Rank 5.08 4.17 9.42 4.33 8.17 9.17 6.75
4.08 4.58 3.83 9.25 9.08

##  6 Discussion

##### Outcome 1: using cost-sensitive models in both levels of stacking is
recommended

The results presented in this paper, have demonstrated that there is a
statistically significant difference in performance between the three
different stacking setups considered in our experiments, namely _CiS-CS_ ,
_CS-CiS_ , and _CS-CS_. Contrary to the majority of cost-sensitive stacking
papers that assumed that one level of cost-sensitive decision-making is
sufficient, our experiments demonstrate that stacking models where the DMECC
was applied in both levels of stacking achieved the highest ranking.

While these conclusions hold for this particular post-training method, cost-
sensitivity can be introduced either before or during training of the learning
algorithm. Further experiments are required to investigate how different cost-
sensitive methods affect our conclusions. Now that we have established how
cost-sensitive stacking should be built, future work can focus on combining
various kinds of cost-sensitive algorithms, including pre-, during- and post-
training cost-sensitive methods [25]. Another interesting avenue for future
research would be investigating homogeneous cost-sensitive stacking, an
example of which was proposed in [5] using cost-sensitive decision trees as
base classifiers and cost-sensitive logistic regression as level-1 classifier.

##### Outcome 2: cost-insensitive classifiers do not perform well when costs
are known, even in stacking

As was previously shown in [20] cost-insensitive classifiers, having no way to
account for differences in misclassification costs, typically perform worse
than cost-sensitive models when evaluated using cost-based performance
metrics. In our study, we observed yet another confirmation to this in the
context of heterogeneous ensembles, where base-learners were cost-sensitive
and meta-learners were cost-insensitive. It is, however, surprising that
applying MEC-weights has no positive impact on the performance of these cost-
insensitive meta-learners. So we conclude that the transfer of cost
information via cost-sensitive decision-making of the base-classifiers, and
via MEC-weights was not sufficient to influence the final decision of the
meta-learner. And even though application of MEC-weights to the meta-inputs
makes the meta-level cost-sensitive, the performance of this method is
inferior to unweighted stacking models. We can hypothesise that it may be
different if the meta-learner used misclassification costs internally, but
this questions is left for future research.

##### Limitations

Our current work is not without limitations, which we address below. The
choice of the algorithms to be used in stacking is likely to impact its
performance. In order to keep our experiments manageable, we limited ourselves
to algorithms used previously in the cost-sensitive stacking literature. No
parameter tuning was performed to preserve the same base-classifier
composition across domains. Those familiar with SVM classifiers could remark
that not tuning this algorithm is a mistake. We are aware of this limitation,
which resulted in possibly poor comparative performance of SVM-based
ensembles, however the purpose of the work was to ensure that the ensembles
compared differ in only one thing, which is the inclusion of cost-sensitive
decision-making into different levels of the ensemble. We are interested in
relative performance of stacking setups, not in optimal performance of every
learning algorithm on every domain. In order to perform statistical tests, we
had to ensure that the classifiers were the same in every ensemble for every
dataset, while parameter tuning will have resulted in different parameter
settings on different datasets, which would have prevented us from performing
statistical comparison. In future work we may experiment with homogeneous
stacking, where the diversity of the ensemble will be created by
hyperparameter tuning of the base classifiers.

##  7 Conclusions

Stacking is a well established state-of-the-art ensemble method, that has been
widely applied to many application domains. In this work we provide insights
into ways to make stacking cost-sensitive. We compare 90 stacking models built
with 15 different compositions of the stacking ensemble using 6 well known
classification algorithms. We evaluate on 12 real-world cost-sensitive
problems with clearly defined, non-synthetic, instance-dependent
misclassification costs. In contrast to the absolute majority of cost-
sensitive literature, our experimental results demonstarate that for the best
results, not one, but two layers of cost-sensitive decision-making are
required.

We also found that applying MEC-weights to the training inputs of the level-1
classifier in stacking did not significantly change the performance of
stacking models where the level-1 algorithm applied the default decision
threshold to classify. Moreover, MEC-weighted stacking models where both
levels were cost-sensitive performed worse than the unweighted stacking of the
same type, indicating that two levels of cost-sensitivity is sufficient for
good performance.

Another contribution of our work is the consolidation of all publically
available datasets with record-dependent costs in one place. In addition to
that we derive instance-dependent costs for the well known _credit-g_ dataset
from the UCI repository.

## References

  * [1] Frederik P Agterberg.  Systematic approach to dealing with uncertainty of geoscience information in mineral exploration.  APCO 89, pages 165–178, 1989.
  * [2] Alejandro Correa Bahnsen, Djamia Aouada, and Björn Ottersten.  Example-dependent cost-sensitive logistic regression for credit scoring.  In 2014 13th International Conference on Machine Learning and Applications, pages 263–269. IEEE, 2014.
  * [3] Alejandro Correa Bahnsen, Djamila Aouada, and Björn Ottersten.  A novel cost-sensitive framework for customer churn predictive modeling.  Decision Analytics, 2(1):1–15, 2015.
  * [4] Alejandro Correa Bahnsen, Aleksandar Stojanovic, Djamila Aouada, and Björn Ottersten.  Improving credit card fraud detection with calibrated probabilities.  In Proceedings of the 2014 SIAM International Conference on Data Mining, pages 677–685. SIAM, 2014.
  * [5] Alejandro Correa Bahnsen, Sergio Villegas, Djamila Aouada, and Björn Ottersten.  Fraud detection by stacking cost-sensitive decision trees.  In Data Science for Cyber-Security, pages 251–266. World Scientific, 2019.
  * [6] R. Mike Cameron-Jones and Andrew Charman-Williams.  Stacking for misclassification cost performance.  In Canadian Conference on AI, 2001.
  * [7] Chenjie Cao and Zhe Wang.  Imcstacking: Cost-sensitive stacking learning with feature inverse mapping for imbalanced problems.  Knowledge-Based Systems, 150:27–37, 2018.
  * [8] Manuel Fernández Delgado, Eva Cernadas, Senén Barro, and Dinani Gomes Amorim.  Do we need hundreds of classifiers to solve real world classification problems?  Journal of Machine Learning Research, 15:3133–3181, 2014.
  * [9] Janez Demšar.  Statistical comparisons of classifiers over multiple data sets.  The Journal of Machine learning research, 7:1–30, 2006.
  * [10] Thomas G. Dietterich.  Approximate statistical tests for comparing supervised classification learning algorithms.  Neural Computation, 10:1895–1923, 1998.
  * [11] Thomas G. Dietterich.  Ensemble methods in machine learning.  In Multiple Classifier Systems, 2000.
  * [12] Dheeru Dua and Casey Graff.  UCI machine learning repository.  http://archive.ics.uci.edu/ml, 2017.
  * [13] Zeinab Eivazpour and Mohammad Reza Keyvanpour.  Cssg: A cost-sensitive stacked generalization approach for software defect prediction.  Software Testing, Verification and Reliability, page e1761, 2021\.
  * [14] Charles Elkan.  The foundations of cost-sensitive learning.  In International Joint Conference on Artificial Intelligence, volume 17, pages 973–978, 2001.
  * [15] David J Hand.  Measuring classifier performance: a coherent alternative to the area under the roc curve.  Machine Learning, 77(1):103–123, 2009.
  * [16] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2.  Springer, 2009.
  * [17] Bao-Gang Hu and Wei-Ming Dong.  A study on cost behaviors of binary classification measures in class-imbalanced problems.  arXiv preprint arXiv:1403.7100, 2014.
  * [18] Nathalie Japkowicz and Mohak Shah.  Evaluating Learning Algorithms: a Classification Perspective.  Cambridge University Press, 2011.
  * [19] S. Kotsiantis.  Stacking cost sensitive models.  2008 Panhellenic Conference on Informatics, pages 217–221, 2008\.
  * [20] Natalie Lawrance, George Petrides, and Marie-Anne Guerry.  Predicting employee absenteeism for cost effective interventions.  Decision Support Systems, page 113539, 2021.
  * [21] Yann-A Le Borgne and Gianluca Bontempi.  Machine learning for credit card fraud detection-practical handbook.  ACM SIGKDD Explorations Newsletter, 6(1):1–6, 2004.
  * [22] Sérgio Moro, Paulo Cortez, and Paulo Rita.  A data-driven approach to predict the success of bank telemarketing.  Decision Support Systems, 62:22–31, 2014.
  * [23] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [24] George Petrides, Darie Moldovan, Lize Coenen, Tias Guns, and Wouter Verbeke.  Cost-sensitive learning for profit-driven credit scoring.  Journal of the Operational Research Society, 73(2):338–350, 2022\.
  * [25] George Petrides and Wouter Verbeke.  Cost-sensitive ensemble learning: a unifying framework.  Data Mining and Knowledge Discovery, 36(1):1–28, 2022.
  * [26] Lior Rokach.  Taxonomy for characterizing ensemble methods in classification tasks: A review and annotated bibliography.  Computational Statistics & Data Analysis, 53(12):4046–4072, 2009\.
  * [27] M Paz Sesmero, Agapito I Ledezma, and Araceli Sanchis.  Generating ensembles of heterogeneous classifiers using stacked generalization.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 5(1):21–34, 2015.
  * [28] Thanaruk Theeramunkong, Cholwich Nattee, Paulo JL Adeodato, Nitesh Chawla, Peter Christen, Philippe Lenca, Josiah Poon, and Graham Williams.  New Frontiers in Applied Data Mining: PAKDD 2009 International Workshops, Bangkok, Thailand, April 27-30, 2010. Revised Selected Papers, volume 5669.  Springer, Bangkok, Thailand, 2010.
  * [29] K. Ting and I. Witten.  Stacking bagged and dagged models.  In ICML, 1997.
  * [30] Kai Ming Ting and Ian H Witten.  Issues in stacked generalization.  Journal of Artificial Intelligence Research, 10:271–289, 1999.
  * [31] David H Wolpert.  Stacked generalization.  Neural Networks, 5(2):241–259, 1992.
  * [32] David H Wolpert and William G Macready.  No free lunch theorems for optimization.  IEEE Transactions on Evolutionary Computation, 1(1):67–82, 1997\.
  * [33] Yueling Xiong, Mingquan Ye, and Changrong Wu.  Cancer classification with a cost-sensitive naive bayes stacking ensemble.  Computational and Mathematical Methods in Medicine, 2021, 2021.
  * [34] Jianhong Yan and Suqing Han.  Classifying imbalanced data sets by a novel re-sample and cost-sensitive stacked generalization method.  Mathematical Problems in Engineering, 2018:1–13, 2018.
  * [35] Bianca Zadrozny and Charles Elkan.  Learning and making decisions when costs and probabilities are both unknown.  In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 204–213. ACM, 2001\.
  * [36] Bianca Zadrozny and Charles Elkan.  Transforming classifier scores into accurate multiclass probability estimates.  In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699, 2002.
  * [37] Zhi-Hua Zhou.  Ensemble methods: foundations and algorithms.  CRC press, 2012.

##  Appendix A Additional results

###  A.1 Pairwise comparison of stacking setups of type-3 and type-1.

Table 7: Wilcoxon test statistics (p-values).

Level-1 algorithm type-3 type-3_acc type-3_exp type-3_ln type-3_sq Ada type-1
0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_acc 0.0 (0.0) 0.0
(0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_exp 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) type-1_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) type-1_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) DT type-1
0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_acc 0.0 (0.0) 0.0
(0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_exp 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) type-1_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) type-1_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) KNN type-1
0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_acc 0.0 (0.0) 0.0
(0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_exp 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) type-1_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) type-1_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) LR type-1
0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_acc 0.0 (0.0) 0.0
(0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_exp 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) type-1_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) type-1_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) RF type-1
0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_acc 0.0 (0.0) 0.0
(0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-1_exp 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) type-1_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) type-1_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) SVM type-1
0.0 (0.0) 0.0 (0.0) 8.0 (0.05) 1.0 (0.0) 6.0 (0.03) type-1_acc 0.0 (0.0) 0.0
(0.0) 8.0 (0.05) 1.0 (0.0) 6.0 (0.03) type-1_exp 0.0 (0.0) 0.0 (0.0) 8.0
(0.05) 1.0 (0.0) 6.0 (0.03) type-1_ln 0.0 (0.0) 0.0 (0.0) 8.0 (0.05) 1.0 (0.0)
6.0 (0.03) type-1_sq 0.0 (0.0) 0.0 (0.0) 8.0 (0.05) 1.0 (0.0) 6.0 (0.03)

###  A.2 Pairwise comparison of stacking setups of type-3 and type-2.

Table 8: Wilcoxon test statistics (p-values).

Level-1 algorithm type-3 type-3_acc type-3_exp type-3_ln type-3_sq Ada type-2
0.0 (0.0) 0.0 (0.0) 3.0 (0.01) 3.0 (0.01) 0.0 (0.0) type-2_acc 0.0 (0.0) 0.0
(0.0) 3.0 (0.01) 3.0 (0.01) 0.0 (0.0) type-2_exp 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) type-2_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) type-2_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) DT type-2
0.0 (0.0) 4.0 (0.01) 3.0 (0.01) 3.0 (0.01) 3.0 (0.01) type-2_acc 0.0 (0.0) 3.0
(0.01) 3.0 (0.01) 3.0 (0.01) 3.0 (0.01) type-2_exp 0.0 (0.0) 3.0 (0.01) 2.0
(0.01) 0.0 (0.0) 3.0 (0.01) type-2_ln 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) type-2_sq 0.0 (0.0) 4.0 (0.01) 3.0 (0.01) 3.0 (0.01) 3.0 (0.01) KNN
type-2 7.0 (0.04) 7.0 (0.04) 7.0 (0.04) 7.0 (0.04) 8.0 (0.05) type-2_acc 7.0
(0.04) 7.0 (0.04) 7.0 (0.04) 7.0 (0.04) 7.0 (0.04) type-2_exp 7.0 (0.04) 7.0
(0.04) 7.0 (0.04) 7.0 (0.04) 8.0 (0.05) type-2_ln 5.0 (0.02) 7.0 (0.04) 7.0
(0.04) 7.0 (0.04) 7.0 (0.04) type-2_sq 7.0 (0.04) 7.0 (0.04) 7.0 (0.04) 7.0
(0.04) 8.0 (0.05) LR type-2 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0)
type-2_acc 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-2_exp 0.0
(0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-2_ln 0.0 (0.0) 0.0 (0.0)
0.0 (0.0) 0.0 (0.0) 0.0 (0.0) type-2_sq 0.0 (0.0) 0.0 (0.0) 0.0 (0.0) 0.0
(0.0) 0.0 (0.0) RF type-2 0.0 (0.0) 3.0 (0.01) 4.0 (0.01) 3.0 (0.01) 4.0
(0.01) type-2_acc 0.0 (0.0) 3.0 (0.01) 3.0 (0.01) 3.0 (0.01) 3.0 (0.01)
type-2_exp 0.0 (0.0) 4.0 (0.01) 4.0 (0.01) 3.0 (0.01) 4.0 (0.01) type-2_ln 0.0
(0.0) 4.0 (0.01) 4.0 (0.01) 3.0 (0.01) 4.0 (0.01) type-2_sq 0.0 (0.0) 3.0
(0.01) 3.0 (0.01) 3.0 (0.01) 4.0 (0.01) SVM type-2 6.0 (0.03) 13.0 (0.16) 25.0
(0.85) 13.0 (0.16) 20.0 (0.49) type-2_acc 6.0 (0.03) 14.0 (0.19) 24.0 (0.77)
14.0 (0.19) 20.0 (0.49) type-2_exp 6.0 (0.03) 13.0 (0.16) 24.0 (0.77) 13.0
(0.16) 19.0 (0.43) type-2_ln 6.0 (0.03) 13.0 (0.16) 25.0 (0.85) 13.0 (0.16)
20.0 (0.49) type-2_sq 6.0 (0.03) 13.0 (0.16) 25.0 (0.85) 13.0 (0.16) 19.0
(0.43)