11institutetext: Department of Systems and Computer Networks,
Faculty of Electronics, Wrocław University of Science and Technology,
Wybrzeże Wyspiańskiego 27, 50-370 Wrocław, Poland
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
# Building an Ensemble of Classifiers via Randomized Models of Ensemble
Members

Pawel Trajdos[0000-0002-4337-6847] 11 Marek Kurzynski[0000-0002-0401-2725] 11

###### Abstract

Many dynamic ensemble selection (DES) methods are known in the literature. A
previously-developed by the authors, method consists in building a randomized
classifier which is treated as a model of the base classifier. The model is
equivalent to the base classifier in a certain probabilistic sense. Next, the
probability of correct classification of randomized classifier is taken as the
competence of the evaluated classifier.

In this paper, a novel randomized model of base classifier is developed. In
the proposed method, the random operation of the model results from a random
selection of the learning set from the family of learning sets of a fixed
size. The paper presents the mathematical foundations of this approach and
shows how, for a practical application when learning and validation sets are
given, one can determine the measure of competence and build a MC system with
the DES scheme.

The DES scheme with the proposed model of competence was experimentally
evaluated on the collection of 67 benchmark datasets and compared in terms of
eight quality criteria with two ensemble classifiers which use the previously-
proposed concepts of randomized model. The proposed approach achieved the
lowest ranks for almost all investigated quality criteria.

###### Keywords:

ensemble classification, probabilistic model, randomized classifier,
competence

##  1 Introduction

In ensemble classification methods the recognition result is obtained by
combining the responses of ensemble members. The literature on MC systems has
grown rapidly in the last 30 years and includes a variety of proposed and
developed concepts, algorithms and techniques of ensemble classification
methods as well as a wide range of their practical applications.

Among the different methods of operation of the MC systems, the dynamic
ensemble selection (DES) scheme seems to be the most mature and promising
approach. In the DES methods, base classifiers are dynamically (individually
for each recognized object) selected from the entire set (pool) and then
classification results of selected classifiers are combined by weighted
majority voting [8]. In most methods, the base classifiers are selected from
the pool according to their accuracy measure (competence) in a local region of
the feature space. It means that only an ensemble containing the most
competent classifiers is selected to predict the label of a specific test
sample. This approach is fully justified because usually, there are no
classifiers in the pool that would be experts for all recognized objects,
rather base classifiers are expert in different local regions.

In [3, 4], we can find a comprehensive review of different DES methods
containing the taxonomy and detailed characteristics of DES schemes and their
comparative experimental analysis.

Although the presented methods differ in algorithms for determining classifier
competence and in the interpretation of competence models justifying the
division of methods into different categories (ranking-based, accuracy-based,
probabilistic-based and behavior-based measures), they are dominated by
solutions in which competence at a point is determined based on the properties
of the tested classifier in terms of correct/incorrect classification of
validation objects located in the neighborhood of the object being recognized.

Such techniques seem to be a justified approach if we notice that there are
serious problems with the natural measure of classifier competence for the
feature vector $x$, that is the probability of correct classification at this
point. Firstly, this requires the adoption of a probabilistic model of the
recognition task which in the case of many classifiers (e.g. artificial neural
network etc.) is simply unjustified. Secondly, for deterministic base
classifiers, the probability of correct classification at the point $x$ is
equal to either 0 or 1. Thus, the probability of correct classification in no
way differentiates (in a continuous scale) base classifiers, but divides them
into classifiers correctly/incorrectly classifying the object $x$.

In this study, we explicitly adopt the probability of correct classification
at point $x$ as a measure of classifier competence at this point, but we will
do it indirectly through the randomized classifier being a probabilistic model
of base classifier. The randomized classifier makes a decision randomly,
depending on the value of a certain random variable, so its probability of
correct classification at a point $x$ is a number belonging to the $[0,1]$
interval. The randomized model is constructed in such a way that, in a certain
probabilistic sense, it is equivalent to the base classifier. Therefore its
probability of correct classification is taken as a point measure of the
competence of the evaluated classifier.

This approach has already been considered by the authors in earlier papers
[14, 13]. The concept of the probabilistic model presented there leads to a
randomized classifier whose values of classifying functions are the observed
values of random variables with a probability distribution related to the
values of classification functions produced by the evaluated base classifier.
In the original probabilistic model, called Randomized Reference Classifier
(RRC), the beta distribution was used. In the modified model, we applied the
truncated Gaussian distribution. The developed concept for determining the
competence of base classifiers through their randomized models proved to be a
very effective approach. This is evidenced by the results of comparative
studies presented in the source publications and above all the results
presented in the review paper containing the state-of-the-art in the field of
MC systems with DES scheme [4]. In experimental comparative studies of 30 MC
systems using 30 benchmark databases, the RRC method took 3rd place in terms
of average rank and 5th place in terms of average accuracy.

In this paper, a novel randomized model of a base classifier is proposed in
which the random operation of the model results from a random selection of the
learning set from the family of learning sets of a fixed size. This approach
means that the concept of the base classifier is its randomized model
determining the algorithm for processing features in the context of a training
set before this set is randomly selected.

Although randomization has been applied for a long time in decision-making
systems (see e.g. [5, 1]), there are not many solutions in the literature that
use a random mechanism to build multiple classifier systems. First of all,
Kleinberg’s method of stochastic discrimination (SD) must be mentioned here
[7] and the random forest algorithm derived from SD. Noteworthy, there are
bagging and boosting methods related to the construction of an ensemble of
classifiers based on different training sets created in the random resampling
process. In [10] interesting concept of building an ensemble of classifiers
through randomization is presented, in which randomization procedure is
associated with the random selection of the parameters of the base
classifiers.

The method presented in the paper follows the trend of using randomization
procedures to build a multiclassifier system. However, this is done originally
and differently from the techniques listed above. Randomization in the
presented approach allows determining the measure of competence, based
directly on the probability of correct classification (at a given point) which
evaluates the quality of the base classifier on a continuous scale.

The paper is organized as follows. Section 2 introduces the formal model of
classification problem and gives an insight into the original method of
determining competence via randomized model of classifier. Section 3 contains
a description of the proposed method. Section 4 presents the multiclassifer
system with DES scheme using the proposed method of determining competence
measure. Section 5 provides results of experimental comparative analysis.
Section 6 concludes the paper.

##  2 Competence of Base Classifier

###  2.1 Preliminaries

Let us consider a pattern recognition problem in which
$x\in\mathcal{X}=\Re^{d}$ denotes feature vector of an object and
$j\in\mathcal{M}=\\{1,2,3,...,M\\}$. is its class number. Let $\psi$ be a
trained classifier which maps the feature space $\mathcal{X}$ into the set of
class numbers $\mathcal{M}$, viz.:

$\displaystyle\psi(x)=i,\ \ x\in\mathcal{X},\ i\in\mathcal{M}.$ (1)

We assume that $\psi$ is described by the canonical model [9], i.e. for a
given $x$, it first produces values of normalized classification functions
(supports) $g_{i}(x),i\in\mathcal{M}$ ($g_{i}(x)\in[0,1],\sum g_{i}(x)=1$) and
then classifies an object according to the maximum support rule:

$\displaystyle\psi(x)=i\Leftrightarrow
g_{i}(x)=\max_{k\in\mathcal{M}}g_{k}(x).$ (2)

Let $c(\psi|x)$ be a competence measure of classifier $\psi$ at the point
$x\in\mathcal{X}$. There exist several methods of building local competence
measure $c(\psi|x)$ based on different paradigms. Their exhaustive review can
be found in [4].

###  2.2 Competence via Randomization

It seems that the most natural competence measure of the classifier $\psi$ for
the feature vector $x$ is the probability of correct classification
$P_{c}(\psi(x))$. If $\psi$ is a deterministic classifier, its operation at
point $x$ is strictly dichotomous: classifier correctly classifies $x$
belonging to the $j$-th class if $\psi(x)=j$, or misclassifies $x$ if
$\psi(x)\neq j$. It means that probability of correct classification
$Pc(\psi(x))$ of classifier $\psi$ at the point $x$ is either $0$ or $1$,
viz.:

$\displaystyle Pc(\psi(x))$
$\displaystyle=\left\llbracket{\psi(x)=j}\right\rrbracket,$ (3)

where $\left\llbracket{\cdot}\right\rrbracket$ denotes the Iverson bracket.

Therefore, a direct application of $Pc(\psi(x))$ as a competence measure
$c(\psi|x)$ is not used in this paper.

In the proposed method, we first built a probabilistic model $\bar{\psi}$ of
classifier $\psi$. The model $\bar{\psi}$ depends on random variable $Y$ with
probability distribution $P(Y)$. It means that the model $\bar{\psi}(Y)$ is
the randomized classifier which makes classification randomly according to a
probability distribution $P(\bar{\psi}(Y))$ dependent on $P(Y)$ and
$\bar{\psi}$ as a function of $Y$ [1]. Formally, $\bar{\psi}$ is also a
function from $\mathcal{X}$ into $\mathcal{M}$ specifying how decisions are
made given observation $y$ of the random variable $Y$. It is obvious that
distribution $P(Y)$ and construction of $\bar{\psi}$ should be proposed in
such a way that the model is equivalent to $\psi$ in a certain probabilistic
sense.

The probability of correct classification of $P_{c}(\bar{\psi}(x))$ at point
$x$ belongs to the interval $[0,1]$. Therefore, we can use the probability of
the correct classification of the randomized classifier as a measure of the
competence of the classifier $\psi$ at point $x$, i.e.:

$\displaystyle c(\psi|x)=P_{c}(\bar{\psi}(x)).$ (4)

The key problem of the proposed approach is related to the definition of
probability distribution $P(Y)$ which results from the adopted relationship
between the classifier $\psi$ and its model $\bar{\psi}$.

####  2.2.1 Randomized Reference Classifier

RRC is a probabilistic classifier which classifying functions
$\\{\delta_{j}(x)\\}_{j\in\mathcal{M}}$ are observed values of
multidimensional random variable $Y=\\{\Delta_{j}(x)\\}_{j\in\mathcal{M}}$
fulfilling the following conditions: (i) $\Delta_{i}(x)\in[0,1]$, (ii)
$\sum_{i\in\mathcal{M}}\Delta_{i}(x)=1$, (iii)
$\mathbf{E}\left[\Delta_{i}(x)\right]=g_{i}(x)$, where $\mathbf{E}$ is the
expected value operator. Conditions (i) and (ii) arise from the normalization
properties of class supports, whereas condition (iii) provides the equivalence
of the randomized model $\bar{\psi}_{RRC}$ and base classifier $\psi$. Based
on the latter condition, the RRC can be used to provide randomized model of
any classifier that returns vector of class-specific supports
$g_{i}(x),i\in\mathcal{M}$.

It is obvious that the probability of correct classification of an object $x$
belonging to the class $j$ using the RRC is as follows:

$\displaystyle
P_{c}(\bar{\psi}_{RRC}(x))=P[\Delta_{j}(x)>\Delta_{k}(x),k\in\mathcal{M}\setminus
j].$ (5)

The probability on the right side of (5) can be easily determined if we assume
– as in the original work [14] – that random variables $\Delta_{i}(x)$ have
the beta distribution.

Since $\bar{\psi}_{RRC}$ acts – on average – as the modeled base classifier
$\psi$, the formula (4) is fully justified.

####  2.2.2 RRC with Gaussian Distribution

In the paper [13], we proposed the use of Gaussian distribution truncated to
the inverval $[0,1]$ instead of the beta distribution suggested in the
original RRC model. It means that in the proposed randomized model
$\bar{\psi}_{GD}$ random variable $Y$ is defined as in the RRC model. The
expected value for each random variable $\Delta_{i}(x)$ is simply determined
using formula (iii), but the standard deviation is calculated using a rescaled
variance of the beta distribution with parameter that should be experimentally
tuned in order to achieve the best classification accuracy.

##  3 The Proposed Model of Competence

###  3.1 The Mathematical Fundamentals

We suppose that classifier $\psi$ is trained in the supervised learning
procedure using a learning set

$\displaystyle s_{n}=\\{(x_{1},j_{1}),(x_{2},j_{2}),\ldots(x_{n},j_{n})\\},$
(6)

containing $n$ learning objects and their true classes. Let $\mathscr{S}_{n}$
be a family of all possible sets of a fixed size $n$.

Since the dependence of the classifier on the learning set is important for
the proposed competence model, we will now treat classifier $\psi$ as a
function of two variables:

$\displaystyle\psi(x,s_{n})=i,\ \ x\in\mathcal{X},\ s_{n}\in\mathscr{S}_{n},\
i\in\mathcal{M}.$ (7)

We assume that over the family $\mathscr{S}_{n}$, random variable $S_{n}$ with
probability distribution $P(S_{n})$ is defined. Learning set $s_{n}$ is the
observed value of $S_{n}$. It means that set $s_{n}$ is randomly selected from
the family $\mathscr{S}_{n}$ according to $P(S_{n})$. In the proposed method,
we adopt $Y=S_{n}$. As a consequence, we get the model
$\bar{\psi}=\psi(S_{n})$.

The difference between $\psi(x,s_{n})$ and its model $\psi(x,S_{n})$ is
obvious. $\psi(x,S_{n})$ is a random variable presenting the concept of
classifier construction or the algorithm of its activity describing how to
process object features $x$ in the context of variables $S_{n}$ to obtain the
classification result. In other words, $\psi(x,S_{n})$ refers to the general
classifier concept before the training set $s_{n}$ has been drawn from
$\mathscr{S}_{n}$. Thus, $\psi(x,S_{n})$ is randomized classifier whose
classification depends on the result of the draw $s_{n}$ from the family
$\mathscr{S}_{n}$. In turn, $\psi(x,s_{n})$ is a classifier trained using a
randomly selected learning set $s_{n}$ which classifies object $x$ into the
specific class $i$ from the set $\mathcal{M}$. As it results from the above
description, in the proposed method the probability of correct classification
of the concept $\psi(x,S_{n})$ averaged over the family $\mathscr{S}_{n}$ is
used as the competence of the classifier $\psi(x,s_{n})$ being the realization
of this concept.

Since random variable $\left\llbracket{\psi(x,S_{n})=j}\right\rrbracket$ has
binary (Bernoulli) distribution (see formula (3)) with success and failure
probability equal to $P_{c}$ and $1-P_{c}=P_{e}$, respectively. We have the
following formula for the probability of correct classification of the model
$\bar{\psi}$ at the point $x$:

$\displaystyle P_{c}(\bar{\psi}(x))$
$\displaystyle=P_{c}(\psi(x,S_{n}))=E_{S_{n}}\left\llbracket{\psi(x,S_{n})=j}\right\rrbracket$
(8)
$\displaystyle=\int_{\mathscr{S}_{n}}\left\llbracket{\psi(x,s_{n})=j}\right\rrbracket\;dP(s_{n}).$
(9)

If probability distribution of the draw of set $s_{n}$ from $\mathscr{S}_{n}$
is uniform, then we have

$\displaystyle
P_{c}(\bar{\psi}(x))=P_{c}(\psi(x,S_{n}))=\frac{\left\|s_{n}:P_{c}(\psi(x,s_{n}))=1\right\|}{\left\|\mathscr{S}_{n}\right\|},$
(10)

where ${\left\|A\right\|}$ denotes Lebesgue measure of $A$.

###  3.2 Empirical Case

Suppose now that the finite set of learning sets of size $n$ is available:

$\displaystyle(s_{n}^{(1)},s_{n}^{(2)},...,s_{n}^{(K)}).$ (11)

Then estimation of the average correct classification probability (10) can be
calculated as follows:

$\displaystyle P_{c}(\psi(x,S_{n}))\approx\frac{K_{c}}{K},$ (12)

where $K_{c}$ denotes the number of learning sequences for which
$P_{c}(\psi(x,s_{n}))=1$.

In this case, we have empirical competence measure of classifier $\psi$ at the
point $x$ which is the estimated value of probability of correct
classification of $\psi$ at this point averaged over learning sets of the
fixed size $n$.

###  3.3 Competence Set

To calculate (12), the true class of the object $x$ must be known. Thus, we
assume that a validation set

$\displaystyle\mathcal{V}=\\{(x_{1},j_{1}),(x_{2},j_{2}),\ldots(x_{N},j_{N})\\}.$
(13)

containing pairs of feature vectors and their true class labels is available.
In this case, from (4), (12) and (13) we have (in the feature space
${\mathcal{X}}$) the set of $N$ validation objects with competences of
classifer $\psi$:

$\displaystyle\mathcal{C}_{\psi}=\\{(x_{1},c(\psi|x_{1})),(x_{2},c(\psi|x_{2})),\ldots(x_{N},c(\psi|x_{N}))\\}.$
(14)

Set $\mathcal{C}_{\psi}$ is called the competence set of classifier $\psi$.

###  3.4 Generalization Method – Potential Function Model

In the second step, based on the competence set (14), the competence function
(measure) $c(\psi|x)$ is determined. In other words, information contained in
the set $\mathcal{C}_{\psi}$, i.e. values of competence for validation points
$x_{k}\in\mathcal{V}$, is generalized to the whole feature space
${\mathcal{X}}$. In the potential function method of generalization, the
feature vectors $x_{k}$ are considered to be the locations of the competence
sources $c(\psi|x_{k})$ that influence the entire feature space $\mathcal{X}$
creating a competence field. The competence at $x$ is a result of the
cumulative influence of the sources, where the influence of each source is
proportional to $c(\psi|x_{k})$ and it decreases as the distance between
$x_{k}$ and $x$ increases. This interpretation allows using the normalized
Gaussian potential function model [15] to construct the competence function as
follows:

$\displaystyle
c(\psi|x)=\frac{1}{D}\sum_{x_{k}\in\mathcal{V}}c(\psi|x_{k})\exp(-\mathrm{dist}(x_{k},x)^{2}),$
(15)

where $D$ denotes the normalizing factor and $\mathrm{dist}(x,y)$ is the
Euclidean distance between $x$ and $y$.

##  4 Multiclassifier System with DES Scheme

In the multiclassifier system (MCS), a set of trained classifiers
$\Psi=\\{\psi_{1},\psi_{2},\ldots,\psi_{L}\\}$ called base classifiers is
given. We assume that all classifiers from the set $\Psi$ meet the assumptions
presented in subsections 2.1 and 3.1. For each classifier $\psi_{l}\in\Psi$,
the competence function $c(\psi_{l}|x)$ is calculated according to (4), (12)
and (15).

The proposed multiclassifier system uses a dynamic ensemble selection (DES)
strategy which consists of two steps. In the first step, an ensemble of
competent classifiers is selected from the entire set $\Psi$ for a given $x$

$\displaystyle\Psi(x)=\\{\psi_{l1},\psi_{l2},...,\psi_{lx}:c(\psi_{lm}|x)>\alpha\\},$
(16)

where the threshold value $\alpha$ is arbitrary, but it is usually assumed
that $\alpha=1/M$. This step eliminates inaccurate classifiers (worse than
random guessing) and keeps the ensemble relatively diverse.

The selected classifiers are combined using a weighted vector of class
supports, where the weights are equal to the competences, viz.:

$\displaystyle
g_{j}^{(MCS)}(x)=\sum_{\psi_{l}\in\Psi(x)}c(\psi_{l}|x)\;g_{j}^{(\psi_{l})}(x).$
(17)

Finally, the MCS system classifies object $x$ using the maximum rule:

$\displaystyle\psi^{(MCS)}(x)=i\;\Leftrightarrow\;g_{i}^{(MCS)}(x)=\max_{j\in\mathcal{M}}g_{j}^{(MCS)}(x).$
(18)

It should be emphasized that the computationally complex procedures for
determining the source competence measure for validation objects – thanks to
the concept of the competence set – is fully implemented in the learning
phase. In the classification phase, only a simple (and quick) procedure of
determining the classifying functions of the ensemble classifier and making
decisions according to the maximum principle remains to be implemented.

##  5 Experiments

The experimental study is aimed at comparing the proposed method of classifier
selection with the original approach proposed in [14], and its modification
using the Gaussian Distribution [13]. This paper does not contain a comparison
between RRC based DES algorithms and the state-of-the-art methods of building
classifier ensembles. This is because that kind of comparison has already been
done [4].

###  5.1 Experimental Setup

During the experimental evaluation the following heterogeneous ensemble
classifiers were compared: $\psi_{\mathrm{C}}$ – DES scheme with the proposed
model of competence, $\psi_{\mathrm{B}}$ – DES scheme proposed in [14],
$\psi_{\mathrm{N}}$ – DES scheme proposed in [13].

The following base classifiers were used to constitute each of the ensembles:
$\psi_{\mathrm{NB}}$ – Naive Bayes classifier with kernel density estimation,
$\psi_{\mathrm{J48}}$ – Weka version of the C4.5 algorithm with Laplace
smoothing, $\psi_{\mathrm{NC}}$ – Nearest centroid (Nearest Prototype),
$\psi_{\mathrm{SVM}}$ – SVM classifier with linear kernel (no kernel),
$\psi_{\mathrm{LDA}}$ – LDA classifier, $\psi_{\mathrm{QDA}}$ – QDA
classifier, $\psi_{\mathrm{KNN}}$ – Nearest neighbour classifier.

Each of the above-mentioned base classifiers was trained using 31 bootstrap
samples from the training dataset. The outcomes of the classifiers built using
bootstrap samples were used to calculate the correct classification
probability using formula (12).

The experimental code was implemented using WEKA framework. The source code of
the algorithms is available online
111https://github.com/ptrajdos/rrcBasedClassifiers/tree/develop.

To evaluate the proposed methods, the following classification-loss criteria
were used [11]: macro-averaged and micro-averaged $\mathrm{FDR}$ (1-
precision), $\mathrm{FNR}$ (1-recall), $F_{1}$, Matthews correlation
coefficient ($\mathrm{MCC}$). Macro and micro averaged criteria were used
because those families of criteria differ significantly in scoring majority
and minority classes. That is, macro averaged criteria are more sensitive to
performance for the minority class. Whereas micro averaged criteria are more
sensitive to majority class performance.

Following the recommendations of [6], the statistical significance of the
obtained results was assessed using the two-step procedure. The first step is
to perform the Friedman test [6] for each quality criterion separately. Since
the multiple criteria were employed, the familywise errors (FWER) should be
controlled [2]. To do so, the Bergman-Hommel [2] procedure of controlling FWER
of the conducted Friedman tests was employed. When the Friedman test shows
that there is a significant difference within the group of classifiers, the
pairwise tests using the Wilcoxon signed-rank test [6] were employed. To
control FWER of the Wilcoxon-testing procedure, the Bergman-Hommel approach
was employed [2]. For all tests the significance level was set to
$\alpha=0.05$.

The experimental evaluation was conducted on the collection of the 67
benchmark datasets taken from the Keel repository. Benchmark datasets are
available online
222https://github.com/ptrajdos/MLResults/blob/master/data/KeelData.tar.xz.

During the preprocessing stage, the datasets underwent a few transformations.
First, all nominal attributes were converted into a set of binary variables.
The transformation is necessary whenever the distance-based algorithms are
employed [12]. To reduce the computational burden and remove irrelevant
information, the PCA procedure with the variance threshold set to 95% was
applied. The features were also normalized to have zero mean value and unit
variance.

###  5.2 Results and Discussion

To compare multiple algorithms on multiple benchmark sets, the average ranks
approach [6] is used. Due to the page limit, full results are published online
333https://github.com/ptrajdos/MLResults/raw/a3b4168a0b0aabee7ef8cd1056baf4a8578a9f6d/RandomizedClassifiers/CORES2021.zip.
.

The numerical results are given in Table 1. The table is divided into eight
sections – one section is related to a single evaluation criterion. The first
row of each section is the name of the quality criterion investigated in the
section. The second row shows the p-value of the Friedman test. The third one
shows the average ranks achieved by algorithms. The following rows show
p-values resulting from pairwise Wilcoxon test. Above each section the names
of the investigated algorithms are placed.

Table 1: Statistical evaluation. Wilcoxon test results. | $\psi_{\mathrm{C}}$ | $\psi_{\mathrm{B}}$ | $\psi_{\mathrm{N}}$ | $\psi_{\mathrm{C}}$ | $\psi_{\mathrm{B}}$ | $\psi_{\mathrm{N}}$ | $\psi_{\mathrm{C}}$ | $\psi_{\mathrm{B}}$ | $\psi_{\mathrm{N}}$ | $\psi_{\mathrm{C}}$ | $\psi_{\mathrm{B}}$ | $\psi_{\mathrm{N}}$
---|---|---|---|---|---|---|---|---|---|---|---|---
Nam. | MaFDR | MaFNR | MaF1 | Macro Matthews
Frd. | 1.000e+00 | 2.523e-01 | 3.470e-01 | 1.000e+00
Rank | 1.955 | 2.061 | 1.985 | 1.795 | 1.992 | 2.212 | 1.803 | 2.000 | 2.197 | 1.939 | 1.932 | 2.129
$\psi_{\mathrm{C}}$ |  | .457 | .457 |  | .174 | .036 |  | .150 | .007 |  | .236 | .224
$\psi_{\mathrm{B}}$ |  |  | .652 |  |  | .148 |  |  | .154 |  |  | .224
Nam. | MiFDR | MiFNR | MiF1 | Micro.Matthews
Frd. | 1.000e+00 | 1.000e+00 | 1.000e+00 | 1.000e+00
Rank | 1.871 | 1.985 | 2.144 | 1.871 | 1.985 | 2.144 | 1.871 | 1.985 | 2.144 | 1.864 | 1.992 | 2.144
$\psi_{\mathrm{C}}$ |  | .411 | .411 |  | .411 | .411 |  | .411 | .411 |  | .386 | .370
$\psi_{\mathrm{B}}$ |  |  | .411 |  |  | .411 |  |  | .411 |  |  | .370

According to the table, the proposed approach achieved the lowest ranks for
all the investigated quality criteria except macro-averaged $\mathrm{MCC}$.
For the macro-averaged $\mathrm{MCC}$, the proposed approach is the second
one; however, the difference between the average ranks is very small.

On the other hand, the approach based on the Gaussian distribution achieves
the lowest rank in terms of all quality criteria except macro-averaged
$\mathrm{FDR}$.

However, the conducted statistical tests show that there are almost no
significant differences between the investigated algorithms. The significant
differences are only between the proposed method and the Gaussian-based RRC-
DES. These differences show that the proposed approach is better in terms of
macro averaged $\mathrm{FNR}$ and macro-averaged $F_{1}$ measure. It means
that the proposed approach is better at discovering minority class objects.
This is done without lowering precision. This property may be useful for the
imbalanced classification task. The results show that using the data-driven
distribution for the RRC model may improve the classification quality.

##  6 Conclusions

In this paper, we proposed a data-driven approach to building the RRC-based
dynamic-ensemble-selection classifier. The classifier was compared with the
original RRC-based approach and the RRC-method built using Gaussian
distribution. The conducted experimental study shows that there are almost no
significant differences between the proposed method and the previously-
proposed algorithms. The only observed differences may suggest that using the
proposed approach in imbalanced classification problems may allow a
significant improvement over the previously mentioned methods. However, this
result is achieved at the cost of training an ensemble of classifiers using
bootstraped datasets. Consequently, future research is aimed at building a
distribution-driven RRC approach with the reduced computational burden.

## Acknowledgment

This work was supported by the statutory funds of the Department of Systems
and Computer Networks, Wroclaw University of Science and Technology.

## References

  * [1] Berger, J.O.: Statistical Decision Theory and Bayesian Analysis.  Springer New York (1985).  DOI 10.1007/978-1-4757-4286-2
  * [2] Bergmann, B., Hommel, G.: Improvements of general multiple test procedures for redundant systems of hypotheses.  In: Multiple Hypothesenprüfung / Multiple Hypotheses Testing, pp. 100–115. Springer Berlin Heidelberg (1988).  DOI 10.1007/978-3-642-52307-6“˙8
  * [3] Britto, A.S., Sabourin, R., Oliveira, L.E.: Dynamic selection of classifiers—a comprehensive review.  Pattern Recognit. 47(11), 3665–3680 (2014).  DOI 10.1016/j.patcog.2014.05.003
  * [4] Cruz, R.M., Sabourin, R., Cavalcanti, G.D.: Dynamic classifier selection: Recent advances and perspectives.  Information Fusion 41, 195–216 (2018).  DOI 10.1016/j.inffus.2017.09.010
  * [5] DeGroot, M.H.: Optimal Statistical Decisions.  Wiley (2004).  DOI 10.1002/0471729000
  * [6] Demšar, J.: Statistical comparisons of classifiers over multiple data sets.  The Journal of Machine Learning Research 7, 1–30 (2006)
  * [7] Kleinberg, E.: Stochastic discrimination.  Ann Math Artif Intell 1(1-4), 207–239 (1990).  DOI 10.1007/bf01531079
  * [8] Ko, A.H., Sabourin, R., Britto Jr., A.S.: From dynamic classifier selection to dynamic ensemble selection.  Pattern Recognit. 41(5), 1718–1731 (2008).  DOI 10.1016/j.patcog.2007.10.015
  * [9] Kuncheva, L.I.: Combining Pattern Classifiers.  John Wiley & Sons, Inc. (2014).  DOI 10.1002/9781118914564
  * [10] Santucci, E., Didaci, L., Fumera, G., Roli, F.: A parameter randomization approach for constructing classifier ensembles.  Pattern Recognit. 69, 1–13 (2017).  DOI 10.1016/j.patcog.2017.03.031.  URL https://doi.org/10.1016/j.patcog.2017.03.031
  * [11] Sokolova, M., Lapalme, G.: A systematic analysis of performance measures for classification tasks.  Information Processing & Management 45(4) (2009).  DOI 10.1016/j.ipm.2009.03.002
  * [12] Tian, Y., Deng, N.: Support vector classification with nominal attributes.  In: Y. Hao, J. Liu, Y. Wang, Y.m. Cheung, H. Yin, L. Jiao, J. Ma, Y.C. Jiao (eds.) Computational Intelligence and Security, pp. 586–591. Springer Berlin Heidelberg (2005).  DOI 10.1007/11596448“˙86
  * [13] Trajdos, P., Kurzynski, M.: Randomized reference classifier with Gaussian distribution and soft confusion matrix applied to the improving weak classifiers.  In: Advances in Intelligent Systems and Computing, pp. 326–336. Springer International Publishing (2019).  DOI 10.1007/978-3-030-19738-4“˙33
  * [14] Woloszynski, T., Kurzynski, M.: A probabilistic model of classifier competence for dynamic ensemble selection.  Pattern Recognition 44(10-11), 2656–2668 (2011).  DOI 10.1016/j.patcog.2011.03.020
  * [15] Woloszynski, T., Kurzynski, M., Podsiadlo, P., Stachowiak, G.W.: A measure of competence based on random classification for dynamic ensemble selection.  Information Fusion 13(3), 207–213 (2012).  DOI 10.1016/j.inffus.2011.03.007