# A Comparative Evaluation of Quantification Methods

Tobias Schumacher
University of Mannheim
<EMAIL_ADDRESS>
&Markus Strohmaier
University of Mannheim,
GESIS - Leibniz Institute for the Social Sciences, and
Complexity Science Hub
<EMAIL_ADDRESS>
&Florian Lemmerich
University of Passau
<EMAIL_ADDRESS>

###### Abstract

Quantification represents the problem of predicting class distributions in a
dataset. It also represents a growing research field in supervised machine
learning, for which a large variety of different algorithms has been proposed
in recent years. However, a comprehensive empirical comparison of
quantification methods that supports algorithm selection is not available yet.
In this work, we close this research gap by conducting a thorough empirical
performance comparison of 24 different quantification methods on 40 data sets,
considering binary as well as multiclass quantification settings. We observe
that no single algorithm generally outperforms all competitors, but identify a
group of methods including the _Median Sweep_ and the _DyS_ framework that
performs best in the binary setting. We also find that tuning the underlying
classifiers has in most cases only a limited impact on the quantification
performance. For the multiclass setting, we observe that a different, broad
group of algorithms yields good performance, including the _Generalized
Probabilistic Adjusted Count_ , the _readme_ method, the _energy distance
minimization_ method, the _EM algorithm for quantification_ , and _Friedman’s
method_. More generally, we find that the performance on multiclass
quantification is inferior to the results obtained in the binary setting. Our
results can guide practitioners who intend to apply quantification algorithms
and help researchers to identify opportunities for future research.

##  1 Introduction

Quantification is the problem of estimating the distribution of target labels
on unseen (test) data. That is, after being trained on a dataset with known
class labels, we want to estimate the number of instances of each class in a
set of instances with given features but no target labels. In contrast to
traditional classification tasks, we are not interested in individual
predictions, but only in aggregated values on a group level. Additionally, we
also assume that the distribution of the features (but not their associations
with the labels) could have shifted between training and application. For this
problem setting, previous research has established that training a
classification algorithm and counting instance-wise predictions does not
generally yield accurate estimates [12, 27]. This has given rise to a
relatively young, but vivid research field within the machine learning
community. As an increasing number of researchers are becoming aware of this
issue, a growing number of novel methods have been proposed.

While a first review of existing quantification methods has been given by
González et al. [15], and recent publications also provide broader frameworks
for quantification learning [20, 21], a thorough, empirical, and independent
comparison of quantification methods has not been presented yet. In this work,
we aim to fill this research gap by providing a comparison of 24 different
quantification algorithms over 40 datasets. Apart from assessing approaches
for the binary class setting, we also include experiments for the multiclass
quantification setting, which has received limited attention in quantification
research so far. For each dataset and algorithm, we evaluate several degrees
of distribution shifts between training data and test data with varying
training set sizes. Further, we evaluate the effect of tuning the classifiers
that the majority of existing quantification methods are making use of. Thus,
overall, our experimental results report results from more than 5 million
algorithm runs.

Our experiments with binary target labels show that there is not a single
algorithm that outperforms all others—but we identify a group of algorithms
including the _Median Sweep_ [12] and the _DyS_ framework [20] that on average
perform significantly better. We also show that algorithms that are based on
optimizing a classifier for the quantification problem show on average worse
performance, implying that their benefits in practice might be restricted to
particular scenarios. Further, we observe that tuning the hyperparameters of
the base classifiers does not produce significant improvements in the
resulting predictions of the corresponding quantifiers.

In the multiclass setting, we find a broader group of algorithms which show a
significantly better average performance than the rest, with the _Generalized
Probabilistic Adjusted Count_ [3, 10], _readme_ [17], _energy distance
minimization_ [19], the _EM algorithm for quantification_ [25], and
_Friedman’s method_ [13] leading in the averaged rankings. These algorithms
share the characteristic that they naturally allow for multiclass
quantification. By contrast, extending predictions from binary quantifiers to
the multiclass case in a one-vs-all fashion does not appear to yield
competitive results, even when using strong base quantifiers such as the
_Median Sweep_ or the _DyS_ framework. Further, we show that fine-tuning the
hyperparameters of the underlying base classifiers appears to have significant
positive effects in this multiclass setting for most of the quantifiers, in
particular for the _EM algorithm_ and _Friedman’s method_ , which appear to
outperform the rest. More generally, we observe significantly weaker
performance for the multiclass case, corroborating that multiclass
quantification constitutes a harder research problem and might need more
research attention in the future.

Overall, our results guide practitioners towards the most propitious
quantification approaches for certain applications, and help researchers to
identify promising future research avenues.

In the following, we first briefly introduce the quantification problem and
describe how it conceptually differs from the classification problem.
Afterwards, Section 3 gives an overview of the algorithms included in our
experimental comparison, providing a summary of the state-of-the-art in
quantification. Next, in Section 4, we provide a thorough description of the
experimental setup of our comparison, before giving an in-depth presentation
of the experimental results in Section 5. Finally, in Section 6, we discuss
the results of our experiments, before closing with a conclusion in Section 7.

##  2 The Quantification Problem

Quantification is a supervised machine learning problem that aims to estimate
the distribution of class labels in a test set instead of predicting the class
of individual instances. Throughout this paper, we use the following
notations. For training, we are given a dataset of instances $D_{Train}$, for
which we know the values of multiple (categorical or continuous) features $X$
and the corresponding target label $Y$. Letting $L$ denote the number of
possible values of the target label, we distinguish between the _binary_ case,
i.e., there are exactly $L=2$ possible values for the target label, and the
_multiclass_ case, i.e., there are $L>2$ options for the target label. The
goal is then to train a model on training data, that allows to predict the
distribution of the target label in some test data $D_{Test}$, for which only
the values of the features $X$ are known. In the following, we will often use
the term _prevalence_ for the relative frequency of single labels in training
or test data. We formally denote the distributions of $X$ and $Y$ in the
training set by $P_{Train}(X)$ and $P_{Train}(Y)$, and their distribution in
the test set by $P_{Test}(X)$ and $P_{Test}(Y)$. Since in the binary case, the
full distribution is already specified by the share of one class, we will
denote for shorter notation the instances from one arbitrary class as
_positives_ , and label their prevalence in training and test data as
$pos_{Train}$ and $pos_{Test}$, respectively.

In contrast to traditional classification, a _shift_ of the distribution of
the target label $Y$, i.e., a difference between the class probabilities
$P_{Train}(Y)$ in the training set and the class probabilities $P_{Test}(Y)$
in the test set, is expected. However, it is assumed that the conditional
distributions $P(X|Y)$ are stable between training and test sets. Furthermore,
compared to classification, it is also more common to expect contradictory
labels for training or test instances with the exact same feature values.

A trivial approach to quantification, known as _Classify and Count (CC)_
method, applies an arbitrary classification method trained on the training
data to the test data and predicts the distribution of the predicted labels.
However, this has been theoretically and empirically shown to achieve
insufficient results in many scenarios [12, 27].

##  3 Algorithms for Quantification

We first outline the quantification algorithms under consideration. Following
a previous categorization [15], we distinguish between (i) adjusted count
adaptations, (ii) distribution matching methods, and (iii) adaptations of
traditional classification algorithms. An overview of the considered
algorithms is given in Table 1. In that table, the column Continuous indicates
whether an algorithm is also able to handle datasets with continuous features,
and Multiclass indicates whether an algorithm is not restricted to binary
quantification. Next to that, this table also lists abbreviations that we are
often going to use when discussing these algorithms.

Quantification Algorithm | Abbreviation | Reference | Multiclass | Continuous
---|---|---|---|---
Adjusted Count | AC | Forman [11] | OVR | Yes
Probabilistic Adjusted Count | PAC | Bella et al. [3] | OVR | Yes
Threshold Sel. Policy X | TSX | Forman [12] | OVR | Yes
Threshold Sel. Policy T50 | TS50 | Forman [12] | OVR | Yes
Threshold Sel. Policy Max | TSMax | Forman [12] | OVR | Yes
Threshold Sel. Policy Median Sweep | MS | Forman [12] | OVR | Yes
Generalized Adjusted Count | GAC | Firat [10] | Yes | Yes
Generalized Prob. Adjusted Count | GPAC | Firat [10] | Yes | Yes
DyS Framework (Topsøe Distance) | DyS | Maletzke et al. [20] | OVR | Yes
Forman’s Mixture Model | FMM | Forman [12] | OVR | Yes
readme | readme |  Hopkins and King [17] | Yes | No
HDx | HDx | González-Castro et al. [16] | Yes | No
HDy | HDy | González-Castro et al. [16] | Yes | Yes
Friedman’s method | FM | Friedman [13] | Yes | Yes
Energy Distance Minimization | ED | Kawakubo et al. [19] | Yes | Yes
EM-Algorithm for Quantification | EM | Saerens et al. [25] | Yes | Yes
CDE Iteration | CDE | Tasche [28] | No | Yes
Classify and Count | CC | Forman [12] | Yes | Yes
Probabilistic Classify and Count | PCC | Bella et al. [3] | Yes | Yes
$\text{SVM}^{perf}$using ${KLD}$ loss | SVM-K | Esuli et al. [8] | No | Yes
$\text{SVM}^{perf}$using Q-measure-loss | SVM-Q | Barranquero et al. [2] | No | Yes
Nearest-Neighbor-Quantification | PWK | Barranquero et al. [1] | Yes | No
Quantification Forest | QF | Milli et al. [22] | Yes | No
AC-corrected Quantification Forest | QF-AC | Milli et al. [22] | No | No

Table 1: Overview of considered quantification algorithms. For each algorithm,
we denote its abbreviation, a reference, and its ability to handle multiclass
labels and continuous features. Regarding the multiclass setting, we
distinguish between algorithms that can naturally handle this setting (Yes),
algorithms that require a one-vs.-rest approach (OVR), and algorithms that
were not considered in our multiclass experiments (No).

###  3.1 Adjusted Count Adaptations

The trivial _Classify and Count (CC)_ method just applies an arbitrary
classifier $c(X)$ on the test data and counts the number of respective
predictions. The core idea behind the _Adjusted Count (AC)_ approach is to
adjust these results post hoc for potential biases. This is accomplished by
exploiting the assumption that the likelihood $P(X|Y)$ of the features $X$
given the target class $Y$ does not vary between training and test data, and
thus the true-positive rate ($tpr$) and false positive rate ($fpr$) of a
classifier, which correspond to the distributions $P(c(X)|Y)$, are expected to
be identical between training and test data. Letting $ppos_{Test}$ denote the
predicted prevalence of positives by the _CC_ method, we can express this
quantity in terms of the true prevalence of positives $pos_{Test}$ and the
(mis)classification rates $tpr$ and $fpr$ via

$ppos_{Test}=pos_{Test}\cdot tpr+(1-pos_{Test})\cdot fpr,$

which we can solve for $pos_{Test}$ to obtain the _AC_ estimation

$pos_{Test}=\frac{ppos_{Test}-fpr}{tpr-fpr}.$ (1)

In practice it can occur that the estimate falls out of the feasible interval
$[0,1]$, hence in such cases one would have to clip the outcome to the
boundary values.

Based on that idea, a few variations of the _AC_ method have been introduced
in the literature, and the following methods will be included in our
experiments:

  1. 1.

Adjusted Count (AC).  As described above, we estimate the true positive and
false positive rates from the training data and utilize them to adjust the
output of the _CC_ method [11].

  2. 2.

Probabilistic Adjusted Count (PAC).  This method adapts the _AC_ approach by
using average class-conditional confidences from a probabilistic classifier
instead of true positive and false positive rates [3].

  3. 3.

Threshold Selection Policies (TSX, TS50, TSMax, MS).  The core idea of these
variations is to shift the decision boundary (e.g., classify an instance as
positive if the original estimate $c(x)$ is larger than 0.7) of the underlying
classifier in order to get numerically more stable _AC_ estimates. Different
strategies involve using the threshold that maximizes the denominator $tpr-
fpr$ (_TSMax_), a threshold for which we have $fpr=1-tpr$ (TSX), a threshold
at which $tpr\approx 0.5$ holds _TS50_ , or, as in the _Median Sweep (MS_)
method, using an ensemble of such threshold-based methods and taking the
median prediction [12].

###  3.2 Distribution Matching Methods

The majority of existing quantification methods can be categorized as
_distribution matching algorithms_. These algorithms are all implicitly based
on the assumption that the distribution of the features $X$ conditioned on the
distribution of the target labels $Y$ does not change between training data
and test data. Under that assumption, with ${\ell_{j}},j\in\\{1,\dots,L\\}$,
denoting the possible values of the labels $Y$, the law of total probability
yields that

$P_{Test}(X)=\sum_{j=1}^{L}P_{Train}(X|Y=\ell_{j})P_{Test}(Y=\ell_{j}).$ (2)

As in this equation, both the left-hand distribution $P_{Test}(X)$ and the
conditional distributions $P_{Train}(X|Y=\ell_{j})$ on the right-hand side can
be seen as represented by given training and test data, only the sought-for
probabilities $P_{Test}(Y=\ell_{j})$ are left as unknowns. To estimate these
target probabilities, there are two main issues to be worked out from a
methodological point of view. First, estimating or modeling the distributions
$P_{Train}(X|Y=\ell_{j})$ and $P_{Test}(X)$ is not at all trivial. There can
be an arbitrary amount of features $X$, and the training data usually does not
provide nearly enough samples to accurately represent the distribution of the
feature space, even more when conditioning on the target labels $Y$. Second,
once an estimation of the distributions $P_{Test}(X)$ and
$P_{Train}(X|Y=\ell_{j})$ has been made, there are also various ways to
predict the target probabilities $P_{Test}(Y=\ell_{j})$ from these
estimations.

The methods discussed in this chapter tackle these issues in various ways. One
basic approach to tackle the first issue has, for instance, already been
introduced when discussing the _Adjusted Count_. In the _Adjusted Count_
approach, information on the distribution of the features $X$ was derived by
the means of applying a classifier $c(X)$ and considering the distribution of
their outputs $P(c(X))$ instead of $P(X)$. That way, Equation 2 would be
transformed to the set of linear equations

$P_{Test}(c(X)={\ell_{i}})=\sum_{j=1}^{L}P_{Train}(c(X)={\ell_{i}}|Y={\ell_{j}})P_{Test}(Y={\ell_{j}}),\quad
i\in\\{1,\dots,L\\}.$ (3)

However, there are also methods that do not apply classifiers, and instead,
for instance, estimate $P(X)$ based on the distributions of single features,
or in terms of distances between individual instances in the data.
Regarding the second issue, most of the presented methods translate Equation 2
into a set of linear equations, and then minimize some distance function
between the left- and right-hand-side expressions, subject to the constraint
that $\sum_{j=1}^{L}P_{Test}(Y=\ell_{j})=1$ has to hold. This common pattern
has already been noted by Firat [10].

Among all the methods of this category, we are comparing the following
methods:

  1. 1.

Generalized Adjusted Count Models (GAC, GPAC). As described above, the most
simple work-around to avoid estimating $P(X)$ is to apply a classifier to
build a system of linear questions as in Equation 3, and solve it via
constrained least-squares regression [10]. That approach can be considered as
a Generalized Adjusted Count (GAC) method, which also naturally includes the
multiclass case. Similarly, one can obtain the Generalized Probabilistic
Adjusted Count (GPAC) method, by utilizing the confidence scores from
probabilistic classifiers as in the _PAC_ method.

  2. 2.

The DyS Framework (DyS, FMM). More recently, Maletzke et al. [20] proposed the
_DyS_ framework, in which the main idea is to utilize confidence scores
resulting from the decision functions of a binary classifier. More precisely,
the confidence scores obtained on the training data are divided into bins, and
then the probability that the confidence of an instance ends up in that bin is
estimated from the training set. Thus, in our context the number of linear
equations we obtain from Equation 2 equals the chosen number of bins, which,
next to the distance function that this set of equations is optimized on, can
be seen as a parameter of this framework. A main drawback of this framework is
that it only works for the binary case, and that many of the distance
functions that were proposed and evaluated for this framework are not convex,
requiring methods such as ternary search to estimate the optimal solution.
Since in their evaluation, using the Topsøe distance [6] has proven to yield
consistently good results, we are applying this setup as _DyS_ method in our
experiments. Furthermore, it is noteworthy that _Forman’s mixture model (FMM)_
[12] can also be regarded as a method of this framework that uses Manhattan
distance along with a particularly large number of bins.

  3. 3.

readme. Instead of applying a classifier, the _readme_ method [17] directly
models the distribution of the feature space by counting the co-occurences of
the individual features $X_{i}$. This method was originally developed to be
used on document-term matrices in the context of text mining. Due to sparsity
issues, it requires all features to be binned, and only a subset of all
features is projected on to form the system as in Equation 2, which is then
optimized by means of general least-squares regression. In practice, one
samples an ensemble of such subsets and averages the corresponding results in
the end to obtain the final prediction.

  4. 4.

Hellinger Distance Minimization (HDx, HDy). González-Castro et al. [16] have
proposed two methods that are based on minimizing the Hellinger distance to
match distributions in Equation 2. Similar to the _readme_ method, in the
_HDx_ method one considers the distributions of single features $X_{i}$, while
in the _HDy_ method, one applies classifiers as in the $GAC$ method to form
the system in Equation 3.

  5. 5.

Friedman’s Method (FM). Similar to the _GPAC_ method, Friedman [13] proposed
to utilize the confidence scores from probabilistic classifiers. However,
instead of averaging the class-conditional confidence scores, he proposes to
utilize the fraction of class-conditional confidence scores that are above and
below the observed class prevalences in the training data.

  6. 6.

Energy Distance Minimization (ED). As the name of this method suggests, its
core idea is to minimize the energy distance between the left-hand and right-
hand side distribution in Equation 2. In that context, the distribution of the
feature space is intrinsically modeled by the Euclidean distances between
individual instances, and therefore no classifiers or additional parameters
are required [19].

  7. 7.

The EM Algorithm for Quantification (EM). This method applies the classic _EM
algorithm_ [5] on the outputs of probabilistic classifiers to adjust them for
potential distribution shift between the target class distributions in
training and test data. While quantification was not the main focus in the
original proposal of the algorithm [25], the sought-for target class
prevalences are obtained as a side-product.

  8. 8.

CDE Iteration (CDE). The _class distribution estimation (CDE) iterator_ [29]
applies principles from cost-sensitive classification to account for changes
in class distributions between training and test data. For that purpose, the
misclassification costs are updated iteratively, and in the original
proposition of the algorithm, the underlying classifier is retrained in every
iteration step. In our experiments, we use the more efficient variant proposed
by Tasche [28], in which each iteration rather updates the decision threshold
of an underlying probabilistic classifier. For this variant, Tasche has also
proven that the iteration will eventually converge.

###  3.3 Classifiers for Quantification

Classifiers for quantification apply established classification methods to the
setting of quantification. The main approach behind most of these methods is
to optimize such established classifiers based on a loss function that
minimizes the quantification error, and then estimate the class distributions
based on the predictions of the individual instances. Thus, these approaches
are all in some sense variants of the _CC_ method. In our experiments, we
included the following methods:

  1. 1.

Classify and Count (CC). This trivial approach applies a classifier and counts
the number of times each class is predicted [12].

  2. 2.

Probabilistic Classify and Count (PCC). This approach takes probabilistic
predictions, i.e., continuous values between zero and one, and averages the
predictions of all instances to estimate the class prevalences [12, 3].

  3. 3.

$\text{SVM}^{perf}$optimization (SVM-Q, SVM-K). This pair of methods applies
the so-called $\text{SVM}^{perf}$classifier, which is an adaptation of
traditional support vector machines which can be optimized for multivariate
loss functions [18]. Based on this algorithm, multiple classifiers with
different quantification-oriented loss functions have been proposed. For
instance, Esuli et al. [8] have proposed to use the Kullback-Leibler
Divergence (SVM-K), while Barranquero et al. [2] have developed the _Q-
measure_ for that purpose (SVM-Q).

  4. 4.

Nearest-Neighbor based Quantification (PWK). Barranquero et al. [1] have
adapted nearest-neighbor based classification to the setting of
quantification. In their $k$-NN approach, they apply a weighting scheme which
applies less weight on neighbors from the majority class.

  5. 5.

Quantification Forests (QF, QF-AC). The decision tree and random forest
classifiers have been adapted for quantification by Milli et al. [22]. Other
than in the traditional approach, the authors propose that the split in each
decision tree is made based on a quantification-oriented loss functions. Since
in their original proposition, applying the _AC_ method on the predictions
from these random forests yielded particularly strong results, we include both
the _Quantification Forests_ as well as the _AC_ adaption of them in our
experiments.

###  3.4 Multiclass Quantification

In the literature on quantification, multiclass quantification has received
surprisingly little attention so far, despite Forman [12] pointing out that
this task is a much harder task than binary quantification. In our comparative
evaluation, we also take a closer look into this scenario. In contrast to
classification, approaches based on a one-vs.-one comparison of two target
labels are to the authors’ knowledge not used for quantification, potentially
since it is unclear how to aggregate the results. Therefore, methods for
multiclass quantification can be broadly distinguished into two categories:

  1. 1.

Natural Multiclass Quantifiers. Like in classification, some quantification
algorithms can naturally handle the multiclass setting as well. This is the
case for most distribution matching methods, as by Equation 2, there is no
constraint on the number of classes that are summated. Further,
quantification-oriented classifiers such as Quantification Forests can handle
the multiclass setting as well, since the underlying classifier allows for it.

  2. 2.

One-vs.-Rest Quantifiers. Traditional quantification methods such as the
_count adaptation methods_ as introduced by Forman [12] have been designed for
the binary setting only. To extend such methods to the multiclass setting, one
can estimate the prevalence of each individual class in a one-vs.-rest
fashion, and then normalize the resulting estimations afterwards so that they
sum to 1. Next to all _count adaptation_ methods, we also applied this
strategy for the distribution matching methods from the $DyS$ framework, as
these do not naturally generalize to the multiclass setting.

An overview regarding which multiclass strategy is used for each
quantification algorithm is provided in Table 1. For the _SVM-K_ , _SVM-Q_ and
_QF-AC_ methods we did not conduct any multiclass experiments, as the
underlying implementations did not provide a multiclass feature. Furthermore,
for the CDE iterator we did not run multiclass experiments either, since the
individual one-vs.-rest predictions yielded extreme predictions of either 0 or
1 regularly.

##  4 Experimental Setup

Overall, we compare 24 algorithms on 40 datasets. In the following, we provide
details regarding the datasets, sampling protocols, algorithmic parameters,
and evaluation measures. The implementation of the algorithms and experiments
can be found on
GitHub111https://github.com/tobiasschumacher/quantification_paper.

###  4.1 Datasets

We applied all algorithms on a broad range of 40 datasets collected from the
UCI machine learning repository222https://archive.ics.uci.edu/ml/index.php and
from Kaggle333https://www.kaggle.com/datasets. An overview of the datasets
that we used in our experiments is provided by Table 2. It includes
information on the amount of instances per dataset $N$, the number of features
$D$, the number $L$ of classes of the corresponding target variable, and an
indicator about whether or not a dataset contains non-categorical (continuous)
features. The non-categorical aspect is relevant for some quantification
algorithms such as _readme_ that require a finite feature space.

Dataset | Abbr. | $D$ | Non-Categorical | $N$ | $L$ | Source
---|---|---|---|---|---|---
Internet Advertisements | ads | 1560 | Yes | 2359 | 2 | UCI
Adult | adult | 89 | Yes | 45222 | 2 | UCI
Student Alcohol Consumption | alco | 57 | Yes | 1044 | 2 | Kaggle
Avila | avila | 10 | Yes | 20867 | 2 | UCI
Breast Cancer Wisconsin (Diagnostic) | bc-cat | 31 | Yes | 569 | 2 | UCI
Breast Cancer Wisconsin (Original) | bc-cont | 10 | Yes | 683 | 2 | UCI
Bike Sharing Dataset | bike | 59 | Yes | 17379 | 4 | UCI
BlogFeedback | blog | 280 | Yes | 52397 | 4 | UCI
MiniBooNE Particle Identification | boone | 50 | Yes | 129569 | 2 | UCI
Credit Approval | cappl | 44 | Yes | 653 | 2 | UCI
Car Evaluation | cars | 22 | No | 1728 | 2 | UCI
Default of Credit Card Clients | ccard | 34 | Yes | 30000 | 2 | UCI
Concrete Compressive Strength | conc | 8 | Yes | 1030 | 3 | UCI
Superconductivity Data | cond | 89 | Yes | 21263 | 4 | UCI
Contraceptive Method Choice | contra | 13 | Yes | 1473 | 3 | UCI
SkillCraft1 Master Table | craft | 18 | Yes | 3338 | 3 | UCI
Diamonds | diam | 22 | Yes | 53940 | 3 | Kaggle
Dota2 Games Results | dota | 116 | No | 102944 | 2 | UCI
Drug Consumption | drugs | 136 | Yes | 1885 | 3 | UCI
Appliances Energy Prediction | ener | 25 | Yes | 19735 | 3 | UCI
FIFA 19 Complete Player Dataset | fifa | 117 | Yes | 14751 | 4 | Kaggle
Solar Flare | flare | 28 | No | 1066 | 2 | UCI
Electrical Grid Stability Simulated Data | grid | 11 | Yes | 10000 | 2 | UCI
MAGIC Gamma Telescope | magic | 10 | Yes | 19020 | 2 | UCI
Mushroom | mush | 111 | No | 8124 | 2 | UCI
Geographical Original of Music | music | 116 | Yes | 1059 | 2 | UCI
Musk (Version 2) | musk | 166 | Yes | 6598 | 2 | UCI
News Popularity in Multiple Social Media Platforms | news | 60 | Yes | 39644 | 4 | UCI
Nursery | nurse | 27 | No | 12960 | 3 | UCI
Occupancy Detection | occup | 5 | Yes | 20560 | 2 | UCI
Phishing Websites | phish | 31 | No | 11055 | 2 | UCI
Spambase | spam | 58 | Yes | 4601 | 2 | UCI
Students Performance in Exams | study | 19 | Yes | 1000 | 2 | Kaggle
Telco Customer Churn | telco | 45 | Yes | 7032 | 2 | Kaggle
First-order Theorem Proving | thrm | 51 | Yes | 6117 | 3 | UCI
Turkiye Student Evaluation | turk | 31 | No | 5820 | 3 | UCI
Video Game Sales | vgame | 132 | Yes | 16719 | 4 | Kaggle
Gender Recognition by Voice | voice | 20 | Yes | 3168 | 2 | Kaggle
Wine Quality | wine | 14 | Yes | 6497 | 4 | UCI
Yeast | yeast | 9 | Yes | 1484 | 5 | UCI

Table 2: Datasets used in our experiments. $D$ indicates the number of
features, $L$ indicates the number of classes of the target variable, $N$
corresponds to the number of instances in the data, and Non-Categorical
indicates whether or not a dataset contains features that required binning.
Note that this latter aspect is relevant for quantification algorithms such as
_readme_ that require a finite feature space

Out of the 40 datasets, 17 had a non-binary set of target labels, or were even
regression datasets. The regression datasets were converted to both multi-
class and binary datasets by binning the values of the target variable. This
was usually done with the abstract goal of achieving groups of similar size
with respect to the number of instances to allow for a more robust basis for
potential shifts in the following steps. The cut-off points for the bins were
determined manually after looking at the distribution of the target values.
Furthermore, the real multiclass datasets were also converted to binary
datasets. In these cases, we kept the most populated class as is, and merged
the other classes into a single class, like in a one-vs.-rest classification
problem. By doing so, we preserved meaningful class semantics that classifiers
and quantifiers could recognize. All datasets have been preprocessed the same
way as for standard classification, including dummy coding their non-ordinal
features, rescaling their continuous features, and removing missing values.
Furthermore, to enable the application of algorithms that require a finite
feature space, we created a variation of each dataset in which all non-
categorical features were binned. All algorithms that could handle a non-
finite feature space were run on the non-binned datasets. While one may argue
that due to these alterations in the datasets the results would be less
comparable, the binning procedure ultimately simulates the loss of information
that one would have to accept when applying such restricted algorithms in the
first place.

Overall, these datasets represent a wide range of domains, and are differently
shaped in terms of their number of instances as well as in the design of their
feature spaces.

###  4.2 Sampling Strategy

As we aim to evaluate quantifiers under a large set of diverse conditions, we
chose a sampling approach in which we varied (i) the training distribution,
(ii) the test distribution and (iii) the (relative) sizes of training and test
datasets. Regarding training and test distributions, in the binary case, we
considered different prevalences of training positives $pos_{Train}$ and test
positives $pos_{Test}$ in the respective sets

$\displaystyle pos_{Train}\in\\{0.05,0.1,0.3,0.5,0.7,0.9\\}\quad\text{and}$
$\displaystyle
pos_{Test}\in\\{0,0.01,0.05,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9\\}.$

In both distributions, we sampled from the full interval $[0,1]$, including
also very unbalanced and thus presumably difficult settings with only very few
(or even no) positive labels in the training and/or test set.

Concerning the multiclass case, we have considered datasets with a varying
number of $L\in\\{3,4,5\\}$ different target classes. For each of these values
of $L$, we fixed a set of three training and five test class distributions,
representing relatively uniform as well as polarized class distributions,
which can be seen in Table 3.

$L$ | Training Distributions $P_{Train}(Y)$ | Test Distributions $P_{Test}(Y)$
---|---|---
3 |  |  $(0.2,0.5,0.3)$,
---
$(0.05,0.8,0.15)$,
$(0.35,0.3,0.35)$
|  $(0.1,0.7,0.2)$,
---
$(0.55,0.1,0.35)$,
$(0.35,0.55,0.1)$,
$(0.4,0.25,0.35)$,
$(0.,0.05,0.95)$
4 |  |  $(0.5,0.3,0.1,0.1)$,
---
$(0.7,0.2,0.1,0.1)$,
$(0.25,0.25,0.25,0.25)$
|  $(0.65,0.25,0.05,0.05)$,
---
$(0.2,0.25,0.3,0.25)$,
$(0.45,0.15,0.2,0.2)$,
$(0.2,0,0,0.8)$,
$(0.3,0.25,0.35,0.1)$
5 |  |  $(0.05,0.2,0.1,0.2,0.45)$,
---
$(0.05,0.1,0.7,0.1,0.05)$,
$(0.2,0.2,0.2,0.2,0.2)$
|  $(0.15,0.1,0.65,0.1,0)$,
---
$(0.45,0.1,0.3,0.05,0.1)$,
$(0.2,0.25,0.25,0.1,0.2)$,
$(0.35,0.05,0.05,0.05,0.5)$,
$(0.05,0.25,0.15,0.15,0.4)$
Table 3: List of training distributions $P_{Train}(Y)$ and test distributions
$P_{Test}(Y)$ considered for experiments in the multiclass setting, ordered by
number of target classes $L$. In both the columns for training and test
distribution, each row represents a distribution of instances that was sampled
from the corresponding data. For instance, assuming that for a dataset with
$L=3$ classes, the corresponding labels are given by $Y\in\\{1,2,3\\}$, the
first row among the column of training distribution indicates that in our
experiments, we have sampled training sets where Label $1$ had a prevalence of
$0.2$, Label $2$ had a prevalence of $0.5$, and Label $3$ had a prevalence of
$0.3$. For each combination of training and test distributions, we generated
ten test scenarios by taking different samples.

Both in the binary and the multiclass settings, we considered splits with
relative amounts of training versus test data samples in

$\\{(0.1,0.9),(0.3,0.7),(0.5,0.5),(0.7,0.3)\\},$

thereby simulating scenarios in which we have little as well as relatively
much information at hand to train our models. We omitted splits with 90%
training data to save computational resources, as the computational complexity
of most algorithms in our experiments is determined by the size of training
data rather than test data. Even without this particular split, in the binary
setting we obtain a large number of 288 combinations of training
distributions, test distributions, training/test-splits, and in the multiclass
setting we obtained 60 of such combinations for each dataset.

To collect experimental data from each dataset which satisfies these
constraints, we use undersampling, selecting from a given dataset as many data
instances as possible. We illustrate this sampling strategy with an example:
Assume a dataset with 1000 instances and a binary class attribute, consisting
of 700 positive and 300 negative instances. As an example evaluation scenario,
we aim to sample data with an 80%/20% split in training and test sets, and
with a 60%/40% distribution of positive and negative instances in both
training and test sets. Splitting the 300 negative instances randomly 80/20,
we have $0.8\cdot 300=240$ negative instances available for training and
$0.2\cdot 300=60$ instances available for the test data. To obtain a 60/40
distribution of positives and negatives in the training data, we therefore
have to choose $240:\frac{40}{60}=360$ positive instances to include in the
training data, which we randomly sample from the full set of positive
instances. The positives for the test data are randomly sampled analogously.
Note that the instance count for each label imposes a constraint on the number
of sampled instances with other labels. Overall, we use the maximum number of
instances for each label that satisfies all constraints.

This sampling procedure can lead to a relatively small subset compared to the
whole corpus in cases where the class distributions we aim to sample strongly
deviates from the natural class distributions in the given dataset. This makes
the quantification task comparatively more challenging in these settings.

To address possible variances in the drawn samples, we made ten independent
draws for each combination of distributions that could occur within our
protocol, and ran all corresponding algorithms on each of these draws. To
ensure the reproducibility of all these draws, we used a set of ten fixed
seeds for the random number generators. In total, for the binary setting, we
therefore performed 2880 draws per dataset, which, considering that we applied
24 algorithms on 40 datasets, yielded 2,764,800 experiments for that setting.
Including 204,000 additional experiments in the multiclass case and 2,666,520
more experiments on tuned alternative base classifiers (cf. Section 5.3), we
conducted for this paper a total number of more than 5 million experiments in
our evaluation.

###  4.3 Algorithms and Parameter Settings

In our experiments, we compared all algorithms described in Section 3 and
listed in overview Table 1. Except for the $\text{SVM}^{perf}$-based
quantifiers and _Quantification Forests_ , all algorithms were implemented
from scratch in Python 3, using scikit-learn as base implementation for the
underlying classifiers, and the package cvxpy [7] to solve constrained
optimization problems. For the $\text{SVM}^{perf}$algorithm, we used the
corresponding open source software package by Joachims [18], and adapted the
code that Esuli et al. [9] have used as a baseline for their _QuaNet_ method
to connect Joachim’s C++ implementation to Python. Regarding _Quantification
Forests_ , we used the original implementation that was kindly provided by the
authors [22].

As the focus of this work is on a general comparison of quantification
algorithms, we initially fixed for all algorithms a set of default parameters
based on which the main experiments were conducted. When choosing the
hyperparameters of each model, we followed recommendations of the original
papers where possible. For all quantification methods that required a base
classifier, we used the same logistic regression classifier for each dataset
split. The logistic regression model was chosen since it is one of the most
established and popular base classifiers and also provides calibrated class
probability scores that are required for quantification methods such as the
_PAC_ , _EM_ , or _FM_ methods. That way, results from different quantifiers
could not be biased by differences in the underlying classification
performances. We acknowledge that fine-tuning the hyperparameters of each
quantifier for each dataset could overall improve the performance, but argue
that fixing parameters once allows for a fairer comparison of individual
approaches and makes larger numbers of algorithm runs computationally
feasible. However, since one could suspect a strong dependence of the
quantification performances from the performance of the underlying
classifiers, we further conducted a series of experiments in which we used
stronger classifiers with tuned parameters, see Sections 4.3.2 and Section
5.3. Below, we will first outline the parameter settings for the main
experiments before giving details on the experiments in which we used tuned
classifiers.

####  4.3.1 Parameter Settings in the Main Experiments

For the main experiments, we made the following choices regarding the
algorithms’ hyperparameters:

  * •

As mentioned above, for all methods that utilize a classifier to perform
quantification, we used the logistic regression classifier with the default
L-BFGS solver along with its built-in probability estimator as provided by
scikit-learn and set the number of maximum iterations to 1000. We always used
stratified 10-fold cross-validation on the training set when estimating the
misclassification rates or computing the set of scores and thresholds that
were needed by the quantifiers.

  * •

In all adaptations of the _Adjusted Count_ that apply threshold selection
policies, namely the _TSX_ , _TS50_ , _TSMax_ and _MS_ methods, as well as in
_Forman’s mixture model (FMM)_ , we reduced the sets of scores and thresholds
obtained from cross-validation by rounding to two decimals. In the _MS_
algorithm, we followed Forman’s recommendation to only use models that yield a
value of at least 0.25 in the denominator of Equation 1.

  * •

In _Forman’s mixture model (FMM)_ , we also reduced the sets of scores and
thresholds obtained from cross-validation by rounding to two decimals.
Afterwards, we have chosen a large number of 100 bins, approximating Forman’s
original proposal of using a single bin for each confidence score occurring in
the data.

  * •

For the _DyS_ algorithm, we chose to divide its confidence scores into 10
bins, as this number of bins appeared to yield consistently strong results in
the study by Maletzke et al. [20].

  * •

For the _EM_ algorithm and the _CDE_ iterators, we chose $\varepsilon=10^{-6}$
as the convergence parameter, and limited the number of iterations to a
maximum of $m=1000$ iterations, which was reached only very rarely.

  * •

For the _readme_ algorithm, we set the size of each feature subset to
$\lfloor\log_{2}(D)+1\rfloor$, with $D$ denoting the number of features in
$X$. We considered an ensemble of 50 subsets, which were all drawn uniformly.

  * •

In the _QF_ and _QF-AC_ algorithms, we used the weka-based implementation that
has kindly been provided by the authors. We left all parameters at their
default values, including the size of the forest of 100 trees.

  * •

For both the _SVM-Q_ and the _SVM-K_ method, we chose $C=1$ as the
regularization coefficient, which was however decreased to $C=0.1$ when there
were more than 10,000 training samples. This adaptation was chosen as in our
experiments we observed that when large amounts of training data were present,
a higher regularization parameter would slow down the convergence of the
optimization significantly.

  * •

For the _PWK_ algorithm, we chose a neighborhood size of $k=10$, and a
weighting parameter of $\alpha=1$, as different weight values did not yield
significantly better results in the study by Barranquero et al. [2].

  * •

In the rare case that in one-vs.-rest quantification all individual class
prevalences were predicted as 0, we returned the uniform distribution as
prevalence estimation.

####  4.3.2 Parameter Settings in the Experiments on Tuned Classifiers

Many quantification methods rely on the predictions of an underlying base
classifier to form their class prevalence estimations. Since the quality of
these underlying classification models could have a strong impact on the
performance of the quantifier, we evaluated the impact of applying more
advanced classification methods with tuned hyperparameters in our second set
of experiments. For that purpose, we conducted experiments with four
classification models, namely random forests, AdaBoost, RBF-kernel support
vector machines, and logistic regression models. For each of these
classifiers, we conducted a grid search to optimize the hyperparameter on
every single dataset split in our experiments. Due to scalability issues,
which in particular occurred for the SVM classifier, we however restricted
ourselves to the 24 datasets which have not more than 10,000 instances in
total. After having determined their optimal parameter configuration for each
dataset split, we used each of the four classification models with their
optimal parameterization as base classifiers for the quantification methods.

For the _CC_ , _AC_ , _GAC_ , and _HDy_ methods, all four classification
models could be applied, as these only require the pure (mis)classification
rates from the training data for their estimations. For all quantifiers which
require scores from a classifier’s decision function, namely the threshold
selection policies _TS50_ , _TSX_ , _TSMax_ , and _MS_ as well as the _DyS_
and _FMM_ method, we only used the support vector classifier and the logistic
regression model, since AdaBoost and random forests do not provide calibrated
decision functions or confidence scores. Furthermore, for all quantifiers
which require probability scores, we only applied the tuned logistic
regression model, since it is the only method that really provides calibrated
probability scores.

In addition to these experiments on tuned classifiers, we also ran additional
experiments on the same datasets with the _SVM-K_ and _SVM-Q_ methods, in
which we applied an RBF-kernel instead of the default linear kernel. Due to
computational reasons, hyperparameter optimization was however not feasible
when using this kernel, and we thus resorted to a default parameter
configuration where we chose $C=1$ as the regularization coefficient and
$\gamma=1/D$ with $D$ denoting the amount of training features. Regarding the
exact parametrizations of the algorithms, we made the following choices:

  * •

All parameters of the quantification methods which do not regard the
underlying classifiers were kept as described in Section 4.3.1.

  * •

For the logistic regression classifier, in the grid search we varied the
regularization weight $C$ within the set
$\\{2^{i}:i\in\\{-15,-13,-11,\dots,13,15\\}\\}$. Like in previous experiments,
we applied the L-BFGS solver to efficiently learn the corresponding models,
and set the number of maximum iterations to 1000.

  * •

For the random forest, we varied the maximum numbers of features considered
per tree among the values $\\{2^{i}:i\in\\{1,2,\dots,11\\}\\}$, and the
minimum number of samples per tree, which we considered as the main parameter
to control the tree size, within the set $\\{2^{i}:i\in\\{1,2,\dots,7\\}\\}$.
Regarding the tree size, we kept a fixed high number of 1000 trees, since it
is well-established that choosing a high number of trees yields more reliable
results than any lower number of trees.

  * •

In the support vector classifier, we varied the regularization weight $C$ and
the kernel parameter $\gamma$. We varied the first in the range
$C\in\\{2^{i}:i\in\\{-5,-3,-1,\dots,13,15\\}\\}$ and the latter in
$\gamma\in\\{2^{i}:i\in\\{-17,-15,-13,\dots,3,5\\}\\}$.

  * •

Finally, for the AdaBoost classifier, there is a well-established trade-off
between the number of classifiers and the learning rate. Therefore, we only
varied the learning rate
$\alpha\in\\{2^{i}:i\in\\{-19,-17,-13,\dots,1,3\\}\\}$ and set the number of
weak classifiers to a medium amount of 100.

###  4.4 Evaluation

Next, we describe the error measures that we used in our evaluation, as well
as the procedure used to rank the quantification algorithms and determine
statistically significant differences in the performance of the algorithms we
have compared.

####  4.4.1 Error Measures for Quantification

The choice of performance measures for quantification is in itself not a
trivial issue, and for a thorough review and discussion of existing
quantification measures we point to a recent survey by Sebastiani [26]. To
evaluate the quantification performances in our experiments, we focus here on
the absolute error (AE) and the normalized Kullback-Leibler divergence (NKLD).
In the following, we let $p\in\mathbb{R}^{L}$ denote the true distribution of
labels $Y$ in an unseen test set, and $\theta\in\mathbb{R}^{L}$ denote the
distribution of labels $Y$ that has been predicted from a given quantifier on
the test set. The absolute error between true distribution $p$ and estimated
distribution $\theta$ is then given by

$e_{AE}(p,\theta):=\sum_{i=1}^{L}|p_{i}-\theta_{i}|\ ,$

whereas the normalized Kullback-Leibler divergence between $p$ and $\theta$ is
defined as

$e_{\text{NKLD}}(p,\theta):=2\cdot\frac{\exp\left\\{e_{\text{KLD}}(p,\theta)\right\\}}{1+\exp\left\\{e_{\text{KLD}}(p,\theta)\right\\}}-1\
,$

with

$e_{\text{KLD}}(p,\theta):=\sum_{i=1}^{L}p_{i}\log\left(\frac{p_{i}}{\theta_{i}}\right)\
$

denoting the Kullback-Leibler divergence. Since the Kullback-Leibler
divergence is not defined when $\theta_{i}=0$ and $p_{i}\not=0$ for some
$i\leq L$, we smoothed the distributions by a small value
$\varepsilon=10^{-8}$ to avoid this problem.

We chose the AE score due to its interpretability as well as its robustness
against outliers. Note that this definition corresponds directly to the total
variation between the predicted categorical probability distribution and the
ground-truth distribution. In contrast to related studies as conducted by
[16], we do not use the _Mean Absolute Error (MAE)_ , i.e., we do not divide
by the number $L$ of predicted classes. This avoids having different upper
bounds for the error depending on $L$, which would make results less
comparable, specifically in multiclass quantification settings. The NKLD
score—in contrast to the AE score—particularly punishes quantifiers which
marginalize the minority class. Both measures are bounded to the same interval
for both binary and multiclass quantification, with both values obtaining
their minimum (and optimal value) at 0, and the maximum of the $AE1$ score
being 2, while the maximum NKLD value is 1.

####  4.4.2 Statistical Evaluation of Performance Rankings

Regarding the actual comparison of the given quantifiers, we adapted a
statistical procedure established by Demšar [4], who, in the context of
classification, suggested to conduct comparisons of multiple algorithms by
statistical tests in a two-step approach that is based on the performance
rankings of all algorithms considered with respect to a number of datasets
they were applied on.

Within that two-step approach, at first a Friedman test [14] is conducted on
the null hypothesis that all algorithms perform equally well over a given
ensemble of datasets with respect to a chosen error measure. If that null
hypothesis is rejected, one may follow up with the Nemenyi post-hoc test [24]
to compare the performance rankings of each algorithm per dataset with each
other, and determine which algorithms differ from each other in a
statistically significant way. The margin of statistical significance is
modeled by the critical distance value, which is determined by both the number
of algorithms and datasets that are considered as well as the chosen
significance level $\alpha$.

While in classification the underlying rankings would usually be obtained
based on a cross-validated accuracy score, in our context we averaged the
quantification errors obtained from all the settings in our protocol over each
dataset. Based on these average errors, for each dataset, we then determined a
ranking of our algorithms for this dataset. To account for outliers, we also
averaged the resulting scores via the mean and not the median value, which by
design of our error measures became more noticeable when averaging with
respect to the NKLD errors.

##  5 Results

This section presents the results of our extensive experimental evaluation for
binary quantification (i.e., labels with exactly two values) and multiclass
quantification (i.e., labels with more than two values). For both types, we
start by first showing the main results, which aggregate the performance of
each algorithm over all datasets and settings. Then, we show more detailed
results for subtypes of scenarios, i.e., for different shifts (differences
between training and test distributions) and for varying amounts of training
data. Finally, we compare the performance of all presented algorithms in the
multiclass case, which is a setting that has not received much attention yet.

(a) Distribution of absolute errors (AE)

(b) Average rankings with respect to the AE

(c) Distribution of NKLD scores

(d) Average rankings with respect to the NKLD

Figure 1: Visual representation of the main results for binary quantification.
The top row shows results for the absolute error (AE), the bottom row for
normalized Kullback-Leibler divergence (NKLD) scores. On the left, letter-
value plots for the distribution of error score across all scenarios per
algorithm are shown. Colors indicate the category of the algorithm, with count
adaptation-based algorithms shown in blue, distribution matching methods in
orange, and adaptations of traditional classification algorithms in green.
Plots are scaled logarithmically above the dotted vertical threshold, and
linearly below. On the right, we plot the distributions of rankings with a
Nemenyi post-hoc test at 5% significance. For each algorithm, we depict the
average performance rank over all algorithms. Horizontal bars indicate which
average rankings do not differ to a degree that is statistically significant.
The critical difference (CD) was $5.6973$. Overall, _DyS_ , _MS_ , and _TSMax_
appear to work best in general.

###  5.1 Binary Quantification

We first describe our results for binary quantification, that is,
quantification with binary class labels.

####  5.1.1 Overall Results

| AC | PAC | TSX | TS50 | TSMax | MS | GAC | GPAC | DyS | FMM | readme | HDx | HDy | FM | ED | EM | CDE | CC | PCC | SVM-K | SVM-Q | PWK | QF | QF-AC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ads | 0.134 | 0.103 | 0.09 | 0.122 | 0.086 | 0.072 | 0.138 | 0.102 | 0.083 | 0.211 | 0.225 | 0.093 | 0.138 | 0.082 | 0.195 | 0.087 | 0.199 | 0.352 | 0.352 | 0.255 | 0.341 | 0.389 | 0.434 | 0.317
adult | 0.04 | 0.022 | 0.018 | 0.027 | 0.017 | 0.014 | 0.041 | 0.022 | 0.014 | 0.031 | 0.032 | 0.031 | 0.041 | 0.018 | 0.024 | 0.017 | 0.225 | 0.467 | 0.443 | 0.447 | 0.528 | 0.272 | 0.570 | 0.118
alco | 0.41 | 0.328 | 0.367 | 0.400 | 0.318 | 0.291 | 0.451 | 0.337 | 0.383 | 0.584 | 0.584 | 0.593 | 0.451 | 0.342 | 0.468 | 0.296 | 0.695 | 0.693 | 0.625 | 0.495 | 0.608 | 0.491 | 0.653 | 0.495
avila | 0.422 | 0.086 | 0.049 | 0.050 | 0.048 | 0.042 | 0.459 | 0.086 | 0.083 | 0.097 | 0.086 | 0.071 | 0.458 | 0.096 | 0.075 | 0.214 | 0.899 | 0.852 | 0.682 | 0.765 | 0.849 | 0.286 | 0.678 | 0.327
bc-cat | 0.136 | 0.112 | 0.088 | 0.158 | 0.078 | 0.064 | 0.193 | 0.112 | 0.105 | 0.246 | 0.276 | 0.121 | 0.193 | 0.062 | 0.093 | 0.207 | 0.315 | 0.38 | 0.390 | 0.304 | 0.753 | 0.091 | 0.315 | 0.151
bc-cont | 0.076 | 0.072 | 0.059 | 0.119 | 0.057 | 0.047 | 0.117 | 0.072 | 0.062 | 0.198 | 0.121 | 0.095 | 0.117 | 0.039 | 0.052 | 0.125 | 0.123 | 0.172 | 0.245 | 0.167 | 0.838 | 0.058 | 0.263 | 0.147
bike | 0.027 | 0.023 | 0.021 | 0.027 | 0.02 | 0.016 | 0.036 | 0.023 | 0.014 | 0.038 | 0.079 | 0.284 | 0.036 | 0.021 | 0.073 | 0.044 | 0.096 | 0.29 | 0.309 | 0.209 | 0.363 | 0.281 | 0.498 | 0.065
blog | 0.109 | 0.036 | 0.033 | 0.030 | 0.03 | 0.027 | 0.072 | 0.036 | 0.027 | 0.052 | 0.042 | 0.027 | 0.072 | 0.033 | 0.055 | 0.042 | 0.569 | 0.643 | 0.575 | 0.387 | 0.654 | 0.394 | 0.625 | 0.213
boone | 0.012 | 0.009 | 0.007 | 0.013 | 0.007 | 0.006 | 0.013 | 0.009 | 0.006 | 0.010 | 0.087 | 0.009 | 0.013 | 0.007 | 0.011 | 0.024 | 0.069 | 0.282 | 0.307 | 0.257 | 0.693 | 0.133 | 0.505 | 0.270
cappl | 0.233 | 0.155 | 0.15 | 0.215 | 0.15 | 0.129 | 0.289 | 0.156 | 0.181 | 0.394 | 0.374 | 0.54 | 0.289 | 0.115 | 0.222 | 0.087 | 0.302 | 0.473 | 0.465 | 0.330 | 0.322 | 0.257 | 0.514 | 0.292
cars | 0.087 | 0.08 | 0.059 | 0.101 | 0.059 | 0.050 | 0.113 | 0.08 | 0.061 | 0.201 | 0.229 | 0.399 | 0.113 | 0.059 | 0.154 | 0.087 | 0.180 | 0.299 | 0.306 | 0.228 | 0.227 | 0.229 | 0.485 | 0.296
ccard | 0.388 | 0.061 | 0.067 | 0.051 | 0.052 | 0.043 | 0.28 | 0.061 | 0.049 | 0.108 | 0.283 | 0.127 | 0.28 | 0.064 | 0.090 | 0.062 | 0.847 | 0.753 | 0.641 | 0.496 | 0.69 | 0.412 | 0.615 | 0.330
conc | 0.31 | 0.171 | 0.116 | 0.153 | 0.12 | 0.093 | 0.369 | 0.171 | 0.179 | 0.382 | 0.258 | 0.198 | 0.369 | 0.155 | 0.184 | 0.336 | 0.745 | 0.699 | 0.608 | 0.304 | 0.601 | 0.300 | 0.627 | 0.275
cond | 0.031 | 0.018 | 0.016 | 0.034 | 0.018 | 0.017 | 0.04 | 0.018 | 0.013 | 0.032 | 0.090 | 0.047 | 0.04 | 0.017 | 0.019 | 0.022 | 0.097 | 0.319 | 0.317 | 0.206 | 0.287 | 0.124 | 0.399 | 0.069
contra | 0.551 | 0.332 | 0.298 | 0.323 | 0.271 | 0.244 | 0.472 | 0.331 | 0.43 | 0.599 | 0.366 | 0.303 | 0.472 | 0.351 | 0.408 | 0.249 | 0.881 | 0.814 | 0.672 | 0.565 | 0.802 | 0.526 | 0.664 | 0.470
craft | 0.178 | 0.084 | 0.071 | 0.082 | 0.075 | 0.056 | 0.219 | 0.084 | 0.059 | 0.221 | 0.211 | 0.227 | 0.219 | 0.067 | 0.070 | 0.144 | 0.528 | 0.602 | 0.543 | 0.344 | 0.684 | 0.319 | 0.568 | 0.222
diam | 0.227 | 0.037 | 0.021 | 0.032 | 0.019 | 0.016 | 0.27 | 0.038 | 0.027 | 0.059 | 0.063 | 0.028 | 0.269 | 0.031 | 0.085 | 0.217 | 0.746 | 0.709 | 0.609 | 0.627 | 0.853 | 0.323 | 0.497 | 0.270
dota | 1.046 | 0.048 | 0.053 | 0.054 | 0.057 | 0.057 | 0.397 | 0.048 | 0.049 | 0.120 | 0.360 | 0.234 | 0.396 | 0.053 | 0.189 | 0.13 | 0.864 | 0.835 | 0.680 | 0.587 | 0.806 | 0.557 | 0.886 | 0.690
drugs | 0.158 | 0.118 | 0.13 | 0.148 | 0.13 | 0.109 | 0.174 | 0.119 | 0.104 | 0.296 | 0.163 | 0.413 | 0.174 | 0.104 | 0.114 | 0.134 | 0.134 | 0.421 | 0.428 | 0.318 | 0.337 | 0.275 | 0.504 | 0.181
ener | 0.248 | 0.04 | 0.036 | 0.040 | 0.036 | 0.032 | 0.224 | 0.04 | 0.034 | 0.102 | 0.207 | 0.102 | 0.224 | 0.041 | 0.067 | 0.12 | 0.699 | 0.672 | 0.596 | 0.399 | 0.742 | 0.270 | 0.741 | 0.500
fifa | 0.76 | 0.035 | 0.032 | 0.037 | 0.031 | 0.025 | 0.055 | 0.035 | 0.022 | 0.065 | 0.137 | 0.021 | 0.055 | 0.03 | 0.040 | 0.031 | 0.204 | 0.461 | 0.447 | 1.240 | 1.188 | 0.329 | 0.418 | 0.056
flare | 0.56 | 0.344 | 0.394 | 0.403 | 0.359 | 0.333 | 0.482 | 0.342 | 0.443 | 0.605 | 0.316 | 0.356 | 0.482 | 0.346 | 0.314 | 0.256 | 0.675 | 0.694 | 0.629 | 0.480 | 0.614 | 0.405 | 0.668 | 0.347
grid | 0.078 | 0.046 | 0.046 | 0.051 | 0.051 | 0.034 | 0.086 | 0.046 | 0.032 | 0.104 | 0.149 | 0.078 | 0.086 | 0.044 | 0.058 | 0.048 | 0.258 | 0.492 | 0.468 | 0.749 | 0.668 | 0.225 | 0.782 | 0.510
magic | 0.288 | 0.043 | 0.044 | 0.048 | 0.046 | 0.042 | 0.236 | 0.043 | 0.039 | 0.095 | 0.103 | 0.087 | 0.236 | 0.044 | 0.044 | 0.057 | 0.469 | 0.606 | 0.542 | 0.607 | 0.587 | 0.314 | 0.661 | 0.477
mush | 0.008 | 0.011 | 0.006 | 0.042 | 0.006 | 0.012 | 0.014 | 0.011 | 0.007 | 0.032 | 0.033 | 0.106 | 0.014 | 0.008 | 0.027 | 0.017 | 0.009 | 0.027 | 0.054 | 0.098 | 0.054 | 0.009 | 0.215 | 0.021
music | 0.48 | 0.324 | 0.372 | 0.379 | 0.333 | 0.312 | 0.462 | 0.324 | 0.392 | 0.593 | 0.682 | 0.748 | 0.462 | 0.328 | 0.404 | 0.257 | 0.840 | 0.748 | 0.651 | 0.465 | 0.572 | 0.449 | 0.741 | 0.609
musk | 0.083 | 0.07 | 0.068 | 0.085 | 0.064 | 0.056 | 0.096 | 0.069 | 0.06 | 0.166 | 0.434 | 0.208 | 0.096 | 0.068 | 0.126 | 0.065 | 0.188 | 0.367 | 0.379 | 0.248 | 0.321 | 0.276 | 0.489 | 0.177
news | 0.439 | 0.052 | 0.058 | 0.052 | 0.057 | 0.048 | 0.243 | 0.052 | 0.05 | 0.146 | 0.433 | 0.209 | 0.243 | 0.053 | 0.089 | 0.058 | 0.866 | 0.772 | 0.651 | 0.475 | 0.842 | 0.470 | 0.842 | 0.570
nurse | 0.000 | 0.002 | 0.84 | 1.036 | 0.839 | 0.841 | 0.000 | 0.002 | 0.004 | 0.155 | 0.448 | 0.891 | 0.000 | 0.000 | 0.024 | 0.02 | 0.002 | 0.000 | 0.024 | 0.001 | 0.000 | 0.128 | 0.065 | 0.005
occup | 0.008 | 0.017 | 0.005 | 0.040 | 0.005 | 0.005 | 0.034 | 0.017 | 0.006 | 0.034 | 0.020 | 0.023 | 0.034 | 0.006 | 0.012 | 0.103 | 0.098 | 0.125 | 0.192 | 0.241 | 0.531 | 0.015 | 0.113 | 0.012
phish | 0.815 | 0.023 | 0.021 | 0.036 | 0.021 | 0.018 | 0.029 | 0.023 | 0.015 | 0.046 | 0.058 | 0.231 | 0.029 | 0.019 | 0.033 | 0.014 | 0.026 | 0.188 | 0.212 | 0.188 | 0.153 | 0.137 | 0.364 | 0.042
spam | 0.103 | 0.069 | 0.04 | 0.061 | 0.04 | 0.034 | 0.236 | 0.069 | 0.046 | 0.111 | 0.177 | 0.442 | 0.236 | 0.047 | 0.082 | 0.265 | 0.603 | 0.595 | 0.537 | 0.261 | 0.638 | 0.204 | 0.667 | 0.298
study | 0.216 | 0.187 | 0.16 | 0.178 | 0.157 | 0.121 | 0.301 | 0.187 | 0.202 | 0.393 | 0.287 | 0.107 | 0.301 | 0.192 | 0.308 | 0.175 | 0.533 | 0.589 | 0.538 | 0.610 | 0.696 | 0.330 | 0.460 | 0.145
telco | 0.252 | 0.075 | 0.072 | 0.082 | 0.07 | 0.061 | 0.211 | 0.075 | 0.06 | 0.204 | 0.097 | 0.087 | 0.211 | 0.07 | 0.065 | 0.059 | 0.401 | 0.571 | 0.525 | 0.373 | 0.476 | 0.387 | 0.609 | 0.304
thrm | 0.488 | 0.318 | 0.294 | 0.323 | 0.274 | 0.240 | 0.462 | 0.318 | 0.383 | 0.553 | 0.534 | 0.348 | 0.462 | 0.309 | 0.355 | 0.266 | 0.861 | 0.773 | 0.655 | 0.491 | 0.629 | 0.444 | 0.698 | 0.495
turk | 0.585 | 0.248 | 0.295 | 0.295 | 0.252 | 0.251 | 0.477 | 0.246 | 0.293 | 0.500 | 0.351 | 0.229 | 0.477 | 0.28 | 0.211 | 0.164 | 0.881 | 0.847 | 0.684 | 0.558 | 0.64 | 0.529 | 0.844 | 0.702
vgame | 0.127 | 0.085 | 0.086 | 0.088 | 0.085 | 0.071 | 0.209 | 0.085 | 0.084 | 0.219 | 0.201 | 0.116 | 0.209 | 0.089 | 0.266 | 0.066 | 0.586 | 0.631 | 0.570 | 0.407 | 0.594 | 0.400 | 0.654 | 0.302
voice | 0.039 | 0.048 | 0.023 | 0.063 | 0.024 | 0.025 | 0.134 | 0.047 | 0.031 | 0.117 | 0.210 | 0.051 | 0.134 | 0.036 | 0.060 | 0.178 | 0.289 | 0.346 | 0.378 | 0.166 | 0.417 | 0.076 | 0.246 | 0.051
wine | 0.395 | 0.095 | 0.078 | 0.082 | 0.085 | 0.064 | 0.372 | 0.095 | 0.102 | 0.263 | 0.198 | 0.155 | 0.372 | 0.102 | 0.162 | 0.233 | 0.815 | 0.75 | 0.637 | 0.662 | 0.905 | 0.350 | 0.649 | 0.319
yeast | 0.57 | 0.238 | 0.237 | 0.283 | 0.213 | 0.189 | 0.471 | 0.241 | 0.295 | 0.523 | 0.343 | 0.359 | 0.471 | 0.261 | 0.246 | 0.38 | 0.873 | 0.839 | 0.680 | 0.569 | 0.881 | 0.428 | 0.672 | 0.450
Mean | 0.278 | 0.107 | 0.123 | 0.146 | 0.117 | 0.105 | 0.224 | 0.107 | 0.113 | 0.223 | 0.234 | 0.22 | 0.224 | 0.102 | 0.139 | 0.134 | 0.467 | 0.529 | 0.481 | 0.414 | 0.585 | 0.297 | 0.547 | 0.290

(a) Absolute error (AE)

| AC | PAC | TSX | TS50 | TSMax | MS | GAC | GPAC | DyS | FMM | readme | HDx | HDy | FM | ED | EM | CDE | CC | PCC | SVM-K | SVM-Q | PWK | QF | QF-AC
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ads | 0.078 | 0.074 | 0.042 | 0.066 | 0.028 | 0.029 | 0.077 | 0.074 | 0.019 | 0.066 | 0.052 | 0.017 | 0.076 | 0.039 | 0.044 | 0.027 | 0.186 | 0.134 | 0.108 | 0.071 | 0.107 | 0.134 | 0.152 | 0.101
adult | 0.013 | 0.005 | 0.003 | 0.004 | 0.002 | 0.002 | 0.017 | 0.005 | 0.001 | 0.003 | 0.003 | 0.003 | 0.016 | 0.002 | 0.003 | 0.001 | 0.199 | 0.175 | 0.139 | 0.129 | 0.184 | 0.061 | 0.294 | 0.052
alco | 0.287 | 0.251 | 0.269 | 0.295 | 0.171 | 0.17 | 0.277 | 0.258 | 0.191 | 0.258 | 0.225 | 0.298 | 0.273 | 0.26 | 0.164 | 0.102 | 0.779 | 0.391 | 0.254 | 0.167 | 0.238 | 0.147 | 0.319 | 0.199
avila | 0.37 | 0.06 | 0.017 | 0.015 | 0.011 | 0.012 | 0.255 | 0.059 | 0.018 | 0.019 | 0.013 | 0.015 | 0.244 | 0.055 | 0.016 | 0.156 | 0.847 | 0.641 | 0.292 | 0.335 | 0.592 | 0.063 | 0.395 | 0.237
bc-cat | 0.067 | 0.064 | 0.037 | 0.093 | 0.02 | 0.021 | 0.088 | 0.064 | 0.024 | 0.105 | 0.075 | 0.028 | 0.087 | 0.023 | 0.016 | 0.16 | 0.405 | 0.182 | 0.123 | 0.08 | 0.316 | 0.013 | 0.083 | 0.033
bc-cont | 0.03 | 0.039 | 0.015 | 0.066 | 0.012 | 0.018 | 0.051 | 0.039 | 0.009 | 0.087 | 0.022 | 0.016 | 0.051 | 0.015 | 0.007 | 0.087 | 0.181 | 0.066 | 0.060 | 0.035 | 0.447 | 0.006 | 0.067 | 0.032
bike | 0.006 | 0.007 | 0.004 | 0.005 | 0.003 | 0.002 | 0.013 | 0.007 | 0.000 | 0.003 | 0.012 | 0.126 | 0.012 | 0.005 | 0.012 | 0.007 | 0.097 | 0.073 | 0.077 | 0.044 | 0.116 | 0.063 | 0.215 | 0.012
blog | 0.061 | 0.009 | 0.008 | 0.007 | 0.005 | 0.005 | 0.035 | 0.009 | 0.002 | 0.007 | 0.004 | 0.003 | 0.033 | 0.008 | 0.006 | 0.004 | 0.572 | 0.305 | 0.214 | 0.102 | 0.333 | 0.103 | 0.375 | 0.074
boone | 0.002 | 0.001 | 0.001 | 0.001 | 0.001 | 0.000 | 0.002 | 0.001 | 0.000 | 0.000 | 0.013 | 0.001 | 0.002 | 0.000 | 0.001 | 0.001 | 0.027 | 0.07 | 0.075 | 0.058 | 0.462 | 0.023 | 0.324 | 0.260
cappl | 0.146 | 0.092 | 0.08 | 0.141 | 0.052 | 0.058 | 0.155 | 0.094 | 0.065 | 0.155 | 0.11 | 0.259 | 0.153 | 0.053 | 0.053 | 0.037 | 0.410 | 0.244 | 0.159 | 0.093 | 0.086 | 0.056 | 0.192 | 0.082
cars | 0.044 | 0.05 | 0.025 | 0.048 | 0.015 | 0.018 | 0.051 | 0.049 | 0.014 | 0.061 | 0.053 | 0.244 | 0.05 | 0.03 | 0.031 | 0.034 | 0.208 | 0.099 | 0.083 | 0.051 | 0.045 | 0.046 | 0.316 | 0.263
ccard | 0.258 | 0.029 | 0.025 | 0.017 | 0.011 | 0.012 | 0.164 | 0.029 | 0.005 | 0.02 | 0.074 | 0.028 | 0.158 | 0.023 | 0.015 | 0.015 | 0.831 | 0.446 | 0.260 | 0.16 | 0.386 | 0.109 | 0.267 | 0.093
conc | 0.222 | 0.129 | 0.057 | 0.091 | 0.037 | 0.041 | 0.155 | 0.129 | 0.072 | 0.149 | 0.066 | 0.064 | 0.153 | 0.09 | 0.045 | 0.325 | 0.795 | 0.494 | 0.245 | 0.074 | 0.306 | 0.072 | 0.323 | 0.132
cond | 0.008 | 0.003 | 0.002 | 0.006 | 0.002 | 0.003 | 0.016 | 0.003 | 0.001 | 0.003 | 0.014 | 0.01 | 0.016 | 0.003 | 0.003 | 0.003 | 0.144 | 0.101 | 0.082 | 0.043 | 0.069 | 0.021 | 0.170 | 0.030
contra | 0.449 | 0.245 | 0.206 | 0.234 | 0.128 | 0.121 | 0.241 | 0.245 | 0.227 | 0.266 | 0.107 | 0.108 | 0.236 | 0.258 | 0.159 | 0.125 | 0.839 | 0.579 | 0.286 | 0.197 | 0.381 | 0.166 | 0.391 | 0.285
craft | 0.119 | 0.048 | 0.025 | 0.035 | 0.015 | 0.019 | 0.105 | 0.048 | 0.010 | 0.058 | 0.048 | 0.093 | 0.103 | 0.026 | 0.013 | 0.089 | 0.728 | 0.317 | 0.199 | 0.09 | 0.306 | 0.076 | 0.310 | 0.150
diam | 0.178 | 0.014 | 0.003 | 0.006 | 0.003 | 0.004 | 0.139 | 0.014 | 0.003 | 0.011 | 0.008 | 0.003 | 0.132 | 0.011 | 0.013 | 0.185 | 0.807 | 0.473 | 0.244 | 0.309 | 0.663 | 0.082 | 0.321 | 0.260
dota | 0.881 | 0.015 | 0.017 | 0.019 | 0.013 | 0.013 | 0.2 | 0.014 | 0.009 | 0.025 | 0.104 | 0.088 | 0.186 | 0.016 | 0.042 | 0.024 | 0.840 | 0.583 | 0.289 | 0.225 | 0.499 | 0.189 | 0.684 | 0.474
drugs | 0.082 | 0.056 | 0.063 | 0.077 | 0.041 | 0.046 | 0.092 | 0.057 | 0.022 | 0.088 | 0.032 | 0.248 | 0.09 | 0.044 | 0.020 | 0.022 | 0.092 | 0.144 | 0.134 | 0.078 | 0.088 | 0.059 | 0.178 | 0.043
ener | 0.199 | 0.017 | 0.009 | 0.012 | 0.006 | 0.008 | 0.125 | 0.016 | 0.004 | 0.019 | 0.047 | 0.032 | 0.121 | 0.013 | 0.013 | 0.066 | 0.799 | 0.408 | 0.232 | 0.112 | 0.376 | 0.06 | 0.518 | 0.427
fifa | 0.775 | 0.013 | 0.008 | 0.009 | 0.005 | 0.006 | 0.027 | 0.013 | 0.002 | 0.009 | 0.025 | 0.002 | 0.025 | 0.008 | 0.005 | 0.002 | 0.252 | 0.164 | 0.141 | 0.979 | 0.951 | 0.081 | 0.132 | 0.007
flare | 0.395 | 0.245 | 0.274 | 0.304 | 0.197 | 0.2 | 0.294 | 0.242 | 0.213 | 0.29 | 0.087 | 0.182 | 0.289 | 0.232 | 0.097 | 0.081 | 0.707 | 0.419 | 0.256 | 0.159 | 0.243 | 0.11 | 0.324 | 0.137
grid | 0.029 | 0.015 | 0.009 | 0.009 | 0.006 | 0.005 | 0.034 | 0.015 | 0.002 | 0.017 | 0.028 | 0.012 | 0.033 | 0.009 | 0.008 | 0.014 | 0.409 | 0.188 | 0.151 | 0.596 | 0.425 | 0.045 | 0.552 | 0.430
magic | 0.266 | 0.009 | 0.008 | 0.009 | 0.006 | 0.007 | 0.101 | 0.009 | 0.005 | 0.015 | 0.016 | 0.019 | 0.099 | 0.008 | 0.006 | 0.014 | 0.523 | 0.377 | 0.197 | 0.371 | 0.358 | 0.077 | 0.468 | 0.406
mush | 0.001 | 0.001 | 0.001 | 0.009 | 0.001 | 0.001 | 0.002 | 0.001 | 0.000 | 0.003 | 0.004 | 0.029 | 0.002 | 0.001 | 0.004 | 0.001 | 0.004 | 0.003 | 0.006 | 0.016 | 0.007 | 0.001 | 0.041 | 0.002
music | 0.329 | 0.246 | 0.255 | 0.274 | 0.182 | 0.185 | 0.257 | 0.246 | 0.184 | 0.256 | 0.291 | 0.405 | 0.252 | 0.222 | 0.134 | 0.082 | 0.825 | 0.473 | 0.270 | 0.136 | 0.204 | 0.131 | 0.473 | 0.389
musk | 0.04 | 0.028 | 0.029 | 0.043 | 0.014 | 0.018 | 0.045 | 0.028 | 0.008 | 0.041 | 0.143 | 0.067 | 0.044 | 0.028 | 0.023 | 0.011 | 0.195 | 0.116 | 0.109 | 0.049 | 0.087 | 0.062 | 0.187 | 0.042
news | 0.345 | 0.015 | 0.017 | 0.016 | 0.011 | 0.011 | 0.107 | 0.015 | 0.006 | 0.031 | 0.139 | 0.056 | 0.102 | 0.015 | 0.013 | 0.006 | 0.839 | 0.483 | 0.268 | 0.144 | 0.466 | 0.134 | 0.623 | 0.479
nurse | 0.000 | 0.000 | 0.704 | 0.813 | 0.703 | 0.704 | 0.000 | 0.000 | 0.000 | 0.059 | 0.129 | 0.592 | 0.000 | 0.000 | 0.002 | 0.005 | 0.010 | 0.000 | 0.002 | 0.000 | 0.000 | 0.017 | 0.017 | 0.001
occup | 0.001 | 0.005 | 0.000 | 0.012 | 0.000 | 0.000 | 0.016 | 0.005 | 0.000 | 0.002 | 0.001 | 0.002 | 0.016 | 0.001 | 0.001 | 0.054 | 0.127 | 0.034 | 0.041 | 0.066 | 0.3 | 0.001 | 0.017 | 0.001
phish | 0.445 | 0.004 | 0.004 | 0.006 | 0.003 | 0.002 | 0.007 | 0.004 | 0.000 | 0.006 | 0.007 | 0.081 | 0.006 | 0.003 | 0.004 | 0.000 | 0.002 | 0.034 | 0.041 | 0.032 | 0.027 | 0.022 | 0.108 | 0.013
spam | 0.058 | 0.035 | 0.009 | 0.026 | 0.007 | 0.008 | 0.121 | 0.035 | 0.005 | 0.02 | 0.036 | 0.179 | 0.118 | 0.013 | 0.012 | 0.218 | 0.714 | 0.351 | 0.200 | 0.061 | 0.298 | 0.04 | 0.299 | 0.082
study | 0.124 | 0.114 | 0.082 | 0.102 | 0.058 | 0.056 | 0.144 | 0.114 | 0.078 | 0.152 | 0.072 | 0.025 | 0.142 | 0.102 | 0.088 | 0.084 | 0.684 | 0.336 | 0.202 | 0.213 | 0.283 | 0.082 | 0.166 | 0.047
telco | 0.204 | 0.037 | 0.027 | 0.035 | 0.017 | 0.023 | 0.119 | 0.039 | 0.012 | 0.05 | 0.015 | 0.027 | 0.116 | 0.031 | 0.012 | 0.007 | 0.527 | 0.283 | 0.186 | 0.099 | 0.151 | 0.104 | 0.377 | 0.272
thrm | 0.352 | 0.249 | 0.22 | 0.250 | 0.142 | 0.146 | 0.223 | 0.249 | 0.202 | 0.245 | 0.2 | 0.162 | 0.22 | 0.219 | 0.129 | 0.191 | 0.833 | 0.532 | 0.275 | 0.164 | 0.295 | 0.129 | 0.401 | 0.268
turk | 0.45 | 0.192 | 0.208 | 0.213 | 0.125 | 0.123 | 0.246 | 0.19 | 0.115 | 0.221 | 0.104 | 0.078 | 0.239 | 0.205 | 0.081 | 0.048 | 0.839 | 0.611 | 0.292 | 0.215 | 0.294 | 0.163 | 0.631 | 0.536
vgame | 0.067 | 0.044 | 0.038 | 0.037 | 0.024 | 0.027 | 0.13 | 0.044 | 0.019 | 0.061 | 0.043 | 0.03 | 0.127 | 0.04 | 0.07 | 0.013 | 0.758 | 0.323 | 0.215 | 0.114 | 0.267 | 0.106 | 0.349 | 0.098
voice | 0.014 | 0.025 | 0.005 | 0.022 | 0.003 | 0.006 | 0.066 | 0.024 | 0.003 | 0.025 | 0.048 | 0.006 | 0.065 | 0.014 | 0.01 | 0.121 | 0.462 | 0.153 | 0.113 | 0.032 | 0.183 | 0.009 | 0.052 | 0.006
wine | 0.316 | 0.049 | 0.025 | 0.027 | 0.016 | 0.014 | 0.164 | 0.048 | 0.022 | 0.074 | 0.041 | 0.056 | 0.161 | 0.047 | 0.058 | 0.211 | 0.827 | 0.523 | 0.262 | 0.248 | 0.513 | 0.089 | 0.349 | 0.174
yeast | 0.483 | 0.178 | 0.167 | 0.206 | 0.099 | 0.101 | 0.199 | 0.181 | 0.128 | 0.216 | 0.096 | 0.156 | 0.196 | 0.188 | 0.079 | 0.373 | 0.838 | 0.65 | 0.291 | 0.228 | 0.634 | 0.121 | 0.392 | 0.262
Mean | 0.205 | 0.068 | 0.075 | 0.091 | 0.055 | 0.056 | 0.114 | 0.068 | 0.043 | 0.08 | 0.065 | 0.096 | 0.111 | 0.059 | 0.038 | 0.075 | 0.504 | 0.299 | 0.177 | 0.159 | 0.3 | 0.077 | 0.296 | 0.172

(b) NKLD scores

Table 4: Main results for binary quantification. We show the averages for
absolute errors (AE) and normalized Kullback-Leibler divergence (NKLD) for all
scenarios per algorithm and dataset. For absolute errors, _MS_ shows the best
performance. For NKLD, _DyS_ achieves the best results in the plurality of
datasets.

We show the general performance results of all quantification algorithms over
all datasets in Figure 1 and Table 4. The letter-value plots in Figures 1(a))
and 1(c)) represent the respective distributions of absolute error (AE) and
normalized Kullback-Leibler divergence (NKLD) scores resulting from all
experiments. Colors in the plot indicate algorithm categories, i.e., count
adaptation-based algorithms are shown in blue, distribution matching methods
in orange, and adaptations of traditional classification algorithms are shown
in green. The plots in 1(b)) and 1(d)) depict the average performance ranks of
all algorithms per dataset along with the critical differences between average
ranks, which indicate whether the difference in average ranks is statistically
significant according to the Nemenyi post-hoc test [4]. Here, horizontal bars
show which average rankings do not differ to a degree that is statistically
significant. Tables 4(a)) and 4(b)) complement these plots by providing the
average absolute errors (AE) and normalized Kullback-Leibler divergences
(NKLD) for all scenarios per algorithm and dataset. Based on these averages,
the rankings for the plots 1(b)) and 1(d)) have been compiled. Further, for
each algorithm, a total average error score over all datasets is provided.

Overall, under both the NKLD and the AE metric, we observe substantial
differences between the algorithms. While there is no single individual best
algorithm for all cases, the results suggest that there is a group of
algorithms that perform particularly well compared to the rest. First and
foremost, the two threshold selection methods _TSMax_ and _MS_ , and the
Topsøe distance optimization from the _DyS_ -Framework share the top three
aggregated ranks with respect to both the AE and the NKLD. When additionally
considering the overall distribution of error scores, the _MS_ and _DyS_
methods even appear to have a slight edge over the _TSMax_ approach. Further,
with respect to both measures, the _TSX_ threshold selection policy as well as
_Friedman’s method (FM)_ do not fall behind these top methods to a degree that
is statistically significant with respect to the Nemenyi post-hoc test.

This general impression is confirmed by Table 4, where we see that the _MS_
algorithm takes the top rank on most datasets with respect to the AE score,
whereas for the NKLD score, the _DyS_ method is most dominant in these
rankings. Interestingly, even overall very good methods can have severe issues
for specific datasets. For example, the _MS_ method can in general be
considered as very good, but faces severe issues for the _nurse_ dataset. This
dataset only contains binary features, which causes the underlying SVM
classifier to extreme decision scores, which may lead to issues due to the
rounding of values. Contrarily, methods such as _FM_ or _ED_ appear to show a
relatively robust performance over all datasets, leading to the lowest total
average error scores with respect to the AE and NKLD, respectively. However,
these algorithms hardly ever show the best performance on a single specific
dataset. Considering the overall distribution of error scores, the _PAC/GPAC_
methods also appear to yield relatively robust performance over all datasets,
even though top ranks with respect to specific datasets are hardly ever
achieved.

When considering the performance of basic algorithms such as _(Probabilistic)
Classify and Count_ and _Adjusted Count_ , we observe that these baselines are
clearly outperformed by the top algorithms. By contrast, the _Probabilistic
Adjusted Count (PAC)_ algorithm shows similar performance to these top
algorithms according to the AE scores, but appear to be weaker in terms of
their ranks according to NKLD scores. Moreover, all algorithms that we have
categorized as classifiers for quantification and also the _CDE iterator_
consistently show the worst performances with respect to both measures.

(a) AE values under minor shift

(b) NKLD values under minor shift

(c) AE values under medium shift

(d) NKLD values under medium shift

(e) AE values under major shift

(f) NKLD values under major shift

Figure 2: Impact of distribution shift. We show the distribution of error
scores split by the amount of shift in the evaluation scenario. The left
column shows results according to the absolute error, the right one according
to NKLD scores. Colors indicate the category of the algorithm. Plots are
scaled logarithmically above the dotted vertical threshold, and linearly
below. _GPAC_ performs best under minor shifts, _MS_ under major shifts.

####  5.1.2 Influence of Distribution Shift

In the context of quantification, a shift of the distribution of the target
labels $Y$ between training and test set is assumed. It could be expected that
the size of distribution shift affects the difficulty of the quantification
task, as we assume that stronger shifts make accurate quantification more
challenging. For that reason, we now take a closer look at the impact of this
distribution shift to find out which methods are more or less sensitive to the
amount of distribution shift. In that context, we categorize all settings into
three scenarios, namely a minor shift, a medium shift, and a major shift in
these distributions. More precisely, we consider the shift to be

  * •

minor, if the distribution shift is lower than 0.4 in $L_{1}$ distance,

  * •

medium, if the distribution shift is bigger or equal to 0.4 and lower than 0.8
in $L_{1}$ distance,

  * •

major, if the distribution shift is bigger or equal to 0.8 in $L_{1}$
distance.

We depict the aggregated performance of the quantification algorithms under
these three kinds of shifts in Figure 2. Unsurprisingly, we can observe that
the performance of all quantification algorithms generally deteriorates with
an increasing shift in target distributions. In that regard, the effect
appears to be the strongest for classification-based approaches, in particular
for the _Quantification Forests_ and the _PCC_ method. The only exception to
this principle appears to be the nearest-neighbor-based _PWK_ quantifier,
which with respect to the NKLD error appears relatively robust towards
distribution shift, as can also be seen by the average rankings (cf.
Appendix). Contrarily, the algorithms which yielded the best overall
performances, that is, the _DyS_ framework and, in particular, the _MS_
method, also appear the most robust to larger shifts. Next to these two
algorithms, also the other threshold selection policies and _Friedman’s method
(FM)_ appear relatively robust to distribution shift. Strong performance under
minor shift is also achieved by the _PAC/GPAC_ algorithms, which, however,
appear to fall significantly behind the top two methods for increasing shifts.

For all other algorithms, the decrease in performance appears to be in between
the aforementioned robust algorithms and the classify and count-based
quantifiers. Aside from the performance of the _PWK_ method, the overall
rankings appear mostly unaffected from distribution shift. That implies that
even though the overall performance deteriorates, the same methods perform
well independent of the amount of shift.

(a) AE values for few training samples

(b) NKLD values for few training samples

Figure 3: Performance with small amounts of training data. Plots are scaled
logarithmically above the dotted vertical threshold, and linearly below.
Colors indicate the category of the algorithm. We observe that _MS_ and _DyS_
still perform best.

####  5.1.3 Influence of Training Set Size

Next, we consider the performance of quantification algorithms when relatively
few training samples are given. For that purpose, we restrict the experimental
data to only those cases in which the given data was split into 10% training
and 90% test samples. The overall distribution of error scores with respect to
the AE and NKLD scores can be found in Figure 3. We observe that in general,
the performance of all algorithms seems to be worse compared to the results
when not restricting ourselves to little training data, which is also to be
expected intuitively. However, again the methods which yielded the overall
best performances, such as _MS_ , _DyS_ , _TSMax_ , _TSX_ , _FM_ , and
_PAC/GPAC_ also appear the most robust towards this scenario. The average
performance rankings of all algorithms per dataset (see Figure 12 in the
Appendix) are mostly in line with the general setting. An exception to this is
that the _DyS_ framework falls somewhat behind the _MS_ method with few
training data samples.

(a) Distribution of absolute errors (AE)

(b) Average rankings with respect to the AE

(c) Distribution of NKLD scores

(d) Average rankings with respect to NKLD

Figure 4: Visual representation of the main results for _multiclass_
quantification. The top row shows results for the absolute error (AE), the
bottom row for normalized Kullback-Leibler divergence (NKLD) scores. On the
left, letter-value plots for the distribution of error score across all
scenarios per algorithm are shown, colors indicate the category of the
algorithm. Plots are scaled logarithmically above the dotted vertical
threshold, and linearly below. On the right, we plot the distributions of
rankings with a Nemenyi post-hoc test at 5% significance. Horizontal bars
indicate which average rankings do not differ to a degree that is
statistically significant. The critical difference (CD) was $7.0045$. Overall,
performance scores are much worse than in the binary setting. Best
performances are achieved by a group of algorithms including the _GPAC_ , _ED_
, _FM_ , _EM_ , and _readme_ methods.

###  5.2 Multiclass Quantification

Next, we present results for multiclass quantification, i.e., quantification
for labels with more than two values.

| AC | PAC | TSX | TS50 | TSMax | MS | GAC | GPAC | DyS | FMM | readme | HDx | HDy | FM | ED | EM | CC | PCC | PWK | QF
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
bike | 0.699 | 0.469 | 0.424 | 0.395 | 0.454 | 0.471 | 0.113 | 0.073 | 0.460 | 0.428 | 0.201 | 0.445 | 0.114 | 0.102 | 0.176 | 0.082 | 0.368 | 0.364 | 0.315 | 0.638
blog | 0.818 | 0.671 | 0.569 | 0.559 | 0.538 | 0.533 | 0.360 | 0.236 | 0.537 | 0.486 | 0.180 | 0.197 | 0.361 | 0.285 | 0.29 | 0.196 | 0.588 | 0.500 | 0.422 | 0.547
conc | 0.623 | 0.566 | 0.528 | 0.517 | 0.455 | 0.459 | 0.486 | 0.473 | 0.589 | 0.627 | 0.432 | 0.626 | 0.484 | 0.51 | 0.457 | 0.498 | 0.915 | 0.692 | 0.480 | 0.662
cond | 0.486 | 0.442 | 0.396 | 0.259 | 0.437 | 0.500 | 0.155 | 0.066 | 0.463 | 0.444 | 0.129 | 0.2 | 0.154 | 0.088 | 0.093 | 0.059 | 0.343 | 0.362 | 0.213 | 0.431
contra | 0.658 | 0.494 | 0.483 | 0.486 | 0.451 | 0.443 | 0.600 | 0.515 | 0.519 | 0.653 | 0.424 | 0.361 | 0.594 | 0.512 | 0.434 | 0.396 | 0.833 | 0.699 | 0.572 | 0.675
craft | 0.496 | 0.525 | 0.420 | 0.393 | 0.394 | 0.400 | 0.296 | 0.190 | 0.519 | 0.547 | 0.412 | 0.336 | 0.288 | 0.190 | 0.274 | 0.191 | 0.752 | 0.654 | 0.442 | 0.763
diam | 0.307 | 0.232 | 0.252 | 0.210 | 0.220 | 0.214 | 0.197 | 0.098 | 0.208 | 0.277 | 0.117 | 0.034 | 0.199 | 0.118 | 0.209 | 0.214 | 0.784 | 0.645 | 0.404 | 0.501
drugs | 0.240 | 0.179 | 0.222 | 0.227 | 0.230 | 0.210 | 0.256 | 0.199 | 0.194 | 0.396 | 0.338 | 0.45 | 0.255 | 0.181 | 0.238 | 0.218 | 0.465 | 0.482 | 0.407 | 0.600
ener | 0.515 | 0.354 | 0.316 | 0.277 | 0.308 | 0.307 | 0.273 | 0.115 | 0.351 | 0.412 | 0.331 | 0.259 | 0.264 | 0.129 | 0.169 | 0.131 | 0.879 | 0.699 | 0.439 | 0.925
fifa | 0.815 | 0.656 | 0.557 | 0.561 | 0.498 | 0.499 | 0.313 | 0.181 | 0.548 | 0.440 | 0.221 | 0.111 | 0.315 | 0.216 | 0.278 | 0.127 | 0.481 | 0.441 | 0.384 | 0.432
news | 0.889 | 0.581 | 0.562 | 0.562 | 0.538 | 0.539 | 0.498 | 0.335 | 0.522 | 0.455 | 0.446 | 0.401 | 0.500 | 0.376 | 0.245 | 0.221 | 0.827 | 0.614 | 0.471 | 0.917
nurse | 0.079 | 0.104 | 0.463 | 0.506 | 0.470 | 0.515 | 0.023 | 0.019 | 0.109 | 0.093 | 0.263 | 0.536 | 0.023 | 0.02 | 0.049 | 0.022 | 0.138 | 0.173 | 0.213 | 0.399
thrm | 1.059 | 0.621 | 0.655 | 0.605 | 0.541 | 0.534 | 0.780 | 0.629 | 0.689 | 0.736 | 0.471 | 0.581 | 0.778 | 0.663 | 0.470 | 0.494 | 1.042 | 0.769 | 0.511 | 0.827
turk | 0.569 | 0.348 | 0.420 | 0.438 | 0.392 | 0.393 | 0.525 | 0.342 | 0.422 | 0.607 | 0.489 | 0.368 | 0.524 | 0.392 | 0.356 | 0.277 | 0.976 | 0.727 | 0.622 | 0.834
vgame | 0.712 | 0.633 | 0.606 | 0.612 | 0.558 | 0.558 | 0.520 | 0.46 | 0.543 | 0.482 | 0.364 | 0.47 | 0.519 | 0.474 | 0.424 | 0.322 | 0.590 | 0.520 | 0.418 | 0.589
wine | 0.873 | 0.663 | 0.604 | 0.611 | 0.525 | 0.532 | 0.656 | 0.575 | 0.656 | 0.537 | 0.428 | 0.564 | 0.653 | 0.605 | 0.44 | 0.757 | 0.965 | 0.636 | 0.496 | 0.613
yeast | 0.705 | 0.544 | 0.492 | 0.475 | 0.447 | 0.457 | 0.567 | 0.408 | 0.435 | 0.448 | 0.474 | 0.395 | 0.563 | 0.413 | 0.289 | 0.613 | 0.878 | 0.612 | 0.295 | 0.526
Mean | 0.620 | 0.476 | 0.469 | 0.453 | 0.439 | 0.445 | 0.389 | 0.289 | 0.457 | 0.474 | 0.336 | 0.373 | 0.387 | 0.31 | 0.288 | 0.284 | 0.696 | 0.564 | 0.418 | 0.640

(a) Average AE values of each algorithm per dataset

| AC | PAC | TSX | TS50 | TSMax | MS | GAC | GPAC | DyS | FMM | readme | HDx | HDy | FM | ED | EM | CC | PCC | PWK | QF
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
bike | 0.667 | 0.376 | 0.296 | 0.342 | 0.300 | 0.310 | 0.044 | 0.016 | 0.274 | 0.249 | 0.05 | 0.433 | 0.042 | 0.032 | 0.045 | 0.016 | 0.116 | 0.105 | 0.092 | 0.244
blog | 0.738 | 0.819 | 0.643 | 0.648 | 0.602 | 0.608 | 0.400 | 0.2 | 0.493 | 0.411 | 0.040 | 0.095 | 0.369 | 0.241 | 0.113 | 0.044 | 0.315 | 0.155 | 0.135 | 0.206
conc | 0.516 | 0.423 | 0.356 | 0.350 | 0.300 | 0.318 | 0.307 | 0.462 | 0.300 | 0.292 | 0.126 | 0.432 | 0.292 | 0.451 | 0.21 | 0.46 | 0.639 | 0.276 | 0.137 | 0.254
cond | 0.662 | 0.520 | 0.373 | 0.257 | 0.388 | 0.436 | 0.088 | 0.011 | 0.308 | 0.292 | 0.022 | 0.089 | 0.084 | 0.027 | 0.017 | 0.004 | 0.166 | 0.098 | 0.044 | 0.130
contra | 0.506 | 0.372 | 0.346 | 0.382 | 0.273 | 0.282 | 0.446 | 0.466 | 0.294 | 0.299 | 0.131 | 0.184 | 0.428 | 0.442 | 0.213 | 0.237 | 0.464 | 0.280 | 0.179 | 0.258
craft | 0.384 | 0.450 | 0.376 | 0.327 | 0.319 | 0.332 | 0.170 | 0.148 | 0.231 | 0.196 | 0.113 | 0.121 | 0.158 | 0.115 | 0.080 | 0.159 | 0.397 | 0.242 | 0.113 | 0.403
diam | 0.324 | 0.157 | 0.153 | 0.131 | 0.129 | 0.127 | 0.101 | 0.061 | 0.109 | 0.158 | 0.016 | 0.002 | 0.087 | 0.091 | 0.09 | 0.169 | 0.530 | 0.254 | 0.096 | 0.225
drugs | 0.152 | 0.096 | 0.150 | 0.123 | 0.104 | 0.119 | 0.178 | 0.148 | 0.063 | 0.136 | 0.085 | 0.285 | 0.170 | 0.124 | 0.053 | 0.049 | 0.151 | 0.147 | 0.112 | 0.204
ener | 0.552 | 0.376 | 0.316 | 0.292 | 0.278 | 0.280 | 0.135 | 0.083 | 0.258 | 0.208 | 0.086 | 0.116 | 0.121 | 0.082 | 0.050 | 0.086 | 0.491 | 0.270 | 0.12 | 0.527
fifa | 0.758 | 0.787 | 0.589 | 0.549 | 0.546 | 0.541 | 0.314 | 0.114 | 0.474 | 0.315 | 0.049 | 0.026 | 0.295 | 0.150 | 0.098 | 0.029 | 0.254 | 0.126 | 0.115 | 0.129
news | 0.834 | 0.454 | 0.419 | 0.424 | 0.377 | 0.383 | 0.537 | 0.316 | 0.309 | 0.266 | 0.143 | 0.205 | 0.486 | 0.398 | 0.075 | 0.059 | 0.524 | 0.227 | 0.16 | 0.608
nurse | 0.055 | 0.062 | 0.142 | 0.158 | 0.144 | 0.211 | 0.010 | 0.005 | 0.020 | 0.009 | 0.055 | 0.556 | 0.008 | 0.007 | 0.005 | 0.001 | 0.025 | 0.033 | 0.049 | 0.115
thrm | 0.842 | 0.550 | 0.625 | 0.593 | 0.430 | 0.442 | 0.602 | 0.643 | 0.534 | 0.472 | 0.145 | 0.495 | 0.582 | 0.716 | 0.245 | 0.439 | 0.692 | 0.340 | 0.151 | 0.382
turk | 0.495 | 0.270 | 0.314 | 0.334 | 0.225 | 0.226 | 0.408 | 0.343 | 0.230 | 0.281 | 0.176 | 0.176 | 0.384 | 0.368 | 0.175 | 0.105 | 0.585 | 0.296 | 0.216 | 0.435
vgame | 0.682 | 0.578 | 0.501 | 0.466 | 0.457 | 0.469 | 0.581 | 0.519 | 0.382 | 0.241 | 0.102 | 0.367 | 0.560 | 0.506 | 0.134 | 0.132 | 0.238 | 0.170 | 0.134 | 0.205
wine | 0.693 | 0.627 | 0.574 | 0.563 | 0.444 | 0.468 | 0.433 | 0.617 | 0.459 | 0.285 | 0.129 | 0.373 | 0.424 | 0.615 | 0.276 | 0.778 | 0.713 | 0.240 | 0.157 | 0.221
yeast | 0.674 | 0.587 | 0.549 | 0.496 | 0.491 | 0.510 | 0.357 | 0.43 | 0.341 | 0.226 | 0.143 | 0.264 | 0.356 | 0.398 | 0.115 | 0.699 | 0.584 | 0.224 | 0.075 | 0.173
Mean | 0.561 | 0.441 | 0.395 | 0.378 | 0.342 | 0.356 | 0.301 | 0.269 | 0.299 | 0.255 | 0.095 | 0.248 | 0.285 | 0.280 | 0.117 | 0.204 | 0.405 | 0.205 | 0.123 | 0.278

(b) Average NKLD scores for each algorithm per dataset

Table 5: Main results for multiclass quantification. We show the averaged
error scores for all scenarios per algorithm and dataset. Overall,
distribution matching methods that naturally generalize to the multiclass
setting perform better than one-vs.-rest or classify and count-based
approaches.

####  5.2.1 Overall Results

Tables 5(a)) and 5(b)) as well as Figure 4 present the main results for
multiclass quantification. Compared to the binary case, we obtain
substantially different results. First of all, the overall prediction
performance is much worse, as both the AE scores and the NKLD scores appear
multiple times higher on average. For instance, AE values below 0.1 and NKLD
values below 0.01 are widespread in the binary case, whereas in the multiclass
case, such scores are only rarely achieved. Instead, the average AE values of
each algorithm over all experiments are mostly around the interval [0.3,0.4],
which is three to four times higher than the average AE values of the best
algorithms in the binary case. The second main difference regards the
algorithms which appear to work best: Algorithms such as the _DyS_ -Framework,
the _Median Sweep (MS)_ , and the other threshold selection policies, which
have worked very well for binary quantification, appear comparatively weak in
their performance. By contrast, the best performances seem to be achieved by
distribution matching algorithms which also naturally extend to the multiclass
setting, namely the _GPAC_ , _ED_ , _FM_ , _EM_ , _readme_ , and _HDx_
methods. In that context, the _GPAC_ , _ED_ , _EM_ , and _FM_ methods show the
best performances with respect to the absolute error, whereas the _readme_ ,
_ED_ , _EM_ , and _FM_ , but also the classification-based _PWK_ method take
the best average rankings with respect to the NKLD. This relatively wide range
of strongly performing algorithms is also underlined in Tables 5(a)) and
5(b)), where we observe a relatively high fluctuation with regard to the best
performance on specific datasets. In addition, from the overall distributions
of errors in Figures 4(a)) and 4(c)) it becomes apparent that these algorithms
also have strong differences in the variance of their performances. In
particular, the _GPAC_ method appears to have a much higher variance in its
error scores compared to the rest, whereas the _ED_ and _readme_ methods
display the lowest variance in their performances. However, the given results
also have one big similarity with the results from the binary setting, i.e.,
all algorithms which are based on the classify and count principle display
subpar performances even when optimizing quantification-based loss functions.

(a) AE values under minor shift

(b) NKLD values under minor shift

(c) AE values under major shift

(d) NKLD values under major shift

Figure 5: Impact of distribution shift in the multiclass setting. We show the
distribution of error scores split by the amount of shift in the evaluation
scenario. The left column shows results according to the absolute error, the
right one according to NKLD scores. Plots are scaled logarithmically above the
dotted vertical threshold, and linearly below. Colors indicate the category of
the algorithm. Performances generally deteriorate under major shift. Best
performances under major shift are achieved by algorithms that also do work
well in general. The _GPAC_ and _FM_ methods appear most robust towards major
shifts.

####  5.2.2 Impact of Distribution Shift

As in the binary case, we also investigate the effect that the shift of the
distribution of the target labels $Y$ between training and test sets has on
the resulting quantification performance. Since we have less experimental data
than in the binary case, this time we only distinguish between a minor shift
and a major shift. We consider the shift to be

  * •

minor, if the distribution shift is lower than 0.5 in $L_{1}$ distance,

  * •

major, if the distribution shift is bigger or equal to 0.5 in $L_{1}$
distance.

The results of multiclass quantification under these scenarios are shown in
Figure 5. Similar to the binary case, we observe that the algorithms which
appeared to work best in general, also appear the most robust with respect to
high distribution shifts. In particular, the GPAC method appears almost
unaffected by a high shift in its average performance, consistently achieving
higher performance ranks under higher shifts, although significant variance is
present in its performance. By contrast, all methods which apply the classify
and count principle are again the most susceptible to higher error rates when
applied in scenarios with higher shifts between training and test
distribution.

####  5.2.3 Influence of Training Set Size

(a) AE values for few training samples

(b) NKLD values for few training samples

Figure 6: Performance under small amount of training data in multiclass
setting. Plots are scaled logarithmically above the dotted vertical threshold,
and linearly below. Colors indicate the category of the algorithm. Overall
results are very similar to the general setting, only with slightly worse
performance scores.

Finally, we consider the performance of the given algorithms when the given
data was split into 10% training and 90% test samples. As before, this serves
to investigate the impact of having a relatively small set of training data.
The distributions of error scores with respect to the AE and NKLD metrics can
be found in Figure 6. Compared to the general distribution of error scores
from all experiments, the performance when only small training sets are given
does deteriorate. In particular, we observe that the _GPAC_ is much less
competitive than in the general scenario, in particular with respect to the
NKLD score. Conversely, the _EM_ and _ED_ algorithms, and, with respect to the
NKLD score, also the _readme_ method appear to be most robust towards this
setting. This implies that those algorithms could be recommended if only small
training sets are available.

(a) Distribution of absolute error (AE) scores

(b) Distribution of NKLD scores

Figure 7: Results of our experiments with quantifiers that apply tuned
classifiers in the binary setting. Plots are scaled logarithmically above the
dotted vertical threshold, and linearly below. Colors indicate the category of
the algorithm. Algorithms based on untuned logistic regression classifiers are
denoted as before (no suffix), alternative tuned base classifiers are marked
with respective suffixes: logistic regressors (LR), support vector machines
(SV), random forests (RF) and AdaBoost (AB). In addition, we present results
of the _RBF-K_ and _RBF-Q_ methods, which are variants of the _SVM-K_ and
_SMV-Q_ that use an RBF-kernel instead of a linear one. We observe that except
for the _CC_ , _PCC_ , _GAC_ and _CDE_ methods, tuning the classifiers does
not seem to have a significant positive effect on the outcome.

###  5.3 Impact of Alternative Classifiers and Tuning

Finally, we present the results from our experiments with quantifiers that
applied tuned base classifiers. We begin with the results on binary data,
before closing with the results on multiclass data.

####  5.3.1 Experiments on Binary Data

In Figure 7, we show the scores of all quantifiers using different tuned base
classifiers aggregated over all considered datasets, see Section 4.3.2. As a
baseline, we also include the results from the quantifiers that are based on
the default logistic regressor. These results yield a few key findings. First,
for most algorithms, tuning the base classifier does not seem to have a
significant positive effect. Instead, for the best performing algorithms _MS_
, _DyS_ , _TSX_ , _FM_ , and _TSMax_ , the performance even appears to
deteriorate. The few exceptions where tuning classifiers appear to benefit the
predictions include the _CC_ , _PCC_ , _CDE_ , and _GAC_. While the first two
directly apply the classify and count principle, where it is to be expected
that more accurate classification will tend to yield more accurate
quantification, the results for the _CDE_ and _GAC_ methods pose as outliers.
Regarding the _SVM-K_ and _SVM-Q_ methods, we observe that applying the
alternative RBF-kernel appears to have a slight positive effect, but these
_RBF-K_ and _RBF-Q_ variants still show inferior performance compared to most
of the other quantifiers, while at the same time coming at very high
computational costs. Overall, the given results also appear consistent over
both the _AE_ and the _NKLD_ metrics.

(a) Distribution of absolute error (AE) scores

(b) Distribution of NKLD scores

Figure 8: Results of our experiments with quantifiers that apply tuned
classifiers in the multiclass setting. Plots are scaled logarithmically above
the dotted vertical threshold, and linearly below. Colors indicate the
category of the algorithm. Algorithms based on untuned logistic regression
classifiers are denoted as before (no suffix), alternative tuned base
classifiers are marked with respective suffixes: logistic regressors (LR),
support vector machines (SV), random forests (RF) and AdaBoost (AB). In
contrast to the binary case, we observe mostly positive effects from tuning
the base classifiers, and the tuned variant of the _EM_ method appears to
outperform the other quantifiers.

####  5.3.2 Experiments on Multiclass Data

The results of our experiments on quantification with tuned base classifiers
in the multiclass setting can be found in Figure 8. Other than in the binary
setting, we observe that tuning classifiers appears to have a significant
positive effect for most quantifiers. This effect is particularly pronounced
for methods which utilize probability scores from their underlying
classifiers, namely the _EM_ , _PAC_ , _GPAC_ , _FM_ , and _CDE_ methods, but
also very notable for the classify and count-based _CC_ and _PCC_ methods. In
that context, the _EM_ method with tuned logistic regressors as its base
classifiers even appears to significantly outperform the other quantifiers in
this experimental setting. Again, these trends are also mostly consistent over
both error metrics that we considered, although with regard to the _NKLD_ ,
the benefits of tuning base classifiers even appear a little more distinct.

##  6 Discussion

Next, we discuss the main results of our study and potential limitations of
our study.

###  6.1 Discussion of Results

Our experiments yield substantially different results for the binary case
compared to the multiclass case, both in terms of overall quality of
performance, and in terms of which algorithms perform best. In the binary
case, we identified a few algorithms which appear to work particularly well
with respect to both absolute error and NKLD, namely the _MS_ , _DyS_ , and
_TSMax_ methods. These methods stand out both in terms of their ranks and
their overall error distribution (though _DyS_ and _MS_ have a slight edge
over _TSMax_ in that regard). Next to these three algorithms, other methods
show similarly strong performances, at least with respect to one of the two
measures that were considered. In that regard, _Friedman’s method_ has shown
very strong performances with respect to the AE score, while the _ED_ method
appears to work particularly well with respect to the NKLD measure.

The strong performance of the _MS_ and _TSMax_ methods indicate that the
simple idea behind the _Adjusted Count_ approach, even when using a rather
unsophisticated baseline classifier, can still yield very decent results as
long as numerical stability, i.e., a big denominator in Equation 1, is
ensured. In that regard, the _MS_ method also benefits from the policy that
all thresholds for which the denominator is below 0.25 are excluded. A similar
argument can be made for the superiority of the _DyS_ framework compared to
other distribution matching methods that utilize predictions from classifiers.
More precisely, the approach of binning confidence scores into more than just
two classes, which ultimately adds more equations to the system in Equation 2,
also appears to yield more robust results. However, using too many bins as in
the _FMM_ method might lead to deteriorating performance.

Interestingly, all quantification methods that do not apply a classifier do
not appear to be as competitive compared to the best methods. Since this also
holds for fully discrete datasets such as _dota_ this is likely not due to
feature binning. While the _readme_ method might yield stronger improvement
through an added parameter optimization step, the _ED_ and _HDx_ methods do
not have any parameters to optimize. On the other hand, for most quantifiers
that apply base classifiers to form their predictions, we observe that tuning
parameters of this base classifiers hardly seems to have a positive effect on
the quality of the resulting predictions. For the quantifiers that utilize
misclassification rates, an explanation might be that having somewhat higher
misclassification rates may actually yield more numerical stability in the
predictions. Next to the _CDE_ method, only the classify and count-based
methods _CC_ and _PCC_ posed on exception to this pattern, but it has also to
be expected that for this kind of algorithm, optimizing the base classifiers
would beneficial. However, these methods displayed the general trend of
yielding worse performance overall than the other classes of quantifiers, even
after hyperparameter tuning. Similarly, classifiers that optimize
quantification-oriented loss-functions also tended to show worse performance
than the majority of other quantifiers. This is overall another strong
indicator that pure classification without adjustments for potential
distributions shifts does not perform well for quantification. The reason for
that is that under a shift in the target distribution, predictions are
strongly biased towards the training distribution, as exemplified in our
experiments. This practical outcome is also clearly in line with Forman’s
Theorem [12], which states that when a distribution shift is given, a bias in
the _CC_ estimates towards the training distribution is to be expected. This
finding bears some contrast to a recent discussion of this kind of approach by
Moreo and Sebastiani [23], who have reassessed the performance of the
_Classify and Count_ approach, and found that when doing careful optimization
of hyperparameters, such quantification-oriented classification approaches
would deliver near-state-of-the-art performance, although still being
inferior. Our experimental outcomes suggest that this type of approach should
be used only carefully for quantification, as a vulnerability towards
distribution shifts in theory as well as in experimental results can be
observed clearly. Further, aside from the base classifiers, we did not
optimize hyperparameters for all other algorithms in our experiments either,
and we would expect positive effects in that regard if, for instance, the
number of bins in the _DyS_ framework was optimized.

Considering the multiclass case, results are qualitatively different. Most
notably, error scores were significantly higher than in binary quantification.
Another key difference is that methods such as _DyS_ , _MS_ or _TSMax_ , which
have excelled in binary quantification, only showed mediocre performance in
the multiclass case. By contrast, distribution matching methods that naturally
extend to the multiclass setting appeared to work best, although there was no
clear best-performing algorithm. This appears to indicate that generalizing
binary quantification results to the multiclass case via a one-vs.-rest
approach is not an optimal strategy for multiclass quantification. A reason
for this might be buried in the normalization of the results from the one-
vs.-rest quantifications, in which small errors in the predictions of
individual class prevalences might get amplified. Further, we found that,
unlike in the binary case, fine-tuning hyperparameters of the underlying base
classifiers appears to have a significant positive effect on the quality of
the resulting predictions for many of the given quantifiers. In particular for
the _EM_ method, a strong effect could be observed, and with tuned base
classifiers this method appeared to clearly outperform the other quantifiers.
However, compared to the other experimental analyses, we had a relatively
small number of only nine multiclass datasets on which we performed this
hyperparameter tuning, so we urge being careful about the generalizability of
these findings.

Ultimately, our results indicate that quantification in the multiclass setting
is a much harder task than in the binary setting. This points to multiclass
quantification as a high-potential problem for future research efforts.

###  6.2 Limitations

This paper presents an extensive empirical comparison of state-of-the-art
quantification methods. As such, our results are necessarily affected by some
experimental design choices.

First of all, in our experiments, we relied on default parameters of the
individual algorithms and did not perform extensive hyperparameter
optimization for the quantification algorithms on each dataset. While on the
one hand this is due to computational considerations—we have performed more
than 295,000 experiments with 10 sampling iterations each, making extra
hyperparameter optimization steps infeasible—this also reflects the
performance an “uninitiated” data analyst would achieve using those methods
off-the-shelf. Obviously, some methods such as _DyS_ and _readme_ that feature
more hyperparameters could profit more from optimizations. However, we
evaluated the effects of tuning the underlying classifiers, see Section 5.3.
These experiments had to be limited to small and medium-sized dataset due to
computational requirements.

Regarding the choice of datasets, training set sizes and distribution shifts,
we aimed to cover a wide range of possible application scenarios. However,
unintended biases in the selection of the datasets or scenarios might lead to
biases in the aggregated results. Yet, we would like to note that this study
does not include algorithms from the authors or collaborators, such that the
authors do not have stakes in any particular outcome. In that regard, the
authors stress that an on average subpar performance of a specific
quantification method does not imply that this method might not excel in other
particular scenarios.

##  7 Conclusions

In this study, we have conducted a thorough experimental comparison of 24
quantification methods over 40 datasets, involving more than 5 million
algorithm runs. In our experiments, we have both considered the binary and the
multiclass case in quantification, and have also specifically considered the
impact of shifting target label distributions between training and test data,
as well as the impact of having relatively small training sets. In the binary
case, we have identified two methods which generally appear to work best,
namely, the distribution matching approach from the _DyS_ framework [20] as
well as the _Median Sweep_ method [12]. Regarding the multiclass case, a group
of distribution matching methods, which naturally extend to multiclass
quantification, appeared to be generally superior to the other evaluated
algorithms. We provide further evidence that the multiclass setting in general
is much harder to solve for established quantification methods, as the
obtained error scores consistently were multiple times higher than in the
binary case. However, in contrast to the binary setting, tuning the base
classifiers appears to have a distinctive positive effect on the resulting
quantifications. This indicates a certain potential for future research on
this specific setting. In addition, our results demonstrate that algorithms
that are based on the classify and count principle, even when the underlying
classifier is optimized for quantification, on average exhibit worse
performance compared to other specialized solutions. Overall, we hope our
findings provide guidance to practitioners in choosing the right
quantification algorithm for a given application and aid researchers in
identifying promising directions for future research.

## References

  * Barranquero et al. [2013] Jose Barranquero, Pablo González, Jorge Díez, and Juan José del Coz.  On the study of nearest neighbor algorithms for prevalence estimation in binary problems.  _Pattern Recognition_ , 46(2):472–482, 2013.
  * Barranquero et al. [2015] Jose Barranquero, Jorge Díez, and Juan José del Coz.  Quantification-oriented learning based on reliable classifiers.  _Pattern Recognition_ , 48(2):591–604, 2015.
  * Bella et al. [2010] Antonio Bella, Cesar Ferri, Jose Hernandez-Orallo, and Maria Jose Ramirez-Quintana.  Quantification via probability estimators.  In _2010 IEEE International Conference on Data Mining_ , pages 737–742, Sydney, Australia, 2010.
  * Demšar [2006] Janez Demšar.  Statistical comparisons of classifiers over multiple data sets.  _Journal of Machine Learning Research_ , 7:1–30, 2006.
  * Dempster et al. [1977] A. P. Dempster, N. M. Laird, and D. B. Rubin.  Maximum likelihood from incomplete data via the EM algorithm.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 39(1):1–22, 1977.
  * Deza and Deza [2009] Michel Marie Deza and Elena Deza.  _Encyclopedia of Distances_.  Springer, Berlin, Heidelberg, Germany, 2009.
  * Diamond and Boyd [2016] Steven Diamond and Stephen Boyd.  CVXPY: A Python-embedded modeling language for convex optimization.  _Journal of Machine Learning Research_ , 17(83):1–5, 2016.
  * Esuli et al. [2010] Andrea Esuli, Fabrizio Sebastiani, and Ahmed Abasi.  AI and opinion mining, part 2.  _IEEE Intelligent Systems_ , 25(4):72–79, 2010\.
  * Esuli et al. [2018] Andrea Esuli, Alejandro Moreo Fernández, and Fabrizio Sebastiani.  A recurrent neural network for sentiment quantification.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , pages 1775–1778, Torino, Italy, 2018.
  * Firat [2016] Aykut Firat.  Unified framework for quantification.  _arXiv preprint arXiv:1606.00868_ , 2016.
  * Forman [2005] George Forman.  Counting positives accurately despite inaccurate classification.  In _Proceedings of the 16th European Conference on Machine Learning_ , volume 3720, pages 564–575, Porto, Portugal, 2005.
  * Forman [2008] George Forman.  Quantifying counts and costs via classification.  _Data Mining and Knowledge Discovery_ , 17(2):164–206, 2008.
  * Friedman [2014] Jerome H. Friedman.  Class counts in future unlabeled samples, 2014.  Presentation at MIT CSAIL Big Data Event.
  * Friedman [1940] Milton Friedman.  A comparison of alternative tests of significance for the problem of m rankings.  _The Annals of Mathematical Statistics_ , 11(1):86–92, 1940.
  * González et al. [2017] Pablo González, Alberto Castaño, Nitesh V. Chawla, and Juan José del Coz.  A review on quantification learning.  _ACM Computing Surveys_ , 50(5):1–40, 2017.
  * González-Castro et al. [2013] Víctor González-Castro, Rocío Alaiz-Rodríguez, and Enrique Alegre.  Class distribution estimation based on the Hellinger distance.  _Information Sciences_ , 218:146–164, 2013.
  * Hopkins and King [2010] Daniel J. Hopkins and Gary King.  A method of automated nonparametric content analysis for social science.  _American Journal of Political Science_ , 54(1):229–247, 2010.
  * Joachims [2005] Thorsten Joachims.  A support vector method for multivariate performance measures.  In _Proceedings of the 22nd International Conference on Machine Learning_ , pages 377–384, Bonn, Germany, 2005.
  * Kawakubo et al. [2016] Hideko Kawakubo, Marthinus Christoffel du Plessis, and Masashi Sugiyama.  Computationally efficient class-prior estimation under class balance change using energy distance.  _IEICE Transactions on Information and Systems_ , 99(1):176–186, 2016.
  * Maletzke et al. [2019] André Maletzke, Denis dos Reis, Everton Cherman, and Gustavo Batista.  DyS: A framework for mixture models in quantification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 4552–4560, Honolulu, Hawaii, 2019.
  * Maletzke et al. [2020] André Maletzke, Waqar Hassan, Denis dos Reis, and Gustavo Batista.  The importance of the test set size in quantification assessment.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , pages 2640–2646, Yokohama, Japan, 2020.
  * Milli et al. [2013] Letizia Milli, Anna Monreale, Giulio Rossetti, Fosca Giannotti, Dino Pedreschi, and Fabrizio Sebastiani.  Quantification trees.  In _2013 IEEE 13th International Conference on Data Mining_ , pages 528–536, Dallas, Texas, 2013.
  * Moreo and Sebastiani [2021] Alejandro Moreo and Fabrizio Sebastiani.  Re-assessing the "classify and count" quantification method.  _arXiv preprint arXiv:2011.02552_ , 2021.
  * Nemenyi [1963] Peter B. Nemenyi.  _Distribution-free multiple comparisons_.  PhD thesis, Princeton University, 1963.
  * Saerens et al. [2002] Marco Saerens, Patrice Latinne, and Christine Decaestecker.  Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure.  _Neural Computation_ , 14(1):21–41, 2002.
  * Sebastiani [2020] Fabrizio Sebastiani.  Evaluation measures for quantification: An axiomatic approach.  _Information Retrieval Journal_ , 23(3):255–288, 2020.
  * Tasche [2016] Dirk Tasche.  Does quantification without adjustments work?  _arXiv preprint arXiv:1602.08780_ , 2016.
  * Tasche [2017] Dirk Tasche.  Fisher consistency for prior probability shift.  _Journal of Machine Learning Research_ , 18:3338–3369, 2017\.
  * Xue and Weiss [2009] Jack Chongjie Xue and Gary M. Weiss.  Quantification and semi-supervised classification methods for handling changes in class distribution.  In _Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 897–906, Paris, France, 2009.

## Appendix

In the appendix, we present additional analytical results regarding the
ranking of algorithms. We compute the average ranks of all algorithms
aggregated per data set filtered by several conditions. Then, we apply a
Nemenyi post-hoc test at 5% significance. In the individual plots, we then
show for each algorithm the average performance rank over all algorithms.
Horizontal bars indicate which algorithm’s average rankings do not differ to a
degree that is statistically significant, cf. [4].

Complementing the results from Section 5.1, Figure 9 displays the
distributions of rankings under varying shifts between training and target
data. Figure 10 displays the rankings of quantification methods when only few
training samples are given. In both figures, we observe that the rankings are
very similar to the general cases. However, we observe a stronger distinction
in the average ranks for high shifts and few training data.

Figure 11 and Figure 12 complement the results from Section 5.2 by presenting
additional rankings in the multiclass settings. Figure 11 displays the
distributions of rankings of quantification algorithms under minor and major
shifts between training and target data. Figure 12 displays the rankings of
multiclass quantifiers when only settings with few training samples are
considered.

Finally, we present additional results from our experiments on quantification
using tuned classifiers. Table 6 displays the average error scores of all
considered algorithms per dataset in the binary setting, whereas Table 7 shows
the corresponding results in the multiclass setting.

(a) MAE values under low shift

(b) NKLD values under low shift

(c) MAE values under medium shift

(d) NKLD values under medium shift

(e) MAE values under high shift

(f) NKLD values under high shift

Figure 9: Impact of distribution shifts on algorithm rankings for binary
labels. We plot the distributions of rankings separated by minor, medium, and
major shifts with a Nemenyi post-hoc test at 5% significance.

(a) MAE values for few training data

(b) NKLD values for few training data

Figure 10: Impact of low training data amount on algorithm rankings for binary
target labels. We plot the distributions of rankings obtained by 10/90
training/tests splits with a Nemenyi post-hoc test at 5% significance.

(a) MAE values under low shift

(b) NKLD values under low shift

(c) MAE values under high shift

(d) NKLD values under high shift

Figure 11: Impact of distribution shifts on algorithm rankings for multiclass
target labels. We plot the distributions of rankings separated by minor,
medium, and major shifts with a Nemenyi post-hoc test at 5% significance.

(a) MAE values for few training data

(b) NKLD values for few training data

Figure 12: Impact of low training data amount on algorithm rankings for
multiclass target labels. We plot the distributions of rankings obtained by
10/90 training/tests splits with a Nemenyi post-hoc test at 5% significance.

| AC | AC-LR | AC-RF | AC-SV | AC-AB | PAC | PAC-LR | TSX | TSX-LR | TSX-SV | TS50 | TS50-LR | TS50-SV | TSMax | TSMax-LR | TSMax-SV | MS | MS-LR | MS-SV | GAC | GAC-LR | GAC-RF | GAC-SV | GAC-AB | GPAC | GPAC-LR | DyS | DyS-LR | DyS-SV | FMM | FMM-LR | FMM-SV | HDy | HDy-LR | HDy-RF | HDy-SV | HDy-AB | FM | FM-LR | EM | EM-LR | CDE | CDE-LR | CC | CC-LR | CC-RF | CC-SV | CC-AB | PCC | PCC-LR | SVM-K | SVM-Q | RBF-K | RBF-Q
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ads | 0.134 | 0.135 | 0.156 | 0.139 | 0.119 | 0.103 | 0.108 | 0.09 | 0.110 | 0.084 | 0.122 | 0.139 | 0.121 | 0.086 | 0.110 | 0.084 | 0.072 | 0.097 | 0.074 | 0.138 | 0.106 | 0.122 | 0.112 | 0.116 | 0.102 | 0.108 | 0.083 | 0.099 | 0.095 | 0.211 | 0.213 | 0.220 | 0.138 | 0.106 | 0.121 | 0.112 | 0.115 | 0.082 | 0.112 | 0.087 | 0.143 | 0.199 | 0.183 | 0.352 | 0.275 | 0.322 | 0.283 | 0.218 | 0.352 | 0.301 | 0.255 | 0.341 | 0.416 | 0.479
alco | 0.410 | 0.547 | 0.506 | 0.490 | 0.581 | 0.328 | 0.503 | 0.367 | 0.541 | 0.427 | 0.400 | 0.546 | 0.432 | 0.318 | 0.544 | 0.416 | 0.291 | 0.524 | 0.397 | 0.451 | 0.425 | 0.407 | 0.415 | 0.477 | 0.337 | 0.503 | 0.383 | 0.447 | 0.42 | 0.584 | 0.568 | 0.539 | 0.451 | 0.425 | 0.410 | 0.415 | 0.477 | 0.342 | 0.563 | 0.296 | 0.446 | 0.695 | 0.714 | 0.693 | 0.730 | 0.741 | 0.695 | 0.746 | 0.625 | 0.645 | 0.495 | 0.608 | 0.692 | 0.658
bc-cat | 0.136 | 0.111 | 0.125 | 0.114 | 0.128 | 0.112 | 0.083 | 0.088 | 0.115 | 0.08 | 0.158 | 0.165 | 0.129 | 0.078 | 0.108 | 0.079 | 0.064 | 0.104 | 0.071 | 0.193 | 0.103 | 0.124 | 0.107 | 0.127 | 0.112 | 0.083 | 0.105 | 0.120 | 0.103 | 0.246 | 0.260 | 0.289 | 0.193 | 0.103 | 0.122 | 0.107 | 0.128 | 0.062 | 0.079 | 0.207 | 0.080 | 0.315 | 0.098 | 0.380 | 0.171 | 0.207 | 0.166 | 0.174 | 0.390 | 0.210 | 0.304 | 0.753 | 0.146 | 0.202
bc-cont | 0.076 | 0.089 | 0.085 | 0.089 | 0.102 | 0.072 | 0.066 | 0.059 | 0.068 | 0.061 | 0.119 | 0.135 | 0.141 | 0.057 | 0.065 | 0.061 | 0.047 | 0.067 | 0.057 | 0.117 | 0.079 | 0.090 | 0.080 | 0.108 | 0.072 | 0.066 | 0.062 | 0.075 | 0.099 | 0.198 | 0.209 | 0.217 | 0.117 | 0.079 | 0.090 | 0.080 | 0.106 | 0.039 | 0.057 | 0.125 | 0.102 | 0.123 | 0.098 | 0.172 | 0.116 | 0.116 | 0.107 | 0.140 | 0.245 | 0.217 | 0.167 | 0.838 | 0.080 | 0.066
cappl | 0.233 | 0.314 | 0.272 | 0.297 | 0.240 | 0.155 | 0.250 | 0.15 | 0.314 | 0.183 | 0.215 | 0.351 | 0.235 | 0.15 | 0.315 | 0.182 | 0.129 | 0.297 | 0.162 | 0.289 | 0.266 | 0.237 | 0.258 | 0.229 | 0.156 | 0.250 | 0.181 | 0.243 | 0.26 | 0.394 | 0.394 | 0.394 | 0.289 | 0.266 | 0.233 | 0.258 | 0.228 | 0.115 | 0.301 | 0.087 | 0.256 | 0.302 | 0.345 | 0.473 | 0.461 | 0.422 | 0.431 | 0.383 | 0.465 | 0.484 | 0.330 | 0.322 | 0.454 | 0.496
cars | 0.087 | 0.092 | 0.106 | 0.086 | 0.087 | 0.080 | 0.075 | 0.059 | 0.081 | 0.061 | 0.101 | 0.117 | 0.107 | 0.059 | 0.079 | 0.061 | 0.050 | 0.071 | 0.055 | 0.113 | 0.085 | 0.093 | 0.079 | 0.077 | 0.080 | 0.075 | 0.061 | 0.070 | 0.071 | 0.201 | 0.186 | 0.171 | 0.113 | 0.085 | 0.093 | 0.079 | 0.077 | 0.059 | 0.073 | 0.087 | 0.073 | 0.180 | 0.091 | 0.299 | 0.186 | 0.195 | 0.140 | 0.181 | 0.306 | 0.196 | 0.228 | 0.227 | 0.499 | 0.540
conc | 0.310 | 0.205 | 0.224 | 0.177 | 0.210 | 0.171 | 0.202 | 0.116 | 0.215 | 0.143 | 0.153 | 0.236 | 0.164 | 0.12 | 0.222 | 0.153 | 0.093 | 0.192 | 0.131 | 0.369 | 0.200 | 0.216 | 0.177 | 0.206 | 0.171 | 0.200 | 0.179 | 0.174 | 0.172 | 0.382 | 0.318 | 0.310 | 0.369 | 0.200 | 0.214 | 0.177 | 0.207 | 0.155 | 0.191 | 0.336 | 0.163 | 0.745 | 0.259 | 0.699 | 0.447 | 0.454 | 0.370 | 0.421 | 0.608 | 0.448 | 0.304 | 0.601 | 0.279 | 0.507
contra | 0.551 | 0.533 | 0.430 | 0.549 | 0.439 | 0.332 | 0.478 | 0.298 | 0.519 | 0.5 | 0.323 | 0.524 | 0.505 | 0.271 | 0.556 | 0.479 | 0.244 | 0.539 | 0.464 | 0.472 | 0.437 | 0.369 | 0.445 | 0.370 | 0.331 | 0.477 | 0.43 | 0.456 | 0.538 | 0.599 | 0.551 | 0.585 | 0.472 | 0.437 | 0.369 | 0.445 | 0.371 | 0.351 | 0.503 | 0.249 | 0.420 | 0.881 | 0.828 | 0.814 | 0.776 | 0.716 | 0.771 | 0.718 | 0.672 | 0.662 | 0.565 | 0.802 | 0.579 | 0.719
craft | 0.178 | 0.145 | 0.146 | 0.169 | 0.183 | 0.084 | 0.111 | 0.071 | 0.118 | 0.108 | 0.082 | 0.128 | 0.110 | 0.075 | 0.125 | 0.105 | 0.056 | 0.108 | 0.09 | 0.219 | 0.144 | 0.142 | 0.164 | 0.176 | 0.084 | 0.111 | 0.059 | 0.196 | 0.11 | 0.221 | 0.206 | 0.252 | 0.219 | 0.144 | 0.142 | 0.164 | 0.176 | 0.067 | 0.111 | 0.144 | 0.088 | 0.528 | 0.272 | 0.602 | 0.516 | 0.509 | 0.549 | 0.509 | 0.543 | 0.492 | 0.344 | 0.684 | 0.324 | 0.520
drugs | 0.158 | 0.218 | 0.316 | 0.236 | 0.250 | 0.118 | 0.140 | 0.13 | 0.200 | 0.119 | 0.148 | 0.208 | 0.137 | 0.13 | 0.219 | 0.128 | 0.109 | 0.195 | 0.108 | 0.174 | 0.191 | 0.261 | 0.208 | 0.227 | 0.119 | 0.140 | 0.104 | 0.176 | 0.142 | 0.296 | 0.308 | 0.324 | 0.174 | 0.191 | 0.263 | 0.208 | 0.227 | 0.104 | 0.218 | 0.134 | 0.156 | 0.134 | 0.299 | 0.421 | 0.471 | 0.536 | 0.474 | 0.476 | 0.428 | 0.486 | 0.318 | 0.337 | 0.520 | 0.620
flare | 0.560 | 0.618 | 0.617 | 0.621 | 0.630 | 0.344 | 0.482 | 0.394 | 0.604 | 0.527 | 0.403 | 0.603 | 0.531 | 0.359 | 0.624 | 0.509 | 0.333 | 0.609 | 0.496 | 0.482 | 0.434 | 0.464 | 0.462 | 0.526 | 0.342 | 0.482 | 0.443 | 0.484 | 0.64 | 0.605 | 0.577 | 0.600 | 0.482 | 0.434 | 0.463 | 0.462 | 0.526 | 0.346 | 0.636 | 0.256 | 0.498 | 0.675 | 0.729 | 0.694 | 0.731 | 0.735 | 0.731 | 0.727 | 0.629 | 0.663 | 0.480 | 0.614 | 0.616 | 0.655
grid | 0.078 | 0.075 | 0.083 | 0.029 | 0.062 | 0.046 | 0.050 | 0.046 | 0.050 | 0.02 | 0.051 | 0.058 | 0.040 | 0.051 | 0.054 | 0.021 | 0.034 | 0.037 | 0.017 | 0.086 | 0.075 | 0.080 | 0.028 | 0.059 | 0.046 | 0.050 | 0.032 | 0.034 | 0.015 | 0.104 | 0.105 | 0.048 | 0.086 | 0.075 | 0.080 | 0.028 | 0.059 | 0.044 | 0.05 | 0.048 | 0.066 | 0.258 | 0.21 | 0.492 | 0.458 | 0.448 | 0.158 | 0.391 | 0.468 | 0.466 | 0.749 | 0.668 | 0.194 | 0.520
mush | 0.008 | 0.009 | 0.010 | 0.018 | 0.009 | 0.011 | 0.008 | 0.006 | 0.007 | 0.012 | 0.042 | 0.043 | 0.053 | 0.006 | 0.007 | 0.012 | 0.012 | 0.015 | 0.016 | 0.014 | 0.009 | 0.010 | 0.018 | 0.009 | 0.011 | 0.008 | 0.007 | 0.009 | 0.025 | 0.032 | 0.030 | 0.033 | 0.014 | 0.009 | 0.010 | 0.018 | 0.009 | 0.008 | 0.007 | 0.017 | 0.007 | 0.009 | 0.006 | 0.027 | 0.013 | 0.018 | 0.012 | 0.010 | 0.054 | 0.019 | 0.098 | 0.054 | 0.022 | 0.364
music | 0.480 | 0.583 | 0.592 | 0.460 | 0.606 | 0.324 | 0.535 | 0.372 | 0.543 | 0.405 | 0.379 | 0.544 | 0.411 | 0.333 | 0.587 | 0.409 | 0.312 | 0.571 | 0.386 | 0.462 | 0.433 | 0.440 | 0.387 | 0.497 | 0.324 | 0.535 | 0.392 | 0.477 | 0.416 | 0.593 | 0.555 | 0.566 | 0.462 | 0.433 | 0.435 | 0.387 | 0.495 | 0.328 | 0.557 | 0.257 | 0.475 | 0.840 | 0.806 | 0.748 | 0.772 | 0.792 | 0.711 | 0.751 | 0.651 | 0.666 | 0.465 | 0.572 | 0.614 | 0.684
musk | 0.083 | 0.088 | 0.129 | 0.066 | 0.093 | 0.070 | 0.074 | 0.068 | 0.075 | 0.047 | 0.085 | 0.093 | 0.087 | 0.064 | 0.069 | 0.048 | 0.056 | 0.060 | 0.044 | 0.096 | 0.079 | 0.105 | 0.062 | 0.087 | 0.069 | 0.074 | 0.06 | 0.067 | 0.051 | 0.166 | 0.155 | 0.138 | 0.096 | 0.079 | 0.107 | 0.062 | 0.087 | 0.068 | 0.075 | 0.065 | 0.101 | 0.188 | 0.131 | 0.367 | 0.281 | 0.359 | 0.180 | 0.277 | 0.379 | 0.292 | 0.248 | 0.321 | 0.313 | 0.509
spam | 0.103 | 0.060 | 0.071 | 0.061 | 0.056 | 0.069 | 0.050 | 0.04 | 0.048 | 0.041 | 0.061 | 0.071 | 0.067 | 0.04 | 0.047 | 0.044 | 0.034 | 0.047 | 0.04 | 0.236 | 0.059 | 0.069 | 0.060 | 0.057 | 0.069 | 0.050 | 0.046 | 0.068 | 0.052 | 0.111 | 0.083 | 0.103 | 0.236 | 0.059 | 0.071 | 0.060 | 0.057 | 0.047 | 0.045 | 0.265 | 0.052 | 0.603 | 0.066 | 0.595 | 0.251 | 0.263 | 0.236 | 0.216 | 0.537 | 0.264 | 0.261 | 0.638 | 0.217 | 0.519
study | 0.216 | 0.201 | 0.190 | 0.199 | 0.174 | 0.187 | 0.190 | 0.16 | 0.209 | 0.176 | 0.178 | 0.216 | 0.186 | 0.157 | 0.216 | 0.183 | 0.121 | 0.184 | 0.147 | 0.301 | 0.189 | 0.182 | 0.189 | 0.176 | 0.187 | 0.190 | 0.202 | 0.159 | 0.161 | 0.393 | 0.322 | 0.327 | 0.301 | 0.189 | 0.182 | 0.189 | 0.176 | 0.192 | 0.19 | 0.175 | 0.164 | 0.533 | 0.205 | 0.589 | 0.385 | 0.428 | 0.386 | 0.385 | 0.538 | 0.383 | 0.610 | 0.696 | 0.567 | 0.641
telco | 0.252 | 0.226 | 0.206 | 0.276 | 0.217 | 0.075 | 0.115 | 0.072 | 0.140 | 0.232 | 0.082 | 0.158 | 0.244 | 0.07 | 0.154 | 0.216 | 0.061 | 0.142 | 0.21 | 0.211 | 0.188 | 0.174 | 0.227 | 0.186 | 0.075 | 0.115 | 0.06 | 0.140 | 0.225 | 0.204 | 0.219 | 0.299 | 0.211 | 0.188 | 0.176 | 0.227 | 0.186 | 0.07 | 0.154 | 0.059 | 0.118 | 0.401 | 0.427 | 0.571 | 0.582 | 0.600 | 0.603 | 0.583 | 0.525 | 0.543 | 0.373 | 0.476 | 0.541 | 0.648
thrm | 0.488 | 0.493 | 0.461 | 0.454 | 0.496 | 0.318 | 0.450 | 0.294 | 0.459 | 0.418 | 0.323 | 0.460 | 0.418 | 0.274 | 0.462 | 0.405 | 0.240 | 0.439 | 0.376 | 0.462 | 0.384 | 0.372 | 0.389 | 0.449 | 0.318 | 0.450 | 0.383 | 0.401 | 0.456 | 0.553 | 0.485 | 0.566 | 0.462 | 0.384 | 0.375 | 0.389 | 0.447 | 0.309 | 0.483 | 0.266 | 0.404 | 0.861 | 0.686 | 0.773 | 0.702 | 0.679 | 0.675 | 0.677 | 0.655 | 0.629 | 0.491 | 0.629 | 0.494 | 0.604
turk | 0.585 | 0.653 | 0.652 | 0.575 | 0.693 | 0.248 | 0.359 | 0.295 | 0.545 | 0.581 | 0.295 | 0.538 | 0.579 | 0.252 | 0.560 | 0.517 | 0.251 | 0.559 | 0.517 | 0.477 | 0.452 | 0.455 | 0.438 | 0.477 | 0.246 | 0.359 | 0.293 | 0.502 | 0.548 | 0.500 | 0.589 | 0.567 | 0.477 | 0.452 | 0.451 | 0.438 | 0.477 | 0.28 | 0.559 | 0.164 | 0.460 | 0.881 | 0.878 | 0.847 | 0.851 | 0.845 | 0.836 | 0.848 | 0.684 | 0.692 | 0.558 | 0.640 | 0.562 | 0.734
vgame | 0.127 | 0.192 | 0.256 | 0.201 | 0.233 | 0.085 | 0.156 | 0.086 | 0.186 | 0.135 | 0.088 | 0.181 | 0.134 | 0.085 | 0.184 | 0.133 | 0.071 | 0.175 | 0.122 | 0.209 | 0.162 | 0.215 | 0.175 | 0.192 | 0.085 | 0.156 | 0.084 | 0.143 | 0.137 | 0.219 | 0.204 | 0.201 | 0.209 | 0.162 | 0.221 | 0.175 | 0.192 | 0.089 | 0.183 | 0.066 | 0.188 | 0.586 | 0.348 | 0.631 | 0.571 | 0.659 | 0.601 | 0.608 | 0.570 | 0.533 | 0.407 | 0.594 | 0.749 | 0.699
voice | 0.039 | 0.033 | 0.033 | 0.031 | 0.031 | 0.048 | 0.028 | 0.023 | 0.023 | 0.022 | 0.063 | 0.063 | 0.069 | 0.024 | 0.025 | 0.025 | 0.025 | 0.031 | 0.025 | 0.134 | 0.033 | 0.033 | 0.031 | 0.031 | 0.047 | 0.028 | 0.031 | 0.025 | 0.022 | 0.117 | 0.088 | 0.097 | 0.134 | 0.033 | 0.033 | 0.031 | 0.031 | 0.036 | 0.022 | 0.178 | 0.025 | 0.289 | 0.025 | 0.346 | 0.092 | 0.089 | 0.080 | 0.077 | 0.378 | 0.133 | 0.166 | 0.417 | 0.103 | 0.323
wine | 0.395 | 0.285 | 0.185 | 0.232 | 0.338 | 0.095 | 0.197 | 0.078 | 0.248 | 0.172 | 0.082 | 0.253 | 0.170 | 0.085 | 0.273 | 0.168 | 0.064 | 0.254 | 0.155 | 0.372 | 0.230 | 0.176 | 0.207 | 0.260 | 0.095 | 0.197 | 0.102 | 0.191 | 0.183 | 0.263 | 0.274 | 0.278 | 0.372 | 0.230 | 0.178 | 0.207 | 0.260 | 0.102 | 0.241 | 0.233 | 0.200 | 0.815 | 0.65 | 0.750 | 0.658 | 0.604 | 0.622 | 0.656 | 0.637 | 0.598 | 0.662 | 0.905 | 0.408 | 0.661
yeast | 0.570 | 0.514 | 0.365 | 0.425 | 0.421 | 0.238 | 0.446 | 0.237 | 0.489 | 0.378 | 0.283 | 0.514 | 0.397 | 0.213 | 0.482 | 0.372 | 0.189 | 0.472 | 0.353 | 0.471 | 0.415 | 0.328 | 0.374 | 0.380 | 0.241 | 0.442 | 0.295 | 0.442 | 0.422 | 0.523 | 0.517 | 0.514 | 0.471 | 0.415 | 0.324 | 0.374 | 0.379 | 0.261 | 0.467 | 0.38 | 0.386 | 0.873 | 0.739 | 0.839 | 0.759 | 0.672 | 0.697 | 0.717 | 0.680 | 0.653 | 0.569 | 0.881 | 0.516 | 0.780