# Naive Automated Machine Learning - A Late Baseline for AutoML

Felix Mohr, Marcel Wever

###### Abstract

Automated Machine Learning (AutoML) is the problem of automatically finding
the pipeline with the best generalization performance on some given dataset.
AutoML has received enormous attention in the last decade and has been
addressed with sophisticated black-box optimization techniques such as
Bayesian Optimization, Grammar-Based Genetic Algorithms, and tree search
algorithms. In contrast to those approaches, we present Naive AutoML, a very
simple solution to AutoML that exploits important meta-knowledge about machine
learning problems and makes simplifying, yet, effective assumptions to quickly
come to high-quality solutions. While Naive AutoML can be considered a
baseline for the highly sophisticated black-box solvers, we empirically show
that those solvers are not able to outperform Naive AutoML; sometimes the
contrary is true. On the other hand, Naive AutoML comes with strong advantages
such as interpretability and flexibility and poses a strong challenge to
current tools.

##  1 Introduction

Automated machine learning (AutoML) is the problem of automatically finding
the data transformation and learning algorithms with the best generalization
performance on a given dataset. The combination of such algorithms is
typically called machine learning _pipeline_ , because several algorithms for
data manipulation and analysis are concatenated sequentially. To optimize such
a machine learning pipeline, important decisions do not only include the
learning algorithm itself but also its parameters as well as a definition of
which features should be used by the learner. The latter decision is often
implicitly resolved by choosing and configuring feature pre-processing
algorithms.

Surprisingly, all common AutoML approaches solve the problem with a general-
purpose black-box optimization technique without explicitly using knowledge
about the AutoML problem [28, 6]. The main argument behind this approach and
the alleged superiority of AutoML tools over human data scientists is that a
lot of decisions for the pipeline need to be made _simultaneously_ , which
cannot be easily achieved in a sequential optimization process a human expert
would pursue. The surprising part is that imitating a human data scientist
seems a natural baseline, but never have the outcomes of such an imitation
process been used as a baseline for the complex tools presented in the
community. In [28], at least the attempt is made to the degree of running a
cross-validation over different learners, but no further steps a data
scientist would usually conduct are considered.

In this paper, we make up for this natural AutoML baseline, which imitates the
process a human expert might go through to find the best pipeline. Instead of
crawling an enormous search space with a black-box optimizer that iteratively
evaluates full pipeline candidates to finally pick the best one, the idea is
to conduct an analytical process and _derive_ the appropriate pipeline from
it. In this, we implicitly assume a form of independence of the optimization
decisions and hence act a bit _naive_ w.r.t. potential interactions between
them; this is why we call the approach Naive AutoML. In this sense, we
complement the “Ex-def” baseline [28] by what a somewhat more experienced data
scientist would maybe do.

Our experimental evaluation evidences the strength of this simple approach and
strongly jeopardizes current AutoML tools. Naive AutoML is highly competitive
with the state of the art and is sometimes even significantly better. In fact,
the experiments convincingly show that, other than reported in [28], even the
simple baseline of only trying out once each base learner with its default
configuration is often _not_ outperformed by the sophisticated optimizers.
This is a somewhat puzzling result, because the justifications to use the
current AutoML tools seem to disappear. Naive AutoML achieves comparable or
better results, is much more simple to understand, more flexible to extend,
and easy to implement. Taking all these observations together, Naive AutoML
imposes a strong challenge to currently advocated approaches.

At this point, we shall stress that the main goal of this paper is _not_ to
present yet another tool for AutoML but to propose a different _view_ on how
automated machine learning could be done. Recent work has revealed that
several aspects of the black-box character of current systems impede experts
from trusting their output, and that there is a need for more interaction
between the tool and the expert[31, 4, 5, 30]. Since Naive AutoML effectively
_imitates_ a data scientist, it makes an important step into this direction by
adopting an optimization process that is much more comprehensive for an expert
who interacts with it. Even if Naive AutoML was less competitive than it is,
we would still consider this perspective an important contribution of the
approach by itself. It paves the way for completely new approaches that are
inconceivable with the current black-box techniques.

##  2 Problem Definition

Even though the vision of AutoML is much broader, the AutoML problem in the
narrower sense as addressed by most AutoML approaches is to automatically
_compose_ and _parametrize_ machine learning algorithms to maximize a given
metric such as predictive accuracy. The available algorithms are typically
related either to preprocessing (feature selection, transformation,
imputation, etc.) or to the core functionality (classification, regression,
ranking, etc.).

In this paper, we are focused on AutoML for supervised learning. Formally, in
the supervised learning context, we assume some _instance space_
$\mathit{\mathcal{X}}\subseteq\mathbb{R}^{d}$ and a _label space_
$\mathit{\mathcal{Y}}$. A _dataset_
$D\subset\\{(x,y)~{}|~{}x\in\mathit{\mathcal{X}},y\in\mathit{\mathcal{Y}}\\}$
is a _finite_ relation between the instance space and the label space, and we
denote as $\mathit{\mathcal{D}}$ the set of all possible datasets. We consider
two types of operations over instance and label spaces:

  1. 1.

_Transformers._ A transformer is a function
$t:\mathit{\mathcal{X}}_{A}\rightarrow\mathit{\mathcal{X}}_{B}$, converting an
instance $x$ of instance space $\mathit{\mathcal{X}}_{A}$ into an instance of
another instance space $\mathit{\mathcal{X}}_{B}$.

  2. 2.

_Predictors._ A predictor is a function
$p:\mathit{\mathcal{X}}_{p}\rightarrow\mathit{\mathcal{Y}}$, assigning an
instance of its instance space $\mathit{\mathcal{X}}_{p}$ a label of the
original label space $\mathit{\mathcal{Y}}$.

In this paper, a _pipeline_ $P=t_{1}\circ..\circ t_{k}\circ p$ is a functional
concatenation in which
$t_{i}:\mathit{\mathcal{X}}_{i-1}\rightarrow\mathit{\mathcal{X}}_{i}$ are
transformers with $\mathit{\mathcal{X}}_{0}=\mathit{\mathcal{X}}$ being the
original instance space, and
$p:\mathit{\mathcal{X}}_{k}\rightarrow\mathit{\mathcal{Y}}$ is a predictor.
Hence, a pipeline is a function
$P:\mathit{\mathcal{X}}\rightarrow\mathit{\mathcal{Y}}$ that assigns a label
to each object of the instance space. We denote as $\mathit{\mathcal{P}}$ the
space of all pipelines of this kind. In general, the first part of a pipeline
could not only be a sequence but also a _transformation tree_ with several
parallel transformations that are then merged [24], but we do not consider
such structures in this paper since they are not necessary for our key
argument. An extension to such tree-shaped pipelines is canonical future work.

In addition to the sequential structure, many AutoML approaches restrict the
search space still a bit further. First, often a particular best order in
which different _types_ of transformation algorithms should be applied is
assumed. For example, we assume that feature selection should be conducted
after feature scaling. So $\mathit{\mathcal{P}}$ will only contain pipelines
compatible with this order. Second, the optimal pipeline uses at most one
transformation algorithm of each type. For example, no optimal pipeline will
concatenate two feature scalers. These assumptions allow us to express _every_
element of $\mathit{\mathcal{P}}$ as a concatenation of $k+1$ functions, where
$k$ is the number of considered transformation algorithm _types_ , e.g.,
feature scalers, feature selectors, etc. If a pipeline does not adopt an
algorithm of one of those types, say the $i$-th type, then $t_{i}$ will simply
be the identity function.

The theoretical goal in supervised machine learning would be to find

$P^{*}~{}\in\underset{P\in\mathit{\mathcal{P}}}{\operatorname{argmin}}~{}\mathit{\mathcal{R}}\big{(}P\big{)}\,,$
(1)

with the _risk_ or expected loss of the pipeline $P$ given by

$\mathit{\mathcal{R}}(P)=\int\limits_{\mathit{\mathcal{X}},\mathit{\mathcal{Y}}}{loss(y,P(x))}\,d\mathbb{P}(x,y)\,.$
(2)

Here, $loss(y,P(x))\in\mathbb{R}$ is the penalty for predicting $P(x)$ for
instance $x\in\mathit{\mathcal{X}}$ when the true label is
$y\in\mathit{\mathcal{Y}}$, and $\mathbb{P}$ is a joint probability measure on
$\mathit{\mathcal{X}}\times\mathit{\mathcal{Y}}$ from which the available
dataset $D$ has been generated.

In practice, (2) cannot be evaluated as the data-generating process
$\mathbb{P}$ is assumed to exist but is obviously not known to the learner.
Hence, the true performance $\mathit{\mathcal{R}}$ is replaced by some
performance estimator
$\mathit{\phi}:\mathit{\mathcal{D}}\times\mathit{\mathcal{P}}\rightarrow\mathbb{R}$
that estimates the performance of a candidate pipeline based on some
validation data. A typical metric used for this evaluation is the average loss
on the validation fold, but, based on the exact problem type, other metrics
such as least squares, AUC/ROC, F1 measure, or others are conceivable.

Consequently, a supervised AutoML problem _instance_ is defined by a dataset
$D\in\mathit{\mathcal{D}}$, a search space $\mathit{\mathcal{P}}$ of
pipelines, and a performance estimation metric
$\mathit{\phi}:\mathit{\mathcal{D}}\times\mathit{\mathcal{P}}\rightarrow\mathbb{R}$
for solutions. An AutoML _solver_
$\mathcal{A}:\mathit{\mathcal{D}}\rightarrow\mathit{\mathcal{P}}$ is a
function that creates a pipeline given some training set $D_{train}\subset D$.
The performance of $\mathcal{A}$ is given by

$\mathbb{E}\left[~{}\mathit{\phi}\big{(}D_{test},\mathcal{A}(D_{train})\big{)}\right]\,,$
(3)

where the expectation is taken with respect to the possible (disjoint) splits
of $D$ into $D_{train}$ and $D_{test}$. In practice, this score is typically
computed taking a series of random binary splits of $D$ and averaging over the
observed scores. Naturally, the goal of any AutoML solver is to optimize this
metric, and we assume that $\mathcal{A}$ has access to $\phi$ (but not to
$D_{test}$) in order to evaluate candidates with respect to the objective
function.

##  3 Related Work

Even though the foundation of AutoML is often attributed to the proposal of
Auto-WEKA, there have been some works on the topic before. These initial
approaches have been based on tree search. Here, a large search tree is given
in which each path encodes a machine learning pipeline. There are mainly three
approaches in this direction, differing in the way how the search space is
defined and how the search process is guided. The very first approach we are
aware of was designed for the configuration of RapidMiner modules based on
hierarchical planning [13, 14] most notably MetaMiner [22, 23, 21]. With ML-
Plan [20], the idea of HTN-based graph definitions was later combined with a
best-first search using random roll-outs to obtain node quality estimates.
Similarly, [26] introduced AutoML based on Monte-Carlo Tree Search, which is
closely related to ML-Plan. However, the authors of [26] do not discuss the
layout of the search tree, which is a crucial detail, because it is the main
channel to inject knowledge into the search problem.

Another line of research based on Bayesian Optimization was initialized with
the advent of Auto-WEKA [28, 16]. Like Naive AutoML, Auto-WEKA assumes a fixed
structure of the pipeline, admitting a feature selection step, and a
predictor. The decisions are encoded into a large vector that is then
optimized using SMAC [12]. Auto-WEKA optimizes pipelines with algorithms of
the Java data analysis library WEKA [10]. For the Python framework scikit-
learn [25], the same technique was adopted by auto-sklearn [6]. In the
original version, auto-sklearn added a data transformation step to the
pipeline; meanwhile the tool has been extended to support some more pre-
processing functionalities in the pipeline. Besides, auto-sklearn features
warm-starting and ensembling. The main difference between these approaches and
tree search is that tree search successively _creates_ solution candidates as
paths of a tree instead of obtaining them from an _acquisition function_ as
done by Auto-WEKA and auto-sklearn.

The idea of warm-starting introduced by auto-sklearn was also examined in
specific works based on recommendations. Approaches here include specifically
collaborative filtering like OBOE [33] and probabilistic matrix factorization
[8]. These approaches not necessarily require but are specifically designed
for cases in which a database of past experiences on other datasets is
available.

Another interesting line of research is the application of evolutionary
algorithms. One of these approaches is TPOT [24]. In contrast to the above
approaches, TPOT allows not just one pre-processing step but an arbitrary
number of feature extraction techniques at the same time. TPOT adopts a
genetic algorithm to find good pipelines and adopts the scikit-learn framework
to evaluate candidates. Another approach is RECIPE [27], which uses a grammar-
based evolutionary approach to evolve pipeline construction. In this, it is
similar to the tree search based approaches. Another genetic approach, focused
on the construction of stacking ensembles, was presented in [3].

A recent line of research adopts a type of black-box optimization relying on
the framework of multipliers (ADMM) [1]. The main idea here is to decompose
the optimization problem into two sub-problems for different variable types,
taking into account that algorithm selection variables are Boolean while most
parameter variables are continuous. This approach was first presented in [18].

All of the above approaches can be considered almost entirely black-box
optimization approaches. Some approaches like ML-Plan indirectly allow the
incorporation of domain knowledge through the search tree definition, and
almost all tools make some algorithmic adjustments to at least somewhat inject
knowledge to the overall process (like warm-starting, ensemble building,
etc.).

Notably, Auto-WEKA is the only tool that has been evaluated against an
(automated) data scientist baseline. In the Auto-WEKA publications, this
baseline is called “Ex-def”, and is simply a for-loop that iterates once over
each learner, assesses their performance in some cross-validation, and chooses
the best one. This is identical to our probing stage as detailed in Sec.
4.3.1. We are not aware of another work that ever came back to a simulated
data scientist baseline.

However, we have two concerns against the “Ex-def” baseline. First, in our
experiments, we could not reproduce the weak performance of this automated
amateur data scientist and found even this strategy to perform well quite
often. But even if this baseline is outperformed, we argue that the goal of
beating this performance is not ambitious enough. An expert data scientist
would hardly ever conduct like “Ex-def” and stop after this initial analysis
but conduct a series of further operations. In [28], a second baseline
includes a grid search, but many data scientists would probably rather focus
on data transformation instead of parameter tuning. In this paper, we
implement such a stronger baseline and show that the state-of-the-art
approaches hardly beat _this_ baseline.

##  4 Naive AutoML

###  4.1 Naivety Assumptions

Naive AutoML makes, among others, the assumption that the optimal pipeline is
the one that is locally best for each of its transformers and the final
predictor. In other words, taking into account pipelines with (up to) $k$
transformers and a predictor, we assume that for all datasets $D$ and all
$1\leq i\leq k+1$

$c_{i}^{*}\in\arg\min_{c_{i}}\mathit{\phi}(D,c_{1}\circ..\circ c_{k+1})$

is _invariant_ to the choices of $c_{1},..c_{i-1},c_{i+1},..,c_{k+1}$, which
are supposed to be fixed in the above equation. Note that we here use the
letter $c$ instead of $t$ for transformers or $p$ for the predictor, because
$c$ may be any of the two types.

We dub the approach Naive AutoML, because there is a direct link to the
assumption made by the Naive Bayes classifier. Consider $\mathit{\mathcal{P}}$
an urn and denote as $Y$ the event to observe an optimal pipeline in the urn.
Then

$\mathbb{P}(Y~{}|~{}c_{1},..,c_{k+1})\propto\mathbb{P}(c_{1},..,c_{k+1}~{}|~{}Y)\mathbb{P}(Y)\overset{naive}{=}\mathbb{P}(c_{i}~{}|~{}Y)\prod_{j=1,j\neq
i}^{k+1}\mathbb{P}(c_{j}~{}|~{}Y)\mathbb{P}(Y),$

in which we consider $c_{j}$ to be fixed components for $j\neq i$, and only
$c_{i}$ being subject to optimization. Applying Bayes theorem again to
$\mathbb{P}(c_{i}~{}|~{}Y)$ and observing that the remaining product is a
constant regardless the choices of $c_{i\neq j}$, it gets clear that the
optimal solution is the one that maximizes the probability of being locally
optimal, and that this choice is _independent_ of the choice of the other
components.

The typical approach to optimize the $c_{i}$ is not to directly construct
those functions but to adopt parametrized model building processes that create
these functions. For example, $c_{1}$ could be a projection obtained by
determining some features which we want to stay with, or $c_{k+1}$ could be a
trained neural network. These induction processes for the components can be
described by an algorithm $a_{i}$ and a parametrization $\theta_{i}$ of the
algorithm. The component $c_{i}$ is obtained by running $a_{i}$ under
parameters $\theta_{i}$ with some training data. So to optimize $c_{i}$, we
need to optimally choose $a_{i}$ and $\theta_{i}$.

Within this regime, Naive AutoML makes the additional assumption that even
each component $c_{i}$ can be optimized by local optimization techniques. More
precisely, it is assumed that the algorithm that yields the best component
when using the default parametrization is also the algorithm that yields the
best component if all algorithms are run with the best parametrization
possible.

Observe that this assumption is less far-fetched than one might expect. Our
preliminary experiments showed that the tuning of parameters has often no or
only a slim improvement over the performance achieved with the default
configuration. Literature reports some exceptions, but these are mostly in the
area of neural networks, in which the network architecture is considered as a
parameter and clearly has a substantial impact on the performance. However,
for most learners, the variance of the variable describing the _improvement_
of a configuration over the performance with default configuration is rather
low. And even if we consider some parameters so important that changing them
effectively completely changes the algorithm behavior, consider these
instantiations as different algorithms. For example, we could simply treat
support vector machines with different kernels and different (orders of)
complexity constants as different algorithms. In the large majority of
algorithms, this practice does not yield an explosion in the algorithm space.
In fact, the only exception we can think of is indeed neural networks.

###  4.2 The Naive AutoML Stage Scheme

The idea of Naive AutoML is to conduct a series of optimization _stages_. Each
stage optimizes one aspect of the pipeline. For example, one stage chooses a
predictor, and another optimizes its parameters. The order of stages does not
need to coincide with the order of the elements in the pipeline, and we also
admit that there are several stages for the _same_ aspect. In a strictly naive
setting, the latter one is not necessary, but it can make sense to adjust an
initial decision later when other decisions have been made.

Each stage produces a _candidate pool_ and _may_ be based on the candidate
pool (and other results) of the preceding stages. In Naive AutoML, a
_candidate_ is simply an encoding that contains all the information necessary
to build and evaluate the corresponding pipeline. For example, in our concrete
prototype presented in Sec. 4.3, a candidate is encoded by a tuple
$\mathit{(scale,F,a,\theta)}$, where $\mathit{scale}$ encodes the feature
transformations (applied to all columns), $F$ is a _set_ of feature indices to
consider, $a$ is the prediction algorithm, and $\theta$ the parametrization
for $a$.

While every stage is entirely free in the way how it modifies the candidate
pool, there is a general pattern underlying the behavior of each stage. We
illustrate this pattern in Fig. 1, in which candidate pools are green and
stages are blue. The general pattern is that a stage creates and evaluates a
new set of candidates, potentially taking into account existing candidates,
and finally may also erase some candidates produced by itself or by previous
stages. Candidates from the incoming candidate pool might be ignored, e.g. if
we just want to try some entirely new type of pipelines. But we can also
consider the _whole_ incoming pool, e.g. when combining every existing
candidate with a particular concept for feature selection. Often, a stage will
_sort_ the candidates of the incoming pool by their performance, and consider
them one by one. For example, in a parameter tuning stage, we might tune the
parameters of the algorithms of the incoming candidates, one by one in the
order of their previous performance. At the end of the stage, we might
eliminate some candidates. For example, a parameter tuning stage may generate
a large set of new candidates, but it is reasonable to stay only with a small
portion of them. It can also make sense to eliminate candidates that came with
the input pool. For example, if we have an initial stage that only considers
learners with configurations that enable a very fast runtime, we might want to
replace them later when other parametrized versions of them have been tried.

Figure 1: Generic schema of a single stage, taking a candidate pool as input
and eventually returning an augmented candidate pool.

With respect to time limitations, Naive AutoML seems to be more flexible than
the black-box approaches, because time can be allocated with respect to
certain optimization steps. That is, besides a global timeout, Naive AutoML of
course also allows specifying timeouts _per stage_ or even for steps _inside_
of a stage. For example, if we have a stage to tune the learner hyper-
parameters, it usually makes sense to limit the time consumed in this stage
and, on top of that, to limit the time used to optimize parameters _per_
candidate. The overall runtime for the stage could be 10 minutes, with a time
budget of 2 minutes for each learner that is optimized.

The output of Naive AutoML can be obtained by taking the candidate with the
best solution observed either in _any_ stage or in the _last_ stage. The
default output is simply the best candidate that has been observed during the
process in _any_ stage. However, we could also introduce a (final) stage with
the sole purpose of _validation and model selection_. For this purpose, we
would hold back data from the beginning that is not used in all but the last
stage. In this case, we would of course use the best candidate of the last
pool only (these scores are not comparable to those of the other stages then
anyway).

The candidate pools enable a certain degree of communication between the
stages, which might seem a bit contradictory to a strictly independent
optimization. However, some optimization steps are just logically dependent.
For example, it does not make sense to optimize hyperparameters of the
predictor before we have chosen a predictor (unless we optimize for all
possible candidates). But even if such logical dependencies do not exist, we
argue that our goal here is not to implement a strictly naive approach but to
rather mimic an expert data scientist’s optimization workflow. While the
expert’s process is still largely naive in the way how it implicitly prunes
large spaces of the search space, it will also employ _some_ interaction
between the stages. For example, we could not only select the features to
stick with for the case of the original data but also for features that have
been scaled before, and where the scaling was determined in a previous stage.
This interaction is still fairly reasonable but far away from simultaneous
optimization of feature scaling and feature selection and can still be
considered “largely” naive.

###  4.3 A Prototype for a Stage Scheme

In this paper, we work with a specific _exemplary_ stage scheme. The purpose
of this scheme is only to illustrate the scheme idea with a concrete example
and to provide a means of evaluating the idea of Naive AutoML. However, this
scheme is _not_ what we refer to as Naive AutoML but only _one prototypical
instantiation_ of the Naive AutoML approach. We certainly do not claim this
prototype to be the last answer in Naive AutoML; better schemes can be found
in future work.

Our prototype stage scheme is sketched in Fig. 2. The first stage simply
cross-validates every learning algorithm once without setting any of its
hyper-parameters. This stage corresponds to the “Ex-def” baseline used in
[28]. The second stage simply pairs all feature scalers with all learners and
observes their performance. The third stage is independent of the first two
stages and adopts filtering techniques to identify a subset of the features
that are expected to bring the best performance (on average). The fourth stage
combines all previous candidates with each homogeneous meta learner, and the
fifth stage tunes the _learner_ parameters of the incoming candidates (with a
given time budget). In the final stage, the most promising candidates are
evaluated against a validation dataset. Below, we discuss each of these stages
in detail.

This stage scheme produces pipelines with up to three components. A candidate
is encoded by a tuple $\mathit{(s,F,a,\theta)}$, where $\mathit{s}$ encodes
the feature transformations (applied to all columns), $F$ is a _set_ of
feature indices to consider, $a$ is the prediction algorithm, and $\theta$ the
parametrization for $a$. The reason why $s$ is a function but $F$ is a set is
simply that the feature selection is a projection that is entirely described
by the features it shall retrain, but the scaling implies a functional
transformation of data that cannot be captured so easily. We will use the
symbol $\mathit{\bot}$ to indicate that a choice has been left blank.

Despite being only an example scheme, one advantage of Naive AutoML over
black-box optimization that already becomes clear here is that it directly
generates important insights that can significantly support the data scientist
working with it. For example, even such a simple question as “what is the
potential of feature selection on the given data?” cannot be directly answered
by the existing black-box approaches. In our scheme, the filtering stage
discussed in Sec. 4.3.3 is a very good basis to give an initial answer to this
question. More complex stage schemes, e.g. including _wrapping_ , can answer
such questions even in much more detail. In this sense, Naive AutoML presents
itself as more amenable to the growing demand for meaningful _interaction_
between the tool and the human [31, 4, 5, 30] compared to the currently
adopted black-box approaches.

Figure 2: Visualization of the stages of the Naive AutoML machine learning
pipeline (top), taking a dataset as an input and returning a final candidate.
The lower part of the figure illustrates the potential shapes of machine
learning pipelines contained in the candidate pool after the execution of the
respective stage.

In the following sub-sections, we will assume that candidates are evaluated
with some standard evaluation scheme. A typical choice to evaluate candidates
is to use standard cross validation techniques. For example, in the evaluation
in Sec. 5, we conduct a Monte-Carlo Cross Validation (MCCV) with a train fold
size of 70% and 5 repetitions. This means that we build 5 random splits of 70%
training data and 30% validation data each, train and test over the 5 folds
using the desired metric, and then average these observations to get to a
score. However, Naive AutoML is not committed to a particular type of scoring
function and could, for example, also be run with a 10-fold cross validation.
We hence just shall assume that every pipeline has some _score_ , and we leave
it to the concrete implementation to implement one or another method.

####  4.3.1 The Probing Stage

This stage corresponds to the “Ex-def” suggested in the evaluation of [28].
Formally, we create and evaluate _all_ candidates of the form
$(\mathit{\bot},\mathit{\bot},a,\mathit{\bot})$, where $a$ is one of the
available _base_ prediction algorithms. Here, we do not consider ensemble
learners like Boosting or Bagging but only those that are already implemented
with a specific base learner, such as Random Forests.

####  4.3.2 The Feature-Scaling Stage

In this step, we examine the benefit of different feature scaling operations
for the base learners. To this end, two or three cheap distance-sensitive
pilot classifiers like kNN or SVMs are used to assess the impact of different
feature scaling techniques such as (mean)-normalization, standardization (mean
0 and std 1), or quantile-based re-scaling. In addition, it can make sense to
add as a pilot the one or two best candidates resulting from the probing
stage. Formally, for each scaler $s$ and each pilot algorithm $a_{p}$, we
evaluate the candidate $(s,\mathit{\bot},a_{p},\mathit{\bot})$.

For each scaling technique $s$, if at least one pilot classifier improves upon
its result of the original data, we evaluate _all_ other candidates
$(s,\mathit{\bot},a,\mathit{\bot})$ as well, where $a$ is any non-pilot base
learner.

####  4.3.3 The Filtering Stage (Classifier-Independent Attribute Selection).

Figure 3: Error rates when applying filtering on the datasets cnae-9 (left)
with 856 features in total and Madelon (right) with 500 features in total.
Each curve shows at point $x$ the error rate obtained when using the first $x$
attributes according to the ranking created by the respective feature
evaluator. In both cases, it is evident that there is little point in using
all the features. In the better case, this only implies a waste of
computational time (cnae-9) while in the worse case this even implies inferior
results (Madelon).

This stage consists of two steps. In the first step, we compute the set $F$ of
features that are considered relevant. In the second step, we examine how the
previously created candidates behave when using only the features of $F$. That
is, for every candidate $(s,\mathit{\bot},a,\mathit{\bot})$, we create and
evaluate the candidate $(s,F,a,\mathit{\bot})$. Here $s$ may be
$\mathit{\bot}$ as well, and the candidates are created in the order of the
performance of the pipeline without feature selection. Typically, one may want
to define a timeout for this stage to not evaluate overly unpromising
candidates if a lot of time has already been used.

The feature set $F$ is computed based on the results of different _filtering_
techniques [11]. Different techniques to _rank_ the relevance of features
exist. In contrast to _wrapping_ [15], filtering techniques do not adopt
(costly) learners to judge feature relevance and hence are typically cheap to
compute. We execute all such available filters, which gives us a set of
rankings. Then, for each of these rankings, we compute the performance when
using the first $l$ features where $l$ is increased until the performance
starts to decrease. To assess the performance, a pilot classifier is evaluated
on the respective feature set in some kind of cheap cross-validation. In this
sense, Naive AutoML uses existing pre-processors only as a source of
_suggestions_ for candidate feature sets.

To motivate this procedure, it is worth to have a look at the _performance
curves_ one obtains for the different filters. For each number $x$ of
features, we can plot the performance of some pilot learner if using only the
first $x$ features in the ranking created by some filter. Fig. 3 illustrates
these curves for two datasets and using different filters from WEKA. In the
case of the cnae-9 dataset (left), we can see that there is only marginal
improvement after $x=100$, and for the madelon dataset (right) we can even
observe that performance _decreases_ quickly. In both cases, it is evident
that only a relatively small portion of the features is needed to obtain the
same or even better results compared to when all features are used (notice
that the plot curves to not even show the full range of features for both
datasets).

Note that the filtering stage optimizes the feature set $F$ independently of
the outcomes of earlier stages. Neither the results of the probing stage nor
those of the scaling stage are used in the determination of $F$. Those
outcomes are only relevant in that they define the candidates to which the
feature selection $F$ should be added, and their order for the evaluation.

One important additional use case for the analysis of such curves is the
reduction of evaluation times during the AutoML process. Even if we do not
_need_ to reduce the dimensionality to obtain good results as in the case of
madelon (right), we can often still _approximate_ the prediction performance
on the full feature set sufficiently well. For example, in the case of cnae-9
(left), we only need to use 100 of the 856 features to obtain comparable
results, which should be good enough to steer the search process. Since we
know that the runtime of many learners often increases super-linearly
(sometimes quadratically) in the number of attributes [19], there is a huge
potential in runtime reduction. For example, the time to train a random forest
on cnae-9 on some reference machine is 2.2s on average but only 0.5s on 100
attributes with almost the same outcome.

####  4.3.4 Meta-Learner Stage.

In this stage, each candidate of the input pool is taken and used as a
template to derive new pipelines in which the base learner is wrapped into a
meta-learner. The feature transformation algorithms, if present, are not
touched in this stage.

The algorithms we consider here as meta-learners are also sometimes called
(_homogeneous_) ensembles. The idea of those learners is to take several
copies of a base learner and somehow combine them into a new augmented
learner. Typical examples are Bagging [2] and AdaBoost [7]. This is opposed to
_heterogeneous_ ensembles like Stacking [32] or majority vote ensembles, in
which different learners of different types are combined. Since we augment
existing pipelines (with one learner), we only work with homogeneous ensembles
here.

Forming a heterogeneous ensemble is a reasonable final step. This would be
identical to the strategy pursued in [6] to eventually take the best $k$
learners seen so far and merge them into a voting ensemble. Alternatively,
more sophisticated approaches could try to optimize such an ensemble using
previous observations.

####  4.3.5 Parameter Tuning.

This stage simply tries to find better hyperparameters for the predictor in
one or more candidate pipelines. Given a total timeout for this phase, the
candidates are optimized in the order of their performance with a (local)
timeout and a maximum number of evaluations. For algorithms with a small
parameter space, all candidates can be enumerated, e.g. k-nearest neighbors
with some reasonable candidate set for the number of neighbors. For all other
algorithms, standard techniques for finding good configurations such as SMAC
[12], Hyperband [17], etc. can be employed. In this paper, we even only adopt
a simple random search.

####  4.3.6 (Validation-Based) Model Selection.

The default decision of Naive AutoML to select the best seen pipeline might
not always be the best. Despite the simplicity and naivity of Naive AutoML,
there is some significant optimization going on, potentially leading to over-
fitting.

The potential need of some validation has been recognized earlier [20], and we
adopt a similar strategy here. More precisely, the idea is to keep back a
certain portion of the original data that is not shown to the optimization
process and _only_ used in this final stage. The main difference to [20] is
that we do not add the validation data to the pool and then run a cross-
validation on the augmented dataset, but we here simply conduct a “classical”
single-fold validation on this hold-out set.

The data portion used for validation can be chosen dynamically based on the
desired guarantees on the generalization performance. For a set of $m$
remaining competitive candidates, the Hoeffding bound allows us to estimate
the out-of-sample error with
$\mathbb{P}(|\mu-\nu|>\varepsilon)<2me^{-2\varepsilon n}$, where $\mu$ is the
true out-of-sample error, $\nu$ is the error on the validation fold, and $n$
is the _size_ of the validation fold. Given sufficient data and for a moderate
number $m$, say $m=10$, this can be quite a good bound and impose an important
remedy against over-fitting after an exhaustive optimization effort.

Unfortunately, the number of validation samples available is often not
sufficient to make the Hoeffding bound meaningful. In most cases, we want to
choose $\varepsilon\approx 0.01$ and have the bound relatively small, say,
$0.1$. To assure such a bound for even only one candidate (effectively
_testing_ its performance), the validation fold must already contain roughly
15000 examples. For $m=10$, we would need 35,000 validation instances, which
are often not available.

We hence propose to use both the “internal” score observed during the
optimization process _and_ the validation score for model selection and weight
the two scores based on the validation set size. To this end, we introduce a
parameter $\bar{n}$ that quantifies the number of validation instances
required to exclusively use validation instances to compute the score.
Intuitively, $\bar{n}$ is the number of instances needed to get the desired
certainty in the Hoeffding bound, e.g., 35,000 instances. The _satisfaction_
of the Hoeffding bound can then be expressed as

$\tau(n)=\min\\{1,\frac{n}{\bar{n}}\\},$

where $n$ is the actual size of the validation fold. However, a close-to-0
satisfaction does not necessarily mean a close-to-0 weight of the validation
score. For example, if we have 500 instances in total and use 100 of them for
validation, then 100 is probably far away from $\bar{n}$, and the satisfaction
is almost 0. Still, the 100 instances are a valuable complement to the 400
instances used for training. In this case, it makes more sense to weight the
validation score based on the _ratio_ between the number of validation
instances and totally available instances; here, this would be 0.2. Hence,
instead of using the satisfaction of the validation fold size directly as a
weight, we use it to determine the point on a linear scale between the above
ratio and 1. Formally, we define the pipeline score in the validation phase
then to be

$\mathit{\phi}_{final}(P)=\mathit{\phi}_{int}(P)(1-\omega(n))+\mathit{\phi}_{validate}(P)\omega(n),$

where $\mathit{\phi}_{int}$ is the internal score obtained using only the data
available for optimization, $\mathit{\phi}_{validate}$ is the performance
obtained using _only_ the validation data (the pipeline has still been defined
not using these data points), $n$ is the number of instances in the validation
fold, and

$\omega(n)=\tau(n)+\frac{n}{N}(1-\tau(n)).$

In the last term, $N$ is the total number of instances available.

##  5 Evaluation

In this section, we want to address the following research questions:

  1. 1.

How much can state-of-the-art tools for Python and Java improve over the
baseline imposed by Naive AutoML?

  2. 2.

What is the independent merit of each technique over the probing stage?

  3. 3.

Are there any synergetic effects observable when employing several stages?

We stress our point of view that Naive AutoML is in fact the baseline here and
not the competing technique. Once more we argue that the baselines proposed in
[28] such as “Ex-def” or a grid search do not reflect an expert data
scientist. Naive AutoML hence _updates_ this baseline by proposing a more
sophisticated data scientist (acknowledging that an expert data scientist
would be even more flexible and hence might even be stronger than our Naive
AutoML). In this sense, our evaluation could be understood as a way of showing
how experimental evaluations of other tools might have looked like if a more
sophisticated baseline, like Naive AutoML, would have been adopted.

###  5.1 Compared Algorithms

To answer the above questions, we ran Naive AutoML in several configurations.
First, we computed results (and runtimes) of Naive AutoML when using no or at
most one of the optional stages. This allows us to see whether a particular
stage alone can improve over the primitive selection algorithm. In addition,
we considered the _monotone_ scenarios in which we consider stage setups of
increasing “complexity” (scenario $i$ takes all stages prior to $i$ and _adds_
stage $i$ to this set). This allows us to observe whether the _combination_ of
two techniques can improve upon the simpler approach.

In the following, we shortcut the two extreme profiles to give a summarized
comparison. We refer to “primitive” as the version of Naive AutoML that _only_
adopts the probing stage and nothing else. Once again, this corresponds to the
“Ex-def” baseline proposed in [28]. The profile containing _all_ the stages
(including validation), is referred to as “full”. The validation stage, if
applied sets $\bar{n}=10000$ and considers the best 10 candidates in the pool.

On the state-of-the-art side, we compare solutions with competitive
performance for both WEKA [10] and scikit-learn [25]. It is well-known that
Auto-WEKA is not competitive even inside the WEKA domain [20], but we still
consider it here to contrast our results of the comparison of AutoML with the
“Ex-def” baseline to those reported in [28]. To our knowledge, ML-Plan [20] is
the best performing AutoML tool for WEKA to date, so we consider it as the
second baseline for WEKA. On the scikit-learn side, we consider only auto-
sklearn as a baseline (without warm-starting and without final ensemble
building). While auto-sklearn has been beaten by several other tools, these
tools either rely on warm-starting ([33, 8]) or do not outpferform auto-
sklearn to such a degree that we would have to consider it a second-class
solution. The latter can be either due to the fact that results are generally
comparable, like in the case of TPOT [24], or that results are only reported
in the way of average ranks, which obscure the factual improvement [26, 18].
Hence, auto-sklearn is still a state-of-the-art solution to us. In this paper,
we consider version 0.12.0, which underwent substantial changes and
improvements compared to the original version [6].

###  5.2 Experiment Setup

The evaluation is based on the dataset portfolio proposed in [9]. This is a
collection of datasets available on openml.org [29]. To complement these
datasets, we have added some of the datasets that were used in the Auto-WEKA
paper [28] and have been used frequently for comparison in publications on
AutoML. Five datasets of [9] (23, 31, 188, 40996, 41161, 42734) were removed
due to technical issues in the data loading process.

For each dataset, 10 random train-test splits were created, and all algorithms
were run once on each of these splits, using the train data for optimizing,
and the test data to assessing the performance. Needless to say, the splits
were the same for all approaches.

Timeouts were configured as follows. For all algorithms, we allowed a total
runtime of 1h, and the runtime for a single pipeline execution was configured
to take up to 1 minute. In Auto-WEKA, it is not possible to configure the
maximum runtime for single executions; this variable is interenally
controlled. In Naive AutoML, we imposed stage time bounds for the meta and the
parameter tuning stages of 5 minutes respectively (the other stages were not
equipped with a dedicated timeout). Of course, on hitting the time bound of
1h, Naive AutoML was stopped regardless the phase in which it was, and the
best seen solution was returned.

The computations were executed in a compute center with Linux machines, each
of them equipped with 2.6Ghz Intel Xeon E5-2670 processors and 32GB memory. In
spite of the technical possibilities, we did _not_ parallelize evaluations.
That is, all the tools were configured to run with a single CPU core111This
was mainly due to technical reasons, because the standard parallelization
mechanism in Python does not support shared memory and copies contexts by
default to the different threads. This can blow-up the memory for some of the
considered datasets, so we refrained from parallelism.. Still, Naive AutoML
has immediate support for parallelization, and the code (both Java and Python)
that was used for the experiments is publicly available222
https://github.com/fmohr/naiveautoml.

###  5.3 Results

Table 1: Mean error rates and standard deviations per machine learning
library.

| WEKA backend | scikit-learn backend
---|---|---
id | auto-weka | mlplan | primitive | full | asklearn | primitive | full
3 |  0.0$\pm$0.0 $\circ$ $\circ$ |  0.01$\pm$0.01 $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 |  0.0$\pm$0.0 $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0
12 |  0.02$\pm$0.0 $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 |  0.03$\pm$0.02 $\circ$ $\circ$ | 0.02$\pm$0.02 | 0.03$\pm$0.02
54 |  0.18$\pm$0.02 $\circ$ $\circ$ |  0.18$\pm$0.03 $\circ$ $\circ$ | 0.17$\pm$0.02 | 0.18$\pm$0.05 | 0.19$\pm$0.03 $\circ$ $\circ$ | 0.16$\pm$0.03 | 0.17$\pm$0.03
181 |  0.39$\pm$0.04 $\circ$ $\circ$ |  0.39$\pm$0.04 $\circ$ $\circ$ | 0.38$\pm$0.03 | 0.39$\pm$0.04 |  0.39$\pm$0.04 $\circ$ $\circ$ | 0.38$\pm$0.03 | 0.37$\pm$0.05
1049 |  0.1$\pm$0.02 $\circ$ $\circ$ |  0.09$\pm$0.01 $\circ$ $\circ$ | 0.09$\pm$0.02 | 0.09$\pm$0.02 |  0.09$\pm$0.02 $\circ$ $\circ$ | 0.09$\pm$0.02 | 0.09$\pm$0.01
1067 |  0.15$\pm$0.01 $\circ$ $\circ$ |  0.15$\pm$0.01 $\circ$ $\circ$ | 0.15$\pm$0.02 | 0.15$\pm$0.01 |  0.14$\pm$0.02 $\circ$ $\circ$ | 0.15$\pm$0.02 | 0.15$\pm$0.01
1111 |  0.02$\pm$0.0 $\circ$ $\circ$ |  0.02$\pm$0.0 $\circ$ $\bullet$ | 0.02$\pm$0.0 | 0.04$\pm$0.0 |  0.02$\pm$0.0 $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0
1457 |  0.29$\pm$0.0 $\circ$ $\circ$ |  0.25$\pm$0.03 $\circ$ $\circ$ | 0.25$\pm$0.03 | 0.24$\pm$0.03 | 0.2$\pm$0.02 $\bullet$ $\circ$ | 0.23$\pm$0.04 | 0.16$\pm$0.03
1461 |  0.1$\pm$0.0 $\circ$ $\circ$ |  0.1$\pm$0.01 $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 |  0.11$\pm$0.0 $\circ$ $\circ$ | 0.11$\pm$0.0 | 0.11$\pm$0.0
1464 |  0.22$\pm$0.03 $\circ$ $\circ$ | 0.24$\pm$0.01 $\circ$ | 0.21$\pm$0.04 | 0.23$\pm$0.03 |  0.22$\pm$0.02 $\circ$ $\circ$ | 0.22$\pm$0.02 | 0.21$\pm$0.03
1468 |  0.06$\pm$0.02 $\circ$ $\circ$ |  0.05$\pm$0.02 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 |  0.05$\pm$0.02 $\circ$ $\circ$ | 0.04$\pm$0.02 | 0.04$\pm$0.01
1475 | 0.38$\pm$0.03 $\circ$ $\circ$ |  0.36$\pm$0.02 $\circ$ $\circ$ | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.39$\pm$0.03 $\circ$ $\circ$ | 0.37$\pm$0.02 | 0.37$\pm$0.02
1485 | 0.39$\pm$0.1 $\circ$ $\circ$ | 0.23$\pm$0.05 $\circ$ $\circ$ | 0.23$\pm$0.02 | 0.11$\pm$0.02 |  0.11$\pm$0.02 $\dagger$ | 0.26$\pm$0.03 | 0.12$\pm$0.01
1486 |  0.03$\pm$0.0 $\circ$ $\circ$ | 0.05$\pm$0.01 $\circ$ $\circ$ | 0.03$\pm$0.0 | 0.03$\pm$0.0 |  0.05$\pm$0.0 $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0
1487 |  0.05$\pm$0.01 $\circ$ $\circ$ |  0.06$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 |  0.05$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01
1489 |  0.09$\pm$0.01 $\circ$ $\circ$ |  0.09$\pm$0.01 $\circ$ $\circ$ | 0.09$\pm$0.01 | 0.09$\pm$0.01 |  0.1$\pm$0.01 $\circ$ $\circ$ | 0.09$\pm$0.01 | 0.09$\pm$0.01
1494 |  0.13$\pm$0.03 $\circ$ $\circ$ |  0.13$\pm$0.04 $\circ$ $\circ$ | 0.13$\pm$0.02 | 0.12$\pm$0.03 | 0.14$\pm$0.03 $\circ$ $\circ$ | 0.13$\pm$0.03 | 0.12$\pm$0.03
1515 |  0.09$\pm$0.03 $\circ$ $\circ$ |  0.13$\pm$0.04 $\circ$ $\circ$ | 0.13$\pm$0.04 | 0.13$\pm$0.04 |  0.07$\pm$0.03 $\bullet$ | 0.13$\pm$0.03 | 0.07$\pm$0.26
1590 |  0.14$\pm$0.0 $\circ$ $\circ$ |  0.14$\pm$0.01 $\circ$ $\circ$ | 0.14$\pm$0.0 | 0.14$\pm$0.01 |  0.16$\pm$0.0 $\circ$ $\circ$ | 0.16$\pm$0.0 | 0.16$\pm$0.0
4134 |  0.23$\pm$0.02 $\circ$ $\circ$ |  0.2$\pm$0.01 $\circ$ $\circ$ | 0.2$\pm$0.01 | 0.22$\pm$0.05 |  0.2$\pm$0.01 $\circ$ $\circ$ | 0.19$\pm$0.02 | 0.2$\pm$0.02
4135 |  0.05$\pm$0.0 $\circ$ $\circ$ |  0.06$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.08$\pm$0.03 |  0.05$\pm$0.0 $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0
4534 |  0.03$\pm$0.0 $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.01 |  0.02$\pm$0.0 $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.0
4538 |  0.32$\pm$0.02 $\circ$ $\circ$ |  0.32$\pm$0.01 $\circ$ $\circ$ | 0.32$\pm$0.01 | 0.31$\pm$0.01 |  0.31$\pm$0.01 $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01
4541 |  0.43$\pm$0.0 $\circ$ $\circ$ |  0.42$\pm$0.01 $\circ$ $\circ$ | 0.42$\pm$0.01 | 0.42$\pm$0.01 |  0.42$\pm$0.0 $\circ$ $\circ$ | 0.42$\pm$0.0 | 0.42$\pm$0.0
23512 | 0.31$\pm$0.0 $\circ$ $\circ$ | 0.32$\pm$0.02 $\circ$ $\circ$ | 0.31$\pm$0.02 | 0.28$\pm$0.01 |  0.28$\pm$0.0 $\circ$ $\circ$ | 0.28$\pm$0.01 | 0.28$\pm$0.01
23517 |  0.48$\pm$0.01 $\circ$ $\circ$ |  0.48$\pm$0.01 $\circ$ $\circ$ | 0.48$\pm$0.0 | 0.48$\pm$0.01 |  0.48$\pm$0.01 $\circ$ $\circ$ | 0.48$\pm$0.0 | 0.48$\pm$0.0
40498 |  0.31$\pm$0.01 $\circ$ $\circ$ |  0.31$\pm$0.02 $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.34$\pm$0.02 $\circ$ $\circ$ | 0.29$\pm$0.01 | 0.29$\pm$0.01
40668 |  0.03$\pm$0.0 $\circ$ $\bullet$ | 0.19$\pm$0.01 $\circ$ $\bullet$ | 0.21$\pm$0.11 | 0.27$\pm$0.0 | 0.17$\pm$0.06 $\dagger$ $\circ$ | 0.26$\pm$0.0 | 0.03$\pm$0.01
40670 |  0.04$\pm$0.01 $\circ$ $\circ$ |  0.05$\pm$0.01 $\circ$ $\circ$ | 0.04$\pm$0.01 | 0.04$\pm$0.02 |  0.03$\pm$0.01 $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01
40685 |  0.0$\pm$0.0 $\circ$ $\circ$ |  0.0$\pm$0.0 $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 |  0.0$\pm$0.0 $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0
40701 |  0.05$\pm$0.01 $\circ$ $\circ$ |  0.04$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 |  0.04$\pm$0.0 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01
40900 |  0.01$\pm$0.0 $\circ$ $\circ$ |  0.01$\pm$0.0 $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 |  0.01$\pm$0.0 $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0
40975 |  0.0$\pm$0.0 $\circ$ $\circ$ |  0.0$\pm$0.01 $\circ$ $\circ$ | 0.0$\pm$0.01 | 0.0$\pm$0.01 |  0.0$\pm$0.0 $\circ$ $\circ$ | 0.01$\pm$0.01 | 0.01$\pm$0.01
40978 |  0.02$\pm$0.01 $\circ$ $\circ$ |  0.03$\pm$0.01 $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.03$\pm$0.01 |  0.03$\pm$0.01 $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01
40981 | 0.14$\pm$0.05 $\circ$ | 0.14$\pm$0.03 $\circ$ | 0.12$\pm$0.03 | 0.13$\pm$0.03 |  0.12$\pm$0.03 $\circ$ $\circ$ | 0.12$\pm$0.03 | 0.12$\pm$0.02
40982 |  0.19$\pm$0.02 $\circ$ $\circ$ |  0.2$\pm$0.02 $\circ$ $\circ$ | 0.2$\pm$0.03 | 0.21$\pm$0.03 |  0.21$\pm$0.04 $\circ$ $\dagger$ | 0.21$\pm$0.02 | 0.25$\pm$0.1
40983 |  0.02$\pm$0.0 $\circ$ $\circ$ |  0.02$\pm$0.0 $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.01 |  0.01$\pm$0.0 $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0
40984 |  0.02$\pm$0.01 $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 |  0.02$\pm$0.01 $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01
41027 | 0.16$\pm$0.02 $\circ$ $\circ$ | 0.15$\pm$0.02 $\circ$ $\circ$ | 0.09$\pm$0.0 | 0.09$\pm$0.02 |  0.07$\pm$0.01 $\dagger$ $\dagger$ | 0.16$\pm$0.01 | 0.14$\pm$0.0
41138 |  0.01$\pm$0.0 $\circ$ $\circ$ |  0.01$\pm$0.0 $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 |  0.01$\pm$0.0 $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0
41142 | 0.29$\pm$0.02 $\circ$ $\circ$ | 0.28$\pm$0.02 $\circ$ $\circ$ | 0.27$\pm$0.02 | 0.26$\pm$0.03 |  0.27$\pm$0.03 $\circ$ $\circ$ | 0.28$\pm$0.02 | 0.28$\pm$0.02
41143 |  0.16$\pm$0.0 $\circ$ $\circ$ |  0.17$\pm$0.02 $\circ$ $\circ$ | 0.17$\pm$0.02 | 0.18$\pm$0.02 |  0.18$\pm$0.02 $\circ$ $\circ$ | 0.19$\pm$0.02 | 0.2$\pm$0.04
41144 |  0.27$\pm$0.01 $\circ$ $\circ$ | 0.19$\pm$0.02 $\circ$ $\circ$ | 0.21$\pm$0.05 | 0.09$\pm$0.01 |  0.1$\pm$0.02 $\dagger$ | 0.24$\pm$0.02 | 0.11$\pm$0.02
41145 | 0.29$\pm$0.03 $\circ$ $\circ$ | 0.26$\pm$0.02 $\circ$ $\circ$ | 0.25$\pm$0.01 | 0.16$\pm$0.03 |  0.18$\pm$0.01 $\dagger$ $\dagger$ | 0.25$\pm$0.02 | 0.24$\pm$0.01
41146 |  0.04$\pm$0.0 $\circ$ $\circ$ | 0.07$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.04$\pm$0.01 |  0.05$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.04$\pm$0.01
41147 | nan $\circ$ $\circ$ |  0.32$\pm$0.01 $\circ$ $\circ$ | 0.33$\pm$0.01 | 0.32$\pm$0.0 | nan $\circ$ $\circ$ | 0.32$\pm$0.0 | nan
41150 |  0.07$\pm$0.0 $\circ$ $\circ$ |  0.08$\pm$0.01 $\circ$ $\circ$ | 0.09$\pm$0.0 | 0.08$\pm$0.01 |  0.06$\pm$0.0 $\circ$ $\circ$ | 0.07$\pm$0.0 | 0.06$\pm$0.0
41156 |  0.15$\pm$0.03 $\circ$ $\circ$ |  0.15$\pm$0.02 $\circ$ $\circ$ | 0.15$\pm$0.01 | 0.15$\pm$0.01 |  0.14$\pm$0.01 $\circ$ $\circ$ | 0.14$\pm$0.01 | 0.14$\pm$0.01
41157 | nan $\circ$ $\circ$ | 0.21$\pm$0.14 $\circ$ | 0.15$\pm$0.09 | 0.22$\pm$0.11 | 0.24$\pm$0.09 $\circ$ | 0.11$\pm$0.09 | 0.26$\pm$0.15
41158 |  0.07$\pm$0.01 $\circ$ $\bullet$ |  0.07$\pm$0.01 $\circ$ $\bullet$ | 0.07$\pm$0.04 | 0.19$\pm$0.05 |  0.05$\pm$0.01 $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01
41159 | nan $\circ$ $\circ$ |  0.2$\pm$0.01 $\circ$ $\circ$ | 0.21$\pm$0.01 | 0.18$\pm$0.03 |  0.17$\pm$0.01 $\circ$ $\circ$ | 0.18$\pm$0.01 | nan
41162 |  0.1$\pm$0.0 $\circ$ $\circ$ |  0.1$\pm$0.0 $\circ$ $\bullet$ | 0.1$\pm$0.0 | 0.22$\pm$0.06 |  0.1$\pm$0.0 $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0
41163 |  0.08$\pm$0.0 $\circ$ $\circ$ |  0.03$\pm$0.0 $\circ$ $\circ$ | 0.03$\pm$0.0 | 0.02$\pm$0.01 |  0.01$\pm$0.0 $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0
41164 | 0.35$\pm$0.01 $\circ$ $\circ$ | 0.32$\pm$0.02 $\circ$ | 0.3$\pm$0.01 | 0.33$\pm$0.02 | 0.31$\pm$0.02 $\circ$ $\circ$ | 0.29$\pm$0.01 | 0.29$\pm$0.01
41165 | nan $\circ$ $\circ$ |  0.62$\pm$0.02 $\circ$ $\circ$ | 0.6$\pm$0.03 | nan |  0.55$\pm$0.02 $\circ$ $\circ$ | 0.55$\pm$0.01 | 0.56$\pm$0.01
41166 |  0.36$\pm$0.03 $\circ$ $\circ$ | 0.35$\pm$0.01 $\circ$ | 0.3$\pm$0.01 | 0.34$\pm$0.1 |  0.28$\pm$0.0 $\dagger$ $\dagger$ | 0.33$\pm$0.0 | 0.32$\pm$0.0
41167 | nan $\circ$ $\circ$ | 0.96$\pm$0.0 $\circ$ $\circ$ | 0.12$\pm$0.07 | 0.09$\pm$0.0 | nan $\circ$ $\circ$ | 0.31$\pm$0.0 | nan
41168 |  0.34$\pm$0.01 $\circ$ $\circ$ |  0.3$\pm$0.0 $\bullet$ | 0.34$\pm$0.01 | 0.29$\pm$0.01 |  0.27$\pm$0.0 $\bullet$ $\bullet$ | 0.29$\pm$0.01 | 0.29$\pm$0.0
41169 |  0.71$\pm$0.0 $\circ$ $\circ$ |  0.72$\pm$0.0 $\circ$ $\bullet$ | 0.72$\pm$0.0 | 0.75$\pm$0.01 | 0.69$\pm$0.01 $\circ$ $\circ$ | 0.65$\pm$0.0 | 0.64$\pm$0.0
42732 | nan $\circ$ $\circ$ |  0.12$\pm$0.0 $\circ$ $\circ$ | 0.12$\pm$0.0 | 0.12$\pm$0.0 |  0.12$\pm$0.0 $\circ$ $\circ$ | 0.12$\pm$0.0 | 0.12$\pm$0.0
42733 |  0.16$\pm$0.0 $\circ$ $\circ$ |  0.17$\pm$0.02 $\circ$ $\bullet$ | 0.17$\pm$0.0 | 0.25$\pm$0.03 |  0.17$\pm$0.0 $\circ$ $\circ$ | 0.17$\pm$0.0 | 0.17$\pm$0.0

The high level results of the experiments are shown in Table 1. The reported
metric is the _error rate_. For each dataset and each approach, we report the
_trimmed_ mean (10% trimmed) with standard deviation. Nan entries are caused
by memory overflows. We refrain from the now somewhat common practice of
reporting average ranks, because, in our view, those obscure a lot of
important details in the comparison, e.g. on _which_ (and _how many_) datasets
which algorithm is better than another and by which _margin_.

The formatting semantics of the table is as follows. The best entries (w.r.t
to the trimmed mean) per machine learning library are formatted in bold, and
those whose result distribution cannot be said to be statistically different
(according to a Wilcoxcon signed rank test with confidence 0.05) or where the
performance difference is “irrelevant” ($<0.01$) are underlined. With respect
to the latter one we of course understand that in some cases such marginal
difference _can_ be relevant, but here we still treat them as equally good to
give more significance to the symbols used to denote improvements. For each of
the state-of-the-art techniques, we use the $\bullet$ or $\circ$ symbol to
indicate a score that is “substantially” better or worse than the one obtained
with Naive AutoML. The symbols are used twice, once to compare against the
“primitive” profile and once comparing against the “full” profile. As above,
substantial here means that it is not only statistically significant but also
that the absolute improvement is at least 0.01. The symbol $\dagger$ is used
instead of a $\bullet$ in cases in which auto-sklearn constructed pipelines
with feature transformation algorithms not supported in the algorithm scheme
of Naive AutoML; examples are the PCA or feature encodings based on trees.
Based on these results, we can now answer the three research questions above.

####  5.3.1 RQ 1: Performance of State-Of-The-Art Approaches over Naive AutoML

When asking how much the currently best AutoML approaches can improve over the
Naive AutoML baseline, the answer contained in Table 1 is, put mildly,
surprising. While one would expect Naive AutoML to be frequently outperformed
by the other approaches, we cannot observe any such type of dominance. In
fact, there are even quite some datasets on which Naive AutoML performs
_better_.

What is more, the above results seem to contradict the first (and only)
comparison of an AutoML tool against a simulated data scientist presented in
[28]. In that paper, Auto-WEKA systematically outperforms the “Ex-def”
baseline, which here corresponds to the performance in the “primitive” column.
On _no_ dataset is Auto-WEKA better than this baseline and is even sometimes
outperformed by it.

This quite striking observation can also be put into words by saying that if
we run a simple for loop over the possible base learners (using default
parameters and ignoring meta-learners or any kind of feature transformation),
then we obtain an equal or better performance than Auto-WEKA, ML-Plan, or
auto-sklearn in more than 90% of the cases. This somewhat shocking observation
alone strongly jeopardizes the justification to apply resource-intense AutoML
tools. As shown in the appendix, the results for the “primitive” solution
arrive often in the range of some few seconds or at least some minutes. So
they are cheap to obtain, and other approaches rarely can improve upon it even
when run for an hour or more. This means that the least we should expect from
evaluations of new AutoML approaches is that they come back to compare against
the “Ex-def” baseline.

When considering the “full” baseline, the situation looks similar. Clearly,
since “primitive” is already such a strong baseline, we would not expect the
“full” baseline to improve upon that too often. However, in some cases, the
“full” baseline performs better than the “primitive” profile. This can almost
always be attributed to advantages gained in the feature scaling or feature
selection. Maybe surprisingly, sometimes the opposite is the case and the
“full” profile performs worse than the “primitive” one. We discuss this in
more detail in Sec. 5.3.2 and 5.3.3, but the short explanation is that “full”
can be weaker than “primitive” because it includes the _validation_ stage. The
latter implies that less data is used to find a model, which is a drawback in
some situations.

In our view, there are two possible explanations for the missing superiority
of the black-box optimization techniques. The first is simply that the global
optimum _is_ in fact obtained by choosing the best algorithm without any
further optimizations. That is, there is simply no _potential_ for
optimization. On some datasets (with almost perfect performance) this is
clearly the case. The second is that there _maybe_ is potential for
optimization, but the resource limitations impede that the black-box
approaches can develop their full potential. Theoretically, all of the black-
box approaches converge to the globally optimal solution. However, only a tiny
fraction of the search space can be examined in any reasonable timeout, and
the number of evaluations that can be made in that time are not enough to
learn enough about the performance landscape to steer the search process in a
meaningful way. In other words, the black-box optimizers _might_ be better in
theory and in the limit, but the limitations imposed by the time constraints
impede that any of the algorithms substantially leaves the exploration phase
and can actually exploit the model it learnt.

####  5.3.2 RQ 2: Individual Merits of the Optimization Stages.

To answer the question which of the stages is responsible for improvements
over the “primitive” profile, we create a tournament table. The appendix
contains the full result tables for the different stages. For each stage, we
check how often a substantial improvement over the primitive stage is obtained
(win), how often this degree of improvement can be achieved _only_ with this
stage for the respective dataset (unique win), how often a substantial
degradation can be observed (loss), and how often there is just no change
(draw).

| WEKA backend | scikit-learn backend
---|---|---
| filtering | meta | tuning | validation | scaling | filtering | meta | tuning | validation
wins | 6 | 4 | 1 | 4 | 7 | 2 | 2 | 3 | 1
unique wins | 5 | 1 | 0 | 3 | 7 | 2 | 0 | 0 | 1
losses | 2 | 5 | 4 | 4 | 1 | 1 | 1 | 2 | 1
draws | 53 | 52 | 56 | 53 | 53 | 58 | 58 | 56 | 59

This summary provides several interesting insights. The first observation is
that, in the majority of cases, none of the stages has any effect at all onto
the overall result. Even in the case of _scaling_ in scikit-learn, which
presents the highest number of changes compared to the _primitive_ setup, less
than 15% of the cases exhibit a substantial change in the results. However, we
can also observe that at least there is the potential to improve upon the
probing stage in some cases.

What is also interesting is that many of the improvements are _unique_ to one
of the techniques. For example, the scaling stage in scikit-learn achieved
wins over the probing stage on 7 datasets, and on all of those datasets,
_only_ the scaling stage was able to improve upon the probing stage.
Summarizing then the row of unique wins, we can assert that out of the 61
examined datasets, we could improve upon 9 datasets with _exactly_ one of the
stages in WEKA and on 10 in the case of scikit-learn. This result highlights
the necessity of complementary stages, and observing the fact that we left out
a stage for feature transformation that was responsible for even better
performance in auto-sklearn (the $\dagger$ cases), this effect can be expected
to be even bigger when more types of algorithms are considered.

####  5.3.3 RQ 3: Synergies Between Stages

We analyze this question in a similar fashion as the previous one via a
tournament table. We now not consider only a single stage but a _range_ of
stages (starting with the probing stage and including _all_ stages up to the
one of interest). For example, the stage range defined by _tuning_ include (i)
probing, (ii) scaling, (iii) filtering, (iv) meta, and (v) tuning. From this
logic, there are four relevant stage ranges; the scaling stage range is the
same as reported above and is hence omitted. For each such range of stages, we
now count the number of datasets on which there was an improvement over the
probing stage that was _better_ than the improvement achieved by _any_ of the
stage contained in the range alone (reported above).

| WEKA backend | scikit-learn backend |
---|---|---|---
| meta | tuning | validation | filtering | meta | tuning | validation
wins | 0 | 0 | 0 | 0 | 2 | 0 | 0
losses | 0 | 0 | 0 | 2 | 4 | 7 | 8
draws | 61 | 61 | 61 | 59 | 55 | 54 | 53

The results show that there are _no_ (positive) synergies between the stages.
While this may seem a bit natural for the filtering stage range (because up to
there, there is essentially no communication between the stages), it may be
surprising for the other stages. The absence of synergies between stages in
_this_ evaluation can be seen as an _indicator_ that the naivity assumption is
indeed justified. However and once more, we do not even claim that the naivity
assumptions are justified in general but only aim at giving evidence that in
_many_ situations, the results obtained with a naive approach are competitive.
In our view, the performance of the state-of-the-art tools is not particularly
challenging this simple view.

Note that the above results do of course _not_ mean that we can run Naive
AutoML with just one of the stages. What we learn is that if there is an
improvement over the probing stage, then this improvement can be attributed to
one of the stages. However, we do not know in advance which of the stages that
could be. Since most stages imply the same number of deteriorations as of
improvements, it seems indeed best to only apply the scaling and filtering
stage and omit meta-learners, tuning and validation.

In this whole evaluation, the effect of the validation stage is quite
disappointing. It seems natural to adscribe the deteriorations in the earlier
stages to an over-fitting effect, and one would hope to combat this over-
fitting in the validation stage. Apparently, this does not work in the desired
way. A nearby explanation is that the reduction in data available for the
optimization process misleads the optimization and proposes sub-optimal
models, and this effect cannot be compensated by validation later. However,
there might be also other reasons, and studying this effect in more detail may
be interesting future work.

##  6 Conclusion

This paper proposes Naive AutoML, a slightly more sophisticated baseline for
AutoML than proposed with “Ex-def” in [28]. Instead of searching a complex
pipeline search space with a black-box optimizer, Naive AutoML imitates the
sequential workflow conducted by a data scientist, which _implicitly_ defines
a solution pipeline. We empirically demonstrate that state-of-the-art tools
are not able to substantially outperform this baseline (sometimes the contrary
is true). Given the increasing demand of transparent and understandable AutoML
[31, 30, 5], this result highly jeopardizes the approach of using black-box
optimization and favors the idea of imitating the data scientist’s workflow.
Producing competitive results, this process is not only much more transparent
and understandable to the expert—and hence allows better interaction between
the expert and the machine—, but also is the approach much more flexible,
because all modifications can be directly realized in the stage
implementations instead of having it to be injected into the solver through
the problem encoding.

Our suggestions for future work go hence also into the direction of extending
and improving the naive approach instead of spending more time on enhancements
of black-box optimization approaches. Besides the obvious option to consider
other stage schemes and more complex pipelines (with tree-shaped pre-
processing), interesting future work lies in the detection of stage
saturation: When can we be sure that a stage will not improve upon the
currently best known solution? Also, creating a more interactive version of
Naive AutoML in which the expert obtains visual summaries of what has been
done and with the option to stop a particular stage and pass to the next seem
highly relevant.

## References

  * [1] Stephen Boyd, Neal Parikh and Eric Chu  “Distributed optimization and statistical learning via the alternating direction method of multipliers”  Now Publishers Inc, 2011
  * [2] Leo Breiman  “Bagging predictors”  In _Machine learning_ 24 Springer, 1996
  * [3] Boyuan Chen et al.  “Autostacker: A compositional evolutionary learning system”  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , 2018, pp. 402–409
  * [4] Anamaria Crisan and Brittany Fiore-Gartland  “Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop”  In _arXiv preprint arXiv:2101.04296_ , 2021
  * [5] Jaimie Drozdal et al.  “Trust in automl: Exploring information needs for establishing trust in automated machine learning systems”  In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ , 2020, pp. 297–307
  * [6] Matthias Feurer et al.  “Efficient and robust automated machine learning”  In _Advances in Neural Information Processing Systems_ , 2015, pp. 2962–2970
  * [7] Y. Freund and R.. Schapire  “Large margin classification using the perceptron algorithm”  In _11th Annual Conference on Computational Learning Theory_ New York, NY: ACM Press, 1998, pp. 209–217
  * [8] Nicolo Fusi, Rishit Sheth and Melih Elibol  “Probabilistic matrix factorization for automated machine learning”  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 3352–3361
  * [9] Pieter Gijsbers et al.  “An open source automl benchmark”  In _arXiv preprint arXiv:1907.00909_ , 2019
  * [10] Mark Hall et al.  “The WEKA data mining software: an update”  In _ACM SIGKDD Explorations_ 11, 2009
  * [11] Mark Andrew Hall  “Correlation-based feature selection for machine learning”  University of Waikato Hamilton, 1999
  * [12] Frank Hutter, Holger H Hoos and Kevin Leyton-Brown  “Sequential Model-Based Optimization for General Algorithm Configuration.”  In _LION_ 5 Springer, 2011, pp. 507–523
  * [13] J Kietz, Floarea Serban, Abraham Bernstein and Simon Fischer  “Towards cooperative planning of data mining workflows”  In _Proceedings of the Third Generation Data Mining Workshop at the 2009 European Conference on Machine Learning_ , 2009, pp. 1–12  Citeseer
  * [14] Jörg-Uwe Kietz, Floarea Serban, Abraham Bernstein and Simon Fischer  “Designing KDD-Workflows via HTN-Planning for Intelligent Discovery Assistance”  In _5th Planning to Learn Workshop WS28 at ECAI 2012_ , 2012, pp. 10
  * [15] Ron Kohavi and George H John  “Wrappers for feature subset selection”  In _Artificial intelligence_ 97.1-2 Elsevier, 1997, pp. 273–324
  * [16] Lars Kotthoff et al.  “Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA”  In _The Journal of Machine Learning Research_ 18.1 JMLR. org, 2017, pp. 826–830
  * [17] Lisha Li et al.  “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”  In _J. Mach. Learn. Res._ 18, 2017, pp. 185:1–185:52  URL: http://jmlr.org/papers/v18/16-558.html
  * [18] Sijia Liu et al.  “An ADMM based framework for automl pipeline configuration”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 34.04, 2020, pp. 4892–4899
  * [19] F. Mohr, M. Wever, A. Tornede and E. Hüllermeier  “Predicting Machine Learning Pipeline Runtimes in the Context of Automated Machine Learning”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021, pp. 1–1  DOI: 10.1109/TPAMI.2021.3056950
  * [20] Felix Mohr, Marcel Wever and Eyke Hüllermeier  “ML-Plan: Automated machine learning via hierarchical planning”  In _Machine Learning_ 107.8 Springer, 2018, pp. 1495–1515
  * [21] Phong Nguyen, Melanie Hilario and Alexandros Kalousis  “Using meta-mining to support data mining workflow planning and optimization”  In _Journal of Artificial Intelligence Research_ 51, 2014, pp. 605–644
  * [22] Phong Nguyen, Alexandros Kalousis and Melanie Hilario  “A meta-mining infrastructure to support kd workflow optimization”  In _Proceedings of the PlanSoKD-11 Workshop at ECML/PKDD_ , 2011, pp. 1–10
  * [23] Phong Nguyen, Alexandros Kalousis and Melanie Hilario  “Experimental evaluation of the e-lico meta-miner”  In _5th planning to learn workshop WS28 at ECAI_ , 2012, pp. 18–19
  * [24] Randal S Olson and Jason H Moore  “TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning”  In _Workshop on Automatic Machine Learning_ , 2016, pp. 66–74
  * [25] Fabian Pedregosa et al.  “Scikit-learn: Machine learning in Python”  In _the Journal of machine Learning research_ 12 JMLR. org, 2011, pp. 2825–2830
  * [26] Herilalaina Rakotoarison, Marc Schoenauer and Michèle Sebag  “Automated Machine Learning with Monte-Carlo Tree Search”  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ ijcai.org, 2019, pp. 3296–3303  DOI: 10.24963/ijcai.2019/457
  * [27] Alex GC Sá, Walter José GS Pinto, Luiz Otavio VB Oliveira and Gisele L Pappa  “RECIPE: a grammar-based framework for automatically evolving classification pipelines”  In _European Conference on Genetic Programming_ , 2017, pp. 246–261  Springer
  * [28] Chris Thornton, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown  “Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms”  In _The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA_ , 2013, pp. 847–855
  * [29] Joaquin Vanschoren, Jan N. Rijn, Bernd Bischl and Luis Torgo  “OpenML: Networked Science in Machine Learning”  In _SIGKDD Explorations_ 15.2 New York, NY, USA: ACM, 2013, pp. 49–60  DOI: 10.1145/2641190.2641198
  * [30] Dakuo Wang et al.  “AutoDS: Towards Human-Centered Automation of Data Science”  In _arXiv preprint arXiv:2101.05273_ , 2021
  * [31] Dakuo Wang et al.  “Human-AI collaboration in data science: Exploring data scientists’ perceptions of automated AI”  In _Proceedings of the ACM on Human-Computer Interaction_ 3.CSCW ACM New York, NY, USA, 2019, pp. 1–24
  * [32] David H. Wolpert  “Stacked generalization”  In _Neural Networks_ 5 Pergamon Press, 1992, pp. 241–259
  * [33] Chengrun Yang, Yuji Akimoto, Dae Won Kim and Madeleine Udell  “OBOE: Collaborative filtering for AutoML model selection”  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 1173–1183

## Appendix

##  Appendix A Datasets

All datasets are available via the openml.org platform [29].

openmlid | name | instances | features | classes
---|---|---|---|---
3 | kr-vs-kp | 3196 | 37 | 2
12 | mfeat-factors | 2000 | 217 | 10
54 | vehicle | 846 | 19 | 4
181 | yeast | 1484 | 9 | 10
1049 | pc4 | 1458 | 38 | 2
1067 | kc1 | 2109 | 22 | 2
1111 | KDDCup09_appetency | 50000 | 231 | 2
1457 | amazon-commerce-reviews | 1500 | 10001 | 50
1461 | bank-marketing | 45211 | 17 | 2
1464 | blood-transfusion-service-center | 748 | 5 | 2
1468 | cnae-9 | 1080 | 857 | 9
1475 | first-order-theorem-proving | 6118 | 52 | 6
1485 | madelon | 2600 | 501 | 2
1486 | nomao | 34465 | 119 | 2
1487 | ozone-level-8hr | 2534 | 73 | 2
1489 | phoneme | 5404 | 6 | 2
1494 | qsar-biodeg | 1055 | 42 | 2
1515 | micro-mass | 571 | 1301 | 20
1590 | adult | 48842 | 15 | 2
4134 | Bioresponse | 3751 | 1777 | 2
4135 | Amazon_employee_access | 32769 | 10 | 2
4534 | PhishingWebsites | 11055 | 31 | 2
4538 | GesturePhaseSegmentationProcessed | 9873 | 33 | 5
4541 | Diabetes130US | 101766 | 50 | 3
23512 | higgs | 98050 | 29 | 2
23517 | numerai28.6 | 96320 | 22 | 2
40498 | wine-quality-white | 4898 | 12 | 7
40668 | connect-4 | 67557 | 43 | 3
40670 | dna | 3186 | 181 | 3
40685 | shuttle | 58000 | 10 | 7
40701 | churn | 5000 | 21 | 2
40900 | Satellite | 5100 | 37 | 2
40975 | car | 1728 | 7 | 4
40978 | Internet-Advertisements | 3279 | 1559 | 2
40981 | Australian | 690 | 15 | 2
40982 | steel-plates-fault | 1941 | 28 | 7
40983 | wilt | 4839 | 6 | 2
40984 | segment | 2310 | 20 | 7
41027 | jungle_chess_2pcs_raw_endgame_complete | 44819 | 7 | 3
41138 | APSFailure | 76000 | 171 | 2
41142 | christine | 5418 | 1637 | 2
41143 | jasmine | 2984 | 145 | 2
41144 | madeline | 3140 | 260 | 2
41145 | philippine | 5832 | 309 | 2
41146 | sylvine | 5124 | 21 | 2
41147 | albert | 425240 | 79 | 2
41150 | MiniBooNE | 130064 | 51 | 2
41156 | ada | 4147 | 49 | 2
41157 | arcene | 100 | 10001 | 2
41158 | gina | 3153 | 971 | 2
41159 | guillermo | 20000 | 4297 | 2
41162 | kick | 72983 | 33 | 2
41163 | dilbert | 10000 | 2001 | 5
41164 | fabert | 8237 | 801 | 7
41165 | robert | 10000 | 7201 | 10
41166 | volkert | 58310 | 181 | 10
41167 | dionis | 416188 | 61 | 355
41168 | jannis | 83733 | 55 | 4
41169 | helena | 65196 | 28 | 100
42732 | sf-police-incidents | 2215023 | 10 | 2
42733 | Click_prediction_small | 39948 | 12 | 2

Table 2: Overview of datasets used in the evaluation.

##  Appendix B Result Tables for Naive AutoML

The following tables contain the results for the different stages and the
respective comparison to the state-of-the-art approaches.

openmlid | auto-weka | mlplan | primitive | filtering | meta | tuning | validation
---|---|---|---|---|---|---|---
3 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.01$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0
12 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01
54 |  0.18$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.18$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.18$\pm$0.02 | 0.17$\pm$0.02 | 0.18$\pm$0.04
181 |  0.39$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.39$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.38$\pm$0.03 | 0.38$\pm$0.02 | 0.38$\pm$0.03 | 0.38$\pm$0.02 | 0.38$\pm$0.03
1049 |  0.1$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.09$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.02 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.02
1067 |  0.15$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.15$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.02 | 0.14$\pm$0.02 | 0.15$\pm$0.02 | 0.15$\pm$0.02 | 0.15$\pm$0.01
1111 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.01
1457 |  0.29$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.25$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.25$\pm$0.03 | 0.25$\pm$0.04 | 0.25$\pm$0.03 | 0.25$\pm$0.03 | 0.25$\pm$0.04
1461 |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.1$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.09$\pm$0.0 | 0.1$\pm$0.0
1464 |  0.22$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.24$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.04 | 0.22$\pm$0.03 | 0.21$\pm$0.04 | 0.22$\pm$0.04 | 0.22$\pm$0.03
1468 |  0.06$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.05$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.02
1475 | 0.38$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.36$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02
1485 | 0.39$\pm$0.1 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.23$\pm$0.05 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.23$\pm$0.02 | 0.11$\pm$0.02 | 0.2$\pm$0.03 | 0.24$\pm$0.04 | 0.24$\pm$0.03
1486 |  0.03$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.0 | 0.03$\pm$0.0 | 0.03$\pm$0.0 | 0.03$\pm$0.0 | 0.03$\pm$0.0
1487 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.06$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
1489 |  0.09$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.09$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01
1494 |  0.13$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.13$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.02 | 0.13$\pm$0.03 | 0.13$\pm$0.03 | 0.12$\pm$0.04 | 0.13$\pm$0.03
1515 |  0.09$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.13$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.04 | 0.13$\pm$0.04 | 0.14$\pm$0.04 | 0.13$\pm$0.04 | 0.14$\pm$0.04
1590 |  0.14$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.14$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0
4134 |  0.23$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.2$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.2$\pm$0.01 | 0.2$\pm$0.02 | 0.2$\pm$0.01 | 0.2$\pm$0.01 | 0.2$\pm$0.01
4135 |  0.05$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.06$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0
4534 |  0.03$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.01 | 0.02$\pm$0.0 | 0.02$\pm$0.0
4538 |  0.32$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.32$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.01 | 0.32$\pm$0.01 | 0.31$\pm$0.01 | 0.32$\pm$0.01 | 0.32$\pm$0.01
4541 |  0.43$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.42$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.42$\pm$0.01 | 0.42$\pm$0.01 | 0.42$\pm$0.01 | 0.42$\pm$0.01 | 0.42$\pm$0.01
23512 | 0.31$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.31$\pm$0.02 | 0.31$\pm$0.01 | 0.31$\pm$0.02 | 0.31$\pm$0.01 | 0.28$\pm$0.01
23517 |  0.48$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.48$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.48$\pm$0.0 | 0.49$\pm$0.0 | 0.48$\pm$0.0 | 0.48$\pm$0.0 | 0.48$\pm$0.01
40498 |  0.31$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.31$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01
40668 |  0.03$\pm$0.0 $\circ$ $\circ$ $\bullet$ $\circ$ $\bullet$ | 0.19$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.11 | 0.18$\pm$0.12 | 0.26$\pm$0.08 | 0.21$\pm$0.11 | 0.24$\pm$0.1
40670 |  0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.03$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01
40685 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0
40701 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
40900 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
40975 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.0$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.01 | 0.0$\pm$0.01 | 0.0$\pm$0.01 | 0.0$\pm$0.01 | 0.0$\pm$0.01
40978 |  0.02$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.03$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01
40981 | 0.14$\pm$0.05 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.14$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.03 | 0.12$\pm$0.04 | 0.13$\pm$0.03 | 0.14$\pm$0.02 | 0.14$\pm$0.02
40982 |  0.19$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.2$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.2$\pm$0.03 | 0.2$\pm$0.03 | 0.19$\pm$0.02 | 0.2$\pm$0.02 | 0.2$\pm$0.03
40983 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0
40984 |  0.02$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01
41027 | 0.16$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.01 | 0.09$\pm$0.0 | 0.09$\pm$0.0
41138 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.01 | 0.01$\pm$0.0
41142 | 0.29$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.28$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.27$\pm$0.02 | 0.25$\pm$0.02 | 0.27$\pm$0.02 | 0.27$\pm$0.02 | 0.27$\pm$0.02
41143 |  0.16$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.17$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.17$\pm$0.02
41144 |  0.27$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.19$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.05 | 0.11$\pm$0.01 | 0.18$\pm$0.03 | 0.2$\pm$0.03 | 0.2$\pm$0.04
41145 | 0.29$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.26$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.25$\pm$0.01 | 0.13$\pm$0.03 | 0.24$\pm$0.01 | 0.25$\pm$0.01 | 0.26$\pm$0.02
41146 |  0.04$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.07$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.06$\pm$0.01 | 0.06$\pm$0.01
41147 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.32$\pm$0.01 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.33$\pm$0.01 | 0.33$\pm$0.0 | 0.41$\pm$0.08 | 0.39$\pm$0.07 | 0.36$\pm$0.06
41150 |  0.07$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.08$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0
41156 |  0.15$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.15$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.01 | 0.15$\pm$0.01 | 0.15$\pm$0.01 | 0.15$\pm$0.02 | 0.15$\pm$0.01
41157 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.14 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.09 | 0.3$\pm$0.14 | 0.16$\pm$0.21 | 0.18$\pm$0.11 | 0.24$\pm$0.14
41158 |  0.07$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.07$\pm$0.01 $\circ$ $\circ$ $\circ$ $\bullet$ | 0.07$\pm$0.04 | 0.06$\pm$0.03 | 0.1$\pm$0.06 | 0.1$\pm$0.06 | 0.07$\pm$0.04
41159 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.2$\pm$0.01 $\circ$ $\circ$ $\circ$ $\bullet$ | 0.21$\pm$0.01 | 0.17$\pm$0.01 | 0.21$\pm$0.01 | 0.22$\pm$0.01 | 0.21$\pm$0.01
41162 |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0
41163 |  0.08$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.03$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.0 | 0.02$\pm$0.01 | 0.03$\pm$0.0 | 0.03$\pm$0.0 | 0.03$\pm$0.0
41164 | 0.35$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.31$\pm$0.01 | 0.31$\pm$0.01 | 0.31$\pm$0.01 | 0.31$\pm$0.01
41165 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.62$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.6$\pm$0.03 | 0.61$\pm$0.03 | 0.59$\pm$0.02 | 0.6$\pm$0.02 | 0.58$\pm$0.02
41166 |  0.36$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.35$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01
41167 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.96$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.07 | 0.09$\pm$0.0 | 0.11$\pm$0.05 | 0.09$\pm$0.0 | 0.11$\pm$0.05
41168 |  0.34$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.3$\pm$0.0 $\bullet$ $\bullet$ $\bullet$ $\bullet$ | 0.34$\pm$0.01 | 0.34$\pm$0.02 | 0.34$\pm$0.01 | 0.34$\pm$0.01 | 0.29$\pm$0.02
41169 |  0.71$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.72$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.72$\pm$0.0 | 0.72$\pm$0.0 | 0.72$\pm$0.01 | 0.72$\pm$0.01 | 0.72$\pm$0.01
42732 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.12$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0
42733 |  0.16$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.17$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.16$\pm$0.0 | 0.17$\pm$0.0

Table 3: Stage-Wise Results in Isolated Configuration for WEKA.

openmlid | asklearn | primitive | scaling | filtering | meta | tuning | validation
---|---|---|---|---|---|---|---
3 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | nan | 0.0$\pm$0.0 | 0.0$\pm$0.01
12 |  0.03$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.02 | 0.02$\pm$0.01 | 0.02$\pm$0.02 | 0.02$\pm$0.02 | 0.02$\pm$0.01 | 0.02$\pm$0.02
54 | 0.19$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.16$\pm$0.03 | 0.16$\pm$0.03 | 0.16$\pm$0.03 | 0.16$\pm$0.03 | 0.18$\pm$0.03 | 0.16$\pm$0.03
181 |  0.39$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.38$\pm$0.03 | 0.39$\pm$0.05 | 0.39$\pm$0.04 | 0.38$\pm$0.03 | 0.39$\pm$0.04 | 0.39$\pm$0.04
1049 |  0.09$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.02 | 0.09$\pm$0.02 | 0.08$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.08$\pm$0.01
1067 |  0.14$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.02 | 0.14$\pm$0.01 | 0.14$\pm$0.02 | 0.14$\pm$0.02 | 0.15$\pm$0.01 | 0.15$\pm$0.01
1111 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | nan | 0.02$\pm$0.0 | 0.02$\pm$0.0
1457 | 0.2$\pm$0.02 $\bullet$ $\circ$ $\bullet$ $\circ$ $\bullet$ $\bullet$ | 0.23$\pm$0.04 | 0.15$\pm$0.03 | 0.23$\pm$0.04 | 0.22$\pm$0.02 | 0.21$\pm$0.06 | 0.23$\pm$0.04
1461 |  0.11$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.11$\pm$0.0 | 0.11$\pm$0.0 | 0.11$\pm$0.0 | nan | 0.11$\pm$0.0 | 0.11$\pm$0.0
1464 |  0.22$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.22$\pm$0.02 | 0.21$\pm$0.02 | 0.23$\pm$0.01 | 0.22$\pm$0.02 | 0.22$\pm$0.03 | 0.23$\pm$0.02
1468 |  0.05$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.02 | 0.04$\pm$0.01 | 0.05$\pm$0.01 | 0.04$\pm$0.02 | 0.04$\pm$0.01 | 0.04$\pm$0.01
1475 | 0.39$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02
1485 |  0.11$\pm$0.02 $\dagger$ $\dagger$ $\circ$ $\dagger$ $\dagger$ $\dagger$ | 0.26$\pm$0.03 | 0.26$\pm$0.03 | 0.12$\pm$0.02 | 0.21$\pm$0.03 | 0.25$\pm$0.03 | 0.27$\pm$0.03
1486 |  0.05$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | nan | 0.05$\pm$0.0 | 0.05$\pm$0.0
1487 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.06$\pm$0.01 | 0.06$\pm$0.01
1489 |  0.1$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01
1494 | 0.14$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.03 | 0.12$\pm$0.03 | 0.12$\pm$0.03 | 0.12$\pm$0.03 | 0.12$\pm$0.03 | 0.13$\pm$0.03
1515 |  0.07$\pm$0.03 $\bullet$ $\circ$ $\bullet$ $\bullet$ $\bullet$ $\bullet$ | 0.13$\pm$0.03 | 0.06$\pm$0.02 | 0.13$\pm$0.03 | 0.13$\pm$0.03 | 0.14$\pm$0.04 | 0.13$\pm$0.03
1590 |  0.16$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.16$\pm$0.0 | 0.16$\pm$0.0 | 0.16$\pm$0.0 | nan | 0.16$\pm$0.0 | 0.16$\pm$0.0
4134 |  0.2$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.19$\pm$0.02 | 0.19$\pm$0.02 | 0.19$\pm$0.02 | 0.19$\pm$0.02 | 0.2$\pm$0.02 | 0.2$\pm$0.02
4135 |  0.05$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | nan | 0.05$\pm$0.0 | 0.05$\pm$0.0
4534 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.0 | nan | 0.02$\pm$0.0 | 0.02$\pm$0.01
4538 | 0.31$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.29$\pm$0.01 | 0.3$\pm$0.01
4541 |  0.42$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.42$\pm$0.0 | 0.42$\pm$0.0 | 0.42$\pm$0.0 | nan | 0.42$\pm$0.0 | 0.42$\pm$0.0
23512 |  0.28$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.28$\pm$0.01 | 0.28$\pm$0.01 | 0.28$\pm$0.01 | 0.28$\pm$0.01 | 0.28$\pm$0.0 | 0.28$\pm$0.01
23517 |  0.48$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.48$\pm$0.0 | 0.48$\pm$0.0 | 0.48$\pm$0.01 | 0.48$\pm$0.01 | 0.48$\pm$0.0 | 0.48$\pm$0.0
40498 | 0.34$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.29$\pm$0.01 | 0.3$\pm$0.02 | 0.29$\pm$0.02 | 0.29$\pm$0.02 | 0.29$\pm$0.01 | 0.3$\pm$0.02
40668 |  0.17$\pm$0.06 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.26$\pm$0.0 | 0.26$\pm$0.0 | 0.26$\pm$0.06 | nan | 0.26$\pm$0.01 | 0.26$\pm$0.07
40670 |  0.03$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | nan | 0.03$\pm$0.01 | 0.03$\pm$0.01
40685 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0
40701 |  0.04$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | nan | 0.05$\pm$0.01 | 0.05$\pm$0.01
40900 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
40975 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.01 | 0.01$\pm$0.01 | 0.01$\pm$0.01 | nan | 0.0$\pm$0.0 | 0.01$\pm$0.01
40978 |  0.03$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | nan | 0.03$\pm$0.01 | 0.03$\pm$0.01
40981 |  0.12$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.03 | 0.13$\pm$0.04 | 0.13$\pm$0.03 | nan | 0.11$\pm$0.03 | 0.13$\pm$0.03
40982 |  0.21$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.02 | 0.22$\pm$0.02 | 0.21$\pm$0.02 | 0.21$\pm$0.03 | 0.21$\pm$0.03 | 0.2$\pm$0.03
40983 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.02$\pm$0.01 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.02$\pm$0.0
40984 |  0.02$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01
41027 |  0.07$\pm$0.01 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.16$\pm$0.01 | 0.14$\pm$0.0 | 0.16$\pm$0.01 | 0.16$\pm$0.01 | 0.15$\pm$0.01 | 0.16$\pm$0.01
41138 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
41142 |  0.27$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.28$\pm$0.02 | 0.27$\pm$0.02 | 0.27$\pm$0.02 | nan | 0.28$\pm$0.02 | 0.27$\pm$0.02
41143 |  0.18$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.19$\pm$0.02 | 0.19$\pm$0.03 | 0.18$\pm$0.02 | nan | 0.19$\pm$0.03 | 0.18$\pm$0.01
41144 |  0.1$\pm$0.02 $\dagger$ $\dagger$ $\circ$ $\dagger$ $\dagger$ $\dagger$ | 0.24$\pm$0.02 | 0.23$\pm$0.01 | 0.1$\pm$0.03 | 0.2$\pm$0.01 | 0.21$\pm$0.02 | 0.24$\pm$0.02
41145 |  0.18$\pm$0.01 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.25$\pm$0.02 | 0.25$\pm$0.01 | 0.24$\pm$0.01 | 0.24$\pm$0.01 | 0.25$\pm$0.02 | 0.24$\pm$0.01
41146 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
41147 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.0 | 0.32$\pm$0.0 | nan | nan | nan | 0.32$\pm$0.0
41150 |  0.06$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.07$\pm$0.0 | 0.06$\pm$0.0 | 0.07$\pm$0.0 | 0.07$\pm$0.0 | 0.07$\pm$0.0 | 0.07$\pm$0.0
41156 |  0.14$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.14$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.01 | 0.13$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.01
41157 | 0.24$\pm$0.09 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.11$\pm$0.09 | 0.15$\pm$0.08 | 0.22$\pm$0.12 | 0.12$\pm$0.09 | 0.14$\pm$0.11 | 0.11$\pm$0.07
41158 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.04$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
41159 |  0.17$\pm$0.01 $\circ$ $\circ$ $\circ$ $\dagger$ $\circ$ $\circ$ | 0.18$\pm$0.01 | 0.18$\pm$0.01 | 0.17$\pm$0.01 | 0.18$\pm$0.01 | 0.19$\pm$0.01 | 0.18$\pm$0.01
41162 |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | nan | 0.1$\pm$0.0 | 0.1$\pm$0.0
41163 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0
41164 | 0.31$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.3$\pm$0.01
41165 |  0.55$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.55$\pm$0.01 | 0.56$\pm$0.01 | 0.55$\pm$0.01 | 0.56$\pm$0.01 | 0.55$\pm$0.01 | 0.56$\pm$0.02
41166 |  0.28$\pm$0.0 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.33$\pm$0.0 | 0.32$\pm$0.0 | 0.33$\pm$0.0 | 0.33$\pm$0.01 | 0.33$\pm$0.01 | 0.33$\pm$0.0
41167 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.31$\pm$0.0 | nan | nan | nan | 0.24$\pm$0.01 | nan
41168 |  0.27$\pm$0.0 $\bullet$ $\bullet$ $\bullet$ $\bullet$ $\bullet$ $\bullet$ | 0.29$\pm$0.01 | 0.29$\pm$0.0 | 0.29$\pm$0.01 | 0.29$\pm$0.0 | 0.29$\pm$0.0 | 0.29$\pm$0.0
41169 | 0.69$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.65$\pm$0.0 | 0.64$\pm$0.0 | 0.65$\pm$0.0 | 0.65$\pm$0.0 | 0.65$\pm$0.0 | 0.64$\pm$0.0
42732 |  0.12$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0 | nan | 0.12$\pm$0.0 | 0.12$\pm$0.0
42733 |  0.17$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.17$\pm$0.0 | nan | 0.17$\pm$0.0 | 0.17$\pm$0.0

Table 4: Stage-Wise Results in Isolated Configuration for scikit-learn.

openmlid | auto-weka | mlplan | primitive | filtering | meta | tuning | full
---|---|---|---|---|---|---|---
3 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.01$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.01 | 0.0$\pm$0.0
12 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.02$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01
54 |  0.18$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.18$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.18$\pm$0.02 | 0.17$\pm$0.02 | 0.18$\pm$0.05
181 |  0.39$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.39$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.38$\pm$0.03 | 0.38$\pm$0.02 | 0.39$\pm$0.02 | 0.39$\pm$0.02 | 0.39$\pm$0.04
1049 |  0.1$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.09$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.02 | 0.09$\pm$0.01 | 0.09$\pm$0.02 | 0.09$\pm$0.02 | 0.09$\pm$0.02
1067 |  0.15$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.15$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.02 | 0.14$\pm$0.02 | 0.14$\pm$0.02 | 0.15$\pm$0.02 | 0.15$\pm$0.01
1111 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.02$\pm$0.0 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.04$\pm$0.0 | 0.04$\pm$0.0 | 0.04$\pm$0.0
1457 |  0.29$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.25$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.25$\pm$0.03 | 0.25$\pm$0.04 | 0.25$\pm$0.03 | 0.24$\pm$0.03 | 0.24$\pm$0.03
1461 |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.1$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0
1464 |  0.22$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.24$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.04 | 0.22$\pm$0.03 | 0.22$\pm$0.03 | 0.23$\pm$0.03 | 0.23$\pm$0.03
1468 |  0.06$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.05$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
1475 | 0.38$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.36$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.08 | 0.37$\pm$0.02
1485 | 0.39$\pm$0.1 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.23$\pm$0.05 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.23$\pm$0.02 | 0.11$\pm$0.02 | 0.11$\pm$0.01 | 0.11$\pm$0.01 | 0.11$\pm$0.02
1486 |  0.03$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.0 | 0.03$\pm$0.0 | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.03$\pm$0.0
1487 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.06$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.06$\pm$0.01 | 0.06$\pm$0.01 | 0.05$\pm$0.01
1489 |  0.09$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.09$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01
1494 |  0.13$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.13$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.02 | 0.13$\pm$0.03 | 0.14$\pm$0.03 | 0.13$\pm$0.03 | 0.12$\pm$0.03
1515 |  0.09$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.13$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.04 | 0.13$\pm$0.04 | 0.13$\pm$0.04 | 0.13$\pm$0.04 | 0.13$\pm$0.04
1590 |  0.14$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.14$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.01
4134 |  0.23$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.2$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.2$\pm$0.01 | 0.2$\pm$0.02 | 0.19$\pm$0.02 | 0.21$\pm$0.05 | 0.22$\pm$0.05
4135 |  0.05$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.06$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.08$\pm$0.03
4534 |  0.03$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.01 | 0.02$\pm$0.0 | 0.02$\pm$0.01
4538 |  0.32$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.32$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.01 | 0.32$\pm$0.01 | 0.32$\pm$0.01 | 0.32$\pm$0.01 | 0.31$\pm$0.01
4541 |  0.43$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.42$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.42$\pm$0.01 | 0.42$\pm$0.01 | 0.42$\pm$0.01 | 0.42$\pm$0.0 | 0.42$\pm$0.01
23512 | 0.31$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.31$\pm$0.02 | 0.31$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.28$\pm$0.01
23517 |  0.48$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.48$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.48$\pm$0.0 | 0.49$\pm$0.0 | 0.48$\pm$0.01 | 0.48$\pm$0.01 | 0.48$\pm$0.01
40498 |  0.31$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.31$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01
40668 |  0.03$\pm$0.0 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.19$\pm$0.01 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.21$\pm$0.11 | 0.18$\pm$0.12 | 0.27$\pm$0.0 | 0.27$\pm$0.0 | 0.27$\pm$0.0
40670 |  0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.02
40685 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0
40701 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
40900 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
40975 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.0$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.01 | 0.0$\pm$0.01 | 0.0$\pm$0.01 | 0.0$\pm$0.01 | 0.0$\pm$0.01
40978 |  0.02$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.03$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01
40981 | 0.14$\pm$0.05 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.14$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.03 | 0.12$\pm$0.04 | 0.13$\pm$0.03 | 0.15$\pm$0.03 | 0.13$\pm$0.03
40982 |  0.19$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.2$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.2$\pm$0.03 | 0.2$\pm$0.03 | 0.2$\pm$0.02 | 0.2$\pm$0.02 | 0.21$\pm$0.03
40983 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.01
40984 |  0.02$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.03$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01
41027 | 0.16$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.02
41138 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.01 | 0.01$\pm$0.0 | 0.01$\pm$0.0
41142 | 0.29$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.28$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.27$\pm$0.02 | 0.25$\pm$0.02 | 0.26$\pm$0.03 | 0.26$\pm$0.02 | 0.26$\pm$0.03
41143 |  0.16$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.17$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.17$\pm$0.02 | 0.18$\pm$0.02
41144 |  0.27$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.19$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.05 | 0.11$\pm$0.01 | 0.11$\pm$0.02 | 0.1$\pm$0.02 | 0.09$\pm$0.01
41145 | 0.29$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.26$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.25$\pm$0.01 | 0.13$\pm$0.03 | 0.14$\pm$0.03 | 0.13$\pm$0.03 | 0.16$\pm$0.03
41146 |  0.04$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.07$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01
41147 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.32$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.33$\pm$0.01 | 0.33$\pm$0.0 | 0.32$\pm$0.0 | 0.32$\pm$0.0 | 0.32$\pm$0.0
41150 |  0.07$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.08$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.08$\pm$0.01
41156 |  0.15$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.15$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.01 | 0.15$\pm$0.01 | 0.15$\pm$0.01 | 0.15$\pm$0.01 | 0.15$\pm$0.01
41157 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.14 $\circ$ $\circ$ $\bullet$ $\circ$ $\circ$ | 0.15$\pm$0.09 | 0.3$\pm$0.14 | 0.27$\pm$0.12 | 0.26$\pm$0.15 | 0.22$\pm$0.11
41158 |  0.07$\pm$0.01 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ |  0.07$\pm$0.01 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.07$\pm$0.04 | 0.06$\pm$0.03 | 0.19$\pm$0.02 | 0.19$\pm$0.02 | 0.19$\pm$0.05
41159 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.2$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ | 0.21$\pm$0.01 | 0.17$\pm$0.01 | 0.17$\pm$0.01 | 0.17$\pm$0.01 | 0.18$\pm$0.03
41162 |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.1$\pm$0.0 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.23$\pm$0.04 | 0.17$\pm$0.07 | 0.22$\pm$0.06
41163 |  0.08$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.03$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.0 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.03$\pm$0.0 | 0.02$\pm$0.01
41164 | 0.35$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.02 $\circ$ $\circ$ $\bullet$ $\bullet$ | 0.3$\pm$0.01 | 0.31$\pm$0.01 | 0.34$\pm$0.01 | 0.34$\pm$0.01 | 0.33$\pm$0.02
41165 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.62$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.6$\pm$0.03 | 0.61$\pm$0.03 | nan | nan | nan
41166 |  0.36$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.35$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.34$\pm$0.1
41167 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.96$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.07 | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0 | 0.09$\pm$0.0
41168 |  0.34$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.3$\pm$0.0 $\bullet$ $\bullet$ $\bullet$ $\bullet$ | 0.34$\pm$0.01 | 0.34$\pm$0.02 | 0.36$\pm$0.0 | 0.36$\pm$0.0 | 0.29$\pm$0.01
41169 |  0.71$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.72$\pm$0.0 $\circ$ $\circ$ $\bullet$ $\bullet$ $\bullet$ | 0.72$\pm$0.0 | 0.72$\pm$0.0 | 0.75$\pm$0.02 | 0.75$\pm$0.02 | 0.75$\pm$0.01
42732 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.12$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0
42733 |  0.16$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ |  0.17$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\bullet$ | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.25$\pm$0.03

Table 5: Stage-Wise Results in Monotone Configuration for WEKA.

openmlid | asklearn | primitive | scale | filtering | meta | tuning | full
---|---|---|---|---|---|---|---
3 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | nan | 0.0$\pm$0.0 | 0.0$\pm$0.0
12 |  0.03$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.02 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.02 | 0.02$\pm$0.01 | 0.03$\pm$0.02
54 | 0.19$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.16$\pm$0.03 | 0.16$\pm$0.03 | 0.16$\pm$0.03 | 0.16$\pm$0.03 | 0.17$\pm$0.03 | 0.17$\pm$0.03
181 |  0.39$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.38$\pm$0.03 | 0.39$\pm$0.05 | 0.39$\pm$0.04 | 0.39$\pm$0.02 | 0.38$\pm$0.03 | 0.37$\pm$0.05
1049 |  0.09$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.02 | 0.09$\pm$0.02 | 0.08$\pm$0.02 | 0.09$\pm$0.01 | 0.1$\pm$0.02 | 0.09$\pm$0.01
1067 |  0.14$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.15$\pm$0.02 | 0.14$\pm$0.01 | 0.15$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.02 | 0.15$\pm$0.01
1111 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | nan | 0.02$\pm$0.0 | 0.02$\pm$0.0
1457 | 0.2$\pm$0.02 $\bullet$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.23$\pm$0.04 | 0.15$\pm$0.03 | 0.15$\pm$0.03 | 0.15$\pm$0.03 | 0.15$\pm$0.03 | 0.16$\pm$0.03
1461 |  0.11$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.11$\pm$0.0 | 0.11$\pm$0.0 | 0.11$\pm$0.0 | nan | 0.11$\pm$0.0 | 0.11$\pm$0.0
1464 |  0.22$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.22$\pm$0.02 | 0.21$\pm$0.02 | 0.21$\pm$0.03 | 0.22$\pm$0.03 | 0.21$\pm$0.02 | 0.21$\pm$0.03
1468 |  0.05$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.04$\pm$0.02 | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.05$\pm$0.02 | 0.04$\pm$0.01 | 0.04$\pm$0.01
1475 | 0.39$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02 | 0.37$\pm$0.02
1485 |  0.11$\pm$0.02 $\dagger$ $\dagger$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.26$\pm$0.03 | 0.26$\pm$0.03 | 0.12$\pm$0.02 | 0.12$\pm$0.02 | 0.12$\pm$0.02 | 0.12$\pm$0.01
1486 |  0.05$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | nan | 0.05$\pm$0.0 | 0.05$\pm$0.0
1487 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.06$\pm$0.01 | 0.05$\pm$0.01
1489 |  0.1$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01 | 0.09$\pm$0.01
1494 | 0.14$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.03 | 0.12$\pm$0.03 | 0.11$\pm$0.03 | 0.12$\pm$0.03 | 0.12$\pm$0.04 | 0.12$\pm$0.03
1515 |  0.07$\pm$0.03 $\bullet$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.13$\pm$0.03 | 0.06$\pm$0.02 | 0.06$\pm$0.02 | 0.07$\pm$0.02 | 0.07$\pm$0.02 | 0.07$\pm$0.26
1590 |  0.16$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.16$\pm$0.0 | 0.16$\pm$0.0 | 0.16$\pm$0.0 | nan | 0.16$\pm$0.0 | 0.16$\pm$0.0
4134 |  0.2$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.19$\pm$0.02 | 0.19$\pm$0.02 | 0.19$\pm$0.01 | 0.19$\pm$0.02 | 0.2$\pm$0.02 | 0.2$\pm$0.02
4135 |  0.05$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.0 | 0.05$\pm$0.0 | 0.05$\pm$0.0 | nan | 0.05$\pm$0.0 | 0.05$\pm$0.0
4534 |  0.02$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | nan | 0.02$\pm$0.01 | 0.02$\pm$0.0
4538 |  0.31$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01 | 0.3$\pm$0.01
4541 |  0.42$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.42$\pm$0.0 | 0.42$\pm$0.0 | 0.42$\pm$0.0 | nan | 0.42$\pm$0.0 | 0.42$\pm$0.0
23512 |  0.28$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.28$\pm$0.01 | 0.28$\pm$0.01 | 0.28$\pm$0.0 | 0.28$\pm$0.01 | 0.28$\pm$0.01 | 0.28$\pm$0.01
23517 |  0.48$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.48$\pm$0.0 | 0.48$\pm$0.0 | 0.48$\pm$0.01 | 0.48$\pm$0.0 | 0.48$\pm$0.0 | 0.48$\pm$0.0
40498 | 0.34$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.29$\pm$0.01 | 0.3$\pm$0.02 | 0.3$\pm$0.01 | 0.29$\pm$0.01 | 0.3$\pm$0.02 | 0.29$\pm$0.01
40668 | 0.17$\pm$0.06 $\dagger$ $\dagger$ $\dagger$ $\circ$ $\circ$ | 0.26$\pm$0.0 | 0.26$\pm$0.0 | 0.23$\pm$0.07 | nan | 0.03$\pm$0.0 | 0.03$\pm$0.01
40670 |  0.03$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | nan | 0.03$\pm$0.02 | 0.03$\pm$0.01
40685 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0
40701 |  0.04$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | nan | 0.05$\pm$0.01 | 0.05$\pm$0.01
40900 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
40975 |  0.0$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.01 | 0.01$\pm$0.01 | 0.0$\pm$0.0 | nan | 0.01$\pm$0.01 | 0.01$\pm$0.01
40978 |  0.03$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.03$\pm$0.01 | 0.03$\pm$0.01 | 0.03$\pm$0.01 | nan | 0.03$\pm$0.01 | 0.03$\pm$0.01
40981 |  0.12$\pm$0.03 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.03 | 0.13$\pm$0.04 | 0.12$\pm$0.03 | nan | 0.12$\pm$0.02 | 0.12$\pm$0.02
40982 |  0.21$\pm$0.04 $\circ$ $\circ$ $\circ$ $\circ$ $\dagger$ | 0.21$\pm$0.02 | 0.22$\pm$0.02 | 0.2$\pm$0.03 | 0.21$\pm$0.02 | 0.21$\pm$0.02 | 0.25$\pm$0.1
40983 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
40984 |  0.02$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.01 | 0.03$\pm$0.01 | 0.02$\pm$0.01 | 0.02$\pm$0.01 | 0.03$\pm$0.02 | 0.02$\pm$0.01
41027 |  0.07$\pm$0.01 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.16$\pm$0.01 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0 | 0.14$\pm$0.0
41138 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0 | 0.01$\pm$0.0
41142 |  0.27$\pm$0.03 $\circ$ $\circ$ $\circ$ $\dagger$ $\circ$ $\circ$ | 0.28$\pm$0.02 | 0.27$\pm$0.02 | 0.26$\pm$0.01 | nan | 0.29$\pm$0.03 | 0.28$\pm$0.02
41143 |  0.18$\pm$0.02 $\circ$ $\circ$ $\bullet$ $\circ$ $\circ$ $\circ$ | 0.19$\pm$0.02 | 0.19$\pm$0.03 | 0.2$\pm$0.03 | nan | 0.19$\pm$0.03 | 0.2$\pm$0.04
41144 |  0.1$\pm$0.02 $\dagger$ $\dagger$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.24$\pm$0.02 | 0.23$\pm$0.01 | 0.1$\pm$0.02 | 0.11$\pm$0.02 | 0.11$\pm$0.02 | 0.11$\pm$0.02
41145 |  0.18$\pm$0.01 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.25$\pm$0.02 | 0.25$\pm$0.01 | 0.25$\pm$0.01 | 0.24$\pm$0.01 | 0.25$\pm$0.02 | 0.24$\pm$0.01
41146 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.04$\pm$0.01 | 0.05$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01
41147 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.32$\pm$0.0 | 0.32$\pm$0.0 | nan | nan | nan | nan
41150 |  0.06$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.07$\pm$0.0 | 0.06$\pm$0.0 | 0.07$\pm$0.0 | 0.06$\pm$0.0 | 0.07$\pm$0.0 | 0.06$\pm$0.0
41156 |  0.14$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.14$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.01 | 0.14$\pm$0.01
41157 | 0.24$\pm$0.09 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.11$\pm$0.09 | 0.15$\pm$0.08 | 0.21$\pm$0.1 | 0.21$\pm$0.05 | 0.26$\pm$0.13 | 0.26$\pm$0.15
41158 |  0.05$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.05$\pm$0.01 | 0.04$\pm$0.01 | 0.04$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01 | 0.05$\pm$0.01
41159 |  0.17$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.18$\pm$0.01 | 0.18$\pm$0.01 | nan | nan | nan | nan
41162 |  0.1$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.1$\pm$0.0 | 0.1$\pm$0.0 | 0.1$\pm$0.0 | nan | 0.1$\pm$0.0 | 0.1$\pm$0.0
41163 |  0.01$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0 | 0.02$\pm$0.0
41164 | 0.31$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01 | 0.29$\pm$0.01
41165 |  0.55$\pm$0.02 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.55$\pm$0.01 | 0.56$\pm$0.01 | 0.55$\pm$0.02 | 0.56$\pm$0.01 | 0.55$\pm$0.01 | 0.56$\pm$0.01
41166 |  0.28$\pm$0.0 $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ $\dagger$ | 0.33$\pm$0.0 | 0.32$\pm$0.0 | 0.31$\pm$0.0 | 0.32$\pm$0.0 | 0.32$\pm$0.0 | 0.32$\pm$0.0
41167 | nan $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.31$\pm$0.0 | nan | nan | 0.2$\pm$0.0 | nan | nan
41168 |  0.27$\pm$0.0 $\bullet$ $\bullet$ $\bullet$ $\bullet$ $\bullet$ $\bullet$ | 0.29$\pm$0.01 | 0.29$\pm$0.0 | 0.29$\pm$0.0 | 0.29$\pm$0.01 | 0.29$\pm$0.0 | 0.29$\pm$0.0
41169 | 0.69$\pm$0.01 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.65$\pm$0.0 | 0.64$\pm$0.0 | 0.64$\pm$0.0 | 0.64$\pm$0.0 | 0.64$\pm$0.0 | 0.64$\pm$0.0
42732 |  0.12$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.12$\pm$0.0 | 0.12$\pm$0.0 | 0.12$\pm$0.0 | nan | 0.12$\pm$0.0 | 0.12$\pm$0.0
42733 |  0.17$\pm$0.0 $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ $\circ$ | 0.17$\pm$0.0 | 0.17$\pm$0.0 | 0.17$\pm$0.0 | nan | 0.17$\pm$0.0 | 0.17$\pm$0.0

Table 6: Stage-Wise Results in Monotone Configuration for scikit-learn.

##  Appendix C Wall-Times per Stage

The following tables show the average runtime of Naive AutoML in the different
stages per dataset.

openmlid | probing | scaling | filtering | meta | tuning | validation | Total
---|---|---|---|---|---|---|---
3 | 123 | 0 | 65 | 80 | 28 | 127 | 423
12 | 558 | 0 | 547 | 33 | 58 | 168 | 1364
54 | 19 | 0 | 17 | 26 | 22 | 42 | 126
181 | 27 | 0 | 6 | 11 | 15 | 26 | 85
1049 | 62 | 0 | 44 | 8 | 21 | 41 | 176
1067 | 53 | 0 | 34 | 21 | 19 | 42 | 169
1111 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
1457 | 1398 | 0 | 1085 | 50 | 10 | 299 | 2842
1461 | 775 | 0 | 441 | 35 | 36 | 248 | 1535
1464 | 2 | 0 | 3 | 3 | 19 | 12 | 39
1468 | 2413 | 0 | 833 | 4 | 14 | 113 | 3377
1475 | 1961 | 0 | 728 | 23 | 40 | 213 | 2965
1485 | 1741 | 0 | 170 | 7 | 21 | 66 | 2005
1486 | 2927 | 0 | 673 | 0 | 0 | 0 | 3600
1487 | 353 | 0 | 195 | 30 | 22 | 174 | 774
1489 | 191 | 0 | 1 | 943 | 74 | 101 | 1310
1494 | 46 | 0 | 47 | 15 | 25 | 40 | 173
1515 | 3569 | 0 | 31 | 0 | 0 | 0 | 3600
1590 | 1662 | 0 | 446 | 32 | 25 | 335 | 2500
4134 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
4135 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
4534 | 297 | 0 | 323 | 18 | 23 | 235 | 896
4538 | 360 | 0 | 254 | 68 | 62 | 167 | 911
4541 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
23512 | 1081 | 0 | 722 | 218 | 243 | 321 | 2585
23517 | 942 | 0 | 422 | 255 | 122 | 308 | 2049
40498 | 140 | 0 | 80 | 2 | 28 | 81 | 331
40668 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
40670 | 446 | 0 | 173 | 118 | 47 | 119 | 903
40685 | 671 | 0 | 639 | 19 | 19 | 440 | 1788
40701 | 274 | 0 | 191 | 15 | 30 | 160 | 670
40900 | 239 | 0 | 96 | 16 | 23 | 129 | 503
40975 | 29 | 0 | 24 | 7 | 23 | 33 | 116
40978 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
40981 | 17 | 0 | 5 | 2 | 20 | 15 | 59
40982 | 384 | 0 | 101 | 1 | 28 | 78 | 592
40983 | 184 | 0 | 122 | 38 | 42 | 138 | 524
40984 | 129 | 0 | 73 | 166 | 27 | 76 | 471
41027 | 520 | 0 | 3 | 36 | 57 | 393 | 1009
41138 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
41142 | 1278 | 0 | 654 | 5 | 4 | 1659 | 3600
41143 | 320 | 0 | 226 | 4 | 26 | 120 | 696
41144 | 652 | 0 | 212 | 2 | 19 | 65 | 950
41145 | 1520 | 0 | 334 | 3 | 32 | 159 | 2048
41146 | 190 | 0 | 84 | 1 | 44 | 99 | 418
41147 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
41150 | 2692 | 0 | 908 | 0 | 0 | 0 | 3600
41156 | 245 | 0 | 183 | 36 | 26 | 154 | 644
41157 | 292 | 0 | 495 | 2 | 1 | 50 | 840
41158 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
41159 | 2493 | 0 | 676 | 42 | 32 | 193 | 3436
41162 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
41163 | 1210 | 0 | 1388 | 24 | 15 | 317 | 2954
41164 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
41165 | 0 | 0 | 0 | 0 | 0 | 0 | 0
41166 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
41167 | 2072 | 0 | 1528 | 0 | 0 | 0 | 3600
41168 | 3351 | 0 | 249 | 0 | 0 | 0 | 3600
41169 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600
42732 | 1395 | 0 | 1091 | 322 | 26 | 766 | 3600
42733 | 3600 | 0 | 0 | 0 | 0 | 0 | 3600

Table 7: Average time spent in stage per dataset in WEKA.

openmlid | probing | scaling | filtering | meta | tuning | validation | Total
---|---|---|---|---|---|---|---
3 | 31 | 25 | 49 | 1122 | 510 | 21 | 1758
12 | 202 | 772 | 544 | 2082 | 0 | 0 | 3600
54 | 9 | 49 | 58 | 1230 | 373 | 1 | 1720
181 | 18 | 75 | 91 | 2415 | 394 | 25 | 3018
1049 | 6 | 49 | 48 | 2049 | 420 | 30 | 2602
1067 | 7 | 52 | 54 | 3410 | 77 | 0 | 3600
1111 | 746 | 381 | 234 | 2239 | 0 | 0 | 3600
1457 | 602 | 1681 | 1204 | 113 | 0 | 0 | 3600
1461 | 495 | 88 | 109 | 2332 | 511 | 22 | 3557
1464 | 3 | 19 | 22 | 1727 | 324 | 9 | 2104
1468 | 48 | 145 | 85 | 2567 | 332 | 46 | 3223
1475 | 135 | 643 | 640 | 2182 | 0 | 0 | 3600
1485 | 81 | 347 | 80 | 2917 | 139 | 4 | 3568
1486 | 636 | 564 | 346 | 2054 | 0 | 0 | 3600
1487 | 16 | 72 | 74 | 1474 | 446 | 29 | 2111
1489 | 23 | 45 | 68 | 1918 | 359 | 12 | 2425
1494 | 8 | 41 | 49 | 912 | 453 | 18 | 1481
1515 | 138 | 710 | 218 | 2534 | 0 | 0 | 3600
1590 | 503 | 78 | 204 | 2038 | 525 | 39 | 3387
4134 | 201 | 913 | 149 | 2337 | 0 | 0 | 3600
4135 | 465 | 269 | 83 | 2172 | 472 | 139 | 3600
4534 | 108 | 99 | 116 | 2207 | 990 | 23 | 3543
4538 | 181 | 884 | 944 | 1591 | 0 | 0 | 3600
4541 | 811 | 480 | 403 | 1906 | 0 | 0 | 3600
23512 | 823 | 2777 | 0 | 0 | 0 | 0 | 3600
23517 | 531 | 765 | 329 | 1908 | 5 | 0 | 3538
40498 | 42 | 255 | 207 | 3096 | 0 | 0 | 3600
40668 | 714 | 585 | 265 | 2036 | 0 | 0 | 3600
40670 | 108 | 47 | 53 | 2192 | 430 | 378 | 3208
40685 | 267 | 1457 | 1539 | 337 | 0 | 0 | 3600
40701 | 16 | 136 | 69 | 3371 | 8 | 0 | 3600
40900 | 17 | 65 | 62 | 1258 | 371 | 11 | 1784
40975 | 12 | 56 | 74 | 3458 | 0 | 0 | 3600
40978 | 278 | 157 | 49 | 2702 | 332 | 53 | 3571
40981 | 6 | 26 | 25 | 1134 | 407 | 9 | 1607
40982 | 32 | 179 | 100 | 3289 | 0 | 0 | 3600
40983 | 12 | 74 | 79 | 1605 | 571 | 98 | 2439
40984 | 33 | 154 | 174 | 3239 | 0 | 0 | 3600
41027 | 217 | 1017 | 1220 | 1146 | 0 | 0 | 3600
41138 | 848 | 2752 | 0 | 0 | 0 | 0 | 3600
41142 | 291 | 1359 | 230 | 1720 | 0 | 0 | 3600
41143 | 77 | 58 | 35 | 1206 | 1262 | 104 | 2742
41144 | 55 | 354 | 115 | 3076 | 0 | 0 | 3600
41145 | 196 | 1064 | 147 | 2193 | 0 | 0 | 3600
41146 | 23 | 133 | 113 | 3331 | 0 | 0 | 3600
41147 | 0 | 0 | 0 | 0 | 0 | 0 | 0
41150 | 739 | 2607 | 254 | 0 | 0 | 0 | 3600
41156 | 12 | 103 | 60 | 3209 | 216 | 0 | 3600
41157 | 40 | 184 | 63 | 1453 | 416 | 11 | 2167
41158 | 96 | 555 | 256 | 2693 | 0 | 0 | 3600
41159 | 0 | 0 | 0 | 0 | 0 | 0 | 0
41162 | 638 | 339 | 170 | 2453 | 0 | 0 | 3600
41163 | 918 | 2682 | 0 | 0 | 0 | 0 | 3600
41164 | 472 | 1051 | 515 | 1562 | 0 | 0 | 3600