# AMLB: an AutoML Benchmark

Pieter Gijsbers1<EMAIL_ADDRESS>Marcos L. P. Bueno1,4
<EMAIL_ADDRESS>Stefan Coors2<EMAIL_ADDRESS>muenchen.de Erin LeDell3<EMAIL_ADDRESS>Sébastien Poirier3<EMAIL_ADDRESS>Janek
Thomas2<EMAIL_ADDRESS>Bernd Bischl2<EMAIL_ADDRESS>muenchen.de Joaquin Vanschoren1<EMAIL_ADDRESS>
1 Eindhoven University of Technology, Eindhoven, The Netherlands
2 Ludwig Maximilian University of Munich, Munich, Germany
3 H2O.ai, Mountain View, CA, United States
4 Radboud University, Nijmegen, The Netherlands

###### Abstract

Comparing different AutoML frameworks is notoriously challenging and often
done incorrectly. We introduce an open and extensible benchmark that follows
best practices and avoids common mistakes when comparing AutoML frameworks. We
conduct a thorough comparison of 9 well-known AutoML frameworks across 71
classification and 33 regression tasks. The differences between the AutoML
frameworks are explored with a multi-faceted analysis, evaluating model
accuracy, its trade-offs with inference time, and framework failures. We also
use Bradley-Terry trees to discover subsets of tasks where the relative AutoML
framework rankings differ. The benchmark comes with an open-source tool that
integrates with many AutoML frameworks and automates the empirical evaluation
process end-to-end: from framework installation and resource allocation to in-
depth evaluation. The benchmark uses public data sets, can be easily extended
with other AutoML frameworks and tasks, and has a website with up-to-date
results.

Keywords: open source, benchmark, automated machine learning, automl

##  1 Introduction

To create useful machine learning (ML) models from data, data scientists must
prepare the data for consumption by ML algorithms (for example, coercing it to
a different format or by encoding categorical features), select an ML
algorithm, and tune its hyperparameters. This requires extensive expertise,
such as knowing which hyperparameters to tune and how (Probst et al., 2019;
Weerts et al., 2020). Even with this knowledge, it is a time-consuming task,
since the best choices are unique to each data set and can be interdependent
on each other (Van Rijn and Hutter, 2018).

The field of automated machine learning (AutoML) is focused on automating the
design and optimization of ML pipelines in a data-driven way (Hutter et al.,
2019). Neural Architecture Search (NAS) is an important part of AutoML that
automates the design decisions of deep neural networks. AutoML aims to free up
valuable time for experts to perform other tasks and allow novice users to
train well-performing ML models.

Many different AutoML approaches have been proposed, including sequential
model-based optimization (Hutter et al., 2011; Snoek et al., 2012),
hierarchical task planning (Erol et al., 1994), and genetic programming (Koza,
1992). Novel systems are being developed in both academia and industry, and a
recent survey by Van der Blom et al. (2021) showed that $69$% of $307$
practitioners (at least partially) adopt automated model selection and
hyperparameter configuration.

###  1.1 The Need for Standardized Benchmarking

With considerable effort being spent on developing and improving AutoML
frameworks as well as increased usage by practitioners, there is a need for
systematic and in-depth comparisons of the the various approaches to track
progress in the field. However, the comparison of AutoML frameworks is prone
to several types of error. First, selection bias may be introduced, even
accidentally, when authors decide which data sets to use in their evaluation.
For example, too few data sets may be selected to accurately evaluate the
framework’s strengths and weaknesses, and the chosen data sets may no longer
be challenging for current AutoML frameworks. Without a standard suite of data
sets to use for evaluation, the selection of data sets is often not reasonably
justified and motivated. Moreover, issues may arise from errors in the
installation, configuration, or use of ‘competitor’ frameworks. Typical
examples are misunderstanding memory management and/or using insufficient
compute resources (Balaji and Allen, 2018), or failing to use comparable
resource budgets (Ferreira et al., 2021).

Several suites of benchmark data sets have been used in the aforementioned
papers, but none have become a standard in the AutoML community. The original
selection of data sets by Thornton et al. (2013) was used in several earlier
papers (Feurer et al., 2015a; Mohr et al., 2018), but fails to highlight
differences between current AutoML frameworks. Most published AutoML papers
use a self-selected suit of data sets on which their methods are evaluated.
For example, Drori et al. (2018), Rakotoarison et al. (2019) and Gil et al.
(2018) were all published at the same time but feature different suites on
which they evaluate their contributions. This inconsistency makes it
impossible to directly compare results across papers and track progress over
time. It also potentially allows for presenting cherry-picked results.

###  1.2 Our Contributions

We introduce a novel AutoML benchmark following best practices to avoid these
common pitfalls while stimulating progress towards more standardized
benchmarking.111A first look of this tool was presented at the ICML 2019
AutoML Workshop (Gijsbers et al., 2019). The current version is significantly
more general, more systematic, and allows much more in-depth analysis. To
ensure reproducibility222Using the ACM definition: an independent group can
obtain the same result using the author’s own artifacts
(https://www.acm.org/publications/policies/artifact-review-and-badging-
current)., we provide an open-source benchmarking tool333Code, results, and
documentation at: https://openml.github.io/automlbenchmark/ that allows easy
integration with AutoML frameworks, and performs end-to-end evaluations
thereof on carefully curated sets of open data sets. Our focus is on tabular
data. Unstructured data are out of scope for this paper, since they are best
tackled with NAS, and benchmarking NAS frameworks imposes different practical
constraints, as discussed in Section 2.2. We also restrict our evaluations to
open-source AutoML frameworks.

Our benchmarking tool, dubbed AMLB (for AutoML Benchmark), can be used to
perform evaluations of AutoML frameworks in a fully automated way. The AutoML
frameworks are integrated with the benchmarking tool in direct agreement and
jointly with the framework’s developers to ensure correctness. We carry out a
large-scale evaluation of 9 well-known AutoML frameworks, some of which with
multiple configurations, across 71 classification and 33 regression tasks. To
better understand how these systems perform across many tasks, we also
introduce techniques for detailed comparison of AutoML frameworks, including
final model accuracy, inference time trade-offs, and failure analysis.
Finally, we provide an interactive visualization tool that may be used for
further exploration of all our results or to reproduce the analyses performed
in this paper.

In the remainder of this paper, we first discuss related work in Section 2 and
cover several key open-source AutoML frameworks in Section 3. Next, we provide
an overview of our proposed benchmarking tool in Section 4 and motivate our
benchmark design choices in Section 5. The results obtained by running this
benchmark are analyzed in Section 6. Finally, Section 7 summarizes our main
conclusions and sketches directions for future work.

##  2 Related Literature

In this section, we motivate why we need benchmarks specifically designed for
AutoML, review other work evaluating AutoML frameworks, and finally discuss
the relevant ML benchmarking literature. Several benchmark suites have been
developed in ML (Van Gestel et al., 2004; Olson et al., 2017; Wu et al., 2018;
Bischl et al., 2021; Fischer et al., 2023). The data sets in these suites
often do not include problematic data characteristics found in real world
tasks (for example, many missing values), as many ML algorithms are not
natively able to handle them. By contrast, in order to be applicable to a wide
range of data, AutoML frameworks should be designed to handle these
problematic data sets. As such, there is opportunity to allow more such
problematic data sets in ML benchmarks and study how well AutoML frameworks
handle these issues. Moreover, runtime budgets are crucial in an AutoML
benchmark, as most AutoML frameworks are designed to run until a given time
budget is exhausted. These runtime budgets are often not specified beforehand
in traditional ML benchmarks, since algorithms must usually run to completion.
444One exception are performance studies, such as Kotthaus et al., 2015.
Consequently, new benchmarks that have been designed specifically for AutoML
frameworks are needed.

###  2.1 Evaluation of Automated Machine Learning Frameworks

To establish new best practices for AutoML benchmarking, it is beneficial to
study the shortcomings encountered in prior benchmarks as well as lessons
learned. Balaji and Allen (2018) conducted one of the first benchmark studies
on AutoML frameworks. They evaluated four open-source frameworks on both
classification and regression tasks sourced from OpenML (Vanschoren et al.,
2014), optimized for weighted F1 score and mean squared error, respectively.
Unfortunately, they encountered technical issues with most AutoML frameworks
that led to a questionable experimental evaluation. For example, H2O AUTOML
(LeDell and Poirier, 2020) was configured to optimize to a different metric
(log loss as opposed to weighted F1 score) and ran with a different setup
(unlike the others, H2O AUTOML was not containerized), and AUTO_ML (Parry,
2018) had its hyperparameter optimization (HPO) disabled, making for
incomparable results. This highlights the need for careful configuration of
all AutoML frameworks involved.

A study by Ferreira et al. (2021) evaluated the effectiveness of AutoML
frameworks for protein abundance prediction. This evaluation compared H2O
AUTOML with a 6 hour limit to GAMA with a one hour limit, and TPOT with a
limit defined by the hyperparameter configuration of its evolutionary
optimization (1000 generations with a population size of 250). While the
comparison was not the main contribution of the paper, the AutoML frameworks
were directly compared against each other, and these very different budgets
were not motivated.

A study on nearly 300 data sets across six different frameworks was conducted
by Truong et al. (2019). Each experiment consisted of a single 80/20 holdout
split on a 15-minute training time budget, which was chosen so that most tools
returned a result on at least 70% of the data sets. We postulate it is
reasonable to assume that the data sets for which no result is returned by a
framework are most often those data sets for which optimization is hard. For
example, a large data set might cause one framework to conduct only few
evaluations while it completely halts another. Unfortunately, this makes the
resulting comparisons uninterpretable, as a framework could seemingly
demonstrate better performance simply because it failed to return models on
data sets for which optimization was difficult. Hence, it is key that failures
must be avoided as much as possible, and any remaining failures should be
analysed and taken into account in subsequent analysis.

On the positive side, Truong et al. (2019) present their results across
different subsets of the benchmark—for example, few versus many categorical
features—which helps to highlight differences between different frameworks.
The authors also conduct small-scale experiments to analyze performance over
time by running the tools on multiple time budgets on a subset of data sets as
well as the ‘robustness’, which denotes the variance in final performance
given the same input data. Unfortunately, both experiments were conducted on
only one data set per sub-category, which does not enable generalizing the
results. Still, studying the impact of data characteristics and budget sizes
should ideally be part of AutoML benchmark design.

(Zöller and Huber, 2021) present a survey on AutoML and combined algorithm
selection and HPO frameworks (CASH, Thornton et al., 2013). Six CASH and five
AutoML frameworks are compared across 137 classification tasks, with the
former limited to 325 iterations and the latter constrained to a one hour time
budget. Among the CASH frameworks, hyperopt (Bergstra et al., 2013) performed
best, although absolute differences were small between all optimizers. The
AutoML frameworks are compared as they are, which might reflect common use.
However, by not controlling their settings, it becomes impossible to draw
conclusions about the effectiveness of individual parts of AutoML systems. A
number of errors of AutoML frameworks are observed, including memory
constraint violations, segmentation faults, and Java server crashes. The
authors also find that most frameworks construct rather modest pipelines (with
few preprocessing operators). As such, it is recommended that AutoML
frameworks are controlled carefully, use a wide operator search space, and are
evaluated on datasets that require non-trivial preprocessing.

Kaggle555https://www.kaggle.com/ , a platform for data science competitions,
is sometimes used to compare AutoML frameworks to human data scientists
(Zöller and Huber, 2021; Erickson et al., 2020). Zöller and Huber, 2021 found
that the rankings of AutoML frameworks on benchmark datasets are very
different from the rankings on competitions. Furthermore, humans still find
better solutions than the examined AutoML frameworks. However, it is hard to
interpret such results. Submissions on Kaggle leaderboards range from serious
attempts to random test runs, and a significant portion of them do not
outperform a simple baseline. Finally, most Kaggle results are several years
old and possibly no longer represent state-of-the-art human-made models
because of advances in the available hardware or algorithms.

A recent AutoML benchmark for multi-label classification (Wever et al., 2021)
proposed a general tool with a configurable search space and optimizer, which
allows for the inclusion of new methods and ablation studies. Unfortunately,
this approach requires that existing AutoML frameworks be re-implemented
within this tool, which is difficult in such a rapidly developing field. As
such, we need a simpler way to include existing and new AutoML frameworks into
benchmarking tools while still allowing control over their configuration.

In addition to AutoML benchmarks, a series of competitions for tabular AutoML
was hosted (Guyon et al., 2019). The first two competitions focused on tabular
AutoML, where data is assumed to be independent and identically distributed.
In these competitions, participants submitted code that automatically builds a
model on given data and produced predictions for a test set. During the
development phase, competitors could make use of a public leaderboard and
several validation data sets. After the development phase, the latest
submissions of each participant would be evaluated on a set of new data sets
to determine the final ranking. These data sets consisted of a mix of both new
data and data taken from public repositories, although they were reformatted
to conceal their identity. In their analysis Guyon et al. (2019) reveal that
most methods failed to return results on at least some data sets due to
practical issues, such as running out of memory.

###  2.2 Other (Benchmark) Literature

Instead of benchmarking whole (Auto-) ML frameworks, it is often useful to
focus on various sub-parts and optimize them step-by-step. One part of such a
system is HPO, and its available algorithms are as numerous and diverse as the
learning algorithms themselves. Consequently, there exist various benchmarking
suites to foster research by comparing those optimization algorithms.

Like non-black-box optimization, black box optimizers are typically evaluated
on synthetic test functions or on real-world tasks. A well-established
benchmarking suite for continuous optimization is COCO (Hansen et al., 2021),
which includes a collection of various synthetic black-box benchmark
functions.
Nevergrad666https://facebookresearch.github.io/nevergrad/benchmarking.html is
a popular platform for gradient-free optimization with benchmarking
functionality. kurobako (Ohta and Yamazaki, 2022) provides various general
black-box optimizers and benchmark problems, while LassoBench (Šehić et al.,
2021) is suitable for benchmarking high-dimensional optimization problems.
Especially for Bayesian Optimization, Bayesmark (Turner, 2022) combines
several benchmarks on real-world tasks.

One of the first benchmarks for empirically evaluating HPO algorithms was
HPOlib (Eggensperger et al., 2013), which allows accessing real-world HPO
tasks, tabular and surrogate benchmarks, and synthetic test functions using a
common API. This benchmark was used by Bergstra et al. (2014) in their
empirical benchmark studies. HPOBench (Eggensperger et al., 2021) is a similar
successor of HPOlib that concentrates on reproducible containerized benchmarks
and multi-fidelity optimization problems. Based on OpenML (Vanschoren et al.,
2014), Arango et al. (2021) recently introduced HPO-B, a large-scale
reproducible benchmark for transfer-HPO methods. In contrast, PROFET (Klein et
al., 2019) uses a generative meta-model to generate synthetic but realistic
benchmark instances.

A related benchmark is NAS-Bench-101 (Ying et al., 2019), which is a tabular
data set that maps convolutional neural network architectures to their trained
and evaluated performance on CIFAR-10. Its goal is to make NAS more
accessible, despite the tremendous demand of computational resources.
Additionally, there exists a series of NAS-Bench systems that includes NAS-
Bench-1shot1 (Zela et al., 2020), NAS-Bench-301 (Siems et al., 2020), and
others.

##  3 AutoML Frameworks

Automated ML pipeline design was first explored by Escalante et al. (2009),
but the first prominent AutoML framework was AUTO-WEKA (Thornton et al.,
2013). AUTO-WEKA used Bayesian optimization to select and tune the algorithms
in an ML pipeline based on WEKA (Hall et al., 2009). Since then, a plethora of
new AutoML frameworks have been developed, either by iteratively improving on
old designs or using novel approaches. In this section, we will discuss the
tools we considered for our benchmark.

Unfortunately, the cost of evaluating all frameworks is prohibitive, so we
selected 9 of them to evaluate in this work. Only open source tools were
considered. From those, we made selections to cover a variety of different
optimization approaches. We considered frameworks developed by industry as
well as academia and included packages whose authors proactively integrated
their AutoML framework, so that we are confident that they are integrated
correctly.

As our experiments are run on machines without GPU, we focus on AutoML
frameworks which are not significantly affected by the lack thereof. For
example, AUTO-KERAS (Jin et al., 2019), and AUTOPYTORCH (Zimmer et al., 2021)
gain large benefits from having a GPU available, thus reporting an evaluation
with only CPU available might give a pessimistic performance estimation
compared to typical use of those frameworks. However, even for some of our
selected frameworks, such as H2O AUTOML and AUTOGLUON, specific base models
support GPU acceleration when it is available.

The most notable omission is AUTO-WEKA, which we decided to exclude based on
the performance in our 2019 evaluation and its lack of updates since Gijsbers
et al. (2019). Some frameworks integrated with the benchmark tool are not
evaluated in this paper. The authors of AUTOXGBOOST (Thomas et al., 2018)
opted out of an evaluation because it is built on deprecated software with no
plans for updates. ML-PLAN (Mohr et al., 2018), MLR3AUTOML777Source and
documentation of MLR3AUTOML at https://github.com/a-hanf/mlr3automl/., and
MLNET888Source and documentation of MLNET at
https://github.com/dotnet/machinelearning/. are excluded because we
encountered significant technical problems evaluating the systems. We hope to
publish results from these frameworks at a later date. There are still many
tools not yet integrated with the AutoML benchmark, and we hope that we can
work together with AutoML developers to add additional integrations in the
future.

###  3.1 Integrated Frameworks

Table 1 offers an overview of the AutoML frameworks evaluated in this paper
alongside a simplified description of their optimization and search space
design, which are elaborated below. We refer the interested reader to the
original publications and our website, which has links to the frameworks’
source code and documentation. NAIVEAUTOML was also evaluated but excluded
from the analysis in Section 6, as explained in Appendix E.

Framework | Optimization and Search Space | Reference
---|---|---
AUTOGLUON | Stacked ensembles of preset pipelines | Erickson et al. (2020)
AUTO-SKLEARN | BO of SCIKIT-LEARN pipelines | Feurer et al. (2015a)
AUTO-SKLEARN 2 | BO of iterative algorithms | Feurer et al. (2020)
FLAML | CFO of iterative algorithms | Wang et al. (2021)
GAMA | EO of SCIKIT-LEARN pipelines | Gijsbers and Vanschoren (2021)
H2O AUTOML | Iterative mix of RS and ensembling | LeDell and Poirier (2020)
LIGHTAUTOML | BO of linear models and GBM | Vakhrushev et al. (2021)
MLJAR | Custom data science pipeline | Płońska and Płoński (2021)
NAIVEAUTOML | Custom data science pipeline | Mohr and Wever (2023)
TPOT | EO of SCIKIT-LEARN pipelines | Olson and Moore (2016)
Table 1: Used AutoML frameworks in the experiments. Different optimization
techniques such as Bayesian optimization (BO), evolutionary optimization (EO),
random search (RS) or cost frugal optimization (CFO) are used across different
search spaces.

####  3.1.1 AutoGluon-Tabular

AUTOGLUON automates ML across a variety of tasks, including image, text, and
tabular data. The subsystem that automates ML on tabular data is called
AUTOGLUON-TABULAR (Erickson et al., 2020), but we will refer to it as
AUTOGLUON. In contrast to other AutoML systems discussed here, AUTOGLUON does
not perform a pipeline search or hyperparameter tuning. Instead, it has a
predetermined set of models that are combined through multi-layer stacking and
ensembling.

AUTOGLUON’s ensemble consists of three layers. The first layer consists of
models from a range of model families trained directly on the data. In the
second layer, the same type of models are considered but rather as a stacking
learner trained with both the input data and the predictions of the first
layer. In the final layer, the predictions of the second-layer models are
combined into an ensemble, using an ensemble method (Caruana et al., 2004)
first used in AutoML by AUTO-SKLEARN (Feurer et al., 2015a).

To adhere to time constraints, AUTOGLUON may stop iterative algorithms
prematurely or forgo training certain models altogether. Given more time,
AUTOGLUON will train additional models using the same algorithms and
hyperparameter configurations on different data splits, which further improves
the generalization of the stacking layer.

AUTOGLUON offers many different presets that affect the trade-off between
final model performance and inference time. In this paper, we evaluated three
presets: best quality, high quality, and high quality with an inference time
limit (denoted as B, HQ, and HQIL, respectively). These respective presets
produce increasingly faster models at the cost of decreased model accuracy.

####  3.1.2 auto-sklearn

Based on the design of AUTO-WEKA, AUTO-SKLEARN (Feurer et al., 2015a) also
uses Bayesian optimization but is instead implemented in Python and optimizes
pipelines built with SCIKIT-LEARN (Pedregosa et al., 2011). Additionally, it
warm-starts optimization through meta-learning, starting pipeline search with
the best pipelines for the most similar data sets (Feurer et al., 2015b).
After pipeline search has concluded, an ensemble is created from pipelines
trained during search using the procedure described by Caruana et al. (2004,
2006). AUTO-SKLEARN has won two AutoML challenges (Guyon et al., 2019),
although for both entries, AUTO-SKLEARN was customized for the competition,
and not all changes are found in the public releases (Feurer et al., 2018).

Based on experience from the challenges, AUTO-SKLEARN 2 was subsequently
developed (Feurer et al., 2020). The most notable changes include reducing the
search space to only iterative learning algorithms and excluding most
preprocessing, use of successive halving (Jamieson and Talwalkar, 2016),
adaptive evaluation strategies, and replacing the data-specific warm-start
strategy with a data-agnostic portfolio of pipelines. Because these changes
make version 2.0 almost entirely different from 1.0, and 1.0 has been updated
since our last evaluation, we evaluate both auto-sklearn versions in this
paper. However, AUTO-SKLEARN 2 does not yet support regression, and its heavy
use of meta-learning made it impossible for us to perform a ‘clean’ evaluation
at this time (see Section 5.3.3).

####  3.1.3 FLAML

The Fast and Lightweight AutoML Library (FLAML, Wang et al., 2021) optimizes
boosting frameworks (XGBOOST, Chen and Guestrin, 2016, CATBOOST, Prokhorenkova
et al., 2018, and LIGHTGBM, Ke et al., 2017) and a small selection of SCIKIT-
LEARN algorithms through a multi-fidelity randomized directed search called
Cost-Frugal Optimization (Wu et al., 2021). This search is based on an
expected cost for improvement, which tracks the expected computational cost of
improving over the best model found so far for each learner. Only after
choosing which learner to tune, hyperparameter optimization proceeds by a
randomized directed search, sampling a new configuration from a unit sphere
around the previous sample point. After evaluating its validation performance,
the next sample point is moved to that direction (if better) or the opposite
direction (if worse). FLAML positions itself as a fast AutoML framework that
is designed to work for small time budgets (Wang et al., 2021).

####  3.1.4 GAMA

Designed as a modular AutoML tool for researchers, GAMA’s search method and
post processing are easily configurable and extensible (Gijsbers and
Vanschoren, 2021). By default, GAMA uses genetic programming to optimize
linear ML pipelines with an arbitrary amount of preprocessing algorithms.
Similar to TPOT, GAMA’s evolutionary algorithm uses NSGA-II to perform multi-
objective optimization (Deb et al., 2002), maximizing performance while
minimizing the number of components in the pipeline. By contrast, GAMA’s
evolutionary algorithm is asynchronous and does not work with distinct
generations, which allows for higher resource utilization. The final model is
constructed through ensemble selection (Caruana et al., 2004, 2006), similar
to AUTO-SKLEARN.

####  3.1.5 H2O AutoML

Built on the H2O distributed machine learning platform (H2O.ai, 2013), H2O
AUTOML (LeDell and Poirier, 2020) evaluates a portfolio of bespoke algorithm
configurations and also performs random searches over the majority of the
supervised learning algorithms offered in H2O. Early stopping is applied to
the searches for efficiency. The amount of time the H2O AUTOML spends
searching each algorithm is pre-specified by the authors, though this can be
customized. This allocates the amount of optimization work done at each step
of the algorithm, and some stronger algorithms (e.g. XGBoost) are favored, or
given more time, over others (e.g. GLM). To further increase model
performance, H2O AUTOML also trains two types of stacked ensemble models at
various stages during the run: an ensemble using all available models at step
$s$, and an ensemble with only the best models of each algorithm type at step
$s$.

H2O AUTOML aims to cover a large search space quickly and relies on stacking
to boost model performance. Additionally, H2O AUTOML uses a predefined
strategy for imputation, normalization, and categorical encoding for each
algorithm and does not currently optimize over preprocessing pipelines by
default, but preprocessing can be turned on as an option. The H2O AUTOML
algorithm is designed to generate models that are very fast at inference time,
rather than strictly focusing on maximizing model accuracy, with the goal of
balancing these two competing factors to produce practical models suited for
production environments.

####  3.1.6 LightAutoML

LIGHTAUTOML is specifically designed with applications in the financial
services industry in mind (Vakhrushev et al., 2021). In this framework,
pipelines are designed for quick inference and interpretability. Only linear
models and boosting frameworks (CATBOOST and LIGHTGBM) are considered. Expert
rules are used to evaluate likely good hyperparameter configurations and
design the search space. Tree-structured Parzen Estimators (Bergstra et al.,
2011) are used to optimize hyperparameters of the boosting frameworks, while
warm-starting and early stopping are used to optimize linear models with grid
search. Different models are combined in either a weighted voting ensemble
(binary classification and regression) or with two levels of stacking (multi-
class classification). In a special ‘compete’ mode for larger time budgets,
the AutoML pipeline is run multiple times with different configurations and
their resulting models are ensembled with weighted voting, which allows for a
more robust model.

####  3.1.7 MLJAR

MLJAR (Płońska and Płoński, 2021) starts its search with a set of
predetermined models and a limited random search—similar to H2O AUTOML. This
is followed by a feature creation and selection step, after which a hill
climbing algorithm is used to further tune the best pipelines. After search,
the models can be stacked, used in a voting ensemble, or both. Learners from
SCIKIT-LEARN are considered, as well as boosting packages XGBOOST, CATBOOST,
and LIGHTGBM. MLJAR features multiple modes for different use cases, including
exploratory data analysis or finding a fast model, a ‘perform’ mode aimed at
finding a good model with good inference time, and a ‘compete’ mode which aims
to find the best possible model. We evaluate both the ‘compete’ and ‘perform’
presets in this work.

####  3.1.8 Naive AutoML

Proposed as a baseline by Mohr and Wever (2023), it mimics a simple workflow a
data scientist might execute. First, a base learner is chosen by evaluating
their performance using their default hyperparameter configuration. Then
several steps are performed in order, including feature scaling, feature
selection, and hyperparameter tuning. Unfortunately, we encountered issues
evaluating NAIVEAUTOML which prohibited us to include its results in Section
6. Appendix E motivates this choice, and shows the results we obtained for
NAIVEAUTOML.

####  3.1.9 TPOT

Tree-based Pipeline Optimization Tool (Olson and Moore, 2016), or TPOT,
optimizes pipelines using genetic programming (e.g., McKay et al. (2010)): ML
pipelines can be expressed as trees where different branches represent
distinct preprocessing pipelines. These pipelines are then optimized through
evolutionary optimization. To reduce overfitting that may arise from the large
search space, multi-objective optimization is used to minimize for pipeline
complexity while optimizing for performance (Olson et al., 2016). It is also
possible to reduce the search space by specifying a pipeline template (Le et
al., 2018), which dictates the high-level steps in the pipeline (for example,
a “Selector-Transformer-Classifier” template will result in pipelines with
only those three steps, in that order). Development has focused around genomic
studies, providing specific options for dealing with this type of high
dimensional data for which prior knowledge may be present (Sohn et al., 2017).
While TPOT supports neural networks in its search (Romano et al., 2021), the
default search space uses SCIKIT-LEARN components and XGBOOST only.

###  3.2 Baselines

In addition to the integrated frameworks, the benchmark tool allows for
running several baselines. The constant predictor always predicts the class
prior or mean target value, regardless of the values of the independent
variables. The Random Forest baseline builds a forest $10$ trees at a time,
until one of two criteria is met: we expect to exceed $90$% of the memory
limit or time limit by building 10 more trees, or $2000$ trees have been
built.

The Tuned Random Forest baseline improves on the Random Forest baseline by
using an optimized max_features value. The max_features hyperparameter defines
how many random features are considered when determining each split and is
found to be the most important hyperparameter (Van Rijn and Hutter,
2018).999The hyperparameter min_samples_leaf is statistically equally
important. The value is optimized by evaluating up to $11$ unique values for
the hyperparameter with $5$-fold cross-validation before training a final
model with the best found value. The Tuned Random Forest is our strongest
baseline and aims to mimic a typical first approach for modeling by a human.

##  4 Software

We developed an open source benchmark tool that may be used for reproducible
AutoML benchmarking, which we will refer to as AMLB.101010Source code and
documentation under MIT license at https://github.com/openml/automlbenchmark.
This tool features robust automated experiment execution and supports multiple
AutoML frameworks, many of which are evaluated in this paper. AMLB is
implemented as a Python application consisting mainly of an _amlb_ module and
a _framework_ folder hosting all the officially supported extensions, which
have been developed together with AutoML framework developers. The main
consideration for the design of AMLB is to produce correct and reproducible
evaluations; the AutoML frameworks are used as intended by their authors with
little to no room for user error, and the same evaluation conditions
(including framework version, data set, and resampling splits) and controlled
computational environments can easily be recreated by anyone. The _amlb_
module provides the following features:

  * •

a data loader to retrieve and prepare data from _OpenML_ or local data sets.

  * •

various benchmark runner implementations:

    * –

a _local_ runner that runs the experiments directly on the local machine. This
is also the runner to which each runner below delegates the final execution.

    * –

_container_ runners (_docker_ and _singularity_ are currently supported),
which allow preinstalling the _amlb_ application together with a full setup of
one framework and consistently run all benchmark tasks against the same setup.
This implementation also makes it possible to run multiple container instances
in parallel.

    * –

an _aws_ runner that allows the user to safely run the benchmark on several
EC2 instances in parallel. Each EC2 instance can itself use a pre-built docker
image, as used for this paper, or can configure the target framework on the
fly, which is useful for experiments in a development environment.

  * •

a job executor responsible for running and orchestrating all the tasks. When
used with the _aws_ runner, this allows distribution of the benchmark tasks
across hundreds of EC2 instances in parallel, with each one being monitored
remotely by the host.

  * •

a post-processor responsible for collecting and formatting the predictions
returned by the frameworks, handling errors, and computing the scoring metrics
before writing the information needed for post-analysis to a file.

Figure 13 in Appendix C provides an architecture overview and description of
the flow of the benchmarking tool, as used in the experiments for this paper.

###  4.1 Extensible Framework Structure

To ensure that AMLB is easily extensible to new AutoML frameworks, we
integrate each tool through a minimal interface. Each of the current tools
requires less than 250 lines of code across at most four files (most of which
is boilerplate). The integration code handles installation of the AutoML
framework as well as its software stack and provides the framework with data
and recording predictions. The integration requirements are minimal, as both
input data and predictions can be exchanged both in Python objects and common
file formats, which makes integration across programming languages possible
(currently integrated frameworks are written in C#, Java, Python and R). By
keeping the integration requirements minimal, we hope that AutoML framework
authors are encouraged to contribute integration scripts for their framework
and, at the same time, avoid influencing the methods or software used to
design and develop new AutoML frameworks, as opposed to providing a generic
starter kit which may bias the developed AutoML frameworks (Guyon et al.,
2019). Frameworks may also be integrated completely locally to allow for
private benchmarking.111111For information on how to add a framework, see
https://openml.github.io/automlbenchmark/docs/extending/framework/.

###  4.2 Extensible Benchmarks

Benchmark suites define the data sets and one or more train/test splits, which
should be used to evaluate the AutoML frameworks. AMLB can work directly with
OpenML tasks and suites (Bischl et al., 2021), allowing for new evaluations
without further changes to the tool or its configuration. This is the
preferred way to use AMLB for scientific experiments, as it guarantees that
the exact evaluation procedure can be reproduced easily by others. However, it
is also possible to use data sets stored in local files with manually defined
splits, e.g., to benchmark private use cases.121212For information on how to
add a new benchmark task or suite, see
https://openml.github.io/automlbenchmark/docs/extending/benchmark/.

###  4.3 Using the Software

To benchmark an AutoML framework, the user must first identify and define:

  * •

the _framework_ against which the benchmark is executed,

  * •

the _benchmark_ suite listing the tasks to use in the evaluation, and

  * •

the _constraints_ that must be imposed on each task. This includes:

    * –

the maximum training time.

    * –

the number of CPU cores that can be used by the framework; not all frameworks
respect this constraint, but when run in _aws_ mode, this constraint
translates to specific EC2 instances, therefore limiting the total number of
CPUs available to the framework.

    * –

the amount of memory that can be used by the framework; not all frameworks
respect this constraint, but when run in _aws_ mode, this constraint
translates to specific EC2 instances, therefore limiting the total amount of
memory available to the framework.

    * –

the amount of disk volume that can be used by the framework (only respected in
_aws_ mode).

Those constraints must then be declared explicitly in a _constraints.yaml_
file (also in the _resources_ folder or as an external extension).

####  4.3.1 Commands

Once the previous parameters have been defined, the user can run a benchmark
on the command line using the basic syntax:

⬇

$ python runbenchmark.py framework_id benchmark_id constraint_id

For example, to evaluate the tuned random forest baseline on the
classification suite for 1 hour on 8 cores, run:

⬇

$ python runbenchmark.py tunedrandomforest openml/s/271 1h8c

Additional options may be used to specify the _mode_ , the _parallelization_ ,
and other details of the experimental setup. For example, the following
command may be used to evaluate the random forest baseline on the regression
benchmark suite across 100 8-core _aws_ instances in parallel with a time
budget of one hour.

⬇

$ python runbenchmark.py randomforest openml/s/269 1h8c -m aws -p 100

##  5 Benchmark Design

In this section, we discuss both the design of the benchmark suite (that is,
the chosen data sets and evaluation procedures, Bischl et al., 2021) and the
experimental setup, as well as their limitations.

###  5.1 Benchmark Suites

To facilitate a reproducible experimental evaluation, we make use of OpenML
Benchmark suites (Bischl et al., 2021). An OpenML benchmark suite is a
collection of OpenML tasks, which each reference a data set, an evaluation
procedure (such as k-fold cross-validation) and its splits, the target
feature, and the type of task (regression or classification). The benchmark
suites are designed to reflect a wide range of realistic use cases for which
the AutoML tools are designed. Resource constraints are not part of the task
definition. Instead, we define them separately in a local file so that each
task can be evaluated with multiple resource constraints. Both the OpenML
benchmark suite (and tasks) and the resource constraints are machine-readable
to ensure automated and reproducible experiments.

####  5.1.1 Data Sets

We created two benchmarking suites, one with 71 classification tasks, and one
with 33 regression tasks. The data sets used in these tasks are selected from
previous AutoML papers (Thornton et al., 2013), competitions (Guyon et al.,
2019), and ML benchmarks (Bischl et al., 2021) according to the following
predefined list of criteria131313The airlines datasets violate these criteria,
but are included for historical reasons.:

  * •

Difficulty of the data set must be sufficiently high. If a problem is easily
solved by almost any algorithm, it will not be able to differentiate the
various AutoML frameworks. This can mean that simple models (such as random
forests, decision trees or logistic regression) achieve a generalization error
of zero, or that the performance of these models and all evaluated AutoML
tools is identical.

  * •

Representative of real-world data science problems to be solved with the
frameworks. In particular, we limit artificial problems. We included a small
selection of such problems, either based on their widespread use (e.g., kr-vs-
kp) or because they pose difficult problems, but we do not want them to
constitute a large part of the benchmark. We also limit computer vision
problems on raw pixel data because those problems are better solved with
dedicated deep learning solutions. However, since they still make for real-
world, interesting, and hard problems, we did not exclude them altogether.

  * •

No free form text features that cannot reasonably be interpreted as a
categorical feature. Most AutoML frameworks do not yet support feature
engineering on text features and will process them as categorical features.
For this reason, we exclude text features, even though we admit their
prevalence in many interesting real-world problems. A first investigation and
benchmark of multimodal AutoML with text features has been carried out by Shi
et al. (2021).

  * •

Diversity in the problem domains. We do not want the benchmark to skew towards
any application domain in particular. There are various software quality
problems in the OpenML-CC18 benchmark (jm1, kc1, kc2, pc1, pc3, pc4), but
adopting them all would lead to a bias in the benchmark to this domain.

  * •

Independent and identically distributed (i.i.d.) data are required for each
task. If the data are of temporal nature or repeated measurements have been
conducted, the task is discarded. Both types of data are generally very
interesting but are currently not supported for most AutoML systems, and we
plan to extend the benchmark in the future in this direction.

  * •

Freely available and hosted on OpenML. Data sets that can only be used on
specific platforms or are not shared freely for any reasons are not included
in the benchmark.

Reasons to exclude a data set included label-leakage and near-duplicates of
other tasks in independent variables (for example, different only in
categorical encoding or imputation) or dependent variable (most commonly the
binarization of a regression or multi-class task).

To study the differences between AutoML systems, the data sets vary in the
number of samples and features by orders of magnitude and vary in the
occurrence of numeric features, categorical features, and missing values.
Figure 1 shows basic properties of the classification and regression tasks,
including the distributions of the number of instances and features, the
frequency of missing values and categorical features, and the number of target
classes (for classification tasks). Properties of the tasks are shown in
Appendix A and can be explored interactively on OpenML.141414Visit
www.openml.org/s/269 for regression and www.openml.org/s/271 for
classification. While the selection spans a wide range of data types and
problem domains, we recognize that there is room for improvement. Restricting
ourselves to open data sets without text features severely limits options,
especially for big data sets. We hope to address this in future work.

Figure 1: Properties of the tasks in both benchmarking suites.

All data sets are available in multiple formats for the AutoML frameworks,
either as files (parquet, arff, or csv) or as Python objects (pandas
dataframe, numpy array), and the used format depends on the AutoML framework.
All frameworks have access to meta-data, such as the data type of the columns,
either directly from the chosen data format or as separate input, so that each
AutoML framework has the same information available regardless of the chosen
data format.

####  5.1.2 Performance Metrics

In our evaluation, we use area under the receiver operating characteristic
curve (AUC) for binary classification, log loss for multi-class classification
and root mean-squared error (rmse) for regression to evaluate model
performance.151515We use the implementations provided by SCIKIT-LEARN 1.2.2.
We chose to use these metrics because they are generally reasonable, commonly
used in practice, and supported by most AutoML tools. The latter is especially
important because it is imperative that AutoML systems optimize for the same
metric on which they are evaluated. However, our tool is not limited to only
these three metrics, and a wide range of performance metrics can be specified
by the user.

Models are not calibrated, unless the AutoML framework does so by default.
Whether or not calibrated models are important is highly dependent on the use
case, and most AutoML frameworks do not support model calibration out of the
box. This is especially important to keep in mind when we discuss results on
our binary classification problems, as AUC is not sensitive to calibration.

####  5.1.3 Missing Values In Experimental Results

As will be discussed in more detail in Section 6.4, not all frameworks are
equally well-behaved. There are situations where search time budgets are
exceeded or the AutoML frameworks crash outright, which results in missing
performance estimates. There are multiple strategies to consider on how to
deal with these missing data.

One naive approach may be to ignore missing values and aggregate over the
obtained results. However, we observe that failures do not occur at random.
Failures correlate with data set properties, such as data set size and class
imbalance, which may be correlated with “problem difficulty” and thus
performance. Ignoring missing values thus means that AutoML frameworks may
fail on harder tasks or folds and consequently obtain higher average
performance estimates. Imputing missing values with performance obtained by
the same AutoML framework on other folds is subject to the same drawback.
Moreover, in case a framework fails to produce predictions on all folds of a
task, neither method specifies how to deal with missing values.

Instead, we propose to impute the missing values with an interpretable and
reliable baseline. An argument may be made for using the random forest
baseline, since this may be a strong fallback that AutoML frameworks could
realistically implement. However, we observe that training a random forest (of
the size used in the baseline) requires a significant amount of time on larger
data sets. Automatically providing this fallback by means of imputation would
provide an unfair advantage to the AutoML frameworks that are not well-
behaved. Moreover, many failures would not be remedied by having a random
forest to fall back on, since the AutoML frameworks crash irrecoverably, for
example, due to segmentation faults.

Therefore, we impute missing values with the constant predictor, or
prior.161616We except one specific error for AUTOGLUON(HQIL) which is already
fixed (c.f. Appendix D). This baseline returns the empirical class
distribution for classification and the empirical mean for regression. This is
a very penalizing imputation strategy, as the constant predictor is often much
worse than results obtained by the AutoML frameworks that produce predictions
for the task or fold. However, we feel this penalization for ill-behaved
systems is appropriate and fairer towards the well-behaved frameworks and hope
that it encourages a standard of robust, well-behaved AutoML frameworks.

###  5.2 Experimental Setup

We execute the experiments on commodity-level hardware with AutoML frameworks
generally in their default configurations.

####  5.2.1 Hardware

As discussed in Section 4, AMLB can be run on any machine. However, for
comparable hardware and easy expandability, we opt to conduct the benchmark on
standard m5.2xlarge171717More information is available at
https://aws.amazon.com/ec2/instance-types/m5/. instances available on Amazon
Web Services (AWS). These represent current commodity-level hardware with $32$
GB memory, $8$ vCPUs (Intel Xeon Platinum $8000$ series Skylake-SP processor
with a sustained all core Turbo CPU clock speed of up to 3.1 GHz). $100$ GB of
gp3-SSD storage is available for storage, which can be necessary for storing a
larger number of evaluated pipelines. The use of AWS also enables others to
fully reproduce and extend our results, as funding permits, since the results
do not depend on private computing infrastructure.

####  5.2.2 Framework Configuration

AutoML frameworks are instantiated with their default configuration, except
that we control the following settings:

  * •

‘Mode’ to declare the user intent. For example, obtaining the best possible
model versus finding an interpretable (less complex) model. The modes used to
evaluate each AutoML framework is chosen by their developers.

  * •

Runtime for the search. Additionally, there is one hour leeway for data
loading, making predictions, and cleanup operations, but this is not
communicated to the AutoML frameworks.

  * •

Resource constraints that specify the number of CPU cores and amount of memory
available.

  * •

Target metric to use for optimization. This is the same metric that is used
for evaluation in the benchmark.

The experiment design intentionally prohibits further customization of other
AutoML system configuration parameters to reflect how these systems are
usually applied in practice as closely as possible (note that AMLB does allow
for this type of customization). An overview of used framework versions, their
‘mode’ configurations, and important integration details are provided in
Appendix C.2.

###  5.3 Limitations

Both the design of the benchmark and the setup for the experiments described
in this paper have some limitations with regard to the interpretation of their
results. Limitations in the design stem from the desire to keep the use of the
frameworks as close as possible to the original vision and usage intended by
developers, whereas the limitations in the experiments are caused by resource
constraints and may be alleviated by running additional experiments with the
benchmark software. In this section, we highlight some important limitations
and stress that this paper and the results within do _not_ state which AutoML
framework is ultimately the best.

####  5.3.1 Limitations of the Design

Perhaps the biggest limitation of the design is the inability to attribute the
performance of an AutoML framework to any one aspect of its design, as is
often done with ablation studies. The evaluated AutoML frameworks differ among
multiple design choices, such as the underlying ML library, search space,
preprocessing, and search algorithm. Concretely, a performance difference
between, for instance, AUTO-SKLEARN and TPOT could be caused by TPOT’s built-
in stacking, AUTO-SKLEARN’s ensembles, the difference in Bayesian optimization
versus genetic programming, the difference in how multiprocessing is employed,
or a combination of these or any other difference between them. Software that
would allow for such conclusions essentially requires each AutoML framework to
be reimplemented on a shared set of algorithms for building models, search,
and evaluation. We acknowledge that this would be incredibly valuable for the
research community. However, it would also no longer resemble the software as
used in practice and thus would be different work altogether. Note that it
_is_ possible to perform ablation studies with AMLB for a specific AutoML
framework, for example by comparing different framework configurations as done
by Erickson et al. (2020).

Another limitation stems from only recording results produced by the final
model. Anytime performance, where information about the performance during
optimization is captured as if they were final models, can be very insightful.
This method allows for the distinction between a framework that converges
quickly from one that does not. This may be especially important for users who
are interested to use the systems with a human-in-the-loop, such as when
designing a search space or data features. Unfortunately, many frameworks do
not support collection of anytime performance and— depending on how they are
recorded—might interfere with resources used during search. We hope to be able
to record anytime performance in the future, but in this work, we only
approximate it by evaluating the tools under two different time constraints
($1$ and $4$ hours).

Finally, the qualitative comparison of the frameworks is also limited. Certain
”quality of life features” like analysis of the pipeline via interpretable ML
methods, reports, usability, or support are not evaluated here but are
important to many users. For a qualitative analysis of those characteristics,
we refer the reader to one of the many existing overview papers on AutoML
(Zöller and Huber, 2021; Truong et al., 2019). We also provide an overview of
various AutoML frameworks with links to their documentation on our
website.181818https://openml.github.io/automlbenchmark/frameworks.html

####  5.3.2 Limitations of the Experiments

Most frameworks are highly configurable and allow the user to configure the
search algorithm or its hyperparameters, among other aspects that affect the
AutoML performance. Some frameworks even provide different configuration
presets for different use cases, such as a performance-oriented competition
mode and a mode that produces fast or interpretable models at the cost of some
performance. However, comparing the effect on model performance of tuning
AutoML hyper-hyperparameters quickly carries prohibitive costs. We evaluated
only a few modes for each framework, if any, based on the developers’
recommendations. It is likely that better results may be achieved by carefully
meta-tuning the AutoML framework or evaluating additional modes. While it is
cost prohibitive for us to evaluate many different scenarios, it is easy to
run the benchmark with custom configurations for the various AutoML
frameworks. This allows users to evaluate AutoML frameworks in a setting that
reflects their interest.

####  5.3.3 Meta-learning

Many AutoML frameworks make use of meta-learning to better initialize and
speed up the search (Yang et al., 2018; Feurer et al., 2015a, 2020). Since all
data in the benchmark is publicly available and many of them are well known in
the AutoML community, it is likely that there is a substantial overlap between
data used by the developers for meta-learning and the data used in the
benchmark. This is a very intricate problem, as we consider AutoML frameworks
as black boxes. Removing the effect of the data set that is to be evaluated
from the meta-learning procedure is not solvable in general.

In this paper specifically, both AUTO-SKLEARN and AUTO-SKLEARN 2 use meta-
learning.
AUTO-SKLEARN’s meta-learning uses $208$ data sets from OpenML, each associated
with well-performing ML pipelines. The search is initialized with $k$-nearest
datasets (Reif et al., 2012) using the $25$-nearest datasets based on $38$
meta-features and looking up the best known pipeline for those datasets. AUTO-
SKLEARN can exclude data sets by name from the lookup, which we make use of in
AMLB to ensure that there is no overlap to the $39$ data sets from Gijsbers et
al. (2019). Even so, it cannot be guaranteed that identical data sets with a
different name might be used for meta learning out-of-the-box.

AUTO-SKLEARN 2’s meta-learning model is more complicated and consists of: a) a
static pipeline portfolio for warm-starting the search, which is computed
across hundreds of data sets using a greedy forward selection, and b) a meta-
model to predict the internal model selection strategy and budget allocation
strategy. Single data sets cannot be excluded from these meta-learning
procedures, and it is not feasible to retrain the meta-models and pipeline
portfolio for each data set in our benchmark. This ultimately means that the
result of AUTO-SKLEARN 2 must be considered very carefully and are likely
optimistic, as their meta-model is built with information about many datasets
in this benchmark. More thought and collaboration is required to address these
issues and allow for the correct evaluation of AutoML systems that use meta-
learning in common benchmarks.

###  5.4 Overfitting the Benchmark

One last issue that plagues any widely-adopted benchmark is the potential of
algorithms overfitting on the data sets used in the benchmark. Since freely
available, interesting, and usable (refer to Section 5.1.1 for our selection
criteria) data sets are scarce, many AutoML developers use these data sets to
benchmark and then improve their systems iteratively. While this is not as
direct of an issue as with meta-learning, these data sets can in general not
be assumed to be truly unseen. The only practical way to avoid this is to
collect a novel set of data sets for the benchmark, which would entail a
prohibitive effort. Moreover, after publishing such a benchmark, the new data
sets are published, which again gives developers the possibility to use them
to improve their systems. On the other hand, should the benchmarking data sets
be kept private to avoid this issue, the benchmark is no longer entirely
reproducible by independent researchers. We hope that the size of our
benchmarking suites is large enough and their design general enough that
overfitting is less of an issue, but this is difficult to guarantee, and a
study as outlined above may be useful to evaluate this phenomenon in the
future.

##  6 Results

In this section, we provide an overview and analysis of the results obtained.
This section is accompanied by an interactive visualization tool191919Results
can be interactively explored at
http://openml.github.io/automlbenchmark/visualization., additional information
in Appendix B, and all data artifacts generated from these
experiments.202020Experiment artifacts can be found at:
http://openml.github.io/automlbenchmark/data. For a more comprehensive
comparison than what we can provide here, we strongly encourage the reader to
explore the data with the interactive visualization tool. This tool enables
users to select any subset of frameworks, task types, performance measures, or
data characteristics iteratively and interactively. The tool includes overview
plots for different task types as well as detailed visualizations for
individual data sets. Moreover, a statistical analysis of the results by
critical difference plots and Bradley-Terry trees is implemented.

In addition to the limitations outlined in section 5.3, we also want to stress
that these experimental results are obtained by running the frameworks as they
were in June of 2023. Some of these frameworks are still under active
development, and results from experiments run on later versions will almost
certainly differ.

We used recent results as much as possible, but we also include some results
of experiments ran of frameworks in September of 2021. In these cases, the
benchmarked framework versions were not substantially different. We motivate
this decision in Appendix C.2 and denote the older results with an asterisk
(*) by the framework name. For example, TPOT denotes the June 2023 version and
TPOT* denotes the September 2021 version. Some results are missing: AUTO-
SKLEARN 2 does not support regression, and we could not complete our 4 hour
classification evaluation of AUTOGLUON(HQIL) due to budget constraints.

We strongly encourage people to run additional experiments that match their
use case with up-to-date frameworks, and use the results in this section as a
reference on how to analyze the results. Results may differ substantially for
different time constraints, amount of parallel processing available,
availability of GPU’s, and more. Table 16 in Appendix C.2 shows an overview of
the versions benchmarked for each framework as well as the most current
version.

###  6.1 Predictive Performance

To report on the results for many AutoML frameworks across whole benchmarking
suites, we propose using critical difference (CD) diagrams (Demšar, 2006)
illustrated in Figure 2. In a CD diagram, the average rank of each framework
as well as which ranks are statistically significantly different from each
other are shown. To calculate the average rank per task, we first impute any
missing values with the constant predictor and then average the performance
over all folds. We may then test for the presence of statistically significant
differences in the average rank distributions using a non-parametric Friedman
test at $p<0.05$ (here, $p\approx 0$ for every diagram) and use a Nemenyi
post-hoc test to find which pairs differ. For each benchmarking suite and time
budget, the CD diagrams are shown in Figure 2, which displays the rank of each
framework (lower is better) averaged over all results from the given
benchmarking suite and budget. AUTO-SKLEARN 2 is excluded in this comparison
due to the meta-learning issues discussed in Section 5.3.3.

(a) Binary Classification, 1 hour

(b) Binary Classification, 4 hours

(c) Multi-class Classification, 1 hour

(d) Multi-class Classification, 4 hours

(e) Regression, 1 hour

(f) Regression, 4 hours

(g) All tasks, 1 hour

(h) All tasks, 4 hours

Figure 2: CD plots with Nimenyi post-hoc test after imputing missing values
with the constant predictor baseline. An asterisk (*) next to a framework name
means the results were obtained in 2021.

Overall, we observe that AUTOGLUON(B) and TPOT respectively achieve the best
and worst rank among AutoML frameworks in almost every setting with respect to
model accuracy. However, while AUTOGLUON(B) is generally not significantly
better than the second best framework, it does rank statistically
significantly better than the worst AutoML frameworks in any given scenario.
Similarly, TPOT is never significantly worse than the second worst framework,
but its rank is always statistically significantly worse than AUTOGLUON(B). In
all cases, the baselines obtain worse ranks than any AutoML framework except
TPOT, although the tuned random forest is a strong baseline. Only AUTOGLUON(B)
and LIGHTAUTOML achieve significantly better ranks across all settings. All
AutoML frameworks except AUTOGLUON(B) and TPOT are generally ranked close to
each other, with small differences across the various suites and budgets.

Figure 3: Boxplots of framework performance across tasks after scaling the
performance values from random forest (0) to best observed (1). The number of
outliers for each framework that are not shown in the plot are denoted on the
x-axis. An asterisk (*) next to a framework name means the results were
obtained in 2021.

To complement the CD diagrams—which obfuscate the relative performance
differences—we show box plots of obtained results (after imputation) across
all tasks in Figure 3. Because the performances are not commensurable across
tasks, we first scale all results per task between the random forest
performance (0) and the best observed performance (1) (that is, higher scores
correspond to better performance). This also makes the scaled value
interpretable. Furthermore, the value scales based on the improvement over the
baseline that is observed to be achievable, and negative values are worse than
the random forest baseline. While the boxplots are calculated over performance
data on all tasks, the plots are cut off to allow a better visualization of
the most relevant area. The number of outliers for each framework that are not
shown in the plot are denoted on the x-axis.

Even if ranks are similar, the performance distribution might be noticeably
different. For example, AUTOGLUON(HQIL) and MLJAR(P) achieve very similar
average ranks on the one hour regression tasks. However, we observe from the
boxplots that while AUTOGLUON(HQIL) achieves lower median normalized
performance in this segment, it outperforms random forest with much greater
consistency than MLJAR(P). Similarly, while TPOT’s average rank is generally
close to that of the Tuned Random Forest baseline, TPOT exhibits much higher
variance in its prediction quality.

Finally, we take a closer look at the difference in performance for different
time constraints in Figure 4. Performance is scaled between 1 hour random
forest performance (0) and best observed performance (1). We only include
frameworks for which we evaluated all tasks on both constraints. We see that
the performance is very similar overall, though generally results improve
slightly with a larger time budget. The performance of AUTOGLUON confirms that
better performance is still possible for other frameworks, but we suspect that
those frameworks are limited by their search space. For example, if a stacking
ensemble is required to reach the best performance, then the other methods
will not achieve the best performance regardless of time constraint.

Figure 4: Scaled performance for each framework under different time
constraints. Only frameworks which have evaluations on all tasks for both time
constraints are shown. Performance generally does not improve much with more
time. Figure 5: Bradley-Terry tree of depth three for classification tasks.
Results from the one hour classification benchmark were used, and missing
values were imputed by constant predictor performance. One observation within
the BT tree equals the preference ranking of one fold on one data set. The $n$
value denotes the number of observations in the leaf node.

###  6.2 Bradley-Terry Trees

Bradley-Terry (BT) trees (Strobl et al., 2011) can be used to statistically
analyse benchmark experiments based on data set characteristics (Eugster et
al., 2014). These trees use data set characteristics—such as the number of
instances, the number of features, the ratio of missing values, and others—to
split paired performance comparisons of the framework to find statistically
significant differences in performance. Bradley-Terry models originate in
psychology to analyze paired comparison experiments of subjects preferring one
stimulus over another. For our benchmark, such a preference ranking can be
easily derived by pairwise performance comparisons of all frameworks with
regard to the data sets and cross-validation folds.

The underlying algorithm of model-based recursive partitioning of
Bradley–Terry models works as follows: In each split of the BT tree, a BT
model is fitted for the paired comparisons based on the underlying data set
characteristics. Following Zeileis and Hornik (2007) and Eugster et al.
(2014), the BT model performs a statistical test of parameter instability for
the chosen data characteristics. If this test reveals a significant
instability in the model parameters, the corresponding tree node splits the
data according the the characteristic yielding the highest instability (lowest
test p-value). The splitting cut-point is then determined such that it has the
highest improvement of the model fit. This procedure is repeated until either
no significant instability is left, a set tree depth is reached, or further
splits would exceed a set minimum number of observations in the leaves.

Numeric values in the tree leaves are _worth parameters_ that can be
interpreted as preferences for the different frameworks (Eugster et al.,
2014). Since these values are in $[0,1]$ and sum up to $1$ within a leaf, they
can be understood as the probability of a framework performing best, given the
data characteristics in the corresponding leaf.

Figure 5 shows a Bradley-Terry tree for classification tasks for a runtime of
one hour. For simplicity, in order to obtain an easily understood tree, only
the number of instances (rows) and the number of features were chosen as data
characteristics. The first split distinguishes between data sets with more
than $8237$ instances and those equal to or below that cut-point. The second
split then further split into leaves based on the number of features in the
datasets. In the nodes, from left to right, we see that:

  * •

For small datasets with a relatively low dimensionality (Node 3, 32 tasks) the
performance differences among the best frameworks are not very pronounced.
Even though AUTOGLUON(B) is preferred for those kinds of data sets,
AUTOGLUON(HQ) and AUTO-SKLEARN 2 show similar performances.

  * •

For very wide datasets, those with a similarly small number of instances but
much higher dimensionality, more than 1301 features (Node 4, 5 tasks),
performance differences increase overall with AUTOGLUON(B) still being
preferred over all others, but now followed by AUTOGLUON(HQ) and LIGHTAUTOML
with similar performances, and larger distance to H2O AUTOML, GAMA(B) and
FLAML.

  * •

For datasets with a larger number of instances, more than 8237 (Node 6 and 7),
we see that AUTOGLUON(B) is preferred by a large margin over AUTOGLUON(HQ) as
second. There are minor differences depending on the number of features, such
as FLAML being a preferred third option when there are few features, at most
43, but AUTOGLUON(HQIL) being preferred if the dataset has a relatively higher
dimensionality.

More Bradley-Terry trees for the subsets of binary and multiclass
classification as well as regression can be found in the appendix. The
findings from the BT trees are essentially the same as those presented here,
as AUTOGLUON(B) is overall the preferred framework in most tree leaves. One
exception is for small balanced binary classification tasks, shown in Figure
10, where various frameworks are preferred over AUTOGLUON.

In conclusion, it can be observed that the performance gap of AUTOGLUON(B)
increases particularly as data sets become more complex. Moreover, the reader
is strongly invited to explore the aforementioned interactive visualization
tool, with which deeper BT trees based on several more data set
characteristics can be constructed on various task types.

###  6.3 Model Accuracy vs. Inference Time Trade-offs

Model accuracy (here measured by AUC, log loss, or RMSE) plays a central role
in evaluating performance of machine learning models. However, maximizing
accuracy can come at the cost of added model complexity. One practical way to
consider the complexity of the model is to measure the inference speed of the
resulting model.

Some of the integrated frameworks offer a “compete” mode (AUTOGLUON, MLJAR,
LIGHTAUTOML and GAMA) that maximizes accuracy, typically at the cost of
increased model complexity, similar to competing in a Kaggle competition. This
can lead to models being built that are highly accurate but are extremely slow
at inference time and are therefore not practical in many real-life use-cases.

However, some frameworks provide multiple presets that allow the user to
balance the trade-off between accuracy and inference time differently. Unless
the use of a specific preset is mentioned (for AUTOGLUON and MLJAR), results
in this section used presets which prioritize accuracy over inference time, or
in the case of H2O AUTOML, a balance of the two. Thus, performing additional
experiments with other presets is advised when inference time is important. We
also recognize that there are applications for which inference time is not
important.

In order to evaluate the limitations of the models produced by each framework
we also measured their inference time. The inference time reported on in this
section is measured by having the frameworks infer on samples of test data,
either on a single row which is in memory, or on ten thousand rows loaded from
disk. Each measurement is done ten times and the median is reported.
Measurements for TPOT are omitted from this section, as TPOT requires data
which is already encoded and the encoding step may consume a considerable part
of the total inference time, making direct comparisons invalid. This metric
provides important insight into the trade-offs that tool authors make in their
algorithm designs.

Figure 6 shows aggregated inference times across all models standardized to
rows per second (fewer is slower). For H2O AUTOML we do not report in-memory
inference speed as technical limitations of our benchmarking tool result in
pessimistic inference time measurements for non-Python frameworks.212121
Transferring data between our Python-based benchmark tool and H2O AUTOML’s
Java back-end requires serialization to disk. Serialization to disk occurs
significant overhead, especially when comparing to in-memory inference. Faster
inference speeds will be obtained by using H2O Frames directly in H2O AUTOML,
or using H2O in it’s production mode, which performs inference on exported
models directly from disk instead of in memory.  Here we see that the high
accuracy of AUTOGLUON(B) comes at the cost of extremely slow inference times.
This is explained by the large models produced by combining both stacking and
ensembling to form ensembles of multiple layers. In general, we see that
ensemble models are slower (AUTO-SKLEARN, GAMA, and MLJAR(B) also do
ensembling), and GAMA and AUTO-SKLEARN with the same search space and ensemble
methods have comparable inference speeds. H2O AUTOML and AUTOGLUON(HQIL)
maintain fast inference speed despite making use of ensembles, by building on
more optimized models or explicitly selecting for fast inference when building
the ensemble. FLAML stand out as having very fast inference time, a potential
explanation is that FLAML’s cost-frugal optimization also indirectly takes
inference time into account, as it is part of the time estimate which is used
to consider which model to tune.

Figure 6: Inference speed on unseen data in rows per second, normalized from
inference on 10 thousand rows from solid-state drive (left) or single rows in
memory (right). Measured for models created with the one hour time constraint
and all scenarios.

In Figure 7, we show the Pareto front for all three scenarios with a one-hour
time constraint, demonstrating the average normalized model accuracy against
corresponding median per-row prediction speeds. Here, it is more apparent that
the frameworks that achieve the highest accuracy do so at the cost of
inference time performance. This demonstrates that when contextualizing any
type of model accuracy results, it is important to consider any trade-offs
that may have been made to achieve the extra performance and how that will
affect the framework’s usability in practice. In this case, measuring accuracy
in isolation does not give the complete picture of the overall utility of a
particular framework.

It should also be noted that with sufficient computing infrastructure and
effort, scoring across rows or chunks of data could be parallelized in a
production system, which would reduce the overall prediction time as compared
to predicting a single test set on a single machine. However, our goal in this
section was to compare the inference time of the high-accuracy models derived
from different AutoML frameworks to each each other rather than evaluate
different techniques for speeding up the inference of any individual system.

Figure 7: Pareto Frontiers of framework performance across tasks after scaling
the performance values from the random forest (0) to best observed (1) for
each task type on a one hour time budget.

###  6.4 Observed AutoML Failures

While most jobs completed successfully, we observed multiple framework errors
during our experiments. In this section, we will discuss where AutoML
frameworks fail, although we want to stress that development for these
packages is ongoing. For that reason, it is likely that the same frameworks
will not experience the same failures in the future (especially after gaining
access to all experiment logs). We categorize the errors into the following
categories:

  * Memory: The framework crashed due to exceeding available memory or encountering other memory-related errors, such as segmentation faults.

  * Time: The framework exceeded the time limit past the leniency period.

  * Data: Errors due to specific data characteristics (such as imbalanced data) occurred.

  * Implementation: Any errors caused by bugs in the AutoML framework code occurred.

These categories are a bit crude and ultimately subjective, since from a
reductive viewpoint, all errors are implementation errors. However, they serve
for a quick overview. We also introduce a ‘fixed’ category in Figure 8(a) to
denote errors from a specific bug in AUTOGLUON(HQIL) which is already fixed in
newer releases. Additional details on this, and other errors encountered, can
be found in Appendix D.

Figure 8(a) shows the errors by type, and Figure 8(b) shows errors by dataset
size and dimensionality on the right. Overall, memory and time constraints are
the main cause for errors, with one major exception.222222 MLJAR(B) has 284
‘implementation errors’ which are almost exclusively caused by only 3 distinct
errors.  We observe that errors are far more common in the classification
benchmark suite than the regression suite. This is largely accounted for by
the difference in benchmarking suite size (33 and 71 tasks) and the fact that
the largest data sets are mostly classification tasks, both in number of
instances and features. Unique to classification, we do observe several
frameworks failing to produce models or predictions on highly imbalanced data
sets. This is also the case for the failures on the two small classification
data sets (‘yeast’ and ‘wine-quality-white’), where careless use of internal
validation splits yields splits that no longer contain all classes.
Interestingly, the distribution of the type of errors observed is different
under different time constraints (as shown in Figure 14(a) of Appendix D).
Both memory and time constraint violations happen more frequently, which may
potentially be explained by frameworks saving increasingly more models or
building increasingly larger pipelines. It is worth noting the stability of
the different systems is varied, with some systems being far more stable than
others. Some types of errors, such as memory errors, could potentially be
easily resolved by running the AutoML system on a machine with more RAM.
However, the implementation errors are more problematic because they represent
an error from the code of the AutoML system. Those are not currently
resolvable without changes to the AutoML system itself, which may or may not
be fixed by code authors in the future.

(a) Errors by type for each framework.

(b) Number of errors by size of the dataset.

Only when the framework exceeds the time budget by more than one hour do we
record a time error. However, as we can see in Figure 9, not all AutoML
frameworks adhere to the runtime constraints equally well, even if they finish
within the leniency period. In the figure, the training duration for each
individual job (task and fold combination) are aggregated, and timeout errors
are shown above each framework, where missing values due to non-time errors
are excluded. These plots reveal different design decisions around the
specified runtime, with some frameworks never exceeding the limit by more than
a few minutes, while others violate it by a larger margin with some
regularity. Interestingly, we see that a number of frameworks consistently
tend to stop far before the specified runtime limit.

Figure 9: Time spent during search with a one hour budget (left) and four hour
budget (right). The grey line indicates the specified time limit, and the red
line denotes the end of the leniency period. The number of timeout errors for
each framework are shown beside it.

##  7 Conclusion

We presented a novel benchmark for measuring and comparing AutoML frameworks.
To ensure reproducibility, fair comparison, and detailed analysis of the
results, our open-source benchmarking tool automates the empirical evaluation
of any integrated framework on any supported task, including installation of
the AutoML framework, provisioning the data for training and inference,
resource allocation, and processing of the results. This greatly simplifies
evaluating AutoML frameworks while enhancing reproducibility and reducing
errors. We worked jointly with the authors of 9 AutoML frameworks to evaluate
their systems in a large-scale study on 71 classification and 33 regression
tasks.

When analyzing the predictive performance of these AutoML systems, we find
that the average ranks of the AutoML frameworks are generally very competitive
with each other. Still, by using Bradley-Terry trees (Strobl et al., 2011), we
find that their relative performance is affected by data characteristics—such
as the data set size, dimensionality, and class balance—highlighting specific
strengths and weaknesses. Overall, in terms of model performance, AUTOGLUON
consistently has the highest average rank in our benchmark. Additionally, in
most scenarios, the AutoML frameworks outperform even our strongest baseline.

Because inference time is an important factor in real-world applications, we
also reviewed the inference time and accuracy trade-off and found large
differences in inference time of the produced models, at times spanning
multiple orders of magnitude. The most accurate frameworks achieve higher
model accuracy at a large cost to performance in terms of inference speed.
Broadly speaking, while the models with higher accuracy also have slower
inference time, not all frameworks produce models that are Pareto optimal.
Furthermore, specific AutoML frameworks allow users to choose presets for
customizing the accuracy-inference time trade-off.

Finally, we analyzed scenarios in which AutoML frameworks fail to produce a
model and found that the main cause for failure was data set size. In other
words, not all methods scale well. To allow further analysis of our results,
we provide an open-source interactive visualization tool, which includes
graphical representations and statistical tests.

###  7.1 Limitations

These quantitative results are obtained by using AutoML frameworks with preset
configurations only. The performance of frameworks under non-default settings
can be very relevant when, for example, there is budget available to also
optimize AutoML hyper-hyperparameters, or when a custom search space is used.
Additionally, many frameworks are under active development, so the results
presented here may not be representative of their performance in the future.

Next, all frameworks differ along multiple design axes, which prohibits
attributing performance differences to any specific component of the AutoML
framework (such as the search algorithm), without additional analysis.

Moreover, since we provide a purely quantitative comparison, it ignores
qualitative aspects of AutoML frameworks that are very relevant in real-world
settings, such as the produced model’s interpretability or the level of
support.

Lastly, the benchmarks were executed on 8-CPU machines, which is very modest
by today’s standards. Therefore, frameworks with better parallelism may show
even greater advantages on higher powered machines (with, for example, 50 or
100 CPUs). On the other hand, even shorter time constraints may be useful for
a human-in-the-loop or green AutoML setting.

###  7.2 Future Work

The benchmarking suites, or sets of tasks, proposed in this paper are meant as
a starting point to be improved upon in collaboration with the AutoML
community. We will seek to update these suites based on community discussion
so that they continue to reflect modern challenges while also decreasing the
risk of AutoML frameworks overfitting to the benchmark. In particular, we are
interested in extending the benchmarking suites with problems that feature
free-form text data. Real-world tabular data often contain instances of text
data with different semantic meaning, such as addresses or URLs, from which
meaningful features can be extracted. This kind of feature engineering is
typically very important for model performance on such data sets, but the
current selection of data sets does not reflect this, in part because not all
AutoML frameworks support text features.

We would also like to extend the benchmarking tool to support new problems,
such as multi-objective optimization tasks. While model accuracy is often an
important metric, ‘secondary’ metrics—such as a model’s inference time or
fairness—are often crucial for real-world applications. Multi-objective
optimization can be used to convey the importance of these metrics to AutoML
frameworks, for example, by using a fairness metric as secondary objective,
(Schmucker et al., 2021)), which can subsequently provide Pareto fronts of
models that optimize this trade-off. In particular for fairness related tasks,
additional support to convey sensitive attributes and protected groups to the
AutoML frameworks must be added. Other interesting problem types include
non-i.i.d. data, such as when temporal relationships are present in the data,
or semi-supervised learning, where not all instances have an associated ground
truth.

Finally, the evaluations in this paper were performed on the equivalent of
commodity-level hardware using only CPU and a limited time budget. In some
cases, it is more desirable to devote a large budget to building a single
model, perhaps even days of compute, potentially with GPU access. In other
cases, much smaller budgets may be desired, for example to reduce carbon
footprints or improve the experience for workflows with a human in the loop.
As different behavior (both in robustness and model performance) on one-hour
and four-hour budgets are already observed, future work may reveal different
behavior when evaluating AutoML frameworks at different scales.

###  7.3 Parting Words

The benchmark tool presented in this work makes producing rigorous
reproducible research both easier and faster. We hope that the open and
extensible nature of this benchmark motivates researchers to not only use the
tool, but also to contribute their own data sets, framework integrations, or
feedback and code contributions to the open source AutoML benchmark. We
strongly encourage this participation so that the benchmark may remain useful
to the community for a long time to come.

Acknowledgments

We would all like to give special thanks to everyone that contributed to the
benchmark, both directly with pull requests and indirectly through opening
issues. We also thank Rinchin Damdinov, Nick Erickson, Matthias Feurer, and
Piotr Płoński for feedback and corrections to this manuscript.

This work made use of the resources and expertise offered by the SURF Public
Cloud Call which is financed by the Dutch Research Council (NWO). We also made
use of research credits provided by the AWS Cloud Credit for Research program.

Pieter Gijsbers and Joaquin Vanschoren would like to acknowledge funding by
AFRL and DARPA under contract FA8750-17-C-0141, and EU’s Horizon Europe
research and innovation program under grant agreement No. 952215 (TAILOR).

Stefan Coors, Janek Thomas, and Bernd Bischl would like to acknowledge funding
by the German Federal Ministry of Education and Research (BMBF) under Grant
No. 01IS18036A.

##  A OpenML Benchmark Suites

Table LABEL:tab:269 and Table LABEL:tab:271 contain an overview of data sets
used in the regression and classification benchmarking suites, respectively.
We hope to continuously update the benchmarking suites with new data sets that
represent current challenges.

Table 2: Tasks in the AutoML regression suite. Task ID | name | n | p
---|---|---|---
359944 | abalone | 4177 | 9
359929 | Airlines_DepDelay_10M | 10000000 | 10
233212 | Allstate_Claims_Severity | 188318 | 131
359937 | black_friday | 166821 | 10
359950 | boston | 506 | 14
359938 | Brazilian_houses | 10692 | 13
233213 | Buzzinsocialmedia_Twitter | 583250 | 78
359942 | colleges | 7063 | 45
233211 | diamonds | 53940 | 10
359936 | elevators | 16599 | 19
359952 | house_16H | 22784 | 17
359951 | house_prices_nominal | 1460 | 80
359949 | house_sales | 21613 | 22
233215 | Mercedes_Benz_Greener_Manufacturing | 4209 | 377
360945 | MIP-2016-regression | 1090 | 145
167210 | Moneyball | 1232 | 15
359943 | nyc-taxi-green-dec-2016 | 581835 | 19
359941 | OnlineNewsPopularity | 39644 | 60
359946 | pol | 15000 | 49
360933 | QSAR-TID-10980 | 5766 | 1026
360932 | QSAR-TID-11 | 5742 | 1026
359930 | quake | 2178 | 4
233214 | Santander_transaction_value | 4459 | 4992
359948 | SAT11-HAND-runtime-regression | 4440 | 117
359931 | sensory | 576 | 12
359932 | socmob | 1156 | 6
359933 | space_ga | 3107 | 7
359934 | tecator | 240 | 125
359939 | topo_2_1 | 8885 | 267
359945 | us_crime | 1994 | 127
359935 | wine_quality | 6497 | 12
317614 | Yolanda | 400000 | 101
359940 | yprop_4_1 | 8885 | 252
Table 3: Tasks in the AutoML classification suite. Task ID | name | n | p | C | class ratio
---|---|---|---|---|---
190411 | ada | 4147 | 49 | 2 | 0.33
359983 | adult | 48842 | 15 | 2 | 0.31
189354 | airlines | 539383 | 8 | 2 | 0.80
189356 | albert | 425240 | 79 | 2 | 1.00
10090 | amazon-commerce-reviews | 1500 | 10001 | 50 | 1.00
359979 | Amazon_employee_access | 32769 | 10 | 2 | 0.06
168868 | APSFailure | 76000 | 171 | 2 | 0.02
190412 | arcene | 100 | 10001 | 2 | 0.79
146818 | Australian | 690 | 15 | 2 | 0.80
359982 | bank-marketing | 45211 | 17 | 2 | 0.13
359967 | Bioresponse | 3751 | 1777 | 2 | 0.84
359955 | blood-transfusion-service-center | 748 | 5 | 2 | 0.31
359960 | car | 1728 | 7 | 4 | 0.05
359973 | christine | 5418 | 1637 | 2 | 1.00
359968 | churn | 5000 | 21 | 2 | 0.16
359992 | Click_prediction_small | 39948 | 12 | 2 | 0.20
359959 | cmc | 1473 | 10 | 3 | 0.53
359957 | cnae-9 | 1080 | 857 | 9 | 1.00
359977 | connect-4 | 67557 | 43 | 3 | 0.15
7593 | covertype | 581012 | 55 | 7 | 0.01
168757 | credit-g | 1000 | 21 | 2 | 0.43
211986 | Diabetes130US | 101766 | 50 | 3 | 0.21
168909 | dilbert | 10000 | 2001 | 5 | 0.93
189355 | dionis | 416188 | 61 | 355 | 0.36
359964 | dna | 3186 | 181 | 3 | 0.46
359954 | eucalyptus | 736 | 20 | 5 | 0.49
168910 | fabert | 8237 | 801 | 7 | 0.26
359976 | Fashion-MNIST | 70000 | 785 | 10 | 1.00
359969 | first-order-theorem-proving | 6118 | 52 | 6 | 0.19
359970 | GesturePhaseSegmentationProcessed | 9873 | 33 | 5 | 0.34
189922 | gina | 3153 | 971 | 2 | 0.97
359988 | guillermo | 20000 | 4297 | 2 | 0.67
359984 | helena | 65196 | 28 | 100 | 0.03
360114 | Higgs | 1000000 | 29 | 2 | 0.89
359966 | Internet-Advertisements | 3279 | 1559 | 2 | 0.16
211979 | jannis | 83733 | 55 | 4 | 0.04
168911 | jasmine | 2984 | 145 | 2 | 1.00
359981 | jungle_chess_2pcs_raw_endgame_complete | 44819 | 7 | 3 | 0.19
359962 | kc1 | 2109 | 22 | 2 | 0.18
360975 | KDDCup09-Upselling | 50000 | 14892 | 2 | 0.08
3945 | KDDCup09_appetency | 50000 | 231 | 2 | 0.02
360112 | KDDCup99 | 4898431 | 42 | 23 | 0.00
359991 | kick | 72983 | 33 | 2 | 0.14
359965 | kr-vs-kp | 3196 | 37 | 2 | 0.91
190392 | madeline | 3140 | 260 | 2 | 0.99
359961 | mfeat-factors | 2000 | 217 | 10 | 1.00
359953 | micro-mass | 571 | 1301 | 20 | 0.18
359990 | MiniBooNE | 130064 | 51 | 2 | 0.39
359980 | nomao | 34465 | 119 | 2 | 0.40
167120 | numerai28.6 | 96320 | 22 | 2 | 0.98
359993 | okcupid-stem | 50789 | 20 | 3 | 0.13
190137 | ozone-level-8hr | 2534 | 73 | 2 | 0.07
359958 | pc4 | 1458 | 38 | 2 | 0.14
190410 | philippine | 5832 | 309 | 2 | 1.00
359971 | PhishingWebsites | 11055 | 31 | 2 | 0.80
168350 | phoneme | 5404 | 6 | 2 | 0.42
360113 | porto-seguro | 595212 | 58 | 2 | 0.04
359956 | qsar-biodeg | 1055 | 42 | 2 | 0.51
359989 | riccardo | 20000 | 4297 | 2 | 0.33
359986 | robert | 10000 | 7201 | 10 | 0.92
359975 | Satellite | 5100 | 37 | 2 | 0.01
359963 | segment | 2310 | 20 | 7 | 1.00
359994 | sf-police-incidents | 2215023 | 9 | 2 | 0.14
359987 | shuttle | 58000 | 10 | 7 | 0.00
168784 | steel-plates-fault | 1941 | 28 | 7 | 0.08
359972 | sylvine | 5124 | 21 | 2 | 1.00
190146 | vehicle | 846 | 19 | 4 | 0.91
359985 | volkert | 58310 | 181 | 10 | 0.11
146820 | wilt | 4839 | 6 | 2 | 0.06
359974 | wine-quality-white | 4898 | 12 | 7 | 0.00
2073 | yeast | 1484 | 9 | 10 | 0.01

##  B Results

This appendix contains additional figures and tables with experimental
results. Tables 4-15 report results per framework per task for different task
types and time budgets. Each value denotes the mean score of completed folds,
the standard deviation in those completed folds, and the number of folds for
which the AutoML framework did not return a result, if applicable. A ‘-’
denotes cases where the AutoML framework was unable to complete any fold of a
task on a specific time budget. A ‘*’ next to a framework name denotes
experiments that were conducted on an older version of the framework in 2021.

| framework | AutoGluon(B) | AutoGluon(HQ) | AutoGluon(HQIL) | autosklearn | autosklearn2 | flaml | GAMA(B) | H2OAutoML
---|---|---|---|---|---|---|---|---|---
task id | task name |  |  |  |  |  |  |  |
146818 | Australi… |  0.941(0.018)${}^{\hskip 1.39998pt}$ |  0.942(0.017)${}^{\hskip 1.39998pt}$ |  0.942(0.017)${}^{\hskip 1.39998pt}$ |  0.931(0.022)${}^{\hskip 1.39998pt}$ |  0.936(0.019)${}^{\hskip 1.39998pt}$ |  0.938(0.023)${}^{\hskip 1.39998pt}$ |  0.941(0.022)1 |  0.935(0.025)${}^{\hskip 1.39998pt}$
146820 | wilt |  0.995(0.008)${}^{\hskip 1.39998pt}$ |  0.994(0.010)${}^{\hskip 1.39998pt}$ |  0.994(0.010)${}^{\hskip 1.39998pt}$ |  0.994(0.009)${}^{\hskip 1.39998pt}$ |  0.995(0.007)${}^{\hskip 1.39998pt}$ |  0.991(0.011)${}^{\hskip 1.39998pt}$ |  0.996(0.004)${}^{\hskip 1.39998pt}$ |  0.992(0.010)${}^{\hskip 1.39998pt}$
167120 | numerai2… |  0.531(0.004)${}^{\hskip 1.39998pt}$ |  0.528(0.004)${}^{\hskip 1.39998pt}$ |  0.529(0.004)${}^{\hskip 1.39998pt}$ |  0.530(0.004)${}^{\hskip 1.39998pt}$ |  0.531(0.004)${}^{\hskip 1.39998pt}$ |  0.528(0.004)${}^{\hskip 1.39998pt}$ |  0.530(0.006)${}^{\hskip 1.39998pt}$ |  0.531(0.004)${}^{\hskip 1.39998pt}$
168350 | phoneme |  0.968(0.009)${}^{\hskip 1.39998pt}$ |  0.968(0.009)${}^{\hskip 1.39998pt}$ |  0.968(0.009)${}^{\hskip 1.39998pt}$ |  0.963(0.009)${}^{\hskip 1.39998pt}$ |  0.970(0.009)${}^{\hskip 1.39998pt}$ |  0.972(0.008)${}^{\hskip 1.39998pt}$ |  0.971(0.010)2 |  0.968(0.009)${}^{\hskip 1.39998pt}$
168757 | credit-g |  0.796(0.041)${}^{\hskip 1.39998pt}$ |  0.788(0.044)${}^{\hskip 1.39998pt}$ |  0.763(0.053)${}^{\hskip 1.39998pt}$ |  0.778(0.030)${}^{\hskip 1.39998pt}$ |  0.796(0.041)${}^{\hskip 1.39998pt}$ |  0.788(0.039)${}^{\hskip 1.39998pt}$ |  0.794(0.033)${}^{\hskip 1.39998pt}$ |  0.779(0.048)${}^{\hskip 1.39998pt}$
168868 | APSFailu… |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.992(0.001)${}^{\hskip 1.39998pt}$ |  0.992(0.002)${}^{\hskip 1.39998pt}$ |  0.990(0.003)${}^{\hskip 1.39998pt}$ |  0.993(0.002)${}^{\hskip 1.39998pt}$
168911 | jasmine |  0.886(0.018)${}^{\hskip 1.39998pt}$ |  0.885(0.019)${}^{\hskip 1.39998pt}$ |  0.885(0.019)${}^{\hskip 1.39998pt}$ |  0.885(0.016)${}^{\hskip 1.39998pt}$ |  0.888(0.017)${}^{\hskip 1.39998pt}$ |  0.887(0.016)${}^{\hskip 1.39998pt}$ |  0.891(0.013)${}^{\hskip 1.39998pt}$ |  0.887(0.018)${}^{\hskip 1.39998pt}$
189354 | airlines |  0.732(0.002)${}^{\hskip 1.39998pt}$ |  0.732(0.002)${}^{\hskip 1.39998pt}$ |  0.732(0.002)${}^{\hskip 1.39998pt}$ |  0.726(0.003)${}^{\hskip 1.39998pt}$ |  0.727(0.002)${}^{\hskip 1.39998pt}$ |  0.731(0.002)${}^{\hskip 1.39998pt}$ |  0.717(0.007)${}^{\hskip 1.39998pt}$ |  0.731(0.002)${}^{\hskip 1.39998pt}$
189356 | albert |  0.782(0.002)${}^{\hskip 1.39998pt}$ |  0.782(0.002)${}^{\hskip 1.39998pt}$ |  0.782(0.003)${}^{\hskip 1.39998pt}$ |  0.756(0.003)${}^{\hskip 1.39998pt}$ |  0.737(0.003)${}^{\hskip 1.39998pt}$ |  0.770(0.002)${}^{\hskip 1.39998pt}$ |  0.726(0.006)${}^{\hskip 1.39998pt}$ |  0.761(0.003)${}^{\hskip 1.39998pt}$
189922 | gina |  0.991(0.005)${}^{\hskip 1.39998pt}$ |  0.992(0.005)${}^{\hskip 1.39998pt}$ |  0.992(0.005)${}^{\hskip 1.39998pt}$ |  0.990(0.005)${}^{\hskip 1.39998pt}$ |  0.988(0.007)${}^{\hskip 1.39998pt}$ |  0.992(0.005)${}^{\hskip 1.39998pt}$ |  0.991(0.006)${}^{\hskip 1.39998pt}$ |  0.991(0.005)${}^{\hskip 1.39998pt}$
190137 | ozone-le… |  0.933(0.016)${}^{\hskip 1.39998pt}$ |  0.931(0.016)${}^{\hskip 1.39998pt}$ |  0.931(0.016)${}^{\hskip 1.39998pt}$ |  0.926(0.019)${}^{\hskip 1.39998pt}$ |  0.927(0.026)${}^{\hskip 1.39998pt}$ |  0.922(0.019)${}^{\hskip 1.39998pt}$ |  0.925(0.030)${}^{\hskip 1.39998pt}$ |  0.930(0.020)${}^{\hskip 1.39998pt}$
190392 | madeline |  0.945(0.008)${}^{\hskip 1.39998pt}$ |  0.944(0.008)${}^{\hskip 1.39998pt}$ |  0.944(0.008)${}^{\hskip 1.39998pt}$ |  0.965(0.007)${}^{\hskip 1.39998pt}$ |  0.945(0.008)${}^{\hskip 1.39998pt}$ |  0.954(0.007)${}^{\hskip 1.39998pt}$ |  0.957(0.009)${}^{\hskip 1.39998pt}$ |  0.943(0.010)${}^{\hskip 1.39998pt}$
190410 | philippi… |  0.878(0.013)${}^{\hskip 1.39998pt}$ |  0.877(0.011)${}^{\hskip 1.39998pt}$ |  0.877(0.011)${}^{\hskip 1.39998pt}$ |  0.915(0.011)${}^{\hskip 1.39998pt}$ |  0.877(0.014)${}^{\hskip 1.39998pt}$ |  0.891(0.014)${}^{\hskip 1.39998pt}$ |  0.892(0.016)${}^{\hskip 1.39998pt}$ |  0.882(0.011)${}^{\hskip 1.39998pt}$
190411 | ada |  0.921(0.018)${}^{\hskip 1.39998pt}$ |  0.920(0.018)${}^{\hskip 1.39998pt}$ |  0.920(0.018)${}^{\hskip 1.39998pt}$ |  0.918(0.018)${}^{\hskip 1.39998pt}$ |  0.919(0.018)${}^{\hskip 1.39998pt}$ |  0.924(0.018)${}^{\hskip 1.39998pt}$ |  0.920(0.019)${}^{\hskip 1.39998pt}$ |  0.921(0.019)${}^{\hskip 1.39998pt}$
190412 | arcene |  0.873(0.155)${}^{\hskip 1.39998pt}$ |  0.873(0.173)${}^{\hskip 1.39998pt}$ |  0.852(0.169)${}^{\hskip 1.39998pt}$ |  0.866(0.141)${}^{\hskip 1.39998pt}$ |  0.845(0.196)${}^{\hskip 1.39998pt}$ |  0.868(0.154)${}^{\hskip 1.39998pt}$ |  0.878(0.154)1 |  0.839(0.228)${}^{\hskip 1.39998pt}$
359955 | blood-tr… |  0.758(0.046)${}^{\hskip 1.39998pt}$ |  0.759(0.043)${}^{\hskip 1.39998pt}$ |  0.759(0.043)${}^{\hskip 1.39998pt}$ |  0.747(0.050)${}^{\hskip 1.39998pt}$ |  0.756(0.049)${}^{\hskip 1.39998pt}$ |  0.730(0.042)${}^{\hskip 1.39998pt}$ |  0.753(0.036)${}^{\hskip 1.39998pt}$ |  0.764(0.041)${}^{\hskip 1.39998pt}$
359956 | qsar-bio… |  0.942(0.032)${}^{\hskip 1.39998pt}$ |  0.941(0.035)${}^{\hskip 1.39998pt}$ |  0.939(0.034)${}^{\hskip 1.39998pt}$ |  0.931(0.034)${}^{\hskip 1.39998pt}$ |  0.936(0.027)${}^{\hskip 1.39998pt}$ |  0.930(0.031)${}^{\hskip 1.39998pt}$ |  0.936(0.032)${}^{\hskip 1.39998pt}$ |  0.939(0.034)${}^{\hskip 1.39998pt}$
359958 | pc4 |  0.952(0.021)${}^{\hskip 1.39998pt}$ |  0.949(0.022)${}^{\hskip 1.39998pt}$ |  0.950(0.023)${}^{\hskip 1.39998pt}$ |  0.942(0.019)${}^{\hskip 1.39998pt}$ |  0.949(0.018)${}^{\hskip 1.39998pt}$ |  0.950(0.019)${}^{\hskip 1.39998pt}$ |  0.950(0.022)${}^{\hskip 1.39998pt}$ |  0.948(0.022)${}^{\hskip 1.39998pt}$
359962 | kc1 |  0.840(0.034)${}^{\hskip 1.39998pt}$ |  0.840(0.035)${}^{\hskip 1.39998pt}$ |  0.840(0.035)${}^{\hskip 1.39998pt}$ |  0.843(0.033)${}^{\hskip 1.39998pt}$ |  0.838(0.037)${}^{\hskip 1.39998pt}$ |  0.841(0.040)${}^{\hskip 1.39998pt}$ |  0.852(0.030)${}^{\hskip 1.39998pt}$ |  0.829(0.030)${}^{\hskip 1.39998pt}$
359965 | kr-vs-kp |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.001)${}^{\hskip 1.39998pt}$ |  1.000(0.001)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  0.961(0.123)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.001)${}^{\hskip 1.39998pt}$
359966 | Internet… |  0.985(0.011)${}^{\hskip 1.39998pt}$ |  0.986(0.012)${}^{\hskip 1.39998pt}$ |  0.978(0.016)${}^{\hskip 1.39998pt}$ |  0.984(0.012)${}^{\hskip 1.39998pt}$ |  0.984(0.013)${}^{\hskip 1.39998pt}$ |  0.987(0.011)${}^{\hskip 1.39998pt}$ |  0.983(0.012)${}^{\hskip 1.39998pt}$ |  0.986(0.010)${}^{\hskip 1.39998pt}$
359967 | Biorespo… |  0.886(0.017)${}^{\hskip 1.39998pt}$ |  0.884(0.016)${}^{\hskip 1.39998pt}$ |  0.878(0.016)${}^{\hskip 1.39998pt}$ |  0.873(0.017)${}^{\hskip 1.39998pt}$ |  0.870(0.018)${}^{\hskip 1.39998pt}$ |  0.884(0.016)${}^{\hskip 1.39998pt}$ |  0.884(0.017)${}^{\hskip 1.39998pt}$ |  0.887(0.016)${}^{\hskip 1.39998pt}$
359968 | churn |  0.922(0.021)${}^{\hskip 1.39998pt}$ |  0.924(0.022)${}^{\hskip 1.39998pt}$ |  0.923(0.023)${}^{\hskip 1.39998pt}$ |  0.922(0.021)${}^{\hskip 1.39998pt}$ |  0.920(0.024)${}^{\hskip 1.39998pt}$ |  0.922(0.023)${}^{\hskip 1.39998pt}$ |  0.920(0.023)${}^{\hskip 1.39998pt}$ |  0.925(0.022)${}^{\hskip 1.39998pt}$
359971 | Phishing… |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.002)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)1 |  0.998(0.001)${}^{\hskip 1.39998pt}$
359972 | sylvine |  0.990(0.003)${}^{\hskip 1.39998pt}$ |  0.989(0.003)${}^{\hskip 1.39998pt}$ |  0.989(0.003)${}^{\hskip 1.39998pt}$ |  0.991(0.003)${}^{\hskip 1.39998pt}$ |  0.990(0.002)${}^{\hskip 1.39998pt}$ |  0.991(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.002)2 |  0.990(0.003)${}^{\hskip 1.39998pt}$
359973 | christine |  0.826(0.014)${}^{\hskip 1.39998pt}$ |  0.827(0.013)${}^{\hskip 1.39998pt}$ |  0.827(0.013)${}^{\hskip 1.39998pt}$ |  0.828(0.015)${}^{\hskip 1.39998pt}$ |  0.819(0.011)${}^{\hskip 1.39998pt}$ |  0.824(0.013)${}^{\hskip 1.39998pt}$ |  0.828(0.017)${}^{\hskip 1.39998pt}$ |  0.825(0.013)${}^{\hskip 1.39998pt}$
359975 | Satellite |  0.996(0.005)${}^{\hskip 1.39998pt}$ |  0.996(0.007)${}^{\hskip 1.39998pt}$ |  0.996(0.007)${}^{\hskip 1.39998pt}$ |  0.994(0.007)${}^{\hskip 1.39998pt}$ |  0.995(0.007)${}^{\hskip 1.39998pt}$ |  0.978(0.041)${}^{\hskip 1.39998pt}$ |  0.996(0.003)${}^{\hskip 1.39998pt}$ |  0.979(0.033)${}^{\hskip 1.39998pt}$
359979 | Amazon_e… |  0.902(0.012)${}^{\hskip 1.39998pt}$ |  0.901(0.011)${}^{\hskip 1.39998pt}$ |  0.901(0.011)${}^{\hskip 1.39998pt}$ |  0.853(0.017)${}^{\hskip 1.39998pt}$ |  0.876(0.014)${}^{\hskip 1.39998pt}$ |  0.876(0.014)${}^{\hskip 1.39998pt}$ |  0.867(0.012)1 |  0.877(0.012)${}^{\hskip 1.39998pt}$
359980 | nomao |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.996(0.001)${}^{\hskip 1.39998pt}$ |  0.996(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.995(0.001)${}^{\hskip 1.39998pt}$ |  0.996(0.001)${}^{\hskip 1.39998pt}$
359982 | bank-mar… |  0.941(0.006)${}^{\hskip 1.39998pt}$ |  0.941(0.006)${}^{\hskip 1.39998pt}$ |  0.940(0.006)${}^{\hskip 1.39998pt}$ |  0.939(0.006)${}^{\hskip 1.39998pt}$ |  0.938(0.007)${}^{\hskip 1.39998pt}$ |  0.937(0.006)${}^{\hskip 1.39998pt}$ |  0.936(0.006)${}^{\hskip 1.39998pt}$ |  0.938(0.007)${}^{\hskip 1.39998pt}$
359983 | adult |  0.932(0.004)${}^{\hskip 1.39998pt}$ |  0.932(0.004)${}^{\hskip 1.39998pt}$ |  0.932(0.004)${}^{\hskip 1.39998pt}$ |  0.930(0.004)${}^{\hskip 1.39998pt}$ |  0.931(0.004)${}^{\hskip 1.39998pt}$ |  0.932(0.004)${}^{\hskip 1.39998pt}$ |  0.929(0.004)${}^{\hskip 1.39998pt}$ |  0.931(0.004)${}^{\hskip 1.39998pt}$
359988 | guillermo |  0.914(0.008)${}^{\hskip 1.39998pt}$ |  0.914(0.008)${}^{\hskip 1.39998pt}$ |  0.914(0.009)${}^{\hskip 1.39998pt}$ |  0.914(0.008)${}^{\hskip 1.39998pt}$ |  0.906(0.009)${}^{\hskip 1.39998pt}$ |  0.919(nan)9 |  0.865(0.031)${}^{\hskip 1.39998pt}$ |  0.897(0.010)${}^{\hskip 1.39998pt}$
359989 | riccardo |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)3 |  0.999(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$
359990 | MiniBooNE |  0.989(0.001)${}^{\hskip 1.39998pt}$ |  0.988(0.001)${}^{\hskip 1.39998pt}$ |  0.988(0.001)${}^{\hskip 1.39998pt}$ |  0.987(0.001)${}^{\hskip 1.39998pt}$ |  0.987(0.001)${}^{\hskip 1.39998pt}$ |  0.987(0.001)${}^{\hskip 1.39998pt}$ |  0.982(0.002)${}^{\hskip 1.39998pt}$ |  0.987(0.001)${}^{\hskip 1.39998pt}$
359991 | kick |  0.791(0.007)${}^{\hskip 1.39998pt}$ |  0.790(0.007)${}^{\hskip 1.39998pt}$ |  0.790(0.007)${}^{\hskip 1.39998pt}$ |  0.785(0.006)${}^{\hskip 1.39998pt}$ |  0.787(0.007)${}^{\hskip 1.39998pt}$ |  0.787(0.007)${}^{\hskip 1.39998pt}$ |  0.791(0.016)${}^{\hskip 1.39998pt}$ |  0.789(0.007)${}^{\hskip 1.39998pt}$
359992 | Click_pr… |  0.710(0.012)${}^{\hskip 1.39998pt}$ |  0.710(0.012)${}^{\hskip 1.39998pt}$ |  0.710(0.012)${}^{\hskip 1.39998pt}$ |  0.697(0.014)${}^{\hskip 1.39998pt}$ |  0.699(0.014)${}^{\hskip 1.39998pt}$ |  0.723(0.010)${}^{\hskip 1.39998pt}$ |  0.655(0.019)${}^{\hskip 1.39998pt}$ |  0.701(0.013)${}^{\hskip 1.39998pt}$
359994 | sf-polic… |  0.697(0.001)${}^{\hskip 1.39998pt}$ |  0.693(0.002)${}^{\hskip 1.39998pt}$ |  0.677(0.002)${}^{\hskip 1.39998pt}$ |  0.694(0.005)${}^{\hskip 1.39998pt}$ |  0.693(0.022)${}^{\hskip 1.39998pt}$ |  0.709(0.004)${}^{\hskip 1.39998pt}$ |  0.623(0.011)${}^{\hskip 1.39998pt}$ |  0.688(0.013)${}^{\hskip 1.39998pt}$
360113 | porto-se… |  0.643(0.004)${}^{\hskip 1.39998pt}$ |  0.643(0.005)${}^{\hskip 1.39998pt}$ |  0.643(0.005)${}^{\hskip 1.39998pt}$ |  0.639(0.004)${}^{\hskip 1.39998pt}$ |  0.640(0.005)${}^{\hskip 1.39998pt}$ |  0.642(0.004)${}^{\hskip 1.39998pt}$ |  0.624(0.006)${}^{\hskip 1.39998pt}$ |  0.642(0.004)${}^{\hskip 1.39998pt}$
360114 | Higgs |  0.838(0.001)${}^{\hskip 1.39998pt}$ |  0.837(0.001)${}^{\hskip 1.39998pt}$ |  0.675(0.001)${}^{\hskip 1.39998pt}$ |  0.838(0.001)${}^{\hskip 1.39998pt}$ |  0.835(0.001)${}^{\hskip 1.39998pt}$ |  0.839(0.001)${}^{\hskip 1.39998pt}$ |  0.784(0.006)1 |  0.832(0.001)${}^{\hskip 1.39998pt}$
360975 | KDDCup09… |  0.908(0.007)${}^{\hskip 1.39998pt}$ |  0.906(0.007)${}^{\hskip 1.39998pt}$ |  0.906(0.007)${}^{\hskip 1.39998pt}$ | - | - | - |  0.558(0.085)8 |  0.903(0.005)${}^{\hskip 1.39998pt}$
3945 | KDDCup09… |  0.849(0.013)${}^{\hskip 1.39998pt}$ |  0.848(0.013)${}^{\hskip 1.39998pt}$ |  0.848(0.013)${}^{\hskip 1.39998pt}$ |  0.838(0.015)${}^{\hskip 1.39998pt}$ |  0.838(0.014)${}^{\hskip 1.39998pt}$ |  0.825(0.017)${}^{\hskip 1.39998pt}$ |  0.818(0.020)${}^{\hskip 1.39998pt}$ |  0.837(0.014)${}^{\hskip 1.39998pt}$
Table 4: Results for binary classification (in AUC) on a one hour budget,denoted as mean(std)${}^{\mbox{{fails}}}$.  | framework | lightautoml | MLJAR(B) | MLJAR(P) | TPOT | constantpredictor | RandomForest |  TunedRandomForest*
---|---|---|---|---|---|---|---|---
task id | task name |  |  |  |  |  |  |
146818 | Australi… |  0.946(0.020)${}^{\hskip 1.39998pt}$ |  0.943(0.020)${}^{\hskip 1.39998pt}$ |  0.944(0.017)${}^{\hskip 1.39998pt}$ |  0.939(0.022)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.940(0.021)${}^{\hskip 1.39998pt}$ |  0.938(0.022)${}^{\hskip 1.39998pt}$
146820 | wilt |  0.994(0.007)${}^{\hskip 1.39998pt}$ |  0.999(0.000)8 |  0.995(0.009)${}^{\hskip 1.39998pt}$ |  0.996(0.004)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.989(0.012)${}^{\hskip 1.39998pt}$ |  0.991(0.010)${}^{\hskip 1.39998pt}$
167120 | numerai2… |  0.531(0.004)${}^{\hskip 1.39998pt}$ |  0.530(0.004)${}^{\hskip 1.39998pt}$ |  0.531(0.005)${}^{\hskip 1.39998pt}$ |  0.528(0.007)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.519(0.006)${}^{\hskip 1.39998pt}$ |  0.520(0.006)${}^{\hskip 1.39998pt}$
168350 | phoneme |  0.966(0.008)${}^{\hskip 1.39998pt}$ | - |  0.967(0.008)${}^{\hskip 1.39998pt}$ |  0.969(0.008)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.965(0.009)${}^{\hskip 1.39998pt}$ |  0.966(0.009)${}^{\hskip 1.39998pt}$
168757 | credit-g |  0.796(0.037)${}^{\hskip 1.39998pt}$ | - |  0.785(0.046)${}^{\hskip 1.39998pt}$ |  0.791(0.052)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.801(0.038)${}^{\hskip 1.39998pt}$ |  0.799(0.035)${}^{\hskip 1.39998pt}$
168868 | APSFailu… |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.995(nan)9 |  0.992(0.002)${}^{\hskip 1.39998pt}$ |  0.989(0.004)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.991(0.002)${}^{\hskip 1.39998pt}$ |  0.992(0.003)${}^{\hskip 1.39998pt}$
168911 | jasmine |  0.880(0.018)${}^{\hskip 1.39998pt}$ |  0.885(0.016)${}^{\hskip 1.39998pt}$ |  0.886(0.019)${}^{\hskip 1.39998pt}$ |  0.886(0.013)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.888(0.016)${}^{\hskip 1.39998pt}$ |  0.888(0.017)${}^{\hskip 1.39998pt}$
189354 | airlines |  0.727(0.002)${}^{\hskip 1.39998pt}$ |  0.731(0.002)${}^{\hskip 1.39998pt}$ |  0.730(0.002)${}^{\hskip 1.39998pt}$ |  0.722(0.002)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.661(0.002)${}^{\hskip 1.39998pt}$ |  0.661(0.002)${}^{\hskip 1.39998pt}$
189356 | albert |  0.780(0.003)${}^{\hskip 1.39998pt}$ |  0.783(0.003)${}^{\hskip 1.39998pt}$ |  0.765(0.003)${}^{\hskip 1.39998pt}$ |  0.718(0.014)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.738(0.003)${}^{\hskip 1.39998pt}$ |  0.738(0.002)${}^{\hskip 1.39998pt}$
189922 | gina |  0.990(0.006)${}^{\hskip 1.39998pt}$ |  0.991(0.005)${}^{\hskip 1.39998pt}$ |  0.990(0.005)${}^{\hskip 1.39998pt}$ |  0.988(0.007)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.984(0.008)${}^{\hskip 1.39998pt}$ |  0.984(0.008)${}^{\hskip 1.39998pt}$
190137 | ozone-le… |  0.930(0.018)${}^{\hskip 1.39998pt}$ | - |  0.929(0.019)${}^{\hskip 1.39998pt}$ |  0.928(0.026)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.920(0.036)${}^{\hskip 1.39998pt}$ |  0.921(0.033)${}^{\hskip 1.39998pt}$
190392 | madeline |  0.935(0.010)${}^{\hskip 1.39998pt}$ |  0.956(0.011)${}^{\hskip 1.39998pt}$ |  0.950(0.012)${}^{\hskip 1.39998pt}$ |  0.948(0.009)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.887(0.017)${}^{\hskip 1.39998pt}$ |  0.932(0.013)${}^{\hskip 1.39998pt}$
190410 | philippi… |  0.866(0.014)${}^{\hskip 1.39998pt}$ |  0.882(0.013)${}^{\hskip 1.39998pt}$ |  0.872(0.012)${}^{\hskip 1.39998pt}$ |  0.879(0.012)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.849(0.016)${}^{\hskip 1.39998pt}$ |  0.858(0.015)${}^{\hskip 1.39998pt}$
190411 | ada |  0.921(0.018)${}^{\hskip 1.39998pt}$ |  0.921(0.018)${}^{\hskip 1.39998pt}$ |  0.921(0.018)${}^{\hskip 1.39998pt}$ |  0.917(0.019)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.903(0.014)${}^{\hskip 1.39998pt}$ |  0.902(0.016)${}^{\hskip 1.39998pt}$
190412 | arcene |  0.848(0.176)${}^{\hskip 1.39998pt}$ |  0.870(0.183)${}^{\hskip 1.39998pt}$ | - |  0.831(0.151)6 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.857(0.175)${}^{\hskip 1.39998pt}$ |  0.828(0.183)${}^{\hskip 1.39998pt}$
359955 | blood-tr… |  0.753(0.058)${}^{\hskip 1.39998pt}$ | - |  0.753(0.053)${}^{\hskip 1.39998pt}$ |  0.724(0.102)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.686(0.064)${}^{\hskip 1.39998pt}$ |  0.686(0.065)${}^{\hskip 1.39998pt}$
359956 | qsar-bio… |  0.934(0.034)${}^{\hskip 1.39998pt}$ |  0.932(nan)9 |  0.937(0.031)${}^{\hskip 1.39998pt}$ |  0.926(0.045)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.932(0.033)${}^{\hskip 1.39998pt}$ |  0.935(0.029)${}^{\hskip 1.39998pt}$
359958 | pc4 |  0.950(0.015)${}^{\hskip 1.39998pt}$ |  0.951(0.016)${}^{\hskip 1.39998pt}$ |  0.953(0.016)${}^{\hskip 1.39998pt}$ |  0.944(0.021)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.947(0.019)${}^{\hskip 1.39998pt}$ |  0.949(0.019)${}^{\hskip 1.39998pt}$
359962 | kc1 |  0.831(0.032)${}^{\hskip 1.39998pt}$ |  0.828(0.030)${}^{\hskip 1.39998pt}$ |  0.824(0.032)${}^{\hskip 1.39998pt}$ |  0.844(0.038)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.837(0.037)${}^{\hskip 1.39998pt}$ |  0.841(0.037)${}^{\hskip 1.39998pt}$
359965 | kr-vs-kp |  1.000(0.001)${}^{\hskip 1.39998pt}$ | - |  1.000(0.001)${}^{\hskip 1.39998pt}$ |  0.999(0.001)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.999(0.001)${}^{\hskip 1.39998pt}$ |  0.999(0.002)${}^{\hskip 1.39998pt}$
359966 | Internet… |  0.986(0.010)${}^{\hskip 1.39998pt}$ |  0.982(0.011)4 |  0.977(0.012)${}^{\hskip 1.39998pt}$ |  0.981(0.012)6 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.986(0.012)${}^{\hskip 1.39998pt}$ |  0.988(0.009)${}^{\hskip 1.39998pt}$
359967 | Biorespo… |  0.884(0.019)${}^{\hskip 1.39998pt}$ |  0.882(0.017)${}^{\hskip 1.39998pt}$ |  0.883(0.016)${}^{\hskip 1.39998pt}$ |  0.880(0.020)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.881(0.018)${}^{\hskip 1.39998pt}$ |  0.881(0.018)${}^{\hskip 1.39998pt}$
359968 | churn |  0.926(0.020)${}^{\hskip 1.39998pt}$ |  0.928(0.021)${}^{\hskip 1.39998pt}$ |  0.925(0.019)${}^{\hskip 1.39998pt}$ |  0.919(0.025)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.916(0.023)${}^{\hskip 1.39998pt}$ |  0.912(0.021)${}^{\hskip 1.39998pt}$
359971 | Phishing… |  0.998(0.001)${}^{\hskip 1.39998pt}$ | - |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.996(0.002)${}^{\hskip 1.39998pt}$ |  0.996(0.002)${}^{\hskip 1.39998pt}$
359972 | sylvine |  0.988(0.003)${}^{\hskip 1.39998pt}$ |  0.992(0.003)${}^{\hskip 1.39998pt}$ |  0.992(0.003)${}^{\hskip 1.39998pt}$ |  0.992(0.002)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.983(0.004)${}^{\hskip 1.39998pt}$ |  0.984(0.004)${}^{\hskip 1.39998pt}$
359973 | christine |  0.831(0.013)${}^{\hskip 1.39998pt}$ |  0.827(0.013)${}^{\hskip 1.39998pt}$ |  0.823(0.012)${}^{\hskip 1.39998pt}$ |  0.811(0.013)2 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.805(0.016)${}^{\hskip 1.39998pt}$ |  0.809(0.013)${}^{\hskip 1.39998pt}$
359975 | Satellite |  0.986(0.025)${}^{\hskip 1.39998pt}$ | - |  0.993(0.007)${}^{\hskip 1.39998pt}$ |  0.984(0.038)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.985(0.029)${}^{\hskip 1.39998pt}$ |  0.984(0.031)${}^{\hskip 1.39998pt}$
359979 | Amazon_e… |  0.879(0.009)${}^{\hskip 1.39998pt}$ |  0.905(0.013)${}^{\hskip 1.39998pt}$ |  0.903(0.013)${}^{\hskip 1.39998pt}$ |  0.864(0.014)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.863(0.015)${}^{\hskip 1.39998pt}$ |  0.863(0.014)${}^{\hskip 1.39998pt}$
359980 | nomao |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.000)8 |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.995(0.001)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.995(0.001)${}^{\hskip 1.39998pt}$ |  0.995(0.001)${}^{\hskip 1.39998pt}$
359982 | bank-mar… |  0.940(0.006)${}^{\hskip 1.39998pt}$ | - |  0.940(0.006)${}^{\hskip 1.39998pt}$ |  0.935(0.007)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.931(0.006)${}^{\hskip 1.39998pt}$ |  0.931(0.007)${}^{\hskip 1.39998pt}$
359983 | adult |  0.932(0.004)${}^{\hskip 1.39998pt}$ | - |  0.931(0.004)${}^{\hskip 1.39998pt}$ |  0.927(0.004)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.910(0.003)${}^{\hskip 1.39998pt}$ |  0.910(0.003)${}^{\hskip 1.39998pt}$
359988 | guillermo |  0.932(0.007)${}^{\hskip 1.39998pt}$ |  0.915(0.009)${}^{\hskip 1.39998pt}$ |  0.912(0.007)3 |  0.826(0.053)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.903(0.009)${}^{\hskip 1.39998pt}$ |  0.903(0.009)${}^{\hskip 1.39998pt}$
359989 | riccardo |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.999(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$
359990 | MiniBooNE |  0.988(0.001)${}^{\hskip 1.39998pt}$ |  0.987(0.001)${}^{\hskip 1.39998pt}$ |  0.987(0.001)${}^{\hskip 1.39998pt}$ |  0.982(0.001)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.982(0.001)${}^{\hskip 1.39998pt}$ |  0.982(0.001)${}^{\hskip 1.39998pt}$
359991 | kick |  0.785(0.008)${}^{\hskip 1.39998pt}$ |  0.754(0.012)${}^{\hskip 1.39998pt}$ |  0.751(0.009)${}^{\hskip 1.39998pt}$ |  0.728(0.009)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.761(0.008)${}^{\hskip 1.39998pt}$ |  0.766(0.009)${}^{\hskip 1.39998pt}$
359992 | Click_pr… |  0.728(0.010)${}^{\hskip 1.39998pt}$ |  0.709(0.014)${}^{\hskip 1.39998pt}$ |  0.709(0.014)${}^{\hskip 1.39998pt}$ |  0.715(0.008)${}^{\hskip 1.39998pt}$ |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.687(0.009)${}^{\hskip 1.39998pt}$ |  0.689(0.009)${}^{\hskip 1.39998pt}$
359994 | sf-polic… |  0.685(0.002)${}^{\hskip 1.39998pt}$ |  0.703(0.007)${}^{\hskip 1.39998pt}$ |  0.694(0.012)8 |  0.615(0.058)4 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.704(0.002)${}^{\hskip 1.39998pt}$ |  0.703(0.002)${}^{\hskip 1.39998pt}$
360113 | porto-se… |  0.641(0.004)${}^{\hskip 1.39998pt}$ |  0.641(0.004)${}^{\hskip 1.39998pt}$ |  0.643(0.004)${}^{\hskip 1.39998pt}$ |  0.577(0.058)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.610(0.005)${}^{\hskip 1.39998pt}$ |  0.618(0.006)${}^{\hskip 1.39998pt}$
360114 | Higgs |  0.838(0.001)${}^{\hskip 1.39998pt}$ |  0.836(0.002)${}^{\hskip 1.39998pt}$ | - |  0.737(0.043)1 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.822(0.001)${}^{\hskip 1.39998pt}$ |  0.819(0.001)${}^{\hskip 1.39998pt}$
360975 | KDDCup09… |  0.910(0.007)${}^{\hskip 1.39998pt}$ |  0.906(0.006)${}^{\hskip 1.39998pt}$ | - | - |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.883(0.008)${}^{\hskip 1.39998pt}$ |  0.883(0.007)${}^{\hskip 1.39998pt}$
3945 | KDDCup09… |  0.851(0.012)${}^{\hskip 1.39998pt}$ | - |  0.837(0.015)${}^{\hskip 1.39998pt}$ |  0.831(0.017)2 |  0.500(0.000)${}^{\hskip 1.39998pt}$ |  0.796(0.018)${}^{\hskip 1.39998pt}$ |  0.795(0.021)${}^{\hskip 1.39998pt}$
Table 5: Results for binary classification (in AUC) on a one hour budget,denoted as mean(std)${}^{\mbox{{fails}}}$.  | framework | AutoGluon(B) | AutoGluon(HQ) | autosklearn |  autosklearn2* | flaml |  GAMA(B)* | H2OAutoML
---|---|---|---|---|---|---|---|---
task id | task name |  |  |  |  |  |  |
146818 | Australi… |  0.941(0.018)${}^{\hskip 1.39998pt}$ |  0.942(0.017)${}^{\hskip 1.39998pt}$ |  0.931(0.023)${}^{\hskip 1.39998pt}$ |  0.940(0.020)${}^{\hskip 1.39998pt}$ |  0.941(0.025)${}^{\hskip 1.39998pt}$ |  0.940(0.019)${}^{\hskip 1.39998pt}$ |  0.935(0.021)${}^{\hskip 1.39998pt}$
146820 | wilt |  0.995(0.008)${}^{\hskip 1.39998pt}$ |  0.994(0.010)${}^{\hskip 1.39998pt}$ |  0.994(0.009)${}^{\hskip 1.39998pt}$ |  0.995(0.008)${}^{\hskip 1.39998pt}$ |  0.991(0.010)${}^{\hskip 1.39998pt}$ |  0.996(0.004)${}^{\hskip 1.39998pt}$ |  0.994(0.008)${}^{\hskip 1.39998pt}$
167120 | numerai2… |  0.531(0.004)${}^{\hskip 1.39998pt}$ |  0.529(0.004)${}^{\hskip 1.39998pt}$ |  0.530(0.005)${}^{\hskip 1.39998pt}$ |  0.531(0.004)${}^{\hskip 1.39998pt}$ |  0.529(0.004)${}^{\hskip 1.39998pt}$ |  0.532(0.004)1 |  0.531(0.004)${}^{\hskip 1.39998pt}$
168350 | phoneme |  0.968(0.009)${}^{\hskip 1.39998pt}$ |  0.968(0.009)${}^{\hskip 1.39998pt}$ |  0.964(0.009)${}^{\hskip 1.39998pt}$ |  0.970(0.009)${}^{\hskip 1.39998pt}$ |  0.972(0.009)${}^{\hskip 1.39998pt}$ |  0.971(0.009)${}^{\hskip 1.39998pt}$ |  0.967(0.010)${}^{\hskip 1.39998pt}$
168757 | credit-g |  0.796(0.041)${}^{\hskip 1.39998pt}$ |  0.788(0.044)${}^{\hskip 1.39998pt}$ |  0.785(0.042)${}^{\hskip 1.39998pt}$ |  0.795(0.038)${}^{\hskip 1.39998pt}$ |  0.783(0.045)${}^{\hskip 1.39998pt}$ |  0.791(0.030)${}^{\hskip 1.39998pt}$ |  0.783(0.043)${}^{\hskip 1.39998pt}$
168868 | APSFailu… |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.992(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.992(0.003)${}^{\hskip 1.39998pt}$ |  0.992(0.002)1 |  0.992(0.002)${}^{\hskip 1.39998pt}$ |  0.992(0.002)${}^{\hskip 1.39998pt}$
168911 | jasmine |  0.886(0.018)${}^{\hskip 1.39998pt}$ |  0.885(0.019)${}^{\hskip 1.39998pt}$ |  0.883(0.016)${}^{\hskip 1.39998pt}$ |  0.887(0.017)${}^{\hskip 1.39998pt}$ |  0.887(0.016)${}^{\hskip 1.39998pt}$ |  0.893(0.014)${}^{\hskip 1.39998pt}$ |  0.887(0.020)${}^{\hskip 1.39998pt}$
189354 | airlines |  0.733(0.002)${}^{\hskip 1.39998pt}$ |  0.733(0.002)${}^{\hskip 1.39998pt}$ |  0.727(0.002)${}^{\hskip 1.39998pt}$ |  0.727(0.002)${}^{\hskip 1.39998pt}$ |  0.732(0.002)${}^{\hskip 1.39998pt}$ | - |  0.732(0.002)${}^{\hskip 1.39998pt}$
189356 | albert |  0.786(0.002)${}^{\hskip 1.39998pt}$ |  0.785(0.002)${}^{\hskip 1.39998pt}$ |  0.762(0.003)${}^{\hskip 1.39998pt}$ |  0.759(0.002)${}^{\hskip 1.39998pt}$ |  0.771(0.002)1 |  0.747(0.009)${}^{\hskip 1.39998pt}$ |  0.769(0.003)${}^{\hskip 1.39998pt}$
189922 | gina |  0.991(0.005)${}^{\hskip 1.39998pt}$ |  0.992(0.005)${}^{\hskip 1.39998pt}$ |  0.991(0.006)${}^{\hskip 1.39998pt}$ |  0.988(0.007)${}^{\hskip 1.39998pt}$ |  0.992(0.005)${}^{\hskip 1.39998pt}$ |  0.991(0.005)${}^{\hskip 1.39998pt}$ |  0.990(0.006)${}^{\hskip 1.39998pt}$
190137 | ozone-le… |  0.933(0.016)${}^{\hskip 1.39998pt}$ |  0.931(0.016)${}^{\hskip 1.39998pt}$ |  0.913(0.029)${}^{\hskip 1.39998pt}$ |  0.933(0.022)${}^{\hskip 1.39998pt}$ |  0.924(0.020)${}^{\hskip 1.39998pt}$ |  0.926(0.032)${}^{\hskip 1.39998pt}$ |  0.925(0.025)${}^{\hskip 1.39998pt}$
190392 | madeline |  0.945(0.008)${}^{\hskip 1.39998pt}$ |  0.944(0.008)${}^{\hskip 1.39998pt}$ |  0.969(0.006)${}^{\hskip 1.39998pt}$ |  0.945(0.008)${}^{\hskip 1.39998pt}$ |  0.954(0.008)${}^{\hskip 1.39998pt}$ |  0.959(0.008)${}^{\hskip 1.39998pt}$ |  0.947(0.010)${}^{\hskip 1.39998pt}$
190410 | philippi… |  0.878(0.013)${}^{\hskip 1.39998pt}$ |  0.877(0.011)${}^{\hskip 1.39998pt}$ |  0.917(0.012)${}^{\hskip 1.39998pt}$ |  0.877(0.014)${}^{\hskip 1.39998pt}$ |  0.893(0.013)${}^{\hskip 1.39998pt}$ |  0.903(0.014)${}^{\hskip 1.39998pt}$ |  0.880(0.015)${}^{\hskip 1.39998pt}$
190411 | ada |  0.921(0.018)${}^{\hskip 1.39998pt}$ |  0.920(0.018)${}^{\hskip 1.39998pt}$ |  0.918(0.018)${}^{\hskip 1.39998pt}$ |  0.920(0.018)${}^{\hskip 1.39998pt}$ |  0.924(0.018)${}^{\hskip 1.39998pt}$ |  0.921(0.018)${}^{\hskip 1.39998pt}$ |  0.921(0.016)${}^{\hskip 1.39998pt}$
190412 | arcene |  0.861(0.176)${}^{\hskip 1.39998pt}$ |  0.869(0.173)${}^{\hskip 1.39998pt}$ |  0.873(0.155)${}^{\hskip 1.39998pt}$ |  0.832(0.156)${}^{\hskip 1.39998pt}$ |  0.881(0.120)${}^{\hskip 1.39998pt}$ |  0.856(0.162)${}^{\hskip 1.39998pt}$ |  0.865(0.156)${}^{\hskip 1.39998pt}$
359955 | blood-tr… |  0.758(0.046)${}^{\hskip 1.39998pt}$ |  0.759(0.043)${}^{\hskip 1.39998pt}$ |  0.748(0.052)${}^{\hskip 1.39998pt}$ |  0.755(0.040)${}^{\hskip 1.39998pt}$ |  0.730(0.042)${}^{\hskip 1.39998pt}$ |  0.757(0.049)${}^{\hskip 1.39998pt}$ |  0.756(0.047)${}^{\hskip 1.39998pt}$
359956 | qsar-bio… |  0.942(0.032)${}^{\hskip 1.39998pt}$ |  0.941(0.035)${}^{\hskip 1.39998pt}$ |  0.932(0.032)${}^{\hskip 1.39998pt}$ |  0.937(0.027)${}^{\hskip 1.39998pt}$ |  0.934(0.028)${}^{\hskip 1.39998pt}$ |  0.937(0.032)${}^{\hskip 1.39998pt}$ |  0.939(0.033)${}^{\hskip 1.39998pt}$
359958 | pc4 |  0.952(0.021)${}^{\hskip 1.39998pt}$ |  0.949(0.022)${}^{\hskip 1.39998pt}$ |  0.942(0.020)${}^{\hskip 1.39998pt}$ |  0.949(0.017)${}^{\hskip 1.39998pt}$ |  0.946(0.021)${}^{\hskip 1.39998pt}$ |  0.951(0.019)${}^{\hskip 1.39998pt}$ |  0.951(0.019)${}^{\hskip 1.39998pt}$
359962 | kc1 |  0.840(0.034)${}^{\hskip 1.39998pt}$ |  0.840(0.035)${}^{\hskip 1.39998pt}$ |  0.843(0.027)${}^{\hskip 1.39998pt}$ |  0.839(0.036)${}^{\hskip 1.39998pt}$ |  0.841(0.039)${}^{\hskip 1.39998pt}$ |  0.851(0.032)${}^{\hskip 1.39998pt}$ |  0.821(0.040)${}^{\hskip 1.39998pt}$
359965 | kr-vs-kp |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  0.976(0.075)${}^{\hskip 1.39998pt}$ |  1.000(0.000)${}^{\hskip 1.39998pt}$ |  1.000(0.001)${}^{\hskip 1.39998pt}$
359966 | Internet… |  0.985(0.011)${}^{\hskip 1.39998pt}$ |  0.986(0.012)${}^{\hskip 1.39998pt}$ |  0.986(0.011)${}^{\hskip 1.39998pt}$ |  0.982(0.015)${}^{\hskip 1.39998pt}$ |  0.986(0.012)${}^{\hskip 1.39998pt}$ |  0.984(0.011)${}^{\hskip 1.39998pt}$ |  0.987(0.009)${}^{\hskip 1.39998pt}$
359967 | Biorespo… |  0.886(0.017)${}^{\hskip 1.39998pt}$ |  0.885(0.016)${}^{\hskip 1.39998pt}$ |  0.872(0.019)${}^{\hskip 1.39998pt}$ |  0.873(0.018)${}^{\hskip 1.39998pt}$ |  0.886(0.016)${}^{\hskip 1.39998pt}$ |  0.885(0.017)${}^{\hskip 1.39998pt}$ |  0.886(0.017)${}^{\hskip 1.39998pt}$
359968 | churn |  0.922(0.021)${}^{\hskip 1.39998pt}$ |  0.924(0.022)${}^{\hskip 1.39998pt}$ |  0.917(0.020)${}^{\hskip 1.39998pt}$ |  0.919(0.021)${}^{\hskip 1.39998pt}$ |  0.926(0.027)${}^{\hskip 1.39998pt}$ |  0.921(0.022)${}^{\hskip 1.39998pt}$ |  0.929(0.015)${}^{\hskip 1.39998pt}$
359971 | Phishing… |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.997(0.001)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$ |  0.998(0.001)${}^{\hskip 1.39998pt}$
359972 | sylvine |  0.990(0.003)${}^{\hskip 1.39998pt}$ |  0.989(0.003)${}^{\hskip 1.39998pt}$ |  0.992(0.003)${}^{\hskip 1.39998pt}$ |  0.990(0.002)${}^{\hskip 1.39998pt}$ |  0.991(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.002)${}^{\hskip 1.39998pt}$ |  0.990(0.003)${}^{\hskip 1.39998pt}$
359973 | christine |  0.827(0.013)${}^{\hskip 1.39998pt}$ |  0.826(0.013)${}^{\hskip 1.39998pt}$ |  0.829(0.014)${}^{\hskip 1.39998pt}$ |  0.818(0.013)${}^{\hskip 1.39998pt}$ |  0.827(0.012)${}^{\hskip 1.39998pt}$ |  0.833(0.014)${}^{\hskip 1.39998pt}$ |  0.824(0.011)${}^{\hskip 1.39998pt}$
359975 | Satellite |  0.996(0.005)${}^{\hskip 1.39998pt}$ |  0.996(0.007)${}^{\hskip 1.39998pt}$ |  0.983(0.039)${}^{\hskip 1.39998pt}$ |  0.995(0.005)${}^{\hskip 1.39998pt}$ |  0.978(0.041)${}^{\hskip 1.39998pt}$ |  0.996(0.002)${}^{\hskip 1.39998pt}$ |  0.993(0.007)${}^{\hskip 1.39998pt}$
359979 | Amazon_e… |  0.902(0.012)${}^{\hskip 1.39998pt}$ |  0.901(0.011)${}^{\hskip 1.39998pt}$ |  0.861(0.017)${}^{\hskip 1.39998pt}$ |  0.878(0.010)${}^{\hskip 1.39998pt}$ |  0.878(0.013)${}^{\hskip 1.39998pt}$ |  0.862(0.013)${}^{\hskip 1.39998pt}$ |  0.875(0.011)${}^{\hskip 1.39998pt}$