* [7] Richard E. Bellman  “Adaptive Control Processes”  Princeton: Princeton University Press, 2015, pp. 94  DOI: 10.1515/9781400874668
  * [8] Yoshua Bengio and Yves Grandvalet  “No unbiased estimator of the variance of k-fold cross-validation”  In _Journal of Machine Learning Research_ 5.Sep, 2004, pp. 1089–1105
  * [9] Christoph Bergmeir, Rob J. Hyndman and Bonsoo Koo  “A note on the validity of cross-validation for evaluating autoregressive time series prediction”  In _Computational Statistics & Data Analysis_ 120, 2018, pp. 70–83  DOI: 10.1016/j.csda.2017.11.003
  * [10] J. Bergstra and Y. Bengio  “Random Search for Hyper-Parameter Optimization”  In _Journal of Machine Learning Research_ 13, 2012, pp. 281–305
  * [11] James Bergstra and Yoshua Bengio  “Random Search for Hyper-Parameter Optimization”  In _Journal of Machine Learning Research_ 13.10, 2012, pp. 281–305
  * [12] James Bergstra, Daniel Yamins and David Cox  “Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures”  In _Proceedings of the 30th International Conference on Machine Learning_ 28.1, Proceedings of Machine Learning Research  Atlanta, Georgia, USA: PMLR, 2013, pp. 115–123
  * [13] Hans-Georg Beyer and Hans-Paul Schwefel  “Evolution strategies - A comprehensive introduction”  In _Natural Computing_ 1, 2002, pp. 3–52  DOI: 10.1023/A:1015059928466
  * [14] H.-G. Beyer and B. Sendhoff  “Evolution Strategies for Robust Optimization”  In _2006 IEEE International Conference on Evolutionary Computation_ , 2006, pp. 1346–1353  DOI: 10.1109/CEC.2006.1688465
  * [15] Andre Biedenkapp et al.  “Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework”  In _ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020)_ 325, Frontiers in Artificial Intelligence and Applications  IOS Press, 2020, pp. 427–434
  * [16] Martin Binder, Julia Moosbauer, Janek Thomas and Bernd Bischl  “Multi-objective hyperparameter tuning and feature selection using filter ensembles”  In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , 2020, pp. 471–479
  * [17] Martin Binder, Florian Pfisterer and Bernd Bischl  “Collecting Empirical Data About Hyperparameters for Data Driven AutoML”  In _Proceedings of the 7th ICML Workshop on Automated Machine Learning (AutoML 2020)_ , 2020
  * [18] Martin Binder et al.  “mlr3pipelines - Flexible Machine Learning Pipelines in R”  In _Journal of Machine Learning Research_ 22.184, 2021, pp. 1–7
  * [19] Mauro Birattari, Thomas Stützle, Luis Paquete and Klaus Varrentrapp  “A Racing Algorithm for Configuring Metaheuristics”  In _Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation_ , GECCO’02  New York City, New York: Morgan Kaufmann Publishers Inc., 2002, pp. 11–18
  * [20] Mauro Birattari, Zhi Yuan, Prasanna Balaprakash and Thomas Stützle  “F-Race and iterated F-Race: An overview”  In _Experimental methods for the analysis of optimization algorithms_ Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 311–336  DOI: 10.1007/978-3-642-02538-9˙13
  * [21] B. Bischl, O. Mersmann, H. Trautmann and C. Weihs  “Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation”  In _Evolutionary Computation_ 20.2, 2012, pp. 249–275  DOI: 10.1162/EVCO˙a˙00069
  * [22] Bernd Bischl et al.  “ASlib: A benchmark library for algorithm selection”  In _Artif. Intell._ 237, 2016, pp. 41–58
  * [23] Bernd Bischl et al.  “MOI-MBO: Multiobjective Infill for Parallel Model-Based Optimization”  In _Learning and Intelligent Optimization Conference_ , Lecture Notes in Computer Science  Florida: Springer, 2014, pp. 173–186  DOI: 10.1007/978-3-319-09584-4˙17
  * [24] Jakob Bossek, Carola Doerr and Pascal Kerschke  “Initial Design Strategies and Their Effects on Sequential Model-Based Optimization: An Exploratory Case Study Based on BBOB”  In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , GECCO ’20  New York, NY, USA: Association for Computing Machinery, 2020, pp. 778–786  DOI: 10.1145/3377930.3390155
  * [25] A-L Boulesteix, Carolin Strobl, Thomas Augustin and Martin Daumer  “Evaluating microarray-based classifiers: an overview”  In _Cancer Informatics_ 6 SAGE Publications Sage UK: London, England, 2008, pp. 77–97  DOI: 10.4137/CIN.S408
  * [26] T. Chen et al.  “MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems”  In _CoRR_ abs/1512.01274, 2015
  * [27] Yutian Chen et al.  “Learning to Learn without Gradient Descent by Gradient Descent”  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ 70, Proceedings of Machine Learning Research  PMLR, 2017, pp. 748–756
  * [28] Clément Chevalier and David Ginsbourger  “Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection”  In _Learning and Intelligent Optimization_ Springer Berlin Heidelberg, 2013, pp. 59–69  DOI: 10.1007/978-3-642-44973-4˙7
  * [29] Michael Chui et al.  “Notes from the AI frontier: Insights from hundreds of use cases”  In _McKinsey Global Institute_ , 2018
  * [30] Carlos A. Coello Coello, Gary B. Lamont and David A. Van Veldhuizen  “Evolutionary algorithms for solving multi-objective problems”  Springer, 2007  DOI: 10.1007/978-1-4757-5184-0
  * [31] Christian Daniel, Jonathan Taylor and Sebastian Nowozin  “Learning Step Size Controllers for Robust Neural Network Training”  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA_ AAAI Press, 2016, pp. 1519–1525
  * [32] George De Ath, Richard M Everson, Alma AM Rahat and Jonathan E Fieldsend  “Greed is good: Exploration and exploitation trade-offs in Bayesian optimisation”  In _ACM Transactions on Evolutionary Learning and Optimization_ 1.1 ACM New York, NY, 2021, pp. 1–22
  * [33] Kevin K Dobbin and Richard M Simon  “Optimally splitting cases for training and testing high dimensional classifiers”  In _BMC medical genomics_ 4.1 BioMed Central, 2011, pp. 1–8
  * [34] Jaimie Drozdal et al.  “Trust in AutoML: exploring information needs for establishing trust in automated machine learning systems”  In _IUI ’20: 25th International Conference on Intelligent User Interfaces, Cagliari, Italy, March 17-20, 2020_ ACM, 2020, pp. 297–307
  * [35] Katharina Eggensperger et al.  “Towards an empirical foundation for assessing bayesian optimization of hyperparameters”  In _NIPS workshop on Bayesian Optimization in Theory and Practice_ 10, 2013, pp. 3
  * [36] Katharina Eggensperger et al.  “Efficient benchmarking of algorithm configurators via model-based surrogates”  In _Mach. Learn._ 107.1, 2018, pp. 15–41
  * [37] Radwa Elshawi, Mohamed Maher and Sherif Sakr  “Automated Machine Learning: State-of-The-Art and Open Challenges”, 2019  arXiv:1906.02287 [cs.LG]
  * [38] T. Elsken, B. Staffler, J. Metzen and F. Hutter  “Meta-Learning of Neural Architectures for Few-Shot Learning”  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ Computer Vision Foundation / IEEE, 2020, pp. 12362–12372
  * [39] Thomas Elsken, Jan Hendrik Metzen and Frank Hutter  “Neural Architecture Search: A Survey”  In _Journal of Machine Learning Research_ 20.55, 2019, pp. 1–21
  * [40] David Eriksson et al.  “Scalable Global Optimization via Local Bayesian Optimization”  In _Advances in Neural Information Processing Systems_ , 2019, pp. 5496–5507
  * [41] Hugo Jair Escalante, Manuel Montes and Luis Enrique Sucar  “Particle swarm model selection.”  In _Journal of Machine Learning Research_ 10.2, 2009
  * [42] Rasool Fakoor et al.  “Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation”  In _Advances in Neural Information Processing Systems_ 33, 2020
  * [43] S. Falkner, A. Klein and F. Hutter  “BOHB: Robust and Efficient Hyperparameter Optimization at Scale”  In _Proceedings of the 35th International Conference on Machine Learning (ICML’18)_ 80 Proceedings of Machine Learning Research, 2018, pp. 1437–1446
  * [44] Manuel Fernández-Delgado, Eva Cernadas, Senén Barro and Dinani Amorim  “Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?”  In _The Journal of Machine Learning Research_ 15.1 JMLR.org, 2014, pp. 3133–3181
  * [45] M. Feurer and F. Hutter  “Hyperparameter Optimization”  In _AutoML: Methods, Sytems, Challenges_ Springer, 2019, pp. 3–33
  * [46] M. Feurer et al.  “Efficient and Robust Automated Machine Learning”  In _Proceedings of the 28th International Conference on Advances in Neural Information Processing Systems (NeurIPS’15)_ , 2015, pp. 2962–2970
  * [47] Matthias Feurer et al.  “Efficient and Robust Automated Machine Learning”  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’15  Montreal, Canada: MIT Press, 2015, pp. 2755–2763
  * [48] Matthias Feurer, Jost Tobias Springenberg and Frank Hutter  “Initializing Bayesian Hyperparameter Optimization via Meta-Learning”  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA_ AAAI Press, 2015, pp. 1128–1135
  * [49] C. Finn, P. Abbeel and S. Levine  “Model-agnostic meta-learning for fast adaptation of deep networks”  In _Proceedings of the 34th International Conference on Machine Learning (ICML’17)_ 70 Proceedings of Machine Learning Research, 2017, pp. 1126–1135
  * [50] L. Franceschi, M. Donini, P. Frasconi and M. Pontil  “Forward and Reverse Gradient-Based Hyperparameter Optimization”  In _Proceedings of the 34th International Conference on Machine Learning (ICML’17)_ 70 Proceedings of Machine Learning Research, 2017, pp. 1165–1173
  * [51] L. Franceschi et al.  “Bilevel Programming for Hyperparameter Optimization and Meta-Learning”  In _Proceedings of the 35th International Conference on Machine Learning (ICML’18)_ 80 Proceedings of Machine Learning Research, 2018, pp. 1568–1577
  * [52] Eduardo C Garrido-Merchán and Daniel Hernández-Lobato  “Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes”  In _Neurocomputing_ 380 Elsevier, 2020, pp. 20–35
  * [53] Pieter Gijsbers et al.  “Meta-learning for symbolic hyperparameter defaults”, 2021, pp. 151–152
  * [54] David Ginsbourger, Rodolphe Le Riche and Laurent Carraro  “Kriging Is Well-Suited to Parallelize Optimization”  In _Computational Intelligence in Expensive Optimization Problems_ Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, pp. 131–162  DOI: 10.1007/978-3-642-10701-6˙6
  * [55] I. Guyon, A. Saffari, G. Dror and G. Cawley  “Model selection: Beyond the Bayesian/frequentist divide”  In _The Journal of Machine Learning Research_ 11.3, 2010, pp. 61–87
  * [56] John T. Hancock and Taghi M. Khoshgoftaar  “Survey on categorical data for neural networks”  In _Journal of Big Data_ 7.1 Springer ScienceBusiness Media LLC, 2020  DOI: 10.1186/s40537-020-00305-w
  * [57] Trevor Hastie, Robert Tibshirani and Jerome Friedman  “The elements of statistical learning: data mining, inference, and prediction”  New York: Springer Science & Business Media, 2009
  * [58] X. He, K. Zhao and X. Chu  “AutoML: A survey of the state-of-the-art”  In _Knowl. Based Syst._ 212, 2021, pp. 106622
  * [59] Xin He, Kaiyong Zhao and Xiaowen Chu  “AutoML: A survey of the state-of-the-art”  In _Knowledge-Based Systems_ 212, 2021, pp. 106622  DOI: https://doi.org/10.1016/j.knosys.2020.106622
  * [60] Philipp Hennig and Christian J. Schuler  “Entropy Search for Information-Efficient Global Optimization”  In _Journal of Machine Learning Research_ 13.6, 2012, pp. 1809–1837
  * [61] Daniel Hernández-Lobato, Jose Hernandez-Lobato, Amar Shah and Ryan Adams  “Predictive entropy search for multi-objective bayesian optimization”  In _International Conference on Machine Learning_ , 2016, pp. 1492–1501
  * [62] Daniel Horn and Bernd Bischl  “Multi-objective parameter configuration of machine learning algorithms using model-based optimization”  In _2016 IEEE symposium series on computational intelligence (SSCI)_ , 2016, pp. 1–8  IEEE  DOI: 10.1109/SSCI.2016.7850221
  * [63] Roman Hornung et al.  “A measure of the impact of CV incompleteness on prediction error estimation with application to PCA and normalization”  In _BMC Medical Research Methodology_ 15 BioMed Central, 2015, pp. 95  DOI: 10.1186/s12874-015-0088-9
  * [64] F. Hutter, H. Hoos and K. Leyton-Brown  “Sequential Model-Based Optimization for General Algorithm Configuration”  In _Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION’11)_ 6683, Lecture Notes in Computer Science  Springer, 2011, pp. 507–523
  * [65] “Automated Machine Learning: Methods, Systems, Challenges” Available for free at http://automl.org/book  In _Automated Machine Learning: Methods, Systems, Challenges_ Springer, 2019
  * [66] Frank Hutter, Youssef Hamadi, Holger H Hoos and Kevin Leyton-Brown  “Performance prediction and automated tuning of randomized and parametric algorithms”  In _International Conference on Principles and Practice of Constraint Programming_ , 2006, pp. 213–228  Springer
  * [67] Frank Hutter, Holger H Hoos and Kevin Leyton-Brown  “Sequential model-based optimization for general algorithm configuration”  In _International conference on learning and intelligent optimization_ Berlin, Heidelberg: Springer, 2011, pp. 507–523  DOI: 10.1007/978-3-642-25566-3˙40
  * [68] Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown  “Parallel Algorithm Configuration”  In _Learning and Intelligent Optimization_ , Lecture Notes in Computer Science 7219  Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 55–70  DOI: 10.1007/978-3-642-34413-8˙5
  * [69] Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown and Thomas Stützle  “ParamILS: An Automatic Algorithm Configuration Framework”  In _J. Artif. Intell. Res._ 36, 2009, pp. 267–306
  * [70] Hamed Jalali, Inneke Van Nieuwenhuyse and Victor Picheny  “Comparison of kriging-based algorithms for simulation optimization with heterogeneous noise”  In _European Journal of Operational Research_ 261.1 Elsevier, 2017, pp. 279–301
  * [71] Kevin Jamieson and Ameet Talwalkar  “Non-stochastic Best Arm Identification and Hyperparameter Optimization”  In _Proceedings of the 19th International Con-ference on Artificial Intelligence and Statistics (AISTATS)_ , 2016, pp. 240–248
  * [72] Nathalie Japkowicz and Mohak Shah  “Evaluating Learning Algorithms: A Classification Perspective”  Cambridge University Press, 2011  DOI: 10.1017/CBO9780511921803
  * [73] Dipti Jasrasaria and Edward O Pyzer-Knapp  “Dynamic Control of Explore/Exploit Trade-Off in Bayesian Optimization”  In _Science and Information Conference_ , 2018, pp. 1–15  Springer
  * [74] Donald R. Jones  “A Taxonomy of Global Optimization Methods Based on Response Surfaces”  In _Journal of Global Optimization_ 21.4, 2001, pp. 345–383  DOI: 10.1023/A:1012771025575
  * [75] Donald R Jones  “Direct Global Optimization Algorithm.”  In _Encyclopedia of optimization_ 1.1, 2009, pp. 431–440
  * [76] Donald R. Jones, Matthias Schonlau and William J. Welch  “Efficient Global Optimization of Expensive Black-Box Functions”  In _Journal of Global Optimization_ 13.4, 1998, pp. 455–492  DOI: 10.1023/A:1008306431147
  * [77] Kirthevasan Kandasamy et al.  “Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly”  In _Journal of Machine Learning Research_ 21.81, 2020, pp. 1–27
  * [78] Rabiya Khalid and Nadeem Javaid  “A survey on hyperparameters optimization algorithms of forecasting models in smart grid”  In _Sustainable Cities and Society_ 61 Elsevier, 2020, pp. 102275
  * [79] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015
  * [80] A. Klein et al.  “Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets”  In _Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS)_ 54 Proceedings of Machine Learning Research, 2017
  * [81] Aaron Klein et al.  “Meta-surrogate benchmarking for hyperparameter optimization”  In _Advances in Neural Information Processing Systems_ 32, 2019, pp. 6270–6280
  * [82] Aaron Klein et al.  “Fast bayesian optimization of machine learning hyperparameters on large datasets”  In _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ 54, Proceedings of Machine Learning Research  Fort Lauderdale, FL, USA: PMLR, 2017, pp. 528–536  PMLR
  * [83] R. Kohavi  “A study of cross-validation and bootstrap for accuracy estimation and model selection”  In _International Joint Conference on Artificial Intelligence (IJCAI)_ , 1995, pp. 1137–1143
  * [84] M. Lang et al.  “Automatic model selection for high-dimensional survival analysis”  In _Journal of Statistical Computation and Simulation_ 85.1 Taylor & Francis, 2015, pp. 62–76  DOI: 10.1080/00949655.2014.929131
  * [85] Michel Lang et al.  “mlr3: A modern object-oriented machine learning framework in R”  In _Journal of Open Source Software_ 4.44, 2019, pp. 1903  DOI: 10.21105/joss.01903
  * [86] Pedro Larrañaga and Jose A Lozano  “Estimation of distribution algorithms: A new tool for evolutionary computation”  Springer Science & Business Media, 2001
  * [87] A. Li et al.  “A Generalized Framework for Population Based Training”  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_ ACM, 2019, pp. 1791–1799
  * [88] Ke Li and Jitendra Malik  “Learning to Optimize”  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_ OpenReview.net, 2017
  * [89] Lisha Li et al.  “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”  In _Journal of Machine Learning Research_ 18.185, 2018, pp. 1–52
  * [90] Rui Li et al.  “Mixed integer evolution strategies for parameter optimization”  In _Evolutionary Computation_ 21.1 MIT Press, 2013, pp. 29–64  DOI: 10.1162/EVCO˙a˙00059
  * [91] Marius Lindauer and Frank Hutter  “Warmstarting of Model-Based Algorithm Configuration”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32.1, 2018
  * [92] Marius Lindauer and Frank Hutter  “Best Practices for Scientific Research on Neural Architecture Search”  In _Journal of Machine Learning Research_ 21.243, 2020, pp. 1–18
  * [93] Hanxiao Liu, Karen Simonyan and Yiming Yang  “DARTS: Differentiable Architecture Search”  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_ OpenReview.net, 2019
  * [94] Manuel López-Ibáñez et al.  “The irace package: Iterated Racing for Automatic Algorithm Configuration”  In _Operations Research Perspectives_ 3, 2016, pp. 43–58  DOI: 10.1016/j.orp.2016.09.002
  * [95] Ilya Loshchilov and Frank Hutter  “SGDR: Stochastic Gradient Descent with Warm Restarts”  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_ OpenReview.net, 2017
  * [96] D. Maclaurin, D. Duvenaud and R. Adams  “Gradient-based Hyperparameter Optimization through Reversible Learning”  In _Proceedings of the 32nd International Conference on Machine Learning (ICML’15)_ 37 Omnipress, 2015, pp. 2113–2122
  * [97] Oded Maron and Andrew W Moore  “Hoeffding races: Accelerating model selection search for classification and function approximation”  In _Advances in Neural Information Processing Systems_ 6, 1994, pp. 59–66
  * [98] Mitchell McIntire, Daniel Ratner and Stefano Ermon  “Sparse Gaussian Processes for Bayesian Optimization”  In _Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence_ , UAI’16  Arlington, Virginia, USA: AUAI Press, 2016, pp. 517–526
  * [99] MD McKay, RJ Beckman and WJ Conover  “Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output from a Computer Code”  In _Technometrics_ 21.2, 1979, pp. 239–245  DOI: 10.2307/1268522
  * [100] F. Mohr, M. Wever and E. Hüllermeier  “ML-Plan: Automated machine learning via hierarchical planning”  In _Machine Learning_ 107.8-10, 2018, pp. 1495–1515
  * [101] Christoph Molnar, Giuseppe Casalicchio and Bernd Bischl  “Quantifying model complexity via functional decomposition for better post-hoc interpretability”  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , 2019, pp. 193–204  Springer
  * [102] Amin Nayebi, Alexander Munteanu and Matthias Poloczek  “A Framework for Bayesian Optimization in Embedded Subspaces”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 4752–4761
  * [103] Andrew Y Ng  “Preventing “overfitting” of cross-validation data”  In _ICML_ 97, 1997, pp. 245–253  Citeseer
  * [104] Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz and Jason H. Moore  “Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science”  In _Proceedings of the Genetic and Evolutionary Computation Conference 2016_ , GECCO ’16  Denver, Colorado, USA: ACM, 2016, pp. 485–492  DOI: 10.1145/2908812.2908918
  * [105] Jack Parker-Holder, Vu Nguyen and Stephen J. Roberts  “Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits”  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems_ , 2020
  * [106] A. Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurIPS’19)_ , 2019, pp. 8024–8035
  * [107] F. Pedregosa et al.  “Scikit-learn: Machine Learning in Python”  In _Journal of Machine Learning Research_ 12, 2011, pp. 2825–2830
  * [108] Valerio Perrone et al.  “Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019
  * [109] Florian Pfisterer et al.  “Learning Multiple Defaults for Machine Learning Algorithms”  In _Proceedings of the Genetic and Evolutionary Computation Conference Companion_ , GECCO ’21  Lille, France: Association for Computing Machinery, 2021, pp. 241–242  DOI: 10.1145/3449726.3459523
  * [110] Victor Picheny, Tobias Wagner and David Ginsbourger  “A benchmark of kriging-based infill criteria for noisy optimization”  In _Structural and Multidisciplinary Optimization_ 48.3 Springer, 2013, pp. 607–626
  * [111] Philipp Probst, Anne-Laure Boulesteix and Bernd Bischl  “Tunability: Importance of Hyperparameters of Machine Learning Algorithms.”  In _Journal of Machine Learning Research_ 20.53, 2019, pp. 1–32
  * [112] J Quinlan and R Cameron-Jones  “Oversearching and layered search in empirical learning”  In _breast cancer_ 286 Citeseer, 1995, pp. 2–7
  * [113] Carl Edward Rasmussen and Christopher K.. Williams  “Gaussian processes for machine learning”  MIT Press, 2006
  * [114] Esteban Real, Alok Aggarwal, Yanping Huang and Quoc V. Le  “Regularized Evolution for Image Classifier Architecture Search”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 33, 2019, pp. 4780–4789  DOI: 10.1609/aaai.v33i01.33014780
  * [115] John R. Rice  “The Algorithm Selection Problem”  In _Adv. Comput._ 15, 1976, pp. 65–118
  * [116] David P. Rodgers  “Improvements in multiprocessor system design”  In _ACM SIGARCH Computer Architecture News_ 13.3 Association for Computing Machinery (ACM), 1985, pp. 225–231  DOI: 10.1145/327070.327215
  * [117] Olivier Roustant, David Ginsbourger and Yves Deville  “DiceKriging, DiceOptim: Two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization”  In _Journal of Statistical Software_ 51, 2012  DOI: 10.18637/jss.v051.i01
  * [118] Michael J Sasena, Panos Papalambros and Pierre Goovaerts  “Exploration of metamodeling sampling criteria for constrained global optimization”  In _Engineering optimization_ 34.3 Taylor & Francis, 2002, pp. 263–278
  * [119] Warren Scott, Peter Frazier and Warren Powell  “The correlated knowledge gradient for simulation optimization of continuous parameters using gaussian process regression”  In _SIAM Journal on Optimization_ 21.3 SIAM, 2011, pp. 996–1026
  * [120] Jasjeet S Sekhon and Walter R Mebane  “Genetic optimization using derivatives”  In _Political Analysis_ 7 JSTOR, 1998, pp. 187–210  DOI: 10.1093/pan/7.1.187
  * [121] Joseph Sexton and Petter Laake  “Standard errors for bagged and random forest estimators”  In _Computational Statistics & Data Analysis_ 53.3 Elsevier, 2009, pp. 801–811  DOI: 10.1016/j.csda.2008.08.007
  * [122] Victor S Sheng and Charles X Ling  “Thresholding for making classifiers cost-sensitive”  In _AAAI’06: Proceedings of the 21st national conference on artificial intelligence_ 6, 2006, pp. 476–481
  * [123] Richard Simon  “Resampling Strategies for Model Assessment and Selection”  In _Fundamentals of Data Mining in Genomics and Proteomics_ Boston, MA: Springer, 2007, pp. 173–186  DOI: 10.1007/978-0-387-47509-7˙8
  * [124] J. Snoek, H. Larochelle and R. Adams  “Practical Bayesian Optimization of Machine Learning Algorithms”  In _Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS’12)_ , 2012, pp. 2960–2968
  * [125] Jasper Snoek, Hugo Larochelle and Ryan P Adams  “Practical Bayesian Optimization of Machine Learning Algorithms”  In _Advances in Neural Information Processing Systems 25_ Curran Associates, Inc., 2012, pp. 2951–2959
  * [126] Jasper Snoek et al.  “Scalable Bayesian Optimization Using Deep Neural Networks”  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ 37, JMLR Workshop and Conference Proceedings  JMLR.org, 2015, pp. 2171–2180
  * [127] David Speck et al.  “Learning Heuristic Selection with Dynamic Algorithm Configuration”  In _Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS’21)_ , 2021
  * [128] Niranjan Srinivas, Andreas Krause, Sham Kakade and Matthias Seeger  “Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design”  In _Proceedings of the 27th International Conference on International Conference on Machine Learning_ , ICML’10  Madison, WI, USA: Omnipress, 2010, pp. 1015–1022
  * [129] K. Swersky, J. Snoek and R. Adams  “Multi-task Bayesian optimization”  In _Proceedings of the 26th International Conference on Advances in Neural Information Processing Systems (NeurIPS’13)_ , 2013, pp. 2004–2012
  * [130] Kevin Swersky et al.  “Raiders of the lost architecture: Kernels for Bayesian optimization in conditional parameter spaces”  In _NeurIPS workshop on Bayesian Optimization in Theory and Practice_ , 2014
  * [131] Kevin Swersky, Jasper Snoek and Ryan P Adams  “Multi-Task Bayesian Optimization”  In _Advances in Neural Information Processing Systems_ 26 Curran Associates, Inc., 2013, pp. 2004–2012
  * [132] Kevin Swersky, Jasper Snoek and Ryan Prescott Adams  “Freeze-thaw bayesian optimization”, 2014  arXiv: http://arxiv.org/abs/1406.3896
  * [133] El-Ghazali Talbi  “Optimization of deep neural networks: a survey and unified taxonomy”, 2020  URL: https://hal.inria.fr/hal-02570804
  * [134] L. Thiede and U. Parlitz  “Gradient based hyperparameter optimization in Echo State Networks”  In _Neural Networks_ 115, 2019, pp. 23–29
  * [135] Chris Thornton, Frank Hutter, Holger H. Hoos and Kevin Leyton-Brown  “Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms”  In _Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’13  Chicago, Illinois, USA: Association for Computing Machinery, 2013, pp. 847–855
  * [136] Louis C Tiao, Aaron Klein, Cedric Archambeau and Matthias Seeger  “Model-based Asynchronous Hyperparameter Optimization”  arXiv: https://arxiv.org/abs/2003.10865
  * [137] Jan N Van Rijn et al.  “OpenML: A collaborative science platform”  In _Joint european conference on machine learning and knowledge discovery in databases_ , 2013, pp. 645–649  Springer
  * [138] Jan N Van Rijn and Frank Hutter  “Hyperparameter importance across datasets”  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 2367–2376
  * [139] Ziyu Wang et al.  “Bayesian optimization in a billion dimensions via random embeddings”  In _Journal of Artificial Intelligence Research_ 55, 2016, pp. 361–387
  * [140] Colin White, Willie Neiswanger and Yash Savani  “BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2021
  * [141] M. Wistuba, N. Schilling and L. Schmidt-Thieme  “Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization”  In _Machine Learning and Knowledge Discovery in Databases (ECML/PKDD’15)_ 9285, Lecture Notes in Computer Science  Springer, 2015, pp. 104–119
  * [142] M. Wistuba, N. Schilling and L. Schmidt-Thieme  “Learning hyperparameter optimization initializations”  In _Proceedings of the International Conference on Data Science and Advanced Analytics (DSAA)_ IEEE, 2015, pp. 1–10
  * [143] David H Wolpert  “Stacked generalization”  In _Neural networks_ 5.2 Elsevier, 1992, pp. 241–259
  * [144] Lin Xu, Frank Hutter, Holger H Hoos and Kevin Leyton-Brown  “SATzilla: portfolio-based algorithm selection for SAT”  In _Journal of artificial intelligence research_ 32, 2008, pp. 565–606
  * [145] L. Yang and A. Shami  “On hyperparameter optimization of machine learning algorithms: Theory and practice”  In _Neurocomputing_ 415, 2020, pp. 295–316
  * [146] Quanming Yao et al.  “Taking Human out of Learning Applications: A Survey on Automated Machine Learning”, 2019  arXiv:1810.13306 [cs.AI]
  * [147] Tong Yu and Hong Zhu  “Hyper-Parameter Optimization: A Review of Algorithms and Applications”, 2020  arXiv:2003.05689 [cs.LG]
  * [148] Arber Zela, Aaron Klein, Stefan Falkner and Frank Hutter  “Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search”  In _ICML 2018 AutoML Workshop_ , 2018
  * [149] Z. Zhang, X. Wang and W. Zhu  “Automated Machine Learning on Graphs: A Survey”  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021_ ijcai.org, 2021, pp. 4704–4712
  * [150] Jun Zheng, Zilong Li, Liang Gao and Guosheng Jiang  “A parameterized lower confidence bounding scheme for adaptive metamodel-based design optimization”  In _Engineering Computations_ Emerald Group Publishing Limited, 2016
  * [151] Lucas Zimmer, Marius Lindauer and Frank Hutter  “Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021, pp. 1–12

## ?refname?

  * [152] M. Abadi et al.  “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems”, 2015  URL: https://www.tensorflow.org/
  * [153] T. Akiba et al.  “Optuna: A Next-Generation Hyperparameter Optimization Framework”  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery $\&$ Data Mining_, KDD ’19, 2019, pp. 2623–2631  DOI: 10.1145/3292500.3330701
  * [154] JJ Allaire and François Chollet  “keras: R Interface to ’Keras”’ R package version 2.3.0.0, 2020  URL: https://CRAN.R-project.org/package=keras
  * [155] A. Aronio de Romblay, N. Cherel, M. Maskani and H. Gerard  “MLBox”  URL: https://mlbox.readthedocs.io/en/latest/index.html
  * [156] Sunil Arya et al.  “An optimal algorithm for approximate nearest neighbor searching fixed dimensions”  In _Journal of the ACM (JACM)_ 45.6 ACM New York, NY, USA, 1998, pp. 891–923  DOI: 10.1145/293347.293348
  * [157] N. Awad, N. Mallik and F. Hutter  “DEHB: Evolutionary Hyperband for Scalable, Robust and Efficient Hyperparameter Optimization”  URL: https://github.com/automl/DEHB
  * [158] Maximilian Balandat et al.  “BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization”  In _Advances in Neural Information Processing Systems_ 33, 2020
  * [159] J. Bergstra, R. Bardenet, Y. Bengio and B. Kégl  “Algorithms for Hyper-Parameter Optimization”  In _Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems (NeurIPS’11)_ , 2011, pp. 2546–2554
  * [160] James Bergstra, Daniel Yamins and David Cox  “Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures”  In _Proceedings of the 30th International Conference on Machine Learning_ 28.1, Proceedings of Machine Learning Research  Atlanta, Georgia, USA: PMLR, 2013, pp. 115–123
  * [161] Alina Beygelzimer et al.  “FNN: Fast Nearest Neighbor Search Algorithms and Applications” R package version 1.1.3, 2019  URL: https://CRAN.R-project.org/package=FNN
  * [162] Martin Binder, Julia Moosbauer, Janek Thomas and Bernd Bischl  “Multi-objective hyperparameter tuning and feature selection using filter ensembles”  In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , 2020, pp. 471–479
  * [163] Martin Binder et al.  “mlr3pipelines - Flexible Machine Learning Pipelines in R”  In _Journal of Machine Learning Research_ 22.184, 2021, pp. 1–7
  * [164] Bernd Bischl et al.  “mlr: Machine Learning in R”  In _Journal of Machine Learning Research_ 17.170, 2016, pp. 1–5
  * [165] Bernd Bischl et al.  “mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions”, 2017  arXiv: http://arxiv.org/abs/1703.03373
  * [166] Bernd Bischl, Julia Schiffner and Claus Weihs  “Benchmarking classification algorithms on high-performance computing clusters”  In _Data Analysis, Machine Learning and Knowledge Discovery_ Springer, 2014, pp. 23–31
  * [167] Andrea Bommert et al.  “Benchmark for filter methods for feature selection in high-dimensional classification data”  In _Computational Statistics & Data Analysis_ 143 Elsevier, 2020, pp. 106839
  * [168] Hans Werner Borchers  “adagio: Discrete and Global Optimization Routines” R package version 0.7.1, 2018  URL: https://CRAN.R-project.org/package=adagio
  * [169] Bernhard E. Boser, Isabelle M. Guyon and Vladimir N. Vapnik  “A training algorithm for optimal margin classifiers”  In _Proceedings of the fifth annual workshop on Computational learning theory_ New York, NY, USA: Association for Computing Machinery, 1992, pp. 144–152  DOI: 10.1145/130385.130401
  * [170] Jakob Bossek  “ecr 2.0: A modular framework for evolutionary computation in R”  In _Proceedings of the Genetic and Evolutionary Computation Conference Companion_ , 2017, pp. 1187–1193  DOI: 10.1145/3067695.3082470
  * [171] X. Bouthillier et al.  “Epistimio/orion: Plotting API and Database commands”  Zenodo, 2020  DOI: 10.5281/zenodo.3478592
  * [172] Leo Breiman  “Random forests”  In _Machine Learning_ 45.1 Springer, 2001, pp. 5–32  DOI: 10.1023/A:1010933404324
  * [173] Leo Breiman, Jerome Friedman, Charles J Stone and Richard A Olshen  “Classification and regression trees”  CRC press, 1984
  * [174] Chih-Chung Chang and Chih-Jen Lin  “LIBSVM: A library for support vector machines”  In _ACM Transactions on Intelligent Systems and Technology_ 2, 2011, pp. 27:1–27:27  DOI: 10.1145/1961189.1961199
  * [175] N.. Chawla, K.. Bowyer, L.. Hall and W.. Kegelmeyer  “SMOTE: Synthetic Minority Over-sampling Technique”  In _Journal of Artificial Intelligence Research_ 16 AI Access Foundation, 2002, pp. 321–357  DOI: 10.1613/jair.953
  * [176] T. Chen et al.  “MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems”  In _CoRR_ abs/1512.01274, 2015
  * [177] Tianqi Chen and Carlos Guestrin  “XGBoost: A Scalable Tree Boosting System”  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16  San Francisco, California, USA: ACM, 2016, pp. 785–794  DOI: 10.1145/2939672.2939785
  * [178] Tianqi Chen et al.  “XGBoost: Extreme Gradient Boosting” R package version 1.2.0.1, 2020  URL: https://CRAN.R-project.org/package=xgboost
  * [179] Corinna Cortes and Vladimir Vapnik  “Support-vector networks”  In _Machine Learning_ 20.3, 1995, pp. 273–297  DOI: 10.1023/A:1022627411411
  * [180] T. Cover and P. Hart  “Nearest neighbor pattern classification”  In _IEEE Transactions on Information Theory_ 13.1, 1967, pp. 21–27  DOI: 10.1109/TIT.1967.1053964
  * [181] “dask”  URL: https://dask.org/
  * [182] A.. G. et al.  “GPflow: A Gaussian process library using TensorFlow”  In _Journal of Machine Learning Research_ 18.40, 2017, pp. 1–6
  * [183] Yufeng Ding and Jeffrey S. Simonoff  “An Investigation of Missing Data Methods for Classification Trees Applied to Binary Response Data”  In _Journal of Machine Learning Research_ 11.6, 2010, pp. 131–170
  * [184] Harris Drucker et al.  “Support Vector Regression Machines”  In _Proceedings of the 9th International Conference on Neural Information Processing Systems_ , NIPS’96  Denver, Colorado: MIT Press, 1996, pp. 155–161
  * [185] Peter Dunn and Gordon Smyth  “Generalized Linear Models With Examples in R”, Springer Texts in Statistics  New York: Springer, 2018  DOI: 10.1007/978-1-4419-0118-7
  * [186] K. Eggensperger, M. Lindauer and F. Hutter  “Pitfalls and Best Practices in Algorithm Configuration”  In _Journal of Artificial Intelligence Research_ , 2019, pp. 861–893
  * [187] Thomas Elsken, Jan Hendrik Metzen and Frank Hutter  “Neural Architecture Search: A Survey”  In _Journal of Machine Learning Research_ 20.55, 2019, pp. 1–21
  * [188] N. Erickson et al.  “AutoGluon”
  * [189] David Eriksson et al.  “Scalable Global Optimization via Local Bayesian Optimization”  In _Advances in Neural Information Processing Systems_ , 2019, pp. 5496–5507
  * [190] S. Estévez-Velarde, Y. Gutiérrez, Y. Almeida-Cruz and A. Montoyo  “General-purpose hierarchical optimisation of machine learning pipelines with grammatical evolution”  In _Information Sciences_ Elsevier, 2020  DOI: 10.1016/j.ins.2020.07.035
  * [191] Daniel Falbel and Javier Luraschi  “torch: Tensors and Neural Networks with ’GPU’ Acceleration” R package version 0.1.1, 2020  URL: https://CRAN.R-project.org/package=torch
  * [192] S. Falkner, A. Klein and F. Hutter  “BOHB: Robust and Efficient Hyperparameter Optimization at Scale”  In _Proceedings of the 35th International Conference on Machine Learning (ICML’18)_ 80 Proceedings of Machine Learning Research, 2018, pp. 1437–1446
  * [193] Manuel Fernández-Delgado, Eva Cernadas, Senén Barro and Dinani Amorim  “Do We Need Hundreds of Classifiers to Solve Real World Classification Problems?”  In _The Journal of Machine Learning Research_ 15.1 JMLR.org, 2014, pp. 3133–3181
  * [194] M. Feurer et al.  “Efficient and Robust Automated Machine Learning”  In _Proceedings of the 28th International Conference on Advances in Neural Information Processing Systems (NeurIPS’15)_ , 2015, pp. 2962–2970
  * [195] F. Fortin et al.  “DEAP: Evolutionary Algorithms Made Easy”  In _Journal of Machine Learning Research_ 13, 2012, pp. 2171–2175
  * [196] Yoav Freund  “Boosting a weak learning algorithm by majority”  In _Information and Computation_ 121.2, 1995, pp. 256–285  DOI: 10.1006/inco.1995.1136
  * [197] Yoav Freund and Robert E Schapire  “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting”  In _Journal of Computer and System Sciences_ 55.1, 1997, pp. 119–139  DOI: 10.1006/jcss.1997.1504
  * [198] Jerome Friedman  “Greedy function approximation: a gradient boosting machine”  In _The Annals of Statistics_ 29.5, 2001, pp. 1189–1232  DOI: 10.1214/aos/1013203451
  * [199] Jerome Friedman, Trevor Hastie and Robert Tibshirani  “Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)”  In _The Annals of Statistics_ 28.2, 2000, pp. 337–407  DOI: 10.1214/aos/1016218223
  * [200] Jerome Friedman, Trevor Hastie and Robert Tibshirani  “Regularization Paths for Generalized Linear Models via Coordinate Descent”  In _Journal of Statistical Software_ 33.1, 2010, pp. 1–22  DOI: 10.18637/jss.v033.i01
  * [201] Unai Garciarena and Roberto Santana  “An extensive analysis of the interaction between missing data types, imputation methods, and supervised classifiers”  In _Expert Systems with Applications_ 89 Elsevier BV, 2017, pp. 52–65  DOI: 10.1016/j.eswa.2017.07.026
  * [202] J. Gardner et al.  “GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration”  In _Advances in Neural Information Processing Systems_ , 2018
  * [203] M. Gelbart, J. Snoek and R. Adams  “Bayesian Optimization with Unknown Constraints”  In _Proceedings of the 30th conference on Uncertainty in Artificial Intelligence (UAI’14)_ AUAI Press, 2014, pp. 250–258
  * [204] Brandon Greenwell, Bradley Boehmke, Jay Cunningham and GBM Developers  “gbm: Generalized Boosted Regression Models” R package version 2.1.8, 2020  URL: https://CRAN.R-project.org/package=gbm
  * [205] Sebastian Gruber and Bernd Bischl  “mlr3hyperband: Hyperband for ’mlr3”’, 2019  URL: https://mlr3hyperband.mlr-org.com
  * [206] Isabelle Guyon and André Elisseeff  “An Introduction to Variable and Feature Selection”  In _The Journal of Machine Learning Research_ 3, 2003, pp. 1157–1182
  * [207] Haibo He and Yunqian Ma  “Imbalanced learning: foundations, algorithms, and applications”  Wiley-IEEE Press, 2013
  * [208] Klaus Hechenbichler and Klaus Schliep  “Weighted k-nearest-neighbor techniques and ordinal classification”, 2004
  * [209] Arthur E. Hoerl and Robert W. Kennard  “Ridge regression: Biased estimation for nonorthogonal problems”  In _Technometrics_ 12.1, 1970, pp. 55–67  DOI: 10.1080/00401706.1970.10488634
  * [210] Torsten Hothorn et al.  “Model-based Boosting 2.0”  In _Journal of Machine Learning Research_ 11, 2010, pp. 2109–2113
  * [211] Torsten Hothorn, Kurt Hornik and Achim Zeileis  “Unbiased recursive partitioning: A conditional inference framework”  In _Journal of Computational and Graphical Statistics_ 15.3, 2006, pp. 651–674  DOI: 10.1198/106186006X1339
  * [212] Torsten Hothorn and Achim Zeileis  “partykit: A modular toolkit for recursive partytioning in R”  In _The Journal of Machine Learning Research_ 16.1 JMLR. org, 2015, pp. 3905–3909
  * [213] F. Hutter, H. Hoos and K. Leyton-Brown  “Sequential Model-Based Optimization for General Algorithm Configuration”  In _Proceedings of the Fifth International Conference on Learning and Intelligent Optimization (LION’11)_ 6683, Lecture Notes in Computer Science  Springer, 2011, pp. 507–523
  * [214] Anil Jadhav, Dhanya Pramod and Krishnan Ramanathan  “Comparison of Performance of Data Imputation Methods for Numeric Dataset”  In _Applied Artificial Intelligence_ 33.10 Informa UK Limited, 2019, pp. 913–933  DOI: 10.1080/08839514.2019.1637138
  * [215] H. Jin, Q. Song and X. Hu  “Auto-Keras: An Efficient Neural Architecture Search System”  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019, pp. 1946–1956  ACM
  * [216] Steven G Johnson  “The NLopt nonlinear-optimization package”, 2014  URL: https://github.com/stevengj/nlopt
  * [217] K. Kandasamy, J. Schneider and B. Póczos  “High Dimensional Bayesian Optimisation and Bandits via Additive Models”  In _Proceedings of the 32nd International Conference on Machine Learning (ICML’15)_ 37 Omnipress, 2015, pp. 295–304
  * [218] K. Kandasamy et al.  “Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly”  In _Journal of Machine Learning Research_ 21.81, 2020, pp. 1–27
  * [219] Alexandros Karatzoglou, Alex Smola, Kurt Hornik and Achim Zeileis  “kernlab – An S4 Package for Kernel Methods in R”  In _Journal of Statistical Software_ 11.9, 2004, pp. 1–20  URL: http://www.jstatsoft.org/v11/i09/
  * [220] N. Knudde, J. van der Herten, T. Dhaene and I. Couckuyt  “GPflowOpt”  URL: https://gpflowopt.readthedocs.io/en/latest/index.html
  * [221] Bartosz Krawczyk  “Learning from imbalanced data: open challenges and future directions”  In _Progress in Artificial Intelligence_ 5.4 Springer, 2016, pp. 221–232
  * [222] Max Kuhn  “Building Predictive Models in R Using the caret Package”  In _Journal of Statistical Software_ 28.5 Foundation for Open Access Statistic, 2008, pp. 1–26  DOI: 10.18637/jss.v028.i05
  * [223] Max Kuhn  “Futility Analysis in the Cross-Validation of Machine Learning Models”, 2014  arXiv: http://arxiv.org/abs/1405.6974
  * [224] Max Kuhn and Kjell Johnson  “Feature engineering and selection: A practical approach for predictive models”  CRC Press, 2019
  * [225] Max Kuhn and Hadley Wickham  “tidymodels: a collection of packages for modeling and machine learning using tidyverse principles.”, 2020  URL: https://www.tidymodels.org
  * [226] Michel Lang et al.  “mlr3: A modern object-oriented machine learning framework in R”  In _Journal of Open Source Software_ 4.44, 2019, pp. 1903  DOI: 10.21105/joss.01903
  * [227] E. LeDell and S. Poirier  “H2O AutoML: Scalable Automatic Machine Learning”  In _7th ICML Workshop on Automated Machine Learning (AutoML)_ , 2020
  * [228] Erin LeDell et al.  “h2o: R Interface for the ’H2O’ Scalable Machine Learning Platform”, 2020  URL: https://CRAN.R-project.org/package=h2o
  * [229] Lisha Li et al.  “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”  In _Journal of Machine Learning Research_ 18.185, 2018, pp. 1–52
  * [230] Yang Li et al.  “OpenBox: A Generalized Black-box Optimization Service”  In _KDD_ ACM, 2021, pp. 3209–3219
  * [231] R. Liaw et al.  “Tune”  URL: https://docs.ray.io/en/master/tune/index.html
  * [232] M. Lindauer et al.  “SMAC v3: Algorithm Configuration in Python”  GitHub, 2017  URL: https://github.com/automl/SMAC3
  * [233] Marius Lindauer et al.  “SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization”, 2021  arXiv:2109.09831 [cs.LG]
  * [234] Manuel López-Ibáñez et al.  “The irace package: Iterated Racing for Automatic Algorithm Configuration”  In _Operations Research Perspectives_ 3, 2016, pp. 43–58  DOI: 10.1016/j.orp.2016.09.002
  * [235] Walter R. Mebane, Jr. and Jasjeet S. Sekhon  “Genetic Optimization Using Derivatives: The rgenoud Package for R”  In _Journal of Statistical Software_ 42.11, 2011, pp. 1–26  DOI: 10.18637/jss.v042.i11
  * [236] Olaf Mersmann  “mco: Multiple Criteria Optimization Algorithms and Related Functions” R package version 1.0-15.1, 2014  URL: https://CRAN.R-project.org/package=mco
  * [237] David Meyer et al.  “e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien” R package version 1.7-3, 2019  URL: https://CRAN.R-project.org/package=e1071
  * [238] Daniele Micci-Barreca  “A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems”  In _ACM SIGKDD Explorations Newsletter_ 3.1 ACM New York, NY, USA, 2001, pp. 27–32
  * [239] K. Moore et al.  “TransmogrifAI”  URL: https://github.com/salesforce/TransmogrifAI
  * [240] Katharine Mullen et al.  “DEoptim: An R Package for Global Optimization by Differential Evolution”  In _Journal of Statistical Software_ 40.6, 2011, pp. 1–26  DOI: 10.18637/jss.v040.i06
  * [241] L. Nardi, D. Koeplinger and K. Olukotun  “Practical design space exploration”  In _2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)_ , 2019, pp. 347–358  IEEE
  * [242] John Ashworth Nelder and Robert WM Wedderburn  “Generalized linear models”  In _Journal of the Royal Statistical Society: Series A (General)_ 135.3, 1972, pp. 370–384  DOI: 10.2307/2344614
  * [243] F. Nogueira  “Bayesian Optimization: Open source constrained global optimization tool for Python”, 2014  URL: https://github.com/fmfn/BayesianOptimization
  * [244] C. Oh, E. Gavves and M. Welling  “BOCK : Bayesian Optimization with Cylindrical Kernels”  In _Proceedings of the 35th International Conference on Machine Learning (ICML’18)_ 80 Proceedings of Machine Learning Research, 2018, pp. 3865–3874
  * [245] R. Olson, N. Bartley, R. Urbanowicz and J. Moore  “Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science”  In _Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’16)_ ACM, 2016, pp. 485–492
  * [246] Florian Pargent, Florian Pfisterer, Janek Thomas and Bernd Bischl  “Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features”, 2021  arXiv:2104.00629 [stat.ML]
  * [247] A. Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurIPS’19)_ , 2019, pp. 8024–8035
  * [248] F. Pedregosa et al.  “Scikit-learn: Machine Learning in Python”  In _Journal of Machine Learning Research_ 12, 2011, pp. 2825–2830
  * [249] John Platt  “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods”  In _Advances in Large Margin Classifiers_ MIT Press, 1999, pp. 61–74
  * [250] Philipp Probst and Anne-Laure Boulesteix  “To tune or not to tune the number of trees in random forest”  In _Journal of Machine Learning Research_ 18.1, 2017, pp. 6673–6690
  * [251] Philipp Probst, Anne-Laure Boulesteix and Bernd Bischl  “Tunability: Importance of Hyperparameters of Machine Learning Algorithms.”  In _Journal of Machine Learning Research_ 20.53, 2019, pp. 1–32
  * [252] Philipp Probst, Marvin N Wright and Anne-Laure Boulesteix  “Hyperparameters and tuning strategies for random forest”  In _WIREs Data Mining and Knowledge Discovery_ 9.3 Wiley Online Library, 2019, pp. e1301  DOI: 10.1002/widm.1301
  * [253] J. Rapin and O. Teytaud  “Nevergrad - A gradient-free optimization platform”  In _GitHub repository_ GitHub, 2018  URL: https://GitHub.com/FacebookResearch/Nevergrad
  * [254] Olivier Roustant, David Ginsbourger and Yves Deville  “DiceKriging, DiceOptim: Two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization”  In _Journal of Statistical Software_ 51, 2012  DOI: 10.18637/jss.v051.i01
  * [255] Md Sahidullah and Goutam Saha  “Design, analysis and experimental evaluation of block based transformation in MFCC computation for speaker recognition”  In _Speech communication_ 54.4 Elsevier, 2012, pp. 543–565
  * [256] Daniel Schalk, Janek Thomas and Bernd Bischl  “compboost: Modular Framework for Component-Wise Boosting”  In _Journal of Open Source Software_ 3.30, 2018, pp. 967  DOI: 10.21105/joss.00967
  * [257] Robert E. Schapire  “The strength of weak learnability”  In _Machine Learning_ 5.2, 1990, pp. 197–227  DOI: 10.1007/BF00116037
  * [258] Klaus Schliep and Klaus Hechenbichler  “kknn: Weighted k-Nearest Neighbors” R package version 1.3.1, 2016  URL: https://CRAN.R-project.org/package=kknn
  * [259] Bernhard Schölkopf, Alex J. Smola, Robert C. Williamson and Peter L. Bartlett  “New support vector algorithms”  In _Neural Computation_ 12.5, 2000, pp. 1207–1245  DOI: 10.1162/089976600300015565
  * [260] “scikit-optimize”  URL: https://scikit-optimize.github.io/
  * [261] J. Snoek  “Spearmint”  URL: https://github.com/HIPS/Spearmint/
  * [262] J. Snoek, H. Larochelle and R. Adams  “Practical Bayesian Optimization of Machine Learning Algorithms”  In _Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems (NeurIPS’12)_ , 2012, pp. 2960–2968
  * [263] J. Snoek, K. Swersky, R. Zemel and R. Adams  “Input Warping for Bayesian Optimization of Non-stationary Functions”  In _Proceedings of the 31th International Conference on Machine Learning, (ICML’14)_ Omnipress, 2014, pp. 1674–1682
  * [264] N. Stander and K. Craig  “On the robustness of a simple domain reduction scheme for simulation‐based optimization”  In _Engineering Computations_ 19, 2002, pp. 431–450
  * [265] Robert E. Sweeney and Edwin F. Ulveling  “A Transformation for Simplifying the Interpretation of Coefficients of Binary Variables in Regression Analysis”  In _The American Statistician_ 26.5 American Statistical Association, Taylor & Francis, Ltd., 1972, pp. 30–32  DOI: 10.2307/2683780
  * [266] Terry Therneau and Beth Atkinson  “rpart: Recursive Partitioning and Regression Trees” R package version 4.1-15, 2019  URL: https://CRAN.R-project.org/package=rpart
  * [267] C. Thornton, F. Hutter, H. Hoos and K. Leyton-Brown  “Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms”  In _The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’13)_ ACM Press, 2013, pp. 847–855
  * [268] Robert Tibshirani  “Regression shrinkage and selection via the lasso”  In _Journal of the Royal Statistical Society: Series B (Methodological)_ 58.1, 1996, pp. 267–288  DOI: 10.1111/j.2517-6161.1996.tb02080.x
  * [269] Heike Trautmann, Olaf Mersmann and David Arnu  “cmaes: Covariance Matrix Adapting Evolution Strategy” R package version 1.0-11, 2011  URL: https://CRAN.R-project.org/package=cmaes
  * [270] Mark J. Laan, Eric C Polley and Alan E. Hubbard  “Super Learner”  In _Statistical Applications in Genetics and Molecular Biology_ 6.1 Walter de Gruyter GmbH, 2007, pp. Article 25  DOI: 10.2202/1544-6115.1309
  * [271] Yap Bee Wah et al.  “Feature Selection Methods: Case of Filter and Wrapper Approaches for Maximising Classification Accuracy.”  In _Pertanika Journal of Science & Technology_ 26.1, 2018
  * [272] Katarzyna Woźnica and Przemysław Biecek  “Does imputation matter? Benchmark for predictive models”, 2020  arXiv: http://arxiv.org/abs/2007.02837
  * [273] Marvin Wright and Andreas Ziegler  “ranger: A fast implementation of random forests for high dimensional data in C++ and R”  In _Journal of Statistical Software_ 77.1, 2017, pp. 1–17  DOI: 10.18637/jss.v077.i01
  * [274] Yang Xiang, Sylvain Gubian, Brian Suomela and Julia Hoeng  “Generalized Simulated Annealing for Efficient Global Optimization: the GenSA Package for R.”  In _The R Journal_ 5.1, 2013, pp. 13–28  DOI: 10.32614/RJ-2013-002
  * [275] Bing Xue, Mengjie Zhang and Will N Browne  “A comprehensive comparison on evolutionary feature selection approaches to classification”  In _International Journal of Computational Intelligence and Applications_ 14.02 World Scientific, 2015, pp. 1550008
  * [276] Yachen Yan  “rBayesianOptimization: Bayesian Optimization of Hyperparameters” R package version 1.1.0, 2016  URL: https://CRAN.R-project.org/package=rBayesianOptimization
  * [277] Yaohui Zeng and Patrick Breheny  “The biglasso Package: A Memory- and Computation-Efficient Solver for Lasso Model Fitting with Big Data in R”, 2018  arXiv:1701.05936 [stat.CO]
  * [278] L. Zimmer, M. Lindauer and F. Hutter  “Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL” IEEE early access; also available under https://arxiv.org/abs/2006.13799  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021, pp. 1–12
  * [279] Barret Zoph, Vijay Vasudevan, Jonathon Shlens and Quoc V. Le  “Learning Transferable Architectures for Scalable Image Recognition”, 2018, pp. 8697–8710
  * [280] Hui Zou and Trevor Hastie  “Regularization and variable selection via the elastic net”  In _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 67.2, 2005, pp. 301–320  DOI: 10.1111/j.1467-9868.2005.00503.x

appendix_A_learners_and_search_spaces

##  ?appendixname? A Preprocessing

Before applying a learner, it may be helpful or even necessary to modify the
data before training in a step called _preprocessing_. Preprocessing serves
mainly two purposes:  (i) it either transforms the data to fit the technical
requirements of a specific learner, e.g., by converting categorical features
to numerical features or by imputing missing values; or  (ii) it modifies the
data to improve predictive performance, e.g., by bringing features on a common
scale or by extracting components of a date column.

###  A.1 Missing Data Imputation

Imputation describes the task of replacing missing values with feasible ones,
since many ML algorithms do not natively handle missing values. In practice,
data is often not missing fully at random, and therefore the fact that part of
the data is missing itself might be informative for predicting the target
label. Hence, it is generally recommended to preserve the information of
whether a certain value has been imputed. For categorical features, we often
simply introduce a new level for missing values. For numerical or integer
features, the simplest techniques impute with a constant mean or median value
per feature column or sample from the empirical univariate distribution of the
feature. To preserve the missing information in this case, an additional
binary dummy column can be added per imputed feature column. When tree-based
methods are used, it is often preferred to impute missing values with very
large or very small values outside the range of the training data [183], as
the tree can then split between available and imputed data, thereby preserving
the information about the fact that data is missing without the need for an
extra column. Much more elaborate methods exist, often based on multiple
imputation, but these are mainly geared towards ensuring proper inference. For
purely predictive tasks, it is still somewhat unclear how beneficial these
more elaborate imputation methods are. More detailed discussions are available
in [201, 214, 272].

###  A.2 Encoding of Categorical Features

Many ML algorithms operate on purely numerical inputs, and hence, we often
need an additional numerical encoding procedure for categorical features.
Dummy encoding, also called one-hot encoding, is arguably the most popular
choice. It replaces a categorical $k$-level feature with $k$ or $k-1$ binary
indicator features. The latter case avoids multicollinearity of the newly
created features by using the $k$-th level as a reference level, encoding it
by setting all indicator features to $0$. For data sets with many categorical
features and data sets with categorical features with many levels (high-
cardinality features), this leads to a considerable increase in
dimensionality, resulting in increased runtime and memory requirements. In
such cases, methods like impact encoding – also called target encoding [265] –
replace the categorical feature with only a single numeric feature, based on
the average outcome value for all observations for which that level has been
observed. [246] have shown that target encoding using generalized linear mixed
models (GLMMs) [238] tends to outperform other common encoding methods in
settings where categorical feature cardinality is high.

Problems arise if factor levels occur during prediction that have not been
encountered during training. To some extent, this can be addressed by merging
infrequent factor levels to a new level with value ?other?, or by stratifying
on the levels during resampling, but can never be completely ruled out. Since
the new level is unknown to the model, it can be considered a special case of
missing data and could therefore be handled by imputation. A typical HP of the
feature encoding process would be the threshold that decides when to perform
target encoding instead of dummy encoding, depending on the number of levels
of a feature.

Several benchmarks of encoding methods have been performed, typically with
different emphasis on setting, such as the work by [246] and the references
contained therein.

###  A.3 Feature Filtering

Feature filtering is used to select a subset of features prior to model
fitting. Subsetting features can serve two purposes:  (i) either to reduce the
computational cost, or  (ii) to eliminate noise variables that may deteriorate
model performance.  Feature filters often assign a numerical score to each
feature, usually based on simple dependency measures between each feature and
the outcome, e.g., mutual information or correlation. The percentage of
features to keep is typically an HP. Note that feature filtering is one of
several possibilities to perform feature selection. See [206] for a
description of feature wrappers and embedded feature selection methods.

See [271] for a benchmark assessing filter methods applied to simulated
datasets and the correctness of selected features. See also [167] as well as
[275] for a benchmark assessing filters with respect to predictive performance
as well as secondary aspects of interest such as runtime. [162] evaluates a
variety of feature filters with respect to their similarity and their
suitability for constructing a feature filter ensemble, which can subsequently
be tuned for a data set at hand with BO.

###  A.4 Data Augmentation and Sampling

Data augmentation has the goal of improving predictive performance by adding
or removing rows of the training data. A common use-case for data augmentation
is classification with imbalanced class labels. Here, we can add additional
observations by oversampling, i.e., repeating observations of the minority
class. A more flexible alternative is SMOTE (“Synthetic Minority Oversampling
Technique”, [175]), where convex combinations of observations from the
minority class are added to the training data. When the available dataset is
very large, it is also possible to undersample in order to correct for class
imbalance, i.e., filtering out observations of the majority class. Also,
subsampling the data set as a whole is an old trick to reduce training time
during HPO, and modern multifidelity approaches exploit this in a more
principled manner. Sampling and data augmentation methods differ from many
other preprocessing methods in that they only affect the training phase of a
model. Typical HPs include the under-, over-, or subsampling ratio or the
number of observations to use for the convex combinations in SMOTE.

For a more thorough description of how to treat imbalanced data in ML, see
[221]. Current challenges and areas of research are described in [207].

###  A.5 Feature Extraction

A crucial step in improving predictive performance of a model is feature
extraction or feature engineering. The key idea is to add new features to the
data that the model can exploit better than the original data. This is often
both domain-specific and model-specific, and therefore is hard to generalize.
For linear models, for example, adding polynomial terms can provide a better
view on the data, effectively allowing the model to capture non-linear effects
and higher-order interactions. Simple tree-based methods might profit from
methods like principal component analysis, which rotate all or a subset of
numerical features to align feature dimensions with the (orthogonal)
directions of major variance. On the other hand, such simple extractions
techniques very often do not provide competitive benefit in terms of
predictive performance when many non-parametric models are included in model
selection and HPO [166].

More beneficial is often domain-specific feature extraction for more complex
data types. For example, for functional data features, where a collection of
columns represents a curve, one can either extract generic characteristics
(mean value, maximum value, average slope, number of peaks, etc.) or run
domain-specific extraction, e.g., we might extract mel-frequency cepstral
coefficients (MFCCs) [255] for an audio signal. For a practical guide to using
feature extraction in predictive modeling, see e.g. [224].

All extracted features can either be used instead or in addition to the
original features. As it is often highly unclear how to perform ideal feature
extraction for a specific task, it seems particularly attractive if an ML
software system allows the user to embed custom extraction code into the
preprocessing with exposed custom HPs so that a flexible feature extraction
piece of code can be automatically configured by the tuner in a data-dependent
manner.

##  ?appendixname? B Evalution Metrics

Name | Formula | Direction | Range |  Description
---|---|---|---|---
Performance measures for regression
Mean Squared Error (MSE) | $\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}}\left(y^{(i)}-\hat{y}^{(i)}\right)^{2}$ | min | $[0,\infty)$ |  Mean of the squared distances between the target variable $y$ and the predicted target $\hat{y}$.
Mean Absolute Error (MAE) | $\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}}\left\lvert y^{(i)}-\hat{y}^{(i)}\right\rvert$ | min | $[0,\infty)$ |  More robust than MSE, since it is less influenced by large errors.
$R^{2}$ | $1-\frac{\sum_{i=1}^{n_{\mathrm{test}}}\left(y^{(i)}-\hat{y}^{(i)}\right)^{2}}{\sum_{i=1}^{n_{\mathrm{test}}}\left(y^{(i)}-\bar{y}\right)^{2}}$ | max | $(-\infty,1]$ |  Compare the sum of squared errors (SSE) of the model to a constant baseline model.
Performance measures for classification based on class labels
Accuracy (ACC) | $\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}}\mathbb{I}_{\left\\{y^{(i)}=\hat{y}^{(i)}\right\\}}$ | max | $[0,1]$ |  Proportion of correctly classified observations.
Balanced Accuracy (BA) | $\frac{1}{g}\sum_{k=1}^{g}\frac{1}{n_{\text{test},k}}\sum_{y^{(i)}:y^{(i)}=k}\mathbb{I}_{\left\\{y^{(i)}=\hat{y}^{(i)}\right\\}}$ | max | $[0,1]$ |  Variant of the accuracy that accounts for imbalanced classes.
Classification Error (CE) | $\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}}\mathbb{I}_{\left\\{y^{(i)}\neq\hat{y}^{(i)}\right\\}}$ | min | $[0,1]$ |  $\mathrm{CE}=1-\mathrm{ACC}$ is the proportion of incorrect predictions.
ROC measures | $\mathrm{TPR}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$ | max | $[0,1]$ |  True Positive Rate: how many observations of the positive class 1 are predicted as 1?
| $\mathrm{FPR}=\frac{\mathrm{FP}}{\mathrm{TN}+\mathrm{FP}}$ | min | $[0,1]$ |  False Positive Rate: how many observations of the negative class 0 are falsely predicted as 1?
| $\mathrm{TNR}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$ | max | $[0,1]$ |  True Negative Rate: how many observations of the negative class 0 are predicted as 0?
| $\mathrm{FNR}=\frac{\mathrm{FN}}{\mathrm{TP}+\mathrm{FN}}$ | min | $[0,1]$ |  False Negative Rate: how many observations of the positive class 1 were falsely predicted as 0?
| $\mathrm{PPV}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$ | max | $[0,1]$ |  Positive Predictive Value: how likely is a predicted 1 a true 1?
| $\mathrm{NPV}=\frac{\mathrm{TN}}{\mathrm{FN}+\mathrm{TN}}$ | max | $[0,1]$ |  Negative Predictive Value: how likely is a predicted 0 a true 0?
$F_{1}$ | $2\frac{\mathrm{PPV}\cdot\mathrm{TPR}}{\mathrm{PPV}+\mathrm{TPR}}$ | max | $[0,1]$ |  $F_{1}$ is the harmonic mean of PPV and TPR. Especially useful for imbalanced classes.
Cost measure | $\sum_{i=1}^{n_{\mathrm{test}}}C(y^{(i)},\hat{y}^{(i)})$ | min | $[0,\infty)$ |  Cost of incorrect predictions based on a (usually non-negative) cost matrix $C\in\mathds{R}^{g,g}$.
Performance measures for classification based on class probabilities
Brier Score (BS) | $\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}}\sum_{k=1}^{g}\left(\hat{\pi}_{k}(\mathbf{x}^{(i)})-\sigma_{k}(y^{(i)})\right)^{2}$ | min | $[0,1]$ |  Measures squared distances of probabilities from the one-hot encoded class labels.
Log-Loss (LL) | $\frac{1}{n_{\mathrm{test}}}\sum_{i=1}^{n_{\mathrm{test}}}\left(-\sum_{k=1}^{g}\sigma_{k}(y^{(i)})\log(\hat{\pi}_{k}(\mathbf{x}^{(i)}))\right)$ | min | $[0,\infty)$ |  A.k.a. Bernoulli, binomial or cross-entropy loss
AUC |  | max | $[0,1]$ |  Area under the ROC curve.
$\hat{y}^{(i)}$ denotes the predicted label for observation
$\mathbf{x}^{(i)}$. ACC, BA, CE, BS, and LL can be used for multi-class
classification with $g$ classes. For AUC, multiclass extensions exist as well.
The notation $\mathbb{I}_{\\{\cdot\\}}$ denotes the indicator function.
$\sigma_{k}(y)=\mathbb{I}_{\\{y=k\\}}$ is 1 if $y$ is class $k$, 0 otherwise
(multi-class one-hot encoding). $n_{\text{test},k}$ is the number of
observations in the test set with class $k$. $\hat{\pi}_{k}(\mathbf{x})$ is
the estimated probability for observation $\mathbf{x}^{(i)}$ of belonging to
class $k$. $TP$ is the number of true positives (observations of class 1 with
predicted class 1), $FP$ is the number of false positives (observations of
class 0 with predicted class 1), $TN$ is the number of true negatives
(observations of class 0 with predicted class 0), and $FN$ is the number of
false negatives (observations of class 1 with predicted class 0).
?tablename? 1: Popular performance measures used for ML, assuming an arbitrary
test set of size $n_{\mathrm{test}}$.

##  ?appendixname? C Software

This section lists relevant software packages in both R and Python. Libraries
providing popular machine learning algorithms are listed first, followed by
additional information about useful packages and relevant considerations for
HPO in both languages.

###  C.1 Software in R

####  C.1.1 Important Machine Learning Algorithms

Here we present a curated list of learners that work well in general and are
well integrated with ML frameworks in R (see Section C.1.3). For a
comprehensive list of R-packages that offer ML models, categorized by the type
of model implemented, see the “CRAN Task View” for ML and Statistical
Learning191919https://cran.r-project.org/view=MachineLearning.

#####  $k$-NN

The basic $k$-NN algorithm is implemented in the class package that comes
installed with R. The distance-weighted $k$-NN is implemented in the kknn
package [258]. For large data sets, one may prefer the approximate algorithms
[156] in the FNN package [161].

##### Regularized Linear Models

The elastic net is implemented in the glmnet package [200], which also offers
an internal, fast optimization of its regularization HPs via the cv.glmnet()
function. Both biglasso [277] and LiblineaR implement regularized linear
models specifically optimized for large datasets.

##### Support Vector Machines

A frequently used implementation of the SVM for R can be found in the e1071
package [237], which uses the LIBSVM library internally [174]. kernlab [219]
is a more comprehensive package with an emphasis on flexibility.

##### Decision Trees

The basic CART algorithm is implemented in the rpart R package [266], while a
more flexible implementation of decision trees is included in partykit [212]

##### Random Forests

Three of the most widely used R implementations of RF are provided in the
packages  (i) randomForest, implementing the original version [172],  (ii)
party, implementing (unbiased) conditional inference forests [211], and  (iii)
ranger, providing a newer implementation of RF, including additional variants
and a wide range of options, optimized for use on high-dimensional data [273].

##### Boosting

Popular implementations of gradient boosting with tree base learners can be
found in gbm [204] and implemented more efficiently in xgboost [177, 178].
Gradient boosting of generalized linear and additive models is implemented in
xgboost as well as in mboost [210], the latter being more efficiently
implemented in compboost [256].

##### Artificial Neural Networks

Modern and fast implementations of NNs are provided by the keras [154] and
torch [191] packages. torch is more low-level than keras, and some properties
that are explicit HPs of keras need to be implemented by the user through R
code in torch.

####  C.1.2 Black-Box and HP Optimizers

##### Evolution Strategies

Popular R packages that implement different ES are rgenoud [235], cmaes [269],
adagio [168], DEoptim [240], mco [236], and ecr [170] and mosmafs [162].

##### Bayesian Optimization

tune, rBayesianOptimization [276] and DiceOptim [254] are R packages that
implement BO with Gaussian processes and for purely numerical search spaces.
mlrMBO [165] and its successor mlr3mbo offer to construct a flexible class of
BO variants with arbitrary regression models and acquisition functions. These
packages can work with mixed and hierarchical search spaces.

##### Other Methods

Hyperband is implemented in the package mlr3hyperband [205]. Iterated F-racing
is provided by irace [234].

####  C.1.3 ML Frameworks

There are many technical difficulties and intricacies to attend to when
practically combining a black-box optimizer with different ML algorithms,
especially pipelines, to perform HPO. Although it would be possible to use one
of the algorithms shown above and write an objective function that performs
performance evaluation, this is not recommended. Instead, one of several ML
frameworks should be used. These frameworks simplify the process of model
fitting, evaluation, and optimization while managing the most common pitfalls
and providing robust and parallel technical execution. In the following
paragraphs, we summarize the tuning capabilities of the most important ML
frameworks for R. The feature matrix in Table 2 gives an encompassing overview
of the implemented capabilities.

| mlr | mlr3 | caret | tidymodels | h2o
---|---|---|---|---|---
Tuners |  |  |  |  |
\- Random Search | ✓ | ✓ | ✓ | ✓ | ✓
\- Grid Search | ✓ | ✓ | ✓ | ✓ | ✓
\- Evolutionary | ✓ | ✓ | ✗ | ✗ | ✗
\- Bayesian Optimization | ✓ | ✓ | ✗ | ✓ | ✗
\- Hyperband | ✗ | ✓ | ✗ | ✗ | ✗
\- Racing | ✓ | $\ast$ | ✓ | ✓ | ✗
\- Other Tuners | NSGA2, SA | NLOpt, SA | - | SA | -
Search Space |  |  |  |  |
\- Transformations | ✓ | ✓ | ✗ | ✓ | ✓
\- Default Search Spaces | ✓ | ✗ | i | ✓ | ✓
Joint Preproc Tuning | ✓ | ✓ | ✗ | ✓ | ✓
$\ast$: not yet released on CRAN but prototype available, i: see description
in main text.
?tablename? 2: Feature matrix of tuning capabilities of R’s ML frameworks.

##### mlr

[164] supports a broad range of optimizers: random search, grid search, CMA-ES
via cmaes, BO via mlrMBO [165], iterated F-racing via irace, NSGA2 via mco,
and simulated annealing via GenSA [274]. Random search, grid search, and NSGA2
also support multi-criteria optimization. Arbitrary search spaces can be
defined using the ParamHelpers package, which also supports HP
transformations. HP spaces of preprocessing steps and ML algorithms can be
jointly tuned with the mlrCPO package.

##### mlr3

[226] superseded mlr in 2019. This new package is designed with a more modular
structure in mind and offers a much improved system for pipelines in
mlr3pipelines [163]. mlr3tuning implements grid search, random search,
simulated annealing via GenSA, CMA-ES via cmaes, and non-linear optimization
via nloptr [216]. Additionally, mlr3hyperband provides Hyperband for
multifidelity HPO, and the miesmuschel package implements a flexible toolbox
for optimization using ES. BO is available through mlrMBO when the mlrintermbo
compatibility bridge package is used. A further extension for BO is currently
in development202020https://github.com/mlr-org/mlr3mbo. mlr3 uses the paradox
package, a re-implementation of ParamHelpers with a comparable set of
features.

##### caret

[222] ships with grid search, random search, and adaptive resampling (AR)
[223] – a racing approach where an iterative optimization algorithm favors
regions of empirically good predictive performance. Default HPs to tune are
encoded in the respective call of each ML algorithm. However, if forbidden
regions, transformations, or custom search spaces are required, the user must
specify a custom design to evaluate. Embedding preprocessing into the tuning
process is possible. However, tuning over the HPs of preprocessing methods is
not supported.

##### tidymodels

[225] is the successor of caret. This package ships with grid search and
random search, and the recently released tune package comes with Bayesian
optimization. The AR racing approach and simulated annealing can be found in
the finetune package. HP defaults and transformations are supported via dials.
HPs of preprocessing operations using tidymodels’ recipes pipeline system can
be tuned jointly with the HPs of the ML algorithm.

##### h2o

[228] connects to the H2O cross-platform ML framework written in Java. Unlike
the other discussed frameworks, which connect third-party packages from CRAN,
h2o ships with its own implementations of ML models. The package supports grid
search and random grid search, a variant of random search where points to be
evaluated are randomly sampled from a discrete grid. The possibilities for
preprocessing are limited to imputation, different encoders for categorical
features, and correcting for class imbalances via under- and oversampling. H2O
automatically constructs a search space for a given set of learning algorithms
and preprocessing methods, and HPs of both can be tuned jointly. It is worth
mentioning that h2o was developed with a strong focus on AutoML and offers the
functionality to perform random search over a pre-defined grid, evaluating
configurations of generalized linear models with elastic net penalty, xgboost
models, gradient boosting machines, and deep learning models. The best
performing configurations found by the AutoML system are then stacked together
[270] for the final model.

###  C.2 Software in Python

####  C.2.1 Machine Learning Package in Python

Scikit-learn [248, sklearn] is a general ML framework implemented in Python
and the default path for ML projects without the need for deep learning (DL).
The framework provides the most important traditional ML models (incl. SVM,
RFs, Boosting, etc.). This is further complemented by a variety of
preprocessing and post-processing methods (e.g., ensembling). The clean and
well-documented API allows users to easily build fairly complex pipelines,
which can provide strong performance on tabular data, e.g., see results on
auto-sklearn [194].

Modern DL frameworks such as Tensorflow [152], PyTorch [247] and MXNet [176]
accelerate large-scale computation task (in particular matrix multiplication)
by executing them on GPUs. Furthermore, GPflow [182] and GPyTorch [202] are
two GP libraries built on top of the two DL frameworks respectively and thus
can be again used as surrogate models for BO.

####  C.2.2 Open-Source HPO Packages in Python

The rapid development of the package landscape in Python has enabled many
developers to also provide HPO tools in Python. Here, we will give an
selective overview of commonly used and well-known packages.

Spearmint [262, 261] was one of the first successful open-source BO packages
for HPO. As proposed by [262], Spearmint implements standard BO with Markov
chain Monte Carlo (MCMC) integration of the acquisition function for a fully
Bayesian treatment of GP’s HPs. Additionally, the second version of Spearmint
allowed warping the input space with a Beta cumulative distribution function
[263] to deal with non-stationary loss landscapes. Spearmint also supports
constrained BO [203] and parallel optimization [262].

Hyperopt [160], as indicated by its name, is a distributed HPO framework.
Unlike other frameworks that estimate the potential performance given a
configuration, Hyperopt implements a Tree of Parzen Estimators (TPE) [159]
that estimates the distribution of configurations given their performance.
This approach allows it to easily run different configurations in parallel, as
multiple density estimators can be executed at the same time.

Scikit-optimize (skopt; scikit_optimize) is a simple yet efficient library to
optimize expensive and noisy black-box functions with BO. Skopt trains RF
(including decision trees and grading-boosting trees) and GP models based on
sklearn [248] as surrogate models. Similar to sklearn, skopt allows for
several ways of pre-processing input-features, e.g. one-hot encoding, log
transformation, normalization, label encoding, etc.

SMAC [213, 232] was originally developed as a tool for algorithm
configurations [186]. However, in recent years, it has also been successfully
used as a key component of several AutoML Tools, such as auto-sklearn [194]
and Auto-PyTorch [278]. SMAC implements both RF and GP as surrogate models to
handle various sorts of HP spaces. Additionally, BOHB [192] is implemented as
a multi-fidelity approach in SMAC; however, instead of the TPE model in BOHB,
SMAC utilized RF as a surrogate model for BO part. The most recent version of
SMAC [233] also allows for parallel optimization with dask [181].

Similarly, HyperMapper [241] also builds an RF as a surrogate model; thus, it
also supports mixed and structured HP configuration spaces. However,
HyperMapper does not implement a Bayesian Optimization approach in a strict
sense. In addition, HyperMapper incorporates previous knowledge by rescaling
the samples with a beta distribution and handles unknown constraints by
training another RF model as a probabilistic classifier. Furthermore, it
supports multi-objective optimization. As HyperMapper cannot be allocated to a
distributed computing environment, it might not be applicable to large-scale
HPO problems.

OpenBox [230] is a general framework for black-box optimization – including
HPO – and supports multi-objective optimization, multi-fidelity, early-
stopping, transfer learning, and parallel BO via distributed parallelization
under both synchronous parallel settings and asynchronous parallel settings.
OpenBox further implements an ensemble surrogate model to integrate the
information from previously seen similar tasks.

Dragonfly [218] is an open source library for scalable BO. Dragonfly extends
standard BO in the following ways to scale to higher dimensional and expensive
HPO problems: it implements GPs with additive kernels and additive GP-UCB
[217] as an acquisition function. In addition, it supports multi-fidelity
approach to scale to expensive HPO problems. To increase the robustness of the
system w.r.t. its HPs, Dragonfly acquires its GP HP by either sampling a set
of GP HPs from the posterior, conditioned on the data or optimizing the
likelihood to handle different degrees of smoothness of the optimized loss
landscape. In addition, Dragonfly maximizes its acquisition function with an
evolutionary algorithm, which enables the system to work on different sorts of
configuration spaces, e.g., different variable types and constraints.

The previously described BO-based HPO tools fix their search space during the
optimization phases. The package Bayesian Optimization [243] can concentrate a
domain around the current optimal values and adjust this domain dynamically
[264]. A similar idea is implemented in TurBO [189], which only samples inside
a trust region while keeping the configuration space fixed. The trust region
shrinks or extends based on the performance of TurBO’s suggested
configuration.

GPflowOpt [220] and BoTorch [158] are the BO-based HPO frameworks that build
on top of GPflow and GPyTorch, respectively. The auto-grading system
facilitates the users to freely extend their own ideas to existing models. Ax
adds an easy-to-use API on top of BoTorch.

DEHB [157] uses differential evolution (DE) and combines it with a multi-
fidelity HPO framework, inspired by BOHB’s combination of hyperband [229] and
BO. DEHB overcomes several drawbacks of BOHB: it does not require a surrogate
model, and thus, its runtime overhead does not grow over time; DEHB can have
stronger final performance compared to BOHB, especially for discrete-valued
and high-dimensional problems, where BO usually fails, as it tends to suggest
points on the boundary of the search space [244]; additionally, it is simpler
to implement an efficient parallel DE algorithm compared to a parallel BO-
based approach. For instance, DEAP [195] and Nevergrad [253] have many EA
implementations that enable parallel computation.

There are several other general HPO tools. For instance, Optuna [153] is an
automatic HP optimization software framework that allows to dynamically
construct the search space (Define-by-run API) and thus makes it much easier
for users to construct a highly complex search space. Oríon [171] is an
asynchronous framework for black-box function optimization. Finally, Tune
[231] is a scalable HP tuning framework that provides APIs for several
optimizers mentioned before, e.g., Dragonfly, SKopt, or HyperOpt.

####  C.2.3 AutoML tools

So far, we discussed several HPO tools in Python, which allow for flexible
applications of different algorithms, search spaces and data sets. Here, we
will briefly discuss several AutoML-packages of which HPO is a part.

Auto-Weka [267] is one of the first AutoML tools that automates the design
choice of entire ML packages, namely Weka, with SMAC. Similarly, the same
optimizer is applied to Auto-sklearn [194], while a meta-learning and ensemble
approach further boost the performance of auto-sklearn. Similarly, Auto-keras
[215] and Auto-PyTorch [278] use BO to automate the design choices of deep
NNs. TPOT [245] automates the design choice of an ML pipeline with genetic
algorithms, while a probabilistic grammatical evolution is implemented in
AutoGOAL [190] for HPO problems. H2O automl [227] and AutoGluon [188] train a
set of base models and ensemble them with a single layer (H2O) or multi-layer
stacks (AutoGluon). Additionally, TransmogrifAI [239] is an AutoML tool built
on the top of Apache Spark and MLBox [155].