For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning.
We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) improved default parameters for GBDTs and RealMLP.
We tune RealMLP and the default parameters on a meta-train benchmark with 71 classification and 47 regression datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 48 classification and 42 regression datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results show that RealMLP offers a better time-accuracy tradeoff than other neural nets and is competitive with GBDTs. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results on medium-sized tabular datasets (1K–500K samples) without hyperparameter tuning.

§ INTRODUCTION

Perhaps the most common type of data in practical machine learning (ML) is tabular data, characterized by a fixed number of numerical or categorical features (columns), lacking the spatiotemporal structure of most other data types such as image or text data. The moderate dimensionality and lack of symmetries make tabular data accessible to a wide variety of machine learning methods. While tabular data is very diverse and no method is dominant on all datasets, gradient-boosted decision trees (GBDTs) have demonstrated excellent results on benchmarks [Shwartz-Ziv and Armon, 2022, Grinsztajn et al., 2022, McElfresh et al., 2023], although their superiority has been challenged by a variety of deep learning methods [Borisov et al., 2022].

GBDTs and neural networks (NNs) are often compared using extensive hyperparameter optimization.
This can be especially expensive for NNs, as multilayer perceptrons (MLPs) and Transformer-based models are roughly one and two orders of magnitude slower than typical GBDTs, respectively [Grinsztajn et al., 2022, McElfresh et al., 2023].
To address this issue, we investigate the potential of improved MLPs and better dataset-independent default parameters for MLPs and GBDTs.
Specifically, we compare the library defaults (D) to our tuned defaults (TD) and (dataset-dependent) hyperparameter optimization (HPO).

Besides offering convenient baselines for quick exploration and benchmarking, good default parameters also play an important role in automated ML (AutoML). AutoGluon [Erickson et al., 2020] demonstrated that stacking and ensembling models with fixed parameters outperforms other AutoML approaches based on hyperparameter optimization [Gijsbers et al., 2022]. Without stacking and ensembling, McElfresh et al., 2023 have argued that light hyperparameter tuning for GBDTs is often more effective than trying out NNs. Here, we argue that even without stacking and ensembling, when using well-chosen default parameters for GBDTs and our improved MLPs, trying GBDTs and MLPs is often faster and more beneficial than (naively) optimizing the hyperparameters of a single method.

§.§ Contribution

The problem of finding better default parameters can be seen as a meta-learning problem [Vanschoren, 2018]. We employ a meta-train benchmark consisting of 118 datasets on which the default hyperparameters are optimized, and a disjoint meta-test benchmark consisting of 90 datasets on which they are evaluated. We consider separate default parameters for classification, optimized for classification error, and for regression, optimized for RMSE. Our benchmarks do not contain missing numerical values, and we restrict ourselves to sizes between 1K and 500K samples, cf. <Ref>.

In <Ref>, we introduce RealMLP, which improves on standard MLPs through a bag of tricks and better default parameters, tuned entirely on the meta-train benchmark. We introduce many novel or nonstandard components,
such as preprocessing using robust scaling and smooth clipping, a new numerical embedding variant, a diagonal weight layer, new schedules, different initialization methods, etc. Our benchmark results demonstrate that it outperforms other comparably fast NNs from the literature and can be competitive with GBDTs. We make RealMLP and the other models available through a scikit-learn interface.

In <Ref>, we provide new default parameters, tuned on the meta-train benchmark, for XGBoost [Chen and Guestrin, 2016], LightGBM [Ke et al., 2017], and CatBoost [Prokhorenkova et al., 2018]. While they cannot match HPO on average, they outperform the library defaults on the meta-test benchmark.

In <Ref>, we evaluate these and other models on the meta-test benchmark and the benchmark by Grinsztajn et al., 2022.
We also investigate several possibilities for algorithm selection and ensembling, demonstrating that algorithm selection over default methods provides a better time-performance tradeoff than HPO, thanks to our new improved default parameters and MLP.

Our code (including scikit-learn interfaces) for meta-train and meta-test benchmarks is available at

At <https://github.com/LeoGrin/tabular-benchmark/tree/better_by_default>, we provide code for the adapted Grinsztajn et al., 2022 benchmark. Our code and benchmark data will be archived at <https://doi.org/10.18419/darus-4255>.

§.§ Related Work

Neural networks [Borisov et al., 2022] review deep learning on tabular data and identify three main classes of methods: Data transformation methods, specialized architectures, and regularization models.
In particular, recent research has mainly focused on specialized architectures based on attention [Arik and Pfister, 2021, Huang et al., 2020, Gorishniy et al., 2021, Chen et al., 2023], including attention between datapoints [Ramsauer et al., 2020, Somepalli et al., 2022, Kossen et al., 2021, Schäfl et al., 2023, Gorishniy et al., 2023]. However, these methods are usually significantly slower than MLPs or even GBDTs [Grinsztajn et al., 2022, McElfresh et al., 2023, Gorishniy et al., 2023]. Our research instead expands on improvements to MLPs for tabular data such as the SELU activation function [Klambauer et al., 2017], bias initialization methods [Steinwart, 2019], regularization methods [Kadra et al., 2021], categorical embedding layers [Guo and Berkhahn, 2016], and numerical embedding layers [Gorishniy et al., 2022].

Benchmarks [Shwartz-Ziv and Armon, 2022] benchmarked three deep learning methods and noticed that they performed better on the datasets from their own papers than on other datasets. We address this issue by using more datasets and evaluating our methods on datasets that they were not tuned on. [Grinsztajn et al., 2022], [McElfresh et al., 2023], and [Ye et al., 2024] propose larger benchmarks and find that GBDTs still outperform deep learning methods on average, analyzing why and when this is the case. [Kohli et al., 2024] also emphasize the need for large benchmarks. We evaluate our methods on the benchmark by [Grinsztajn et al., 2022] as well as datasets from the AutoML benchmark [Gijsbers et al., 2022] and the OpenML-CTR23 regression benchmark [Fischer et al., 2023].

Better defaults Probst et al., 2019 study the tunability of ML methods, i.e., the difference in benchmark scores between the best fixed hyperparameters and tuned hyperparameters. While their approach involves finding better defaults, they do not evaluate them on a separate meta-test benchmark, only consider classification, and do not provide defaults for LightGBM, CatBoost, and NNs.

Meta-learning The problem of finding the best fixed hyperparameters is a meta-learning problem [Brazdil et al., 2008, Vanschoren, 2018].
Although we do not introduce or employ a fully automated method to find good defaults, we use a meta-learning benchmark setup to properly evaluate them.
[Wistuba et al., 2015] and [Pfisterer et al., 2021] learn portfolios of configurations and [van Rijn et al., 2018] learn symbolic defaults, but neither of the three papers considers GBDTs or NNs. [Salinas and Erickson, 2023] learn large portfolios of configurations on an extensive benchmark, without studying the best defaults for individual model families. Such portfolios are successfully applied in modern AutoML methods [Erickson et al., 2020, Feurer et al., 2022].
At the other end of the meta-learning spectrum, TabPFN [Hollmann et al., 2022] meta-learns a (tuning-free) learning method on small synthetic datasets. Unlike TabPFN, we only meta-learn hyperparameters and can therefore use fewer but larger and more realistic meta-train datasets, resulting in methods that scale to larger datasets.

§ METHODOLOGY

To evaluate a fixed hyperparameter configuration $\calH$, we need a collection $\calB$ of benchmark datasets and a scoring function that computes a benchmark score $\calS(\calB, \calH)$ by aggregating the errors attained by the method with hyperparameters $\calH$ on each dataset. However, when optimizing $\calH$ on $\calB$, we might overfit to the benchmark and therefore ideally need a second benchmark $\calB'$ to get an unbiased score for $\calH$.
We refer to $\calB, \calB'$ as meta-train and meta-test benchmarks and subdivide them into classification and regression benchmarks $\Ctrc$, $\Ctrr$, $\Ctec$, and $\Cter$.
Since the meta-train benchmark contains groups of datasets that are variants of the same dataset, for example by using different columns as targets, we use weighting factors inversely proportional to the group size.

<Ref> shows some characteristics of the meta-train and meta-test benchmarks. The meta-test benchmark includes datasets that are more extreme in several dimensions, allowing us to test whether our default parameters generalize out of distribution. For all datasets, we remove rows with missing numerical values and encode missing categorical values as a separate category. For regression, we standardize the targets on all datasets, such that $1$ is the RMSE of the best constant predictor.

§.§ Benchmark Data Selection

Characteristics of the meta-train and meta-test sets.

The meta-train set consists of medium-sized datasets from the UCI Repository [32], adapted from [Steinwart, 2019]. The meta-test set consists of datasets from the AutoML Benchmark [Gijsbers et al., 2022] and additional regression datasets from the OpenML-CTR23 benchmark [Fischer et al., 2023]. We subsample some large datasets and remove datasets that are already contained in the meta-train set, are too small, or have categories with too large cardinality. More details on the datasets and preprocessing can be found in <Ref>.

§.§ Aggregate Benchmark Score

To optimize the default parameters, we need to define a single benchmark score. To this end, we evaluate a method on $\Nsplits=10$ random training-validation-test splits (60%-20%-20%) on each dataset.
As metrics on individual dataset splits, we use classification error ($100\% - \text{accuracy}$) for classification and RMSE for regression.
There are various options to aggregate these errors into a single score. Some, such as average rank or mean normalized error, depend on which other methods are included in the evaluation, hindering an independent optimization. We would like to use the geometric mean error because arguably, an error reduction from $0.02$ to $0.01$ is more valuable than an error reduction from $0.42$ to $0.41$. However, since the geometric mean error is too sensitive to cases with zero error (especially for classification error), we instead use a shifted geometric mean error, where a small value $\eps \equalDef 0.01$ is added to the errors $\err_{ij}$ before taking the geometric mean:
SGM_ exp(∑_i=1^ w_i/ ∑_j=1^ log(_ij + )).

Here, we use weights $w_i = 1/\Nds$ on the meta-test set. On the meta-train set, we make the $w_i$ dependent on the number of related datasets, cf. <Ref>.

§ IMPROVED MLP

Here, we introduce RealMLP-TD, our improved MLP with tuned defaults, which we designed based on the meta-train benchmark. We also introduce a simplified version called RealMLP-TD-S.

Preprocessing and NN architecture for RealMLP-TD.
node distance=0.3cm,
every node/.style=draw,rounded corners,text centered,
boxnode/.style=draw,rectangle,minimum height=2cm,minimum width=2.5cm,

(onehot) [graynode] One-hot encoding;
(rescale) [below=of onehot, graynode] Robust scale;
(softclip) [below=of rescale, graynode] Smooth-clip;

(emb) [below=0.7cm of softclip] Num./cat. embeddings;
(scaling) [below=of emb] Learnable scaling;

(linear) [below=of scaling] Linear;
(activation) [below=of linear] Parametric activation;
(dropout) [below=of activation] Dropout;

(lastlinear) [below=of dropout] Linear;

[arrow] ++(0, 0.5) – (onehot.north);
[arrow] (onehot) – (rescale);
[arrow] (rescale) – (softclip);
[arrow] (softclip) – (emb);
[arrow] (emb) – (scaling);
[arrow] (scaling) – (linear);
[arrow] (linear) – (activation);
[arrow] (activation) – (dropout);
[arrow] (dropout) – (lastlinear);
[arrow] (lastlinear.south) – ++(0, -0.3);

[rounded corners=2mm] ($ (linear.north) + (-2.0, 0.15) $) rectangle ($ (dropout.south) + (2.0, -0.15) $);
[left=0.4cm of activation, draw=none] $3\times$;

The $\operatorname{coslog}_4$ and $\operatorname{flat\_cos}$ schedules.

From a vanilla MLP to RealMLP-TD.

Components of RealMLP-TD. In (c), we add one component in each step, tuning the best default learning rate for each step separately. The vanilla MLP uses categorical embeddings, a quantile transform to preprocess numerical features, default PyTorch initialization, ReLU activation, early stopping, and is optimized with Adam with default parameters. For more details, see <Ref>.
The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

Data preprocessing We first apply one-hot encoding to categorical columns with at most eight distinct values (not counting missing values). Binary categories are encoded to a single feature with values $\{-1, 1\}$. Missing values in categorical columns are encoded to zero. We then preprocess all numerical columns, including the one-hot encoded ones, independently as follows: Let $x_1, \hdots, x_n \in \bbR$ be the values in column $i$, and let $q_p$ be the $p$-quantile of $(x_1, \hdots, x_n)$ for $p \in [0, 1]$. Then,
x_j,processed         f(s_j ·(x_j - q_1/2)),     f(x) x/√(1 + (x/3)^2),

1/q_3/4 - q_1/4    ,  if  q_3/4 ≠q_1/4

2/q_1 - q_0    ,  if  q_3/4 = q_1/4 and q_1 ≠q_0

0    ,  otherwise.

In scikit-learn [Pedregosa et al., 2011], this corresponds to applying a  (first case) or  (second case), and then the function $f$, which smoothly clips its input to the range $(-3, 3)$. Smooth clipping functions like $f$ have been used by, e.g., Holzmüller et al., 2023 and Hafner et al., 2023. Intuitively, when features have large outliers, smooth clipping prevents the outliers from affecting the result too strongly, while robust scaling prevents the outliers from affecting the inlier scaling.

NN architecture Our architecture, visualized in <Ref> (a), is a multilayer perceptron (MLP) with three hidden layers containing 256 neurons each, except for the following additions and modifications:

* We use categorical embedding layers [Guo and Berkhahn, 2016] to embed the remaining categorical features with cardinality $> 8$.

* For numerical features, excluding the one-hot encoded ones, we introduce PBLD (periodic bias linear densenet) embeddings, which concatenate the original value to the PL embeddings proposed by Gorishniy et al., 2022 and use a different periodic embedding with biases, inspired by [Rahimi and Recht, 2007]. PBLD embeddings apply separate small two-layer MLPs to each feature $x_i$ as
(x_i, ^(2,i)_emb
cos(2π^(1,i)_emb x_i + ^(1,i)_emb) + ^(2,i)_emb ) ∈^4.

For efficiency reasons, we use 4-dimensional embeddings with $\bfw^{(1,i)}_{\text{emb}}, \bfb^{(1,i)}_{\text{emb}} \in \bbR^{16}, \bfb^{(2,i)}_{\text{emb}} \in \bbR^3, \bfW^{(2,i)}_{\text{emb}} \in \bbR^{3 \times 16}$.

* To encourage (soft) feature selection, we introduce a scaling layer before the first linear layer, which is simply a matrix-vector product with a diagonal weight matrix. In other words, it computes $x_{i, \mathrm{out}} = s_i \cdot x_{i, \mathrm{in}}$, with a learnable scaling factor $s_i$ for each feature $i$. We found it beneficial to use a larger learning rate for this layer.

* Our linear layers use the neural tangent parametrization (NTP) as proposed by Jacot et al., 2018, i.e., they compute $\bfz^{(l+1)} = d_l^{-1/2}\bfW^{(l)} \bfx^{(l)} + \bfb^{(l)}$,
where $d_l$ is the dimension of the layer input $\bfx^{(l)}$. The motivation behind the use of the NTP here is that it effectively modifies the learning rate for the weight matrices depending on the input dimension $d_l$, hopefully preventing too large steps whenever the number of columns is large.
We did not observe improvements when using the Adam version of the maximal update parametrization [Yang et al., 2021].

* We use parametric activation functions inspired by PReLU [He et al., 2015]. In general, for an activation function $\sigma$, we define a parametric version with separate learnable $\alpha_i$ for each neuron $i$:
σ_α_i(x_i)     =     (1-α_i) x_i + α_i σ(x_i) .

When $\alpha_i = 1$, we recover $\sigma$, and when $\alpha_i = 0$, the activation function is linear. As activation functions, we use SELU [Klambauer et al., 2017] for classification and Mish [Misra, 2020] for regression.

* We use dropout after each activation function. We do not use the Alpha-dropout variant originally proposed for SELU [Klambauer et al., 2017], as we were not able to obtain good results with it.

* For regression, at test time, we clip the MLP outputs to the observed range during training. (We observed that this is mainly helpful for suboptimal hyperparameters.)

Initialization We initialize the parameters $s_i$ of the scaling layer to $1$, making it an identity function at initialization. We initialize the parameters $\alpha_i$ of the parametric activation functions to $1$, recovering the standard activation functions at initialization. We initialize weights and biases in a data-dependent fashion during a forward pass on the (possibly subsampled) training set. We rescale rows of standard-normal-initialized weight matrices to scale the variance of the output pre-activations over the dataset to one. For the biases, we use the data-dependent  initialization method [Steinwart, 2019].

Training We use the Adam optimizer [Kingma and Ba, 2015] in the AdamW variant [Loshchilov and Hutter, 2018] for weight decay.
We set its momentum hyperparameters to $\beta_1 = 0.9$ and $\beta_2 = 0.95$ instead of the default $\beta_2 = 0.999$. The idea to use a smaller value for $\beta_2$ is adopted from the fastai tabular MLP [Howard and Gugger, 2020]. We optimize for 256 epochs with a batch size of 256. As a loss function for classification, we use softmax + cross-entropy with label smoothing [Szegedy et al., 2016] with parameter $\eps = 0.1$. To make the binary and multi-class cases more similar, we also use this loss function for binary classification instead of using a single output neuron with sigmoid and log-loss. For regression, we use the MSE loss and affinely transform the targets to have zero mean and unit variance on the training and validation set.

We allow parameter-specific scheduled hyperparameters computed in each iteration using a base value, optional parameter-specific factors, and a schedule, as
base_value · param_factor · schedule(iteration/#iterations),

allowing us, for example, to use a high learning rate factor for scaling layer parameters.
Because we do not tune the number of epochs separately on each dataset, we use a multi-cycle learning rate schedule, providing multiple valleys that are usually preferable for stopping the training, while allowing high learning rates in between. Our schedule is similar to Loshchilov and Hutter, 2017 and Smith, 2017, but with a simpler analytical expression:
coslog_k(t)         1/2(1 - cos(2πlog_2(1 + (2^k - 1)t))) .

We set $k=4$ to obtain four cycles as shown in <Ref> (b). To allow stopping at different levels of regularization, we schedule dropout and weight decay using the following schedule, cf. <Ref> (b):[inspired by a similar schedule in <https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer>]
flat_cos(t)         1/2 (1+cos(π(max{1,2t}-1))).

The detailed hyperparameters can be found in <Ref>.

Best-epoch selection Due to the multi-cycle learning rate schedule, we do not perform classical early stopping. Instead, we always train for the full 256 epochs and then revert the model to the epoch with the lowest validation error, which in this paper is based on classification error, or RMSE for regression. In case of a tie, we found it beneficial to use the last of the tied best epochs.

RealMLP-TD-S Since certain aspects of RealMLP-TD are somewhat complex to implement, we introduce a simplified (and faster) variant called RealMLP-TD-S.
Among the simplifications (see <Ref>) are: omitting embedding layers, using non-parametric activations, using a simpler initialization method, and omitting dropout and weight decay.

§ GRADIENT-BOOSTED DECISION TREES

To find better default hyperparameters for GBDTs, we employ a semi-automatic approach: We use hyperparameter optimization libraries like hyperopt [Bergstra et al., 2013] and SMAC3 [Lindauer et al., 2022] to explore a reasonably large hyperparameter space, evaluating the benchmark score of each configuration on the meta-train benchmarks, and then perform some small manual adjustments like rounding the best obtained hyperparameters. To balance efficiency and accuracy, we fix the number of estimators to 1000 and use the  method for XGBoost. We only consider the libraries' default tree-building strategies since it is one of their main differences.
The tuned defaults (TD) for LightGBM (LGBM), XGBoost (XGB), and CatBoost can be found in <Ref>, <ref>, and <ref>, respectively.

While some of the obtained hyperparameter values might be sensitive to the tuning and benchmark setup, we observe some general trends. First, row subsampling is applied in all settings, while column subsampling is rarely applied. Second, trees are generally allowed to be deeper for regression than for classification. Third, the Bernoulli bootstrap in CatBoost is competitive with the Bayesian bootstrap while also being faster.

§ EXPERIMENTS

In the following, we evaluate different methods with library defaults (D), tuned defaults (TD), and hyperparameter optimization (HPO). Recall that TD uses fixed parameters optimized on the meta-train benchmarks, while HPO tunes hyperparameters on each dataset split independently. All methods except random forests select the best iteration/epoch on the validation set of the respective dataset split. All NN-based regression methods standardize the labels for training.

§.§ Methods

We use the following abbreviations, see <Ref> for more details:

* MLP-D, ResNet-D: MLP and ResNet from Gorishniy et al., 2021 with default parameters from McElfresh et al., 2023.

* TabR-S-D: TabR-S with default parameters [Gorishniy et al., 2023], using context freeze for one slow dataset.

* RF-SKL-D: Random Forest [Breiman, 2001] implementation of scikit-learn [Pedregosa et al., 2011] with default parameters.

* CatBoost-D, LGBM-D, and XGB-D: CatBoost, LightGBM, and XGBoost defaults.

* XGB-PBB-D: XGBoost with optimized defaults (for AUC) from Probst et al., 2019, using hist boosting.

* RealMLP-TD and RealMLP-TD-S: Our MLPs with tuned defaults from <Ref>.

* CatBoost-TD, LGBM-TD, and XGB-TD: Tuned defaults for CatBoost, LightGBM, and XGBoost, respectively, as presented in <Ref>.

* RealMLP-HPO, MLP-HPO, CatBoost-HPO, LGBM-HPO, and XGB-HPO: We apply 50 random search steps to each of the models, see <Ref>.

* Best-HPO, Best-TD, Best-D: Selecting the best model out of RealMLP, CatBoost, XGBoost, and LightGBM in the respective HPO, TD, or D variant based on the validation error on each dataset. For Best-D, we use MLP-D instead of RealMLP.

* Ensemble-HPO, Ensemble-TD, Ensemble-D: Similar to Best, but creating a weighted ensemble using the validation set as proposed by Caruana et al., 2004, with 40 greedy selection steps as in Salinas and Erickson, 2023.

§.§ Results

<Ref> shows the results of the aforementioned methods on the meta-train and meta-test benchmarks, along with their runtimes on a CPU. We also show results for the benchmark of [Grinsztajn et al., 2022] in <Ref>, specifically the version with medium-size datasets ($\leq 10$K training samples). The latter include scikit-learn's  (GBT-SKL) as well as two transformer-based models, FT-Transformer [Gorishniy et al., 2021] and SAINT [Somepalli et al., 2022]. For the baselines, we use (slightly adapted) HPO search spaces from the original paper, see <Ref>.

Benchmark scores on $\Ctrc$, $\Ctrr$, $\Ctec$, and $\Cter$ vs. average training time.
The $y$-axis shows the shifted geometric mean ($\operatorname{SGM}_\eps$) classification error (left) or RMSE (right) as explained in <Ref>.
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>.
The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

Results on the benchmarks of Grinsztajn et al., 2022.
The $y$-axis (inverted) shows the normalized accuracy / R2 score used in the original paper (see <Ref>). The $x$-axis shows average training times per 1000 samples, using GPUs for NNs as in [Grinsztajn et al., 2022], see <Ref>.

How good are tuned defaults on new datasets?
To answer this question, we compare the relative gaps between TD and HPO benchmark scores on the meta-test benchmarks to those on the meta-train benchmarks. The gap between RealMLP-HPO and RealMLP-TD is about equally large on the meta-train and meta-test benchmarks, indicating that the tuned defaults transfer very well to the meta-test benchmark. For GBDTs, tuned defaults are competitive with HPO on the meta-train set, but not as good on the meta-test set. Still, they are considerably better than the untuned defaults on the meta-test set. Note that we did not limit the TD parameters to the literature search spaces for the HPO models (cf. <Ref>); for example, XGB-TD uses a smaller value of min_child_weight for classification and CatBoost-TD uses deeper trees and Bernoulli boosting. The XGBoost defaults XGB-PBB-D from Probst et al., 2019 outperform XGB-TD on $\Ctec$, perhaps because their benchmark is more similar to $\Ctec$ or because XGB-PBB-D uses more estimators (4168) and deeper trees.

Among GBDTs, CatBoost defaults are better and slower. Several papers have found CatBoost to perform favorably among GBDTs while being more computationally expensive to train [Prokhorenkova et al., 2018, McElfresh et al., 2023, Chen et al., 2023, Kim et al., 2024, Ye et al., 2024].
We observe the same for our tuned defaults on the meta-test benchmark.

RealMLP performs favorably among NNs. We can see on the meta-test benchmark as well as the benchmark by [Grinsztajn et al., 2022] that RealMLP-TD performs significantly better than MLP-D and ResNet-D. RealMLP-TD is slower on average, mainly due to the use of numerical embeddings and the lack of early stopping. However, the latter allows efficient cross-validation and ensemble training through vectorization.
On the benchmark by [Grinsztajn et al., 2022], RealMLP-TD achieves equal or better results than other NNs, and in the HPO setting, it outperforms transformer-based models in terms of speed and benchmark scores.
In addition, RealMLP-TD is faster than TabR-S-D while also achieving better meta-test benchmark scores. When looking at rank-based metrics in <Ref>, RealMLP-TD is worse than TabR-S-D on $\Ctrr$ but better on $\Ctrc$ and $\Ctec$.

NN improvements <Ref> (c) shows how adding the proposed RealMLP components to a simple MLP improves the meta-train benchmark performance. However, these results depend on the order in which components are added, which is addressed by a separate ablation study in <Ref>. We also show in <Ref> that our architectural improvements alone are beneficial when applied to MLP-D directly, although non-architectural aspects are at least as important. In particular, our numerical preprocessing is easy to adopt and often beneficial for other NNs as well.

RealMLP is competitive with tree-based models. On the meta-test benchmark, in the TD and HPO setting, our MLP performs better than GBDTs in terms of shifted geometric mean error. When considering arithmetic mean error and average rank, RealMLP is comparable to or slightly better than GBDTs (<Ref>). The improvement in shifted geometric mean error reflects the fact that on several datasets, the NN has a lower error by roughly an order of magnitude. The win rate of RealMLP-TD vs. CatBoost-TD is around 51–62% and the win rate of RealMLP-HPO vs. CatBoost-HPO is around 55–60% (<Ref>). On the benchmark of [Grinsztajn et al., 2022], RealMLP performs worse for classification but comparably for regression.

Simply trying all default algorithms is very often better than (naive) single-algorithm HPO, and always faster.
When comparing Best-TD to 50-step HPO on any individual algorithm, we notice that Best-TD is always faster, while also being better on the meta-train and meta-test benchmarks and only slightly worse than the best HPO models on the benchmark of [Grinsztajn et al., 2022]. We also note that ensemble selection [Caruana et al., 2004] usually gives 1–3% improvement on the benchmark score compared to selecting the best model, and can potentially be further improved [Caruana et al., 2006]. Our results indicate that our methods have the potential to be useful within AutoML methods, which may implement more sophisticated methods such as training-time-aware HPO or stacking.
We note that this surprising performance directly results from the quality of our tuned default hyperparameters and RealMLP, as selecting or ensembling non-tuned defaults is matched by LGBM-HPO and outperformed by RealMLP-HPO (<Ref>).

Further insights In <Ref>, we present further experimental results, including an ablation study. We compare bagging and refitting for RealMLP-TD and LGBM-TD, finding that refitting multiple models is often better on average. We demonstrate that GBDTs benefit from high early stopping patiences for classification, especially when using accuracy as the stopping metric. When considering ROC AUC as a stopping metric, we show that stopping on cross-entropy is preferable to accuracy, and that label smoothing is harmful. We also investigate robust scaling and smooth clipping for other NNs, showing that it is often helpful.

Limitations In our benchmarks, we have considered medium-to-large tabular datasets with random train-test splits, using classification error and RMSE as metrics, with additional results for AUROC in <Ref>.
It is unclear to which extent the obtained defaults can generalize to very small datasets, distribution shifts, datasets with missing numerical values, and other metrics such as log-loss. Additionally, runtimes and the resulting tradeoffs may change with different parallelization, different (time-aware) HPO algorithms, or different hardware.
For computational reasons, we only use a single training-validation split per train-test split. This means that HPO can overfit the validation set more easily than in a cross-validation setup.
While we extensively benchmark different NN models from the literature, we do not attempt to equalize non-architectural aspects, and our work should therefore not be seen as a comparison of architectures.
We compared to TabR-S-D as a recent promising method with good default parameters [Gorishniy et al., 2023, Ye et al., 2024]. However, due to a surge of recently published deep tabular models <cit.>, it is unclear what the current best deep tabular model is.

§ CONCLUSION

In this paper, we studied the potential of improved default parameters for GBDTs and an improved MLP, evaluated on a large separate meta-test benchmark as well as the benchmark by [Grinsztajn et al., 2022], and investigated the time-accuracy tradeoffs of various algorithm selection and ensembling scenarios. Our improved MLP mostly outperforms other NNs from the literature with moderate runtime and is competitive with GBDTs. Since many of the proposed improvements to NNs are orthogonal to the improvements in other papers, they offer exciting opportunities for combinations.
While the NNs vs GBDTs debate remains interesting,
our results demonstrate that with good default parameters, it is worth trying both algorithm families even with a moderate training time budget.

We thank Gaël Varoquaux, Frank Sehnke, Katharina Strecker, Ravid Shwartz-Ziv, Lennart Purucker, and Francis Bach for helpful discussions. We thank Katharina Strecker for help with code refactoring.

Funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 – 390740016. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting David Holzmüller. LG acknowledges support in part by the French Agence Nationale de la Recherche under
Grant ANR-20-CHIA-0026 (LearnI). Part of this work was performed on the computational resource bwUniCluster funded by the Ministry of Science, Research and the Arts Baden-Württemberg and the Universities of the State of Baden-Württemberg, Germany, within the framework program bwHPC. Part of this work was performed using HPC resources from GENCI–IDRIS (Grant 2023-AD011012804R1 and 2024-AD011012804R2).

Contribution statement DH and IS conceived the project. DH implemented and experimentally validated the newly proposed methods and wrote the initial paper draft. DH and LG contributed to benchmarking, plotting, and implementing baseline methods. LG and IS helped revise the draft. IS supervised the project and contributed dataset downloading code.

[Arik and Pfister, 2021]
Sercan O. Arik and Tomas Pfister.
TabNet: Attentive interpretable tabular learning.
In AAAI Conference on Artificial Intelligence, 2021.

[Bergstra et al., 2013]
James Bergstra, Daniel Yamins, and David Cox.
Making a science of model search: Hyperparameter optimization in
hundreds of dimensions for vision architectures.
In International Conference on Machine Learning, 2013.

[Borisov et al., 2022]
Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin
Pawelczyk, and Gjergji Kasneci.
Deep neural networks and tabular data: A survey.
IEEE Transactions on Neural Networks and Learning Systems,

[Brazdil et al., 2008]
Pavel Brazdil, Christophe Giraud Carrier, Carlos Soares, and Ricardo Vilalta.
Metalearning: Applications to Data Mining.
Springer Science & Business Media, 2008.

[Breiman, 2001]
Leo Breiman.
Random forests.
Machine learning, 45:0 5–32, 2001.

[Caruana et al., 2004]
Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes.
Ensemble selection from libraries of models.
In International Conference on Machine Learning, 2004.

[Caruana et al., 2006]
Rich Caruana, Art Munson, and Alexandru Niculescu-Mizil.
Getting the most out of ensemble selection.
In International Conference on Data Mining, pages
828–833. IEEE, 2006.

[Chen et al., 2023]
Jintai Chen, Jiahuan Yan, Danny Ziyi Chen, and Jian Wu.
ExcelFormer: A neural network surpassing GBDTs on tabular data.
arXiv:2301.02819, 2023a.

[Chen et al., 2023]
Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, and Tien-Hao
Trompt: Towards a better deep neural network for tabular data.
In International Conference on Machine Learning,

[Chen and Guestrin, 2016]
Tianqi Chen and Carlos Guestrin.
XGBoost: A scalable tree boosting system.
In International Conference on Knowledge Discovery and
Data Mining, 2016.

[Erickson et al., 2020]
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy,
Mu Li, and Alexander Smola.
AutoGluon-Tabular: Robust and accurate AutoML for structured
In 7th ICML Workshop on Automated Machine Learning,

[Feurer et al., 2022]
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and
Frank Hutter.
Auto-sklearn 2.0: Hands-free automl via meta-learning.
The Journal of Machine Learning Research, 230 (261),

[Fischer et al., 2023]
Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl.
OpenML-CTR23–A curated tabular regression benchmarking suite.
In AutoML Conference 2023 (Workshop), 2023.

[Gijsbers et al., 2022]
Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien
Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren.
AMLB: An AutoML benchmark.
arXiv:2207.12560, 2022.

[Gneiting and Raftery, 2007]
Tilmann Gneiting and Adrian E Raftery.
Strictly proper scoring rules, prediction, and estimation.
Journal of the American Statistical Association, 1020
(477):0 359–378, 2007.

[Gorishniy et al., 2021]
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.
Revisiting deep learning models for tabular data.
Neural Information Processing Systems, 2021.

[Gorishniy et al., 2022]
Yury Gorishniy, Ivan Rubachev, and Artem Babenko.
On embeddings for numerical features in tabular deep learning.
Neural Information Processing Systems, 2022.

[Gorishniy et al., 2023]
Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim
Kotelnikov, and Artem Babenko.
TabR: Tabular deep learning meets nearest neighbors in 2023.
arXiv:2307.14338, 2023.

[Grinsztajn et al., 2022]
Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux.
Why do tree-based models still outperform deep learning on typical
tabular data?
Neural Information Processing Systems, 2022.

[Guo and Berkhahn, 2016]
Cheng Guo and Felix Berkhahn.
Entity embeddings of categorical variables.
arXiv:1604.06737, 2016.

[Hafner et al., 2023]
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.
Mastering diverse domains through world models.
arXiv:2301.04104, 2023.

[He et al., 2015]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level performance on
imagenet classification.
In IEEE International Conference on Computer Vision,
pages 1026–1034, 2015.

[Herbold, 2020]
Steffen Herbold.
Autorank: A Python package for automated ranking of classifiers.
Journal of Open Source Software, 50 (48):0
2173, 2020.
URL <https://doi.org/10.21105/joss.02173>.
Publisher: The Open Journal.

[Hollmann et al., 2022]
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter.
TabPFN: A transformer that solves small tabular classification
problems in a second.
In International Conference on Learning Representations,

[Holzmüller et al., 2023]
David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart.
A framework and benchmark for deep batch active learning for
Journal of Machine Learning Research, 240 (164), 2023.

[Howard and Gugger, 2020]
Jeremy Howard and Sylvain Gugger.
Fastai: A layered API for deep learning.
Information, 110 (2):0 108, 2020.

[Huang et al., 2020]
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin.
TabTransformer: Tabular data modeling using contextual
arXiv:2012.06678, 2020.

[Jacot et al., 2018]
Arthur Jacot, Franck Gabriel, and Clément Hongler.
Neural tangent kernel: Convergence and generalization in neural
Neural Information Processing Systems, 2018.

[Joseph and Raj, 2024]
Manu Joseph and Harsh Raj.
GANDALF: Gated Adaptive Network for Deep Automated
Learning of Features.
arXiv:2207.08548, 2024.

[Kadra et al., 2021]
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka.
Well-tuned simple nets excel on tabular datasets.
In Neural Information Processing Systems, 2021.

[Ke et al., 2017]
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei
Ye, and Tie-Yan Liu.
LightGBM: A highly efficient gradient boosting decision tree.
In Neural Information Processing Systems, 2017.

[32]
Markelle Kelly, Rachel Longjohn, and Kolby Nottingham.
The UCI Machine Learning Repository.
URL <https://archive.ics.uci.edu>.

[Kim et al., 2024]
Myung Jun Kim, Léo Grinsztajn, and Gaël Varoquaux.
CARTE: pretraining and transfer for tabular learning.
arXiv:2402.16785, 2024.

[Kingma and Ba, 2015]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In International Conference on Learning Representations,

[Klambauer et al., 2017]
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.
Self-normalizing neural networks.
In Neural Information Processing Systems, 2017.

[Kohli et al., 2024]
Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank
Towards Quantifying the Effect of Datasets for Benchmarking:
A Look at Tabular Machine Learning.
In ICLR 2024 Data-centric Machine Learning Research
Workshop, 2024.

[Kossen et al., 2021]
Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Thomas Rainforth, and
Yarin Gal.
Self-attention between datapoints: Going beyond individual
input-output pairs in deep learning.
In Neural Information Processing Systems, 2021.

[Lindauer et al., 2022]
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp,
Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter.
SMAC3: A versatile Bayesian optimization package for
hyperparameter optimization.
Journal of Machine Learning Research, 230 (54), 2022.

[Loshchilov and Hutter, 2017]
Ilya Loshchilov and Frank Hutter.
SGDR: Stochastic gradient descent with warm restarts.
In International Conference on Learning Representations,

[Loshchilov and Hutter, 2018]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In International Conference on Learning Representations,

[Marton et al., 2024]
Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt.
GRANDE: Gradient-based decision tree ensembles for tabular data.
In International Conference on Learning Representations,

[McCarter, 2023]
Calvin McCarter.
The kernel density integral transformation.
Transactions on Machine Learning Research, 2023.

[McElfresh et al., 2023]
Duncan McElfresh, Sujay Khandagale, Jonathan Valverde, Ganesh Ramakrishnan,
Micah Goldblum, and Colin White.
When do neural nets outperform boosted trees on tabular data?
arXiv:2305.02997, 2023.

[Mishkin and Matas, 2016]
Dmytro Mishkin and Jiri Matas.
All you need is a good init.
In International Conference on Learning Representations,

[Misra, 2020]
Diganta Misra.
Mish: A self regularized non-monotonic activation function.
In British Machine Vision Conference, 2020.

[Moritz et al., 2018]
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw,
Eric Liang, Melih Elibol, Zongheng Yang, William Paul, and Michael I. Jordan.
Ray: A distributed framework for emerging AI applications.
In USENIX Symposium on Operating Systems Design and
Implementation, 2018.

[Paszke et al., 2019]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory
Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca Antiga.
PyTorch: An imperative style, high-performance deep learning
Neural Information Processing Systems, 32, 2019.

[Pedregosa et al., 2011]
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand
Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, and
Vincent Dubourg.
Scikit-learn: Machine learning in Python.
Journal of Machine Learning Research, 120 (85), 2011.

[Pfisterer et al., 2021]
Florian Pfisterer, Jan N. van Rijn, Philipp Probst, Andreas Müller, and Bernd
Learning multiple defaults for machine learning algorithms.
arXiv:1811.09409, 2021.

[Probst et al., 2019]
Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl.
Tunability: Importance of hyperparameters of machine learning
Journal of Machine Learning Research, 200 (53), 2019.

[Prokhorenkova et al., 2018]
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush,
and Andrey Gulin.
CatBoost: Unbiased boosting with categorical features.
In Neural Information Processing Systems, 2018.

[Rahimi and Recht, 2007]
Ali Rahimi and Benjamin Recht.
Random features for large-scale kernel machines.
In Neural Information Processing Systems, 2007.

[Ramsauer et al., 2020]
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael
Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, and
Michael K. Kopp.
Hopfield networks is all you need.
In International Conference on Learning Representations,

[Salinas and Erickson, 2023]
David Salinas and Nick Erickson.
TabRepo: A Large Scale Repository of Tabular Model
Evaluations and its AutoML Applications.
arXiv:2311.02971, 2023.

[Schäfl et al., 2023]
Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, and Sepp Hochreiter.
Modern Hopfield networks as memory for iterative learning on
tabular data.
In NeurIPS Workshop on Associative Memory & Hopfield
Networks in 2023, 2023.

[Shen et al., 2023]
Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham
Neubig, and Ameet Talwalkar.
Cross-Modal Fine-Tuning: Align then Refine.
arXiv:2302.05738, 2023.

[Shwartz-Ziv and Armon, 2022]
Ravid Shwartz-Ziv and Amitai Armon.
Tabular data: Deep learning is not all you need.
Information Fusion, 81:0 84–90, 2022.

[Smith, 2017]
Leslie N. Smith.
Cyclical learning rates for training neural networks.
In Winter Conference on Applications of Computer
Vision, 2017.

[Somepalli et al., 2022]
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom
SAINT: Improved neural networks for tabular data via row
attention and contrastive pre-training.
In NeurIPS 2022 Table Representation Learning
Workshop, 2022.

[Steinwart, 2019]
Ingo Steinwart.
A sober look at neural network initializations.
arXiv:1903.11482, 2019.

[Szegedy et al., 2016]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Rethinking the inception architecture for computer vision.
In Computer Vision and Pattern Recognition, 2016.

[van Rijn et al., 2018]
Jan N. van Rijn, Florian Pfisterer, Janek Thomas, Andreas Muller, Bernd Bischl,
and Joaquin Vanschoren.
Meta learning for defaults: Symbolic defaults.
In NeurIPS 2018 Workshop on Meta-Learning, 2018.

[Vanschoren, 2018]
Joaquin Vanschoren.
Meta-learning: A survey.
arXiv:1810.03548, 2018.

[Vanschoren et al., 2014]
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo.
OpenML: Networked science in machine learning.
ACM SIGKDD Explorations Newsletter, 150 (2):0
49–60, 2014.
Publisher: ACM New York, NY, USA.

[Wistuba et al., 2015]
Martin Wistuba, Nicolas Schilling, and Lars Schmidt-Thieme.
Learning hyperparameter optimization initializations.
In International Conference on Data Science and
Advanced Analytics, pages 1–10, 2015.

[Xu et al., 2024]
Chenwei Xu, Yu-Chao Huang, Jerry Yao-Chieh Hu, Weijian Li, Ammar Gilani,
Hsi-Sheng Goan, and Han Liu.
BiSHop: Bi-directional cellular learning for tabular data with
generalized sparse modern Hopfield model.
arXiv:2404.03830, 2024.

[Yang et al., 2021]
Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi,
Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao.
Tuning large neural networks via zero-shot hyperparameter transfer.
In Neural Information Processing Systems, 2021.

[Ye et al., 2024]
Han-Jia Ye, Si-Yang Liu, Hao-Run Cai, Qi-Le Zhou, and De-Chuan Zhan.
A closer look at deep learning on tabular data.
arXiv:2407.00956, 2024.

§ FURTHER DETAILS ON REALMLP-TD AND REALMLP-TD-S

The detailed hyperparameter settings for RealMLP-TD and RealMLP-TD-S are listed in <Ref>.

Overview of hyperparameters for RealMLP-TD and RealMLP-TD-S.

2cRealMLP-TD     2cRealMLP-TD-S

Hyperparameter     classification     regression     classification     regression

Num. embedding type     PBLD     PBLD     None     None

Num. embedding periodic init std.     0.1     0.1     —     —

Num. embedding hidden dimension     16     16     —     —

Num. embedding dimension     4     4     —     —

Max one-hot size (without missing)     8     8     $\infty$     $\infty$

Num. preprocessing     4crobust scale + smooth clip

Categorical embedding dimension     8     8     —     —

Categorical embedding initialization     $\calN(0, 1)$     $\calN(0, 1)$     —     —

Use scaling layer     4cyes

Scaling layer initialization     4c1.0 (constant)

Number of linear layers     4c4

Hidden layer sizes     4c[256, 256, 256]

Activation function     SELU     Mish     SELU     Mish

Use parametric activation function     yes     yes     no     no

Parametric activation function initialization     1.0     1.0     —     —

Linear layer parametrization     4cNTP

Last linear layer weight initialization     data-driven     data-driven     zero     zero

Other linear layer weight initialization     data-driven     data-driven     std normal     std normal

Last linear layer bias initialization     he+5     he+5     zero     zero

Other linear layer bias initialization     he+5     he+5     std normal     std normal

Optimizer     4cAdamW

Batch size     4c256

Number of epochs     4c256

Adam $\beta_1$     4c0.9

Adam $\beta_2$     4c0.95

Adam $\eps$     4c1e-8

Learning rate (base value)     0.04     0.2     0.04     0.07

Learning rate schedule     4c$\coslog_4$

Learning rate (num. emb. factor)     0.1     0.1     —     —

Learning rate (scaling layer factor)     4c6

Learning rate (bias factor)     4c0.1

Learning rate (param. act. factor)     0.1     0.1     —     —

Dropout probability (base value)     0.15     0.15     0.0     0.0

Dropout schedule     $\flatcos$     $\flatcos$     —     —

Weight decay (base value)     0.02     0.02     0.0     0.0

Weight decay schedule     $\flatcos$     $\flatcos$     —     —

Weight decay (bias factor)     0.0     0.0     —     —

Loss function     cross-entropy     MSE     cross-entropy     MSE

Label smoothing $\eps$     0.1     —     0.1     —

Standardize targets during training     —     yes     —     yes

Output min-max clipping     —     yes     —     no

Best epoch selection metric     class. error     MSE     class. error     MSE

Best epoch selection method     4clast best validation error

§.§ RealMLP-TD Details

Initialization We initialize categorical embedding parameters from $\calN(0, 1)$. We initialize the components of $\bfw^{(1,i)}_{\text{emb}}$ from $\calN(0, 0.1^2)$ and of $\bfb^{(1,i)}_{\text{emb}}$ from $\calU[-\pi, \pi]$. The other numerical embedding parameters are initialized according to PyTorch's default initialization, that is, from the uniform distribution $\calU[-1/\sqrt{16}, 1/\sqrt{16}]$. For weights and biases of the linear layers, we use a data-dependent initialization. The initialization is performed on the fly during a first forward pass of the network on the training set (which can be subsampled adaptively not to use more than 1 GB of RAM). We realize this by providing  methods similar to a pipeline in scikit-learn. For the weight matrices, we use a custom two-step procedure: First, we initialize all entries from $\calN(0, 1)$. Then, we rescale each row of the weight matrix such that the outputs $\frac{1}{\sqrt{d_l}}\bfW^{(l)} \bfx_j^{(l)}$ have variance $1$ over the dataset (i.e. when considering the sample index $j \in \{1, \hdots, n\}$ as a uniformly distributed random variable). This is somewhat similar to the LSUV initialization method [Mishkin and Matas, 2016]. For the biases, we use the data-dependent  initialization method [Steinwart, 2019].

Training We implement weight decay as in PyTorch using $\theta \assign \theta - \text{lr} \cdot \text{wd} \cdot \theta$, which includes the learning rate unlike the original version [Loshchilov and Hutter, 2018].

§.§ RealMLP-TD-S Details

For RealMLP-TD-S, we make the following changes compared to RealMLP-TD:

* We apply one-hot encoding to all categorical variables and do not apply categorical embeddings.

* We do not apply numerical embeddings.

* We use the standard non-parametric versions of the SELU and Mish activation functions.

* We do not use dropout and weight decay.

* We use simpler weight and bias initializations: We initialize weights and biases from $\calN(0, 1)$, except in the last layer, where we initialize them to zero.

* We do not clip the outputs, even in the regression case.

* We apply a different base learning rate in the regression case.

§.§ Details on Cumulative Ablation

Here, we provide more details on the vanilla MLP and the ablation steps from <Ref> (c). For each step, we choose the best default learning rate out of a learning rate grid, using $\{0.0004, 0.0007, 0.001, 0.0015, 0.0025, 0.004, 0.007, 0.01, 0.015\}$ for NNs using standard parametrization and $\{0.01, 0.02, 0.03, 0.04, 0.07, 0.1, 0.2, 0.3, 0.4\}$ for NNs using neural tangent parametrization.

* Vanilla MLP: We use three hidden layers with 256 hidden neurons in each layer, just like RealMLP-TD, and the ReLU activation function. Each linear layer uses standard parametrization and the PyTorch default initialization, which is uniform from $[-1/\sqrt{\text{fan\_in}}, 1/\sqrt{\text{fan\_in}}]$ for both weights and biases, where fan_in is the input dimension. Categorical features are embedded using embedding layers, using eight-dimensional embeddings for each feature. Numerical features are transformed using a scikit-learn  to approximately normal-distributed features. Optimization is performed using Adam with constant learning rate and default parameters $\beta_1 = 0.9, \beta_2 = 0.999, \eps = 10^{-8}$ for at most 256 epochs with batch size 256, with constant learning rate. If the best validation error (classification error or RMSE) does not improve for 40 epochs, training is stopped. In each case, the model is reverted to the parameters of the epoch with the best validation score, using the first best epoch in case of a tie.

* Robust scale + smooth clip: We replace the  with robust scaling and smooth clipping.

* One-hot for small cat.: As in RealMLP-TD, we use one-hot encoding for categories with at most eight values, not counting missing values.

* No early stopping: We always train the full 256 epochs.

* Last best epoch: In case of a tie, we use the last of the best epochs.

* $\coslog_4$ lr sched: We use the $\coslog_4$ learning rate schedule instead of a constant one.

* Adam $\beta_2 = 0.95$: We set $\beta_2 = 0.95$.

* Label smoothing (class.): We enable label smoothing with $\eps = 0.1$ in the classification case.

* Output clipping (reg.): For regression, outputs are clipped to the min-max range observed during training.

* NT parametrization: We use the neural tangent parametrization for linear layers, setting the bias learning rate factor to $0.1$.

* Act. fn. SELU / Mish: We change the activation function from ReLU to SELU (classification) or Mish (regression).

* Parametric act. fn.: We use parametric versions of the activation functions, with a learning rate factor of $0.1$ for the parameters.

* Scaling layer: We use a scaling layer with a learning rate factor of $6$ before the first linear layer.

* Num. embeddings: PL: We apply the PL embeddings [Gorishniy et al., 2022] to numerical features.

* Num. embeddings: PBLD: We apply our PBLD embeddings instead.

* Dropout $p=0.15$: We apply dropout with probability $0.15$.

* Dropout sched: $\flatcos$: We apply the $\flatcos$ schedule to the dropout probability.

* Weight decay wd $=$ $0.02$: We apply weight decay (as in AdamW, PyTorch version) with value $0.02$.

* wd sched: $\flatcos$: We apply the $\flatcos$ schedule to weight decay.

* Bias init: he+5: We apply the he+5 bias initialization method from [Steinwart, 2019].

* Weight init: data-driven: We apply our data-driven weight initialization method.

§.§ Discussion

Here, we discuss some of the design decisions behind RealMLP-TD and possible trade-offs. First, our implementation allows us to train RealMLP-TD in a vectorized fashion on multiple train-validation-test splits at the same time. On the one hand, this can lead to speedups on GPUs when training multiple models in parallel, including on the benchmarks. On the other hand, it can hinder the implementation of certain methods like patience-based early stopping or loss-based learning rate schedules. While our ablations in <Ref> show the advantage of our multi-cycle schedule over decreasing learning rate schedules, the latter ones could potentially enable a faster average training time through low-patience early stopping. An interesting follow-up question could be whether the multi-cycle schedule still works well with larger-patience early stopping.

Regarding categorical embeddings, our meta-train benchmark does not contain many high-cardinality categorical variables, and we were not able to conclude whether categorical embeddings are helpful or harmful compared to one-hot encoding (see <Ref>). Our motivation to include categorical embeddings stems from [Guo and Berkhahn, 2016] as well as their potential to be more efficient for high-cardinality categorical variables. However, in practice, we find pure one-hot encoding to be faster on most datasets. Regarding the embedding size, we found that 4 already gave good results for numerical embeddings and decided to use 8 for categorical variables.

Additionally, other speed-accuracy tradeoffs are possible. Especially for regression, we observed that more epochs and larger hidden layers can be helpful. When faster networks are desired, the omission of numerical and categorical embedding layers as well as parametric activations from RealMLP-TD can be helpful, while the other omissions in RealMLP-TD-S do not considerably affect the training time. Of course, using larger batch sizes can also be helpful for larger datasets.

One caveat for classification is that cross-entropy with label smoothing is not a proper scoring rule, that is, in the infinite-sample limit, it is not minimized by the true probabilities $P(y|x)$ [Gneiting and Raftery, 2007]. Hence, label smoothing might not be suitable when other classification error metrics are used, as demonstrated in <Ref> for AUROC.

§ MORE EXPERIMENTS

§.§ MLP Ablations

To assess the importance of different improvements in RealMLP-TD, we perform an ablation study. We perform the ablation study only on the meta-train benchmarks, first because they are considerably faster to run, and second because we tune the default parameters only on the meta-train benchmarks. Since the hyperparameters of RealMLP-TD have been tuned on the meta-train benchmarks, the ablation scores are not unbiased but represent some of the considerations that have been made when tuning the defaults. For each ablation, we multiply the default learning rate by learning rate factors from the grid $\{0.1, 0.15, 0.25, 0.35, 0.5, 0.7, 1.0, 1.4, 2.0, 3.0, 4.0\}$ and pick the best one. <Ref> shows the results of the ablation study in terms of the relative increase of the benchmark score for each ablation.

In general, we observe that ablations often lead to much larger changes for regression than for classification. Perhaps this is because RMSE is more sensitive compared to classification error. Another factor could be that the classification benchmark contains more datasets than the regression benchmark. For the specific ablations, we observe a few things:

* For the numerical embeddings, we see that PBLD outperforms PL, PLR, and no numerical embeddings. Contrary to Gorishniy et al., 2022, PL embeddings perform better than PLR embeddings in our setting. While the configurations with PLR and no numerical embeddings appear extremely bad for regression, we observed that they can perform more benignly with lower weight decay values.

* Using the Adam default value of $\beta_2 = 0.999$ instead of our default $\beta_2 = 0.95$ leads to considerably worse performance, especially for regression. As for numerical embeddings, we observed that the difference is less pronounced at lower weight decay values.

* Using a cosine decay learning rate schedule instead of our multi-cycle schedule leads to small deteriorations. A constant learning rate schedule performs even worse, especially for regression.

* Not employing label smoothing for classification is detrimental by around 1.8%.

* The learnable scaling layer yields improvements around 1.2% on both benchmarks.

* The use of parametric activations results in a considerable 4.8% improvement for regression but is insignificant for classification. We observed that parametric activations can sometimes alleviate optimization difficulties with weight decay.

* The differences between activation functions are rather small. For classification, Mish is competitive with SELU in this ablation but we found it to be worse in some other hyperparameter settings, so we keep SELU as the default. For regression, Mish performs best.

* For dropout and weight decay, we observe that they yield comparable but not always significant benefits for classification and regression. Scheduling dropout and weight decay parameters with the $\flatcos$ schedule is helpful for regression, but not for classification in this setting.

* When comparing the standard parametrization (SP) to the neural tangent parametrization (NTP), we disable weight decay for a fair comparison. Moreover, for SP, we set the learning rate factors for weight and bias layers to $1/16 = 1/\sqrt{256}$. This is because, for the weights in NTP, the effective updates by Adam are damped by this factor in all hidden layers except the first one. Compared to NTP without weight decay, SP without weight decay performs insignificantly worse on both benchmarks. It is unclear to us why the parametrization, which has a considerable influence on how the effective learning speed of the first linear layer scales with the number of features, is apparently of little importance.

* When comparing the data-dependent initialization of RealMLP-TD to a vanilla initialization with standard normal weights and zero biases, we see that the data-dependent initialization gains around 1% on both benchmarks.

* For selecting the best epoch, we consider selecting the first best epoch instead of the last best epoch in case of a tie. This is only relevant for classification metrics like classification error, where ties are somewhat likely to occur, especially on small and easy datasets. We observe a non-significant 0.4% deterioration in the benchmark score.

* We do not observe a significant difference when using one-hot encoding for all categorical variables, since our benchmarks contain only very few datasets with large-cardinality categorical variables.

Ablation experiments for RealMLP-TD. We re-tune the learning rate (picking the one with the best $\SGM_\eps$ benchmark score) for each ablation separately. For each ablation, we specify the increase in the benchmark score ($\SGM_\eps$) relative to RealMLP-TD, with approximate 95% confidence intervals (<Ref>), and the best learning rate factor found. In the cases where values are missing, the corresponding option is already the default.

§.§ MLP Preprocessing

In <Ref>, we compare different preprocessing methods for numerical features. Since we want to compare these methods in a relatively conventional setting, we apply them to RealMLP-TD-S (without numerical embeddings) and before one-hot encoding. We compare the following methods:

* Robust scaling and smooth clipping, our method used in RealMLP-TD and RealMLP-TD-S and described in <Ref>.

* Robust scaling without smooth clipping.

* Standardization, i.e. subtracting the mean and dividing by the standard deviation. If the standard deviation of a feature is zero, we set the feature to zero.

* Standardization followed by smooth clipping.

* The quantile transformation from scikit-learn [Pedregosa et al., 2011] with normal output distribution, which is popular in recent works [Grinsztajn et al., 2022, Gorishniy et al., 2022, Gorishniy et al., 2023, McElfresh et al., 2023].

* A variant of the quantile transform, which we call the RTDL version, used by [Gorishniy et al., 2021] and [Gorishniy et al., 2023]. This version uses a dataset size dependent number of quantiles and adds some noise before fitting the transformation.

* The recent kernel density integral transform [McCarter, 2023], which interpolates between the quantile transformation and min-max scaling, with default parameter $\alpha=1$.

<Ref> shows that on the meta-train benchmark, robust scaling and smooth clipping performs best for both classification and regression.

Effects of different preprocessing methods for numerical features for RealMLP-TD-S. We report the relative increase in the shifted geometric mean benchmark scores compared to the standard method used in RealMLP-TD and RealMLP-TD-S, which is robust scaling and smooth clipping. We also report approximate 95% confidence intervals. To have a more common setting, we do not apply the preprocessing methods to one-hot encoded categorical features. In each column, the best score is highlighted in bold, and errors whose confidence interval contains the best score are underlined.

§.§ Bagging, Refitting, and Ensembling

In our benchmark, for each training-test split, we only train one model on one training-validation split for efficiency reasons. However, ensembling and cross-validation techniques usually allow additional improvements to models. Here, we study multiple variants for RealMLP-TD and LGBM-TD. Let $\calD$ be the available data for training and validation, split into five equal-size subsets $\calD_1, \hdots, \calD_5$. (When $|\calD|$ is not divisible by five, $\calD_1 \cup \hdots \cup \calD_5 \subsetneq \calD$ since we need equal-size validation sets for vectorized NNs.) Let $f_{\calD, t}(X)$ be the predictions on inputs $X$ of the model trained on training set $\calD$ after $t \in \{1, \hdots, T\}$ epochs (for NNs) or iterations (for LGBM). For classification, we consider the class probabilities as predictions. Let $L_{\calD'}(f_{\calD, t})$ be the loss of $f_{\calD, t}$ on dataset $\calD'$. Then, we compare the test errors of an ensemble of $M=1$ or $M=5$ models, trained using bagging or refitting, with individual or joint stopping (best-epoch selection), which is formally given as follows:

1/M ∑_i=1^M f__i, t_i^*(),    (M models)

∖_i     (bagging)

_t ∈{1, , T} L__i(f_∖_i, t)     (indiv. stopping)

_t ∈{1, , T} ∑_j=1^5 L__j(f_∖_j, t)     (joint stopping).

Here, each model is trained with a different random seed. For LGBM, since we use an early stopping patience of 300 for each of the individual models, the $\argmin$ in the definition of $t_i^*$ can only go up to the minimum stopping iteration $T$ across the considered models.

Improvements for LGBM-TD by bagging or (ensembled) refitting. We perform 5-fold cross-validation, stratified for classification, and 5-fold refitting. We compare compare bagging vs. refitting, one model vs. five models, and individual stopping vs. joint stopping. The table shows the relative reduction in shifted geometric mean benchmark scores, including approximated 95% confidence intervals (<Ref>). In each column, the best score is highlighted in bold, and errors whose confidence interval contains the best score are underlined.

Improvements for RealMLP-TD by bagging or (ensembled) refitting. We perform 5-fold cross-validation, stratified for classification, and 5-fold refitting. We compare bagging vs. refitting, one model vs. five models, and individual stopping vs. joint stopping. The table shows the relative reduction in shifted geometric mean benchmark scores, including approximated 95% confidence intervals (<Ref>). In each column, the best score is highlighted in bold, and errors whose confidence interval contains the best score are underlined.

The results of our experiments can be found in <Ref> for LGBM-TD and in <Ref> for RealMLP-TD. As expected, five models are considerably better than one. We find that refitting is mostly better than bagging, although a disadvantage of refitted models is that no validation scores are available, and it is unclear how HPO would affect this comparison. Comparing individual stopping to joint stopping, we find that individual stopping has a slight advantage in five-model bagging, while joint stopping performs better for single-model refitting. In the other two scenarios, joint stopping appears slightly better for RealMLP-TD and slightly worse for LGBM-TD. We also observe that the benefit of using five models instead of one appears to be larger for RealMLP-TD than for LGBM-TD.

§.§ Early stopping for GBDTs

In <Ref> and <Ref>, we study the influence of different early stopping patiences and metrics on the resulting benchmark performance of XGB-TD, LGBM-TD, and CatBoost-TD. While the regression results only deteriorate slightly for low patiences of 10 or 20 iterations, classification results are much more hurt by low patiences. In the classification setting, we evaluate the use of different losses for early stopping and for best-epoch selection: classification error, Brier score, and cross-entropy loss. In each case, cross-entropy loss is used as the training loss, and classification error is used for evaluating the models on the test sets in the computation of the benchmark score. We observe that models stopped on classification error strongly deteriorate at low patiences ($\lesssim 100$), while our default patience of 300 achieves close-to-optimal results. Models stopped on cross-entropy loss deteriorate much less at low patiences, but achieve roughly 2% worse benchmark score at high patiences. Stopping on Brier loss achieves very good high-patience performance and is still only slightly more sensitive to the patience than stopping on cross-entropy loss. An interesting follow-up question would be if HPO can attenuate the differences between different settings.

Effect of stopping patiences and metrics on the performance of GBDTs on $\Ctrc$. We run the XGB-TD, LGBM-TD, and CatBoost-TD with different early stopping patiences (). We compare three different metrics used for stopping and best-epoch selection: classification error, Brier loss, and cross-entropy loss. The $y$-axis reports the relative increase in the benchmark score relative to stopping on classification error with patience $1000$ (i.e., never stopping early). The shaded areas are approximate 95% confidence intervals, cf. <Ref>.

Effect of stopping patiences on the performance of GBDTs on $\Ctrr$. We run the TD configurations of XGB, LGBM, and CatBoost with different early stopping patiences (). As in the remainder of the paper, we use RMSE for early stopping and best-epoch selection. The $y$-axis reports the relative increase in the benchmark score relative to stopping on classification error with patience $1000$ (i.e., never stopping early). The shaded areas are approximate 95% confidence intervals, cf. <Ref>.

§.§ Results for AUROC

For classification, there are many different metrics to capture model performance. In the main paper, we use classification error to evaluate models. All TD configurations were tuned for classification error, early stopping and best-epoch selection were performed for classification error, and HPO was performed for classification error. Here, we evaluate models on the area under the ROC curve, also known as AUROC, AUC ROC, or AUC. For the multi-class case, we use the one-vs-rest formulation of AUC, which is faster to evaluate than one-vs-one. Higher AUC values are better and the optimal value is $1$. Since we are interested in the shifted geometric mean error, we use $1-\mathrm{AUC}$ instead.

We compare two settings:

* A variant of the original setting where early stopping and the selection of the best epoch/iteration is based on accuracy but HPO is performed on $1-\mathrm{AUC}$. (Thanks to using random search, we do not have to re-run the HPO for this.)

* A setting where we use the cross-entropy loss for stopping and selecting the best epoch/iteration. While it would be possible to stop on AUC directly, this can be significantly slower since AUC is slower to evaluate. We do not perform HPO in this setting since it is expensive to run.

In both settings, we also evaluate RealMLP without label smoothing (no ls). <Ref> shows the results optimized for accuracy and <Ref> shows the results optimized for cross-entropy. We make a few observations:

* Stopping for cross-entropy generally performs better than stopping for classification error.

* Label smoothing harms RealMLP for AUC, perhaps because the stopping metric does not use label smoothing, or because it encourages near-constant logits in areas where the model is relatively certain.

* Tuned defaults are mostly still better than the library defaults, except for XGBoost on $\Ctec$.

* RealMLP without label smoothing is still competitive with GBDTs on the meta-test benchmark but does not perform better than GBDTs unlike what we observed for classification error.

Benchmark scores on $\Ctrc$ and $\Ctec$ vs. average training time for AUC, optimized for cross-entropy. BestModel-TD uses RealMLP-TD without label smoothing.
The $y$-axis shows the shifted geometric mean ($\operatorname{SGM}_\eps$) $1-\mathrm{AUC}$ as explained in <Ref>.
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>.
The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

Benchmark scores on $\Ctrc$ and $\Ctec$ vs. average training time for AUC. Stopping and best-epoch selection are performed on accuracy, while HPO is performed on AUC.
The $y$-axis shows the shifted geometric mean ($\operatorname{SGM}_\eps$) $1-\mathrm{AUC}$ as explained in <Ref>.
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>.
The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

§.§ Results Without Missing-Value Datasets

To assess whether the results are influenced by our choices in missing value handling and exclusion, <Ref> presents results on all meta-test datasets that originally did not contain missing values. Only six meta-test datasets originally contain missing values: Three from $\Ctec$ (kick, okcupid-stem, and porto-seguro) and three from $\Cter$ (fps_benchmark, house_prices_nominal, SAT11-HAND-runtime-regression). While RealMLP deteriorates slightly, especially due to the exclusion of fps_benchmark, qualitative takeaways remain similar.

Benchmark scores on $\Ctec$ and $\Cter$ without missing value datasets vs. average training time.
The $y$-axis shows the shifted geometric mean ($\operatorname{SGM}_\eps$) classification error (left) or RMSE (right) as explained in <Ref>.
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>.
The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

§.§ Results for Varying Architecture and Preprocessing

<Ref> shows the effects of including the preprocessing and architecture of RealMLP within other models. In particular,

* we include robust scaling and smooth clipping (RS+SC) into MLP-D, ResNet-D, and TabR-S-D. By default, these models use a version of the quantile transform (see <Ref>). We observe that it is beneficial on average but not on every benchmark for MLP-D and ResNet-D, while providing notable gains on every benchmark for TabR-S-D.

* we study the benefits of our architectural changes, cf. <Ref> (c), when applied directly to the setting of MLP-D. To this end, we approximately reproduce MLP-D in our codebase without weight decay (since the optimal value changes when including the NTP) and with marginally different early stopping thresholding logic. We also determine the best default learning rate on the meta-train benchmark, similar to <Ref>. Our reproduction achieves benchmark scores within 1% of the benchmark scores of the MLP-D (RS+SC) version. Adding the PL embeddings from [Gorishniy et al., 2022] with our default settings sometimes gives good results but is significantly worse on $\Ctec$, indicating that they need more tuning. In contrast, incorporating the RealMLP architectural changes (including their associated learning rate factors) improves scores on all benchmarks by around 5% or more, although they alone do not match the results of TabR-S-D. However, the non-architectural changes in RealMLP-TD make an even larger difference.

Comparison of preprocessing and architecture for different models. We include variants with robust scaling and smooth clipping (RS+SC), as well as other modified aspects, cf. <Ref>. We report the relative decrease in the shifted geometric mean benchmark scores compared to MLP-D. We also report approximate 95% confidence intervals, cf. <Ref>.

§.§ Comparing HPO Methods

In <Ref>, we compare two different HPO methods for GBDTs:

* Random search (HPO), as used in the main paper, with 50 steps.

* Tree parzen estimator (HPO-TPE) as implemented in hyperopt [Bergstra et al., 2013], with 50 steps. The first 20 of these steps use random search.

While TPE often performs slightly better, the differences in benchmark scores are relatively small.

Benchmark scores of selected methods on $\Ctrc$, $\Ctrr$, $\Ctec$, and $\Cter$ vs. average training time.
The $y$-axis shows the shifted geometric mean ($\operatorname{SGM}_\eps$) classification error (left) or RMSE (right) as explained in <Ref>.
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>.
The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

§.§ More Time-Error Plots

Here, we provide more time-vs-error plots.
<Ref> shows results for the arithmetic mean error, and <Ref> shows results for the arithmetic mean rank, which is also shown in <Ref> for the Grinsztajn et al., 2022 benchmark. In <Ref> we also reproduce the main plots from [Grinsztajn et al., 2022] with added methods.

Benchmark scores (arithmetic mean) vs. average training time.
The $y$-axis shows the arithmetic mean classification error (left) or RMSE (right).
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>. The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

Benchmark scores (ranks) vs. average training time.
The $y$-axis shows the arithmetic mean rank, averaged over all splits and datasets.
The $x$-axis shows average training times per 1000 samples (measured on $\Ctr$ for efficiency reasons), see <Ref>. The error bars are approximate 95% confidence intervals for the limit #splits $\to$ $\infty$, see <Ref>.

Results on the benchmarks of Grinsztajn et al., 2022.
The $y$-axis shows the arithmetic mean rank of each model. The $x$-axis shows average training times per 1000 samples, see <Ref>.

Results on the benchmarks of Grinsztajn et al., 2022, for classification (left) and regression (right).
The plot is similar to the one in the main part of [Grinsztajn et al., 2022], with our algorithms added. The $y$-axis shows the result of the best (on val, but evaluated on test) hyperparameter combination up to n steps of random step ($x$-axis). As in the original paper, we normalize each score between the max and the 10% quantile (classification) or  50% (regression), and truncate scores below 0 for regression.

§.§ Critical Difference Diagrams

Critical difference diagrams on the meta-train and meta-test benchmarks. The plots show the average rank of methods on each benchmark. Horizontal bars indicate groups of algorithms that are not statistically significantly different at a 95% confidence level according to a Friedman test and post-hoc Nemenyi test implemented in  [Herbold, 2020].

<Ref> analyzes the external validity of differences in average ranks between methods, i.e., whether they will generalize to new datasets from a distribution. While establishing external validity requires a large number of datasets, our benchmarks consistently show the improvements of RealMLP-TD over MLP-D to be externally valid.

§.§ Win-rate Plots

For pairs of methods, we analyze the percentage of (dataset, split) combinations on which the first method has a lower error than the second method. We plot these win-rates in marix plots: <Ref> shows the results on $\Ctrc$, <Ref> shows the results on $\Ctec$, <Ref> shows the results on $\Ctrr$, and <Ref> shows the results on $\Cter$.

Percentages of wins of row algorithms vs column algorithms on $\Ctrc$. Wins are averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average winrate (i.e., the average of the values in the row). When averaging, we use dataset-dependent weighting as explained in <Ref>.

Percentages of wins of row algorithms vs column algorithms on $\Ctec$. Wins are averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average winrate (i.e., the average of the values in the row).

Percentages of wins of row algorithms vs column algorithms on $\Ctrr$. Wins are averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average winrate (i.e., the average of the values in the row). When averaging, we use dataset-dependent weighting as explained in <Ref>.

Percentages of wins of row algorithms vs column algorithms on $\Cter$. Wins are averaged over all datasets and splits. Ties count as half-wins. Methods are sorted by average winrate (i.e., the average of the values in the row).

§ BENCHMARK DETAILS

§.§ Default Configurations

The parameters for RealMLP-TD and RealMLP-TD-S have already been given in <Ref>. <Ref> shows the hyperparameters of LGBM-TD and LGBM-D. <Ref> shows the hyperparameters of XGB-TD and XGB-D. <Ref> shows the hyperparameters of CatBoost-TD and CatBoost-D. The parameters for LGBM-D, XGB-D, and CatBoost-D have been taken from the respective libraries at the time of writing and are given here for completeness. The parameters for MLP-D are given in <Ref> and the parameters for ResNet-D are given in <Ref>. By RTDL quantile transform, we refer to the version adding noise before fitting the quantile transform. The parameters for TabR-S-D are given in <Ref>. On the dionis dataset, we activate context freeze for TabR-S-D after four epochs, since training can take up to 30 GPU-hours even with context freeze.

For XGB-PBB-D, we use the default parameters from [Probst et al., 2019], with the following modifications: We use  gradient boosting since it is the new default in XGBoost 2.0. Moreover, since we have high-cardinality categories, we limit one-hot encoding to categories with less than 20 distinct values (not counting missing values), and use XGBoost's native categorical feature handling for the remaining categorical features.

Hyperparameters for LGBM-TD and LGBM-D. Italic hyperparameters have not been tuned.

Hyperparameter     2cLGBM-TD     LGBM-D

classif.     reg.

num_leaves     50     100     31

learning_rate     0.04     0.05     0.1

subsample     0.75     0.7     1.0

colsample_bytree     1.0     1.0     1.0

min_data_in_leaf     40     3     20

min_sum_hessian_in_leaf     1e-7     1e-7     1e-3

n_estimators     1000     1000     100

bagging_freq     1     1     1

max_bin     255     255     255

early_stopping_rounds     300     300     1000

Hyperparameters for XGB-TD and XGB-D. Italic hyperparameters have not been tuned for XGB-TD.

Hyperparameter     2cXGB-TD     XGB-D

classif.     reg.

max_depth     6     9     6

learning_rate     0.08     0.05     0.3

subsample     0.65     0.7     1.0

colsample_bytree     1.0     1.0     1.0

colsample_bylevel     0.9     1.0     1.0

min_child_weight     5e-6     2.0     1.0

lambda     0.0     0.0     1.0

tree_method     hist     hist     hist

n_estimators     1000     1000     100

max_bin     256     256     256

early_stopping_rounds     300     300     1000

Hyperparameters for CatBoost-TD and CatBoost-D. Italic hyperparameters have not been tuned for CatBoost-TD.

Hyperparameter     2cCatBoost-TD     CatBoost-D

classif.     reg.

boosting_type     Plain     Plain     Plain

bootstrap_type     Bernoulli     Bernoulli     Bayesian

max_depth     7     9     6

learning_rate     0.08     0.09     automatic

subsample     0.9     0.9     —

bagging_temperature     —     —     1.0

l2_leaf_reg     1e-5     1e-5     3.0

random_strength     0.8     0.0     1.0

one_hot_max_size     15     20     2

leaf_estimation_iterations     1     20     None

n_estimators     1000     1000     1000

max_bin     254     254     256

od_wait     300     300     None

od_type     Iter     Iter     Iter

Hyperparameters for MLP-D, adapted from [McElfresh et al., 2023].

Hyperparameter     Space

lr scheduler     None

n_layers     3

d_layers     [128, 256, 128]

Dropout prob.     0.1

lr     1e-3

Optimizer     AdamW

d_embedding     8

batch_size     128

max_epochs     1000

early stopping patience     20

Preprocessing     RTDL quantile transform

Activation function     ReLU

Initialization     PyTorch default

Weight decay     0.01

Hyperparameters for ResNet-D, adapted from [McElfresh et al., 2023].

Hyperparameter     Space

lr scheduler     None

Activation     ReLU

Normalization     BatchNorm

n_layers     2

d_layers     [128, 128]

d_hidden_factor     2

hidden_dropout     0.25

residual_dropout     0.1

lr     1e-3

weight_decay     0.01

Optimizer     AdamW

d_embedding     8

batch_size     128

max_epochs     1000

early stopping patience     20

Preprocessing     RTDL quantile transform

Hyperparameters for TabR-S-D, taken from [Gorishniy et al., 2023].

Hyperparameter     Space

num_embeddings     None

d_main     265

context_dropout     0.38920071545944357

d_multiplier     2.0

encoder_n_blocks     0

predictor_n_blocks     1

mixer_normalization     auto

dropout0     0.38852797479169876

dropout1     0.0

normalization     LayerNorm

activation     ReLU

batch_size     auto

patience     16

n_epochs     100,000

context_size     96

optimizer     AdamW

lr     0.0003121273641315169

weight_decay     1.2260352006404615e-06

Preprocessing     RTDL quantile transform

§.§ Hyperparameter Optimization

For LGBM-HPO, XGB-HPO, and CatBoost-HPO, we run 50 steps of random search using the search spaces presented in <Ref>, <ref>, and <ref>, respectively. These search spaces are adapted from the literature, using  in each case.

For RealMLP-HPO, we run 50 steps of random search. We provide a custom search space specified in <Ref>. For MLP-HPO, we also run 50 steps of random search with the search space in <Ref>, adapted from [Gorishniy et al., 2021].

Hyperparameter seach space for LGBM-HPO, adapted from [Prokhorenkova et al., 2018] with 1000 estimators instead of 5000.

Hyperparameter     Space

n_estimators     1000

bagging_freq     1

early_stopping_rounds     300

num_leaves     LogUniformInt[1, $e^7$]

learning_rate     LogUniform[$e^{-7}$, 1]

subsample     Uniform[0.5, 1]

feature_fraction     Uniform[0.5, 1]

min_data_in_leaf     LogUniformInt[1, $e^6$]

min_sum_hessian_in_leaf     LogUniform[$e^{-16}$, $e^5$]

lambda_l1     Random$\{$0, LogUniform[$e^{-16}$, $e^2$]$\}$

lambda_l2     Random$\{$0, LogUniform[$e^{-16}$, $e^2$]$\}$

Hyperparameter search space for XGB-HPO, adapted from [Grinsztajn et al., 2022]. We use the  method, which is the new default in XGBoost 2.0 and supports native handling of categorical values, while the old  method selection is not available in XGBoost 2.0. We also increase  to 300.

Hyperparameter     Space

tree_method     hist

n_estimators     1000

early_stopping_rounds     300

max_depth     UniformInt[1, 11]

learning_rate     LogUniform[1e-5, 0.7]

subsample     Uniform[0.5, 1]

colsample_bytree     Uniform[0.5, 1]

colsample_bylevel     Uniform[0.5, 1]

min_child_weight     LogUniformInt[1, 100]

alpha     LogUniform[1e-8, 1e-2]

lambda     LogUniform[1, 4]

gamma     LogUniform[1e-8, 7.0]

Hyperparameter search space for CatBoost-HPO, adapted from [Shwartz-Ziv and Armon, 2022], who did not specify the number of estimators.

Hyperparameter     Space

boosting_type     Plain

bootstrap_type     Bayesian

n_estimators     1000

max_depth     6

od_wait     300

od_type     Iter

learning_rate     LogUniform[$e^{-5}$, 1]

bagging_temperature     Uniform[0, 1]

l2_leaf_reg     LogUniform[0, 10]

random_strength     UniformInt[1, 20]

one_hot_max_size     UniformInt[0, 25]

leaf_estimation_iterations     UniformInt[1, 20]

Hyperparameter search space for RealMLP-HPO. The remaining hyperparameters are set as in RealMLP-TD.

Hyperparameter     classif.     reg.

Num. embedding type     Choice([None, PBLD, PL, PLR])     same

Use scaling layer     Choice([True, False], p=[0.6, 0.4])     same

Learning rate     LogUniform([2e-2, 3e-1])     same

Dropout prob.     Choice([0.0, 0.15, 0.3], p=[0.3, 0.5, 0.2])     same

Activation fct.     Choice([ReLU, SELU, Mish])     same

Hidden layer sizes     Choice([[256, 256, 256], [64, 64, 64, 64, 64], [512]], p=[0.6, 0.2, 0.2])     same

Weight decay     Choice([0.0, 2e-2])     same

$\bfw^{(1,i)}_{\text{emb}}$ init std.     LogUniform([0.05, 0.5])

Label smoothing $\eps$     Choice([0.0, 0.1], p=[0.3, 0.7])     no label smoothing

Hyperparameter search space for MLP-HPO, adapted from [Gorishniy et al., 2021]. We reduced the embedding dimension upper bound, the maximum number of epochs, and the number of layers to have a more acceptable runtime on the meta-test benchmarks. We also used a minimum batch size of 256. As in the original paper, the size of the first and the last layers are tuned and set separately, while the size for
in-between layers is the same for all of them.

Hyperparameter     2cSpace

#samples$\leq$100,000     #samples$>$100,000

n_layers     UniformInt[1, 8]     UniformInt[1, 11]

d     UniformInt[64, 512]     UniformInt[64, 1024]

d_hidden_factor     2cUniformInt[1, 4]

hidden_dropout     2cUniform[0, 0.5]

residual_dropout     2cChoice(0, Uniform[0, 0.5])

lr     2cLogUniform[1e-5, 1e-2]

weight decay     2cChoice(0, LogUniform[1e-6, 1e-3])

d_embedding     2cUniformInt[8, 31]

batch_size     2c256 if train_size < 30K

2c else 512 if train_size < 100K else 1024

lr_scheduler     2cNone

Optimizer     2cAdamW

max #epochs     2c400

early stopping patience     2c16

Preprocessing     2cRTDL quantile transform

§.§ Dataset Selection and Preprocessing

§.§.§ Meta-train Benchmarks

For the meta-train benchmarks, we adapt code from [Steinwart, 2019] to collect all datasets from the UCI repository that follow certain criteria:

* Between 2,500 and 50,000 samples.

* Number of features at most 1,000.

* Labeled as classification or regression task.

* Description made it straightforward to convert the original dataset into a numeric .csv format.

* Uploaded before 2019-05-08.

We remove rows with missing values and keep only those datasets that still have at least 2,500 samples.[We noticed later that the ozone_level_1hr and ozone_level_8hr datasets contain less than 2,500 samples, but we decided to keep them since we already used them for tuning the hyperparameters.] Some datasets are labeled both as regression and classification datasets, in which case we use them for both. We standardize the targets for regression datasets, such that a constant predictor can achieve an RMSE of 1. Some datasets contain different versions (e.g., different target columns), in which case we use all of them. To avoid biasing the results towards one dataset, we compute benchmark scores using weights proportional to $1/\#\text{versions}$. In total, we obtain 71 classification datasets (including versions) out of 46 original datasets, and 47 regression datasets (including versions) out of 26 original datasets. Tables <ref> and <ref> summarize key characteristics of these datasets. We count datasets with the same prefix (before the first underscore) as being versions of the same dataset for weighting, except for the two facebook datasets in $\Ctrr$, which we count as distinct. For regression, we standardize the targets to have mean zero and variance 1.

During earlier development of the MLP, the meta-train benchmark used to include an epileptic seizure recognition dataset, which has since been removed from the UCI repository, hence we do not report results on it.

Datasets in the meta-train classification benchmark.

Datasets in the meta-train regression benchmark.

§.§.§ Meta-test Benchmarks

The meta-test benchmarks consist of datasets from the AutoML Benchmark [Gijsbers et al., 2022] and additional regression datasets from the OpenML-CTR23 benchmark [Fischer et al., 2023], obtained from OpenML [Vanschoren et al., 2014].

We make the following modifications:

* We use brazilian_houses from OpenML-CTR23 and exclude Brazilian_houses from the AutoML regression benchmark, since the latter contains three additional features that should not be used for predicting the target.

* We use another version of the sarcos dataset where the original test set is not included, since the original test set consists of duplicates of training samples.

* We excluded the following datasets because versions of them were already contained in the meta-training set:

* For classification: kr-vs-kp, wilt, ozone-level-8hr, first-order-theorem-proving, GesturePhaseSegmentationProcessed, PhishingWebsites, wine-quality-white, nomao, bank-marketing, adult

* For regression: wine_quality, abalone, OnlineNewsPopularity, Brazilian_houses, physicochemical_protein, naval_propulsion_plant, superconductivity, white_wine, red_wine, grid_stability

We preprocess the datasets as follows:

* We remove rows with missing continuous values

* We subsample large datasets to contain at most 500,000 samples. Since the dionis dataset was particularly slow to train with GBDT models due to its 355 classes, we subsampled it to 100,000 samples.

* We standardize the targets for regression datasets, such that a constant predictor can achieve an RMSE of 1.

* We encode missing categorical values as a separate category.

* For regression, we standardize the targets to have mean zero and variance 1.

After preprocessing, we

* exclude datasets with less than 1,000 samples, these were

* for classification: albert, APSFailure, arcene, Australian, blood-transfusion-service-center, eucalyptus, KDDCup09_appetency, KDDCup09-Upselling, micro-mass, vehicle

* for regression: boston, cars, colleges, energy_efficiency, forest_fires, Moneyball, QSAR_fish_toxicity, sensory, student_performance_por, tecator, us_crime

* exclude datasets that have more than 10,000 features after one-hot encoding. These were Amazon_employee_access, Click_prediction_small, and sf-police-incidents (all classification).

Datasets in the meta-test classification benchmark.

Datasets in the meta-test regression benchmark.

§.§ Grinsztajn et al. benchmark

The datasets are taken from the benchmarks described in [Grinsztajn et al., 2022].
When a dataset is used both in benchmarks with and without categorical features, we use the version with categorical features. We preprocess the datasets following the same steps as in [Grinsztajn et al., 2022]:

* For neural networks, we quantile-transform the features to have a Gaussian distribution.
For TabR [Gorishniy et al., 2023], we use the modified quantile transform from the TabR paper.
For RealMLP, we use the preprocessing described in <Ref>, namely robust scaling and smooth clipping.

* For neural networks, we add as a hyperparameter the possibility to normalize the target variable for the model fit and transform it back for evaluation (via scikit-learn's TransformedTargetRegressor and StandardScaler, which differs from the QuantileTransformer from the original paper, as we found it to work better). The same standardization is also applied to all default-parameter versions of neural networks.

* For models that do not handle categorical variables natively, we encode categorical features using  from scikit-learn.

* Train size is restricted to 10,000 samples and test and validation size to 50,000 samples.

Note that the datasets from the original benchmark are already slightly preprocessed, e.g., heavy-tailed targets are standardized and missing values are removed.
More details can be found in the original paper.

*Results normalization
For Figure <ref>, as in the original paper, we normalize the R2 or accuracy score for each dataset before averaging them.
We use an affine normalization between 0 and 1, 1 corresponding to the score of the best model for each dataset, and 0 corresponding
to the score of the worst model (for classification) and the 10th percentile of the scores (for regression). We use slightly different
percentiles compared to the original paper as we normalize across the scores of the tuned and default models, and not all steps of the random search,
which reduces the number of outliers. Other aggregation metrics are shown in <Ref>.

*Other details
We rerun classification results for neural networks compared to the original results to early stop on accuracy rather than on cross-entropy,
to make results more comparable with the rest of this paper.

§.§ Confidence Intervals

Here, we specify how our confidence intervals are computed. Let $X_{ij}$ denote the score (error/rank) of a method on dataset $i$ and split $j$, with $i \in \{1, \hdots, n\}$ and $j \in \{1, \hdots, m\}$. Then, the benchmark score $\calS$ can be written as
= g(∑_i=1^n w_i/m ∑_j=1^m f(X_ij)),

where $f = g = \id$ for the arithmetic mean. For the shifted geometric mean, we instead have $g = \exp$ and $f(x) = \log(x + \eps)$, $\eps = 0.01$. We interpret the benchmark datasets as fixed, but the splits as random. For each dataset $i$, $X_{i1}, \hdots, X_{im}$ are i.i.d. random variables. We first take the dataset averages
Z_j ∑_i=1^n w_i f(X_ij) .

The random variables $X_{1j}, \hdots, X_{nj}$ are independent but not identically distributed. Still, for lack of a better option, we assume that the $Z_j$ are normally distributed with unknown mean and variance. We know that the $Z_j$ are i.i.d., hence we use the confidence intervals from the Student's $t$-distribution for normally distributed random variables with unknown mean and variance. This gives us a confidence interval $[a, b]$ for $\frac{1}{m} \sum_{j=1}^m Z_j$. Since $g$ is increasing, we hence obtain a confidence interval $[g(a), g(b)]$ for $\calS = g\left(\frac{1}{m} \sum_{j=1}^m Z_j\right)$.

Comparison of two methods We often compute the error increase in % in the benchmark score of method A compared to method B with the shifted geometric mean, given by
100 ·(^(A)/^(B) - 1) .

Here, we leverage that the shifted geometric mean uses $g=\exp$ to write
^(A)/^(B) = g(∑_i=1^n w_i/m ∑_j=1^m (f(X_ij^(A)) - f(X_ij^(B)))) ,

which is of the same form as Eq. (<ref>). Hence, we obtain confidence intervals for this quantity using the same method.

§.§ Time Measurements

For our meta-train and meta-test benchmarks, we report training times measured as follows: We run all methods on a single compute node with a 32-core AMD Ryzen Threadripper Pro 3975 WX CPU, using 32 threads for GBDTs and the PyTorch default settings for MLPs. No method is run on GPUs.
We run methods sequentially on one split on each dataset of the meta-train-class and meta-train-reg benchmarks. For RealMLP-HPO, we only run two random search steps and extrapolate the runtime to 50 steps. Runtimes for combinations of models (Best and Ensemble) are computed as the sum of the individual runtimes. We compute the runtime per 1000 samples on each dataset and then average them. For simplicity, we do not use the dataset-dependent weighting employed otherwise on the meta-train benchmark.
For the Grinsztajn et al., 2022 benchmark, we follow the original paper and run neural networks on a GPU and the other models on 1 core of AMD EPYC 7742 64-Core Processor, and we average the time across all random steps (for each random step, the time is averaged across splits).
To compute the runtime of neural networks, we restrict ourselves to steps ran on the same GPU model (NVIDIA A100-40GB), which means that we exclude datasets for which we have less than 15 steps of each model on this GPU (leaving us with 11 datasets for classification and 15 for regression). We then compute the average runtime per 1000 samples on each dataset and average them.

§.§ Compute Resources

While we did not measure compute resources precisely, our experiments required at least around 2000 hours on RTX 3090 GPUs and other GPUs, as well as roughly 10,000 hours on HPC CPU nodes (32–64 cores).

§.§ Used Libraries

Our implementation uses various libraries, out of which we would like to particularly acknowledge PyTorch [Paszke et al., 2019],  Scikit-learn [Pedregosa et al., 2011], Ray [Moritz et al., 2018], XGBoost [Chen and Guestrin, 2016], LightGBM [Ke et al., 2017], and CatBoost [Prokhorenkova et al., 2018]. For using XGBoost, LightGBM, and CatBoost, we adapted wrapping code from the CatBoost quality benchmarks [Prokhorenkova et al., 2018].
For the [Grinsztajn et al., 2022] benchmark, we adapt the code from the original paper, which is available at the new branch better_by_default in the repository <https://github.com/LeoGrin/tabular-benchmark/tree/better_by_default>.

§ RESULTS FOR INDIVIDUAL DATASETS

Here, we provide and compare the results of central methods per dataset. First, Figures <ref> – <ref> show scatterplot comparisons for different models.

<Ref> and <Ref> show results on $\Ctrc$. <Ref> and <Ref> show results on $\Ctrr$. <Ref> and <Ref> show results on $\Ctec$. <Ref> and <Ref> show results on $\Cter$.

Best-TD vs CatBoost-HPO on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

Best-TD vs Best-HPO on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

RealMLP-TD vs CatBoost-TD on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

RealMLP-HPO vs CatBoost-HPO on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

Ensemble-TD vs Best-TD on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

RealMLP-TD vs RealMLP-HPO on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

CatBoost-TD vs CatBoost-HPO on individual datasets. Each point represents the errors of both models on a dataset, averaged across 10 train-valid-test splits. The black line represents equal errors ($x=y$).

Classification error of untuned methods on datasets in $\Ctrc$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

Classification error of tuned methods on datasets in $\Ctrc$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

RMSE of untuned methods on datasets in $\Ctrr$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

RMSE of tuned methods on datasets in $\Ctrr$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

Classification error of untuned methods on datasets in $\Ctec$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

Classification error of tuned methods on datasets in $\Ctec$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

RMSE of untuned methods on datasets in $\Cter$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

RMSE of tuned methods on datasets in $\Cter$, averaged over ten train-validation-test splits. When we write $a \pm b$, $a$ is the mean error on the dataset and $[a-b, a+b]$ is an approximate 95% confidence interval for the mean in the #splits $\to$ $\infty$ limit. The confidence interval is computed from the $t$-distribution using a normality assumption as in <Ref>. In each row, the lowest mean error is highlighted in bold, and errors whose confidence interval contains the lowest error are underlined.

§ BROADER IMPACT

We present NN models with an improved speed-accuracy tradeoff and hope that this can reduce the resource consumption of tabular models in applications and further benchmarks. While tabular ML has many potential applications, we feel that none must be particularly highlighted here.

§ NEURIPS PAPER CHECKLIST

* Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Justification: See the paper.

* The answer NA means that the abstract and introduction do not include the claims made in the paper.

* The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

* The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

* It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

* Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Justification: We provide a paragraph on limitations in <Ref>.

* The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

* The authors are encouraged to create a separate "Limitations" section in their paper.

* The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

* The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

* The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

* The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

* If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

* While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

* Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Justification: We do not have theoretical results.

* The answer NA means that the paper does not include theoretical results.

* All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

* All assumptions should be clearly stated or referenced in the statement of any theorems.

* The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

* Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

* Theorems and Lemmas that the proof relies upon should be properly referenced.

* Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Justification: We provide implementation details in the paper and the appendix, especially <Ref>. We cannot provide all details on dataset preprocessing, but these are provided with the code.

* The answer NA means that the paper does not include experiments.

* If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

* If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

* Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

* While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

* If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

* If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

* If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

* We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

* Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Justification: We provide code for the meta-train and meta-test benchmarks in the supplementary material. For the camera-ready version, we will provide more complete documentation, the code for the [Grinsztajn et al., 2022] benchmark, and the experimental data.

* The answer NA means that paper does not include experiments requiring code.

* Please see the NeurIPS code and data submission guidelines (<https://nips.cc/public/guides/CodeSubmissionPolicy>) for more details.

* While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

* The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (<https://nips.cc/public/guides/CodeSubmissionPolicy>) for more details.

* The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

* The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

* At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

* Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

* Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Justification: These details are provided in the paper and appendix.

* The answer NA means that the paper does not include experiments.

* The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

* The full details can be provided either with the code, in appendix, or as supplemental material.

* Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Justification: We provide error bars for fixed datasets, quantifying the uncertainty over the random splits, as described in the appendix. We also provide critical-difference diagrams in <Ref>.

* The answer NA means that the paper does not include experiments.

* The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

* The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

* The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

* The assumptions made should be given (e.g., Normally distributed errors).

* It should be clear whether the error bar is the standard deviation or the standard error of the mean.

* It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

* For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

* If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

* Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Justification: We did not track these resources in detail, but a rough estimate for the total resources can be found in <Ref>.

* The answer NA means that the paper does not include experiments.

* The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

* The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

* The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

* Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics <https://neurips.cc/public/EthicsGuidelines>?

Justification: The research appears to conform to the code of ethics but we did not check all $\approx$ 200 datasets used in this paper.

* The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

* If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

* The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

* Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Justification: See <Ref>, but this work is foundational research and the impact is unclear.

* The answer NA means that there is no societal impact of the work performed.

* If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

* Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

* The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

* The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

* If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

* Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Justification: We only use publicly available tabular datasets without obvious safety risks and do not release pretrained models.

* The answer NA means that the paper poses no such risks.

* Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

* Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

* We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

* Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Justification: We use $\approx$ 200 datasets from online repositories, so we cite the repositories / benchmark curators but not the individual datasets.

* The answer NA means that the paper does not use existing assets.

* The authors should cite the original paper that produced the code package or dataset.

* The authors should state which version of the asset is used and, if possible, include a URL.

* The name of the license (e.g., CC-BY 4.0) should be included for each asset.

* For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

* If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, <paperswithcode.com/datasets> has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

* For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

* If this information is not available online, the authors are encouraged to reach out to the asset's creators.

* New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Justification: We only provide download code for existing datasets.

* The answer NA means that the paper does not release new assets.

* Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

* The paper should discuss whether and how consent was obtained from people whose asset is used.

* At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

* Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

* The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

* Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

* According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

* Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

* The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

* Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

* We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

* For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.