# LaTable: Towards Large Tabular Models

Boris van Breugel Jonathan Crabbé Rob Davis Mihaela van der Schaar
University of Cambridge
<EMAIL_ADDRESS>
###### Abstract

Tabular data is one of the most ubiquitous modalities, yet the literature on
tabular generative foundation models is lagging far behind its text and vision
counterparts. Creating such a model is hard, due to the heterogeneous feature
spaces of different tabular datasets, tabular metadata (e.g. dataset
description and feature headers), and tables lacking prior knowledge (e.g.
feature order). In this work we propose LaTable: a novel tabular diffusion
model that addresses these challenges and can be trained across different
datasets. Through extensive experiments we find that LaTable outperforms
baselines on in-distribution generation, and that finetuning LaTable can
generate out-of-distribution datasets better with fewer samples. On the other
hand, we explore the poor zero-shot performance of LaTable, and what it may
teach us about building generative tabular foundation models with better zero-
and few-shot generation capabilities.

##  1 Introduction

Motivation. Foundation models (FMs) in the image and text domains epitomize
the value of large scale training, achieving impressive results and pushing
the boundaries of AI capabilities. In this work, we propose a tabular
diffusion model that can be trained across vastly different datasets, and
could possibly be the basis for a tabular foundation model.

Tabular foundation models, or Large Tabular Models (LTMs), lag far behind
their text and vision counterparts, and existing LTM research focuses on
representation and supervised learning [1]. Yet, despite this
_underrepresentation_ in research, tabular data is _ubiquitous_ in the real
world—including in medicine [2, 3], finance [4], census data [5], data science
competitions [6, 7], and natural science [8]. A generative LTM could transform
these fields, through enabling few- and zero-shot generation of synthetic data
(e.g. for privacy, debiasing, data democratization, augmentation, adaptation,
simulation, etc. [9]), better imputation, and providing a base model that can
be finetuned to other tasks (e.g. representation learning or prediction). Few-
shot capabilities are especially interesting for the tabular domain, where
datasets are often smaller. In Box A we expand on applications of a generative
LTM.

Aim. Developing a generative LTM for tabular data is hard; different datasets
cover vastly different features, there is no prior knowledge in the form of
feature order, and data is messy—e.g. different datasets use different
formats, contain different features, and can contain categorical, numerical,
and missing values. As a result of this, training models meaningfully on large
amounts of heterogeneous data remains underexplored.

We aim to take a step towards Large Tabular Models by developing a versatile
tabular generative model with the following desiderata [1]:

  1. D1

Cross-dataset generation. We need a generative model to be able to generate
different tables, which requires it being able to generate different features
and variable number of features.

  2. D2

Categorical and numerical feature generation. Tables consist of combinations
of both continuous and discrete data, and we want to be able to model
each.111In contrast, an LLM-based approach like [10] implicitly models each
numerical feature as a series of discrete tokens, see Section 2.

  3. D3

Use textual context. To achieve generating across tables, the model will need
to understand contextual metadata—including the dataset description, feature
names, and allowed categories (for categorical features).

  4. D4

Equivariance w.r.t. column order. Table column order is usually arbitrary, so
we want an LTM to be equivariant to this. In other words, define generative
model $G:\mathcal{S}^{L}\times\mathcal{M}\rightarrow\mathcal{X}^{L}$ with
$\mathcal{S}$ the column-wise input space (e.g. feature names), $\mathcal{X}$
the output space, and $\mathcal{M}$ metadata (e.g. dataset description); for
input $\mathbf{s}\in\mathcal{S}^{L}$, $r\in\mathcal{M}$, and $T$ some
permutation, we desire $G(T(\mathbf{s}),r)=T(G(\mathbf{s},r))$.

Contributions. Our contributions are as follows:

  1. 1.

We introduce LaTable, a novel tabular diffusion model that satisfies these
desiderata and can thus be trained across vastly different tabular datasets
(including different features, number of features, and different types) and
with context of table metadata.

  2. 2.

We empirically show how LaTable outperforms existing generative models
significantly on in-distribution generation, and when finetuned on new, out-
of-distribution datasets.

  3. 3.

We explore the limitations of LaTable in terms of zero-shot capabilities. We
argue that this stems from the limited pretraining of LaTable and discuss the
challenges ahead for generative LTM research. Our findings points towards the
central role that the training data should play in future research.

##  2 Related work

Tabular single-dataset generative models. Tabular data is challenging due to
its lack of structural meaning (cf. images), mixed-type variables, and often
limited size. A growing body of work is developing generative models tailored
to tabular data [11, 12, 13], including diffusion/score-based models [14, 15,
16]. Nonetheless, most methods ignore or naively one-hot-encode categorical
data (e.g. feature names and categories), which loses information and context
that could help overcome data availability issues (D3).222Of course, without
cross-table training, it would not be possible to learn much from the dataset
description, feature names, and categories, due to lack of diversity—each
dataset provides just one description, and not many unique features or
categories to learn from. Naturally, methods developed for single datasets are
also not designed for cross-datasets generation (D1, or equivariance w.r.t.
column order (D4).

LM-based approaches. Pretrained language models (LMs) may provide a solution
to cross-dataset generation with (D1-D4), as they contain general knowledge
and can represent strings as numerical vectors in a space structured by
language supervision. These works all convert the rows of tables into strings
that can be processed by the LM. The string generated by the LM is then
converted back to a tabular format. Borisov et al. [10] finetune GPT-2 [17] on
tables. They aim to achieve approximate equivariance (D4) through randomly
permuting feature permutations, and sample tokens autoregressively from GPT’s
predicted token probabilities to model distributions (cf. sample only the most
likely sentences). Solatorio and Dupriez [18] note that [10]’s approach
retains GPT-2’s original vocabulary, yet that most tokens may not appear in
the tables of interest. They instead use a fixed-set vocabulary[19], which
reduces the chances of generating invalid samples and thus improves
efficiency. Last, Zhao et al. [20] show that an untrained, smaller LM can
generate more accurate data and do so more cheaply than [10]. We note that in
contrast to us, previous works [10, 18, 20] train/finetune their model on just
a single tabular dataset—i.e. they do not attempt cross-dataset training and
do not aim to create a generative model that generalizes beyond the training
data.

The advantage of LM-based generation is the simplicity and the fact that it
does not require any manual preprocessing of data, since the raw data can be
presented to the LM directly (e.g. prompting and in-context examples).
However, adapting LMs as LTMs comes with serious disadvantages [1]. First, LMs
are expensive, during both finetuning and inference. A 1-min training job for
CTGAN, takes a whopping $540\times$ longer for [10] (see their Appendix B.5,
Table 6). One of the core sources of this inefficiency is the linearization of
table rows as sentences, and generating these autoregressively. For tabular
data, where many columns are typically numerical, this is problematic—it means
that a single numerical variable is implicitly modelled as an autoregressive
series of categorical variables (e.g. $1.23$ is modelled as
$1\rightarrow.\rightarrow 23$). Hence, generating this number with an LM
requires 3 expensive forward calls to the model (plus one for the field
separation token that follows). Fitting the whole data in context might also
become difficult with datasets containing hundreds of columns. Furthermore,
the multinomial training objective of LM is not apt at approximating
continuous distributions—e.g. Renda et al. [21] show how LMs do not accurately
generate simple distributions (e.g. uniform) and van Breugel and van der
Schaar [1] visualize how modelling a simple Gaussian distribution with an LM
is non-trivial. By opting for an end-to-end numerical representation of the
data and a diffusion model framework, LaTable circumvents these limitations.

##  3 Method

Summary. We propose LaTable, which satisfies the desiderata. LaTable uses an
encoder-only transformer as backbone for mixed-type diffusion. The input of
the model consists of noised feature values (i.e. from the forward noising
process), but also the dataset description, column names, conditioning mask
(in case of conditional generation), and boolean missingness mask. All
metadata strings (e.g. dataset description and feature names) and categorical
features are first encoded using a pretrained LLM into a fixed-length
embedding. All inputs are subsequently mapped to a common hidden space using
separate element-wise MLPs, and added to provide the direct input of the
transformer. The output of the transformer is decoded on a feature-by-feature
basis, where categorical features are mapped to probabilities over the
original features. See Figure 1.

###  3.1 Set-up and notation

We index datasets with $k$, samples with $i$, and features with $j$.333To
avoid excessive clutter, we will leave out indices when they are irrelevant or
clear from context. Let us assume we have access to a metadataset
$\mathcal{D}=\\{D^{(k)}\\}_{k=1}^{n_{D}}$, where each dataset $D^{(k)}$
consists of a description, column names, and the data itself. Let
$d_{k}\in\mathbb{N}$ denote the number of columns for dataset $k$. Feature
spaces will generally differ for each dataset, hence let us define the $j$th
feature space for dataset $k$ as $\mathcal{X}^{(k)}_{j}$—i.e. for categorical
data we let this be a finite set of strings, for numerical columns it may be
some subset of $\mathbb{R}$. Let us denote the set of all strings with
$\mathcal{S}$. We can thus write,
$D^{(k)}=(r^{(k)},\mathbf{s}^{(k)},\mathbf{X}^{(k)})$, with
$r^{(k)}\in\mathcal{S}$ the dataset description,
$\mathbf{s}^{(k)}\in\mathcal{S}^{d_{k}}$ feature names, and samples
$\mathbf{X}^{(k)}_{i}\in\prod_{j=1}^{d_{k}}\mathcal{X}^{(k)}_{j}$. Finally,
let us denote the indices of the categorical features by
$\mathcal{I}_{cat}^{(k)}$.

###  3.2 Generative model choice

We use a diffusion model [22] (i.e., score-based model [23]) for modelling the
data distribution. Diffusion models have increased in popularity thanks to
their capability to produce samples of higher quality than GANs and VAEs. We
use the discrete-time formulation with Denoising Diffusion Implicit Model
(DDIM) noise scheduler [24].

To satisfy the desiderata, however, we need to think carefully about what
architecture backbone we use for the diffusion model, how we model both
continuous and categorical variables, and how the tabular metadata fits in.
Let us look into each element of LaTable in more detail, and how it helps to
satisfy desiderata D1-D4.

###  3.3 Satisfying desiderata

####  3.3.1 Transformer backbone (D1, D4)

We use an encoder-only transformer at the core of LaTable. The transformer
backbone is useful for our setting, as it allows variable-length input and
output—this allows us to train across datasets (D1). We _do not_ use a
positional encoding for the input, such that the transformer is equivariant
w.r.t. the input (D4). In contrast, a recurrent architecture would not
necessarily satisfy this equivariance property.444Importantly, removing the
positional encoding means the transformer cannot know which column is which.
We remove this undesirable symmetry by adding column information to each input
element, in the form of a feature name embedding—see Section 3.3.2. We denote
the transformer’s hidden dimension by $d_{h}$.

####  3.3.2 Using context (D3)

To make use of context (D3), we will encode the category names, column names,
and dataset description using a pretrained LLM encoder. Note that the dataset
description is the same across all features, but that the other metadata are
defined on the feature level. By using a pretrained LLM encoder, we avoid
having to learn embeddings for descriptions and categories from scratch. It
also enables us to use textual similarities, even if our tabular data is
limited. For example, one dataset may contain column “gender” and another
“sex”, and we need not learn from the data alone that these are closely
related.

We denote the pretrained LLM by $f_{\mathrm{LM}}$. Text encodings are fixed
length $d_{f}$ for any input string $\mathcal{S}$, such that
$f_{\mathrm{LM}}:\mathcal{S}\rightarrow\mathbb{R}^{d_{f}}$. Whenever we encode
text data, we will denote this with a bar, e.g. $\bar{\mathbf{r}}^{(k)}$, as
follows:

  1. 1.

Each string dataset description $r^{(k)}$ is encoded by the LLM into a vector
$\bar{\mathbf{r}}^{(k)}$.

  2. 2.

Each categorical variable is converted into a string “[column name $s$] is
[category $c$]". Categories $(c_{1},...,c_{n})$ are then encoded by the LLM
and stacked into an _embedding matrix_ $\bar{\mathbf{C}}^{(k)}_{j}$

$\bar{\mathbf{C}}^{(k)}_{j}=(\bar{\mathbf{c}}_{1},...,\bar{\mathbf{c}}_{n}).$
(1)

  3. 3.

Feature names $(s_{j})_{j}$ are encoded by the LLM into vectors
$(\bar{\mathbf{s}}_{j})_{j}$

We condition on the encoded metadata by mapping the dataset description
$\bar{\mathbf{r}}$ and feature embeddings $\bar{\mathbf{s}}_{j}$ to size
$d_{h}$ using shallow MLPs $g_{r},g_{s}$ and adding it to the transformer
input (see Figure 1 in blue).

For the encoder $f_{\mathrm{LM}}$, we use model WhereIsAI/UAE-Large-V1 for
three reasons: it is open source and easily
available,555https://huggingface.co/WhereIsAI/UAE-Large-V1; it achieved SOTA
on the Massive Text Embedding Benchmark (MTEB) [25] until December 31 2023;
and it is very lightweight (size: 1.34 GB). All encodings are cached to disk
before training and testing to avoid LLM forward calls during training.

####  3.3.3 Mixed-type variables (D2)

In contrast to standard LLM transformers and diffusion models, we need to
generate numerical values as well as categoricals (D2). Let us start with the
numerical variables.

Numerical variables. For numerical variables we can use relatively standard
diffusion. Given some time step $t\in T$, the forward denoising process is
applied to all numerical features $x_{j}$, giving the noisy $(x_{j})_{t}$. A
shallow MLP $g^{i}_{n}:\mathbb{R}\rightarrow\mathbb{R}^{d_{h}}$ is used to map
each numerical variable to the transformers input space. The transformer is
subsequently applied to the sequence of all variables. Subsequently, each
sequence element in the transformer’s output corresponding to a numerical
variable is mapped back to scalars using output network
$g^{o}_{n}:\mathbb{R}^{d_{h}}\rightarrow\mathbb{R}$. For each numerical
$(x_{j})_{0}$, this yields output $(\hat{x}_{j})_{0}$, and the loss (e.g.
sample-loss, $\epsilon$-loss, or $v$-prediction) is computed per feature.

Categorical variables. To handle categorical variables, we draw inspiration
from Continuous Diffusion for Categorical Data [14]. The original CDCD learns
high-dimensional embeddings for each category separately, before applying
$L_{2}$ normalization to each embedding (after which noise is added for the
denoising). During training, CDCD adds noise to these categorical embeddings
(identical to continuous diffusion), and trains a model that predicts
probabilities for the original categories. This model is trained using some
classification loss, e.g. cross-entropy. During sampling, the outputted
probabilities are used to predict the score in the continuous space.
Specifically, they use _score interpolation_ —which is effectively a weighted
mean of all category embeddings, where weights are given by the probabilities
predicted by the score model.

In our case, we do _not_ want to learn embeddings and predict probabilities
for all possible categories in all datasets. By learning an embedding
separately for each category, this loses context that is captured in the
category name (e.g. how categories female and woman are related). It also
makes it hard to scale to very large number of total categories, especially if
some of these categories are hardly ever observed. Instead, we use the
pretrained LLM $f_{\mathrm{LM}}$ for acquiring ${d_{f}}$-dimensional
embeddings for each category $c$, which _do_ carry the contextual knowledge of
the category name.666As discussed in Section 3.3.2, for all categorical
features we embed the categorical $c$ as $\bar{\mathbf{c}}=$“[feature name] is
[$c$]”. We fine-tune these embeddings using a shallow MLP followed by an
$L_{2}$ normalization layer (similar to CDCD), which we together denote by
$g_{f}:\mathbb{R}^{d_{f}}\rightarrow S^{d_{f}}$.777Without the normalization
layer, it would be difficult for the model to give embedded categories with
small $L_{2}$-norm a high probability (see Eq. 2). We add the shallow MLP’s
finetuning, to avoid projecting embeddings that are quite different in
$\mathbb{R}^{d}$ to the same point on the unit sphere. The forward diffusion
process corrupts these vectors by adding Gaussian noise, giving
$\bar{\mathbf{c}}_{t}$. Similar to the numericals, we use a network
$g^{i}_{c}:\mathbb{R}^{d_{f}}\rightarrow\mathbb{R}^{d_{h}}$ to map these noisy
embeddings to the transformer’s input space, and network
$g^{o}_{c}:\mathbb{R}^{d_{h}}\rightarrow\mathbb{R}^{d_{f}}$ is applied to the
categorical sequence elements in the transformer’s output—giving predicted
embeddings in $\mathbb{R}^{d_{f}}$.

One key question remains: considering each categorical feature may have a
different number of categories, how do we map the transformer output (in
$\mathbb{R}^{d_{f}}$) back to the individual categories? We achieve this using
an attention-like layer. Letting $\hat{\mathbf{c}}_{j}\in\mathbb{R}^{d_{f}}$
be the output of the model, we let the final predicted probability for some
category $i$ be proportional to the similarity of the generated vector and the
original categories’ embeddings (see Eq. 1):

$p(c_{i})=\text{softmax}(g_{f}(\bar{\mathbf{C}})^{T}\hat{\mathbf{c}})_{i},\forall
i,$ (2)

thereby allowing us to predict probabilities for each datasets’ and
categoricals’ categories individually. Identical to CDCD, we use cross-entropy
loss $\mathrm{CE}$ for the categorical variables and use score interpolation.
Combining the numerical and categorical losses, we get:

$\mathcal{L}(x,\hat{x})=\frac{1}{d_{k}}\left[\sum_{j\in\mathcal{I}_{cat}^{(k)}}\mathrm{CE}(x_{j},\hat{x}_{j})+\sum_{j\not\in\mathcal{I}_{cat}^{(k)}}\mathrm{MSE}(x_{j},\hat{x}_{j})\right].$
(3)

Bringing it together. Overall, the model thus consists of a number of
independent MLPs $g_{\cdot}$, that allow us to bring differently sized
variables (dataset description, feature column name embeddings, categorical
features, numerical features) all to the same space $d_{h}$. The input of the
transformer consists of a sequence of inputs $h_{j}\in\mathbb{R}^{d_{h}}$,
$j=1,...,d_{k}$ , with
$h_{j}=g^{i}_{c}((\bar{x}_{j})_{t})+g_{r}(\bar{r})+g_{s}(\bar{s}_{j})+g_{t}(t)$
for categoricals, and
$h_{j}=g^{i}_{n}(x_{j})_{t})+g_{r}(\bar{r})+g_{s}(\bar{s}_{j})+g_{t}(t)$ for
numericals. The transformers output sequence is again decoded using
independent shallow networks $g_{c}^{o},g_{n}^{o}$, to provide score estimates
for both numerical and categorical variables. See Figure 1 for a full
overview.

Figure 1: The diffusion pipeline, for both training and generation. Note that
the whole pipeline is flexible with respect to the number of numerical and
categorical features as input. The LLM is encoder is frozen and the
transformer is an encoder-only model without positional encodings or causal
masking. Additional conditioning (e.g. missingness mask, conditioning
information) can be trivially added to the transformer input, or through
cross-attention layers.

##  4 Experiments

In this section we explore the tabular data generation capabilities of
LaTable. We show how LaTable achieves significantly better in-distribution
generation, as well as few-shot out-of-distribution generation through minor
finetuning. These observations suggest that LaTable’s data generation
capabilities generalize across tables. At the frontier of these capabilities,
we discuss zero-shot tabular data generation as a limitation of LaTable. We
explore how this relates to the limited diversity and quality of the training
data, drawing lessons for future research into generative LTMs.

Data. We combine curated metadatasets [26] and [27] from OpenML, which
respectively cover classification and regression tasks. After removing image
datasets, these total 83 sets (678k samples, 3021 unique columns)—see Figure 4
for number of samples and features per dataset. We split up these datasets
into in-distribution $\mathcal{D}_{\mathrm{id}}$ and out-of-distribution
$\mathcal{D}_{\mathrm{ood}}$, including respectively 78 and 5 datasets. In
contrast to other large metadatasets (e.g. TabLib [28]) that may contain
(processed) duplicate sets, the datasets we use are curated, hence there is no
data leakage. All experimental details can be found in Appendix B.

###  4.1 In-distribution generation

We use independent train-val-test splits of each set in
$\mathcal{D}_{\mathrm{id}}$, and train LaTable on all training sets in
$\mathcal{D}_{\mathrm{id}}$. We compare the generation quality of LaTable to
popular synthetic data baselines. We choose tabular generators in different
classes, including TVAE, CTGAN [12], TabDDPM (diffusion) [15], and ARF (tree-
based) [13]. Note that for each of these models, we train a new model on each
individual dataset and for each seed. We compare downstream model train-on-
synthetic-test-on-real AUC performance (of an XGBoost classifier and taking
the label of the original dataset); synthetic data metrics precision and
recall [29, 30]; and density and coverage [31]. Note that precision and
density aim to quantify realism or fidelity of samples, whereas recall and
coverage measure diversity.

As we observe in Table 1, LaTable achieves significantly better results than
baselines on most metrics. The closest competitor is the tree-based ARF, which
scores comparably on downstream AUC and recall but significantly poorer on the
other metrics. Metric Recall gives a high score when data samples are very
diverse (i.e. the synthetic distribution has a large support), but this does
necessarily reflect well on data quality (see Fig. 1b in [31]). This might
explain why recall is high for all methods, even though they fail on other
metrics.

Table 1: Average performance of each generative model across 78 OpenML datasets in $\mathcal{D}_{\mathrm{id}}$. We report the mean and standard deviation over 5 random seeds. Downstream AUC is computed for an XGBoost model trained on synthetic, tested on real data (only including classification datasets). LaTable is significantly better on all metrics except recall. _Generator_ |  _Down. AUC_ [$\uparrow$] |  _Density_ [$\uparrow$] |  _Coverage_ [$\uparrow$] |  _Precision_ [$\uparrow$] |  _Recall_ [$\uparrow$]
---|---|---|---|---|---
Real | 0.895 (0.007) | 0.961 (0.005) | 0.955 (0.002) | 0.953 (0.003) | 0.945 (0.001)
ARF | 0.853 (0.009) | 0.700 (0.008) | 0.832 (0.003) | 0.836 (0.002) | 0.942 (0.001)
CTGAN | 0.842 (0.008) | 0.584 (0.006) | 0.741 (0.003) | 0.720 (0.002) | 0.912 (0.004)
TVAE | 0.828 (0.007) | 0.739 (0.010) | 0.805 (0.004) | 0.812 (0.007) | 0.877 (0.002)
TabDDPM | 0.757 (0.014) | 0.414 (0.015) | 0.535 (0.022) | 0.469 (0.021) | 0.823 (0.011)
LaTable | 0.874 (0.006) | 0.865 (0.009) | 0.900 (0.001) | 0.866 (0.004) | 0.900 (0.002)

It is insightful to more closely analyze the origin of LaTable’s gains against
the baselines. In Figure 2 we plot the metrics for each individual dataset and
as a function of the number of samples in each dataset. We observe that
LaTable performs more reliably for datasets with smaller sample size, and on
average better across the board. This can be understood as a consequence of
LaTable being trained on other datasets besides the small datasets. Hence,
this suggests that LaTable outperforms the baselines by transferring data
generation capabilities across tables. We note that despite hyperparameter
tuning attempts, TabDDPM performed poorly for small datasets.

(a) Density

(b) Coverage

Figure 2: LaTable outperforms baselines especially for smaller datasets.
Generation results for different generative models across 78 datasets (sorted
by size), with 5 seeds per (model, dataset) and a LOWESS curve fitted for
smoothing.

###  4.2 Zero- and few-shot out-of-domain generation

An important characteristic of foundation models is their ability to be
adapted to new datasets, preferably using relatively little data (i.e. few-
shot). In this section we thus aim to use the trained model $G$ on some target
datasets $D_{T}\in\mathcal{D}_{\mathrm{ood}}$ that it has not seen during
training, and see how it compares to baselines.

For LaTable, we would ideally want good zero-shot performance—i.e. to generate
new data based on the new data’s feature names and metadata alone. However, we
will show that this constitutes the limits of LaTable’s capabilities and
pretraining. A more realistic scenario is to adapt LaTable to the new datasets
through finetuning. To measure the effect of how much data is needed for
finetuning, we artificially reduce the target dataset to a smaller number of
samples. We compare to baselines that are trained on the target dataset alone.

In Figure 3 we plot the performance of LaTable and baselines as a function of
the number of target samples, which is averaged over 5 seeds and 5 different
datasets in $\mathcal{D}_{\mathrm{ood}}$. Baseline _Real_ is again the real
training data, but in contrast to last section, this dataset is now very
small.

As aforementioned, the zero-shot performances of LaTable does not match the
baselines—it has not seen enough data during training to generalize well. When
finetuned, however, LaTable exhibits a more promising behavior. On _Density_
(i.e. fidelity) it achieves an almost perfect score of 1, on par with real
data. Among the baselines, TVAE stands out. However, its density larger than 1
suggests that the model collapsed to a high-density region of the
distribution. When we look at _Coverage_ (i.e. diversity), LaTable appears
even stronger. We hypothesize that due to the pretraining and its metadata, it
is able to generate more diverse realistic samples in a few-shot setting. This
hypothesis is reinforced by noting that LaTable achieves significantly better
diversity than the real data—this is possible, considering the real training
data includes just a few samples.

(a) Density

(b) Coverage

Figure 3: A finetuned LaTable outperforms baselines significantly on out-of-
distribution datasets with few samples. Generation performance on out-of-
distribution datasets as a function of the training dataset size. Metrics are
averaged over 5 datasets in $\mathcal{D}_{\mathrm{ood}}$, where datasets are
reduced to $n_{samples}$. Mean and error bars (95%) are computed over 5 seeds.

##  5 Challenges to generalization

Let us explore possible directions for future research into better few- and
zero-shot LaTable.

Why LaTable achieves poor zero-shot performance. We cannot expect good zero-
shot performance of LaTable, as $\mathcal{D}_{\mathrm{ood}}$ is not covered by
$\mathcal{D}_{\mathrm{{id}}}$. In Appendix C.1 we analyse the features of
$\mathcal{D}_{\mathrm{id}}$ and $\mathcal{D}_{\mathrm{ood}}$. We observe that
most features in $\mathcal{D}_{\mathrm{ood}}$ do not have a close feature in
$\mathcal{D}_{\mathrm{id}}$. Similarly, even for the features that are
observed during training, there may be a significant shift in the feature
distribution—let alone, how it interacts (e.g. correlates) with other
features.

Improving out-of-distribution performance through scaling. The straightforward
solution to this problem is to increase the training dataset size—this has
been one of the key drivers behind text and vision FM progress [32, 33, 34,
35]. Multiple large tabular metadatasets exist: WebTables [36] consists of 10M
crawled HTML tables, WikiTables [37] includes 580K Wikipedia tables, and the
recent Tablib [28] includes 627M crawled tables with 827B context tokens.

Though these metadatasets are large, unfortunately they primarily include a
significantly different type of table. In Appendix C.2 we show they consist of
smaller tables, both in terms of features and samples (see Figure 7). We also
show this affects LaTable’s generation. WikiTables is curated, covers a wide
range of topics with context from the Wikipedia article, and is guaranteed to
not have overlap with $\mathcal{D}_{\mathrm{ood}}$ (vs Tablib or WebTables,
which may bias our results), hence let us consider this for training LaTable.
We discard any table with less than 20 samples, which yields 100k+ tables, and
repeat the experiment from Section 4.2 for a model trained on WikiTables.
Though we observe (Figure 8) comparable performance to the original LaTable
when finetuned, the zero-shot performance is significantly worse.

Going forward. Our findings emphasize the importance of good quality data in
improving generalization performance of LaTable. The issue of data quality may
be resolved by better understanding of large metadatasets like TabLib, and
using heavy-handed curation and selection. To avoid data leakage between the
training and evaluation sets, this also requires robust tools for identifying
and removing duplicates, which is hard for tabular datasets since preprocessed
and filtered copies are common. Another possible avenue for future work is to
use more specialized databases for training, e.g. of high-throughput
biological data [38].

##  6 Discussion

Limitations. LaTable can be extended and improved in multiple ways:

  1. 1.

Scale and data. As discussed in the previous section, LaTable is a relatively
small model trained on a relatively small dataset. Scaling up is one of the
key challenges for future research.

  2. 2.

Extending variable and table types. We restrict ourselves to numerical and
categorical data, but future work could consider datetime, full string
descriptions, time-series, and relational databases to increase applicability
and impact.

  3. 3.

Bias. In this work, we did not explore possible bias in the data and
pretrained LLM, or how LaTable copies it. More research into bias is required
before applying LaTable to the real-world, see broader impact below.

Broader impact. We believe the broader impact of LaTable, and continued work
in this area, is primarily positive. LaTable can enable better synthetic data
using fewer samples, which could promote more responsible AI—e.g. to improve
minority representation using data augmentation [39, 40, 41], ML model testing
through data simulation [41], and data democratization through private
synthetic data [42]. The latter is especially interesting, since few-shot
generation may require a smaller privacy budget than standard synthetic
data—after all the model need not learn the full distribution from the private
data and may be less likely to memorize this data [43].

Nonetheless, we need to acknowledge the risk: LaTable may make errors and may
reflect or exacerbate societal biases that are present in the data, or in the
pretrained LLM embeddings. More research into possible biases is required. The
quality and fairness of generated data should always be evaluated before
applying LaTable to real-world sensitive settings like healthcare and finance.

## References

  * van Breugel and van der Schaar [2024] Boris van Breugel and Mihaela van der Schaar.  Why Tabular Foundation Models Should Be a Research Priority.  In _International Conference on Machine Learning_ , May 2024.  doi: 10.48550/arXiv.2405.01147.  URL http://arxiv.org/abs/2405.01147.  arXiv:2405.01147 [cs].
  * Fatima and Pasha [2017] Meherwar Fatima and Maruf Pasha.  Survey of Machine Learning Algorithms for Disease Diagnostic.  _Journal of Intelligent Learning Systems and Applications_ , 09(01):1, 2017.  doi: 10.4236/jilsa.2017.91001.  URL http://www.scirp.org/journal/PaperInformation.aspx?PaperID=73781&#abstract.  Number: 01 Publisher: Scientific Research Publishing.
  * Borisov et al. [2022] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.  Publisher: IEEE.
  * Dastile et al. [2020] Xolani Dastile, Turgay Celik, and Moshe Potsane.  Statistical and machine learning models in credit scoring: A systematic literature survey.  _Applied Soft Computing_ , 91:106263, June 2020.  ISSN 1568-4946.  doi: 10.1016/j.asoc.2020.106263.  URL https://www.sciencedirect.com/science/article/pii/S1568494620302039.
  * Office for National Statistics [2021] Office for National Statistics.  2021 Census, 2021.  URL https://www.ons.gov.uk/census.
  * Kaggle [2017] Kaggle.  2017 Kaggle Machine Learning & Data Science Survey, 2017.  URL https://www.kaggle.com/datasets/kaggle/kaggle-survey-2017.
  * Huang et al. [2020] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin.  TabTransformer: Tabular Data Modeling Using Contextual Embeddings, December 2020.  URL https://arxiv.org/abs/2012.06678v1.  arXiv: 2012.06678.
  * Shwartz-Ziv and Armon [2022] Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  _Information Fusion_ , 81:84–90, May 2022.  ISSN 1566-2535.  doi: 10.1016/J.INFFUS.2021.11.011.  arXiv: 2106.03253 Publisher: Elsevier.
  * van Breugel and van der Schaar [2023] Boris van Breugel and Mihaela van der Schaar.  Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Data, April 2023.  URL http://arxiv.org/abs/2304.03722.  arXiv:2304.03722 [cs].
  * Borisov et al. [2023] Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci.  Language Models are Realistic Tabular Data Generators.  In _The Eleventh International Conference on Learning Representations_ , October 2023.  ISBN 2210.06280v2.  URL https://arxiv.org/abs/2210.06280v2.  arXiv: 2210.06280.
  * Choi et al. [2017] Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F Stewart, and Jimeng Sun.  Generating Multi-label Discrete Patient Records using Generative Adversarial Networks.  In _Machine Learning for Healthcare_ , March 2017.  doi: 10.48550/arxiv.1703.06490.  arXiv: 1703.06490.
  * Xu et al. [2019] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Modeling Tabular data using Conditional GAN.  _Advances in Neural Information Processing Systems_ , 32, July 2019.  ISSN 10495258.  URL https://arxiv.org/abs/1907.00503v2.  arXiv: 1907.00503 Publisher: Neural information processing systems foundation.
  * Watson et al. [2023] David S Watson, Kristin Blesch, Jan Kapar, and Marvin N Wright.  Adversarial Random Forests for Density Estimation and Generative Modeling.  In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics_ , volume 206, pages 5357–5375. PMLR, November 2023.  URL https://proceedings.mlr.press/v206/watson23a.html.  Series Title: Proceedings of Machine Learning Research.
  * Dieleman et al. [2022] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler.  Continuous diffusion for categorical data, November 2022.  URL https://arxiv.org/abs/2211.15089v3.  arXiv: 2211.15089.
  * Kotelnikov HSE et al. [2023] Akim Kotelnikov HSE, Dmitry Baranchuk, Ivan Rubachev HSE, and Yandex Artem Babenko Yandex.  TabDDPM: Modelling Tabular Data with Diffusion Models.  In _Proceedings of the 40th International Conference on Machine Learning_ , pages 17564–17579, September 2023.  doi: 10.48550/arxiv.2209.15421.  URL https://arxiv.org/abs/2209.15421v1.  arXiv: 2209.15421.
  * Zhang et al. [2024] Hengrui Zhang, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Xiao Qin, Christos Faloutsos, Huzefa Rangwala, and George Karypis.  Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=4Ay23yeuz0.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Solatorio and Dupriez [2023] Aivin V. Solatorio and Olivier Dupriez.  REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers, February 2023.  URL https://arxiv.org/abs/2302.02041v1.  arXiv: 2302.02041.
  * Padhi et al. [2021] Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, and Erik Altman.  Tabular Transformers for Modeling Multivariate Time Series.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 3565–3569, June 2021.  doi: 10.1109/ICASSP39728.2021.9414142.  URL https://ieeexplore.ieee.org/abstract/document/9414142?casa_token=FJ8ftI46EKUAAAAA:-ensrUQ1Jmtfpu1tAw2PsfLN3eR3sZOwV9dHgJ-5l-a-IjqZC0Vb_e5guzWKhri5Pic0xOIc8-A.  ISSN: 2379-190X.
  * Zhao et al. [2023] Zilong Zhao, Robert Birke, and Lydia Y Chen.  TabuLa: Harnessing Language Models for Tabular Data Synthesis.  _Proceedings of ACM Conference (Conference’17)_ , 1, October 2023.  URL http://arxiv.org/abs/2310.12746.  arXiv: 2310.12746.
  * Renda et al. [2023] Alex Renda, Aspen K Hopkins, and Michael Carbin.  Can LLMs Generate Random Numbers? Evaluating LLM Sampling in Controlled Domains.  In _ICML 2023 Workshop: Sampling and Optimization in Discrete Space_ , 2023.  URL http://people.csail.mit.edu/renda/llm-sampling-paper.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models.  _Advances in Neural Information Processing Systems_ , 2020-December, June 2020.  ISSN 10495258.  doi: 10.48550/arxiv.2006.11239.  arXiv: 2006.11239 Publisher: Neural information processing systems foundation.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative Modeling by Estimating Gradients of the Data Distribution.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising Diffusion Implicit Models.  In _International Conference on Learning Representations_ , October 2021.  ISBN 2010.02502v4.  URL https://arxiv.org/abs/2010.02502v4.  arXiv: 2010.02502.
  * Muennighoff et al. [2023] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers.  MTEB: Massive Text Embedding Benchmark, March 2023.  URL http://arxiv.org/abs/2210.07316.  arXiv:2210.07316 [cs].
  * Bischl et al. [2021] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G. Mantovani, Jan N. van Rijn, and Joaquin Vanschoren.  OpenML Benchmarking Suites, November 2021.  URL http://arxiv.org/abs/1708.03731.  arXiv:1708.03731 [cs, stat].
  * Fischer [2023] Sebastian Fischer.  OpenML-CTR23 – A curated tabular regression benchmarking suite.  In _AutoML Conference 2023 (Workshop)_ , 2023.
  * Eggert et al. [2023] Gus Eggert, Kevin Huo, Mike Biven, and Justin Waugh.  TabLib: A Dataset of 627M Tables with Context, October 2023.  URL https://arxiv.org/abs/2310.07875v1.  arXiv: 2310.07875 ISBN: 2310.07875v1.
  * Sajjadi et al. [2018] Mehdi S M Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly.  Assessing Generative Models via Precision and Recall.  In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi, and R Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Kynkäänniemi et al. [2019] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved Precision and Recall Metric for Assessing Generative Models.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.  URL https://github.com/kynkaat/improved-precision-and-recall-metric.
  * Naeem et al. [2020] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo.  Reliable Fidelity and Diversity Metrics for Generative Models.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119, pages 7176–7185. PMLR, June 2020.  URL http://proceedings.mlr.press/v119/naeem20a.html.  Series Title: Proceedings of Machine Learning Research.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners.  In _Advances in Neural Information Processing Systems_ , May 2020.  doi: 10.48550/arxiv.2005.14165.  arXiv: 2005.14165 Publisher: Neural information processing systems foundation.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling Laws for Neural Language Models, January 2020.  URL http://arxiv.org/abs/2001.08361.  arXiv:2001.08361 [cs, stat].
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, December 2022.  doi: 10.48550/arxiv.2112.10752.  arXiv: 2112.10752.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre.  An empirical analysis of compute-optimal large language model training.  _Advances in Neural Information Processing Systems_ , 35:30016–30030, December 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html.
  * Lehmberg et al. [2016] Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer.  A Large Public Corpus of Web Tables containing Time and Context Metadata.  In _Proceedings of the 25th International Conference Companion on World Wide Web - WWW ’16 Companion_ , pages 75–76, Montréal, Québec, Canada, 2016. ACM Press.  ISBN 978-1-4503-4144-8.  doi: 10.1145/2872518.2889386.  URL http://dl.acm.org/citation.cfm?doid=2872518.2889386.
  * Bhagavatula et al. [2015] Chandra Sekhar Bhagavatula, Thanapon Noraset, and Doug Downey.  TabEL: Entity Linking in Web Tables.  In Marcelo Arenas, Oscar Corcho, Elena Simperl, Markus Strohmaier, Mathieu d’Aquin, Kavitha Srinivas, Paul Groth, Michel Dumontier, Jeff Heflin, Krishnaprasad Thirunarayan, and Steffen Staab, editors, _International Semantic Web Conference_ , volume 9366, pages 425–441, Cham, 2015. Springer International Publishing.  doi: 10.1007/978-3-319-25007-6_25.  URL http://link.springer.com/10.1007/978-3-319-25007-6_25.  Book Title: The Semantic Web - ISWC 2015 Series Title: Lecture Notes in Computer Science.
  * Clough and Barrett [2016] Emily Clough and Tanya Barrett.  The Gene Expression Omnibus Database.  In Ewy Mathé and Sean Davis, editors, _Statistical Genomics: Methods and Protocols_ , Methods in Molecular Biology, pages 93–110. Springer, New York, NY, 2016.  ISBN 978-1-4939-3578-9.  doi: 10.1007/978-1-4939-3578-9_5.  URL https://doi.org/10.1007/978-1-4939-3578-9_5.
  * Chawla et al. [2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  SMOTE: Synthetic Minority Over-sampling Technique.  _Journal of Artificial Intelligence Research_ , 16:321–357, 2002.
  * Das et al. [2021] Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni-Vincentelli, and Costas J. Spanos.  Conditional Synthetic Data Generation for Robust Machine Learning Applications with Limited Pandemic Data.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(11):11792–11800, September 2021.  ISSN 2159-5399.  doi: 10.48550/arxiv.2109.06486.  URL https://arxiv.org/abs/2109.06486v1.  arXiv: 2109.06486 Publisher: Association for the Advancement of Artificial Intelligence (AAAI).
  * van Breugel et al. [2023a] Boris van Breugel, Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar.  Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data.  In _Advances in Neural Information Processing (NeurIPS 2023)_ , October 2023a.  arXiv: 2310.16524.
  * Ho et al. [2021] Stella Ho, Youyang Qu, Bruce Gu, Longxiang Gao, Jianxin Li, and Yong Xiang.  DP-GAN: Differentially private consecutive data publishing using generative adversarial nets.  _Journal of Network and Computer Applications_ , 185:103066, July 2021.  ISSN 1084-8045.  doi: 10.1016/J.JNCA.2021.103066.  Publisher: Academic Press.
  * van Breugel et al. [2023b] Boris van Breugel, Hao Sun, Zhaozhi Qian, and Mihaela van der Schaar.  Membership Inference Attacks against Synthetic Data through Overfitting Detection.  In _Proceedings of the 26th International Conference on Artificial Intelligence and Statistics_ , 2023b.
  * Rubin and Schenker [1991] Donald B. Rubin and Nathaniel Schenker.  Multiple imputation in health-care databases: an overview and some applications.  _Statistics in medicine_ , 10(4):585–598, 1991.  ISSN 0277-6715.  doi: 10.1002/SIM.4780100410.  Publisher: Stat Med.
  * Qian et al. [2023] Zhaozhi Qian, Rob Davies, and Mihaela van der Schaar.  Synthcity: a benchmark framework for diverse use cases of tabular synthetic data.  In _Advances in Neural Information Processing Systems_ , volume 36, January 2023.  doi: 10.48550/arxiv.2301.07573.  arXiv: 2301.07573.

##  Appendix A Broader impact of generative LTMs

The potential impact of a foundational tabular models is large. Generative
models require a lot of data to be trained, inhibiting their use in smaller
domains. On the other hand, if we would able to learn from multiple datasets,
combine knowledge, and translate knowledge from one dataset’s domain to
another, this could enable generating data and making predictions with less
data. We highlight some applications in Box A.

Box 1: Applications of generative Large Tabular Model (LTM). _(i)_ Because
LTMs have seen a lot of data already, their adaptation to new datasets through
transfer learning is likely to be more data-efficient (e.g. it can leverage
feature correlations that hold across datasets). _(ii)_ An LTM would enable to
generate data for small datasets or datasets with underrepresented groups, by
using the knowledge from other datasets. _(iii)_ Data augmentation using
knowledge from other datasets could also allow more powerful
science—implicitly leveraging data from other domains that are usually not
considered. _(iv)_ A conditional generative model (i.e. that allows generation
based on a subset of the variables), allows (probabilistic) zero or few-shot
prediction. These predictions can be more powerful than just training a
predictive model (because the LTM may have seen data from different, but
similar domains), and can give uncertainty estimates (because generative
models are probabilistic). _(v)_ Similarly, a conditional LTM permits
(multiple) imputation of missing data[44]—few-shot. _(vi)_ Because LTMs can be
trained on public data only and require fewer data point to adapt to a new
task, they can be used for private generation or prediction on new datasets
while requiring a smaller privacy budget—e.g. no need to use private data to
learn “global” correlations that can be learnt from other datasets.

##  Appendix B Implementation details

Encoding example. As mentioned in 3.3.2, we use text encoder WhereIsAI/UAE-
Large-V1 for encoding all string values. Let us consider an example:

Table 2: This describes the dataset. Age | Gender | Income
---|---|---
42 | Male | 30,000
33 | Female | 46,000

In this case, we encode and cache “This describes the dataset” (
$\bar{\mathbf{r}}$), feature names “Age”, “Gender”, and “Income”
($\bar{\mathbf{s}}_{j}$), and “Gender is Male” and “Gender is Female”.

###  B.1 Data

Data is loaded from OpenML dataset benchmarks [27, 26]. Datasets with more
than 500 columns are discarded as these correspond mostly to image datasets,
which results in 83 datasets. We take 78 datasets for training LaTable
($\mathcal{D}_{\mathrm{id}}$) and 5 datasets for testing out-of-distribution
performance ($\mathcal{D}_{\mathrm{ood}}$)—the latter includes OpenML datasets
with dataset IDs 31 (credit-g), 37 (diabetes), 44 (spambase), 1063 (kc2), and
1570 (adult). Additional train-val-test splits are created for each dataset
individually, and all training sets in $\mathcal{D}_{\mathrm{id}}$ are
combined for training LaTable. The number of samples and features per dataset
is visualized in Figure 4. Numerical features are normalized.

Figure 4: Number of samples per dataset plotted against number of features,
with in-distribution datasets $\mathcal{D}_{\mathrm{id}}$ in blue and
$\mathcal{D}_{\mathrm{ood}}$ in orange. Note: double log-scale.

###  B.2 Model and training

For LaTable, we use the architecture described in Section 3 and visualized in
Figure 1. All MLPs $g_{\cdot}$ consist of 3 layers, and the transformer
consists of 10 layers. All hidden layers are of dimension $1024$. We train and
finetune for 500 epochs using Adam, with a cosine scheduler and learning rate
$5\cdot 10^{-5}$. These values were chosen based on the in-distribution
validation loss and validation generation performance, and are kept the same
across all experiments. During training we weight points by their respective
dataset size to ensure small datasets are not ignored, with dataset $k$’s
weight $w^{(k)}\propto\frac{1}{\sqrt{|D^{(k)}|}}$. We use $T=200$ inference
steps for DDIM, and use the standard Diffusers implementation of DDIM. Note
that zero-shot generation, we _do_ assume we know the mean and standard
deviations of numerical variables (as we retain scaling) and we _do_ assume
which categories are allowed to be generated for each categorical feature
(i.e. $\mathbf{\hat{C}}$ is provided).

###  B.3 Baselines and metrics.

We use the Synthcity library [45] for baselines and metrics. Baselines use
standard recommended settings for all datasets. For TabDDPM, we use 1000
denoising steps. We have tried changing the MLP size, but this did not yield
reliably better results. To compute the metrics, we generate
$\min\\{|D_{\mathrm{test}}|,5000\\}$ samples. For the “Real” baseline in
Figure 3, we oversample the small real datasets to acquire the required number
of points for computing the metrics.

###  B.4 Compute

The model as described above can be trained on a 16GB GPU with batch size 32.
Training LaTable on all datasets in $\mathcal{D}_{\mathrm{id}}$ takes around 2
days on a single GPU. Finetuning takes about 3 seconds per sample (e.g. 5
minutes for 100 samples). Generation speed is around 30 samples per second.
For model development, we trained 33 models with different hyperparameters on
all of $\mathcal{D}_{id}$.

##  Appendix C Improving out-of-distribution generation (accompanying Section
5)

###  C.1 Comparing in- and out-of-distribution datasets

Though the number of features and samples in datasets is comparable across
$\mathcal{D}_{\mathrm{ood}}$ and $\mathcal{D}_{\mathrm{id}}$ (see Figure 4),
the small size of $\mathcal{D}_{\mathrm{id}}$ means it does not cover
$\mathcal{D}_{\mathrm{ood}}$’s features. We plot the cosine similarities
between all feature embeddings of $\mathcal{D}_{\mathrm{ood}}$ and
$\mathcal{D}_{\mathrm{id}}$ in Figure 5. Though there is some semantic overlap
in features, about three quarters of the features in
$\mathcal{D}_{\mathrm{ood}}$ have no similar feature in
$\mathcal{D}_{\mathrm{id}}$.

Even when similar feature names appear in both datasets, trivially these may
still pertain to vastly different features. Many datasets contain a column
“class”, yet these have vary different meaning and categories. Similarly,
marginals (and joint) distribution can be significantly shifted—see Fig 6

Figure 5: Features in $\mathcal{D}_{\mathrm{id}}$ do not cover all features in
$\mathcal{D}_{\mathrm{ood}}$. Cosine similarity matrix between feature
embeddings in $\mathcal{D}_{\mathrm{ood}}$ and $\mathcal{D}_{\mathrm{id}}$.
For 0.752 of the ood-features, all id-features are dissimilar (
similarity$<0.8$). Note: the three little diagonal lines refer to dataset “Kc2
Software Defects” (OpenML ID 1063) and 3 similar datasets that contain the
same features.

(a) Education

(b) Sex

(c) Age

Figure 6: Examples of feature “matches”. Distributions and categories can vary
significantly: in (a) annotation is different, (b) there is a significant
variability in female/male proportion over datasets, and in (c) there are
significant differences in distribution, with one of the datasets having a
vastly different scale (where age does not refer to human age).

###  C.2 Using other metadatasets

In Figure 7 we compare the number of features and samples in OpenML,
WikiTables, and Tablib. We observe that the latter two have significantly
different characteristics. This is to be expected—many online datasets consist
of informational tables (with few samples and features). TabLib consists of a
large number of datasets with many features. This is likely due to a good
amount of their data being sourced from Github, which may contain wide SQL and
other database structures, which do not resemble typical tabular ML tasks.

In Figure 8 we show that indeed, training on WikiTables does not yield better
out-of-distribution performance. Training on WikiTables for 100 epochs took 24
days on a single GPU.

Figure 7: Large online metadatasets have significantly different
characteristics than ML tabular datasets. Plotting different sources’ number
of samples and features per table. Note: double log-scale.

(a) Density

(b) Coverage

Figure 8: WikiTables training does not help out-of-distribution generation.
Repeating Section 4.2’s experiments for a model trained on WikiTables (vs.
OpenML as before). We observe no significant different in finetuned
performance, but significant degradation in zero-shot generation.