# Balanced Mixed-Type Tabular Data Synthesis with Diffusion Models

Zeyu Yang  Peikun Guo  Khadija Zanna  Akane Sano

###### Abstract

Diffusion models have emerged as a robust framework for various generative
tasks, such as image and audio synthesis, and have also demonstrated a
remarkable ability to generate mixed-type tabular data comprising both
continuous and discrete variables. However, current approaches to training
diffusion models on mixed-type tabular data tend to inherit the imbalanced
distributions of features present in the training dataset, which can result in
biased sampling. In this research, we introduce a fair diffusion model
designed to generate balanced data on sensitive attributes. We present
empirical evidence demonstrating that our method effectively mitigates the
class imbalance in training data while maintaining the quality of the
generated samples. Furthermore, we provide evidence that our approach
outperforms existing methods for synthesizing tabular data in terms of
performance and fairness.

Machine Learning, Diffusion Models

##  1 Introduction

Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) are a class of
energy-based generative models utilizing a stochastic process to transform
desired data samples from noise iteratively and have demonstrated remarkable
performance in generative modeling for various tasks, especially text-to-image
synthesis. Researchers use diffusion frameworks to model images paired with
text descriptions and show promising results (Nichol et al., 2021; Saharia et
al., 2022). Advancing further from diffusion models in pixel space, latent
diffusion frameworks optimize image representation in a low-dimensional space,
thereby enhancing synthetic image quality and efficiency (Rombach et al.,
2022; Ramesh et al., 2022). The impressive results of diffusion models in
image synthesis extended to the generation of sequential data such as audio
and time series (Kong et al., 2020; Rasul et al., 2021; Li et al., 2022). This
advancement has gradually overshadowed generative adversarial networks
(previous research spot in generative modeling), as diffusion models
consistently demonstrate superior performance across diverse tasks (Dhariwal &
Nichol, 2021; Stypułkowski et al., 2023; Mazé & Ahmed, 2023).

Researchers have recently been motivated by the versatility of diffusion
models to explore the potential in synthesizing tabular data (Kotelnikov et
al., 2023; He et al., 2023). Tabular data often involves sensitive user
information, which raises significant privacy concerns when used for model
training. Moreover, such datasets suffer from limitations in size and
imbalanced class distribution (Jesus et al., 2022), which leads to challenges
in training robust and fair machine learning models. To address these issues,
there is a growing demand for effective tabular data synthesis. However,
generative modeling of tabular data is challenging due to the combination of
discrete and continuous features and their varying value distributions.
Interpolation-based techniques, such as SMOTE (Chawla et al., 2002), are
simple but effective and challenging to beat. Researchers have used
adversarial generative networks and variational autoencoders to generate
tabular data with promising results (Ma et al., 2020; Xu et al., 2019).
Research on diffusion models for tabular data is ongoing, and the primary
challenge is to handle continuous and discrete features simultaneously. In
(Austin et al., 2021), the authors suggested using diffusion frameworks for
discrete features with Markov transition matrices. Recent developments such as
TabDDPM (Kotelnikov et al., 2023), which combines Markov transition functions
(called diffusion kernels) for continuous and discrete features, mark
significant progress in this field and underscore the potential of diffusion
models as a comprehensive solution for tabular data synthesis.

Despite the success of diffusion models in various generative tasks, ethical
concerns regarding potential bias and unsafe content in the synthetic data
persist. Generative models are inherently data-driven, thereby propagating
these biases into the synthetic data. Studies such as (Friedrich et al., 2023;
Schramowski et al., 2023) have shown that latent diffusion models,
particularly in text-to-image synthesis, can create images biased toward
certain privileged groups and may contain inappropriate elements, such as
nudity and violence. In response, they propose methods to apply hazardous or
sensitive guidance in latent space and then eliminate bias and offensive
elements in the synthetic images. In addition to text and images, tabular
datasets frequently exhibit notable bias, often stemming from issues related
to the disproportionate representation of sensitive groups. For instance, some
datasets exhibit significant imbalances in demographic groups regarding
sensitive attributes such as sex and race (Le Quy et al., 2022). However, in
our review of the literature, research on the safety and fairness of diffusion
models in areas beyond text-to-image synthesis is limited. Although diffusion
models have shown remarkable performance in tabular data synthesis (Sattarov
et al., 2023; He et al., 2023; Kotelnikov et al., 2023), they are prone to
learning and replicating these biases in training data. To address this issue,
we propose a novel diffusion framework to accurately model distributions of
mixed-type tabular data, with a specific focus on outcomes and sensitive
attributes. We employ balanced sampling techniques to reduce disparities in
group representation within the synthetic data. In summary, our main
contributions are as follows:

  1. 1.

We introduce a diffusion-based framework tailored to learn distributions of
mixed-type tabular data given multiple attributes.

  2. 2.

We generate tabular data that is balanced for a predefined set of sensitive
attributes, addressing inherent biases in the data.

  3. 3.

Our model surpasses existing models in specific areas of performance and
fairness, as evidenced through comprehensive evaluations.

The remainder of this paper is organized as follows: Section 2 provides an
overview of fairness-aware machine learning and diffusion models. Section 3
provides an overview of mixed-type modeling with diffusion models. In Section
4, we detail our approach to multivariate conditioning and balanced sampling.
In Section 5, we illustrate the experiments and present the results. Finally,
in Section 6, we conclude our findings and discuss their implications.

##  2 Related Work

The concept of bias in machine learning that involves assigning significance
to specific features to enhance overall generalization is essential for the
effective performance of models (Mehrabi et al., 2021). However, it is crucial
to acknowledge that bias in machine learning can also have negative
implications. Negative bias is an inaccurate assumption made by machine
learning algorithms that reflects systematic or historical prejudices against
certain groups of people (Zanna et al., 2022). Decisions derived from such
biased algorithms can lead to adverse effects, particularly impacting specific
social groups defined by factors such as sex, age, disability, race, and more.

This concept of bias and fairness in machine learning has been widely studied,
the aim being to mitigate algorithmic bias of sensitive attributes from
machine learning models (Mehrabi et al., 2021; Pessach & Shmueli, 2022). Many
methods for bias mitigation in classification tasks have been proposed, and
they can be classified into three categories: pre-processing, in-processing,
and post-processing methods. Pre-processing methods (Kamiran & Calders, 2012;
Calmon et al., 2017) transform the collected data to remove discrimination and
train machine learning models on discrimination-free datasets. In-processing
methods (Zhang et al., 2018; Agarwal et al., 2018; Kamishima et al., 2011;
Zafar et al., 2017; Kamishima et al., 2012) regulate machine learning
algorithms by incorporating fairness constraints or regularization terms into
the objective functions. Lastly, post-processing methods (Hardt et al., 2016;
Pleiss et al., 2017), implemented after training machine learning models on
collected datasets, directly override the predicted labels to improve
fairness. Fairness-aware generative models can be classified as pre-processing
methods. In one of such methods proposed to generate fair data, (Xu et al.,
2018) introduced fairness-aware adversarial generative networks that employ a
fair discriminator to maintain equality in the joint probability of features
and labels conditioned on subgroups within sensitive attributes. However, this
approach does not effectively tackle the imbalance in sensitive classes within
synthetic data, and diffusion-based tabular data synthesis methods such as
TabDDPM (Kotelnikov et al., 2023) excel in producing synthetic data of
superior quality.

With impressive capabilities in generative modeling, diffusion models are a
class of deep generative models that utilize a Markov process that starts from
noise and ends with the target data distribution. Although diffusion models
are widely studied, the research on fairness and safety of diffusion models is
limited, and existing works (Schramowski et al., 2023; Friedrich et al., 2023)
mainly focus on text-to-image synthesis tasks. Researchers in (Friedrich et
al., 2023) utilize sensitive content, such as sex and race, within text
prompts to guide training diffusion models. Subsequently, during the sampling
phase, a uniform distribution of sensitive attributes is applied to generate
images that are fair and do not exhibit preference towards privileged groups.
Similarly, (Schramowski et al., 2023) uses unsafe text, such as nudity and
violence, to guide diffusion models and remove unsafe content in the sampling
phase. In addition, this method applies some tricks to remove unsafe content
from synthetic images while keeping changes minimal. However, these fairness-
aware diffusion models have not been adapted to tabular data synthesis, even
though bias commonly exists in tabular datasets (Le Quy et al., 2022). This
paper studies how to model mixed-type tabular data conditioning on labels and
multiple sensitive attributes in latent space. Our method can generate
balanced tabular data considering multiple sensitive attributes and
subsequently achieve improved fairness scores without compromising the quality
of the synthetic data.

##  3 Background

Diffusion models, taking inspiration from thermodynamics, are probabilistic
generative models that operate under the Markov assumption. Diffusion models
involve two major components: a forward process and a reverse process. The
forward process gradually transforms the given data $\mathbf{x}_{0}$ into
noise $\mathbf{x}_{T}$ passing through $T$ steps of the diffusion kernel
$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$. In the reverse process, $\mathbf{x}_{T}$
is restored to the original data $\mathbf{x}_{0}$ by $T$ steps of posterior
estimator $p_{\mathbf{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ with
trainable parameters $\mathbf{\theta}$. In this section, we explain the
Gaussian diffusion kernel for continuous features and the multinomial
diffusion kernel for discrete features. Furthermore, we show how to optimize
the parameters $\mathbf{\theta}$ of the posterior estimator considering mixed-
type data with both continuous and discrete features.

###  3.1 Gaussian Diffusion Kernel

The Gaussian diffusion kernel is used in the forward process for continuous
features. After a series of Gaussian diffusion kernels with $T$ timesteps,
$\mathbf{x}_{T}$ follows the standard Gaussian distribution
$\mathcal{N}(\mathbf{0},\mathbf{I})$. Specifically, with the values of
$\beta_{t}$ predefined according to a schedule such as linear, cosine, etc.
(Chen, 2023), the Gaussian diffusion kernel
$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$ can be written as:

$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t}|\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$

With the starting point $\mathbf{x}_{0}$ known, the posterior distribution in
the forward process can be calculated analytically as follows,

$q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})=\mathcal{N}(\mathbf{x}_{t-1}|\tilde{\mathbf{\mu}}_{t}(\mathbf{x}_{t},\mathbf{x}_{0}),\tilde{\beta}_{t}\mathbf{I})$
$\tilde{\mathbf{\mu}}_{t}(\mathbf{x}_{t},\mathbf{x}_{0})=\frac{1}{\sqrt{\alpha_{t}}}(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\mathbf{\epsilon})\\\
$ (1)
$\tilde{\beta}_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$

where $\alpha_{t}=1-\beta_{t}$, $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$.
In the reverse process, a posterior estimator with parameters
$\mathbf{\theta}$ is used to approximate
$p_{\mathbf{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ at each timestep $t$
given $\mathbf{x}_{t}$.

$p_{\mathbf{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1}|\hat{\mathbf{\mu}}(\mathbf{x}_{t}),\hat{\mathbf{\Sigma}}(\mathbf{x}_{t}))$

Researchers in (Ho et al., 2020) set the estimated covariance matrix
$\hat{\mathbf{\Sigma}}(\mathbf{x}_{t})$ as diagonal matrix with constant
values $\beta_{t}$ or $\tilde{\beta}_{t}$, and calculate the estimated mean
$\hat{\mathbf{\mu}}(\mathbf{x}_{t})$ as follows,

$\hat{\mathbf{\mu}}(\mathbf{x}_{t})=\frac{1}{\sqrt{\alpha_{t}}}(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\hat{\mathbf{\epsilon}}(\mathbf{x}_{t}))$
(2)

where $\hat{\mathbf{\epsilon}}(\mathbf{x}_{t})$ is the estimated noise at time
$t$.

###  3.2 Multinomial Diffusion Kernel

The multinomial diffusion kernel is applied in the forward process for
discrete features with the same noise schedule $\beta_{t}$ used by the
Gaussian diffusion kernel. For discrete features, in the final timestep $T$,
$\mathbf{x}_{T}$ follows the discrete uniform distribution
$\operatorname{Uniform}(K)$, where $K$ is the number of categories. The
multinomial diffusion kernel can be written as:

$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\operatorname{Cat}(\mathbf{x}_{t}|(1-\beta_{t})\mathbf{x}_{t-1}+\beta_{t}/K)$

Similarly to the continuous case, the posterior distribution of the
multinomial diffusion kernel has an analytical solution.

$q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})=\mathrm{Cat}(\mathbf{x}_{t-1}|\tilde{\mathbf{\theta}}/\sum_{k=1}^{K}\tilde{\theta}_{k})$
$\tilde{\mathbf{\theta}}=[\alpha_{t}\mathbf{x}_{t}+(1-\alpha_{t})/K]\odot[\bar{\alpha}_{t-1}\mathbf{x}_{0}+(1-\bar{\alpha}_{t-1})/K]$

The posterior estimator for discrete features is preferably parameterized by
the estimated starting point $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})$ as
suggested by (Hoogeboom et al., 2021).

$p_{\mathbf{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\hat{\mathbf{x}}_{0}(\mathbf{x}_{t}))$
(3)

###  3.3 Model Fitting

The parameters ${\mathbf{\theta}}$ of the posterior estimator are learned by
minimizing the variational lower bound,

$\displaystyle\mathcal{L}(\mathbf{x}_{0})=\mathbb{E}_{q(\mathbf{x}_{0})}[D(q(\mathbf{x}_{T}|\mathbf{x}_{0})\|p(\mathbf{x}_{T}))-\log
p_{\mathbf{\theta}}(\mathbf{x}_{0}|\mathbf{x}_{1})$ (4)
$\displaystyle+\sum_{t=2}^{T}D(q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})\|p_{\mathbf{\theta}}(\mathbf{x}_{t-1}|\mathbf{x}_{t}))]$

where Kullback–Leibler divergence is denoted by $D$. The variational lower
bound can be simplified for continuous variables as the mean squared error
between true noise $\mathbf{\epsilon}$ in Equation 1 and estimated noise
$\hat{\mathbf{\epsilon}}$ in Equation 2, denoted by $\mathcal{L}_{\text{G}}$.

$\mathcal{L}_{\text{G}}=\mathbb{E}_{q(\mathbf{x}_{0})}[\|\mathbf{\epsilon}-\hat{\mathbf{\epsilon}}({\mathbf{x}_{t}})\|^{2}]$

For mixed-type data containing continuous features and $C$ categorical
features, the total loss $\mathcal{L}_{\text{T}}$ can be expressed as the
summation of the Gaussian diffusion loss term $\mathcal{L}_{\text{G}}$ and the
average of the multinomial diffusion loss terms in Equation 4 for all
categorical features:

$\mathcal{L}_{\text{T}}=\mathcal{L}_{\text{G}}+\frac{\sum_{i\leq
C}\mathcal{L}^{(i)}}{C}$

To train latent diffusion models, let $\mathbf{z}_{t}$ be the latent
representation of $\mathbf{x}_{t}$, the estimated noise
$\hat{\mathbf{\epsilon}}(\mathbf{x}_{t})$ for continuous features in Equation
2 can be reformulated as $\hat{\mathbf{\epsilon}}(\mathbf{z}_{t})$. The
estimated original input $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})$ in Equation 3
can be reformulated as $\hat{\mathbf{x}}_{0}(\mathbf{z}_{t})$.

##  4 Methods

As explained in Section 1, generative diffusion models for tabular data are
prone to learning the sensitive group size disparity in training data. In this
section, we describe the process of synthesizing balanced mixed-type tabular
data considering sensitive attributes. We first introduce conditional
generation given multivariate guidance while preserving the synthesis quality.
Then, we explain the architecture of the backbone deep neural network used as
the posterior estimator in our method. Lastly, we explain the procedure for
sampling balanced data with consideration for sensitive attributes. The high-
level structure of our method is shown in Figure 1.

Figure 1: The diagram of our model architecture. In the forward process, the
input data point $\mathbf{x}$ is pre-processed into numerical part
$\mathbf{x}_{\text{num}}$ and categorical part $\mathbf{x}_{\text{cat}}$, and
then passing through $T$ steps of the diffusion kernel to get
$\mathbf{x}_{1},\cdots,\mathbf{x}_{T}$. $\mathbf{z}_{1},\cdots,\mathbf{z}_{T}$
is the latent representation of $\mathbf{x}_{1},\cdots,\mathbf{x}_{T}$. In the
reverse process, the posterior estimator iteratively denoises noisy input
$\mathbf{z}_{T}$ conditioning on an outcome $\mathbf{c}$ and $N$ sensitive
attributes $\mathbf{s}^{(1)},\cdots,\mathbf{s}^{(N)}$. The estimated data
point is $\hat{\mathbf{x}}$.

###  4.1 Multivariate Latent Guidance

Conditional data generation is important in many tasks like text-to-image
synthesis. To achieve conditional data generation with diffusion models,
classifier-free guidance (Ho & Salimans, 2022) is a simple but effective
approach. Intuitively, the classifier-free guidance method tries to let the
posterior estimator “know” the label of the data it models. In latent space,
it uses one neural network to receive $\mathbf{z}_{t}$ and condition
$\mathbf{c}$ to make estimations. We can denote the estimated noise given
corresponding condition $\mathbf{c}$ for continuous features as
$\bar{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})$,

$\bar{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})=\hat{\mathbf{\epsilon}}(\mathbf{z}_{t})+w_{g}(\underbrace{\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})-\hat{\mathbf{\epsilon}}(\mathbf{z}_{t})}_{\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c})})$
(5)

where
$\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c})=\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})-\hat{\mathbf{\epsilon}}(\mathbf{z}_{t})$
is guidance of $\mathbf{c}$ and it is scaled by guidance weight $w_{g}$.

Equation 5 can be extended to support multivariate guidance. Especially in
tabular synthesis, we want to generate samples conditioned on labels
$\mathbf{c}$ and sensitive features $\mathbf{s}$. In this case, we can let our
neural network take $\mathbf{z}_{t}$, $\mathbf{c}$, and $\mathbf{s}$ as
inputs, resulting in:

$\bar{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c},\mathbf{s})=\hat{\mathbf{\epsilon}}(\mathbf{z}_{t})+w_{g}(\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c})+\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{s}))$
(6)
$\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{s})=\mathbf{\mu}(\mathbf{c},\mathbf{s};w_{s},\lambda)(\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{s})-\hat{\mathbf{\epsilon}}(\mathbf{z}_{t}))$

The sensitive guidance $\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{s})$
is controlled element-wisely by a “security gate”
$\mathbf{\mu}(\mathbf{c},\mathbf{s};w_{s},\lambda)$,

$\mathbf{\mu}(\mathbf{c},\mathbf{s};w_{s},\lambda)=\begin{cases}\phi,&\text{where
}\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})\ominus\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{s})<\lambda\\\
0,&\text{otherwise }\end{cases}$
$\phi=\max(1,w_{s}|\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c})-\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{s})|)$

where $w_{s}$ is the sensitive guidance weight. The purpose of
$\mathbf{\mu}(\mathbf{c},\mathbf{s};w_{s},\lambda)$ is to keep the alterations
caused by the sensitive guidance steady and minimal. It deactivates the
sensitive guidance when the difference between the estimation conditioned on
$\mathbf{c}$ and the estimation conditioned on $\mathbf{s}$ is larger than a
threshold $\lambda$. Furthermore, sensitive guidance is deactivated in the
early phase of the diffusion process when $t$ is smaller than a warm-up
timestep $\delta$ (a positive integer indicating the timestep at which
sensitive guidance starts):

$\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{s}):=\mathbf{0}\text{ if
}t<\delta$

Additionally, a momentum term with momentum weight $w_{m}$ is added to the
sensitivity guidance to make guidance direction stable,

$\mathbf{\gamma}_{t}(\mathbf{z}_{t},\mathbf{c},\mathbf{s})=\mathbf{\mu}(\mathbf{c},\mathbf{s};w_{s},\lambda)(\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{s})-\hat{\mathbf{\epsilon}}(\mathbf{z}_{t}))+w_{m}\mathbf{\nu}_{t}$
$\mathbf{\nu}_{t+1}=\beta\mathbf{\nu}_{t}+(1-\beta)\mathbf{\gamma}_{t}$

where $\mathbf{\nu}_{0}=\mathbf{0}$ and $\beta\in[0,1]$. To include not only
one sensitive feature, but $N$ sensitive features
$\mathbf{S}=\\{\mathbf{s}^{(1)},\cdots,\mathbf{s}^{(N)}\\}$, we can extend the
sensitive guidance as follows:

$\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{S})=\sum_{i=1}^{N}\mathbf{\mu}(\mathbf{c},\mathbf{s}^{(i)};w_{s},\lambda)(\hat{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{s}^{(i)})-\hat{\mathbf{\epsilon}}(\mathbf{z}_{t}))$

In this way, we can model continuous features given labels $\mathbf{c}$ and a
set of sensitive features $\mathbf{S}$. Similarly, the estimated starting
point given $\mathbf{c}$ and $\mathbf{S}$ for discrete features can be
formulated as follows,

$\bar{\mathbf{x}}_{0}(\mathbf{z}_{t},\mathbf{c},\mathbf{S})=\hat{\mathbf{x}}_{0}(\mathbf{z}_{t})+w_{g}(\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c})+\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{S}))$
(7)

with $\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c})$ and
$\mathbf{\gamma}(\mathbf{z}_{t},\mathbf{c},\mathbf{S})$ parameterized by
$\hat{\mathbf{x}}_{0}$.

###  4.2 Backbone

In this research, we model mixed-type tabular data in latent space following a
similar setup in (Rombach et al., 2022). The tabular data are encoded into
latent space and decoded back with multilayer perceptrons. We choose U-Net
(Ronneberger et al., 2015) with transformers as a posterior estimator to
predict $\bar{\mathbf{\epsilon}}(\mathbf{z}_{t},\mathbf{c},\mathbf{S})$ and
$\bar{\mathbf{x}}_{0}(\mathbf{z}_{t},\mathbf{c},\mathbf{S})$. This choice is
based on the hypothesis that the U-Net, augmented by the contextual
understanding capabilities of transformers, can effectively manage the
heterogeneous nature of tabular data. This integration aims to leverage the
U-Net’s ability to capture spatial correlation and the transformers’ strengths
in sequence modeling, thereby offering a novel approach to latent space
modeling of tabular data.

###  4.3 Balanced Sampling

After training diffusion models on tabular data paired with labels
$\mathbf{c}$ and a set of sensitive features $\mathbf{S}$, we apply
multivariate latent guidance for conditional tabular data synthesis. We
customize the marginal distributions of the provided labels and sensitive
features in the sampling phase. Fair synthetic tabular data are typically
anticipated to exhibit the same label distribution as the real data. These
data are expected to mirror the label distribution of the real data and
exhibit a balanced distribution of sensitive features, which addresses the
common issue of label imbalance in real-world datasets. To achieve this, we
can compute the empirical distribution of the labels in the actual data and
use it together with uniform categorical distributions for each sensitive
feature to guide the sampling process. By doing this, we generate balanced
synthetic data regarding the sensitive features, and the synthetic data have
the same label distribution as the actual data.

##  5 Experiments

We first introduce experiment datasets, data processing, baselines, and how we
assess synthetic tabular data. Then we present computational results on
experimental datasets.

###  5.1 Datasets

We evaluate the effectiveness and fairness of our proposed method using seven
tabular datasets for classification tasks. These datasets contain numerical
and categorical features, including some sensitive attributes with observed
class imbalances. We provide a summary of the datasets, including their
corresponding sensitive attributes in Table 1, and offer further details, such
as data sources, sensitive class label distributions, etc., in Appendix A.
Common sensitive attributes such as sex and race can be found in data sets
like UCI Adult, Cardio, Credit Card, Depression, KDD Census, and Law School.
Additionally, other sensitive attributes such as marital status and education
are present in the Bank Marketing and Credit Card datasets. Each data set is
divided into training, validation, and test sets using a 50/25/25 split.
Following the training of our diffusion models with multivariate guidance, we
proceed to generate synthetic data that adheres to the original label
distribution while incorporating a uniform distribution for sensitive
features.

Table 1: Experimental datasets Abbreviation | Dataset | Size | Sensitive | Target
---|---|---|---|---
AD | UCI Adult | 48842 | Sex, Race | Income
BA | Bank Marketing | 45221 | Marital | Subscription
CA | Cardio | 70000 | Sex | Diagnosis
CR | Credit Card | 30000 | Sex, Education | Payment
DE | Depression | 11000 | Sex, Race | Diagnosis
KD | KDD Census | 199523 | Sex, Race | Income
LA | Law School | 20800 | Sex, Race | Admission

###  5.2 Data Processing and Baselines

Data Processing. Following the data processing steps in (Kotelnikov et al.,
2023), we split the numerical and categorical features for each dataset and
pre-process them separately. Numerical features are converted through quantile
transformation, a non-parametric technique transforming variables into a
standard Gaussian distribution based on quantiles, thus normalizing the data
and enhancing the robustness to outliers. In preparation for the Markov
transition process in the latent diffusion model, we encode categorical
features into one-hot vectors and concatenate the resulting vectors with
normalized numerical features.

Baselines. In assessing the efficacy of our approach for various datasets, we
conduct a comparative analysis against several baselines for tabular data
synthesis. Within learning-based methods, we categorize them into diffusion
models, generative adversarial networks, and variational autoencoders,
selecting the top-performing model from each category with publicly available
code. In addition to learning-based techniques, we incorporate SMOTE, a robust
and traditional method for tabular data synthesis, into our set of baselines.
Specifically, the selected baseline methods for comparison are described in
detail as follows.

  * •

TabDDPM (Kotelnikov et al., 2023): a recent diffusion-based model that has
demonstrated to surpass existing methods for synthesizing tabular data.
Leveraging multilayer perceptrons as its backbone, TabDDPM excels in learning
unbalanced distributions present in the training data.

  * •

CTGAN (Xu et al., 2019): a tabular data synthesis approach based on generative
adversarial networks, providing publicly available code that is accessible and
user-friendly in (Patki et al., 2016).

  * •

TVAE (Xu et al., 2019): a state-of-the-art variational autoencoder
specifically designed to generate tabular data with a user-friendly
implementation in (Patki et al., 2016).

  * •

SMOTE (Chawla et al., 2002): a robust interpolation-based method that creates
synthetic data points by combining a real data point with its $k$ nearest
neighbors from the dataset. In our application, we leverage SMOTE to generate
additional tabular data samples, employing interpolation among existing
samples with the same label. The implementation of SMOTE is available in
(Lemaître et al., 2017).

###  5.3 Assessing Synthetic Tabular Data

There are two primary approaches to evaluating synthetic tabular data. One
approach is a model-based method that involves training machine learning
models on synthetic tabular data and evaluating the trained models on
validation data extracted from the original dataset. The other approach is a
data-based method that operates independently of machine learning models and
directly compares synthetic and real tabular data.

Model-Based Evaluation. The computation procedure of model-based evaluation of
synthetic tabular data in classification tasks involves the following steps.

  1. 1.

Train a generative model on the original training set.

  2. 2.

Generate synthetic data with the generative model.

  3. 3.

Train classifiers on the synthetic data.

  4. 4.

Evaluate classifiers on the original validation data.

Specifically, when assessing model accuracy on the original validation data,
it is referred to as machine learning efficiency as used in (Choi et al.,
2017). Beyond accuracy assessments, the evaluation of synthetic data can
extend to fairness scores of classifiers, providing insights into the fairness
of the generated data. We evaluate the fairness of machine learning models
trained on the synthetic data, focusing on two key aspects: equal allocation
and equal performance (Agarwal et al., 2018).

  * •

Equal allocation, a fundamental principle in model-based fairness evaluations,
entails the idea that a model should distribute resources or opportunities
proportionally among different groups, irrespective of their affiliation with
any privileged group. Equal allocation is quantified using the demographic
parity ratio. This ratio reveals the extent of the imbalance by comparing the
lowest and highest selection rates (the proportion of examples predicted as
positive) between groups.

  * •

Equal performance is grounded in the principle that a fair model should
exhibit consistent performance across all groups. This entails maintaining the
same precision level for each group. The equalized odds ratio, which
represents the smaller of two ratios comparing true and false positive rates
between groups, serves as a metric for assessing performance fairness in this
context.

The evaluation of the demographic parity ratio and equalized odds ratio
depends on the selection of sensitive features. When evaluating fairness
scores in our experiments, we choose the first sensitive feature specified in
Table 1. Both the demographic parity ratio and equalized odds ratio are within
the range $[0,1]$. The demographic parity ratio highlights the importance of
distributing resources equally among different groups, whereas the equalized
odds ratio prioritizes impartial decision-making within each specific group.
Higher values in either metric indicate progress toward fairness. However,
attaining perfect fairness, represented by a score of $1$, may not always be
feasible or appropriate depending on the specific context. Our proposed method
is expected to have enhanced fairness scores compared to baseline models, and
we interpret the results using SHAP feature importance (Lundberg & Lee, 2017).

Data-Based Evaluation. The data-based evaluation directly compares synthetic
and real tabular data. To address the privacy concerns when sharing tabular
data publicly, the synthetic tabular data is expected not to be a “copy” of
the original data. The originality of synthetic data is quantified by
utilizing a nearest-neighbor distance metric suggested by (Park et al., 2018),
known as the “distance to the closest record” (DCR), calculated as the
Euclidean distance between each synthetic sample and its nearest real data
point. Larger DCR values indicate that the synthetic data has a greater
dissimilarity to the original data. In addition, when evaluating the fairness
of synthetic tabular data with a data-based method, we examine the imbalance
in class characteristics within sensitive features such as sex or race.

###  5.4 Computational Results

This section presents the computational results of our evaluation of synthetic
tabular data generated with five different random seeds. To ensure the
robustness of our model-based evaluations, we employed a diverse set of
classic classifiers, including decision trees, random forests, support vector
machines, multilayer perceptrons, and XGBoost. We determined the optimal model
for each synthetic dataset through meticulous training and evaluation across
separate training and validation sets, utilizing five random seeds for each
model. The averaged validation accuracy, along with the standard deviation, as
shown in Table 2, revealed XGBoost as the dominant performer across most
datasets, with random forest excelling in one exceptional case (see fairness
scores in Section A.3).

Table 2: The best and average values of validation accuracy on experimental datasets (the averaged validation accuracy $\pm$ the standard deviation) Dataset | Model Choice | Best | Average
---|---|---|---
AD | XGBoost | $0.866_{\pm 0.000}$ | $0.795_{\pm 0.041}$
BA | XGBoost | ${0.900_{\pm 0.000}}$ | $0.869_{\pm 0.031}$
CA | XGBoost | ${0.733_{\pm 0.000}}$ | $0.658_{\pm 0.015}$
CR | Random Forest | ${0.816_{\pm 0.000}}$ | $0.757_{\pm 0.018}$
DE | XGBoost | ${0.941_{\pm 0.000}}$ | $0.929_{\pm 0.001}$
KD | XGBoost | ${0.956_{\pm 0.000}}$ | $0.946_{\pm 0.000}$
LA | XGBoost | ${1.000_{\pm 0.000}}$ | $0.942_{\pm 0.004}$

Performance. In our exploration of the machine learning efficiency of
synthetic data compared to baselines, we employ two rigorous protocols. The
first protocol calculates the average machine learning efficiency across all
classifiers, while the second identifies and utilizes the best classifier for
each dataset. This multifaceted approach ensures a robust and insightful
evaluation. As illustrated in Table 3, our proposed method maintains
competitiveness in terms of average machine learning efficiency, even when
incorporating a uniformly distributed direction for sensitive features during
the sampling phase. It generally outperforms CTGAN and TVAE. Moreover, our
method excels against TabDDPM on the majority of datasets. While the
interpolation-based method, SMOTE, proves highly competitive and challenging
to outperform, our model achieves a machine learning efficiency very close to
SMOTE, underscoring its effectiveness. The best machine learning efficiency
exhibits a similar pattern with the average machine learning efficiency, with
SMOTE and our model emerging as the two most competitive methods.

Table 3: The values of machine learning efficiency averaged on all types of
classifiers Average Accuracy
---
| AD | BA | CA | CR | DE | KD | LA
TabDDPM | $0.777_{\pm 0.093}$ | $0.874_{\pm 0.017}$ | $0.640_{\pm 0.025}$ | $0.768_{\pm 0.032}$ | $0.920_{\pm 0.001}$ | $0.937_{\pm 0.001}$ | $0.905_{\pm 0.012}$
CTGAN | $0.742_{\pm 0.148}$ | $0.873_{\pm 0.019}$ | $0.600_{\pm 0.061}$ | $0.746_{\pm 0.058}$ | $0.867_{\pm 0.050}$ | $0.931_{\pm 0.014}$ | $0.882_{\pm 0.112}$
TVAE | $0.786_{\pm 0.023}$ | $0.854_{\pm 0.026}$ | $0.613_{\pm 0.058}$ | $0.784_{\pm 0.021}$ | $0.881_{\pm 0.051}$ | $0.938_{\pm 0.008}$ | $0.919_{\pm 0.119}$
SMOTE | $0.807_{\pm 0.038}$ | $0.880_{\pm 0.022}$ | $0.632_{\pm 0.074}$ | $0.770_{\pm 0.044}$ | $0.928_{\pm 0.022}$ | $0.939_{\pm 0.013}$ | $0.930_{\pm 0.124}$
Ours | $0.803_{\pm 0.022}$ | $0.880_{\pm 0.011}$ | $0.629_{\pm 0.001}$ | $0.734_{\pm 0.045}$ | $0.926_{\pm 0.004}$ | $0.932_{\pm 0.004}$ | $0.921_{\pm 0.003}$
Best Accuracy
| AD | BA | CA | CR | DE | KD | LA
TabDDPM | $0.834_{\pm 0.004}$ | $0.895_{\pm 0.003}$ | $0.708_{\pm 0.003}$ | $0.816_{\pm 0.002}$ | $0.937_{\pm 0.003}$ | $0.945_{\pm 0.001}$ | $0.964_{\pm 0.002}$
CTGAN | $0.817_{\pm 0.002}$ | ${0.888}_{\pm 0.002}$ | ${0.652}_{\pm 0.006}$ | ${0.716}_{\pm 0.009}$ | ${0.870}_{\pm 0.008}$ | ${0.941}_{\pm 0.001}$ | ${0.975}_{\pm 0.002}$
TVAE | ${0.810}_{\pm 0.004}$ | ${0.849}_{\pm 0.008}$ | ${0.662}_{\pm 0.006}$ | ${0.773}_{\pm 0.006}$ | ${0.900}_{\pm 0.003}$ | ${0.944}_{\pm 0.001}$ | ${0.993}_{\pm 0.001}$
SMOTE | $0.853_{\pm 0.000}$ | $0.898_{\pm 0.000}$ | $0.704_{\pm 0.000}$ | $0.817_{\pm 0.001}$ | $0.941_{\pm 0.000}$ | $0.949_{\pm 0.000}$ | $1.000_{\pm 0.000}$
Ours | $0.838_{\pm 0.001}$ | $0.892_{\pm 0.003}$ | $0.709_{\pm 0.004}$ | $0.806_{\pm 0.004}$ | $0.942_{\pm 0.003}$ | $0.930_{\pm 0.003}$ | $0.995_{\pm 0.004}$

Privacy. As explained in Section 5.3, we employ DCR values to assess the
privacy score of the synthetic tabular data. The median DCR values, obtained
across five random seeds, are detailed in Table 4. Remarkably, our method
consistently attains a superior privacy score in comparison to both SMOTE and
TabDDPM. This suggests that our approach excels in maintaining the
distinctiveness and privacy of the synthetic data.

Table 4: The median value of distance to the closest record | AD | BA | CA | CR | DE | KD | LA
---|---|---|---|---|---|---|---
TabDDPM | $0.179$ | $0.195$ | $0.021$ | $0.123$ | $0.520$ | $2.089$ | $0.176$
CTGAN | $0.340$ | $0.303$ | $0.028$ | $0.202$ | $0.428$ | $2.451$ | $0.276$
TVAE | $0.253$ | $1.415$ | $0.032$ | $0.208$ | $0.443$ | $2.018$ | $1.414$
SMOTE | $0.170$ | $0.191$ | $0.021$ | $0.117$ | $0.398$ | $2.005$ | $0.178$
Ours | $0.511$ | $0.229$ | $0.023$ | $0.255$ | $0.430$ | $2.831$ | $0.216$

Fairness. A key contribution of our work is the generation of balanced tabular
data while considering sensitive features. To substantiate our claim and
provide empirical evidence, we conduct a thorough assessment of the fairness
of our proposed model. After choosing the best-performing classifier for each
dataset, we assess its fairness by evaluating two metrics: demographic parity
ratio and equalized odds ratio, and the results are presented in Table 5. Our
approach outperforms baseline methods notably in terms of equal allocation, as
evidenced by higher values of demographic parity ratio across nearly all
datasets. For instance, on the KDD Census dataset, our method achieves a
demographic parity ratio of $0.613$, surpassing baseline methods like SMOTE
and TabDDPM by around fifty percent, while SMOTE only attains $0.141$, and
TabDDPM reaches $0.097$. Similarly, on the UCI Adult dataset, our model
demonstrates a demographic parity ratio of $0.529$, surpassing the ratios of
$0.306$ by SMOTE and $0.312$ by TabDDPM. In addition, our model demonstrates
competitiveness in terms of equal performance metrics, as indicated by the
equalized odds ratio. Particularly noteworthy are datasets such as UCI Adult
and KDD Census, where our model achieves an equalized odds ratio of $0.641$
and $0.884$, respectively, surpassing baseline models that register less than
$0.250$ in these metrics. Furthermore, employing equitable synthetic data to
train classifiers can yield higher fairness scores compared to training with
the original data, as illustrated in Table 7. In contrast, baseline models
generally achieve comparable fairness scores.

Table 5: The values of demographic parity ratio and equalized odds ratio with
the best-performing classifier Demographic Parity Ratio
---
| AD | BA | CA | CR | DE | KD | LA
TabDDPM | $0.312_{\pm 0.033}$ | $0.590_{\pm 0.070}$ | $0.957_{\pm 0.036}$ | $0.791_{\pm 0.063}$ | $0.634_{\pm 0.020}$ | $0.097_{\pm 0.032}$ | $0.858_{\pm 0.007}$
TabGAN | ${0.188}_{\pm 0.039}$ | ${0.749}_{\pm 0.060}$ | ${0.760}_{\pm 0.027}$ | ${0.915}_{\pm 0.051}$ | ${0.706}_{\pm 0.021}$ | ${0.112}_{\pm 0.037}$ | ${0.844}_{\pm 0.006}$
TabVAE | ${0.249}_{\pm 0.017}$ | ${0.624}_{\pm 0.045}$ | ${0.965}_{\pm 0.011}$ | ${0.740}_{\pm 0.088}$ | ${0.664}_{\pm 0.012}$ | ${0.092}_{\pm 0.027}$ | ${0.847}_{\pm 0.006}$
SMOTE | ${0.306}_{\pm 0.000}$ | ${0.625}_{\pm 0.000}$ | ${0.975}_{\pm 0.000}$ | ${0.787}_{\pm 0.000}$ | ${0.671}_{\pm 0.000}$ | ${0.141}_{\pm 0.000}$ | ${0.851}_{\pm 0.000}$
Ours | $0.529_{\pm 0.046}$ | $0.839_{\pm 0.098}$ | $0.963_{\pm 0.017}$ | $0.828_{\pm 0.048}$ | $0.706_{\pm 0.005}$ | $0.613_{\pm 0.057}$ | $0.862_{\pm 0.007}$
Equalized Odds Ratio
| AD | BA | CA | CR | DE | KD | LA
TabDDPM | $0.247_{\pm.0.033}$ | $0.496_{\pm.0.077}$ | $0.919_{\pm.0.068}$ | $0.680_{\pm.0.081}$ | $0.764_{\pm.0.110}$ | $0.134_{\pm.0.050}$ | $0.593_{\pm.0.484}$
CTGAN | ${0.125}_{\pm 0.047}$ | ${0.760}_{\pm 0.074}$ | ${0.669}_{\pm 0.038}$ | ${0.895}_{\pm 0.048}$ | ${0.815}_{\pm 0.062}$ | ${0.178}_{\pm 0.067}$ | ${0.611}_{\pm 0.172}$
TVAE | ${0.205}_{\pm 0.023}$ | ${0.621}_{\pm 0.046}$ | $0.950_{\pm 0.027}$ | ${0.667}_{\pm 0.115}$ | ${0.825}_{\pm 0.067}$ | ${0.126}_{\pm 0.046}$ | ${0.562}_{\pm 0.175}$
SMOTE | ${0.231}_{\pm 0.000}$ | ${0.593}_{\pm 0.000}$ | ${0.939}_{\pm 0.000}$ | ${0.683}_{\pm 0.000}$ | ${0.827}_{\pm 0.000}$ | ${0.202}_{\pm 0.000}$ | ${1.000}_{\pm 0.000}$
Ours | $0.641_{\pm 0.108}$ | $0.760_{\pm 0.043}$ | $0.901_{\pm 0.035}$ | $0.802_{\pm 0.056}$ | $0.254_{\pm 0.089}$ | $0.884_{\pm 0.055}$ | $0.170_{\pm 0.340}$

Figure 2: Comparison in SHAP values on KDD dataset: balanced scenario (top)
vs. unbalanced scenario (bottom) using XGBoost

In Figure 2, we illustrate the feature importance through absolute SHAP values
when making predictions with XGBoost on the KDD Census dataset. On the top, we
present the scenario where synthetic data generated by our approach exhibits a
uniformly distributed sensitive feature, sex, across all samples. In contrast,
the figure on the bottom showcases data generated by TabDDPM with biased
sampling in the sex feature. It is evident that in the data generated by
TabDDPM, the variable sex significantly influences decision-making, while our
method mitigates the impact of sex on the final decision to a considerable
extent.

Apart from fairness evaluation dependent on machine learning models, we also
analyze the class distribution within the sensitive attributes. For example,
in the UCI Adult dataset, the distribution of sensitive attributes has an
inherent bias. In contrast, the synthetic data generated through our approach
exhibits more balanced distributions, as depicted in Figure 3, where the
y-axis represents the percentage of the respective attribute. We present
visualizations of class imbalance in sensitive attributes for the rest of
experimental datasets in Figure 4.

Figure 3: Comparison in the real vs. synthetic distribution of sensitive
attributes in the UCI Adult dataset

###  5.5 Limitations and Discussion

Our proposed method is designed to generate balanced synthetic tabular data
considering sensitive attributes. Empirically, classifiers trained on such
balanced synthetic tabular data demonstrate higher fairness scores at equal
allocation metrics indicated by demographic parity ratios. However, there is
no guarantee that equal performance, reflected by equalized odds ratios, will
always improve. We use U-Net with attention layers as the backbone neural
network for the posterior estimator in the diffusion framework, but it is
time-consuming. Despite our model exhibiting superior machine learning
efficiency and higher fairness scores compared to other deep generative models
in our experiments, the computational time is approximately five times longer.
In the future, we may explore the incorporation of constraints to ensure equal
performance and seek ways to reduce computational costs.

##  6 Conclusion

In this work, we propose a novel diffusion model framework for mixed-type
tabular data conditioned on both outcome and sensitive feature variables. Our
approach leverages a multivariate guidance mechanism and performs balanced
sampling considering sensitive features while ensuring a fair representation
of the generated data. Extensive experiments on real-world datasets containing
sensitive demographics demonstrate that our model achieves competitive
performance and superior fairness compared to existing baselines.

## Impact Statements

The objective of this study is to make progress in the area of fair machine
learning. Tabular datasets sometimes contain inherent bias, such as imbalanced
distributions in sensitive attributes. Training machine learning models on
biased datasets may result in decisions that could negatively affect minority
groups. By mitigating biases present in tabular datasets through the
generation of equitable synthetic data, our approach contributes to fostering
equitable decision-making processes across industries such as finance,
healthcare, and employment. Moreover, in instances where sharing datasets
becomes necessary, the utilization of fair synthetic data ensures the
preservation of user privacy and avoids causing emotional distress to minority
groups. An adverse possibility is that bad individuals could manipulate
distributions in sensitive attributes to generate biased (even stronger than
original) data to harm minority groups.

## References

  * Agarwal et al. (2018) Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H.  A reductions approach to fair classification.  In _International conference on machine learning_ , pp. 60–69. PMLR, 2018.
  * Austin et al. (2021) Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R.  Structured denoising diffusion models in discrete state-spaces.  _Advances in Neural Information Processing Systems_ , 34:17981–17993, 2021.
  * Borisov et al. (2022) Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., and Kasneci, G.  Deep neural networks and tabular data: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Calmon et al. (2017) Calmon, F., Wei, D., Vinzamuri, B., Natesan Ramamurthy, K., and Varshney, K. R.  Optimized pre-processing for discrimination prevention.  _Advances in neural information processing systems_ , 30, 2017.
  * Chawla et al. (2002) Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16:321–357, 2002.
  * Chen (2023) Chen, T.  On the importance of noise scheduling for diffusion models.  _arXiv preprint arXiv:2301.10972_ , 2023.
  * Choi et al. (2017) Choi, E., Biswal, S., Malin, B., Duke, J., Stewart, W. F., and Sun, J.  Generating multi-label discrete patient records using generative adversarial networks.  In _Machine learning for healthcare conference_ , pp. 286–305. PMLR, 2017.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Friedrich et al. (2023) Friedrich, F., Schramowski, P., Brack, M., Struppek, L., Hintersdorf, D., Luccioni, S., and Kersting, K.  Fair diffusion: Instructing text-to-image generation models on fairness.  _arXiv preprint arXiv:2302.10893_ , 2023.
  * Hardt et al. (2016) Hardt, M., Price, E., and Srebro, N.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ , 29, 2016.
  * He et al. (2023) He, H., Zhao, S., Xi, Y., and Ho, J. C.  Meddiff: Generating electronic health records using accelerated denoising diffusion model.  _arXiv preprint arXiv:2302.04355_ , 2023.
  * Ho & Salimans (2022) Ho, J. and Salimans, T.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hoogeboom et al. (2021) Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M.  Argmax flows and multinomial diffusion: Learning categorical distributions.  _Advances in Neural Information Processing Systems_ , 34:12454–12465, 2021.
  * Jesus et al. (2022) Jesus, S., Pombal, J., Alves, D., Cruz, A., Saleiro, P., Ribeiro, R., Gama, J., and Bizarro, P.  Turning the tables: Biased, imbalanced, dynamic tabular datasets for ml evaluation.  _Advances in Neural Information Processing Systems_ , 35:33563–33575, 2022.
  * Kamiran & Calders (2012) Kamiran, F. and Calders, T.  Data preprocessing techniques for classification without discrimination.  _Knowledge and information systems_ , 33(1):1–33, 2012.
  * Kamishima et al. (2011) Kamishima, T., Akaho, S., and Sakuma, J.  Fairness-aware learning through regularization approach.  In _2011 IEEE 11th International Conference on Data Mining Workshops_ , pp. 643–650. IEEE, 2011.
  * Kamishima et al. (2012) Kamishima, T., Akaho, S., Asoh, H., and Sakuma, J.  Fairness-aware classifier with prejudice remover regularizer.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23_ , pp. 35–50. Springer, 2012.
  * Kohavi et al. (1996) Kohavi, R. et al.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.  In _Kdd_ , volume 96, pp. 202–207, 1996.
  * Kong et al. (2020) Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020.
  * Kotelnikov et al. (2023) Kotelnikov, A., Baranchuk, D., Rubachev, I., and Babenko, A.  Tabddpm: Modelling tabular data with diffusion models.  In _International Conference on Machine Learning_ , pp. 17564–17579. PMLR, 2023.
  * Le Quy et al. (2022) Le Quy, T., Roy, A., Iosifidis, V., Zhang, W., and Ntoutsi, E.  A survey on datasets for fairness-aware machine learning.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 12(3):e1452, 2022.
  * Lemaître et al. (2017) Lemaître, G., Nogueira, F., and Aridas, C. K.  Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning.  _Journal of machine learning research_ , 18(17):1–5, 2017.
  * Li et al. (2022) Li, Y., Lu, X., Wang, Y., and Dou, D.  Generative time series forecasting with diffusion, denoise, and disentanglement.  _Advances in Neural Information Processing Systems_ , 35:23009–23022, 2022.
  * Lundberg & Lee (2017) Lundberg, S. M. and Lee, S.-I.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30, 2017.
  * Ma et al. (2020) Ma, C., Tschiatschek, S., Turner, R., Hernández-Lobato, J. M., and Zhang, C.  Vaem: a deep generative model for heterogeneous mixed type data.  _Advances in Neural Information Processing Systems_ , 33:11237–11247, 2020.
  * Mazé & Ahmed (2023) Mazé, F. and Ahmed, F.  Diffusion models beat gans on topology optimization.  In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Washington, DC_ , 2023.
  * Mehrabi et al. (2021) Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A.  A survey on bias and fairness in machine learning.  _ACM computing surveys (CSUR)_ , 54(6):1–35, 2021.
  * Nichol et al. (2021) Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Park et al. (2018) Park, N., Mohammadi, M., Gorde, K., Jajodia, S., Park, H., and Kim, Y.  Data synthesis based on generative adversarial networks.  _arXiv preprint arXiv:1806.03384_ , 2018.
  * Patki et al. (2016) Patki, N., Wedge, R., and Veeramachaneni, K.  The synthetic data vault.  In _2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA)_ , pp. 399–410. IEEE, 2016.
  * Pessach & Shmueli (2022) Pessach, D. and Shmueli, E.  A review on fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 55(3):1–44, 2022.
  * Pleiss et al. (2017) Pleiss, G., Raghavan, M., Wu, F., Kleinberg, J., and Weinberger, K. Q.  On fairness and calibration.  _Advances in neural information processing systems_ , 30, 2017.
  * Ramesh et al. (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rasul et al. (2021) Rasul, K., Seward, C., Schuster, I., and Vollgraf, R.  Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting.  In _International Conference on Machine Learning_ , pp. 8857–8868. PMLR, 2021.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pp. 234–241. Springer, 2015.
  * Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Sattarov et al. (2023) Sattarov, T., Schreyer, M., and Borth, D.  Findiff: Diffusion models for financial tabular data generation.  In _Proceedings of the Fourth ACM International Conference on AI in Finance_ , pp. 64–72, 2023.
  * Schramowski et al. (2023) Schramowski, P., Brack, M., Deiseroth, B., and Kersting, K.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22522–22531, 2023.
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pp. 2256–2265. PMLR, 2015.
  * Stypułkowski et al. (2023) Stypułkowski, M., Vougioukas, K., He, S., Zieba, M., Petridis, S., and Pantic, M.  Diffused heads: Diffusion models beat gans on talking-face generation.  _arXiv preprint arXiv:2301.03396_ , 2023.
  * Xu et al. (2018) Xu, D., Yuan, S., Zhang, L., and Wu, X.  Fairgan: Fairness-aware generative adversarial networks.  In _2018 IEEE International Conference on Big Data (Big Data)_ , pp. 570–575. IEEE, 2018.
  * Xu et al. (2019) Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K.  Modeling tabular data using conditional gan.  _Advances in neural information processing systems_ , 32, 2019.
  * Yeh & Lien (2009) Yeh, I.-C. and Lien, C.-h.  The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients.  _Expert systems with applications_ , 36(2):2473–2480, 2009.
  * Zafar et al. (2017) Zafar, M. B., Valera, I., Rogriguez, M. G., and Gummadi, K. P.  Fairness constraints: Mechanisms for fair classification.  In _Artificial intelligence and statistics_ , pp. 962–970. PMLR, 2017.
  * Zanna et al. (2022) Zanna, K., Sridhar, K., Yu, H., and Sano, A.  Bias reducing multitask learning on mental health prediction.  In _2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)_ , pp. 1–8. IEEE, 2022.
  * Zhang et al. (2018) Zhang, B. H., Lemoine, B., and Mitchell, M.  Mitigating unwanted biases with adversarial learning.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 335–340, 2018.
  * Zhang et al. (2023) Zhang, C., Zhang, C., Zhang, M., and Kweon, I. S.  Text-to-image diffusion model in generative ai: A survey.  _arXiv preprint arXiv:2303.07909_ , 2023.

##  Appendix A Details about Datasets

###  A.1 Data Sources

####  A.1.1 UCI Adult

The UCI Adult dataset (Kohavi et al., 1996) is widely used as a benchmark for
exploring fairness and bias in machine learning. It contains 48842 data
points. Each data point has 14 attributes and a binary target variable
indicating whether an individual earns over fifty thousand dollars annually.
The dataset encompasses employment, education, and demographic information,
and sensitive features are sex and race. Download it from OpenML.

####  A.1.2 Bank Marketing

The Bank Marketing dataset is collected from direct marketing campaigns of a
Portuguese banking institution. Comprising 45211 instances with 16 features,
the predicted outcome is a binary variable indicating whether a client
subscribed to a term deposit. Demographic, economic, and past marketing
campaign data are included in features, and marital status is sensitive.
Download it from OpenML.

####  A.1.3 Cardio

The Cardio dataset focuses on diagnosing cardiovascular disease (yes or no)
and it contains 70000 instances. It incorporates 11 features, which contain
demographics and factual information (blood pressure, cholesterol levels,
etc.), as well as self-reported labels such as whether smoking or not. The
sensitive feature is sex. Download it from OpenML.

####  A.1.4 Credit Card

The Credit Card dataset (Yeh & Lien, 2009) explores customer default payments
(yes or no) among credit card clients in Taiwan, and it contains 30000 data
points. Each data point has 23 features including the information of
demographics and financial records (monthly payments, etc.). The sensitive
features are sex and education. Download it from OpenML.

####  A.1.5 Depression

The Depression dataset contains demographic and clinical data (concentration,
rumination, etc.) to predict a diagnosis of depression. It has 11000
instances, each with 18 features, and sensitive features are sex and race.
Download it from OpenML.

####  A.1.6 KDD Census

The KDD Census dataset is an expanded version of the UCI Adult dataset. It
involves 199523 instances, each featuring 41 attributes and a binary target
variable indicating income over fifty thousand dollars per year. The dataset
covers employment, education, and demographic details, with sex and race as
sensitive features. Download it from OpenML.

####  A.1.7 Law School

The Law School dataset presents a survey on the law school admission process.
It has 20800 instances. Each instance has 12 attributes and a binary target
variable indicating admission to law school. Demographic and academic
information, such as GPA, are included in features. The sensitive features are
sex and race. Download it from OpenML.

###  A.2 Number of Features

Dataset | Number of Numerical Features | Number of Categorical Features
---|---|---
UCI Adult | 6 | 8
Bank Marketing | 7 | 9
Cardio | 5 | 6
Credit Card | 14 | 9
Depression | 14 | 4
KDD Census | 7 | 34
Law School | 6 | 6
Table 6: The number of numerical attributes and categorical attributes in
experimental datasets

###  A.3 Evaluation of Best-Performing Classifiers on Original Datasets

Dataset | Model Choice | Accuracy | Demographic Parity Ratio | Equalized Odds Ratio
---|---|---|---|---
UCI Adult | XGBoost | $0.866$ | $0.353$ | $0.273$
Bank Marketing | XGBoost | $0.900$ | $0.619$ | $0.540$
Cardio | XGBoost | $0.733$ | $0.992$ | $0.989$
Credit Card | Random Forest | $0.816$ | $0.778$ | $0.834$
Depression | XGBoost | $0.941$ | $0.696$ | $0.644$
KDD Census | XGBoost | $0.956$ | $0.151$ | $0.170$
Law School | XGBoost | $1.000$ | $0.825$ | $1.000$
Table 7: The values of accuracy and fairness scores of the best-performing
classifier on experimental datasets

###  A.4 Class Imbalance

(a) Bank Marketing

(b) Cardio

(c) Credit Card

(d) Depression

(e) KDD Census

(f) Law School

Figure 4: Visualizations of class imbalance in sensitive attributes in
experimental datasets except UCI Adult