# Differentially Private Tabular Data Synthesis
using Large Language Models

Toan V. Tran
Emory University, GA, USA
<EMAIL_ADDRESS>
&Li Xiong
Emory University, GA, USA
<EMAIL_ADDRESS>

###### Abstract

Synthetic tabular data generation with differential privacy is a crucial
problem to enable data sharing with formal privacy. Despite a rich history of
methodological research and development, developing differentially private
tabular data generators that can provide realistic synthetic datasets remains
challenging. This paper introduces DP-LLMTGen – a novel framework for
differentially private tabular data synthesis that leverages pretrained large
language models (LLMs). DP-LLMTGen models sensitive datasets using a two-stage
fine-tuning procedure with a novel loss function specifically designed for
tabular data. Subsequently, it generates synthetic data through sampling the
fine-tuned LLMs. Our empirical evaluation demonstrates that DP-LLMTGen
outperforms a variety of existing mechanisms across multiple datasets and
privacy settings. Additionally, we conduct an ablation study and several
experimental analyses to deepen our understanding of LLMs in addressing this
important problem. Finally, we highlight the controllable generation ability
of DP-LLMTGen through a fairness-constrained generation setting.

##  1 Introduction

Tabular data is one of the most popular forms of data for data analytic
applications (Borisov et al., 2022). In many domains such as healthcare and
finance, real data may be privacy sensitive and can not be directly shared and
utilized. To address this issue, synthetic data, generated from real datasets,
enables the use of realistic datasets without exposing real-world data.
However, synthetic data can still retain sample-level patterns and details
from the original datasets. This poses privacy risks for reidentification and
attribute disclosure by membership inference or by linking attributes,
especially with sensitive information (Zhu et al., 2024). A promising approach
is to integrate differential privacy (DP), which introduces randomness in the
data generation (Dwork et al., 2006). This process ensures that the data
remain useful for analysis but significantly reduce the risk of re-identifying
individuals (Cummings et al., 2024; Zhu et al., 2024). Therefore, synthetic
data generation with differential privacy is an important problem that has
received growing attention by the research community (Aydore et al., 2021;
Vietri et al., 2022; Liu et al., 2023, 2021; Yoon et al., 2019).

Generating synthetic tabular data in both non-DP and DP settings have been
extensively investigated. One notable approach involves deep learning-based
Generative Adversarial Networks (GANs). Xu et al. (2019) has designed a
suitable variant of GANs for tabular data (named CTGAN) which outperformed
previous statistical methods (Chow and Liu, 1968; Zhang et al., 2017).
However, the utility of GANs struggles under DP settings when the models are
trained by differentially private stochastic gradient descent (DPSGD) (Abadi
et al., 2016) due to the complexity of the model architecture and adversarial
training of GANs. Consequently, an alternative approach has emerged by
leveraging marginal distributions. This approach privately measures marginal
distributions and then generates synthetic datasets from those distributions
(McKenna et al., 2022; Liu et al., 2021; Vietri et al., 2022; Aydore et al.,
2021). The marginal-based methods significantly outperformed GAN-based methods
and created a notable gap between marginal-based and deep learning-based
methods.

Furthermore, inspired by the development of large language models (LLMs),
Borisov et al. (2023) proposed a method that leverages pre-trained LLMs for
synthesizing tabular data in non-private settings. Compared to previous
methods, pre-trained LLMs with background knowledge offer a basic
understanding of contextual knowledge. For example, in the Adult dataset Dua
and Graff (2017), features like gender and occupation clearly have a
correlation, e.g., a mechanic is more likely to be a male. This basic
knowledge can be already gained by LLMs during their pretrained stage (Dong et
al., 2023). As a result, LLMs outperformed GANs for synthesizing tabular data
in non-DP settings (Borisov et al., 2023). However, whether LLMs can operate
effectively under DP settings remains an open question.

In this paper, we demonstrate that naively enhancing the existing approach
from (Borisov et al., 2023) by DPSGD do not work well. This is mainly because
such conventional DP fine-tuned LLMs fail at generating tabular data with
format compliance due to the injected noise. Additionally, the standard cross
entropy loss used in causal language modeling does not work well for tabular
data. To tackle the challenges, we propose DP-LLMTGen (an abbreviation of
Differentially Private Large Language Models for Tabular data Generation),
which is the first study leveraging pretrained LLMs for synthetic tabular
generation under DP settings.

In summary, our contributions are as the follows. 1) Technical novelty: we
propose a two-stage fine-tuning mechanism using a novel loss function. Our
fine-tuning mechanism distinguishes learning targets (format compliance and
data modeling) and treats them suitably. The loss function is designed to
align with the proposed fine-tuning procedure and improve numerical
understanding of LLMs. 2) Empirical evaluation: DP-LLMTGen outperforms
marginal-based methods in tight differential privacy settings, closing the gap
between marginal-based and deep learning-based methods. Additionally, our
ablation study demonstrates the two-stage fine tuning is necessary for format
compliance and the proposed loss function helps to improve their numerical
understanding and boost the overall performance. We also perform several
experimental analyses. Our findings include: $\bullet$ Data contamination
between the pre-training and experimental datasets of LLama-2 7b models is not
a significant problem. $\bullet$ LLMs incorporate feature names to enable
context-aware learning for tabular data synthesis. $\bullet$ Models without
dialogue optimization generate better synthetic datasets. $\bullet$ Larger
models are not always better. 3) Controllable generation: we empirically
demonstrate the effectiveness of controllable generation of DP-LLMTGen in a
fairness setting. Our controllable generator can significantly reduce biases
with only a minor utility tradeoff.

##  2 Related Works

Synthetic tabular data generation. This field has a rich history of
methodological research and development. Classical methods treat each feature
as a variable to model the data as a joint multivariate probability
distribution, from which synthetic samples are drawn. Various modeling methods
have been employed, such as trees (Chow and Liu, 1968), Bayesian networks
(Young et al., 2009), and copulas (Kamthe et al., 2021). With the advent of
advanced neural networks, Variational Autoencoders (VAEs) (Akrami et al.,
2022; Xu et al., 2019), Generative Adversarial Networks (GANs) (Park et al.,
2018; Xu et al., 2019), and diffusion models (Kotelnikov et al., 2023) have
enhanced the modeling of high-dimensional dependencies. Recent studies have
explored transformer-based architectures (Vaswani et al., 2017; Badaro et al.,
2023). Gulati and Roysdon (2023) conducted masked pretraining for encoder-only
models. Canale et al. (2022) trained decoder-only models from scratch.
Concurrently, Borisov et al. (2023) fine-tuned LLMs (GPT-2 (Radford et al.,
2019) and distilled GPT-2 (Sanh et al., 2019)) which have been pre-trained on
a massive amount of text data.

Differentially private tabular data generation. The advancements in standard
synthetic data generation have led to various mechanisms that can ensure
differential privacy. Classical methods have been enhanced to achieve the DP
property, e.g., private Bayesian networks (Zhang et al., 2017) and DP-copula
(Li et al., 2014). Additionally, many neural networks have been trained by
differentially private stochastic gradient descent (DPSGD) (Abadi et al.,
2016) such as DP-GAN (Xie et al., 2018) and DP-Conditonal GAN (Torkzadehmahani
et al., 2019). To improve privacy guarantees, Yoon et al. (2019) trained GANs
by the PATE framework (Papernot et al., 2018) which utilizes non-labeled
public data. Despite GANs’ effectiveness in non-DP settings, they do not align
well with DPSGD due to the complexity of the model architecture and
adversarial training of GANs. Consequently, marginal-based methods have
emerged as more effective under DP constraints (McKenna et al., 2022; Aydore
et al., 2021; Vietri et al., 2022; Liu et al., 2023). Recent efforts involve
training from scratch transformer-based networks with DPSGD (Castellon et al.,
2023; Sablayrolles et al., 2023). However, training from scratch usually
requires significant privacy budgets and thus still lags behind marginal-based
methods in most cases. Our study is the first to leverage pre-trained LLMs and
introduces a novel fine-tuning approach for tight DP settings. We provide
additional related works and clarify the differences in Appendix A.

##  3 Preliminaries

Differential Privacy (DP). Differential privacy is a golden privacy notation
for quantifying and bounding the privacy risks of releasing data statistics
(Dwork et al., 2006). Informally, DP ensures the outputs of computational
algorithms do not change significantly when a single record is added to or
removed from the dataset. In this work, we use a popular notion of
differential privacy – $(\epsilon,\delta)$-DP.

###### Definition 1

A mechanism $\mathcal{M}:{D}\rightarrow{S}$ is $(\epsilon,\delta)$-DP if for
any two neighbouring datasets D and D’, and for any subset of output responses
$S\in Range(\mathcal{M})$,

$P[\mathcal{M}(D))\in S]\leq e^{\epsilon}P[\mathcal{M}(D^{\prime}))\in
S]+\delta$

Smaller values of $\epsilon$ and $\delta$ (close to zero) provide stronger
privacy guarantees. The post-processing property of DP mechanism ensures that
any computations, which are applied to the output of $\mathcal{M}$ and do not
access the original sensitive data, remain the same DP guarantee.

##  4 DP-LLMTGen: Differentially Private LLM-based Tabular data Generators

The previous work (Borisov et al., 2023) has leveraged pretrained LLMs for
non-private tabular data synthesis by simply converting tabular data to text
and fine tuning with standard causal language modeling. However, this
conventional fine-tuning approach fails under DP training due to format
compliance issues. To address this, we introduce DP-LLMTGen, as shown in
Figure 1. Initially, a random tabular dataset is generated using public
general knowledge. Subsequently, both the original sensitive tabular dataset
and the random tabular dataset are converted to textual data by the tabular-
to-text encoding. Next, the LLM undergoes the two-stage fine-tuning process.
Synthetic data are then generated by sampling from the fine-tuned LLM and
converted back into tabular format through the text-to-tabular decoding.

Figure 1: The process flow of DP-LLMTGen

###  4.1 Data processing

Random Tabular Data Generation. In this paper, we assume that general
knowledge, including feature names, categorical names, and numerical ranges,
are not sensitive. Instead, the actual sensitive information comprises
individual records. Consequently, we propose the generation of a dataset
(referred to as "random tabular data") that lacks realistic feature
distributions and exhibits random dependencies among features. The sole
requirements for this random data are accurate feature names, correct
categorical names, and numerically valid ranges. It is worth noting that this
assumption is similar to previous transformer-based works (Castellon et al.,
2023; Sablayrolles et al., 2023) which require creating a specialized
vocabulary.

Tabular-to-Text Encoding. We use a common template – "{feature_1 name} is
{value}, {feature_2 name} is {value}, etc." which has been used in some
previous works (Borisov et al., 2023; Dinh et al., 2022; Hegselmann et al.,
2023). Moreover, we permute the features randomly to remove the feature-order
dependencies which naturally does not exist in tabular data (Borisov et al.,
2022, 2023). This permutation step makes the sampling step more flexible and
controllable. Conversely, Text-to-Tabular Decoding is the reverse process of
Tabular-to-Text Encoding.

###  4.2 Two-stage Fine Tuning

Our fine tuning include two stages: 1) Format Learning and 2) Differentially
Private Fine Tuning. This procedure helps LLMs to ensure format compliance
while achieving DP properties. We implement a parameter-efficient fine tuning
method – LoRA (Hu et al., 2022) for both stages.

Stage 1: Format Learning. This stage utilizes the random data that does not
require any privacy protection. Therefore, we apply standard causal language
modeling for fine tuning the pretrained LLM. This stage helps the LLM to learn
the format, feature names, corresponding categorical names, and numerical
ranges without spending any privacy budget. Let $\\{t_{1},t_{2},t_{3},...\\}$
denote the sequence of tokens $\mathbb{S}^{(i)}$, where each token is an
element in vocabulary $V$. The loss function for this stage is the standard
cross entropy loss across all tokens, shown in Equation 1, where $N$ is the
number of sequences, $\theta$ is the model parameters, and
$p\left(t_{j}|t_{j-1},t_{j-2},...,t_{1},\theta\right)$ denotes the model
output probability of predicting token $t_{j}$ given the previous tokens.

$\mathcal{L}(\theta)=-\dfrac{1}{N}\sum_{i=1}^{N}\sum_{t_{j}\in\mathbb{S}^{(i)}}\log
p\left(t_{j}|t_{j-1},t_{j-2},...,t_{1},\theta\right)$ (1)

Stage 2: Differentially Private Fine Tuning. This stage aims to learn the
feature distributions and dependencies of the actual datasets. To protect the
sensitive data, the LLM is fine-tuned by differential private stochastic
gradient descent (DPSGD) (Abadi et al., 2016). To enhance the performance, we
propose a novel loss function (denoted as $\mathcal{L}(\theta)$ in Equation 7)
which includes two main components: Weighted Cross Entropy Loss (WCEL) and
Numerical-Understanding Loss (NUL).

WCEL: Given a sequence of tokens
$\mathbb{S}^{(i)}=\\{t_{1},t_{2},t_{3},...\\}$, the WCEL discriminates between
format tokens $\mathbb{F}^{(i)}$ and tabular tokens $\mathbb{T}^{(i)}$. More
specifically, words (such as feature names and "is") and commas are format
tokens while tabular tokens are derived from the numerical and categorical
values. For instance, considering the sentence "Age is 20, Education is high
school", the format and tabular tokens are "Age is , Education is" and "20
high school", respectively. This discrimination helps the LLMs to efficiently
focus on learning the feature distributions and dependencies instead of format
compliance, which has been achieved in the first stage. The WCEL is defined in
Equation 2.

$\mathcal{L}_{\text{WCE}}(\mathbb{S}^{(i)},\theta)=(1-\alpha)\sum_{t_{j}\in\mathbb{F}^{(i)}}-\log
p\left(t_{j}|t_{j-1},...,t_{1},\theta\right)+\alpha\sum_{t_{j}\in\mathbb{T}^{(i)}}-\log
p\left(t_{j}|t_{j-1},...,t_{1},\theta\right)$ (2)

NUL: There is a fundamental problem of cross entropy loss while dealing with
numbers. The cross entropy loss does not differentiate well between numerical
errors in different contexts. For instance, considering a ground truth of
10.0, the cross entropy loss treats identically between predicting 10.1 and
99.9. To address this, our Numerical-Understanding Loss considers the exact
difference between the prediction and the ground truth numbers. Let
$\hat{t}_{j}$ be the highest-probability prediction token for the token at
index $j$, formalized in Equation 3, where $w$ is a token in the model
vocabulary; $\theta$ is the model parameters; $f$ represents the model
forwarding. Subsequently, Equation 4 represents numeric token decoding, which
converts the consecutive tokens from index $j$ to $k$ into a number, denoted
by $\hat{n}_{\\{j,k\\}}$. For each sequence of tokens $\mathbb{S}^{(i)}$,
$\mathbb{N}^{(i)}$ is the set of all pairs $\\{j,k\\}$ which represents the
starting and ending indices of tabular numerical values. The NUL (Equation 6)
is the sum of the squared errors ($SE$ – defined in Equation 5) between
predicted numerical values $\hat{n}_{j,k}$ and their corresponding ground
truth values $n_{j,k}$. Notably, the numeric token decoding can fail if the
prediction tokens do not form a number. In success cases, the errors are
scaled with a factor $\lambda$. For failed cases, the error is set to 1.0,
which is typically higher than the success squared errors to enhance format
compliance.

$\hat{t}_{j}={\text{argmax}_{w}\left[f(w|t_{j-1},t_{j-2},...,t_{1},\theta))\right]}$
(3)

$\hat{n}_{\\{j,k\\}}=\text{decoding(}\\{\hat{t}_{j};\hat{t}_{j+1};...\hat{t}_{k}\\}\text{)}$
(4)

$SE(\\{j,k\\},\theta)=\begin{cases}1.0&\text{if decoding failed}\\\
\dfrac{1}{2}\left(\dfrac{n_{\\{j,k\\}}-\hat{n}_{\\{j,k\\}}}{\lambda}\right)^{2},&\text{otherwise}\end{cases}$
(5)

$\mathcal{L}_{\text{NU}}(\mathbb{S}^{(i)},\theta)=\sum_{\\{j;k\\}\in\mathbb{N}^{(i)}}SE(\\{j,k\\},\theta)$
(6)

The final loss function is the weighted sum of $\mathcal{L}_{\text{WCE}}$ and
$\mathcal{L}_{\text{NU}}$ over $N$ sentences:

$\mathcal{L}(\theta)=\dfrac{1}{N}\sum_{i=1}^{N}\left[\mathcal{L}_{\text{WCE}}(\mathbb{S}^{(i)},\theta)+\beta\mathcal{L}_{\text{NU}}(\mathbb{S}^{(i)},\theta)\right]$
(7)

###  4.3 Sampling

To generate synthetic samples, we initialize short prompts and ask the fine-
tuned LLMs to generate the remainder. Following the approach outlined by
Borisov et al. (2023), our LLMs facilitate both types of sampling: random-
initialized and value-specified. In random-initialized sampling, the prompts
consist of a randomly chosen feature name, such as "Age is" or "Workclass is".
In contrast, value-specified sampling involves initializing prompts with
specific feature values to enable controllable generation, e.g., "Age is 26,"
or "Income is >50K, Sex is female,". The fine-tuned LLMs can complete the
remainder of the sentences. The full sentences can be converted back to
tabular format by Text-to-Tabular Decoding, which is just a simple string
processing algorithm.

Table 1: Comparative performance of DP-LLMTGen against the baselines across
five datasets. For each metric, we report both the average value and standard
deviation of three times running.

Dataset Privacy Method Statistical Fidelity (TVD) ($\downarrow$) Xgboost
($\uparrow$) Budget 1-way 2-way 3-way 4-way 5-way ACC AUC Bank - Training Set
0.009±0.001 0.021±0.001 0.043±0.001 0.076±0.002 0.121±0.002 0.853±0.009
0.924±0.005 $\epsilon$ = 0.5 DP-GAN 0.196±0.019 0.302±0.023 0.368±0.023
0.411±0.021 0.440±0.017 0.528±0.010 0.547±0.032 DP-CTGAN 0.175±0.017
0.266±0.019 0.327±0.019 0.374±0.018 0.411±0.016 0.497±0.024 0.509±0.042 PATE-
CTGAN 0.200±0.003 0.291±0.004 0.351±0.004 0.400±0.003 0.442±0.002 0.420±0.061
0.331±0.027 RAP 0.165±0.006 0.268±0.005 0.345±0.003 0.404±0.002 0.446±0.001
0.636±0.006 0.699±0.005 RAP++ 0.071±0.005 0.129±0.005 0.185±0.005 0.240±0.005
0.293±0.006 0.635±0.068 0.697±0.062 GSD 0.166±0.001 0.245±0.002 0.302±0.001
0.354±0.002 0.401±0.001 0.635±0.017 0.692±0.017 DP-LLMTGen 0.050±0.003
0.091±0.003 0.134±0.003 0.181±0.004 0.232±0.004 0.637±0.007 0.669±0.032
$\epsilon$ = 1.0 DP-GAN 0.244±0.013 0.357±0.013 0.418±0.013 0.453±0.011
0.473±0.009 0.460±0.071 0.396±0.056 DP-CTGAN 0.157±0.019 0.244±0.020
0.303±0.022 0.347±0.023 0.382±0.023 0.499±0.025 0.518±0.214 PATE-CTGAN
0.194±0.007 0.287±0.008 0.350±0.008 0.402±0.007 0.445±0.006 0.533±0.027
0.583±0.111 RAP 0.159±0.005 0.263±0.003 0.341±0.002 0.400±0.001 0.443±0.001
0.655±0.008 0.705±0.006 RAP++ 0.058±0.007 0.105±0.009 0.151±0.010 0.201±0.011
0.252±0.011 0.674±0.017 0.747±0.019 GSD 0.166±0.001 0.244±0.002 0.301±0.001
0.353±0.002 0.400±0.001 0.671±0.041 0.726±0.032 DP-LLMTGen 0.047±0.010
0.085±0.012 0.125±0.011 0.169±0.010 0.218±0.009 0.638±0.004 0.703±0.008 Adult
- Training Set 0.004±0.001 0.011±0.001 0.024±0.001 0.044±0.001 0.073±0.001
0.872±0.002 0.928±0.001 $\epsilon$ = 0.5 DP-GAN 0.201±0.013 0.316±0.015
0.386±0.014 0.429±0.011 0.456±0.008 0.765±0.010 0.692±0.166 DP-CTGAN
0.151±0.012 0.239±0.012 0.299±0.010 0.343±0.007 0.378±0.005 0.759±0.001
0.606±0.089 PATE-CTGAN 0.262±0.005 0.365±0.004 0.425±0.004 0.464±0.003
0.488±0.002 0.518±0.248 0.379±0.146 RAP 0.152±0.001 0.253±0.002 0.326±0.002
0.382±0.002 0.425±0.002 0.791±0.003 0.819±0.004 RAP++ 0.066±0.003 0.116±0.004
0.162±0.004 0.208±0.006 0.251±0.008 0.766±0.012 0.758±0.023 GSD 0.245±0.001
0.337±0.000 0.395±0.000 0.438±0.000 0.470±0.000 0.759±0.001 0.768±0.019 DP-
LLMTGen 0.058±0.008 0.095±0.010 0.126±0.011 0.154±0.011 0.183±0.010
0.833±0.005 0.887±0.003 $\epsilon$ = 1.0 DP-GAN 0.259±0.112 0.370±0.091
0.427±0.058 0.460±0.034 0.479±0.018 0.761±0.002 0.589±0.126 DP-CTGAN
0.178±0.026 0.277±0.027 0.339±0.024 0.380±0.021 0.409±0.018 0.550±0.272
0.597±0.110 PATE-CTGAN 0.256±0.002 0.362±0.001 0.423±0.001 0.463±0.001
0.487±0.000 0.589±0.295 0.486±0.071 RAP 0.148±0.000 0.248±0.001 0.322±0.001
0.378±0.001 0.421±0.001 0.801±0.006 0.831±0.005 RAP++ 0.054±0.002 0.099±0.003
0.142±0.003 0.183±0.002 0.225±0.002 0.803±0.003 0.807±0.017 GSD 0.245±0.001
0.337±0.000 0.395±0.000 0.438±0.000 0.470±0.000 0.759±0.001 0.778±0.006 DP-
LLMTGen 0.038±0.002 0.068±0.002 0.096±0.003 0.124±0.003 0.153±0.002
0.831±0.003 0.879±0.003 Food - Training Set 0.062±0.002 0.134±0.004
0.221±0.010 0.299±0.013 0.354±0.014 0.902±0.027 0.880±0.143 $\epsilon$ = 0.5
DP-GAN 0.229±0.026 0.345±0.019 0.425±0.012 0.472±0.006 0.492±0.003 0.658±0.381
0.401±0.184 DP-CTGAN 0.229±0.009 0.344±0.004 0.427±0.004 0.475±0.003
0.494±0.001 0.637±0.186 0.358±0.145 PATE-CTGAN 0.175±0.010 0.295±0.006
0.393±0.004 0.456±0.002 0.486±0.002 0.868±0.020 0.346±0.060 RAP 0.152±0.015
0.266±0.017 0.377±0.013 0.451±0.008 0.485±0.004 0.739±0.241 0.489±0.217 RAP++
0.250±0.007 0.369±0.002 0.437±0.003 0.476±0.002 0.493±0.001 0.650±0.212
0.609±0.031 GSD 0.169±0.012 0.287±0.006 0.389±0.002 0.455±0.000 0.485±0.001
0.641±0.077 0.693±0.023 DP-LLMTGen 0.149±0.012 0.264±0.009 0.372±0.004
0.446±0.002 0.482±0.001 0.872±0.013 0.665±0.130 $\epsilon$ = 1.0 DP-GAN
0.249±0.023 0.364±0.019 0.437±0.012 0.477±0.006 0.494±0.003 0.372±0.219
0.650±0.055 DP-CTGAN 0.242±0.010 0.357±0.010 0.436±0.006 0.479±0.002
0.495±0.001 0.470±0.368 0.355±0.076 PATE-CTGAN 0.135±0.010 0.258±0.008
0.362±0.004 0.433±0.002 0.473±0.001 0.872±0.013 0.411±0.004 RAP 0.132±0.017
0.243±0.020 0.354±0.019 0.433±0.016 0.473±0.011 0.812±0.052 0.623±0.088 RAP++
0.228±0.024 0.346±0.020 0.423±0.014 0.468±0.007 0.489±0.003 0.850±0.030
0.601±0.045 GSD 0.158±0.010 0.272±0.007 0.376±0.004 0.446±0.002 0.481±0.001
0.632±0.075 0.692±0.193 DP-LLMTGen 0.113±0.006 0.220±0.008 0.333±0.009
0.419±0.007 0.468±0.004 0.872±0.013 0.482±0.173 Apple - Training Set
0.025±0.001 0.093±0.000 0.242±0.004 0.407±0.005 0.482±0.002 0.896±0.016
0.959±0.005 $\epsilon$ = 0.5 DP-GAN 0.270±0.003 0.371±0.003 0.433±0.002
0.476±0.000 0.494±0.001 0.498±0.006 0.579±0.053 DP-CTGAN 0.183±0.020
0.272±0.026 0.350±0.023 0.431±0.013 0.483±0.004 0.498±0.007 0.490±0.031 PATE-
CTGAN 0.119±0.003 0.199±0.005 0.288±0.005 0.400±0.003 0.478±0.001 0.505±0.036
0.505±0.065 RAP 0.202±0.002 0.315±0.000 0.434±0.001 0.491±0.000 0.499±0.000
0.467±0.033 0.529±0.069 RAP++ 0.120±0.030 0.270±0.028 0.411±0.017 0.479±0.006
0.497±0.001 0.616±0.047 0.665±0.047 GSD 0.124±0.005 0.243±0.002 0.377±0.001
0.468±0.000 0.496±0.000 0.526±0.011 0.547±0.010 DP-LLMTGen 0.076±0.009
0.156±0.008 0.311±0.006 0.448±0.002 0.493±0.000 0.588±0.041 0.684±0.033
$\epsilon$ = 1.0 DP-GAN 0.295±0.007 0.395±0.005 0.451±0.004 0.482±0.002
0.496±0.001 0.415±0.023 0.372±0.019 DP-CTGAN 0.282±0.027 0.373±0.025
0.431±0.018 0.473±0.009 0.493±0.003 0.491±0.021 0.585±0.057 PATE-CTGAN
0.116±0.006 0.193±0.007 0.283±0.006 0.398±0.008 0.477±0.005 0.542±0.009
0.597±0.027 RAP 0.199±0.002 0.312±0.001 0.432±0.001 0.491±0.000 0.499±0.000
0.487±0.048 0.506±0.040 RAP++ 0.090±0.006 0.233±0.005 0.386±0.008 0.471±0.004
0.495±0.001 0.654±0.011 0.719±0.010 GSD 0.093±0.004 0.204±0.002 0.345±0.002
0.454±0.002 0.493±0.000 0.563±0.016 0.582±0.016 DP-LLMTGen 0.068±0.003
0.139±0.003 0.277±0.003 0.425±0.002 0.488±0.001 0.639±0.011 0.687±0.033
Shipping - Training Set 0.010±0.001 0.026±0.001 0.060±0.001 0.123±0.003
0.218±0.005 0.693±0.005 0.756±0.004 $\epsilon=0.5$ DPGAN 0.228±0.023
0.334±0.022 0.395±0.018 0.435±0.016 0.463±0.012 0.513±0.093 0.493±0.081
DPCTGAN 0.269±0.015 0.368±0.021 0.421±0.019 0.452±0.014 0.473±0.009
0.537±0.107 0.485±0.039 PATECTGAN 0.105±0.001 0.173±0.003 0.230±0.003
0.296±0.004 0.376±0.004 0.593±0.009 0.572±0.030 RAP 0.109±0.006 0.177±0.005
0.236±0.004 0.303±0.002 0.378±0.002 0.577±0.006 0.518±0.012 RAP++ 0.080±0.002
0.145±0.004 0.214±0.006 0.290±0.007 0.364±0.007 0.594±0.011 0.624±0.031 GSD
0.057±0.002 0.100±0.002 0.149±0.001 0.218±0.002 0.306±0.003 0.606±0.014
0.663±0.022 DP-LLMTGen 0.048±0.002 0.084±0.002 0.126±0.003 0.186±0.003
0.271±0.005 0.590±0.006 0.510±0.031 $\epsilon=1.0$ DPGAN 0.219±0.067
0.327±0.061 0.395±0.044 0.439±0.029 0.467±0.017 0.510±0.102 0.498±0.133
DPCTGAN 0.274±0.025 0.372±0.022 0.422±0.018 0.453±0.013 0.474±0.009
0.468±0.109 0.489±0.036 PATECTGAN 0.070±0.002 0.124±0.002 0.176±0.002
0.243±0.004 0.328±0.005 0.592±0.013 0.480±0.152 RAP 0.109±0.005 0.178±0.004
0.237±0.003 0.303±0.001 0.377±0.001 0.582±0.007 0.541±0.060 RAP++ 0.070±0.007
0.123±0.011 0.182±0.013 0.253±0.013 0.332±0.010 0.623±0.006 0.673±0.014 GSD
0.053±0.002 0.093±0.001 0.140±0.000 0.208±0.002 0.297±0.003 0.616±0.009
0.689±0.011 DP-LLMTGen 0.042±0.002 0.074±0.002 0.114±0.000 0.174±0.004
0.262±0.008 0.594±0.010 0.452±0.061

##  5 Experiments and Results

###  5.1 Overall Evaluation

Datasets. We conduct experiments on five datasets, which vary in size from
less than 400 to more than 40,000 samples. These datasets also exhibit a
diverse range of ratios between numerical and categorical features. The
details of datasets and their key characteristics are provided in Appendix B,
Table 3. Notably, three of the five datasets were published after the
knowledge cutoff date of the Llama 2 models (et al., 2023). We split all
datasets into 80:20 for training and testing. All our and baseline models are
trained and tested on identical sets. To provide robust results, for each
dataset, we split into three different train-test sets using multiple random
seeds.

Baselines. Our baselines include state-of-the-art GAN-based and marginal-based
methods. Regarding GANs, we implement DP-GAN as a baseline, which trains
Generative Adversarial Networks by DPSGD. We also consider Conditional Tabular
GAN (CTGAN) (Xu et al., 2019) which is a variant designed for tabular data. We
train the CTGANs by both DPSGD (named DP-CTGAN) and Private Aggregation of
Teacher Ensembles (PATE) framework (Papernot et al., 2018) (referred as PATE-
CTGAN). For marginal-based methods, we implement three methods: RAP (Aydore et
al., 2021), RAP++ (Vietri et al., 2022), and GSD (Liu et al., 2023).

Metrics. We focus on two main kinds of metrics: Statistical Fidelity (SF) and
Machine Learning (ML) downstream performance, which are similar to previous
works (Liu et al., 2023; Vietri et al., 2022; Aydore et al., 2021). For
statistical fidelity, we calculate the average of total variation distances
(TVD) of single/joint distributions (ranging from 1-way to 5-way) between the
synthetic and testing sets. For ML downstream performance, we train XGBoost
models using the synthetic data and report their performance (accuracy and
AUC) on the testing sets. To ensure robust results, we run each experiment
three times and report the average values and standard deviations. Due to the
space limit, we present the details of the evaluation metrics in Appendix C.

Results. Table 1 presents the overall performance of DP-LLMTGen (using
LLama-2-chat-hf 7B backbone (et al., 2023)) and the baselines under different
privacy budget settings across five datasets. Generally, DP-LLMTGen
outperforms the baselines in most cases. For the Bank and Adult datasets,
which have large numbers of samples (11K and 48K, respectively), DP-LLMTGen
delivers better statistical fidelity for capturing both low- and high-
dimensional dependencies among features by around 15% compared to the
baselines. RAP++ is the only method that comes close. Furthermore, DP-LLMTGen
not only achieves the highest machine learning performance on the Adult
dataset but also demonstrates competitive results for the Bank dataset. For
the Food dataset, which has the smallest number of samples (i.e., 388), while
RAP++ fails to provide a high-quality synthetic dataset, DP-LLMTGen still
achieves the best statistical fidelity. For the Apple dataset, which consists
entirely of numerical features, DP-LLMTGen excels at capturing low-dimensional
dependencies. However, the GAN-based methods outperform the LLMs in modeling
high-dimensional joint relationships. Notably, better statistical fidelity
usually leads to higher performance for ML downstream model but not always.
This is consistent to the previous results (Tao et al., 2022). Due to the
space limit, we provide distance to closest record histogram in Appendix E.2
which demonstrates that DP-LLMTGen does not memorize and replicate the
training datasets. We also provides some visualizations regarding to 1-way and
2-way TVD in Appendices E.3 and E.4.

###  5.2 Ablation Study

####  5.2.1 Two-stage fine tuning

(a) Format-compliance performance of our 2-stage FT and conventional DP-FT
under a tight-privacy setting

(b) Performance on perplexity and format compliance of conventional DP-FT
under a loose privacy setting

Figure 2: Analyzing Conventional Differentially-Private Fine Tuning (DP-FT)
for Format Compliance on the Adult dataset

Two-stage fine tuning ensures format compliance while conventional DP fine
tuning fails. To verify the advantage of two-stage fine tuning versus
conventional DP fine tuning, we conduct an experiment on the Adult dataset.
Figure 2(a) presents the format-compliance capability of conventional DP-FT
(in green) and our 2-stage fine tuning (in orange and blue) under the privacy
budget of $\left(1,10^{-6}\right)$-DP. While the proposed 2-stage FT remains
nearly perfect format compliance, conventional DP-FT fails entire time. We
also investigate the conventional fine-tuning approach when the privacy
setting is loose (i.e., $\epsilon=8$ presented in Figure 2(b)). The noise
added is fairly small, so the model achieves the capability of 0.4 in the
second epoch. However, this capability diminishes with continued training,
although the evaluation perplexity consistently decreases. It can be because
perplexity or cross entropy loss, which is used for standard causal language
modeling, are calculated at token levels. However, format compliance is
considered at sentence levels. For example, a missing comma in a 200-token
sentence does not lead to high perplexity, but is considered as a failure at
format compliance.

####  5.2.2 Weighted Cross Entropy Loss

Weighted Cross Entropy Loss (WCEL) improves performance. Notably, if
$\alpha=0.5$, WCEL (in Equation 2) becomes the standard cross entropy loss
which is widely used in causal language modeling. Figure 3 depicts performance
of DP-LLMTGen with varying $\alpha$. Generally, using any $\alpha$ s.t.
$1>\alpha>0.5$, i.e., focusing more on the tabular tokens than the format
tokens, the model provides better perplexity and statistical fidelity than the
model trained by the standard cross entropy loss (i.e., $\alpha=0.5$), as
shown in Figures 3(a), 3(b), 3(c). However, the larger $\alpha$ can lead to
some loss of format-compliance capability, as demonstrated in Figure 3(d). At
$\alpha=1.0$, the second fine-tuning stage ignores entirely format tokens,
causing a significant drop in format compliance capability.

(a) Perplexity

(b) 1-way TVD

(c) 2-way TVD

(d) Format Compliance

Figure 3: Performance of DP-LLMTGen while varying $\alpha$ on three subsets of
the Adult dataset.

####  5.2.3 Numerical-Understanding Loss

Numerical-Understanding Loss (NUL) helps the LLMs to deal better with numbers.
To verify the significance of the Numerical-Understanding Loss, we conduct an
ablation study using the Apple dataset, where all features are numerical.
Table 2 presents the performance of DP-LLMTGen trained with and without NUL
under different settings of privacy budget. Generally, DP-LLMTGen trained with
NUL provides better statistical fidelity by about 5%. Moreover, we observe
that better Causal Language Modeling does not always lead to a better data
generator for numerical features. Table 2 provides the evaluation perplexity
and statistical fidelity. There is no direct causal relationships between the
two metrics. A higher-perplexity model can provide a better synthetic dataset.
This result confirms our statement about the fundamental problem of LLMs using
the standard cross entropy loss in representing and understanding numbers in
the previous section.

Table 2: Performance of DP-LLMTGen with and without the numerical-
understanding loss

Privacy Budget Method Eval PPL ($\downarrow$) 1-TVD ($\downarrow$) 2-TVD
($\downarrow$) 3-TVD ($\downarrow$) 4-TVD ($\downarrow$) $\epsilon=0.5$ DP-
LLMTGen w.o. NUL 3.882±0.007 0.081±0.005 0.162±0.009 0.316±0.012 0.452±0.007
DP-LLMTGen w. NUL 3.886±0.002 0.076±0.008 0.156±0.008 0.311±0.006 0.448±0.002
$\epsilon=1.0$ DP-LLMTGen w.o. NUL 3.865±0.002 0.064±0.003 0.139±0.004
0.287±0.007 0.434±0.005 DP-LLMTGen w. NUL 3.859±0.008 0.069±0.003 0.139±0.003
0.277±0.003 0.425±0.003

###  5.3 Additional Experimental Analyses

Data contamination testing. Data contamination between the pre-training and
experimental datasets of the LLama-2 7b model is not a significant problem.

Figure 4: Feature completion testing performance of GPT-4 and Llama-2-7b
models. The values represent for the percentage of correct completion.

To ensure no DP violence, we conduct the feature completion testing proposed
by Bordt et al. (2023) for Llama-2 7b and GPT-4 models for the Adult and Bank
datasets. The results (illustrated in Figure 4) indicate that GPT-4 (et al.,
2024c) could have been trained on those public datasets and actually memorizes
the training data to achieve the superior performance compared to the
baselines, including random-choice and majority-choice strategies. In
contrast, Llama-2 performs comparably to the random-choice strategy but
significantly worse than the majority-choice strategy. It indicates that
Llama-2 has no memorization and zero knowledge about feature distributions of
the datasets.

Do LLMs incorporate feature names for context-aware learning? LLMs achieve
better performance on datasets with meaningful feature names compared to those
lacking feature names.

Figure 5: Performance of DP-LLMTGen for the Adult subsets with and without
feature names.

We conduct an experiment using identical datasets: one dataset is maintained
in its original form, while the feature names in the other are changed to
Feature-1, Feature-2, etc. Figure 5 presents the performance of DP-LLMTGen for
these two datasets. Generally, DP-LLMTGen provides better synthetic datasets
when fine tuning on the datasets with feature names. This result demonstrates
that LLMs utilize pre-trained general knowledge to incorporate both feature
names and values. This ability is a unique characteristic of LLMs compared to
earlier approaches like GANs. GANs generate outcomes that are independent of
feature names, as they can only learn from the feature values. This aligns
with the results in (Dinh et al., 2022; Hegselmann et al., 2023) in
classification settings.

Does dialogue optimization provide benefit for DP-LLMTabGen? Models without
dialogue optimization can generate better synthetic datasets.

Figure 6: Performance of DP-LLMTGen when using Llama-2-7b-hf and
Llama-2-7b-chat-hf as the model backbone on subsets of the Adult dataset.

Most LLMs were developed for assistant tasks, which typically require dialogue
optimization. To understand the effects of dialogue optimization, we conduct
an experiments using Llama-2-hf and Llama-2-chat-hf models. Figure 6
illustrates the performance of DP-LLMTGen using those two models. Generally,
models which have not been optimized for conversational tasks exhibit a
marginal (though not significant) improvement – an average decrease of 2.85%
in 5-way TVD. This could be because synthetic data generation tasks might not
require dialogue-related features. Therefore, models like Llama-2-hf, which
have been optimized for more general tasks, could align better for
synthesizing tabular data.

Are Larger LLMs always better? Llama-2 7b can outperform LLama-2 13b under DP
settings.

Figure 7: Performance of DP-LLMTGen when using Llama-2-7b-chat-hf and
Llama-2-13b-chat-hf on three subsets of the Adult dataset.

To understand the effects of model size, we perform an experiment using two
Llama-2 models, 7b and 13b, on the Adult subsets. We configure identical
hyperparameters of LoRA (i.e., scaling factor and rank). It is noteworthy that
given the same rank in LoRA configuration, Llama-2 13b has bigger matrices
which leads to more trainable parameters. More specifically, Llama-2 7b and
13b require 8M and 13M trainable parameters, respectively. Figure 7 presents
the performance of the two LLMs where LLama-2 7b provides better synthetic
datasets. One of the reasons is that given fewer trainable parameters, DPSGD
employs a lower gradient clipping factor, resulting in smaller total noise.
This aligns with existing findings (Kurakin et al., 2023; Yu et al., 2022)
which demonstrated parameter-efficient fine tuning outperforms full fine-
tuning under differential privacy settings.

###  5.4 Controllable Generator – Fairness-Aware Generation

Figure 8: Fairness-Aware Generation Performance. DPDiff denotes Demographic
Parity Difference; EODiff stands for Equalized of Odd Difference.

Controllable generator can significantly reduce biases with a minor tradeoff
in utility. For controllable generation, DP-LLMTGen uses value-specified
sampling. For example, in the Adult dataset, to generate a sample that is a
male earning more than $50K/year, the initialized prompt can be "Gender is
male, Income is >50K,". We conduct an experiment where the amount of
controllable-generated data is varied from 5% to 50%. The remainder is
produced through random-initialized sampling. A value of 0.0% denotes non-
controllable generation. We also train XGBoost models on the generated
datasets. Figure 8 illustrates the fairness in data and ML downstream tasks.
Utilizing a controllable generator for a small fraction of the data (e.g.,
10%), the demographic parity difference (detailed in Appendix C.3) in data can
be decreased by $\sim$85%. This also leads to a substantial improvement in
fairness for ML downstream models while only reducing classification accuracy
by $\sim$1%. When the amount is significant, DP-LLMTGen can generate datasets
with perfect demographic parity. However, data demographic parity does not
guarantee the complete absence of bias in ML models, as bias elimination in ML
still requires in-processing fairness-aware training models (Li and Liu, 2022;
Han et al., 2024).

##  6 Conclusion

We have presented DP-LLMTGen which leverages pretrained LLMs for
differentially private tabular data synthesization. Overall, DP-LLMTGen
outperformed the baselines across multiple datasets and privacy settings,
closing the gap between deep learning-based and marginal-based methods. DP-
LLMTGen offers controllable generators which are important for some
applications that require fairness guarantees (Liu et al., 2024; Wang et al.,
2024). However, one key limitation of DP-LLMTGen is the high computational
demand. Despite the limitations, with the rapid development of LLMs, we
believe utilizing those models can offer a new paradigm in privacy-preserving
tabular data synthesis.

## References

  * Abadi et al. [2016] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , CCS ’16, page 308–318, New York, NY, USA, 2016. Association for Computing Machinery.  ISBN 9781450341394.  doi: 10.1145/2976749.2978318.  URL https://doi.org/10.1145/2976749.2978318.
  * Akrami et al. [2022] Haleh Akrami, Anand A. Joshi, Jian Li, Sergül Aydöre, and Richard M. Leahy.  A robust variational autoencoder using beta divergence.  _Knowledge-Based Systems_ , 238:107886, 2022.  ISSN 0950-7051.  doi: https://doi.org/10.1016/j.knosys.2021.107886.  URL https://www.sciencedirect.com/science/article/pii/S0950705121010534.
  * Aydore et al. [2021] Sergul Aydore, William Brown, Michael Kearns, Krishnaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva.  Differentially private query release through adaptive projection.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 457–467. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/aydore21a.html.
  * Badaro et al. [2023] Gilbert Badaro, Mohammed Saeed, and Paolo Papotti.  Transformers for tabular data representation: A survey of models and applications.  _Transactions of the Association for Computational Linguistics_ , 11:227–249, 2023.  doi: 10.1162/tacl_a_00544.  URL https://aclanthology.org/2023.tacl-1.14.
  * Bordt et al. [2023] Sebastian Bordt, Harsha Nori, and Rich Caruana.  Elephants never forget: Testing language models for memorization of tabular data.  In _NeurIPS 2023 Second Table Representation Learning Workshop_ , 2023.
  * Borisov et al. [2022] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–21, 2022.  doi: 10.1109/TNNLS.2022.3229161.
  * Borisov et al. [2023] Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci.  Language models are realistic tabular data generators.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=cEygmQNOeI.
  * Canale et al. [2022] Luca Canale, Nicolas Grislain, Grégoire Lothe, and Johan Leduc.  Generative modeling of complex data, 2022.
  * Carey et al. [2024] Alycia N. Carey, Karuna Bhaila, Kennedy Edemacu, and Xintao Wu.  Dp-tabicl: In-context learning with differentially private tabular data, 2024.
  * Castellon et al. [2023] Rodrigo Castellon, Achintya Gopal, Brian Bloniarz, and David Rosenberg.  Dp-tbart: A transformer-based autoregressive model for differentially private tabular data generation.  _ArXiv_ , abs/2307.10430, 2023.  URL https://arxiv.org/abs/2307.10430.
  * Chen and Guestrin [2016] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, page 785–794, New York, NY, USA, 2016. Association for Computing Machinery.  ISBN 9781450342322.  doi: 10.1145/2939672.2939785.  URL https://doi.org/10.1145/2939672.2939785.
  * Chow and Liu [1968] C. Chow and C. Liu.  Approximating discrete probability distributions with dependence trees.  _IEEE Transactions on Information Theory_ , 14(3):462–467, 1968.  doi: 10.1109/TIT.1968.1054142.
  * Cummings et al. [2024] Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Yangsibo Huang, Matthew Jagielski, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, Nicolas Papernot, Ryan Rogers, Milan Shen, Shuang Song, Weijie Su, Andreas Terzis, Abhradeep Thakurta, Sergei Vassilvitskii, Yu-Xiang Wang, Li Xiong, Sergey Yekhanin, Da Yu, Huanyu Zhang, and Wanrong Zhang.  Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment.  _Harvard Data Science Review_ , 6(1), jan 16 2024.  https://hdsr.mitpress.mit.edu/pub/sl9we8gh.
  * Dinh et al. [2022] Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee.  LIFT: Language-interfaced fine-tuning for non-language machine learning tasks.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=s_PJMEGIUfa.
  * Dong et al. [2023] Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li.  Statistical knowledge assessment for large language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=pNtG6NAmx0.
  * Dua and Graff [2017] Dheeraj Dua and Casey Graff.  Uci machine learning repository: Adult data set, 2017.  URL https://archive.ics.uci.edu/ml/datasets/adult.  April 29 2024.
  * Dwork et al. [2006] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith.  Calibrating noise to sensitivity in private data analysis.  In _Proceedings of the Third Conference on Theory of Cryptography_ , TCC’06, page 265–284, Berlin, Heidelberg, 2006. Springer-Verlag.  ISBN 3540327312.  doi: 10.1007/11681878_14.  URL https://doi.org/10.1007/11681878_14.
  * et al. [2024a] Gemma Team et al.  Gemma: Open models based on gemini research and technology, 2024a.
  * et al. [2023] Hugo Touvron et al.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * et al. [2024b] Marah Abdin et al.  Phi-3 technical report: A highly capable language model locally on your phone, 2024b.
  * et al. [2024c] OpenAI et al.  Gpt-4 technical report, 2024c.
  * Gulati and Roysdon [2023] Manbir S Gulati and Paul F Roysdon.  TabMT: Generating tabular data with masked transformers.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=qs4swxtIAQ.
  * Han et al. [2024] Xiaotian Han, Jianfeng Chi, Yu Chen, Qifan Wang, Han Zhao, Na Zou, and Xia Hu.  FFB: A fair fairness benchmark for in-processing group fairness methods.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=TzAJbTClAz.
  * Hegselmann et al. [2023] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag.  Tabllm: Few-shot classification of tabular data with large language models.  In _International Conference on Artificial Intelligence and Statistics_ , pages 5549–5581. PMLR, 2023.
  * Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Kamthe et al. [2021] Sanket Kamthe, Samuel A. Assefa, and Marc Peter Deisenroth.  Copula flows for synthetic data generation.  _ArXiv_ , abs/2101.00598, 2021.  URL https://api.semanticscholar.org/CorpusID:230435729.
  * Kingma and Ba [2017] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * Kotelnikov et al. [2023] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko.  Tabddpm: modelling tabular data with diffusion models.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023.
  * Kurakin et al. [2023] Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and A. Terzis.  Harnessing large-language models to generate private synthetic text.  _ArXiv_ , abs/2306.01684, 2023.  URL https://arxiv.org/abs/2306.01684v2.
  * Kurakin et al. [2024] Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis.  Harnessing large-language models to generate private synthetic text, 2024.
  * Li et al. [2014] Haoran Li, Li Xiong, and Xiaoqian Jiang.  Differentially private synthesization of multi-dimensional data using copula functions.  _Advances in database technology : proceedings. International Conference on Extending Database Technology_ , 2014:475–486, 2014.  URL https://api.semanticscholar.org/CorpusID:14422031.
  * Li and Liu [2022] Peizhao Li and Hongfu Liu.  Achieving fairness at no utility cost via data reweighing with influence.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 12917–12930. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/li22p.html.
  * Li et al. [2023] Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin.  Synthetic data generation with large language models for text classification: Potential and limitations.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 10443–10461, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.647.  URL https://aclanthology.org/2023.emnlp-main.647.
  * Liu et al. [2021] Terrance Liu, Giuseppe Vietri, and Steven Wu.  Iterative methods for private synthetic data: Unifying framework and new methods.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=jcCatp6oWZK.
  * Liu et al. [2023] Terrance Liu, Jingwu Tang, Giuseppe Vietri, and Zhiwei Steven Wu.  Generating private synthetic data with genetic algorithms.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org, 2023.
  * Liu et al. [2024] Tongyu Liu, Ju Fan, Nan Tang, Guoliang Li, and Xiaoyong Du.  Controllable tabular data synthesis using diffusion models.  _Proc. ACM Manag. Data_ , 2(1), mar 2024.  doi: 10.1145/3639283.  URL https://doi.org/10.1145/3639283.
  * Mattern et al. [2022] Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan.  Differentially private language models for secure data sharing.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4860–4873, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.323.  URL https://aclanthology.org/2022.emnlp-main.323.
  * McKenna et al. [2022] Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau.  Aim: an adaptive and iterative mechanism for differentially private synthetic data.  _Proc. VLDB Endow._ , 15(11):2599–2612, jul 2022.  ISSN 2150-8097.  doi: 10.14778/3551793.3551817.  URL https://doi.org/10.14778/3551793.3551817.
  * Papernot et al. [2018] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ulfar Erlingsson.  Scalable private learning with PATE.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rkZB1XbRZ.
  * Park et al. [2018] Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim.  Data synthesis based on generative adversarial networks.  _Proc. VLDB Endow._ , 11(10):1071–1083, jun 2018.  ISSN 2150-8097.  doi: 10.14778/3231751.3231757.  URL https://doi.org/10.14778/3231751.3231757.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  _PyTorch: an imperative style, high-performance deep learning library_.  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Remerscheid et al. [2022] Nicolas W. Remerscheid, Alexander Ziller, Daniel Rueckert, and Georgios Kaissis.  Smoothnets: Optimizing cnn architecture design for differentially private deep learning, 2022.
  * Sablayrolles et al. [2023] Alexandre Sablayrolles, Yue Wang, and Brian Karrer.  Privately generating tabular data using language models, 2023.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  In _NeurIPS EMC2 Workshop_ , 2019.
  * [47] Michael Shoemate, Andrew Vyrros, Chuck McCallum, Raman Prasad, Philip Durbin, Sílvia Casacuberta Puig, Ethan Cowan, Vicki Xu, Zachary Ratliff, Nicolás Berrios, Alex Whitworth, Michael Eliot, Christian Lebeda, Oren Renard, and Claire McKay Bowen.  OpenDP Library.  URL https://github.com/opendp/opendp.
  * Shwartz-Ziv and Armon [2022] Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  _Information Fusion_ , 81:84–90, 2022.  ISSN 1566-2535.  doi: https://doi.org/10.1016/j.inffus.2021.11.011.  URL https://www.sciencedirect.com/science/article/pii/S1566253521002360.
  * Tao et al. [2022] Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau.  Benchmarking differentially private synthetic data generation algorithms, 2022.
  * team [2024] Llama team.  Llama-3.  https://ai.meta.com/blog/meta-llama-3/, 2024.
  * Torkzadehmahani et al. [2019] Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten.  Dp-cgan: Differentially private synthetic data and label generation.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 98–104, 2019.  doi: 10.1109/CVPRW.2019.00018.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Vietri et al. [2022] Giuseppe Vietri, Cédric Archambeau, Sergul Aydore, William Brown, Michael Kearns, Aaron Roth, Ankit Siva, Shuai Tang, and Zhiwei Steven Wu.  Private synthetic data for multitask learning and marginal queries.  In _NeurIPS 2022_ , 2022.  URL https://www.amazon.science/publications/private-synthetic-data-for-multitask-learning-and-marginal-queries.
  * Wang et al. [2024] Shiyu Wang, Yuanqi Du, Xiaojie Guo, Bo Pan, Zhaohui Qin, and Liang Zhao.  Controllable data generation by deep learning: A review.  _ACM Comput. Surv._ , 56(9), apr 2024.  ISSN 0360-0300.  doi: 10.1145/3648609.  URL https://doi.org/10.1145/3648609.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In Qun Liu and David Schlangen, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Xie et al. [2024] Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, Bo Li, and Sergey Yekhanin.  Differentially private synthetic data via foundation model apis 2: Text, 2024.
  * Xie et al. [2018] Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou.  Differentially private generative adversarial network.  _ArXiv_ , abs/1802.06739, 2018.
  * Xu et al. [2019] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Modeling tabular data using conditional gan.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Yoon et al. [2019] Jinsung Yoon, James Jordon, and Mihaela van der Schaar.  PATE-GAN: Generating synthetic data with differential privacy guarantees.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=S1zk9iRqF7.
  * Young et al. [2009] Jim Young, Patrick Graham, and Richard Penny.  Using bayesian networks to create synthetic data.  _Journal of Official Statistics_ , 25:549–567, 12 2009.
  * Yousefpour et al. [2021] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov.  Opacus: User-friendly differential privacy library in PyTorch.  _arXiv preprint arXiv:2109.12298_ , 2021.
  * Yu et al. [2022] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang.  Differentially private fine-tuning of language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=Q42f0dfjECO.
  * Yue et al. [2023] Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David Levitan, and Robert Sim.  Synthetic text generation with differential privacy: A simple and practical recipe.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1321–1342, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.74.  URL https://aclanthology.org/2023.acl-long.74.
  * Zhang et al. [2017] Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao.  Privbayes: Private data release via bayesian networks.  _ACM Trans. Database Syst._ , 42(4), oct 2017.  ISSN 0362-5915.  doi: 10.1145/3134428.  URL https://doi.org/10.1145/3134428.
  * Zhu et al. [2024] Chaoyi Zhu, Jiayi Tang, Hans Brouwer, Juan F. Pérez, Marten van Dijk, and Lydia Y. Chen.  Quantifying and mitigating privacy risks for tabular generative models, 2024.

Appendix

Due to the space limit, many details are presented in the Appendix.

[sections] [sections] 1

##  Appendix A Additional Related Works

Transformer-based approach for differentially private synthesizing tabular
data. Although our method (DP-LLMTGen) and the methods from [Castellon et al.,
2023, Sablayrolles et al., 2023] (named DP-TBART and SynLM) are all based on
transformer networks, there are significant differences. DP-TBART and SynLM
train from scratch generative models specialized for each dataset. DP-TBART
and SynLM create a small vocabulary including categorical values, and
numerical quantiles. In contrast, DP-LLMTGen aims to fine tune pretrained
large language models. DP-LLMTGen converts tabular data into textual sentences
and uses a (human-language) semantic vocabulary. Additionally, the most
important point is that DP-TBART and SynLM underperform the marginal-based
methods in most cases [Castellon et al., 2023, Sablayrolles et al., 2023]. DP-
LLMTGen is the first approach that leverages powerful LLMs for differentially
private synthesizing tabular data and outperforms the marginal-based methods.

Large Language Models for (differentially private) synthetic textual data
generation. Large language models have been a powerful tool for synthetic data
generation. Li et al. [2023] investigated in-context learning for text
generation to enable text classifiers. Xie et al. [2024] utilized LLMs to
privately paraphrase texts. Meanwhile, Kurakin et al. [2024], Mattern et al.
[2022], Yue et al. [2023] fine tuning LLMs by DPSGD. Kurakin et al. [2024]
pointed out the data contamination problems of previous works and filtered the
experimental datasets to address this problem. Non-fine-tuning approaches also
work well for textual data because LLMs have been pretrained on massive texts.
However, the performance of non-fine-tuning LLMs for tabular data is still far
behind conventional methods [Carey et al., 2024]. Therefore, fine-tuning is
necessary for tabular data. This has been demonstrated in some tabular
classification problems [Dinh et al., 2022, Hegselmann et al., 2023].

##  Appendix B Details of datasets

We use two main dataset sources: UCI and Kaggle. The UCI machine learning
datasets are at larger scales and have been extensively studied by the
research community. The datasets from Kaggle are more recent and have been
published after the knowledge cutoff date of the Llama-2 models, which are
used for DP-LLMTGen’s model backbone.

Table 3: Key characteristics of our experimental datasets. Short Name |  Full Name | Source |  Number of samples |  Number of categorical features |  Number of numerical features |  Published after Llama-2 knowledge cutoff date
---|---|---|---|---|---|---
Bank |  Bank marketing | UCI |  11162 |  9 |  7 |  ✗
Adult |  Adult census income | UCI |  48842 |  7 |  6 |  ✗
Food |  Online food order | Kaggle |  388 |  6 |  5 |  ✓
Apple |  Apple quality | Kaggle |  4000 |  0 |  7 |  ✓
Shipping |  Ontime Shipping Classification | Kaggle |  10999 |  4 |  7 |  ✓

##  Appendix C Evaluation Metrics

###  C.1 Statistical fidelity.

Our work evaluates statistical fidelity based on the total variation distance
(TVD). To calculate TVD, numerical features are quantiled into 20 groups. Let
$[x_{1},x_{2},x_{3},...,x_{N_{\mathcal{F}}}]$ be a record/sample (i.e., a row
of tables), ${N_{\mathcal{F}}}$ is the number of features. We define $k$-way
TVD (also abbreviated as $k$-TVD) where $k$ is the number of features that are
involved. For example, $k=1$ denotes the single feature distributions
(Equation 8). Meanwhile, $k=2$ means the total variation distances of joint
two-feature distributions (Equation 9), where $f_{i}$ is the feature at index
$i$; $\mathcal{F}$ stands for the set of all features; $c$, $c_{1}$ and
$c_{2}$ denote the feature values; $\mathcal{D}$ and $\mathcal{D^{\prime}}$
represent for the synthetic and testing datasets. Inductively, we calculate
3-way TVD, 4-way TVD, and 5-way TVD.

$1\text{-way
TVD}(\mathcal{D},\mathcal{D^{\prime}})=\dfrac{1}{2N_{\mathcal{F}}}\sum_{f_{i}\in\mathcal{F}}\sum_{c\in
f_{i}}\ \left|p(x_{i}=c|\mathcal{D})-p(x_{i}=c|\mathcal{D^{\prime}})\right|$
(8)

$\begin{split}2\text{-way
TVD}(\mathcal{D},\mathcal{D^{\prime}})=\dfrac{1}{2N_{\mathcal{F}}\left(N_{\mathcal{F}}-1\right)}\sum_{f_{i}\in\mathcal{F}}\sum_{f_{j}\in\mathcal{F}\backslash
f_{i}}\sum_{c_{1}\in f_{i}}\sum_{c_{2}\in
f_{j}}|p(x_{i}=c_{1},x_{j}=c_{2}|\mathcal{D})\\\
-p(x_{i}=c_{1},x_{j}=c_{2}|\mathcal{D^{\prime}})|\end{split}$ (9)

###  C.2 Machine learning downstream performance.

We evaluate the ML downstream tasks using XGBoost [Chen and Guestrin, 2016]
which has achieved state-of-the-art performance in many benchmarks for tabular
data [Shwartz-Ziv and Armon, 2022]. We perform a grid search for 5-fold cross
validation to achieve the robust and realistic testing performance. The grid
search hyperparameters include number of estimators {100, 200, 300}, maximum
tree depth {3, 5, 10, 20}, and learning rate {0.01, 0.05, 0.1}.

###  C.3 Fairness metrics.

Demographic Parity, also known as statistical parity, is a fairness criterion
where particular attributes or model predictions are equally distributed
across different demographic groups. The demographic parity difference of a
mechanism $\mathcal{M}$ is calculated in Equation 10, where $x$ is the
features of a sample and $a$, $b$ are demographic groups. Similarly, the
demographic parity difference of a dataset can be formalized as Equation 11
where $Y(x)$ represents for the label of $x$.

$\mathcal{M}\\_\text{DPDiff}=|p(\mathcal{M}(x)=1|A=a)-p(\mathcal{M}(x)=1|A=b)|$
(10)

$\text{Data\\_DPDiff}=|p(Y(x)=1|A=a)-p(Y(x)=1|A=b)|$ (11)

Equalized Odds can guarantee a machine learning model performs equally well
for different groups. Let $TPR_{a}$, $TPR_{b}$, $FPR_{a}$, and $FPR_{b}$ be
the True Positive Rate and the False Positive Rate for groups $a$, $b$. The
Equalized Odds difference can be defined as the follow:

$\text{EO\\_DPDiff}=\text{max}(|TPR_{a}-TPR_{b}|,|FPR_{a}-FPR_{b}|)$ (12)

##  Appendix D Implementation details

For the GAN-based methods, we implemented using SmartNoise SDK
https://github.com/opendp/smartnoise-sdk which is a part of OpenDP library
[Shoemate et al., ]. For RAP, RAP++, and GSD, we use the code provided the
authors. For the baselines, we used recommended hyperparameters in the papers
and default hyperparameters of their github repositories (if no recommendation
in the papers). DP-LLMTGen’s implementation is based one Huggingface [Wolf et
al., 2020] for LLMs, Pytorch [Paszke et al., 2019], and Opacus [Yousefpour et
al., 2021] for DPSGD. Regarding to the hyperparameters of DP-LLMTGen, for the
first fine-tuning stage (i.e., format learning), we fine tune LLMs using Adam
optimizer [Kingma and Ba, 2017] and the learning rate at 1e-4. We fine tune 5
epochs for the Bank, Adult, and Shipping datasets and 10 epochs for the Food
and Apple datasets. The reason is the number of samples in the Food and Apple
datasets is small, so the LLMs need more epochs to learn. For the second
stage, $\alpha$ iws set at 0.65; the learning rate is 5e-4. For the Adult
dataset, we fine tune for 2 epochs. The remainder datasets are fined tune with
4 epochs. All the experiments are conducted at least three times by different
random train test splits to ensure the robust results. It should be noted that
the size of the synthetic datasets matches that of the training datasets.

##  Appendix E In-Depth Analyses of Overall Evaluation

###  E.1 Training time of DP-LLMTGen

The main factors affect to the training time is the dataset size, sample
length, and number of epochs. Various datasets have different number of
samples. Moreover, after converting to sentences (using Tabular-to-Text
decoding), the sentence length (i.e, number of tokens) depends on the number
of features and categorical/numerical values. It is worth noting that, given
the same dataset, there is no significant difference of training time when
using SGD or DPSGD. It is because, LLMs have billions of parameters, there are
only a dozen of millions of trainable parameters when using LoRA. Most of the
training time is spent on the forward pass of the LLMs rather than the
backward pass for LoRA parameters.

Table 4: Training time of DP-LLMTGen on a single H100 GPU Dataset | Training time for 1 epoch (Llama 2 7b)
---|---
Bank | 15 minutes
Adult | 1 hour 45 minutes
Food | 1 minute
Apple | 12 minutes
Shipping | 16 minutes

###  E.2 Distance to closest record histograms

Since LLMs have a strong capability to memorize the fine-tuning datasets, it
is essential to verify that LLMs actually generate new samples or just
replicate the training datasets. In this study, to demonstrate that LLMs are
capable of generating novel samples, we calculate and distance from each
record of the synthetic datasets to the closest record in the corresponding
training datasets. We normalize the datasets before distance calculation to
ensure equal importance of all features. A distance at 0 represents for a
replication from the training datasets to the generation. Figures 9 and 10
depicts the histograms of distances for the Shipping dataset under different
privacy budgets. Generally, there is no replication of training data when
using DP-LLMTGen. When the privacy constraints are more relaxed, all methods
tend to generate samples that are closer to each other than those produced
under a stricter privacy budget. Additionally, DP-LLMTGen and RAP++ generates
less outliers compared to the other methods.

Figure 9: Histograms of the distance to closest record between the
testing/synthetic and training sets for the Shipping dataset. The privacy
budget is set at $\epsilon=0.5$

Figure 10: Histograms of the distance to closest record between the
testing/synthetic and training sets for the Shipping dataset. The privacy
budget is set at $\epsilon=1.0$

###  E.3 Single feature distribution visualizations

DP-LLMTGen produces better single features distributions than the baselines.
That leads to a lower 1-way TVD. Especially, DP-LLMTGen replicates well for
minority feature categories while RAP, RAP++, and GSD amplify them (Figures
11(a) and 12(b)).

(a) Age distribution

(b) Family size distribution

(c) Output distribution

(d) Income distribution

Figure 11: [Food dataset] Single feature distributions of synthetic and real
sets. Numerical features are binned into 10 groups for this visualization. The
colors represent for categorical values and numerical groups.

(a) Product importance distribution

(b) Discount offered distribution

(c) Weight in gms distribution

(d) Warehouse block distribution

Figure 12: [Shipping dataset] Single feature distributions of synthetic and
real sets. Categorical features are binned into 10 groups for this
visualization. The colors represent for categorical values and numerical
groups.

###  E.4 Joint feature distribution visualizations

Figure 13: [Adult dataset] Joint distribution plot of two features (age and
relationship) of synthetic and real sets.

Figure 14: [Bank dataset] Joint distribution plot of two features (age and
job) of synthetic and real sets.

##  Appendix F Additional results for experimental analyses

Section 5.3 presents some experimental analyses using the Adult subsets with
8192 samples. In this section, we provide results of the same settings but
using 4096 samples. Generally, the trends are consistent to those of the
previous experiments 8192 samples.

###  F.1 Do large language models truly comprehend feature names?

Figure 15: Performance of DP-LLMTGen for the Adult subsets of 4096 samples
with and without feature names.

###  F.2 Does dialogue optimizing provide benefit DP-LLMTabGen?

Figure 16: Performance of DP-LLMTGen when using Llama-2-7b-hf and
Llama-2-7b-chat-hf as the model backbone on the Adult subsets of 4096 samples.

###  F.3 Are Larger LLMs always better?

Figure 17: Performance of DP-LLMTGen when using Llama-2-7b-chat-hf and
Llama-2-13b-chat-hf on three Adult subsets of 4096 samples.

##  Appendix G Limitations

LLM diversity. In our previous experiment on the Adult dataset, larger models
were not always better. The main reason is the more number of trainable
parameters causing bigger gradient clipping factor and larger total noise
added. The number of trainable parameters can be adjusted through the
parameter-efficient fine-tuning configuration. However, we did not tune this
number due to the limitation of computation resources. Therefore, a suitable
configuration of parameter-efficient fine-tuning for larger models can provide
promising results. Additionally, we acknowledge that there are many other
open-source models such as Mistral [Jiang et al., 2023] and Gemma [et al.,
2024a]. However, our experiments were mainly based on the Llama-2 models only.
It is still unknown about the performance of other model classes for DP
tabular data synthesis. Additionally, the higher-quality pre-training corpus
can deliver better LLMs given the same number of parameters [et al., 2024b].
For example, Phi-3 [et al., 2024b], which was pre-trained on a selected corpus
(composed of heavily filtered web data and synthetic data), outperformed all
other LLMs with the same number of parameters. Llama-3 8b is better Llama-2
70b in multiple benchmarks [team, 2024]. Given existing evidence that smaller
models can deal better with DPSGD training [Remerscheid et al., 2022], DP-
LLMTGen employing these parameter-efficient LLMs as the model backbone
potentially enhances the current performance.

ML downstream performance. DP-LLMTGen using Llama-2 outperformed the baselines
in terms of statistical fidelity, but did not entirely surpass them in
downstream machine learning performance. Our results exhibited a mismatch
between statistical fidelity and ML downstream performance, which is
consistent to a previous benchmark [Tao et al., 2022]. This is because
statistical fidelity treats the target feature and other features, equally.
However, for ML downstream performance, the target feature can be more
important compared to the other features. Only an mistake in the target
feature may not lead to a high error of the statistical fidelity but will
significantly hurt the ML downstream performance. Currently, DP-LLMTGen does
not discriminate target features and other features. Therefore, a possible
enhancement could be implementing a weighted loss function to increase the
importance of the target feature. Additionally, we evaluated DP-LLMTGen using
only the random sampling strategy for the general evaluation. The random
sampling does not consider data characteristics and marginal distributions.
DP-LLMTGen also offers controllable generation, which has been demonstrated in
our fairness-aware experiment. To improve the ML performance, it is possible
to generate synthetic datasets with diversity and uncertainty controls by
leveraging the controllable generation capability of DP-LLMTGen. Another
potential approach is to combine DP-LLMTGen’s controllable generation with
active learning or shaley-value methods which can determine which are worthy
samples.