# High-Quality Tabular Data Generation using Post-Selected VAE

Volodymyr Shulakov
Department of Computer Science and System Analysis
Cherkasy State Technological University
Shevchenka Blvd, 460, Cherkasy, Cherkasy Oblast, 18000, Ukraine
<EMAIL_ADDRESS>

###### Abstract

Synthetic tabular data is becoming a necessity as concerns about data privacy
intensify in the world. Tabular data can be useful for testing various
systems, simulating real data, analyzing the data itself or building
predictive models. Unfortunately, such data may not be available due to
confidentiality issues. Previous techniques, such as TVAE (Xu et al.,, 2019)
or OCTGAN (Kim et al.,, 2021), are either unable to handle particularly
complex datasets, or are complex in themselves, resulting in inferior run time
performance. This paper introduces PSVAE, a new simple model that is capable
of producing high-quality synthetic data in less run time. PSVAE incorporates
two key ideas: loss optimization and post-selection. Along with these ideas,
the proposed model compensates for underrepresented categories and uses a
modern activation function, Mish (Misra,, 2019).

_Keywords:_ generative models , variational autoencoder , synthetic data ,
multivariate sampling , tabular synthesis

##  1 Introduction

The importance of maintaining privacy of user data continues to grow today,
and real data, including tabular data that can be used in applied data
modelling (such as building a predictive model), socio-economic and physical
processes, is confidential. Usage of such data in information systems is
therefore becoming increasingly difficult. Techniques based on a combination
of statistical methods and neural networks are being incorporated into modern
data generation systems. As a result, implementation of such generative
algorithms for publishing data that is completely different numerically but
similar to the original statistical characteristics, is becoming increasingly
necessary.

In particular, synthetic data is relevant for a variety of purposes, such as
testing and validating machine learning algorithms, testing integration
software (Taneja et al.,, 2011), building simulation models, researching
statistical characteristics of data in various fields, such as healthcare
(Alsmadi et al.,, 2021). The use of synthetic data generation systems makes it
possible to obtain data for a research without directly using private data.

Synthetic data generation is not just an innovation, but a solution for
accurate, secure, and cost-effective data modelling. According to Gartner,
synthetic data will replace real data in artificial intelligence by 2030
(Ramos and Subramanyam,, 2021).

The objective of this work is to develop a simple and robust method for
generating synthetic data of high quality, thereby providing a model for
integration into various systems for a wide range of needs. The model
presented here, PSVAE, builds upon the variational autoencoder (Kingma and
Welling,, 2014) architecture and employs a custom loss adjustment algorithm
and a bespoke post-selection mechanism to filter the output of the decoder
network. This approach allows for extraction of the optimal univariate
statistics from complex datasets, while maintaining the quality of
multivariate relationships. The algorithm can simulate both discrete and
continuous distributions. It appears that this concept of post-selection has
not previously been employed in this manner.

##  2 Related Work

The most basic approach to generating random numerical data according to some
distribution is to try to mimic the distribution by closely matching its
univariate distributions. Doing so neglects the most crucial part, namely the
dependencies between variables, which represent the essence of high-quality
synthetic data.

The recent advances in deep generative models have opened up a wide range of
possibilities. Specifically, powerful neural network-based methods have
emerged, most notably CTGAN and TVAE (Xu et al.,, 2019) which can learn better
distributions than Bayesian networks such as those proposed by Chow and Liu,
(1968) or Zhang et al., (2017). Kim et al., (2021) proposed OCT-GAN, a
generative model based on neural ordinary differential equations that
outperforms TVAE in some cases. The synthesis quality of Invertible Tabular
GANs (Lee et al.,, 2021) is comparable to that of TVAE. Afonja et al., (2024)
introduce MargCTGAN, an improved version of CTGAN for generating synthetic
tabular data, particularly in low-sample regimes. CTAB-GAN+ (Zhao et al.,,
2024) uses Wasserstein loss with gradient penalty for better training
convergence and adds downstream losses to conditional GANs for higher utility
synthetic data.

It seems reasonable to posit that the VAE architecture offers an elegant and
efficient solution for developing synthetic data generative models.
Consequently, this work is focused on the implementation of this type of
neural network in order to achieve this objective.

##  3 Post-Selected VAE-based Data Generator

PSVAE builds on a VAE and is similar to TVAE, but does not include specific
layers for ‘multi-modal’ normalization. The encoder transforms multiple one-
hot encoded categories through two feature-extracting 256-neuron linear
layers. These features are then passed through another two 256-neuron layers
to obtain 128 $\mu$ and 128 $\sigma$ values, which account for one input layer
and two output layers. $\mu$ and $\sigma$ are used to re-parameterize the
latent space. No activation is applied after the output layers. The decoder
differs only in that its input dimension is of the size of the latent space
(128 float values) and output size is the same as the encoder’s input size.
Like the authors of TVAE, we found that a batch size of 500 performs the best
(Xu et al.,, 2019). To generate continuous synthetic variables, real data is
simply discretized into buckets. The number of buckets is $\min(\sqrt{N},100)$
where N is the number of records in a dataset.

To improve generalization, in-between hidden layers a modern smooth,
continuous, self regularized, non-monotonic activation function called mish is
used, which provides better empirical results than Swish, ReLU, and Leaky ReLU
(Misra,, 2019). It is incorporated into most ML frameworks, including Pytorch.
In PSVAE it is used instead of ReLU, in contrast to TVAE. Mish function is
defined as follows.

$f(x)=x\tanh(\ln(1+e^{x})).$

As with the regular VAE implementation, reconstruction loss (categorical cross
entropy) and KL-divergence of $\mu$ and $\sigma$ are used in combination.
However, in the case of TVAE, a factor of 2 is applied to the reconstruction
loss, which has the effect of impairing the model’s performance on certain
datasets. In order to achieve balance between these two losses, a simple
algorithm has been devised. This balancing takes inspiration from $\beta$-VAE
(Higgins et al.,, 2017), where our approach does not just tune $\beta$, but
optimizes it automatically in every training session (Algorithm 1). It also
improves the performance of the post-selection mechanism.

$Loss=L_{RE}+\beta L_{KL},$

where $L_{RE}$ is the reconstruction loss and $L_{KL}$ is the regularization
loss.

To address the issue of imbalanced datasets like credit, the cross-entropy
loss is weighted by the inverse frequency of the category occurrence
($\omega_{i}$)

$L(x,y)=-\sum_{c=1}^{C}{\omega_{i}\log\frac{e^{x_{c}}}{\sum_{i=1}^{C}{x_{i}}}*y_{c}}.$

Experimental results indicate that this approach yields superior results in
terms of synthetic data quality compared to the TVAE architecture.

Algorithm 1 PSVAE loss adjustment algorithm

$\theta\leftarrow$ network parameters

$\beta\leftarrow 1$

for each epoch do

$L_{KL}\leftarrow 0$

$L_{R}\leftarrow 0$

for each batch do

$l_{kl},l_{r}\leftarrow\textsc{model}($)

$L\leftarrow L_{KL}+\beta L_{RE}$

$\theta\leftarrow\textsc{optimize}(\theta,L)$

$L_{KL}\leftarrow L_{KL}+l_{kl}$

$L_{R}\leftarrow L_{R}+l_{r}$

end for

$s\leftarrow L_{KL}+L_{RE}$

$d\leftarrow 1/3*s$

if $L_{KL}>d$ then

$\beta\leftarrow\beta*1.04$

else

$\beta\leftarrow\beta/1.04$

end if

end for

Sampling random latent variables Decoding Post-Selection Synthetic data batch
N cyclesInitializationImprovement Figure 1: Illustration of the synthetic data
generation workflow of PSVAE

One of the most important parts of PSVAE is post-selection (Figure 1). This
mechanism plays the role of enhancing VAE output. To achieve this, an initial
batch of synthetic data is sampled from the decoder and is refined by
replacing samples with new ones from subsequent generated batches, provided
that the new samples exert a positive influence on the distribution of samples
(Algorithm 2).

Algorithm 2 PSVAE post-selection algorithm

$V\leftarrow$ normally-distributed random 128 floats

$S\leftarrow$ decoder(V)

$P\leftarrow$ pdf(S)

for each selection cycle do

$V\leftarrow$ normally-distributed random 128 floats

$R\leftarrow$ decoder(V)

for $i=1$ to $n$ do $\triangleright$ for each sample in the batch

$c\leftarrow S_{i}$

$r\leftarrow R_{i}$

if influence($P$, $c$, $r$) $>0$ then $\triangleright$ check if it improves
PDF similarity

$S_{i}\leftarrow r$

$P\leftarrow$ newpdf($P$, $c$, $r$)

end if

end for

end for

##  4 Experiments

TVAE, OCT-GAN and CTAB-GAN+ are selected to assess their performance in
comparison to PSVAE. Interestingly, despite the post-selection step utilizing
only univariate similarity, PSVAE is able to successfully reproduce
statistical intricacies of the original data, thereby confirming that the
quality of learned multivariate dependencies remains intact.

In the course of the experiments, L1-distance, F1 classification score and
Pearson correlation coefficients have been employed. Three datasets have been
selected for analysis (see Table 1).

  * –

Brain Stroke. URL: https://www.kaggle.com/datasets/jillanisofttech/brain-
stroke-dataset

  * –

Diabetes Health Indicators. URL:
https://www.kaggle.com/datasets/alexteboul/diabetes-health-indicators-dataset

  * –

Credit Card Fraud Detection. Has 31 columns, where the last one (Class) takes
value 1 in case of fraud and 0 otherwise. There are only 0.1% 1s out of 284k
0s. Thus, it is a challenging dataset to reproduce statistically. URL:
https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud

Table 1: datasets used in experiments Name | # records | # variables | Identity F1
---|---|---|---
Brain Stroke | 4981 | 11 | 0.57
Diabetes | 253680 | 22 | 0.6
Credit | 284807 | 31 | 0.9

Table 2 presents evaluations of all datasets of the models that were selected
for analysis. $t_{e}$ denotes the training run time of an epoch, in seconds.
In order to obtain an L1 metric, all L1-distances across all distributions of
columns in a dataset are averaged. A classifier network is trained on
synthetic data and subsequently tested against real data from the selected
datasets to calculate a mean F1-score of all prediction labels. Table 1 also
presents Identity-F1 metrics calculated on 20% of the data of each dataset,
with the remaining 80% used to train the classifier networks. The sum of all
absolute Pearson correlation coefficient differences of each pair of column
data distributions is denoted by $\rho$. Categorical values in the form of
alphanumeric labels are converted to unsigned sequential integers.

PSVAE is trained for 100 epochs for each dataset, with 10 post-selection
cycles performed for each synthetic data batch generation. Adam optimizer is
used with a learning rate of $10^{-3}$. A single batch consists of 500
randomly selected records. During experimentation, the performance of OCT-GAN
and CTAB-GAN was observed to not increase significantly after training for 50
and 100 epochs, respectively.

Table 2: Evaluation of different synthetic data generation models |  | Brain stroke | Diabetes | Credit
---|---|---|---|---
Model | $t_{e}$ | L1 | F1 | $\rho$ | L1 | F1 | $\rho$ | L1 | F1 | $\rho$
OCT-GAN | 30 | 0.19 | 0.5 | 5.12 | 0.14 | 0.38 | 10.6 | 0.08 | 0.84 | 25.1
CTAB-GAN+ | 5 | 0.08 | 0.51 | 3.46 | 0.18 | 0.41 | 6.07 | 0.07 | 0.82 | 23.5
TVAE | 1 | 0.14 | 0.49 | 5.54 | 0.11 | 0.6 | 67.6 | 0.08 | 0.51 | 32.4
PSVAE | 2 | 0.01 | 0.55 | 1.76 | 0.01 | 0.45 | 12.1 | 0.01 | 0.83 | 15.8

A modern GPU (RTX 4060) was used to train all the models. OCT-GAN and CTAB-
GAN+ took considerable time to achieve performance comparable to that of
PSVAE. When training on the diabetes dataset, a single epoch took 2 seconds
for PSVAE, whereas CTAB-GAN+ took 5 seconds per epoch and OCT-GAN performed a
single epoch in 30 seconds. This slowdown may be due to high complexity of the
underlying neural architectures.

As illustrated in Figure 1(a), PSVAE is capable of capturing all the intricate
details of the original datasets. The previous models are unable to identify
much of the correlations present in the last column of the credit dataset,
likely due to imbalanced data distribution. Interestingly, TVAE has better F1
performance, likely due to data distillation by approximating continuous
variables with Gaussian mixtures.

(a)

(a)

(b)

Figure 2: Correlations between variables of (a) brain stroke dataset, (b)
diabetes dataset, (c) credit dataset. It can be seen in (c) that TVAE does not
have proper correlations in the last column.

The results of ablation study are as follows.

  * –

Replacing Mish activation with ReLU results in higher losses and thus slower
training time.

  * –

Removal of either categorical loss weighting or loss adjustment algorithm
precludes the model from learning intricate correlations ($\rho$), such as
those observed in the credit dataset.

  * –

Removing the post-selection deteriorates the L1 accuracy down to the level of
TVAE, requiring a larger network and a longer training time to address
underfitting.

##  5 Conclusion

The study presents a new approach to generating high-quality synthetic tabular
data called PSVAE that augments a VAE with improved loss computation
algorithm, data balancing, modern activation function mish and a post-
selection mechanism.

Ultimately, PSVAE offers superior performance on L1-distance, comparable
F1-scores, and in some cases better correlation synthesis of datasets of
varying complexity, compared to previous solutions. TVAE tends to poorly
reproduce complex data correlations, although OCT-GAN manages to achieve a
good $\rho$-distance on the diabetes dataset. Nevertheless, previous methods
that produce competitive results have inferior run times in comparison to
PSVAE.

In future work, vector-quantization may be used (Van Den Oord et al.,, 2017)
to investigate its capabilities within VAE in generating synthetic data of
even better quality.

## References

  * Afonja et al., (2024) Afonja, T., Chen, D., and Fritz, M. (2024).  MargCTGAN: a “marginally” better CTGAN for the low sample regime.  In Köthe, U. and Rother, C., editors, Pattern Recognition, pages 524–537, Cham. Springer Nature Switzerland.
  * Alsmadi et al., (2021) Alsmadi, T., Alqudah, N., and Najadat, H. (2021).  Prediction of Covid-19 patients states using data mining techniques.  In International Conference on Information Technology (ICIT), pages 251–256.
  * Chow and Liu, (1968) Chow, C. and Liu, C. (1968).  Approximating discrete probability distributions with dependence trees.  IEEE Transactions on Information Theory, 14(3):462–467.
  * Higgins et al., (2017) Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017).  beta-VAE: Learning basic visual concepts with a constrained variational framework.  In International Conference on Learning Representations.
  * Kim et al., (2021) Kim, J., Jeon, J., Lee, J., Hyeong, J., and Park, N. (2021).  OCT-GAN: neural ODE-based conditional tabular GANs.  In The Web Conference, page 1506–1515, New York, NY, USA.
  * Kingma and Welling, (2014) Kingma, D. P. and Welling, M. (2014).  Mw (2014). Auto-encoding variational bayes.  In the International Conference on Learning Representations (ICLR).
  * Lee et al., (2021) Lee, J., Hyeong, J., Jeon, J., Park, N., and Cho, J. (2021).  Invertible tabular gans: killing two birds with one stone for tabular data synthesis.  In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 4263–4273. Curran Associates, Inc.
  * Misra, (2019) Misra, D. (2019).  Mish: A self regularized non-monotonic neural activation function.  arXiv preprint arXiv:1908.08681.
  * Ramos and Subramanyam, (2021) Ramos, L. and Subramanyam, J. (2021).  Maverick* research: Forget about your real data — synthetic data is the future of ai.  Gartner Research.
  * Taneja et al., (2011) Taneja, K., Grechanik, M., Ghani, R., and Xie, T. (2011).  Testing software in age of data privacy: a balancing act.  In ESEC/FSE ’11, page 201–211, New York, NY, USA. Association for Computing Machinery.
  * Van Den Oord et al., (2017) Van Den Oord, A., Vinyals, O., et al. (2017).  Neural discrete representation learning.  Advances in neural information processing systems, 30.
  * Xu et al., (2019) Xu, L., Skoularidou, M., Cuesta-Infante, A., and Veeramachaneni, K. (2019).  Modeling tabular data using conditional GAN.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  * Zhang et al., (2017) Zhang, J., Cormode, G., Procopiuc, C. M., Srivastava, D., and Xiao, X. (2017).  PrivBayes: Private Data Release via Bayesian Networks.  ACM Trans. Database Syst., 42(4).
  * Zhao et al., (2024) Zhao, Z., Kunar, A., Birke, R., Van der Scheer, H., and Chen, L. Y. (2024).  CTAB-GAN+: enhancing tabular data synthesis.  Frontiers in Big Data, 6.