# SOS: Score-based Oversampling for Tabular Data

Jayoung Kim<EMAIL_ADDRESS>0000-0002-6327-1006 ,  Chaejeong Lee
<EMAIL_ADDRESS>Yonsei UniversitySouth Korea ,  Yehjin Shin
<EMAIL_ADDRESS>Yonsei UniversitySouth Korea ,  Sewon Park
<EMAIL_ADDRESS>Samsung SDSSouth Korea ,  Minjung Kim
<EMAIL_ADDRESS>Samsung SDSSouth Korea ,  Noseong Park
<EMAIL_ADDRESS>Yonsei UniversitySouth Korea and  Jihoon Cho
<EMAIL_ADDRESS>Samsung SDSSouth Korea

(2022)

###### Abstract.

Score-based generative models (SGMs) are a recent breakthrough in generating
fake images. SGMs are known to surpass other generative models, e.g.,
generative adversarial networks (GANs) and variational autoencoders (VAEs).
Being inspired by their big success, in this work, we fully customize them for
generating fake tabular data. In particular, we are interested in oversampling
minor classes since imbalanced classes frequently lead to sub-optimal training
outcomes. To our knowledge, we are the first presenting a score-based tabular
data oversampling method. Firstly, we re-design our own score network since we
have to process tabular data. Secondly, we propose two options for our
generation method: the former is equivalent to a style transfer for tabular
data and the latter uses the standard generative policy of SGMs. Lastly, we
define a fine-tuning method, which further enhances the oversampling quality.
In our experiments with 6 datasets and 10 baselines, our method outperforms
other oversampling methods in all cases.

score-based generative model; tabular data synthesis; oversampling

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2022; Washington, DC, USA††booktitle: Proceedings of the 28th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August
14–18, 2022, Washington, DC, USA††price: 15.00††doi:
10.1145/3534678.3539454††isbn: 978-1-4503-9385-0/22/08††ccs: Computing
methodologies Artificial intelligence

##  1\. Introduction

Tabular data is one of the most frequently occurring data types in the field
of data mining and machine learning. However, imbalanced situations happen
from time to time, e.g., one class significantly outnumbers other classes.
Oversampling minor classes (and as a result, making class sizes even) is a
long-standing research topic. Many methods have been proposed, from classical
statistical methods (Chawla et al., 2002; Han et al., 2005; He et al., 2008),
to recent deep generative methods (Mariani et al., 2018; Mullick et al.,
2019).

(a) The overall workflow of score-based generative models, where the score
function is approximated by a score network, i.e.,
$S_{\theta}(\mathbf{x}_{t},t)\approx\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x})$. We note that it means the gradient of the log probability
w.r.t. $\mathbf{x}$ at time $t$. (b) The proposed concept where i) we transfer
a non-target (or major) record $\mathbf{x}^{+}_{0}$ to a synthesized target
(or minor) record $\hat{\mathbf{x}}^{-}_{0}$ or ii) we sample a noisy vector
$\mathbf{z}$ and generate $\hat{\mathbf{x}}^{-}_{0}$. (c) The proposed fine-
tuning method where we i) compare the two gradients, and ii) fine-tune
$\theta_{-}$ to enhance the sampling quality.

Figure 1. In this figure, we assume one major (non-target) and one minor
(target) classes for ease of discussion but our propose method can be applied
to multiple minor classes. (a) The reverse SDE can be considered as a
generator since it produces data from noise. (b) The combination of the major
(non-target) class’s forward SDE and the minor (target) class’s reverse SDE
can be considered as a _style transfer_ -based oversampling process. (c) The
fine-tuning process can further enhance the minor (target) class’s score
network.

Among many deep generative models, score-based generative models (SGMs (Song
et al., 2021)) have gained much attention recently. They adopt a stochastic
differential equation (SDE) based framework to model i) the (forward)
diffusion process and ii) the (reverse) denoising process (cf. Fig. 1 (a)).
The diffusion process is to add increasing noises to the original data
$\mathbf{x}_{0}$, and the denoising process is the reverse of the diffusion
process to remove noises from the noisy data $\mathbf{x}_{T}$. Since the space
of $\mathbf{x}$ can be tremendously large, the score function,
$\nabla_{\mathbf{x}}\log p_{t}(\mathbf{x})$ is approximated by a _score neural
network_ (or simply a score network), where $t\in[0,T]$ and the step sizes of
the diffusion and the denoising processes are typically 1. The ground-truth
score values are collected during the forward SDE process. In comparison with
generative adversarial networks (GANs), SGMs are known to be superior in terms
of generation quality (Song et al., 2021; Dockhorn et al., 2022; Xiao et al.,
2022). However, one drawback of SGMs is that they require large-scale
computation, compared to other generative models. The main reason is that SGMs
sometimes require long-step SDEs, e.g., $T=1,000$ for images in (Song et al.,
2021). It takes long time to conduct the denoising process with such a large
number of steps. At the same time, however, it is also known that small $T$
values work well for time series (Rasul et al., 2021).

In this paper, we propose a novel Score-based OverSampling (SOS) method for
tabular data and found that i) $T$ can be small in our case because of our
model design, ii) the score network architecture should be significantly
redesigned, and iii) a fine-tuning method, inspired by one characteristic of
imbalanced tabular data that we have both target and non-target classes, can
further enhance the synthesis quality.

As shown in Fig. 1 (b), we separately train two SGMs, one for the non-target
(or major) class and the other for the target (or minor) class to oversample —
for simplicity but without loss of generality, we assume the binary class
data, but our model is such flexible that multiple minor classes can be
augmented. In other words, we separately collect the ground-truth score values
of the target and non-target classes and train the score networks. Therefore,
their reverse SDEs will be very different. After that, we have two model
designs denoted as Option 1 and 2 in Fig. 1 (b). By combining the non-target
class’s forward SDE and the target class’s reverse SDE, firstly, we can create
a style transfer. A non-target record $\mathbf{x}^{+}_{0}$ is converted to a
noisy record $\mathbf{x}^{+}_{T}$, from which a fake target record
$\hat{\mathbf{x}}^{-}_{0}$ is created. In other words, the style (or class
label) of $\mathbf{x}^{+}_{0}$ is transferred to that of the target class.
Secondly, we sample a noisy vector $\mathbf{z}$ and generate a fake target
record.

Since the forward SDE is a pure Brownian motion process shared by the non-
target and the target classes, it constitutes a single noisy space. Since we
maintain a single noisy space for both of them, $\mathbf{x}^{+}_{T}$ is a
correct noisy representation of the non-target record $\mathbf{x}^{+}_{0}$
even when it comes to the target class’s reverse SDE. Therefore, the first
option in Fig. 1 (b) can be considered as a style transfer-based oversampling
method. Its main motivation is to oversample around class boundaries since the
non-target record is (little) perturbed to generate the fake target record
(cf. Fig. 6).

We further enhance our method by adopting the fine-tuning method in Fig. 1
(c), a post-processing procedure after training the two score networks. Our
fine-tuning concept is to compare the two gradients (approximated) by
$S_{\theta_{-}}$ and $S_{\theta_{+}}$ at a certain
$(\mathbf{x}_{t},t=\epsilon_{t})$ pair, where $\mathbf{x}\in\mathcal{T}$ is a
real record, $\mathcal{T}$ is tabular data, and $\epsilon_{t}$ is a time point
close to 0 — note that at this step, we do not care about the class label of
$\mathbf{x}$ but use all records in the tabular data. If the angle between the
two gradients is small, we regulate $S_{\theta_{-}}$ to discourage the
sampling direction toward the measured gradient (since the direction will lead
us to a region where the non-target and target records co-exist).

We conduct experiments with 6 benchmark tabular datasets and 10 baselines: 4
out of 6 datasets are for binary classification and 2 are for multi-class
classification. Our careful choices of the baselines include various types of
oversampling algorithms, ranging from classical statistical methods,
generative adversarial networks (GANs), variational autoencoders (VAEs) to
neural ordinary differential equation-based GANs. In those experiments, our
proposed method significantly outperforms existing methods by large margins —
in particular, our proposed method is the first one increasing the test score
commonly for all those benchmark datasets. Our contributions can be summarized
as follows:

  1. (1)

We, for the first time, design a score-based generative model (SGM) for
oversampling tabular data. Existing SGMs mainly study image synthesis.

  2. (2)

We design a neural network to approximate the score function for tabular data
and propose a style transfer-based oversampling technique.

  3. (3)

Owing to one characteristic of tabular data that we always have class labels
for classification, we design a fine-tuning method, which further enhances the
sampling quality.

  4. (4)

In our oversampling experiments with 6 benchmark datasets and 10 baselines,
our method not only outperforms them but also increases the test score for all
cases in comparison with the test score before oversampling (whereas existing
methods fail in at least one case).

  5. (5)

In Appendix E, our method overwhelms baselines when generating full fake
tabular data (instead of oversampling minor classes) as well.

##  2\. Related Work and Preliminaries

In this section, we review papers related to our research. We first review the
recent score-based generative models, followed by tabular data synthesis.

###  2.1. Score-based Generative Models

SGMs use the following Itô SDE to define diffusive processes:

(1) $\displaystyle d\mathbf{x}=\mathbf{f}(\mathbf{x},t)dt+g(t)d\mathbf{w},$

where $\mathbf{f}(\mathbf{x},t)=f(t)\mathbf{x}$ and its reverse SDE is defined
as follows:

(2) $\displaystyle
d\mathbf{x}=\big{(}\mathbf{f}(\mathbf{x},t)-g^{2}(t)\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x})\big{)}dt+g(t)d\mathbf{w},$

where this reverse SDE process is a generative process (cf. Fig. 1 (a)).
According to the types of $f$ and $g$, SGMs can be further classified as i)
variance exploding (VE), ii) variance preserving (VP), and ii) sub-variance
preserving (Sub-VP) models (Song et al., 2021).

In general, $\mathbf{x}_{0}$ means a real (image) sample and following the
diffusive process in Eq. (1), we can derive $\mathbf{x}_{t}$ at time
$t\in[0,T]$. These above specific design choices are used to easily
approximate the transition probability (or the perturbation kernel)
$p(\mathbf{x}_{t}|\mathbf{x}_{0})$ at time $t$ with a Gaussian distribution.
The reverse SDE in Eq. (2) maps a noisy sample at $t=T$ to a data sample at
$t=0$.

We can collect the gradient of the log transition probability,
$\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{t}|\mathbf{x}_{0})$, during the
forward SDE since it can be analytically calculated. Therefore, we can easily
train a score network
$S_{\theta}:\mathbb{R}^{\dim(\mathbf{x})}\times[0,T]\rightarrow\mathbb{R}^{\dim(\mathbf{x})}$
as follows, where $\dim(\mathbf{x})$ means the dimensionality of $\mathbf{x}$,
e.g., the number of pixels in the case of images or the number of columns in
the case of tabular data:

(3)
$\displaystyle\operatorname*{arg\,min}_{\theta}\mathbb{E}_{t}\mathbb{E}_{\mathbf{x}_{t}}\mathbb{E}_{\mathbf{x}_{0}}\Big{[}\lambda(t)\|S_{\theta}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\|_{2}^{2}\Big{]},$

where $\lambda(t)$ is to control the trade-off between synthesis quality and
likelihood. This training method is known as _denoising score matching_ and it
is known that $\theta^{*}$ solving Eq. (3) can correctly solve the reverse SDE
in Eq. (2). In other words, the optimal solution $\theta^{*}$ of the denoising
score matching training is equivalent to the optimal solution of the exact
score matching training — its proof is in (Vincent, 2011).

There exist several other improvements on SGMs. For instance, the adversarial
score matching model (Jolicoeur-Martineau et al., 2020) is to combine SGMs and
GANs. Therefore, the reverse SDE is not solely determined as a reverse of the
forward SDE but as a trade off between the denoising score matching and the
adversarial training of GANs (cf. Fig. 2 (a)).

(a) Adversarial score matching where the score network $S_{\theta}$ is trained
with i) the denoising score matching loss in Eq. (3) and ii) the adversarial
training. (b) Langevin corrector highlighted in purple

Figure 2. Two ideas further improving the synthesis quality of SGMs. (a) The
adversarial score matching model combining SGMs and GANs. (b) The correction
algorithm, called _Langevin corrector_ , where $\hat{\mathbf{x}}^{\prime}_{0}$
has a higher log probability than $\hat{\mathbf{x}}_{0}$ since we move
following the gradient of the log probability.

Once the training process is finished, one can synthesize fake data samples
with the _predictor-corrector_ framework. In the prediction phase, we solve
the reverse SDE in Eq. (2) after sampling a Gaussian noise. Let
$\hat{\mathbf{x}}_{0}$ be a solution of the reverse SDE. In the correction
phase, we then enhance it by using the Langevin corrector (Song et al., 2021).
The key idea of the correction phase is in Fig. 2 (b). Since we correct the
predicted sample $\hat{\mathbf{x}}_{0}$, following the direction of the
gradient of the log probability, the corrected sample has a higher log
probability. However, only the VE and VP models use this correction mechanism
whereas the Sub-VP model does not.

###  2.2. Tabular Data Synthesis

Generative adversarial networks (GANs) are the most popular deep learning-
based model for tabular data synthesis, and they generate fake samples by
training a generator and a discriminator in an adversarial way (Goodfellow et
al., 2014). Among many variants of GANs (Arjovsky et al., 2017; Gulrajani et
al., 2017; Park et al., 2018a; Adler and Lunz, 2018; PARIMALA and
Channappayya, 2019; Chen et al., 2016), WGAN is widely used providing a more
stable training of GANs (Arjovsky et al., 2017). Especially, WGAN-GP is one of
the most sophisticated model and is defined as follows (Gulrajani et al.,
2017):

$\displaystyle\begin{split}\mathop{min}_{\begin{subarray}{c}\mathtt{G}\end{subarray}}\mathop{max}_{\begin{subarray}{c}\mathtt{D}\end{subarray}}\mathbb{E}_{p_{d}}[\mathtt{D}(\mathbf{x})]-\mathbb{E}_{p_{z}}[\mathtt{D}(\mathtt{G}(\mathbf{z}))]-\lambda\mathbb{E}_{p_{\tilde{\mathbf{x}}}}[(\|\nabla_{\tilde{\mathbf{x}}}\mathtt{D}(\tilde{\mathbf{x}})\|_{2}-1)^{2}],\end{split}$

where $\mathtt{G}$ is a generator function, $\mathtt{D}$ is a Wasserstein
critic function, $p_{\mathbf{z}}$ is a prior distribution for a noisy vector
$\mathbf{z}$, which is typically $\mathcal{N}(\mathbf{0},\mathbf{1})$, $p_{d}$
is a data distribution, $\tilde{\mathbf{x}}$ is a randomly weighted
combination of $\mathtt{G}(\mathbf{z})$ and $\mathbf{x}$, and $\lambda$ is a
regularization weight.

Tabular data synthesis generates realistic table records by modelling a joint
probability distribution of features in a table. CLBN (Chow and Liu, 1968) is
a Bayesian network built by the Chow-Liu algorithm representing a joint
probability distribution. TVAE (Xu et al., 2019) is a variational autoencoder
that effectively handles mixed types of features in tabular data. There are
also many variants of GANs for tabular data synthesis. RGAN (Esteban et al.,
2017) generates continuous time-series records while MedGAN (Choi et al.,
2017) generates discrete medical records using non-adversarial loss. TableGAN
(Park et al., 2018b) generates tabular data using convolutional neural
networks. PATE-GAN (Jordon et al., 2019) was proposed to synthesize
differentially private tabular records.

However, one notorious challenge in synthesizing tabular data with GANs is
addressing the issue of mode collapse. In many cases, the fake data
distribution synthesized by GANs is confined to only a few modes and fails to
represent many others. Some variants of GANs tried to resolve this problem,
proposing to force the generator to produce samples from the various modes as
evenly as possible. VEEGAN (Srivastava et al., 2017) has a reconstructor
network, which maps the output of the generator to the noisy vector space, and
jointly trains the generator and the reconstructor network. CTGAN (Xu et al.,
2019) has a conditional generator and use a mode separation process. OCT-GAN
(Kim et al., 2021) exploits the homeomorphic characteristic of neural ordinary
differential equations, when designing its generator, and now shows the state-
of-the-art synthesis quality for many tabular datasets. However, one downside
is that it requires a much longer training time than other models.

Table 1. The qualitative comparison among our method and other methods. There are no existing minor-class oversampling models that adopt the score-based generative model. Moreover, many of the existing minor-class oversampling methods are for images. In our experiments, we modify some of them to be able to process tabular data and compare with our method. Method | Multiple Generators | Generative Model Type | Domain
---|---|---|---
SMOTE | N/A | Classical Method | Tabular
B-SMOTE | N/A | Classical Method | Tabular
ADASYN | N/A | Classical Method | Tabular
BAGAN | X | GAN | Image
cWGAN | X | GAN | Tabular
GL-GAN | X | GAN | Tabular
GAMO | O | GAN | Image
SOS (Ours) | O (Score Networks) | Score-based Model | Tabular

###  2.3. Minor-class Oversampling

Multi-class data commonly suffers from imbalanced classes, and two different
approaches have been proposed to address the issue; classical statistical
methods and deep learning-based methods.

Classical statistical methods are mostly based on the synthetic minority
oversampling technique (SMOTE (Chawla et al., 2002)) method, which generates
synthetic samples by interpolating two minor samples. Borderline SMOTE
(B-SMOTE (Han et al., 2005)) improves SMOTE to consider only the minor samples
on the class boundary where the major samples occupy more than half of the
neighbors. The adaptive synthetic sampling (ADASYN (He et al., 2008)) applies
SMOTE on the minority samples close to the major samples.

Deep learning-based methods, based on GANs, mostly aim at oversampling minor
class in images. In ACGAN (Odena et al., 2017), a generator generates all
classes including the minor ones, and a discriminator returns not only whether
the input is fake or real but also which class it belongs to. Inspired by
ACGAN, BAGAN (Mariani et al., 2018) constructs the discriminator which
distinguishes the fake from the rest of the classes at once, showing good
performance for minor-class image generation. cWGAN (Engelmann and Lessmann,
2021) is a WGAN-GP based tabular data oversampling model, which is equipped
with an auxiliary classifier to consider the downstream classification task.
GL-GAN (Wang et al., 2020) is a GAN that employs an autoencoder and SMOTE to
learn and oversample the latent vectors and includes an auxiliary classifier
to provide label information to the networks. GAMO (Mullick et al., 2019) uses
multiple generators, each of which is responsible for generating a specific
class of data, and two separate critics; one for the real/fake classification
and another for the entire class classification.

##  3\. Proposed Method

We describe our proposed score-based oversampling method. We first sketch the
overall workflow of our method, followed by its detailed designs.

(a) Option 1: Style transfer-based oversampling which corresponds to
_oversampling around class boundary_ (b) Option 2: Plain score-based
oversampling

Figure 3. Another diagram showing our model architectures. In this figure, we
assume multiple minor classes. Given a set of classes
$\\{\mathcal{C}_{m}\\}_{m=1}^{M}$, suppose that a target minor class is
$\mathcal{C}_{1}$, for which we aim at oversampling. (a) We randomly choose
$\mathbf{x}^{+}_{0}$ from other non-target classes
$\\{\mathcal{C}_{m}\\}_{m=2}^{M}$. (b) We randomly sample a noisy vector
$\mathbf{z}$.

###### Definition 3.1.

Let $\mathcal{T}$ be a table (or tabular data) consisting of $M$ classes,
$\\{\mathcal{C}_{m}\\}_{m=1}^{M}$, where $\mathcal{C}_{m}$ is $m$-th class
(subset) of $\mathcal{T}$, i.e., classes are disjoint subsets of
$\mathcal{T}$. We define a minor class as $\mathcal{C}_{j}$ whose cardinality
is smaller than the maximum cardinality, i.e.,
$|\mathcal{C}_{j}|<\max_{m}|\mathcal{C}_{m}|$. In general, therefore, we have
one major (with the largest cardinality) and multiple minor classes in
$\mathcal{T}$. Our task is to oversample those minor classes, i.e., add
artificial fake minor records, until
$|\mathcal{C}_{j}|=\max_{m}|\mathcal{C}_{m}|$ for all $j$.

###  3.1. Overall Workflow

Our proposed score-based oversampling, shown in Fig. 3, is done by the
following workflow:

  1. (1)

We train a score-based generative model for each class $\mathcal{C}_{m}$. Let
$S_{\theta_{m}}$ be a trained score network for $\mathcal{C}_{m}$.

  2. (2)

Let $\mathcal{C}_{j}$ be a minor class to oversample, i.e., target class. We
fine-tune $S_{\theta_{j}}$ using all records of $\mathcal{T}$ as shown in Fig.
1 (c).

  3. (3)

In order to oversample each target (minor) class $\mathcal{C}_{j}$, we use one
of the following two options as shown in Fig. 3. We repeat this until
$|\mathcal{C}_{j}|=\max_{m}|\mathcal{C}_{m}|$.

    1. (a)

After feeding a record randomly chosen from all other non-target classes
except $\mathcal{C}_{j}$, we derive its noisy vector. By using
$S_{\theta_{j}}$, we synthesize a fake target (minor) record for
$\mathcal{C}_{j}$. (cf. Fig. 3 (a)).

    2. (b)

After sampling a noisy vector $\mathbf{z}\sim\mathcal{N}(\mu,\sigma^{2})$, we
generate a fake target (minor) record (cf. Fig. 3 (b)).

  4. (4)

After generating a fake record for the target class $\mathcal{C}_{j}$, we can
apply the Langevin corrector, as noted in Fig. 2 (b). However, this step is
optional and for some types of SGMs, we also do not use this step sometimes —
see Appendix A for its detailed settings in each dataset for our experiments.
For instance, the type of Sub-VP does not use this step for reducing
computation in (Song et al., 2021) since it shows good enough quality even
without the corrector.

###### Definition 3.2.

Let $\mathcal{C}_{j}$ be a target minor class, for which we aim at
oversampling. $\mathbf{x}^{-}_{0}\in\mathcal{C}_{j}$ is a target (minor)
record in the target class.

###### Definition 3.3.

Let $\mathcal{C}_{j}$ be a target minor class, for which we aim at
oversampling. $\mathbf{x}^{+}_{0}\in\mathcal{C}_{i}$, where $i\neq j$, is a
record in other non-targeted classes. In the case of binary classification,
$\mathcal{C}_{j}$ is a minor class and $\mathcal{C}_{i}$ is a major class. In
general, however, $\mathbf{x}^{+}_{0}$ means a record from other non-target
classes. For ease of our convenience for writing, we use the symbol ‘$+$’ to
denote all other non-targeted classes. Fig. 3 follows this notation.

In this process, our main contributions lie in i) designing the score network
$S_{\theta_{m}}$ for each class $\mathcal{C}_{m}$, and ii) designing the fine-
tuning method. Note that the fine-tuning happens after all score networks are
trained.

##### Option 1 vs. Option 2

In Fig. 3, we show two options of how to sample a noisy vector. In Fig. 3 (a),
we perturb $\mathbf{x}^{+}_{0}$ in a non-target class to generate
$\hat{\mathbf{x}}^{-}_{0}$. This strategy can be considered as _oversampling
around class boundary_. In Fig. 3 (b), we sample from random noisy vectors and
therefore, there is no guarantee that $\hat{\mathbf{x}}^{-}_{0}$ is around
class boundary. We refer to Fig. 6 for its intuitive visualization result.

One more advantage of the first option is that we do not require that
$\mathbf{x}^{+}_{T}$ follows a Gaussian distribution. Regardless of the
distribution of $\mathbf{x}^{+}_{T}$, we can derive $\hat{\mathbf{x}}^{-}_{0}$
using the reverse SDE with $S_{\theta_{-}}$ — recall that we do not sample the
noisy vector from a Gaussian in the first option but from $\mathbf{x}^{+}_{0}$
using the forward SDE. Therefore, we can freely choose a value for $T$, which
is one big advantage in our design since a large $T$ increases computational
overheads. For the second option, we also found that small $T$ settings work
well as in (Rasul et al., 2021).

The relationship between the first and the second option is analogous to that
between Borderline-SMOTE (or B-SMOTE) and SMOTE. In this work, we propose
those two mechanisms for increasing the usability of our method.

###  3.2. Score-based Model for Tabular Data

We use the forward and reverse SDEs in Eqs. (1) and (2). As noted earlier,
there exist three types of SGMs, depending on the types of $f$ and $g$ in Eqs.
(1) and (2). We introduce the definitions of $f$ and $g$ in Appendix B and the
proposed architecture for our score network in Appendix C.

Since the original score network design is for images, we redesign it with
various techniques used by tabular data synthesis models (Xu et al., 2019; Lee
et al., 2021; Grathwohl et al., 2018). Its details are in Appendix C. However,
we train this score network using the denoising score matching loss as in the
original design.

Our method provides two options for oversampling $\mathcal{C}_{j}$. In the
first option, we adopt the style transfer-based architecture, where i) we
choose $\mathbf{x}^{+}_{0}$ from $\mathcal{C}_{i}$, $i\neq j$, ii) we derive a
noisy vector from it using the forward SDE, and iii) we finally transfer the
noisy vector to a fake target record. The rationale behind this design is that
the forward SDE is a pure Brownian motion regardless of classes — in other
words, all classes share the same forward SDE and thereby, share the same
noisy space as well. Therefore, $\mathbf{x}^{+}_{T}$ contains small
information on its original record $\mathbf{x}^{+}_{0}\in\mathcal{C}_{i}$,
which is still effective when it comes to generating a fake target record for
$\mathcal{C}_{j}$. Thus, this option can be considered as oversampling around
class boundary.

In the second option, we follow the standard use of SGMs, where we sample
$\mathbf{z}\sim\mathcal{N}(\mu,\sigma^{2})$ and generate a fake target record.
We note that the Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$ is
defined by the types of SDEs, i.e., VE, VP, or Sub-VP, as follows:

(4)
$\displaystyle\mathbf{z}\sim\begin{cases}\mathcal{N}(\mathbf{0},\sigma^{2}_{max}),&\textrm{
if VE,}\\\ \mathcal{N}(\mathbf{0},\mathbf{1}),&\textrm{ if VP,}\\\
\mathcal{N}(\mathbf{0},\mathbf{1}),&\textrm{ if Sub-VP,}\\\ \end{cases}$

where $\sigma_{max}$ is a hyperparameter.

Figure 4. Suppose that we want to fine-tune $\theta_{j}$ for class
$\mathcal{C}_{j}$. At $(\mathbf{x}_{t},t)$, the two gradients evaluated with
Eq. (5), $\mathbf{g}^{j}_{\mathbf{x},t}$ and $\mathbf{g}^{i}_{\mathbf{x},t}$,
create a small angle. Then, we suppress the denoising process (or reverse SDE)
toward $\mathbf{g}^{j}_{\mathbf{x},t}$ by a factor of $w$, as noted in Eq.
(6), to avoid the gray region where both target and non-target class records
are likely to co-exist.

###  3.3. Fine-tuning Method

Once score networks $\\{S_{\theta_{m}}\\}_{m=1}^{M}$ are trained, we can
sample fake records for all target classes as described earlier. In the case
of tabular data, however, we can exploit class information to further enhance
the sampling quality. Our proposed fine-tuning procedures are as follows — for
simplicity, we describe how to fine-tune for $\mathcal{C}_{j}$ only. However,
we repeat the following method for each $j$:

  1. (1)

Firstly, given a record $\mathbf{x}\in\mathcal{T}$ regardless of its class
label and a chosen time $t=\epsilon_{t}$111$\epsilon_{t}$ is a small value
close to 0, and the range $[0,\epsilon_{t}]$ means the last moment of the
reverse SDE in Eq. (2), which we will fine-tune., we evaluate with each score
network $S_{\theta_{m}}$ to know the gradient of the log probability w.r.t.
$\mathbf{x}_{t}$ at time $t$, i.e., time-dependant score, as follows:

(5)
$\displaystyle\mathbf{g}^{m}_{\mathbf{x},t}=S_{\theta_{m}}\bigg{\rvert}_{\mathbf{x}_{t},t},\forall
m,$

where $\mathbf{g}^{m}_{\mathbf{x},t}$ is the gradient evaluated at
$(\mathbf{x}_{t},t)$ with the score network $S_{\theta_{m}}$. The long
vertical bar means the ‘evaluated-at’ operator.

    1. (a)

Our goal is to fine-tune the model around $(\mathbf{x}_{t},t)$. Therefore, we
add a random noisy vector to $\mathbf{x}$.

  2. (2)

Secondly, we compare the angle between $\mathbf{g}^{j}_{\mathbf{x},t}$ and
$\mathbf{g}^{i}_{\mathbf{x},t}$ where $i\neq j$. Suppose that the angle
$arccos\Big{(}\frac{\mathbf{g}^{j}_{\mathbf{x},t}\cdot\mathbf{g}^{i}_{\mathbf{x},t}}{\|\mathbf{g}^{j}_{\mathbf{x},t}\|\|\mathbf{g}^{i}_{\mathbf{x},t}\|}\Big{)}$
is large enough. It then says that $\mathcal{C}_{j}$ and $\mathcal{C}_{i}$ can
be well separated around the location and time of $(\mathbf{x},t)$, which is a
preferred situation. However, a problematic situation happens when the angle
is smaller than a threshold $0<\xi<\pi$ — in other words, their directions are
similar. In such a case, we use the following loss to fine-tune the gradient
$S_{\theta_{j}}$ at $(\mathbf{x}_{t},t)$:

(6) $\displaystyle
L(\mathbf{x},t,j)=\|S_{\theta_{j}}(\mathbf{x}_{t},t)-w\mathbf{g}^{j}_{\mathbf{x},t}\|_{2}^{2},$

where $0<w<1$. In other words, we want to decrease the gradient by a factor of
$w$.

The rationale behind the proposed fine-tuning method is that by decreasing the
scale of the gradient by a factor of $w$ at $(\mathbf{x}_{t},t)$, we can
prevent that the reverse SDE process moves too much toward the _gray region_
where both target and non-target class records are mixed (cf. Fig. 4). By
controlling the damping coefficient $w$, we control how aggressively we
suppress the direction.

1

2Initialize $\theta_{m}$ for all $1\leq m\leq M$

3

4for  _each class $\mathcal{C}_{m}$_ do

5 Train its score network $S_{\theta_{m}}$ using the denoising score matching
loss in Eq. (3)

6

/* Let $\mathcal{C}_{i}$ be the major class, where
$i=\operatorname*{arg\,max}_{m}|\mathcal{C}_{m}|$. */

7 for  _each record $\mathbf{x}\in\mathcal{T}$_ do

8 for  _each minor class $\mathcal{C}_{j}$_ do

9 if
_$arccos\Big{(}\frac{\mathbf{g}^{j}_{\mathbf{x},t}\cdot\mathbf{g}^{i}_{\mathbf{x},t}}{\|\mathbf{g}^{j}_{\mathbf{x},t}\|\|\mathbf{g}^{i}_{\mathbf{x},t}\|}\Big{)}
<\xi$_ then

10 Fine-tune $\theta_{j}$ with $L(\mathbf{x},t,j)$ in Eq. (6)

11

12for  _each minor class $\mathcal{C}_{j}$_ do

13 Oversample fake minor records with Option 1 or 2 for $\mathcal{C}_{j}$
until $|\mathcal{C}_{j}|=\max_{m}|\mathcal{C}_{m}|$

14

return oversampled minor classes

Algorithm 1 How to train our proposed model

###  3.4. Training Algorithm

Algorithm 1 shows the overall training process for our method. At the
beginning, we initialize the score network parameters for all $m$. We then
train each score network $S_{\theta_{m}}$ using each $\mathcal{C}_{m}$ in Line
1. At this step, we use the standard denoising score matching loss in Eq. (3).
After finishing this step, we can already generate fake target records.
However, we further enhance each score network in Line 1. The complexity of
the proposed fine-tuning is $\mathcal{O}(|\mathcal{T}|m)$, which is affordable
in practice since $m$, the number of classes, is not large in comparison with
the table size $|\mathcal{T}|$. For oversampling $\mathcal{C}_{j}$, we can use
either the first or the second option of our model designs in Fig. 3. They
correspond to i) oversampling around class boundary or ii) regular
oversampling, respectively.

##  4\. Experiments

We describe the experimental environments and results. In Appendix E, we
introduce our additional experiments that we generate entire fake tabular data
(instead of making imbalanced tabular data balanced after oversampling minor
classes). We also analyze the space and time overheads of our method in
Appendix D.

###  4.1. Experimental Environments

####  4.1.1. Datasets

We describe the tabular datasets that are used for our experiments. Buddy and
Satimage are for multi-class classification, and the others for binary
classification.

Default (Lichman, 2013) is a dataset describing the information on credit card
clients in Taiwan regarding default payments, demographic factors, credit
data, and others. The ‘default payment’ column includes 23,364 (77.9$\%$) of
major records and 6,636 (22.1$\%$) minor records. Shoppers (Sakar et al.,
2019) is a binary classification dataset, where out of 12,330 sessions, 10,422
(84.5$\%$) are negative records which did not end with purchase, and the rest
1,908 (15.5$\%$) are positive records ending with purchase. Surgical (Sessler
et al., 2011) is a binary classification dataset, which contains general
surgical patient information. There are 10,945 (74.8$\%$) records for the
major class, and 3,690 (25.2$\%$) records for the minor class. WeatherAUS
(wea, logy) contains 10 years of daily weather observations from several
weather stations in Australia. ‘RainTomorrow’ is the target variable to
classify, where 43,993 (78.0$\%$) records belong to the major class, which
means the rain for that day was over 1mm, and 12,427 (22.0$\%$) records belong
to the minor class.

Buddy (bud, uddy) consists of parameters, including a unique ID, assigned to
each animal that is up for adoption, and a variety of physical attributes. The
target variable is ‘breed’, which includes 10,643 (56.5$\%$) records for the
major class, and 7,021 (37.3$\%$) and 1,170 (6.2$\%$) records for minor
classes. Satimage (Dua and Graff, 2017) is to classify types of land usage,
which is created by the Australian Centre for Remote Sensing. The dataset
contains 2,282 (35.8$\%$) records for the major class, and 1,336 (21.0$\%$),
1,080 (16.9$\%$), 562 (8.8$\%$), 573 (9.0$\%$), and 539 (8.5$\%$) records for
minor classes.

####  4.1.2. Baselines

We use a set of baselines as follows, which include statistical methods and
generative models based on deep learning:

Identity is the original tabular data without oversampling. SMOTE (Chawla et
al., 2002), B-SMOTE (Han et al., 2005), and Adasyn (He et al., 2008) are
classical methods of data augmentation for the minority class; MedGAN (Choi et
al., 2017), VEEGAN (Srivastava et al., 2017), TableGAN (Park et al., 2018b),
and CTGAN (Xu et al., 2019) are GAN models for tabular data synthesis; TVAE
(Xu et al., 2019) is proposed to learn latent embedding in VAE by
incorporating deep learning metric. BAGAN (Mariani et al., 2018) is a type of
GAN model for oversampling images, but we replace its generator and
discriminator with ours for tabular data synthesis. OCT-GAN (Kim et al., 2021)
is one of the state of the art model for tabular data systhesis.

We exclude cWGAN (Engelmann and Lessmann, 2021) and GL-GAN (Wang et al.,
2020), since they do not support multiple minority cases and synthesis quality
is also relatively lower than others in our preliminary experiments.

####  4.1.3. Hyperparameters

We refer to Appendix A for detailed hyperparameter settings. We also provide
all our codes and data for reproducibility and one can easily reproduce our
results.

####  4.1.4. Evaluation Methods

Let $\mathcal{C}_{j}$ be a minor class, where
$|\mathcal{C}_{j}|<\max_{m}|\mathcal{C}_{m}|$. As mentioned earlier, our task
is to oversample $\mathcal{C}_{j}$ until
$|\mathcal{C}_{j}|=\max_{m}|\mathcal{C}_{m}|$ for all $j$. However, some
baselines are not for oversampling minor classes but generating one entire
fake tabular data, in which case we train such a baselines for each
$\mathcal{C}_{j}$ — in other words, one dedicated baseline model for each
minor class. For our method, we follow the described process.

For evaluation, we first train various classification algorithms with the
augmented (or oversampled) training data — we note that the augmented data is
fully balanced after oversampling — including Decision Tree (Breiman et al.,
1984), Logistic Regression (Cox, 1958), AdaBoost (Schapire, 1999), and MLP
(Bishop, 2006). We then conduct classification with the testing data. We take
the average of the results — the same evaluation scenario had been used in (Xu
et al., 2019; Lee et al., 2021) and we strictly follow their evaluation
protocols. We repeat with 5 different seed numbers and report their mean and
std. dev. scores. We use the weighted F1 since the testing data is also
imbalanced. We visualize the column-wise histogram of original and fake
records and use t-SNE (van der Maaten and Hinton, 2008) to display them.

Table 2. Experimental results (Weighted F1). The best results are in boldface — if tie, the winner is determined by comparing in the 4th decimal place. | Methods |  |  | Single Minority |  |  | Multiple Minority |  |
---|---|---|---|---|---|---|---|---|---
|  |  | Default | Shoppers | Surgical | WeatherAUS |  |  | Buddy | Satimage |  |
Identity |  |  | 0.515±0.035 | 0.601±0.039 | 0.687±0.004 | 0.657±0.016 |  |  | 0.603±0.010 | 0.817±0.004 |  |
Baselines | SMOTE |  |  | 0.561±0.025 | 0.648±0.004 | 0.678±0.008 | 0.674±0.025 |  |  | 0.584±0.005 | 0.846±0.005 |  |
B-SMOTE |  |  | 0.561±0.029 | 0.640±0.042 | 0.671±0.004 | 0.663±0.022 |  |  | 0.595±0.003 | 0.845±0.005 |  |
Adasyn |  |  | 0.558±0.023 | 0.630±0.045 | 0.662±0.007 | 0.658±0.022 |  |  | 0.608±0.002 | 0.841±0.008 |  |
MedGAN |  |  | 0.532±0.028 | 0.620±0.062 | 0.686±0.003 | 0.656±0.022 |  |  | 0.598±0.011 | 0.835±0.019 |  |
VEEGAN |  |  | 0.495±0.076 | 0.607±0.065 | 0.680±0.117 | 0.661±0.025 |  |  | 0.555±0.036 | 0.840±0.031 |  |
TableGAN |  |  | 0.423±0.115 | 0.571±0.097 | 0.704±0.001 | 0.579±0.066 |  |  | 0.570±0.019 | 0.813±0.013 |  |
TVAE |  |  | 0.536±0.035 | 0.610±0.060 | 0.681±0.004 | 0.652±0.018 |  |  | 0.552±0.044 | 0.846±0.031 |  |
CTGAN |  |  | 0.545±0.022 | 0.605±0.059 | 0.701±0.004 | 0.659±0.020 |  |  | 0.593±0.009 | 0.833±0.015 |  |
OCT-GAN |  |  | 0.531±0.018 | 0.639±0.029 | 0.692±0.082 | 0.656±0.018 |  |  | 0.551±0.015 | 0.837±0.011 |  |
BAGAN |  |  | 0.525±0.005 | 0.610±0.005 | 0.668±0.004 | 0.663±0.002 |  |  | 0.555±0.013 | 0.834±0.011 |  |
SOS | VE |  |  | 0.571±0.003 | 0.675±0.004 | 0.709±0.003 | 0.672±0.002 |  |  | 0.607±0.007 | 0.854±0.002 |  |
VP |  |  | 0.559±0.006 | 0.658±0.003 | 0.712±0.002 | 0.680±0.002 |  |  | 0.607±0.011 | 0.857±0.006 |  |
Sub-VP |  |  | 0.574±0.003 | 0.673±0.002 | 0.714±0.001 | 0.680±0.003 |  |  | 0.608±0.002 | 0.855±0.004 |  |

###  4.2. Experimental Results

We compare various oversampling methods in Table 2. As mentioned earlier,
Identity means that we do not oversampling minor classes, train classifiers,
and test with testing data. Therefore, the score of Identity can be considered
as a minimum requirement upon oversampling.

Classical methods, such as SMOTE, B-SMOTE, and Adasyn, show their
effectiveness to some degree. For instance, SMOTE improves the test score from
0.515 to 0.561 after oversampling in Default. All these methods, however, fail
to increase the test score in Surgical. SMOTE and B-SMOTE fail in Buddy as
well.

VEEGAN and TableGAN are two early GAN-based methods and their test scores are
relatively lower than other advanced methods, such as CTGAN and OCT-GAN.
However, TableGAN shows the best result among all baselines for Surgical.
MedGAN fails in Surgical, WeatherAUS, and Buddy. TVAE also fails in multiple
cases. Among the baselines, CTGAN and OCT-GAN show the best outcomes in many
cases and in general, their synthesis quality looks stable. In general, those
deep learning-based methods show better effectiveness than the classical
methods.

However, our method, SOS, clearly outperforms all those methods in all cases
by large margins. In Default, the original score without oversampling, i.e.,
Identity, is 0.515, and the best baseline score is 0.561 by SMOTE. However,
our method with Sub-VP achieves 0.574, which is about 11% improvement over
Identity. The biggest enhancement is achieved in Shoppers, i.e., a score of
0.601 by Identity vs. 0.675 by our method with VE. For other datasets, our
method marks significant enhancements after oversampling. In Buddy, moreover,
other methods except Adasyn and SOS oversample noisy fake records and decrease
the score below that of Identity.

###  4.3. Sensitivity on Hyperparameters

We summarize the sensitivity results w.r.t. some key hyperparameters in Table
3. We used $T=50$ in Table 2. Others for $T$ produce sub-optimal results in
Table 3. $\beta_{min}$ and $\beta_{max}$ means the lower and upper bounds of
$\beta(t)$ of the drift $f$ and the diffusion $g$. In general, all settings
produce reasonable outcomes, outperforming Identity. $\xi$ is the angle
threshold for our fine-tuning method. As shown in Appendix A, the best setting
for $\xi$ varies from one dataset to another but in general, $\xi\in[80,100]$
produces the best fine-tuning results. For $w$, we recommend a value close to
0.99 as shown in Table 3. $\epsilon_{t}=5e-04$ produces optimal results in
almost all cases.

Table 3. Results by some selected key hyperparameters Hyper |  | WeatherAUS |  |  | Satimage |
---|---|---|---|---|---|---
Param. |  | Setting |  | Weighted F1 |  |  | Setting |  | Weighted F1 |
$T$ |  | 10 |  |  0.679±0.002 |  |  | 10 |  |  0.841±0.008 |
| 50 |  | 0.680±0.003 |  |  | 50 |  | 0.857±0.006 |
| 100 |  |  0.670±0.001 |  |  | 100 |  |  0.846±0.004 |
| 150 |  |  0.660±0.002 |  |  | 150 |  |  0.847±0.005 |
| 200 |  |  0.666±0.002 |  |  | 200 |  |  0.845±0.002 |
| 300 |  |  0.669±0.002 |  |  | 300 |  |  0.849±0.004 |
$(\beta_{min},\beta_{max})$ |  | (0.01, 5.0) |  |  0.671±0.000 |  |  | (0.01, 1.0) |  |  0.843±0.002 |
| (0.01, 10.0) |  |  0.676±0.001 |  |  | (0.01, 10.0) |  | 0.848±0.003 |
| (0.1, 1.0) |  | 0.676±0.002 |  |  | (0.1, 1.0) |  |  0.820±0.004 |
| (0.1, 5.0) |  |  0.673±0.002 |  |  | (0.1, 5.0) |  |  0.834±0.008 |
| (0.1, 10.0) |  |  0.668±0.003 |  |  | (0.1, 10.0) |  |  0.834±0.014 |
$\xi$ |  | 80 |  |  0.679±0.003 |  |  | 70 |  |  0.783±0.016 |
| 90 |  | 0.680±0.002 |  |  | 90 |  | 0.854±0.005 |
$w$ |  | 0.99 |  |  0.680±0.003 |  |  | 0.99 |  | 0.856±0.006 |
| 0.90 |  | 0.680±0.003 |  |  | 0.90 |  |  0.854±0.006 |
$\epsilon_{t}$ |  | 5e-04 |  | 0.680±0.003 |  |  | 5e-04 |  | 0.857±0.006 |
| 1 |  |  0.661±0.002 |  |  | 1 |  |  0.850±0.004 |
| 2 |  |  0.661±0.002 |  |  | 2 |  |  0.857±0.004 |
| 3 |  |  0.661±0.003 |  |  | 3 |  |  0.854±0.002 |

###  4.4. Sensitivity on Boundary vs. Regular Oversampling

The first option of our sampling strategy (cf. Fig. 3 (a)) corresponds to
oversampling around class boundary, and the second option (cf. Fig. 3 (b))
corresponds to regular oversampling. We compare their key results in Table 4.
It looks like that there does not exist a clear winner, but the boundary
oversampling produces better results in more cases.

###  4.5. Ablation study on the reverse SDE solver (or predictor)

To solve $\hat{\mathbf{x}}^{-}_{0}$ from $\mathbf{x}^{+}_{T}$ with Eq. (2), we
can adopt various strategies. In general, the Euler-Maruyama (EM) method
(Platen, 1999) is popular in solving SDEs. However, one can choose different
SDE solvers, such as ancestral sampling (Ho et al., 2020), reverse diffusion
(Song et al., 2021), probability flow (Song et al., 2021) and so on. Following
the naming convection of (Song et al., 2021), we call them as _predictor_
(instead of solver) in this subsection. In the context of SGMs, therefore, the
predictor means the SDE solver to solve Eq. (2).

In Table 5, the Euler-Maruyama method leads to the best outcomes for Surgical.
For Shoppers, the ancestral sampling method produces the best outcomes.
Likewise, predictors depend on datasets. The probability flow also shows
reasonable results in our experiments. Especially, we observe that the
probability flow always outperforms other predictors in Buddy.

Table 4. Comparison between the boundary (Bnd.) vs. regular (Reg.) oversampling SDE |  | WeatherAUS |  |  | Default |  |  | Satimage |
---|---|---|---|---|---|---|---|---|---
Type |  | Bnd. | Reg. |  |  | Bnd. | Reg. |  |  | Bnd. | Reg. |
SOS (VE) |  |  0.670±0.002 |  0.672±0.002 |  |  |  0.569±0.001 |  0.571±0.002 |  |  |  0.854±0.004 |  0.854±0.002 |
SOS (VP) |  |  0.673±0.002 |  0.680±0.002 |  |  |  0.559±0.005 |  0.557±0.004 |  |  |  0.855±0.005 | 0.857±0.006 |
SOS (Sub-VP) |  | 0.680±0.003 |  0.651±0.002 |  |  | 0.574±0.003 |  0.574±0.002 |  |  |  0.854±0.005 |  0.855±0.004 |
Table 5. Results by the reverse SDE solver (or predictor). We consider the
predictors of Euler-Maruyama (EM), Ancestral Sampling (AS), Reverse Diffusion
(RD), and Probability Flow (PF).

Predictor |  |  | VE |  |  | VP |
---|---|---|---|---|---|---|---
|  | Pred. only | Pred. Corr. |  |  | Pred. only | Pred. Corr. |
EM |  |  |  0.709±0.002 | 0.709±0.003 |  |  |  0.710±0.004 | 0.712±0.002 |
AS |  |  |  0.706±0.002 |  0.707±0.003 |  |  |  0.706±0.004 |  0.711±0.002 |
RD |  |  |  0.701±0.003 |  0.708±0.003 |  |  |  0.707±0.002 |  0.711±0.001 |
PF |  |  |  0.704±0.004 |  |  |  0.712±0.002 |
(a) Surgical

Predictor |  |  | VE |  |  | VP |
---|---|---|---|---|---|---|---
|  | Pred. only | Pred. Corr. |  |  | Pred. only | Pred. Corr. |
EM |  |  |  0.668±0.003 |  0.664±0.005 |  |  |  0.648±0.007 |  0.651±0.005 |
AS |  |  | 0.675±0.006 |  0.661±0.003 |  |  |  0.646±0.002 | 0.657±0.002 |
RD |  |  |  0.673±0.005 |  0.671±0.006 |  |  |  0.640±0.004 |  0.655±0.004 |
PF |  |  |  0.674±0.004 |  |  |  0.640±0.005 |
(b) Shoppers

###  4.6. Ablation study on Adversarial Score Matching vs. Fine-tuning

We also compare the two enhancement strategies, the adversarial score matching
(cf. Fig. 2) and our fine-tuning, in Table 6. First of all, we found that the
adversarial score matching for tabular data is not as effective as that for
image data. The original method of the adversarial score matching had been
designed for images to integrate SGMs with GANs. We simply took the
discriminator of CTGAN, one of the best performing GAN models for tabular
data, and combine it with our model, when building the adversarial score
matching for our tabular data. Although there can be a better design, the
current implementation with the best existing discriminator model for tabular
data does not show good performance in our experiments. In many cases, it is
inferior to our method even without fine-tuning. However, our fine-tuning
successfully improves in almost all cases.

Table 6. Results of No Fine-tuning vs. Adversarial SGM vs. Fine-tuning SDE |  | Shoppers |  |  | Surgical |  |  | Default |
---|---|---|---|---|---|---|---|---|---
Type |  | No Fine-Tune | Adv. SGM | Fine-Tune |  |  | No Fine-Tune | Adv. SGM | Fine-Tune |  |  | No Fine-Tune | Adv. SGM | Fine-Tune |
SOS (VE) |  |  0.672±0.003 |  0.651±0.005 | 0.675±0.004 |  |  |  0.709±0.004 |  0.692±0.003 |  0.709±0.003 |  |  |  0.567±0.001 |  0.565±0.002 |  0.571±0.002 |
SOS (VP) |  |  0.652±0.004 |  0.651±0.005 |  0.658±0.002 |  |  |  0.708±0.002 |  0.689±0.001 |  0.712±0.002 |  |  |  0.558±0.004 |  0.557±0.004 |  0.559±0.006 |
SOS (Sub-VP) |  |  0.667±0.005 |  0.659±0.003 |  0.673±0.002 |  |  |  0.712±0.003 |  0.705±0.002 | 0.714±0.001 |  |  |  0.570±0.001 |  0.555±0.003 | 0.574±0.003 |

###  4.7. Visualization

We introduce several key visualization results, ranging from histograms of
columns to t-SNE plots.

####  4.7.1. Column-wise Histogram

Fig. 5 shows two histogram figures. The real histogram of values and the
histogram by our method are similar to each other in both figures. However,
CTGAN fails to capture the real distribution of the ‘bmi’ column of Surgical
in Fig. 5 (a), which we consider the well-known mode-collapse problem of GANs,
i.e., values in certain regions are more actively sampled than those in other
regions. In Fig. 5 (b), CTGAN’s histogram shows three peaks whereas the real
histogram has only one peak. Our method successfully captures it.

(a) Surgical (bmi) (b) WeatherAUS (temperature)

Figure 5. Column-wise Histogram Figure 6. t-SNE plot of real/fake records by
our method with boundary oversampling on WeatherAUS. Blue dots mean non-target
records, i.e., $\mathbf{x}^{+}_{0}$, and red dots mean fake target records,
i.e., $\hat{\mathbf{x}}^{-}_{0}$. Black arrows show which non-target records
are style-transferred to which target records.

####  4.7.2. t-SNE plot of fake/real records

In Fig. 6, we visualize real and fake records — we use SOS (Sub-VP) with the
boundary oversampling option in WeatherAUS. The translucent dots mean real
records, and the solid blue dots are transferred to the solid red dots in the
figure. As described earlier, those solid red dots are around the class
boundary.

##  5\. Conclusions & Limitations

Oversampling minor classes is a long-standing research problem in data mining
and machine learning. Many different methods have been proposed so far. In
this paper, however, we presented a score-based oversampling method, called
SOS. Since SGMs were originally invented for images, we carefully redesigned
many parts of them. For instance, we designed a neural network to approximate
the score function of the forward SDE and a fine-tuning method. In our
experiments with 6 datasets and 10 baselines, our method overwhelms other
oversampling methods in all cases. Moreover, our method does not decrease the
F1 score after oversampling in all cases whereas existing methods fail to do
so from time to time. From all these facts, we think that our method makes a
great step forward in oversampling minor classes.

Even though SGMs perform well in general, they are still in an early phase
when it comes to generating tabular data, and we hope that our work will bring
much follow-up research. For instance, one can invent a drift and/or a
diffusion function different from the three existing types and specialized to
tabular data.

###### Acknowledgements.

Jayoung Kim and Chaejeong Lee equally contributed. Noseong Park is the
corresponding author. This work was supported by the Institute of Information
& Communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (10% from No. 2020-0-01361, Artificial Intelligence
Graduate School Program at Yonsei University and 70% from No. 2021-0-00231,
Development of Approximate DBMS Query Technology to Facilitate Fast Query
Processing for Exploratory Data Analysis) and the National Research Foundation
of Korea (NRF) grant funded by the Korea government (MSIT) (20% from No.
2021R1F1A1063981).

## References

  * (1)
  * wea (logy) Commonwealth of Australia 2010 Bureau of Meteorology.  https://www.kaggle.com/jsphyg/weather-dataset-rattle-package.
  * bud (uddy) HackerEarth Machine Learning Challenge—Adopt a buddy.  https://www.kaggle.com/akash14/adopt-a-buddy.
  * Adler and Lunz (2018) Jonas Adler and Sebastian Lunz. 2018.  Banach Wasserstein GAN.  In _NeurIPS_.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017\.  Wasserstein Generative Adversarial Networks. In _ICML_.
  * Bishop (2006) Christopher M. Bishop. 2006\.  _Pattern Recognition and Machine Learning (Information Science and Statistics)_.  Springer-Verlag.
  * Breiman et al. (1984) L. Breiman, J. Friedman, C.J. Stone, and R.A. Olshen. 1984\.  _Classification and Regression Trees_.  Taylor & Francis.  https://books.google.co.kr/books?id=JwQx-WOmSyQC
  * Chawla et al. (2002) Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002.  SMOTE: Synthetic Minority over-Sampling Technique.  _J. Artif. Int. Res._ 16, 1 (2002).
  * Chen et al. (2016) Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016\.  InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. In _NeurIPS_.
  * Choi et al. (2017) Edward Choi, Siddharth Biswal, A. Bradley Maline, Jon Duke, F. Walter Stewart, and Jimeng Sun. 2017\.  Generating Multi-label Discrete Electronic Health Records using Generative Adversarial Networks.  (2017).  arXiv:1703.06490
  * Chow and Liu (1968) C. Chow and C. Liu. 1968.  Approximating discrete probability distributions with dependence trees.  _IEEE Transactions on Information Theory_ 14, 3 (1968), 462–467.
  * Cox (1958) David R Cox. 1958\.  The regression analysis of binary sequences.  _Journal of the Royal Statistical Society: Series B (Methodological)_ 20, 2 (1958), 215–232.
  * Dockhorn et al. (2022) Tim Dockhorn, Arash Vahdat, and Karsten Kreis. 2022\.  Score-Based Generative Modeling with Critically-Damped Langevin Diffusion. In _ICLR_.
  * Dua and Graff (2017) Dheeru Dua and Casey Graff. 2017.  UCI Machine Learning Repository.  http://archive.ics.uci.edu/ml
  * Engelmann and Lessmann (2021) Justin Engelmann and S. Lessmann. 2021.  Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning.  _Expert Syst. Appl._ 174 (2021), 114582.
  * Esteban et al. (2017) Cristóbal Esteban, L. Stephanie Hyland, and Gunnar Rätsch. 2017.  Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs.  arXiv:1706.02633
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets.  In _NeurIPS_.
  * Grathwohl et al. (2018) Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. 2018.  Ffjord: Free-form continuous dynamics for scalable reversible generative models.  _arXiv preprint arXiv:1810.01367_ (2018).
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017\.  Improved Training of Wasserstein GANs. In _NeurIPS_.
  * Han et al. (2005) Hui Han, Wen-Yuan Wang, and Bing-Huan Mao. 2005.  Borderline-SMOTE: A New over-Sampling Method in Imbalanced Data Sets Learning. In _ICIC_.
  * He et al. (2008) Haibo He, Yang Bai, Edwardo A. Garcia, and Shutao Li. 2008\.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In _IJCNN_.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising Diffusion Probabilistic Models. In _NeurIPS_.
  * Jolicoeur-Martineau et al. (2020) Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioannis Mitliagkas. 2020\.  Adversarial score matching and improved sampling for image generation.  _arXiv preprint arXiv:2009.05475_ (2020).
  * Jordon et al. (2019) James Jordon, Jinsung Yoon, and V. D. Mihaela Schaar. 2019\.  PATE-GAN: Generating Synthetic Data with Differential Privacy Guarantees. In _International Conference on Learning Representations_.
  * Kim et al. (2021) Jayoung Kim, Jinsung Jeon, Jaehoon Lee, Jihyeon Hyeong, and Noseong Park. 2021.  OCT-GAN: Neural ODE-Based Conditional Tabular GANs. In _TheWebConf_.
  * Lee et al. (2021) Jaehoon Lee, Jihyeon Hyeong, Jinsung Jeon, Noseong Park, and Jihoon Cho. 2021.  Invertible Tabular GANs: Killing Two Birds with One Stone for Tabular Data Synthesis. In _NeurIPS_.
  * Lichman (2013) M. Lichman. 2013\.  UCI Machine Learning Repository.  http://archive.ics.uci.edu/ml
  * Mariani et al. (2018) Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, and A. Cristiano I. Malossi. 2018\.  BAGAN: Data Augmentation with Balancing GAN.  _CoRR_ abs/1803.09655 (2018).
  * Mullick et al. (2019) Sankha Subhra Mullick, Shounak Datta, and Swagatam Das. 2019\.  Generative Adversarial Minority Oversampling. In _ICCV_.
  * Odena et al. (2017) Augustus Odena, Christopher Olah, and Jonathon Shlens. 2017\.  Conditional Image Synthesis With Auxiliary Classifier GANs.  arXiv:1610.09585
  * PARIMALA and Channappayya (2019) KANCHARLA PARIMALA and Sumohana Channappayya. 2019.  Quality Aware Generative Adversarial Networks. In _NeurIPS_.
  * Park et al. (2018a) Noseong Park, Ankesh Anand, Joel Ruben Antony Moniz, Kookjin Lee, Jaegul Choo, David Keetae Park, Tanmoy Chakraborty, Hongkyu Park, and Youngmin Kim. 2018a.  MMGAN: Manifold-Matching Generative Adversarial Networks. In _ICPR_.
  * Park et al. (2018b) Noseong Park, Mahmoud Mohammadi, Kshitij Gorde, Sushil Jajodia, Hongkyu Park, and Youngmin Kim. 2018b.  Data Synthesis based on Generative Adversarial Networks.  (2018).  arXiv:1806.03384
  * Platen (1999) Eckhard Platen. 1999\.  An introduction to numerical methods for stochastic differential equations.  _Acta Numerica_ 8 (1999), 197–246.  https://doi.org/10.1017/S0962492900002920
  * Rasul et al. (2021) Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. 2021.  Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting. In _ICML_.
  * Sakar et al. (2019) C Okan Sakar, S Olcay Polat, Mete Katircioglu, and Yomi Kastro. 2019.  Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and LSTM recurrent neural networks.  _Neural Computing and Applications_ 31 (10 2019), 6893–6908.
  * Schapire (1999) Robert E. Schapire. 1999\.  A Brief Introduction to Boosting. In _IJCAI_.
  * Sessler et al. (2011) Daniel Sessler, Andrea Kurz, Leif Saager, and Jarrod Dalton. 2011\.  Operation Timing and 30-Day Mortality After Elective General Surgery.  _Anesthesia and analgesia_ 113 (09 2011), 1423–8.  https://doi.org/10.1213/ANE.0b013e3182315a6d
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021.  Score-Based Generative Modeling through Stochastic Differential Equations. In _ICLR_.
  * Srivastava et al. (2017) Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. 2017\.  VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning.  In _NeurIPS_.
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing Data using t-SNE.  _Journal of Machine Learning Research_ 9, 86 (2008).
  * Vincent (2011) Pascal Vincent. 2011\.  A Connection between Score Matching and Denoising Autoencoders.  _Neural Comput._ 23, 7 (2011), 1661–1674.
  * Wang et al. (2020) Wentao Wang, Suhang Wang, Wenqi Fan, Zitao Liu, and Jiliang Tang. 2020.  Global-and-local aware data generation for the class imbalance problem. In _ICDM_. SIAM, 307–315.
  * Xiao et al. (2022) Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. 2022\.  Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In _ICLR_.
  * Xu et al. (2019) Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. 2019.  Modeling Tabular data using Conditional GAN.  In _NeurIPS_.

##  Appendix A Settings & Reproducibility

Our software and hardware environments are as follows: Ubuntu 18.04 LTS,
Python 3.8.2, Pytorch 1.8.1, CUDA 11.4, and NVIDIA Driver 470.42.01, i9 CPU,
and NVIDIA RTX 3090. Our codes and data are at
https://github.com/JayoungKim408/SOS.

In Tables 7 and 8, we list the best hyperparameters. We have three layer types
as shown in Appendix C: Concat, Squash, and Concatsquash (Grathwohl et al.,
2018). $(\beta_{min},\beta_{max})$ (resp. $(\sigma_{min},\sigma_{max})$) are
two hyperparameters of the function $\beta(t)$ (resp. $\sigma(t)$). In fact,
the full writing of $\beta$ and $\sigma$ are
$\beta(t;\beta_{min},\beta_{max})$ and $\sigma(t;\sigma_{min},\sigma_{max})$,
respectively. We search for (min, max), in total, with 9 combinations using
$min=\\{0.01,0.1,0.5\\}$ and $max=\\{1.0,5.0,10.0\\}$. We use a learning rate
in {$1\times 10^{-i}$, $2\times 10^{-i}$ — $i$ = {3, 4, 5}}. Our predictors
are in {AS, RD, EM, PF} and the corrector is the Langevin corrector (Song et
al., 2021). SNR is the signal-to-noise ratio set to $\\{0.01,0.05,0.16\\}$
when using the corrector.

We also consider the following hyperparameter configurations for the fine-
tuning process: the fine-tuning learning rate is {$2\times 10^{-i}$ — $i$ =
{4, 5, …, 8}}, and the angle threshold $\xi$ is {$60+5\times i$ — $i$ = {0, 1,
…, 10}}. The number of fine-tuning epochs is {1, 2, 3, 4, 5}, and $w$ is
{0.99, 0.95, 0.9, 0.8, 0.7, 0.6}. $\epsilon_{t}$ is {5e-04, 1, 2, 3} in all
datasets.

Table 7. The best hyperparameters of SOS we used in Table 2. ‘(min, max)’ means $(\beta_{min},\beta_{max})$ for VP and Sub-VP or $(\sigma_{min},\sigma_{max})$ for VE, respectively. Dataset | SDE |  |  Hyperparameters for SOS |
---|---|---|---|---
Type |  | Layer Type | $(\dim(h_{1}),\dim(h_{1}),\dots,\dim(h_{d_{N}}))$ | Activation | Learn. Rate | (min, max) | Pred. | Corr. | SNR |
Default | VE |  | Concat | (512, 1024, 1024, 512) | LeakyReLU | 2e-03 | (0.01, 5.0) | RD | Langevin | 0.05 |
VP |  | (256, 512, 1024, 1024, 512, 256) | - | (0.1, 5.0) | PF | None | - |
Sub-VP |  | (512, 1024, 1024, 512) |  | (0.01, 1.0) | EM | None | - |
Shoppers | VE |  | Concatsquash | (512, 1024, 2048, 1024, 512) | ReLU | 1e-03 | (0.1, 5.0) | AS | None | - |
VP |  | - | (0.01, 5.0) | RD | Langevin | 0.05 |
Sub-VP |  |  | (0.1, 10.0) | EM | None | - |
Surgical | VE |  | Squash | (256, 512, 1024, 1024, 512, 256) | ReLU | 2e-03 | (0.1, 10.0) | EM | Langevin | 0.05 |
VP |  | (0.01, 5.0) | EM | Langevin | 0.16 |
Sub-VP |  | Concat | (512, 1024, 1024, 512) | (0.1, 10.0) | PF | None | - |
WeatherAUS | VE |  | Concat | (512, 1024, 2048, 1024, 512) | LeakyReLU | 2e-04 | (0.01, 5.0) | AS | Langevin | 0.16 |
VP |  | (0.01, 10.0) | RD | None | - |
Sub-VP |  | 2e-03 | (0.01, 1.0) | EM | None | - |
Buddy | VE |  | Concat | (256, 512, 1024, 1024, 512, 256) | SoftPlus | 2e-03 | (0.5, 5.0) | PF | None | - |
VP |  | Squash | (0.5, 10.0) | PF | None | - |
Sub-VP |  | (0.1, 10.0) | PF | None | - |
Satimage | VE |  | Concat | (512, 1024, 2048, 2048, 1024, 512) | LeakyReLU | 2e-04 | (0.01, 5.0) | RD | None | - |
VP |  | - | (0.01, 5.0) | RD | Langevin | 0.05 |
Sub-VP |  |  | (0.1, 5.0) | EM | None | - |
Table 8. The best hyperparameters of the fine-tuning process we used in Table 2 Dataset | SDE |  | Hyperparameters for fine-tuning |
---|---|---|---|---
Type |  | $\epsilon_{t}$ | Learn. Rate | $\xi$ | $w$ | Epoch | Option |
Default | VE |  | 5e-04 | 2e-06 | 80 | 0.90 | 4 | Reg. |
VP |  | 2 | 2e-08 | 80 | 0.95 | 1 | Bnd. |
Sub-VP |  | 5e-04 | 2e-06 | 90 | 0.95 | 1 | Bnd. |
Shoppers | VE |  | 5e-04 | 2e-07 | 90 | 0.80 | 1 | Bnd. |
VP |  | 5e-04 | 2e-06 | 100 | 0.95 | 3 | Reg. |
Sub-VP |  | 5e-04 | 2e-05 | 80 | 0.95 | 2 | Bnd. |
Surgical | VE |  | 5e-04 | 2e-08 | 80 | 0.99 | 3 | Reg. |
VP |  | 5e-04 | 2e-06 | 100 | 0.90 | 5 | Reg. |
Sub-VP |  | 2 | 2e-07 | 80 | 0.95 | 3 | Bnd. |
WeatherAUS | VE |  | 5e-04 | 2e-05 | 100 | 0.70 | 4 | Reg. |
VP |  | 5e-04 | 2e-07 | 60 | 0.60 | 1 | Reg. |
Sub-VP |  | 5e-04 | 2e-07 | 100 | 0.95 | 1 | Bnd. |
Buddy | VE |  | 5e-04 | 2e-07 | 80 | 0.99 | 3 | Bnd. |
VP |  | 5e-04 | 2e-07 | 80 | 0.90 | 3 | Bnd. |
Sub-VP |  | 2 | 2e-03 | 80 | 0.90 | 4 | Bnd. |
Satimage | VE |  | 5e-04 | 2e-07 | 80 | 0.95 | 3 | Reg. |
VP |  | 5e-04 | 2e-06 | 80 | 0.95 | 4 | Reg. |
Sub-VP |  | 5e-04 | 2e-08 | 80 | 0.70 | 5 | Reg. |
Table 9. Results of experiments synthesizing full fake tabular data Method |  |  | Shoppers |  |  | Surgical |  |  | Satimage |
---|---|---|---|---|---|---|---|---|---|---
|  | Acc. | F1 |  |  | Acc. | F1 |  |  | Acc. | F1 |
Identity |  |  |  0.883±0.002 |  0.503±0.005 |  |  |  0.827±0.002 |  0.608±0.005 |  |  |  0.846±0.002 |  0.799±0.003 |
CTGAN |  |  |  0.778±0.009 |  0.440±0.012 |  |  |  0.716±0.004 |  0.472±0.003 |  |  |  0.735±0.010 |  0.699±0.007 |
OCT-GAN |  |  |  0.835±0.011 |  0.490±0.010 |  |  |  0.695±0.026 |  0.527±0.014 |  |  |  0.798±0.008 |  0.767±0.009 |
TableGAN |  |  |  0.854±0.024 |  0.515±0.013 |  |  |  0.752±0.003 |  0.496±0.009 |  |  |  0.783±0.007 |  0.710±0.007 |
TVAE |  |  |  0.855±0.004 |  0.476±0.009 |  |  |  0.778±0.004 |  0.427±0.019 |  |  |  0.825±0.004 |  0.779±0.008 |
SOS |  |  | 0.874±0.002 | 0.618±0.003 |  |  | 0.798±0.003 | 0.549±0.005 |  |  | 0.852±0.003 | 0.815±0.004 |

##  Appendix B VE, VP, and Sub-VP SDEs

The definitions of $f$ and $g$ as follows:

(7) $\displaystyle f(t)$ $\displaystyle=\begin{cases}0,&\textrm{ if VE,}\\\
-\frac{1}{2}\beta(t)\mathbf{x},&\textrm{ if VP,}\\\
-\frac{1}{2}\beta(t)\mathbf{x},&\textrm{ if Sub-VP,}\\\ \end{cases}$ (8)
$\displaystyle g(t)$
$\displaystyle=\begin{cases}\sqrt{\frac{\texttt{d}[\sigma^{2}(t)]}{\texttt{d}t}},&\textrm{
if VE,}\\\ \sqrt{\beta(t)},&\textrm{ if VP,}\\\
\sqrt{\beta(t)(1-e^{-2\int_{0}^{t}\beta(s)\,\texttt{d}s})},&\textrm{ if Sub-
VP,}\\\ \end{cases}$

where $\sigma(t)$ and $\beta(t)$ are noise functions at time $t$.

##  Appendix C Score Network Architecture

The proposed score network $S_{\theta}(\mathbf{x},t)$ is as follows:

$\displaystyle\mathbf{h}_{0}$ $\displaystyle=\mathbf{x}_{t},$
$\displaystyle\mathbf{h}_{i}$
$\displaystyle=\omega(\mathtt{H}_{i}(\mathbf{h}_{i-1},t)\oplus\mathbf{h}_{i-1}),1\leq
i\leq d_{N}$ $\displaystyle S_{\boldsymbol{\theta}}(\mathbf{x}_{t},t)$
$\displaystyle=\mathtt{FC}(\mathbf{h}_{d_{N}}),$

where $\mathbf{x}_{t}$ is a record (or a row) at time $t$ in tabular data and
$\omega$ is an activation function. $d_{N}$ is the number of hidden layers.
For various layer types of $\mathtt{H}_{i}(\mathbf{h}_{i-1},t)$, we provide
the following options:

$\displaystyle\mathtt{H}_{i}(\mathbf{h}_{i-1},t)$
$\displaystyle=\begin{cases}\texttt{FC}_{i}(\mathbf{h}_{i-1})\odot\psi(\texttt{FC}^{t}_{i}(t)),&\textrm{
if Squash,}\\\ \texttt{FC}_{i}(t\oplus\mathbf{h}_{i-1}),&\textrm{ if
Concat,}\\\
\texttt{FC}_{i}(\mathbf{h}_{i-1})\odot\psi(\texttt{FC}^{gate}_{i}(t)+\texttt{FC}^{bias}_{i}(t)),&\textrm{
if Concatsquash,}\end{cases}$

where we can choose one of the three possible layer types as a hyperparameter,
$\odot$ means the element-wise multiplication, $\oplus$ means the
concatenation operator, $\psi$ is the Sigmoid function, and $\mathtt{FC}$ is a
fully connected layer.

##  Appendix D Space and Time Overheads

Figure 7. The training loss curves of SOS in Default

Some SGMs are notorious for their high computational overheads. In the case of
tabular data, however, it is not the case since the number of columns is
typically much smaller than other cases, e.g., the number of pixels of images.
We introduce our training and synthesis overheads. As shown in Fig. 7, our
training curves are smooth and in Table 10, the space and time overheads of
our method are well summarized. The GPU memory requirements are around 4GB,
which is small. It takes well less than a second per epoch for training. For
the total generation time, i.e, the time taken to accomplish the entire
oversampling task, our method shows less than 5 seconds in all cases.
Sometimes, it takes less than a second.

Table 10. The runtime and GPU memory usage (averaged across classes) of SOS (Sub-VP) with the EM predictor. Dataset | Avg. | GPU Memory | Train Time | Total Gen.
---|---|---|---|---
Class size | Usage (MB) | per Epoch (sec) | Time (sec)
Satimage | 3,186 | 4,424 | 0.036 | 4.240
Shoppers | 6,165 | 4,608 | 0.067 | 0.824
Surgical | 7,318 | 4,616 | 0.185 | 0.952
Buddy | 9,417 | 4,357 | 0.157 | 1.718
Default | 15,000 | 4,196 | 0.395 | 0.901
WeatherAUS | 28,210 | 4,812 | 0.606 | 1.766
Figure 8. The training loss curves of the MLP classifier before and after
oversampling in Satimage

##  Appendix E Full Fake Table Synthesis

Instead of oversampling minor classes, one can use our method for generating
fake tabular data entirely — in other words, fake tabular data consists of
only fake records. To this end, we train one score network of SOS (Sub-VP)
with all classes without the fine-tuning process since we do not distinguish
major/minor classes but try to generate all classes. In Table 9, we summarize
its results in three datasets. As shown, it overwhelms all other existing
methods. Even in comparison with Identity, it shows higher F1 scores in
Shoppers and Satimage. However, its accuracy is lower than Identity in
Shoppers and Surgical.

One interesting point is that for Satimage, our fake table shows better scores
than Identity for all metrics. We compare the loss curves of the MLP
classifier with the original data and our fake data in Fig. 8. Surprisingly,
the fake tabular data by SOS yields lower training loss values than Identity.
For some reasons, the fake tabular data is likely to provide better training,
which is very promising. We think that it needs more study on synthesizing
fake tabular data with SGMs to fully understand the phenomena.


# Normal ordering of degenerate integral powers of number operator and its
applications

Taekyun Kim  Department of Mathematics, Kwangwoon University, Seoul 139-701,
Republic of Korea<EMAIL_ADDRESS>,  Dae San Kim  Department of Mathematics,
Sogang University, Seoul 121-742, Republic of Korea<EMAIL_ADDRESS>and
Hye Kyung Kim  Department of Mathematics Education, Daegu Catholic University,
Gyeonsang, Republic of Korea<EMAIL_ADDRESS>
###### Abstract.

The normal ordering of an integral power of the number operator in terms of
boson operators is expressed with the help of the Stirling numbers of the
second kind. As a ‘degenerate version’ of this, we consider the normal
ordering of a degenerate integral power of the number operator in terms of
boson operators, which is represented by means of the degenerate Stirling
numbers of the second kind. As an application of this normal ordering, we
derive two equations defining the degenerate Stirling numbers of the second
kind and a Dobinski-like formula for the degenerate Bell polynomials.

###### Key words and phrases:

normal ordering; coherent states; degenerate Stirling numbers of the second
kind; degenerate Bell numbers

###### 2010 Mathematics Subject Classification:

11B73; 11B83

##  1\. Introduction

The Stirling number of the second $S_{2}(n,k)$ is the number of ways to
partition a set of $n$ objects into $k$ nonempty subsets. The Stirling numbers
of the second kind have been extensively studied and repeatedly and
independently rediscovered during their long history. The Stirling numbers of
the second kind appear in many different contexts and have numerous
applications, for example to enumerative combinatorics and quantum mechanics.
They are given either by (5) or by (7). The study of degenerate versions of
some special numbers and polynomials began with Carlitz’s paper in [2], where
the degenerate Bernoulli and Euler numbers were investigated. It is remarkable
that in recent years quite a few degenerate versions of special numbers and
polynomials have been explored with diverse tools and yielded many interesting
results (see [4-6] and the references therein). It turns out that the
degenerate Stirling numbers of the second play an important role in this
exploration for degenerate versions of many special numbers and polynomials.
The normal ordering of an integral power of the number operator $a^{\dagger}a$
in terms of boson operators $a$ and $a^{\dagger}$ can be written in the form

$(a^{\dagger}a)^{k}=\sum_{l=0}^{k}S_{2}(k,l)(a^{\dagger})^{l}a^{l}.$

In addition, the normal ordering of the degenerate $k$th power of the number
operator $a^{\dagger}a$, namely $(a^{\dagger}a)_{k,\lambda}$, in terms of
boson operators $a,a^{\dagger}$ can be written in the form

(1)
$(a^{\dagger}a)_{k,\lambda}=\sum_{l=0}^{k}S_{2,\lambda}(k,l)(a^{\dagger})^{l}a^{l},$

where the generalized falling factorials $(x)_{n,\lambda}$ are given by (3)
and the degenerate Stirling numbers $S_{2,\lambda}(k,l)$ by (4) and (6).

The aim of this paper is to use the normal ordering in (1) in order to derive
two equations defining the degenerate Stirling numbers of the second kind (see
(4), (6)) and a Dobinski-like formula for the degenerate Bell numbers (see
(37)). In more detail, our main results are as follows. Firstly, by applying
the degenerate $k$th power $(a^{\dagger}a)_{k,\lambda}$ to any number states
$|m\rangle,\ m=0,1,2,\dots$ and using (1) we obtain the equation (4).
Secondly, by comparing one expression of $\langle
z|e_{\lambda}^{a^{\dagger}a}(t)|z\rangle$ obtained by using (1) and another
one of it obtained by solving a differential equation we get the equation (6).
Thirdly, we obtain a Dobinski-like fromula for the degenerate Bell numbers
$\phi_{k,\lambda}$ by showing that $\langle
z|(a^{\dagger}a)_{k,\lambda}|z\rangle$ is equal to $\phi_{k,\lambda}(|z|^{2})$
by using (1) and that it is also equal to some other expression coming from
the representation of coherent state in terms of number states. For the rest
of this section, we recall what are needed throughout this paper.

For any $\lambda\in\mathbb{R}$, the degenerate exponential functions are
defined by

(2)
$e_{\lambda}^{x}(t)=\sum_{k=0}^{\infty}\frac{(x)_{n,\lambda}}{k!}t^{k},\quad(\mathrm{see}\
[2,4,5,6]),$

where the generalized falling factorials $(x)_{n,\lambda}$ are defined by

(3)
$(x)_{0,\lambda}=1,\quad(x)_{n,\lambda}=x(x-\lambda)\cdots\big{(}x-(n-1)\lambda\big{)},\quad(n\geq
1).$

When $x=1$, we let $\displaystyle
e_{\lambda}(t)=e_{\lambda}^{1}(t)=\sum_{k=0}^{\infty}\frac{(1)_{k,\lambda}}{k!}t^{k}.\displaystyle$
The degenerate Stirling numbers of the second kind are defined by

(4) $(x)_{n,\lambda}=\sum_{k=0}^{n}S_{2,\lambda}(n,k)(x)_{k},\quad(n\geq
0),\quad(\mathrm{see}\ [4]),$

where $(x)_{0}=1,\ (x)_{n}=x(x-1)\cdots(x-n+1),\ (n\geq 1)$.
Note that $\displaystyle\lim_{\lambda\rightarrow
0}S_{2,\lambda}(n,k)=S_{2}(n,k)\displaystyle$ are the ordinary Stirling
numbers of the second kind given by

(5) $x^{n}=\sum_{k=0}^{n}S_{2}(n,k)(x)_{k},\quad(n\geq 0),\quad(\mathrm{see}\
[1,4]).$

From (4), we note that

(6)
$\frac{1}{k!}\big{(}e_{\lambda}(t)-1\big{)}^{k}=\sum_{n=k}^{\infty}S_{2,\lambda}(n,k)\frac{t^{n}}{n!},\quad(k\geq
0),\quad(\mathrm{see}\ [4]).$

By letting $\lambda\rightarrow 0$ in (6), we see that the Stirling numbers of
the second kind are also given by

(7)
$\frac{1}{k!}\big{(}e^{t}-1\big{)}^{k}=\sum_{n=k}^{\infty}S_{2}(n,k)\frac{t^{n}}{n!},\quad(k\geq
0),\quad(\mathrm{see}\ [4]).$

In [6], the degenerate Bell polynomials are defined by

(8)
$e^{x(e_{\lambda}(t)-1)}=\sum_{n=0}^{\infty}\phi_{n,\lambda}(x)\frac{t^{n}}{n!}.$

Thus, by (8), we get

$\phi_{n,\lambda}(x)=\sum_{k=0}^{n}S_{2,\lambda}(n,k)x^{k},\quad(n\geq
0),\quad(\mathrm{see}\ [5,6]).$

When $x=1$, $\phi_{n,\lambda}=\phi_{n,\lambda}(1)$ are called the degenerate
Bell numbers.

From (4), we note that

(9)
$S_{2,\lambda}(n+1,k)=S_{2,\lambda}(n,k-1)+(k-n\lambda)S_{2,\lambda}(n,k),\quad(\mathrm{See}\
[4]),$

where $n,k\in\mathbb{N}$ with $n\geq k$.

In this paper, we pay attention to some properties with the boson operators
$a,a^{\dagger}$ that satisfy

$[a,a^{\dagger}]=aa^{\dagger}-a^{\dagger}a=1,\quad(\mathrm{see}\ [3,7]).$

The normal ordering of an integral power of the number operator $a^{\dagger}a$
in terms of boson operators $a$ and $a^{+}$ can be written in the form

(10)
$(a^{\dagger}a)^{k}=\sum_{l=0}^{k}S_{2}(k,l)(a^{\dagger})^{l}a^{l},\quad(\mathrm{see}\
[3,7]).$

The number states $|m\rangle,\ m=0,1,2,\dots$, are defined as

(11) $a|m\rangle=\sqrt{m}|m-1\rangle,\quad
a^{\dagger}|m\rangle=\sqrt{m+1}|m+1\rangle.$

By (11), we get $a^{\dagger}a|m\rangle=m|m\rangle$. The coherent states
$|z\rangle$, where $z$ is a complex number, satisfy
$a|z\rangle=z|z\rangle,\quad\langle z|z\rangle=1$. To show a connection to
coherent states, we recall that the harmonic oscillator has Hamiltonian
$H=a^{\dagger}a$ (neglecting the zero point energy) and the usual eigenstates
$|n\rangle$ (for $n\in\mathbb{N}$) satisfying $H|n\rangle=n|n\rangle$ and
$\langle m|n\rangle=\delta_{m,n}$, where $\delta_{m,n}$ is Kronecker’s symbol.

In this paper, we show that the combinatorial properties of the degenerate
Bell numbers and polynomials are derived from the algebraic properties of the
boson operators.

##  2\. Normal ordering of degenerate integral powers of the number operator
and its applications

First, we recall the definition of coherent states $a|z\rangle=z|z\rangle$,
equivalently $\langle z|a^{\dagger}=\langle z|\overline{z}$, where
$z\in\mathbb{C}$. For the coherent states $|z\rangle$, we write

(12) $|z\rangle=\sum_{n=0}^{\infty}A_{n}|n\rangle.$

Then, by (12), we get

(13)
$a|z\rangle=\sum_{n=0}^{\infty}A_{n}a|n\rangle=\sum_{n=1}^{\infty}A_{n}\sqrt{n}|n-1\rangle,$

and

(14)
$a|z\rangle=z|z\rangle=z\sum_{n=0}^{\infty}A_{n}|n\rangle=\sum_{n=1}^{\infty}zA_{n-1}|n-1\rangle.$

From (13) and (14), we have

(15) $\displaystyle A_{n}$
$\displaystyle=\frac{z}{\sqrt{n}}A_{n-1}=\frac{z}{\sqrt{n}}\frac{z}{\sqrt{n-1}}A_{n-2}=\cdots=\frac{z^{n}}{\sqrt{n!}}A_{0}.$

So, by (12) and (15), we get

(16) $|z\rangle=A_{0}\sum_{n=0}^{\infty}\frac{z^{n}}{\sqrt{n!}}|n\rangle.$

By the property of coherent state $|z\rangle$, we get

(17) $\displaystyle 1$ $\displaystyle=\langle
z|z\rangle=\overline{A}_{0}\sum_{m=0}^{\infty}\frac{\overline{z}^{m}}{\sqrt{m!}}A_{0}\sum_{n=0}^{\infty}\frac{z^{n}}{\sqrt{n!}}\langle
m|n\rangle$
$\displaystyle=|A_{0}|^{2}\sum_{n=0}^{\infty}\frac{|z|^{2n}}{n!}=|A_{0}|^{2}e^{|z|^{2}}.$

Thus, by (17), we get

$A_{0}=e^{-\frac{1}{2}|z|^{2}}\cdot e^{i(\textnormal{artitrary phase})}.$

Discarding the phase factors, by (16), we get

(18)
$|z\rangle=e^{-\frac{|z|^{2}}{2}}\sum_{n=0}^{\infty}\frac{z^{n}}{\sqrt{n!}}|n\rangle.$

For $x,y\in\mathbb{C}$, we have

$\displaystyle\langle x|y\rangle$
$\displaystyle=e^{-\frac{|x|^{2}}{2}}\sum_{m=0}^{\infty}\frac{(\overline{x})^{m}}{\sqrt{m!}}e^{-\frac{|y|^{2}}{2}}\sum_{n=0}^{\infty}\frac{y^{n}}{\sqrt{n}!}\langle
m|n\rangle$
$\displaystyle=e^{-\frac{|x|^{2}}{2}-\frac{|y|^{2}}{2}}\sum_{n=0}^{\infty}\frac{(\overline{x}y)^{n}}{n!}=e^{-\frac{1}{2}(|x|^{2}+|y|^{2})+\overline{x}y}.$

We recall that the standard bosonic commutation relations
$[a,a^{\dagger}]=aa^{\dagger}-a^{\dagger}a=1$ can be considered formally, in a
suitable space of functions $f$, by letting $a=\frac{d}{dx}$ and
$a^{\dagger}=x$ (the operator of multiplication by $x$). From (3), we note
that

(19)
$\Big{(}x\frac{d}{dx}\Big{)}_{n,\lambda}f(x)=\sum_{k=1}^{n}S_{2,\lambda}(n,k)x^{k}\Big{(}\frac{d}{dx}\Big{)}^{k}f(x),$

where $n$ is a positive integer.

Now, we consider the normal ordering of a degenerate integral power of the
number operator $a^{\dagger}a$ in terms of the boson operators
$a,a^{\dagger}$. In view of (19), the normal ordering of the degenerate $k$th
power of the number operator $a^{\dagger}a$ in terms of boson operators
$a,a^{\dagger}$ can be written in the form

(20)
$(a^{\dagger}a)_{k,\lambda}=\sum_{l=0}^{k}S_{2,\lambda}(k,l)(a^{\dagger})^{l}a^{l},\quad(k\in\mathbb{N}).$

From (11) and (20), we note that

(21)
$(a^{\dagger}a)_{k,\lambda}|m\rangle=(a^{\dagger}a)(a^{\dagger}a-\lambda)\cdots(a^{\dagger}a-(k-1)\lambda|m\rangle=(m)_{k,\lambda}|m\rangle,$

and

(22)
$(a^{\dagger}a)_{k,\lambda}|m\rangle=\sum_{l=0}^{k}S_{2,\lambda}(k,l)(a^{\dagger})^{l}a^{l}|m\rangle=\sum_{l=0}^{k}S_{2,\lambda}(k,l)(m)_{l}|m\rangle.$

Thus, by (21) and (22), we get

(23) $(m)_{k,\lambda}=\sum_{l=0}^{k}S_{2,\lambda}(k,l)(m)_{l},\quad(k\geq 1).$

This is the classical expression for the degenerate $k$th power of $m$ in
terms of the falling factorials $(m)_{l}$. This shows that (4) holds for all
nonnegative integers $x=m=0,1,2,\dots$, which in turn implies (4) itself holds
true. On the other hand, by (11) and (20), we get

(24) $\displaystyle\langle z|(a^{\dagger}a)_{k,\lambda}|z\rangle$
$\displaystyle=\sum_{l=0}^{k}S_{2,\lambda}(k,l)\langle
z|(a^{\dagger})^{l}a^{l}|z\rangle=\sum_{l=0}^{k}S_{2,\lambda}(k,l)(\overline{z})^{l}z^{l}\langle
z|z\rangle$
$\displaystyle=\sum_{l=0}^{k}S_{2,\lambda}(k,l)|z|^{2l}=\phi_{k,\lambda}(|z|^{2}).$

Let $f(t)=\langle z|e_{\lambda}^{a^{\dagger}a}(t)|z\rangle$. Then, by (24), we
get

(25) $\displaystyle f(t)$ $\displaystyle=\langle
z|e_{\lambda}^{a^{\dagger}a}(t)|z\rangle=\sum_{k=0}^{\infty}\frac{t^{k}}{k!}\langle
z|(a^{\dagger}a)_{k,\lambda}|z\rangle$
$\displaystyle=\sum_{k=0}^{\infty}\frac{t^{k}}{k!}\sum_{l=0}^{k}S_{2,\lambda}(k,l)|z|^{2l}=\sum_{k=0}^{\infty}\phi_{k,\lambda}(|z|^{2})\frac{t^{k}}{k!}.$

Indeed, the equation (25) says that $f(t)$ is the generating function of the
degenerate Bell polynomials. To obtain an explicit expression for $f(t)$, we
differentiate $f(t)$ with respect to $t$. It is not difficult to show that

(26)
$a^{\dagger}a\,e_{\lambda}^{a^{\dagger}a-\lambda}(t)=e_{\lambda}^{a^{\dagger}a-\lambda}(t)a^{\dagger}a=a^{\dagger}e_{\lambda}^{aa^{\dagger}-\lambda}(t)a=a^{\dagger}e_{\lambda}^{a^{\dagger}a+1-\lambda}(t)a.$

From (25) and (26), we note that

(27) $\displaystyle\frac{\partial f(t)}{\partial t}$
$\displaystyle=\frac{\partial}{\partial t}\langle
z|e_{\lambda}^{a^{\dagger}a}(t)|z\rangle=\langle
z|a^{\dagger}a\,e_{\lambda}^{a^{\dagger}a-\lambda}(t)|z\rangle$
$\displaystyle=\langle
z|a^{\dagger}e_{\lambda}^{a^{\dagger}a+1-\lambda}(t)a|z\rangle=e_{\lambda}^{1-\lambda}(t)\langle
z|a^{\dagger}e_{\lambda}^{a^{\dagger}a}(t)a|z\rangle$
$\displaystyle=e_{\lambda}^{1-\lambda}(t)\overline{z}z\langle
z|e_{\lambda}^{a^{\dagger}a}(t)|z\rangle=e_{\lambda}^{1-\lambda}(t)|z|^{2}f(t).$

Thus, we have

(28) $\frac{\partial f(t)}{\partial t}=e_{\lambda}^{1-\lambda}(t)|z|^{2}f(t)\
\Longleftrightarrow\
\frac{f^{\prime}(t)}{f(t)}=e_{\lambda}^{1-\lambda}(t)|z|^{2},\quad\bigg{(}f^{\prime}(t)=\frac{d}{dt}f(t)\bigg{)}.$

Assume that $f(0)=1$, for the initial value. Then, by (28), we get

(29) $\log
f(t)=\int_{0}^{t}\frac{f^{\prime}(t)}{f(t)}dt=\int_{0}^{t}|z|^{2}e_{\lambda}^{1-\lambda}(t)dt=\big{(}e_{\lambda}(t)-1\big{)}|z|^{2}.$

The equation (29) can be rewritten as

(30)
$f(t)=e^{|z|^{2}(e_{\lambda}(t)-1)}=\sum_{l=0}^{\infty}|z|^{2l}\frac{1}{l!}(e_{\lambda}(t)-1)^{l}.$

From (25), we note that

(31) $\displaystyle
f(t)=\sum_{k=0}^{\infty}\frac{t^{k}}{k!}\bigg{(}\sum_{l=0}^{k}S_{2,\lambda}(k,l)|z|^{2l}\bigg{)}=\sum_{l=0}^{\infty}\bigg{(}\sum_{k=l}^{\infty}S_{2,\lambda}(k,l)\frac{t^{k}}{k!}\bigg{)}|z|^{2l}.$

Therefore, by (30) and (31), we get

$\frac{1}{l!}\big{(}e_{\lambda}(t)-1\big{)}^{l}=\sum_{k=l}^{\infty}S_{2,\lambda}(k,l)\frac{t^{k}}{k!},$

which agrees with (6). From (24), we have

(32) $\langle
z|(a^{\dagger}a)_{k,\lambda}|z\rangle=\sum_{l=0}^{k}S_{2,\lambda}(k,l)|z|^{2l}=\phi_{k,\lambda}(|z|^{2}).$

Setting $|z|=1$, we obtain

$\langle z|(a^{\dagger}a)_{k,\lambda}|z\rangle=\phi_{k,\lambda},\quad(k\geq
1).$

Differentiating (25) with respect to $t$, we obtain

(33) $\frac{\partial f(t)}{\partial
t}=\sum_{k=1}^{\infty}\frac{t^{k-1}}{(k-1)!}\phi_{k,\lambda}(|z|^{2})=\sum_{k=0}^{\infty}\phi_{k+1,\lambda}(|z|^{2})\frac{t^{k}}{k!}.$

On the other hand, by (28), we get

(34) $\displaystyle\frac{\partial f(t)}{\partial t}$
$\displaystyle=e_{\lambda}^{1-\lambda}(t)|z|^{2}f(t)=e_{\lambda}^{1-\lambda}(t)|z|^{2}\sum_{k=0}^{\infty}\frac{t^{k}}{k!}\phi_{k,\lambda}(|z|^{2})$
$\displaystyle=|z|^{2}\sum_{k=0}^{\infty}\bigg{(}\sum_{l=0}^{k}\binom{k}{l}(1-\lambda)_{k-l,\lambda}\phi_{l,\lambda}(|z|^{2})\bigg{)}\frac{t^{k}}{k!}.$

Thus, by (33) and (34), we get

$\phi_{k+1,\lambda}(|z|^{2})=|z|^{2}\sum_{l=0}^{k}\binom{k}{l}(1-\lambda)_{k-l,\lambda}\phi_{l,\lambda}(|z|^{2}).$

In particular, for $|z|=1$, we have

$\phi_{k+1,\lambda}=\sum_{l=0}^{k}\binom{k}{l}(1-\lambda)_{k-l,\lambda}\phi_{l,\lambda}.$

Evaluating the left hand side of (32) by using the representation of the
coherent state in terms of the number state in (18), we have

(35) $\displaystyle\langle z|(a^{\dagger}a)_{k,\lambda}|z\rangle$
$\displaystyle=e^{-\frac{|z|^{2}}{2}}\cdot
e^{-\frac{|z|^{2}}{2}}\sum_{m,n=0}^{\infty}\frac{\bar{z}^{m}z^{n}}{\sqrt{m!}\sqrt{n!}}(n)_{k,\lambda}\langle
m|n\rangle$
$\displaystyle=e^{-|z|^{2}}\sum_{n=0}^{\infty}\frac{|z|^{2n}}{n!}(n)_{k,\lambda}.$

Thus, by (32) and (35), we get

(36) $\displaystyle\phi_{k,\lambda}(|z|^{2})$
$\displaystyle=\sum_{l=0}^{k}|z|^{2l}S_{2,\lambda}(k,l)=e^{-|z|^{2}}\sum_{n=0}^{\infty}\frac{|z|^{2n}}{n!}(n)_{k,\lambda}$
$\displaystyle=e^{-|z|^{2}}\sum_{n=1}^{\infty}\frac{|z|^{2n}}{(n-1)!}(n-\lambda)_{k-1,\lambda},\quad(k\in\mathbb{N}).$

In particular, by letting $|z|=1$, we get

(37)
$\displaystyle\phi_{k,\lambda}=\frac{1}{e}\sum_{n=0}^{\infty}\frac{1}{n!}(n)_{k,\lambda}=\frac{1}{e}\sum_{n=1}^{\infty}\frac{1}{(n-1)!}(n-\lambda)_{k-1,\lambda},\quad(k\in\mathbb{N}).$

This is a Dobinski-like formula for the degenerate Bell numbers.

##  3\. conclusion

Intensive studies have been done for degenerate versions of quite a few
special polynomials and numbers by using such tools as combinatorial methods,
generating functions, mathematical physics, umbral calculus techniques,
$p$-adic analysis, differential equations, special functions, probability
theory and analytic number theory.

As a degenerate version of the well known normal ordering of an integral power
of the number operator, we considered the normal ordering of a degenerate
integral power of the number operator in terms of boson operators. By using
this normal ordering we derived two equations defining the degenerate Stirling
numbers of the second kind and a Dobinski-like formula for the degenerate Bell
numbers.

It is one of our future projects to continue to explore various degenerate
versions of many special polynomials and numbers by using aforementioned
tools.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This work was supported by the Basic Science Research Program, the National
Research Foundation of Korea, (NRF-2021R1F1A1050151).

## References

  * [1] Araci, S. _A new class of Bernoulli polynomials attached to polyexponential functions and related identities_ , Adv. Stud. Contemp. Math. (Kyungshang) 31 (2021), no .2, 195-204.
  * [2] Carlitz, L. _Degenerate Stirling, Bernoulli and Eulerian numbers._ Utilitas Math. 15 (1979), 51-88.
  * [3] Katriel, J. _Bell numbers and coherent states._ Phys. Lett. A 273 (2000), no. 3, 159-161.
  * [4] Kim, D. S.; Kim, T. _A note on a new type of degenerate Bernoulli numbers._ Russ. J. Math. Phys. 27 (2020), no. 2, 227–235.
  * [5] Kim, T.; Kim, D. S. _On some degenerate differential and degenerate difference operator._ Russ. J. Math. Phys. 29 (2022), no. 1, 37–46.
  * [6] Kim, T.; Kim, D. S.; Dolgy, D. V. _On partially degenerate Bell numbers and polynomials._ Proc. Jangjeon Math. Soc. 20 (2017), no. 3, 337-345.
  * [7] Perelomov, A. _Generalized coherent states and their applications._ Texts and Monographs in Physics. Springer-Verlag, Berlin, 1986. xii+320 pp.