# Easy steps towards a sane IT policy in hospitals

Edouard Klein

###### Abstract

We witnessed the low quality of IT solutions in Paris hospitals. The price
paid to private companies for these solutions and the cost incurred from their
inefficiency constitute a gross and appalling waste of public resources. We
propose to bootstrap a change in IT policy by having heads of department hire
IT workers ; we give advice to the central decision making body on how to
incentivize them. Easily measurable efficiency gains as well as hard-to-
quantify positive externalities will follow.

###### Contents

  1. 1 Introduction
  2. 2 The data-entry task
    1. 2.1 Manual data-entry
    2. 2.2 Automatic data-entry
  3. 3 The economics of automatic data-entry
    1. 3.1 Comparative advantage
    2. 3.2 Compounding benefit
    3. 3.3 Cost of opportunity
  4. 4 Using our software
  5. 5 The big picture
    1. 5.1 Up to date software
    2. 5.2 Good solutions come from the bottom up
    3. 5.3 The right incentives
      1. Intrinsic motivation
      2. Kill the common sense
  6. 6 Related work
  7. 7 Conclusion

##  1 Introduction

In Paris hospitals, there is no easy way to access medical records although
those are computerized since circa 2011.

In order to gather data prior to statistical analysis, one has to tackle the
task of manually searching and extracting this data from a software called
Cristal-Net.

This software is technologically outdated (it can only run on Windows XP with
IE 6111Those pieces of software first came out in 2001 (more than 13 years ago
at the time of writing), and support from Microsoft was finally dropped in
April 2014.). It is also slow, prone to crashing, not devoid of bugs and has a
terrible user experience. On top of that, it has been very costly to build and
is, I believe, costly to maintain. Cristal-Net being presented as modern and
efficient [5], it is my understanding that most medical record management
systems are as bad or even worse.

The boring and error-prone task of manual data entry is usually devolved to
those at the bottom of the hospital/med-school food-chain : the medical
residents.

Those brave soon-to-be-doctors, outrageously overworked as they are [6, in
french], have to spend countless hours on this desk-drone task, despite the
fact that they have been studying medicine for close to or more than ten
years.

This is not a problem of pride at all. Our point is not to say that medical
residents are somehow too good for lowly tasks like mindless data entry. They
themselves do not pipe up and do this job with application.

Our point is that there are two grave, disturbing, compounding problems here.
The first is that the skilled manpower directly lost in data entry tasks could
be used elsewhere where it is scarce and useful : in the operating room,
talking with and listening to the patients, in staff reviews or in the
classroom (or catching up on sleep or social life).

The second is that medical residents are for the vast majority greatly under-
skilled in all things related to Information Technology (IT). Because of that,
they will be quite inefficient at their data-entry tasks. They may pollute the
data by making entry mistakes or make errors that will lead to data loss, data
corruption or loss of patient confidentiality. Datasets that are actually easy
to create will never see the light of day because the IT-unskilled operators
will see the task as too daunting or downright impossible.

The cost of managing the mistakes of the (IT-)unskilled operators, and the
cost of opportunity of never seeing those datasets that-could-have-been come
in addition to the direct loss of manpower and money that the creation, use
and maintenance of low-quality software such as Cristal-Net entails.

In this context of funds and manpower shortage in pubic hospitals in France,
it is a shame to witness such a gross misallocation of public resources.

We propose a two-fold solution to this problem. We first give a technical
emergency fix in the form a small piece of software.

  * •

This software has successfully been used to reliably extract specific
variables from more than a thousand medical records accessible only through
Cristal-Net (subsection 2.2).

  * •

We provide a link to the source code and documentation of this software as
well as suggestions to hospital managers and heads of department regarding its
deployment and adaptation to any record management software (not just Cristal-
Net) (section 4).

  * •

We provide a rationale for the quick and generalized hiring of IT workers in
any hospital where there is no solution for automated data gathering from
medical records (section 3).

The second part of the solution is political.

  * •

We propose our vision of what a well-thought out medical record management
system would look like (subsection 5.1).

  * •

We explain how such a solution will not come from a central decision making
body but from interaction, discussion and collaboration between IT field
workers (subsection 5.2).

  * •

We finally advise higher-ups about what kind of incentives need to be put in
place for this to happen, and also what kind of schemes ought _not_ to be put
in place in order to avoid perverse incentives (subsection 5.3).

We provide these ideas to sparkle discussion about software and IT policies in
public health institutions and to pave the way to a saner management of
skilled human resources and public funds. We show these ideas align with what
happen in the field of software development and with what is called for by
members of the medical community with respect to the computerization of the
field (section 6).

##  2 The data-entry task

###  2.1 Manual data-entry

The task at hand is quite straightforward. We have a list of patients
(identified by a unique number) that have given birth with a planned
C-section, and we want to know the hemoglobin rate well before the C-section
(base rate) and just after. This data will be used as part of a comparative
study between oxytocin and carbetocin. All the women before a certain date
were injected with oxytocin, and then the protocol changed and all the women
were injected with carbetocin instead. There is no doubt this data will
deliver beautiful insights about the respective risks and benefits of the two
products involved as there probably are no confounding factors : the change of
protocol was well-defined in time and systematic, and quite a lot of data has
been gathered (in the hundreds of patients in each group).

The two points of data we want exist in a database somewhere, but the digital
medical record management system of Paris hospitals make them available only
through Cristal-Net.

To access data for one patient, in the best case222If Cristal-Net does not
stop working, if the medical records are complete and if the operator does not
make any mistake., a minimum of 41 actions (clicks, or hotkey presses) are
required. The constant lagging of Cristal-Net prevents the operator from
mindlessly executing these operations as one would play a tune one knows well
on a piano, for example. The operator has to wait sometimes up to 15 or 20
seconds for feedback from the interface before he can proceed with the next
action. The video that we use to describe the software (see subsection 2.2)
gives a good idea of the tediousness of this task.

Three hours of manual work yielded data for around 80 patients, that is close
to two minutes and fifteen seconds per record. Over 1100 records had to be
analyzed, which made for an expected workload of over 41 man-hours.

###  2.2 Automatic data-entry

We were able, in less than a third of that time, to create a piece of software
automating this process. This software contains very little actual algorithmic
logic333We only need to compare the dates of the records to the date of the
C-section.. The grunt of its work is to reproduce the various clicks and
keypresses the operator would make, wait for visual feedback from Cristal-Net,
and gracefully handle edge-cases and errors. The software really, actually
takes the place of the human operator : it simulates mouse and keyboard events
and analyses images on the screen to make decisions444Yeah, data entry is so
dumb a machine can do it..

A video of this software in action can be found at
https://www.youtube.com/watch?v=nGMSP_wBr-8. Bear in mind that all throughout
the video, there is no human input, everything is automated.

The software runs with little human supervision. If everything fails, it can
be relaunched back from before everything went awry. An (IT-)unskilled medical
resident was trained in its use in less than half an hour and was able to
supervise its execution, analyze and report problems, and relaunch the system
until completion of the data-gathering task. Supervising the execution of the
program does not need to be done in real time. One can go have a look once in
a while, and, if everything is not running smoothly, purge the error logs and
relaunch the thing.

Now that we overcame the various specific problems of creating this software
for the first time, we are fairly confident in our ability (or any mid-level
software engineer’s ability) to adapt it to a new data gathering task in, say,
two to six hours.

##  3 The economics of automatic data-entry

###  3.1 Comparative advantage

The costs-benefits analysis of hiring an IT specialist to do (or automate
away) the data-entry jobs instead of using medical residents is quite
straightforward. The gain (in monetary units) is the difference between the
cost of the medical resident and the cost of the IT specialist for the same
data-entry task.

$G=cost(med)-cost(IT)$ (1)

We will take the naive approach of thinking that the cost of work is simply
the hourly cost $C$ times the time spent $t$. We will see later (subsection
3.2) that the benefits of hiring an IT specialist go beyond just him being
faster at data-entry than a resident, and that the cost of making a resident
do data entry are higher than just paying him. But for now let’s just focus on
the _comparative advantage_ [12, 27-36] of the IT worker for a data entry
task.

$G=t_{med}\cdot C_{med}-t_{IT}\cdot C_{IT}$ (2)

It is economically sound to hire an IT worker if the gain $G$ is positive, or
at the very least zero :

$\displaystyle G=t_{med}\cdot C_{med}-t_{IT}\cdot C_{IT}$ $\displaystyle\geq
0$ (3) $\displaystyle t_{med}\cdot C_{med}$ $\displaystyle\geq t_{IT}\cdot
C_{IT}$ (4) $\displaystyle\frac{t_{med}\cdot C_{med}}{t_{IT}}$
$\displaystyle\geq C_{IT}$ (5)

We can now talk about the relative (to the resident) efficiency $\alpha$ of
the IT worker.

$\alpha=\frac{t_{med}}{t_{IT}}$ (6)

For example if the IT worker can, by working one hour on a task555Either by
doing it himself, or by automating part of it., spare two hours of a medical
resident’s time, then $\alpha=2$. Whether the gain $G$ is positive, and by how
much, depends on $\alpha$ :

$G\geq 0\Leftrightarrow C_{IT}\leq\alpha C_{med}$ (7)

A visual representation of this inequality is shown Figure 1. We show
$max(C_{IT})$ : how much an IT worker can cost to the hospital and still have
the hospital benefit. Ticks for $max(C_{IT})$ are the cost associated with [7]
:

  * •

an intern (4.3 €/h),

  * •

the minimum legal wage (12.1 €/h),

  * •

an entry level salary for a software engineer (22.9 €/h),

  * •

the maximum wage for a software project leader in Paris (55.9 €/h).

Ticks for the cost of the medical personnel are located at the approximate
actual666Computed with respect to how many hours these people actually spend
working, and not how many they are supposed to spend. The theoretical hourly
cost is much higher, especially for residents. Using it would strenghten our
argument.
costs for

  * •

residents (8.5 €),

  * •

nurses and doctors (14.7 €)777Doctors (”Chef de clinique”) are paid twice as
much nurses, but work twice as many hours.,

  * •

professors (35.6 €).

Finally, ticks for the relative efficiency $\alpha$ of the IT worker are at :

  * •

$1$, a theoretical lower bound at which the IT worker is as (in)efficient at
data entry as an IT-unskilled worker,

  * •

$3.5$ the efficiency we observed in our experiment (Equation 8),

  * •

$10$, the efficiency we can expect if one tackle a new data entry task using
our software (subsection 3.2),

  * •

$40$, the efficiency that the use and deployment of high-quality software in
hospitals would allow to attain reliably (subsection 5.1).

Figure 1: Maximum price one should rationally pay to hire an IT worker. Notice
the log scale for $\alpha$ and $C_{IT}$. The datapoint at $\alpha=3.5$ and
$C_{med}=8.5$ means we could have been paid an entry level salary (for a cost
to the hospital of $22.9$€ per hour) and the hospital still would have
benefited from the exchange. Code for plotting this can be found at
https://gist.github.com/edouardklein/2d3937eac5c0c4c1b79f

Thanks to our small program, a task that should have taken a resident 41 hours
of work (recall subsection 2.1) has been done in less than two hours. The time
gain for the resident was then $t_{med}=41-2=39$ hours. The time spent writing
and testing the program was a bit less than $t_{IT}=11$ hours. We have, from
the start, an efficiency of

$\alpha=\frac{t_{med}}{t_{IT}}=\frac{39}{11}\approx 3.5$ (8)

Our experiment shows that from the start, a skilled IT worker will provide
enough value to the hospital to compensate for the cost of his employment.
This is despite the low cost of the people he is replacing (mainly the
residents who cost a whooppingly low 8.5 €888Combine ridiculously high working
hours with a salary computed on theoretically worked hours (instead of
actually worked hours) just shy above minimum wage and you’ve got a salary
between half and two third of the legal minimum wage.) and the difficulty of
automating data entry for the first time, as bugs and hoops have to be
discovered.

Considering this, it should be a no-brainer to hire a programmer/sysadmin in
every unit of every hospital. Or, at the very least, to open positions for
engineering schools interns. Yet, the hospital I did this work (for free) in
has no dedicated IT support. The one IT correspondent they can talk to is in
another hospital, and is tasked with maintenance, not development. This basic,
but quantitative analysis shows that there is an urgent need for the APHP to
reevaluate its IT policy.

###  3.2 Compounding benefit

If we were to tune our program to a new data entry task, it would not take as
long as the 11 hours it took to write it. This is because part of the hard
work has already been done. As an IT worker spends time in an environment, he
will little by little build a library of useful utilities and learn tricks to
compensate for the defects of the existing software. He will also gain domain
knowledge both in medical science and with the clinical aspect of the hospital
environment. Because of this, his relative efficiency will grow as he has to
spend less time writing code, and has a better understanding of his role
within the medical research process.

This is what we hinted at earlier when we said that the value an IT worker
provides goes beyond just freeing up some time for the medical personnel. As
he works, the IT worker not only solves the problem at hand, but makes it
easier to solve a similar problem in the future.

While we were working in the residents office in the hospital, another
resident was busy with another data-entry task. While this task could not be
fully automated because it required the resident to analyze a textual entry
made by a doctor999Reliable natural language processing is an open problem in
artificial intelligence., our program could be used to make it easier. We
could have tweaked our program to fetch the textual entries and scan them for
keywords, and then let the resident decide, using a well-thought user
interface, what parts were relevant to her problem. This would have let the
resident concentrate on the interesting problem and use her hard-gained
medical skills while automating away the tedious, repetitive and error-prone
bits (multiple clicks and keypresses to access one record). We estimate that
we could have tweaked the program in about one hour, freeing between 5 and 20
hours for the resident, depending on the number of entries she had to process.
This bumps our efficiency $\alpha$ between $5$ and $20$. Expecting an
efficiency of $\alpha=10$ for an average IT worker after some time spent in
the hospital looks like a conservative estimate to us.

An IT worker can also discover ways to access data that was previously deemed
too hard to gather to bother with it. Easy access to data means it is easier
to do research. This in turn means better research, which gives better
outreach, brings funds to the hospital and globally makes a better world for
everyone.

###  3.3 Cost of opportunity

In a world where there are infinitely many good doctors, nurses and residents,
diverting one of these people’s time to anything else than clinical activities
or medical research costs only as much as this person’s cost of employment.

We do not live in such a world. Residents don’t count their hours, but still
do not manage to work more than 120 hours a week101010Lazy bastards :/. A
resident-hour is a scarce resource, because there is not an infinite amount of
resident that can be summoned and fired at will. Diverting a resident’s time
to a task like data-entry costs more than the resident’s salary and some
taxes. One must take into account the cost of opportunity of not using the
residents where they shine : learning from doctors, teaching younger
residents, trying to heal people and doing research.

Some of these tasks can be accomplished by other personnels (nurses or doctors
depending on the task), but then the resident won’t gain experience. They can
be accomplished by stretching hours, but a tired medical practitioner is not
as efficient as, and certainly more dangerous than a well-rested one. Or they
can be foregone alltogether, which will degrade any useful metric of the
hospital’s utility.

Hiring an IT worker can be seen as hiring medical workers by proxy, at a very
low cost. Hiring an IT worker with an efficiency of $\alpha=3.5$ (our
baseline, we expect $\alpha=10$) at $21$ €/h is equivalent to hiring $3.5$
more residents at $6$ €/h (i.e. less than their already ridiculously low
cost).

##  4 Using our software

The source code of our software can be found at
https://gist.github.com/edouardklein/15b1895b64496267fc06.

Our technical solution to the problem of diverting skilled personnel to dumb
tasks can be used by anyone with a modicum of power. An intern from any
software or Computer Science (CS) cursus should be able to understand, copy
and adapt this software quite quickly. If the analysis of subsection 3.1 has
not convinced the reader of hiring a full fledged IT worker, or if the reader
does not hold enough power to hire more than an intern, then starting with
that to test the waters is an easy enough step to take. For follow-up
research, we welcome data about the costs and benefits of actually going
through with our advice.

If our reader was convinced by our arguments, or hired an intern and was happy
with the result, we suggest hiring, at an entry- to mid-level salary, a
programmer with systems administration knowledge, a desire to learn (and self-
teaching capabilities), a scholarly interest in the medical field, and a
capacity to listen to, understand and adapt to what the users (in this case
the professors, doctors, medical residents, midwives and nurses of the unit)
say.

This person will be tasked with making the life of the hospital personnel as
painless as possible with regards to IT. Provided he proved he can be trusted,
he is to be let free to pursue what he assesses to be useful, and should only
answer to one person, preferably the head of department, who will refrain from
ever micro-managing him, and will try his best to understand the nuts and
bolts of the IT side of things.

One can not emphasize how important it is that the hired person is kind and
understanding with the users, whom he was hired to serve, and that in return
he is virtually free to set his own goals, deadlines and priorities.

A successful hire will bear fruits very quickly. Medical records will be
created faster and with less mistakes, the time thus gained by the medical
staff will be spend actually doing medicine or research, which will increase
the quality of life of the patients and the throughput of the department,
which always looks good during a department review. Little by little, the IT
worker will stop working against the hindrances of the badly conceived
hospital IT system, and will actually begin to create new, better solutions.

In an environment where most workers are highly skilled, one must not
underestimate the cost of wasting a worker’s time, and the benefits of making
his life easier by hiring a complementary-skilled worker. There is a reason
why doctors don’t repair the coffee machine, do the electrical wiring or
replace a broken window. Why should they have to tackle any IT or
administrative tasks ?

##  5 The big picture

We just explained how, at the department level, one IT worker will provide far
more value than what he costs. We propose here a set of decision to be taken
at a higher level of power in the institution that we hope will allow for a
generalize optimization of IT expenditures and will have many positive
externalities.

We will first define what we want to happen, and who is susceptible to make
those changes happen. We will then try to devise incentives that will push IT
workers in the right direction, and warn the reader about ”common-sense”
policies that actually present perverse incentives that would have the
opposite of the desired effect.

###  5.1 Up to date software

If the software that manages medical records had been well crafted, not even
state-of-the-art or ”advanced” by any measure, but just good and well-thought,
then it would have, for example, presented us with a RESTful API [11] serving
JSON. In other words, we could have pointed our program at an URL like
https://example.com/patients/123456789/record.json and got all the medical
records for patient number 123456789, in a format readable by humans yet
easily parsable by a computer, the JSON format. Such a record may look like
this :

⬇

1"id":123456789,

2"records":[

3...

4{

5...

6"Executing_MU":"Hematologie (LMR)",

7"date":"2014-11-11T11:11:11.111Z",

8"data":{

9...

10"Hemoglobine":10.6,

11...

12},

13...

14},

15...

API and encoding formats should be documented, but one can see that a properly
constructed API and format such as the ones we suggest here are more or less
self documenting. Once we get a list of records such as these, finding what we
need (the newest hemoglobin stat before the C-section, and the oldest after)
is trivial and can be done in 5 lines of code or less.

Serving a request should take the server less than one second. Writing the
code that send the request and parse the answer is less than an hour’s job.
Execution of the record-selection code is almost instantaneous (even if there
are thousands of records). So, all in all, if we add the time it takes to
write the program (less than an hour) and the time it takes to execute it
(around one second per patient, so less than 20 minutes for 1100 patients), we
get between one and two hours for $t_{IT}$. As we remember, the time it would
have taken the resident was 41 hours. This puts the efficiency $\alpha$ of an
IT worker dealing with good software somewhere between $20$ and $40$. This is
highly desirable. So how do we get from the dreadful state of hospital IT we
are in now to a state where we can truly begin to leverage technology’s power
?

###  5.2 Good solutions come from the bottom up

IT is a field job. Most problems happen locally with the users. They should be
part of the solution. The best solutions are those that answer a real need,
and evolve in response to changes in the eco-system (here the users’ needs,
the legal constraints, the medical advances etc.).

Therefore, an IT solution can only be good if it is deployed with boots on the
ground. That is to say, with IT workers that can train the users, see the
problems a submit corrections. Sadly, most of IT is organized the exact
opposite way : we pay a high amount (in the order of millions) some external
company to build a piece of software that barely answer the specifications,
but that does not really matter because the specifications were defined by a
central decision making body that was almost completely disconnected with the
field. No one will ever ask the actual users what they needs are, or god
forbid come and see them interact with the software to identify the pain
points. The system will cost an arm to develop and the poor souls in the
centralized (because centralized means more efficient, of course !) IT support
team will have to maintain it for years to come, while nobody actually using
it can say a word about what should be updated or re-thought.

Cristal-Net is not an isolated case of a centrally ordered failed software
project. Louvois [1, in french], the payroll management system for the army,
is another appalling example of waste and mismanagement of public funds.

There is a need for a central influence, a driving force defining strategic
objectives. But this does not mean a central body of decision makers giving
away millions of euros for crap software.

One IT worker in a hospital unit can greatly increase, through tricks and ad-
hoc solutions, the productivity of his immediate surroundings. A dozen or so
IT workers, working in close collaboration with their medical colleagues in
their respective department, but also collaborating as an IT group, exchanging
ideas, code, software and information can bring any useful metrics of an
hospital through the roof through the network effect. Once IT-workers are
numerous and well integrated in whatever department they work in, consensus
solutions will appear as if by magic. And they will share the qualities of
modern open source software : relevance, reliability, security, and overall
quality. In other words, low cost, high value. It is possible to manage the IT
resources of a big state institution this way, as demonstrated those recent
years by the French Gendarmerie (see section 6).

###  5.3 The right incentives

#### Intrinsic motivation

In section 4, we saw how to bootstrap the political solution by deploying our
emergency technical patch. Hiring an IT worker can be done at various levels
of decision making (as an intern to a doctor, by a department head, by the
hospital manager or at the central decision making level), and it will
snowball from there. The decision is easy to make and good consequences will
automatically follow (once again, we welcome data about implementing our
advice).

Once the professors and doctors start hiring IT people in their department,
and these IT people, thanks to the network effect, start creating new and
better technical solutions to the various challenges of information management
in hospitals, the time comes to take a more formal political approach. The
goal is to make official and manage what may have started as different people
at various level in the hierarchy following our advice of hiring an IT crew,
and also to drive global optimization in order to harvest as much value as
possible from the IT workers. Somewhat paradoxically, the best way to do that
is to do as little as possible and let the knowledgeable people sort it out
themselves. Collaboration and cooperation is deeply ingrained in IT and CS
research culture111111As things like open source software, open formats [8] or
the IETF [3] prove. and competent people will be able to work together.

We know from scientific studies [16] on motivation that for creative work,
intrinsic motivation work best. Therefore, once the matter of salary is
settled121212The APHP is able to throw away millions of euros for big software
projects that don’t work, they can set aside some of it to pay in-house
developers., and we saw section 3 that the value provided by an IT worker will
be higher than what a senior project manager costs to his employer, one should
concentrate on fostering an atmosphere of creativity and collaboration.

_Autonomy_ is one of the tenants of intrinsic motivation, and we talked in
section 4 of how an IT worker should answer only to the head of the department
he works in. _Purpose_ should be familiar to anyone working in an hospital :
people are not in it for the money. Hospital matters and the IT worker will be
happy to help oil the cogs of a machine that tries its best to heal people.
Working in close collaboration with medical personnel will yield rewards as
one sees nurses and doctor becoming more efficient as the IT system gets
better. _Mastery_ is the third tenant of intrinsic motivation. It consists in
getting better at something that matters. This is where a central driving
force can help. One should push the IT workers to hold conferences and
workshops to discuss tips, tricks, technical advances etc, using _peer-
recognition_ as an incentive for innovation and documentation.

Hiring people with a scholar background (such as CS PhDs) will also foster
collaboration with the medical personnel in research activities. Being able to
jump back into a research position after a technical bout may appeal to a lot
of young graduates.

We also suggest that every decision impacting IT in hospitals should be vetted
by a qualified majority of those IT workers. In other world, a opposition from
small number of IT worker should be enough to cancel a project. This will
avoid wasteful fiascos such as Cristal-Net and Louvois.

#### Kill the common sense

This system will auto-organize and work if no perverse incentives are present.
Perverse incentives are well-intentioned mechanisms that will in fact be
detrimental to the pursued goal. The role of the central decision makers will
be, far more than taking action to help the IT workers, to avoid taking
detrimental decisions such as trying to control and check everything.

The IT workers should have job security. Otherwise, they will avoid
documenting the systems they create and voluntarily obfuscate code in order to
become essential. Their pay, benefits and working conditions should be subject
only to one person in the hierarchy (preferably the head of department), and
not tied to any metric whatsoever. This would lead to pernicious optimization
where people optimize the metric instead of doing their job. The same kind of
problem appear when trying to assess scientific research output (see e.g.
[2]). The problem is that bureaucrats with little to no training in the
subject matter try to measure efficiency by imposing quantified goals (e.g.
for IT ”tickets solved per day”, ”number of lines of code written” or ”number
of hours spent in the office”) the optimization of which is only loosely
related to the actual goal.

The IT budget should not be subject to negative change over years. Savings
from one year should be transferable to the following year. This will avoid
end-of-year stupid purchases where one tries to burn the budget in order to
get the same amount the year after. Equipment roll-out is expensive and
usually happen in bursts. Only when a 10-year accumulation of funds is clearly
unneeded a plan to give them back to the administration should be created
under the IT worker’s supervision.

IT workers at the department level are not only the missing link between
decision makers and users, but should be an unavoidable part of the decision
making process. No need to consult them, and create commissions, and hold
meetings, and take minutes, and write reports. Just let them vote and veto any
silly project that come from above. Good IT solutions come from the ground,
organically.

##  6 Related work

The agile trend in software development methodologies [14] put emphasis in
continued and meaningful interaction between the developers and the end users.
This is typically lacking in the big, costly and failed software projects that
plague the IT budgets of state institutions.

[13] studied the motivations of open source contributors and found that
external motivators were not the primary driving force. On the contrary,
intrinsic motivators such as creativity, fulfilling user’s need and bettering
one’s programming skills were the primary incentives. This is in line with the
incentives we propose to present hospital IT workers with.

The French Gendarmerie started converting their IT to open source in the early
2000’s, with great success : IT costs reduction and better IT security ([4],
in French).

The _Write The Docs_ [9] initiative regroups tools and methodologies to help
write software documentation. One can forgive the volunteers writing open
source software when they don’t document their code. This is harder to do for
a very expensive professionally developed solutions such as Cristal-Net.

[15] authored a study of IT in Florida hospitals finding a positive
relationship between IT use and efficiency. IT use is linked with higher
expenses, though.

Another study by [10] surveys statistics about IT use in hospitals. The study
is broader than what we are talking about here, but one take-away with regards
to IT efficiency in hospitals is that data is lacking. We reproduce part of
their conclusion here :

> This review suggests several important future directions in the field.
> First, additional studies need to evaluate commercially developed systems in
> community settings, and additional funding for such work may be needed.
> Second, more information is needed regarding the organizational change,
> workflow redesign, human factors, and project management issues involved
> with realizing benefits from health information technology. Third, a high
> priority must be the development of uniform standards for the reporting of
> research on implementation of health information technology, similar to the
> Consolidated Standards of Reporting Trials (CONSORT) statements for
> randomized, controlled trials and the Quality of Reporting of Meta-analyses
> (QUORUM) statement for meta-analyses (109, 110). Finally, additional work is
> needed on interoperability and consumer health technologies, such as the
> personal health record.

We believe our suggested approach would help address those issues, especially
the second and last ones.

##  7 Conclusion

Appalled by the dreadful state of IT in Paris hospitals, and ashamed of the
wasteful IT policies that spend millions of euros to end up in such a sad
state, we propose mid- to long-term low-budget provisions that will allow for
:

  * •

a decrease, down to complete removal, of loss of skilled manpower to data-
entry tasks thanks to the use of up-to-date technology (subsection 5.1);

  * •

a decrease, down to near-optimal levels, of software creation, deployment and
maintenance cost, due to the hiring of people skilled in IT, immersed in the
day-today life of a hospital unit and with a say in the decision process of
all things IT-related (subsection 5.3);

  * •

an increase in medical research productivity and scope due to the complete
availability of rich, ever growing, structured datasets, and the freeing of
precious medically skilled man-hours, that could be reinvested in research or
clinical endeavours (subsection 3.2).

Based on our experience of writing a piece of software to save 41 hours of
work to one medical resident, we provide one data point of quantitative
empirical evidence backing up our claims.

It is our hope that our advice will be followed by some heads of department,
which will allow for data gathering about the efficiency of IT workers and
will either give weight to our arguments, or give us an opportunity to revise
our proposal. We are quite confident that our advice is good, though, as the
situation is so bad that to make it worse one would have to actively try.

## References

  * [1] URL: http://www.tubbydev.com/2013/11/louvois-et-steria-d%C3%A9faite-militaire-mais-victoire-financi%C3%A8re-pour-la-ssii-.html
  * [2] URL: http://am.ascb.org/dora/
  * [3] , 1993  URL: http://www.ietf.org/
  * [4] , 2009  URL: http://www.zdnet.fr/actualites/la-gendarmerie-cas-d-ecole-d-une-migration-a-grande-echelle-vers-les-logiciels-libres-39602252.htm
  * [5] , 2014  URL: http://www.alma.fr/Solutions-sante/Cristal-link
  * [6] , 2014  URL: http://isni.fr/temps-de-travail-des-internes-pourquoi-le-gouvernement-ne-repond-pas-aux-attentes-de-la-commission-europeenne/#more-2041
  * [7] , 2014  URL: http://www.urbanlinker.com/salaire-moyen/salaire-technique-2014/
  * [8] , 2014  URL: www.linfo.org/free_file_format.html
  * [9] , 2014  URL: http://docs.writethedocs.org/
  * [10] Basit Chaudhry, Jerome Wang, Shinyi Wu, Margaret Maglione, Walter Mojica, Elizabeth Roth, Sally C Morton and Paul G Shekelle  “Systematic review: impact of health information technology on quality, efficiency, and costs of medical care”  In _Annals of internal medicine_ 144.10 Am Coll Physicians, 2006, pp. 742–752
  * [11] Roy Thomas Fielding  “Architectural styles and the design of network-based software architectures”, 2000
  * [12] Paul Krugman and Maurice Obstfeld  “International Economics: Theory and Policy”  In _International Economics: Theory and Policy_ New York: Prentice Hall, 1988
  * [13] Karim R Lakhani and Robert G Wolf  “Why hackers do what they do: Understanding motivation and effort in free/open source software projects”  In _Perspectives on free and open source software_ 1, 2005, pp. 3–22
  * [14] Robert Cecil Martin  “Agile software development: principles, patterns, and practices”  Prentice Hall PTR, 2003
  * [15] Nir Menachemi, Jeffrey Burkhardt, Richard Shewchuk, Darrell Burke and Robert G Brooks  “Hospital information technology and positive financial performance: a different approach to finding an ROI.”  In _Journal of healthcare management/American College of Healthcare Executives_ 51.1, 2005, pp. 40–58
  * [16] Daniel H Pink  “Drive: The surprising truth about what motivates us”  Penguin, 2011


# TabPFGen – Tabular Data Generation with TabPFN

Junwei Ma, Apoorv Dankar, George Stein, Guangwei Yu, Anthony Caterini
Layer 6 AI
Toronto, Canada
{jeremy, apoorv, george, guang<EMAIL_ADDRESS>

###### Abstract

Advances in deep generative modelling have not translated well to tabular
data. We argue that this is caused by a mismatch in structure between popular
generative models, and _discriminative_ models of tabular data. We thus devise
a technique to turn TabPFN – a highly performant transformer initially
designed for in-context discriminative tabular tasks – into an energy-based
generative model, which we dub _TabPFGen_. This novel framework leverages the
pre-trained TabPFN as part of the energy function and does not require any
additional training or hyperparameter tuning, thus inheriting TabPFN’s in-
context learning capability. We can sample from TabPFGen analogously to other
energy-based models. We demonstrate strong results on standard generative
modelling tasks, including data augmentation, class-balancing, and imputation,
unlocking a new frontier of tabular data generation.

##  1 Introduction

Tabular data is pervasive and important across various domains [3, 35, 10, 33,
36], yet the application of deep generative modelling – successful in
modalities like images [6, 37, 30] and text [11, 24] – has lagged behind in
the tabular setting [25]. Previous works [34, 17, 28] have argued that
attempts in this direction have not used high performing _discriminative_
models of tabular data effectively. These discriminative models can assume
various forms, encompassing tree-based and deep learning models. The non-
differentiability of tree-based models makes it challenging to integrate them
into gradient-based methods. Conversely, conventional deep learning-based
models demand substantial time and effort in terms of training and
hyperparameter tuning, rendering them hard to use across diverse datasets
[31]. An exception to this trend is TabPFN [19], a transformer-based model
designed for tabular data, which has demonstrated remarkable in-context
learning capability for discriminative tasks. It is thus worth considering if
TabPFN can be leveraged for _generative_ tasks.

We answer this in the affirmative by introducing _TabPFGen_ , a novel energy-
based model that harnesses the power of TabPFN for tabular tasks in generative
modelling. TabPFGen defines a class-conditional energy within the pre-trained
TabPFN, and employs the workhorse stochastic gradient Langevin dynamics
algorithm [38] for sample generation. Notably, TabPFGen inherits TabPFN’s in-
context learning capabilities, requiring no additional training or
hyperparameter tuning. We conduct experiments on 18 well-established datasets
from OpenML-CC18 [4]. Our results show a substantial improvement in downstream
model performance through the utilization of TabPFGen for data augmentation,
surpassing competitive baselines. Moreover, TabPFGen also proves valuable for
class balancing and imputation by producing samples that closely align with
the training data distribution, showcasing its exciting potential for tabular
data generation in practice.

##  2 Background & Related Work

TabPFN, introduced by Hollmann et al. [19], is a transformer-based
architecture designed for in-context learning of discriminative tabular data
tasks. It is trained using a prior-fitting procedure [27] that exposes the
network to a massive number of possible inductive biases which may be observed
in the tabular setting. After training, the learned TabPFN model accepts
training samples ($x_{\text{train}}$, $y_{\text{train}}$) and test features
$x_{\text{test}}$, and yields predictions $\hat{y}_{\text{test}}$ for the
entire test set in a single forward pass. In our work, we probabilistically
invert these inductive biases in order to generate synthetic data
$x_{\text{synth}}$ conditioned on synthetic labels $y_{\text{synth}}$.

The field of generative modeling for tabular data has witnessed significant
advancements. Initially, GAN-based approaches [20, 40, 14, 28] dominated,
followed by diffusion models [42, 21] and large language models (LLMs) [5,
32]. However, surprisingly, the simplest interpolation methods such as SMOTE
[8, 26] remain competitive [7]. We conjecture that the aforementioned
generative techniques may not have adequately captured the inductive biases of
successful _discriminative_ approaches, undermining their effectiveness.
Meanwhile, generative modelling using discriminators has expanded mainly
outside of the tabular domain. Early work by Tu [34] showed promising results
on computer vision tasks, and recent investigations have further demonstrated
efficacy in image synthesis [1, 22]. Nock and Guillame-Bert [28] also shed
light on this strategy for tabular data synthesis using decision trees.

Energy-based models (EBMs) have also gained significant traction across
machine learning domains. For example, Florence et al. [15] utilized EBMs in
the context of robot behavioral cloning, while Liu et al. [23] applied an
energy score for out-of-distribution detection. It is worth noting that energy
scores inherently lack class information; our contribution builds upon the
existing work by introducing a class-conditional energy for sample generation.
Grathwohl et al. [17] highlighted that any classifier can be treated as an
EBM, a concept that has been applied to various image generation works [39,
16, 12, 41]. Notably, our work stands out by generating tabular data while
leveraging a pre-trained model without any additional training or
hyperparameter tuning.

##  3 Method

Figure 1: TabPFGen overview. We backpropagate from the class-conditional
energy to $x_{\text{synth}}$ for gradient generation. CE denotes cross
entropy; blue and red arrows represent attention.

We leverage the strong in-context discriminative performance of TabPFN to
devise a class-conditional generative model. In particular, given a synthetic
label $y_{\text{synth}}\in\\{1,\ldots,K\\}$, we seek to define a generative
model $p(x_{\text{synth}}\mid y_{\text{synth}})$ which can synthesize new
samples $x_{\text{synth}}\in\mathbb{R}^{D}$ while maintaining a link to the
classification task solved by TabPFN.

TabPFN, like many classification models, induces a conditional distribution
$p(y\mid x)$ given by $\sigma(f(x))[y]$, where $x$ is the network input, $y$
is the label for $x$, $f:\mathbb{R}^{D}\rightarrow\mathbb{R}^{K}$ represents
the TabPFN, $\sigma$ is the softmax function, and $[y]$ denotes an indexing
operation. The training data $(x_{\text{train}},y_{\text{train}})$ is also
passed to TabPFN, but we omit it in the notation for simplicity.

Next, recalling Bayes’ rule, we have $p(x\mid y)\propto p(y\mid x)\cdot p(x)$.
To specify $p(x)$ we take inspiration from Grathwohl et al. [17] and employ an
energy function $E(x)$ – termed the _class-agnostic energy_ – so that
$p(x)\propto\exp(-E(x))$:

$E(x)=-\operatorname{\text{LogSumExp}}_{y^{\prime}}(f(x)[y^{\prime}])$ (1)

The class-agnostic energy also closely resembles the energy score introduced
in Liu et al. [23]. However, since we are interested in class-_conditional_
generation, we also require _class-specific_ information. To do this, first
rewrite $p(y\mid x)$ as

$p(y\mid
x)=\frac{\exp\left(f(x)[y]\right)}{\sum_{y^{\prime}}\exp\left(f(x)[y^{\prime}]\right)}=\exp\left(f(x)[y]-\operatorname{\text{LogSumExp}}_{y^{\prime}}(f(x)[y^{\prime}])\right),$
(2)

and thus $p(x\mid y)\propto p(y\mid x)\cdot p(x)\propto\exp(f(x)[y])$, as the
$\operatorname{\text{LogSumExp}}$ terms cancel out. As a result, the _class-
conditional energy_ can be defined as simply:

$E(x\mid y)\coloneqq-f(x)[y]$ (3)

We have thus arrived at an energy-based class-conditional generative model
$p(x\mid y)$ which relies on TabPFN in a principled manner. As depicted in
Figure 1, we first obtain $f(x_{\text{synth}})$ using
$(x_{\text{train}},y_{\text{train}})$ as training data. Then, we can directly
compute (3) using $E(x_{\text{synth}}\mid
y_{\text{synth}})=-f(x_{\text{synth}})[y_{\text{synth}}]$, eliminating the
need to calculate the class-agnostic energy and cross entropy separately.

1: Input: $x_{\text{train}}$, $y_{\text{train}}$ 2: Initialize
$x_{\text{synth}}^{0}$ $\triangleright$ SGLD Initialization; detailed in
Appendix C.2 3: for $t\in$[1, 2, …, $\eta$] do 4: $E(x_{\text{synth}}^{t}\mid
y_{\text{synth}})=-f(x_{\text{synth}}^{t}\mid(x_{\text{train}},y_{\text{train}}))[y_{\text{synth}}]$
$\triangleright$ Class-conditional Energy  5:
$x_{\text{synth}}^{t+1}=x_{\text{synth}}^{t}-\alpha\cdot\frac{\partial
E(x_{\text{synth}}^{t}|y_{\text{synth}})}{\partial
x_{\text{synth}}^{t}}+\sigma\cdot\mathcal{N}(0,I)$ $\triangleright$ SGLD  6:
end for 7: Output: $x_{\text{synth}}^{\eta}$

Algorithm 1 TabPFGencore: Given TabPFN model $f$, SGLD step size $\alpha$,
SGLD noise $\sigma$, SGLD steps $\eta$, manually defined synthetic labels
$y_{\text{synth}}$

In order to sample from this energy-based model, we employ the stochastic
gradient Langevin dynamics (SGLD) method [38]. We are able to generate a batch
of $x_{\text{synth}}$ every $\eta$ steps, taking advantage of their
independence. The choice of $\eta$ is determined based on the classification
performance on $x_{\text{train}}$ using a frozen TabPFN. This method is
referred to as TabPFGencore and is detailed in Algorithm 1.

To further enhance the core methodology, we propose an alternative
configuration by switching $(x_{\text{synth}},y_{\text{synth}})$ and
$(x_{\text{train}},y_{\text{train}})$ in e.g. Figure 1, and integrating the
resulting class-conditional energy $E(x_{\text{train}}\mid y_{\text{train}})$
– which implicitly depends on $(x_{\text{synth}},y_{\text{synth}})$ – into
$E(x_{\text{synth}}\mid y_{\text{synth}})$. We posit this modification
introduces a regularization effect, although we leave the proof of this to
future work. Empirically, this extended approach exhibits slightly superior
performance and greater stability. An ablation analysis can be found in
Appendix D.2. The configuration details are detailed in Appendix C.2. We
henceforth denote the full approach as _TabPFGen_.

##  4 Experiments & Analysis

We conduct a comprehensive set of experiments utilizing the 18 numerical
datasets without missing values, which are previouly employed by Hollmann et
al. [19]. These datasets are obtained from the OpenML-CC18 suite [4] and their
specifics are outlined in Appendix B.

Experimental Setup: We partition each dataset into equal-sized training and
test sets, and then train or utilize generative models to synthesize samples
given the training data. To evaluate the efficacy of a generative model we use
its synthetic samples to either replace, augment, or class-balance the
training data, then train a variety of discriminative models to predict the
class label and evaluate the AUC (Area Under the ROC Curve) performance on the
test set. We present the mean and standard deviation of the above evaluation
process over three runs with unique seeds.

Downstream Models: For all experiments we train and evaluate 4 distinct
downstream models: XGBoost [9], random forest [18], logistic regression, and
TabPFN [19]. To ensure fair comparisons we adopt the optimized hyperparameters
of downstream models previously published by Hollmann et al. [19], if
available. Experiments with a range of alternative hyperparameter
configurations consistently demonstrate that TabPFGen outperforms baseline
methods, as detailed in Appendix D.5.

Baseline Models: We employ a diverse set of baseline generative models,
including the traditional interpolation approach SMOTE [8], generative
adversarial networks represented by CTGAN [40], variational autoencoder-based
methods including TVAE [40] and RTVAE [2], the normalizing flow-based Neural
Spline Flows (NF) [13], and the recent diffusion-based methods TabDDPM [21].
All baseline generative models are trained and evaluated using the publicly
available synthcity package [29]. The details and hyperparameters of each
generative model can be found in Appendix C.1.

###  4.1 Results

Replacement & Augmentation: We assess the quality of synthetic samples from a
given model through two tasks: augmentation, which augments the training
dataset with an equal volume and class-ratio of synthetic data, and
replacement, which replaces the original training set with synthetic data. The
replacement task assesses whether the generated samples closely resemble the
training data, while the augmentation task assesses whether adding synthetic
samples improves classification performance. Table 1 displays the downstream
model performance on the test sets when utilizing synthetic samples from
various generative models during training. We find that TabPFGen facilitates a
significant performance enhancement over other generative models – TabPFGen is
the only generative model that consistently improves downstream performance
when augmenting the original training set. For instance, with XGBoost, the
average AUC (0.936) achieved is better than a model trained on the original
data (0.924). Individual results on each of the 18 datasets can be found in
Appendix D.4. TabPFGen also outperforms all other generative methods on the
replacement task, indicating that its generated samples most closely resemble
the training data. We highlight that TabPFGen requires no training or fine-
tuning, and no hyperparameter tuning to adapt to specific datasets.

Table 1: Average AUC over OpenML-CC18 test sets, with error bars over 3 runs. Top 4 rows: synthetic data as augmentation. Bottom 4 rows: synthetic data as replacement. Model | Original | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---
XGB | $0.924\scriptscriptstyle\pm 3e-4$ | $0.926\scriptscriptstyle\pm 3e-4$ | $0.912\scriptscriptstyle\pm 2e-4$ | $0.914\scriptscriptstyle\pm 7e-4$ | $0.912\scriptscriptstyle\pm 4e-4$ | $0.917\scriptscriptstyle\pm 3e-4$ | $0.927\scriptscriptstyle\pm 3e-4$ | $\mathbf{0.934}\scriptscriptstyle\pm 3e-4$
RF | $0.906\scriptscriptstyle\pm 3e-4$ | $0.906\scriptscriptstyle\pm 2e-3$ | $0.898\scriptscriptstyle\pm 1e-3$ | $0.904\scriptscriptstyle\pm 1e-3$ | $0.894\scriptscriptstyle\pm 3e-4$ | $0.907\scriptscriptstyle\pm 2e-3$ | $0.911\scriptscriptstyle\pm 7e-4$ | $\mathbf{0.912}\scriptscriptstyle\pm 4e-4$
LR | $0.920\scriptscriptstyle\pm 7e-4$ | $0.914\scriptscriptstyle\pm 3e-3$ | $0.904\scriptscriptstyle\pm 3e-3$ | $0.909\scriptscriptstyle\pm 6e-3$ | $0.901\scriptscriptstyle\pm 9e-4$ | $0.906\scriptscriptstyle\pm 8e-3$ | $0.885\scriptscriptstyle\pm 3e-4$ | $\mathbf{0.921}\scriptscriptstyle\pm 2e-4$
TabPFN | $0.934\scriptscriptstyle\pm 2e-3$ | $0.927\scriptscriptstyle\pm 1e-3$ | $0.930\scriptscriptstyle\pm 1e-3$ | $0.931\scriptscriptstyle\pm 1e-3$ | $0.928\scriptscriptstyle\pm 3e-4$ | $0.932\scriptscriptstyle\pm 1e-3$ | $0.929\scriptscriptstyle\pm 5e-4$ | $\mathbf{0.935}\scriptscriptstyle\pm 3e-4$
XGB | N/A | $0.907\scriptscriptstyle\pm 4e-4$ | $0.842\scriptscriptstyle\pm 8e-4$ | $0.858\scriptscriptstyle\pm 2e-3$ | $0.700\scriptscriptstyle\pm 6e-4$ | $0.795\scriptscriptstyle\pm 9e-4$ | $0.812\scriptscriptstyle\pm 3e-4$ | $\mathbf{0.927}\scriptscriptstyle\pm 3e-4$
RF | N/A | $0.894\scriptscriptstyle\pm 1e-3$ | $0.837\scriptscriptstyle\pm 6e-4$ | $0.844\scriptscriptstyle\pm 5e-4$ | $0.676\scriptscriptstyle\pm 2e-3$ | $0.774\scriptscriptstyle\pm 3e-4$ | $0.814\scriptscriptstyle\pm 9e-4$ | $\mathbf{0.906}\scriptscriptstyle\pm 6e-4$
LR | N/A | $0.893\scriptscriptstyle\pm 2e-3$ | $0.843\scriptscriptstyle\pm 6e-4$ | $0.873\scriptscriptstyle\pm 1e-3$ | $0.722\scriptscriptstyle\pm 3e-3$ | $0.854\scriptscriptstyle\pm 7e-4$ | $0.876\scriptscriptstyle\pm 3e-4$ | $\mathbf{0.920}\scriptscriptstyle\pm 1e-3$
TabPFN | N/A | $0.920\scriptscriptstyle\pm 8e-4$ | $0.888\scriptscriptstyle\pm 4e-4$ | $0.887\scriptscriptstyle\pm 3e-4$ | $0.705\scriptscriptstyle\pm 2e-3$ | $0.862\scriptscriptstyle\pm 1e-3$ | $0.894\scriptscriptstyle\pm 7e-4$ | $\mathbf{0.934}\scriptscriptstyle\pm 2e-4$
Table 2: Average AUC using synthetic data for class balancing. Error bars can be found in the Appendix. These datasets are the five most class-imbalanced datasets in the OpenML-CC18 suite. Dataset | Original | Sampling | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---|---
KC | $0.823$ | $0.875$ | $0.872$ | $0.859$ | $0.848$ | $0.862$ | $0.866$ | $0.805$ | $\mathbf{0.877}$
PC | $0.824$ | $0.811$ | $0.836$ | $0.827$ | $0.835$ | $0.825$ | $\mathbf{0.841}$ | $0.825$ | $\mathbf{0.841}$
BL | $0.731$ | $0.757$ | $0.756$ | $0.743$ | $0.714$ | $0.755$ | $0.706$ | $\mathbf{0.774}$ | $0.767$
CL | $0.925$ | $0.935$ | $0.949$ | $0.793$ | $0.771$ | $0.795$ | $0.909$ | $0.915$ | $\mathbf{0.955}$
DI | $0.837$ | $0.832$ | $0.832$ | $0.831$ | $0.813$ | $0.832$ | $0.837$ | $0.843$ | $\mathbf{0.844}$

(a) Original

(b) CTGAN

(c) NF

(d) TabDDPM

(e) TabPFGen

Figure 2: Contour and marginal density plots of: (a) original two-moons
dataset; (b)-(d) synthetic data generated using baseline methods; (e)
synthetic data generated by TabPFGen

Class Balancing: We create class-balanced datasets by generating synthetic
data for minority classes until every class has an equal number of samples. We
then train an XGBoost model on the class-balanced data, and report its test
AUC in Table 2. We find that TabPFGen generally outperforms alternative
methods, including even class-balancing the original data by sampling with
replacement, which we denote “sampling”. Additional details and an expanded
table can be found in Appendix D.3.

Imputation: Promising results on imputation using TabPFGen are shown in
Appendix D.1.

Qualitative Assessment: The above analyses demonstrate that TabPFN’s generated
samples have a high degree of utility for downstream applications. As a
further confirmation of sample quality we display contour and the marginal
plots for generative models on the popular two-moons dataset in Figure 2. We
show that the synthetic data distribution generated by TabPFGen is more
similar to the original data distribution than other baseline methods.
Surprisingly, we find that some generative models perform poorly even on such
a simple dataset.

##  5 Conclusion

In this work, we present TabPFGen, an efficient approach harnessing a pre-
trained TabPFN as an energy based model for tabular data generation without
additional training. TabPFGen outperforms competitive baselines across
augmentation, class balancing, and imputation tasks. Nonetheless, it is
essential to recognize that TabPFN’s current limitations, such as input size
constraints and a focus on numerical datasets, limits our method’s current
applicability for large-scale datasets. We continue this discussion in
Appendix A, but anticipate that these limitations will gradually recede as
transformers and extensions of TabPFN advance, and argue that re-purposing
powerful pretrained discriminative models for tabular generation is an
impactful and promising avenue for future research.

## References

  * san [2019] Image synthesis with a single (robust) classifier.  In _Santurkar, Shibani and Ilyas, Andrew and Tsipras, Dimitris and Engstrom, Logan and Tran, Brandon and Madry, Aleksander_ , 2019.
  * Akrami et al. [2022] Haleh Akrami, Anand A. Joshi, Jian Li, Sergül Aydöre, and Richard M. Leahy.  A robust variational autoencoder using beta divergence.  _Knowledge-Based Systems_ , 238:107886, 2022.
  * Benjelloun et al. [2020] Omar Benjelloun, Shiyu Chen, and Natasha Noy.  Google dataset search by the numbers.  In _The Semantic Web – ISWC 2020_ , pages 667–682, 2020.
  * Bischl et al. [2021] Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael Gomes Mantovani, Jan van Rijn, and Joaquin Vanschoren.  OpenML benchmarking suites.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , 2021.
  * Borisov et al. [2022] Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci.  Language models are realistic tabular data generators.  In _International Conference on Learning Representations_ , 2022.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In _International Conference on Learning Representations_ , 2018.
  * Camino et al. [2020] Ramiro D. Camino, Radu State, and Christian A. Hammerschmidt.  Oversampling tabular data with deep generative models: Is it worth the effort?  In _Proceedings on "I Can’t Believe It’s Not Better!" at NeurIPS Workshops_ , pages 148–157, 2020.
  * Chawla et al. [2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  SMOTE: Synthetic minority over-sampling technique.  _Journal of Artificial Intelligence Research_ , 16:321–357, 2002.
  * Chen and Guestrin [2016] Tianqi Chen and Carlos Guestrin.  XGBoost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SigKDD International Conference on Knowledge Discovery and Data Mining_ , pages 785–794, 2016.
  * Clements et al. [2020] Jillian M Clements, Di Xu, Nooshin Yousefi, and Dmitry Efimov.  Sequential deep learning for credit risk monitoring with tabular financial data.  _arXiv preprint arXiv:2012.15330_ , 2020.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4171–4186, 2019.
  * Du and Mordatch [2019] Yilun Du and Igor Mordatch.  Implicit generation and modeling with energy based models.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Durkan et al. [2019] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  Neural spline flows.  _Advances in Neural Information Processing systems_ , 2019.
  * Engelmann and Lessmann [2021] Justin Engelmann and Stefan Lessmann.  Conditional Wasserstein GAN-based oversampling of tabular data for imbalanced learning.  _Expert Systems with Applications_ , 174:114582, 2021.
  * Florence et al. [2022] Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson.  Implicit behavioral cloning.  In _Conference on Robot Learning_ , pages 158–168, 2022.
  * Gao et al. [2017] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu.  Learning energy-based models as generative ConvNets via multi-grid modeling and sampling.  _arXiv preprint arXiv:1709.08868_ , 2017.
  * Grathwohl et al. [2020] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky.  Your classifier is secretly an energy based model and you should treat it like one.  In _International Conference on Learning Representations_ , 2020.
  * Ho [1995] Tin Kam Ho.  Random decision forests.  In _Proceedings of 3rd International Conference on Document Analysis and Recognition_ , pages 278–282, 1995.
  * Hollmann et al. [2023] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter.  TabPFN: A transformer that solves small tabular classification problems in a second.  In _International Conference on Learning Representations_ , 2023.
  * Jordon et al. [2018] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar.  PATE-GAN: Generating synthetic data with differential privacy guarantees.  In _International Conference on Learning Representations_ , 2018.
  * Kotelnikov et al. [2023] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko.  TabDDPM: Modelling tabular data with diffusion models.  In _International Conference on Machine Learning_ , pages 17564–17579, 2023.
  * Li [2022] Haoyang Li.  Use classifier as generator.  _arXiv preprint arXiv:2209.09210_ , 2022.
  * Liu et al. [2020] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  In _Advances in Neural Information Processing Systems_ , pages 21464–21475, 2020.
  * Ma et al. [2019] Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy.  FlowSeq: Non-autoregressive conditional sequence generation with generative flow.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ , pages 4282–4292, 2019.
  * Manousakas and Aydöre [2023] Dionysis Manousakas and Sergül Aydöre.  On the usefulness of synthetic tabular data generation.  In _Data-centric Machine Learning Research (DMLR) Workshop at the 40th International Conference on Machine Learning_ , 2023.
  * More [2016] Ajinkya More.  Survey of resampling techniques for improving classification performance in unbalanced datasets.  _arXiv preprint arXiv:1608.06048_ , 2016.
  * Müller et al. [2022] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter.  Transformers can do Bayesian inference.  In _International Conference on Learning Representations_ , 2022.
  * Nock and Guillame-Bert [2022] Richard Nock and Mathieu Guillame-Bert.  Generative trees: Adversarial and copycat.  In _International Conference on Machine Learning_ , pages 16906–16951, 2022.
  * Qian et al. [2023] Zhaozhi Qian, Bogdan-Constantin Cebere, and Mihaela van der Schaar.  Synthcity: Facilitating innovative use cases of synthetic data in different data modalities.  _arXiv preprint arXiv:2301.07573_ , 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Shwartz-Ziv and Armon [2022] Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  _Information Fusion_ , 81:84–90, 2022.
  * Solatorio and Dupriez [2023] Aivin V. Solatorio and Olivier Dupriez.  REaLTabFormer: Generating realistic relational and tabular data using transformers.  _arXiv preprint arXiv:2302.02041_ , 2023.
  * Tang et al. [2020] Qi Tang, Guoen Xia, Xianquan Zhang, and Feng Long.  A customer churn prediction model based on XGBoost and MLP.  In _2020 International Conference on Computer Engineering and Application (ICCEA)_ , pages 608–612, 2020.
  * Tu [2007] Zhuowen Tu.  Learning generative models via discriminative approaches.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1–8, 2007.
  * Ulmer et al. [2020] Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà.  Trust issues: Uncertainty estimation does not enable reliable OOD detection on medical tabular data.  In _Machine Learning for Health_ , pages 341–354, 2020.
  * Urban and Gates [2021] Christopher J Urban and Kathleen M Gates.  Deep learning: A primer for psychologists.  _Psychological Methods_ , 26(6):743, 2021.
  * Vahdat and Kautz [2020] Arash Vahdat and Jan Kautz.  NVAE: A deep hierarchical variational autoencoder.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Welling and Teh [2011] Max Welling and Yee Whye Teh.  Bayesian learning via stochastic gradient Langevin dynamics.  In _International Conference on Machine Learning_ , pages 681–688, 2011.
  * Xie et al. [2016] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu.  A theory of generative ConvNet.  In _International Conference on Machine Learning_ , pages 2635–2644, 2016.
  * Xu et al. [2019] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Modeling tabular data using conditional GAN.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Yin et al. [2021] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov.  See through gradients: Image batch recovery via GradInversion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16337–16346, 2021.
  * Zheng and Charoenphakdee [2022] Shuhan Zheng and Nontawat Charoenphakdee.  Diffusion models for missing value imputation in tabular data.  In _NeurIPS 2022 First Table Representation Workshop_ , 2022.

##  Appendix A Limitations and Impact

Our method builds upon TabPFN, and consequently inherits certain limitations.
Presently, TabPFN imposes restrictions on input size, allowing for a maximum
of 2000 tokens, 100 features, and 10 classes. These constraints arise from the
quadratic complexity inherent in the underlying transformer architecture. It
is important to note that TabPFN and TabPFGen algorithms are designed to be
architecture-agnostic, therefore the improvement of transformer architectures
will directly translate to our method. As the research of transformer
architectures continues to advance, we anticipate these constraints will
diminish.

##  Appendix B Datasets Details

Our experiments are conducted using the 18 numerical datasets from OpenML-CC18
[4] (available at http://openml.org). Similar to Hollmann et al. [19], we use
datasets with maximum 2000 samples, 100 numerical features, and 10 classes,
without missing values. The details of the datasets are listed in Table 3.

Table 3: 18 datasets from OpenML-CC18 Name | #Feat. | #Inst. | #Class. | Minor. Class Size | OpenML ID
---|---|---|---|---|---
balance-scale | 5 | 625 | 3 | 49 | 11
mfeat-fourier | 77 | 2000 | 10 | 200 | 14
mfeat-karhunen | 65 | 2000 | 10 | 200 | 16
mfeat-morphological | 7 | 2000 | 10 | 200 | 18
mfeat-zernike | 48 | 2000 | 10 | 200 | 22
diabetes | 9 | 768 | 2 | 268 | 37
vehicle | 19 | 846 | 4 | 199 | 54
analcatdata_auth… | 71 | 841 | 4 | 55 | 458
pc4 | 38 | 1458 | 2 | 178 | 1049
pc3 | 38 | 1563 | 2 | 160 | 1050
kc2 | 22 | 522 | 2 | 107 | 1063
pc1 | 22 | 1109 | 2 | 77 | 1068
banknote-authenti… | 5 | 1372 | 2 | 610 | 1462
blood-transfusion-… | 5 | 748 | 2 | 178 | 1464
qsar-biodeg | 42 | 1055 | 2 | 356 | 1494
wdbc | 31 | 569 | 2 | 212 | 1510
steel-plates-fault | 28 | 1941 | 7 | 55 | 40982
climate-model-simu… | 21 | 540 | 2 | 46 | 40994

##  Appendix C Experimental Details

###  C.1 Baseline Details

We use the synthcity package [29] to run all of our baselines. The synthcity
code repository can be found at: https://github.com/vanderschaarlab/synthcity.
We keep the same hyperparameters as Manousakas and Aydöre [25] when possible.
The exact hyperparameters and training details can be found in Table 13.

###  C.2 TabPFGen Details

We use a pre-trained TabPFN for all of our experiments. The pre-trained TabPFN
weights are available for download from the following link:
https://github.com/automl/TabPFN/raw/main/tabpfn/models_diff/prior_diff_real_checkpoint_n_0_epoch_42.cpkt.

We first initialize $x_{\text{synth}}$ using the training data
$x_{\text{train}}$ injected with Gaussian noise. We fix the mean of the noise
to be 0 and standard deviation to be 0.01 throughout all of our experiments.
We then perform SGLD for $\eta$ steps guided by the classification AUC on
$x_{\text{train}}$ with $x_{\text{synth}}$ as in-context training data. Since
TabPFN accepts independent inputs, we are able to generate a batch of
independent samples for each iteration in the SGLD process. Therefore, we
return the best batch of $x_{\text{synth}}$ at the end of the SGLD process.

##  Appendix D Additional Results

###  D.1 Imputation with TabPFGen

$70$$60$$50$$40$$30$$20$$0.88$$0.9$$0.92$$0.94$$0.96$$0.98$$1$$1.02$$1.04$Percentage
of Data Missing [%]RMSEBalance ScaleTabPFGenMean

$70$$60$$50$$40$$30$$20$$0.78$$0.82$$0.86$$0.9$$0.94$$0.98$$1.02$Percentage of
Data Missing [%]RMSEMfeat MorphologicalTabPFGenMean

$70$$60$$50$$40$$30$$20$$0.84$$0.87$$0.9$$0.93$$0.96$$0.99$$1.02$Percentage of
Data Missing [%]RMSEVehicleTabPFGenMean

$70$$60$$50$$40$$30$$20$$0.9$$0.92$$0.94$$0.96$$0.98$$1$$1.02$Percentage of
Data Missing [%]RMSEBanknote AuthenticationTabPFGenMean

Figure 3: Imputation results. TabPFGen consistently has lower RMSE than the
baseline method and also has decreasing RMSE with a decreasing fraction of
missing data.

We conduct imputation analysis on 4 randomly selected datasets including:
balance scale, mfeat morphological, vehicle, and banknote authentication.
Details of the datasets can be found in Appendix B. For all datasets, we
uniform randomly mask out a certain percentage of the entries in the table as
NaNs. For the baseline method, we simply use the mean of the entire column as
the predicted value.

For our method, we initialize the missing entries as the mean value of the
feature. Then we start sampling with TabPFGen on $x_{\text{synth}}$, but we
fix the values of the non-missing features. Finally, we measure the RMSE
between the original values and the sampled values for all missing entries. As
shown in Figure 3, for all datasets, TabPFGen results in lower RMSE than using
the mean feature value as imputation. We can also observe that TabPFGen is
able to reduce RMSE further as the percentage of data missing decreases, while
the mean imputation method does not improve. Even though these experiments do
not compare to a robust set of baselines, they nevertheless demonstrate that
using TabPFGen for imputation is a fruitful direction for further study.

###  D.2 Ablation Studies

Table 4: Average AUC over OpenML-CC18 with error bars over 3 runs. Results shown across different TabPFGen variants. Top 4 rows: synthetic data as augmentation. Bottom 4 rows: synthetic data as replacement. Model |  TabPFGencore |  TabPFGen
---|---|---
XGB |  $\mathbf{0.934}\scriptscriptstyle\pm 5e-4$ |  $\mathbf{0.934}\scriptscriptstyle\pm 3e-4$
RF |  $\mathbf{0.913}\scriptscriptstyle\pm 9e-4$ |  $0.912\scriptscriptstyle\pm 4e-4$
LR |  $0.919\scriptscriptstyle\pm 6e-4$ |  $\mathbf{0.921}\scriptscriptstyle\pm 2e-4$
TabPFN |  $\mathbf{0.937}\scriptscriptstyle\pm 3e-4$ |  $0.935\scriptscriptstyle\pm 3e-4$
XGB |  $\mathbf{0.927}\scriptscriptstyle\pm 1e-3$ |  $\mathbf{0.927}\scriptscriptstyle\pm 3e-4$
RF |  $\mathbf{0.906}\scriptscriptstyle\pm 7e-4$ |  $\mathbf{0.906}\scriptscriptstyle\pm 6e-4$
LR |  $0.918\scriptscriptstyle\pm 6e-4$ |  $\mathbf{0.920}\scriptscriptstyle\pm 1e-3$
TabPFN |  $\mathbf{0.934}\scriptscriptstyle\pm 1e-3$ |  $\mathbf{0.934}\scriptscriptstyle\pm 2e-4$

We conduct ablation studies over the 2 TabPFGen variants in Table 4, recalling
that the Core variant is the basic technique outlined in Algorithm 1, and the
full TabPFGen variant also includes the swapping of
$(x_{\text{synth}},y_{\text{synth}})$ and
$(x_{\text{train}},y_{\text{train}})$ as described previous. The top 4 rows
indicate the performance of different TabPFGen variants when augmenting the
training set with synthetic samples, and the bottom 4 rows show the downstream
model performance when trained only on synthetic data. We see that the Core
variant is generally competitive with the full TabPFGen – even outperforming
in some instances – but we decided to use the full TabPFGen as the main
approach because of its performance on the replacement task.

###  D.3 Additional Class Balancing Experiment Details

Table 5: Average AUC using synthetic data for class balancing. Error bars shown over 3 runs. | Original | Sampling | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---|---
KC | $0.823\scriptscriptstyle\pm 1e-3$ | $0.875\scriptscriptstyle\pm 4e-4$ | $0.872\scriptscriptstyle\pm 2e-4$ | $0.859\scriptscriptstyle\pm 7e-4$ | $0.848\scriptscriptstyle\pm 6e-4$ | $0.862\scriptscriptstyle\pm 9e-4$ | $0.866\scriptscriptstyle\pm 2e-3$ | $0.805\scriptscriptstyle\pm 8e-4$ | $\mathbf{0.877}\scriptscriptstyle\pm 6e-4$
PC | $0.824\scriptscriptstyle\pm 4e-3$ | $0.811\scriptscriptstyle\pm 4e-3$ | $0.836\scriptscriptstyle\pm 4e-4$ | $0.827\scriptscriptstyle\pm 4e-4$ | $0.835\scriptscriptstyle\pm 5e-3$ | $0.825\scriptscriptstyle\pm 1e-3$ | $\mathbf{0.841}\scriptscriptstyle\pm 5e-4$ | $0.825\scriptscriptstyle\pm 5e-3$ | $\mathbf{0.841}\scriptscriptstyle\pm 8e-4$
BL | $0.731\scriptscriptstyle\pm 4e-4$ | $0.757\scriptscriptstyle\pm 4e-3$ | $0.756\scriptscriptstyle\pm 3e-3$ | $0.743\scriptscriptstyle\pm 7e-4$ | $0.714\scriptscriptstyle\pm 4e-4$ | $0.755\scriptscriptstyle\pm 5e-4$ | $0.706\scriptscriptstyle\pm 4e-3$ | $\mathbf{0.774}\scriptscriptstyle\pm 4e-4$ | $0.767\scriptscriptstyle\pm 4e-4$
CL | $0.925\scriptscriptstyle\pm 5e-4$ | $0.935\scriptscriptstyle\pm 3e-3$ | $0.949\scriptscriptstyle\pm 1e-3$ | $0.793\scriptscriptstyle\pm 4e-4$ | $0.771\scriptscriptstyle\pm 4e-4$ | $0.795\scriptscriptstyle\pm 4e-4$ | $0.909\scriptscriptstyle\pm 4e-4$ | $0.915\scriptscriptstyle\pm 5e-4$ | $\mathbf{0.955}\scriptscriptstyle\pm 3e-3$
DI | $0.837\scriptscriptstyle\pm 4e-3$ | $0.832\scriptscriptstyle\pm 4e-4$ | $0.832\scriptscriptstyle\pm 7e-4$ | $0.831\scriptscriptstyle\pm 5e-4$ | $0.813\scriptscriptstyle\pm 5e-4$ | $0.832\scriptscriptstyle\pm 3e-3$ | $0.837\scriptscriptstyle\pm 5e-4$ | $0.843\scriptscriptstyle\pm 4e-4$ | $\mathbf{0.844}\scriptscriptstyle\pm 4e-3$
Table 6: Information of datasets used for class-balancing experiments Dataset | Full Name | MinMaj
---|---|---
KC | kc2 | 0.26
PC | pc3 | 0.11
BL | blood-transfusion-service-center | 0.31
CL | climate-model-simulation-crashes | 0.09
DI | diabetes | 0.54

Table 5 shows class balancing results with multiple runs for each experimental
setup. The datasets KC, PC, BL, CL, and DI represent kc2, pc3, blood-
transfusion-service-center, climate-model-simulation-crashes, and diabetes
datasets, respectively. All of them are chosen from the OpenML-CC18 suite. We
find that TabPFGen generally outperforms baseline methods, with only TabDDPM
on blood-transfusion outperforming TabPFGen throughout the entire suite of
experiments.

Table 6 shows the full name for each dataset along with the ratio of minority
class size to majority class size (MinMaj).

###  D.4 Synthetic Data as Augmentation Details

Table 7: Detailed “synthetic data as augmentation” experiment using XGBoost Dataset | Original | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---
balance-scale | $\mathbf{0.9939}$ | $0.9931$ | $0.8374$ | $0.8236$ | $0.8135$ | $0.8236$ | $0.9620$ | $0.9931$
mfeat-fourier | $\mathbf{0.9803}$ | $0.9801$ | $0.9761$ | $0.9773$ | $0.9754$ | $0.9770$ | $0.9768$ | $\mathbf{0.9803}$
mfeat-karhunen | $0.9983$ | $\mathbf{0.9989}$ | $0.9977$ | $0.9974$ | $0.9961$ | $0.9977$ | $0.9977$ | $0.9985$
mfeat-morphological | $0.9612$ | $0.9608$ | $0.9574$ | $0.9556$ | $0.9562$ | $0.9600$ | $0.9612$ | $\mathbf{0.9613}$
mfeat-zernike | $0.9735$ | $\mathbf{0.9749}$ | $0.9729$ | $0.9731$ | $0.9725$ | $0.9733$ | $0.9721$ | $0.9740$
diabetes | $0.8378$ | $0.8109$ | $0.8390$ | $0.8273$ | $0.8384$ | $\mathbf{0.8401}$ | $0.8295$ | $0.8378$
vehicle | $\mathbf{0.9282}$ | $\mathbf{0.9282}$ | $0.9219$ | $0.9248$ | $0.9147$ | $0.9074$ | $0.9234$ | $0.9272$
analcatdata_authorship | $0.9997$ | $\mathbf{1.0000}$ | $0.9999$ | $0.9999$ | $0.9997$ | $0.9998$ | $0.9999$ | $0.9999$
pc4 | $0.9291$ | $0.9291$ | $0.9245$ | $0.9259$ | $0.9209$ | $0.9230$ | $0.9279$ | $\mathbf{0.9351}$
pc3 | $0.8288$ | $0.8261$ | $0.8187$ | $0.8251$ | $0.8304$ | $0.8323$ | $0.8279$ | $\mathbf{0.8392}$
kc2 | $0.8227$ | $0.8567$ | $0.8760$ | $0.8762$ | $0.8793$ | $\mathbf{0.8831}$ | $0.8801$ | $0.8716$
pc1 | $0.8489$ | $0.8683$ | $0.8428$ | $0.8465$ | $0.8602$ | $0.8556$ | $0.8800$ | $\mathbf{0.8971}$
banknote-authentication | $\mathbf{1.0000}$ | $0.9999$ | $0.9990$ | $0.9974$ | $0.9990$ | $0.9997$ | $\mathbf{1.0000}$ | $\mathbf{1.0000}$
blood-transfusion-service-center | $0.7312$ | $0.7495$ | $0.7221$ | $0.7662$ | $0.7626$ | $0.7580$ | $\mathbf{0.7651}$ | $0.7579$
qsar-biodeg | $0.9191$ | $0.9151$ | $0.9120$ | $0.9091$ | $0.9078$ | $0.9113$ | $0.9181$ | $\mathbf{0.9212}$
wdbc | $\mathbf{0.9904}$ | $0.9800$ | $0.9512$ | $0.9491$ | $0.9687$ | $0.9674$ | $0.9798$ | $0.9855$
steel-plates-fault | $0.9656$ | $0.9653$ | $0.9597$ | $\mathbf{0.9668}$ | $0.9627$ | $0.9604$ | $0.9585$ | $0.9654$
climate-model-simulation-crashes | $0.9255$ | $0.9447$ | $0.9081$ | $0.9106$ | $0.8586$ | $0.9389$ | $0.9347$ | $\mathbf{0.9586}$
Table 8: Detailed “synthetic data as augmentation experiment” using Random Forest. Dataset | Original | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---
balance-scale | $0.8001$ | $0.8142$ | $0.7491$ | $0.7717$ | $0.7125$ | $0.7717$ | $0.7990$ | $\mathbf{0.8294}$
mfeat-fourier | $0.9794$ | $\mathbf{0.9806}$ | $0.9766$ | $0.9774$ | $0.9775$ | $0.9765$ | $0.9793$ | $0.9785$
mfeat-karhunen | $0.9968$ | $\mathbf{0.9974}$ | $0.9955$ | $0.9970$ | $0.9946$ | $0.9963$ | $0.9971$ | $0.9973$
mfeat-morphological | $0.9509$ | $0.9482$ | $0.9471$ | $0.9459$ | $0.9449$ | $0.9510$ | $\mathbf{0.9529}$ | $0.9502$
mfeat-zernike | $\mathbf{0.9714}$ | $0.9672$ | $0.9662$ | $0.9664$ | $0.9677$ | $0.9684$ | $0.9694$ | $0.9707$
diabetes | $0.8159$ | $0.8020$ | $0.8068$ | $0.7813$ | $0.8087$ | $0.8070$ | $0.8209$ | $\mathbf{0.8307}$
vehicle | $0.9188$ | $0.9153$ | $0.9155$ | $0.9183$ | $0.9109$ | $0.9167$ | $\mathbf{0.9237}$ | $0.9175$
analcatdata_authorship | $0.9998$ | $\mathbf{0.9999}$ | $0.9998$ | $\mathbf{0.9999}$ | $0.9996$ | $\mathbf{0.9999}$ | $0.9995$ | $\mathbf{0.9999}$
pc4 | $0.9220$ | $0.9225$ | $0.9131$ | $0.9213$ | $0.9145$ | $0.9166$ | $\mathbf{0.9277}$ | $0.9261$
pc3 | $0.8047$ | $0.8173$ | $0.8101$ | $0.8090$ | $0.8085$ | $\mathbf{0.8318}$ | $0.7906$ | $0.8200$
kc2 | $0.8348$ | $\mathbf{0.8594}$ | $0.8259$ | $0.8497$ | $0.8484$ | $0.8230$ | $0.8377$ | $0.8348$
pc1 | $0.8853$ | $0.8851$ | $0.8740$ | $0.9063$ | $0.8902$ | $\mathbf{0.9066}$ | $0.8919$ | $0.8867$
banknote-authentication | $0.9996$ | $0.9998$ | $0.9992$ | $0.9992$ | $\mathbf{1.0000}$ | $0.9994$ | $\mathbf{1.0000}$ | $0.9998$
blood-transfusion-service-center | $0.7016$ | $0.6914$ | $0.6868$ | $0.7081$ | $0.6905$ | $0.7136$ | $\mathbf{0.7261}$ | $0.6984$
qsar-biodeg | $0.9158$ | $0.9156$ | $0.9076$ | $0.9106$ | $0.9076$ | $0.9072$ | $\mathbf{0.9198}$ | $0.9158$
wdbc | $0.9838$ | $0.9856$ | $0.9837$ | $0.9897$ | $0.9844$ | $\mathbf{0.9906}$ | $0.9860$ | $0.9802$
steel-plates-fault | $0.9577$ | $\mathbf{0.9601}$ | $0.9579$ | $0.9600$ | $0.9579$ | $0.9584$ | $0.9578$ | $0.9574$
climate-model-simulation-crashes | $0.8758$ | $0.8580$ | $0.8581$ | $0.8614$ | $0.7908$ | $0.8951$ | $\mathbf{0.9301}$ | $0.9261$
Table 9: Detailed “synthetic data as augmentation experiment” using Logistic Regression. Dataset | Original | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---
balance-scale | $\mathbf{0.9574}$ | $0.9568$ | $0.8933$ | $0.8778$ | $0.8517$ | $0.8778$ | $0.9556$ | $0.9559$
mfeat-fourier | $0.9631$ | $0.9612$ | $0.9611$ | $\mathbf{0.9636}$ | $0.9605$ | $0.9601$ | $0.9570$ | $0.9584$
mfeat-karhunen | $0.9933$ | $0.9930$ | $0.9903$ | $0.9895$ | $0.9885$ | $0.9907$ | $0.9863$ | $\mathbf{0.9948}$
mfeat-morphological | $\mathbf{0.9664}$ | $0.9654$ | $0.9578$ | $0.9538$ | $0.9534$ | $0.9526$ | $0.9629$ | $0.9660$
mfeat-zernike | $\mathbf{0.9759}$ | $0.9750$ | $0.9754$ | $0.9757$ | $0.9708$ | $0.9744$ | $0.9645$ | $0.9757$
diabetes | $0.8384$ | $0.8292$ | $0.8330$ | $0.8304$ | $\mathbf{0.8440}$ | $0.8255$ | $0.8132$ | $0.8365$
vehicle | $0.9369$ | $\mathbf{0.9404}$ | $0.8914$ | $0.8917$ | $0.8607$ | $0.8803$ | $0.7469$ | $0.9388$
analcatdata_authorship | $0.9999$ | $0.9999$ | $0.9832$ | $0.9909$ | $0.9976$ | $0.9782$ | $0.9989$ | $\mathbf{0.9999}$
pc4 | $0.8880$ | $0.8869$ | $\mathbf{0.8922}$ | $0.8743$ | $0.8597$ | $0.8615$ | $0.8500$ | $0.8830$
pc3 | $\mathbf{0.8201}$ | $0.8010$ | $0.7649$ | $0.7986$ | $0.8031$ | $0.8129$ | $0.7144$ | $0.8162$
kc2 | $0.8524$ | $0.8084$ | $0.8640$ | $0.8428$ | $\mathbf{0.8781}$ | $0.8497$ | $0.8283$ | $0.8727$
pc1 | $\mathbf{0.8233}$ | $0.8015$ | $0.7731$ | $0.8393$ | $0.8072$ | $0.8185$ | $0.7827$ | $0.8224$
banknote-authentication | $\mathbf{0.9999}$ | $\mathbf{0.9999}$ | $0.9981$ | $0.9983$ | $0.9986$ | $0.9918$ | $\mathbf{0.9999}$ | $\mathbf{0.9999}$
blood-transfusion-service-center | $0.7616$ | $0.7610$ | $0.7487$ | $\mathbf{0.7682}$ | $0.7669$ | $0.7520$ | $0.7610$ | $0.7639$
qsar-biodeg | $0.9060$ | $0.9046$ | $0.9037$ | $0.9055$ | $0.8789$ | $0.9024$ | $0.8664$ | $\mathbf{0.9073}$
wdbc | $0.9834$ | $0.9836$ | $0.9832$ | $0.9805$ | $0.9873$ | $\mathbf{0.9896}$ | $0.9729$ | $0.9841$
steel-plates-fault | $0.9375$ | $0.9374$ | $0.9189$ | $0.9295$ | $0.9175$ | $0.9247$ | $\mathbf{0.9500}$ | $0.9377$
climate-model-simulation-crashes | $0.9621$ | $0.9544$ | $0.9453$ | $0.9581$ | $0.9037$ | $\mathbf{0.9687}$ | $0.8192$ | $0.9610$
Table 10: Detailed “synthetic data as augmentation experiment” using TabPFN. Dataset | Original | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---
balance-scale | $\mathbf{0.9984}$ | $0.9983$ | $0.9510$ | $0.9582$ | $0.9508$ | $0.9582$ | $0.9792$ | $0.9955$
mfeat-fourier | $\mathbf{0.9810}$ | $0.9804$ | $0.9783$ | $0.9774$ | $0.9764$ | $0.9773$ | $0.9805$ | $0.9802$
mfeat-karhunen | $0.9980$ | $0.9979$ | $0.9976$ | $0.9978$ | $0.9975$ | $0.9976$ | $\mathbf{0.9986}$ | $0.9978$
mfeat-morphological | $\mathbf{0.9646}$ | $0.9637$ | $0.9644$ | $0.9627$ | $0.9638$ | $0.9630$ | $0.9638$ | $0.9638$
mfeat-zernike | $0.9818$ | $0.9817$ | $0.9806$ | $0.9807$ | $0.9816$ | $0.9810$ | $\mathbf{0.9819}$ | $\mathbf{0.9817}$
diabetes | $\textbf{}{0.8428}$ | $0.7766$ | $0.8423$ | $0.8307$ | $0.8418$ | $0.8308$ | $0.8224$ | $0.8138$
vehicle | $0.9568$ | $0.9516$ | $0.9419$ | $0.9477$ | $0.9467$ | $0.9450$ | $\mathbf{0.9541}$ | $0.9523$
analcatdata_authorship | $\mathbf{1.0000}$ | $\mathbf{1.0000}$ | $\mathbf{1.0000}$ | $0.9999$ | $0.9999$ | $0.9999$ | $0.9999$ | $0.9999$
pc4 | $0.9233$ | $0.9283$ | $0.9320$ | $0.9326$ | $0.9254$ | $0.9306$ | $0.9146$ | $\mathbf{0.9329}$
pc3 | $0.8481$ | $0.8183$ | $0.8396$ | $0.8449$ | $0.8369$ | $0.8516$ | $0.8365$ | $\mathbf{0.8533}$
kc2 | $0.8777$ | $0.8565$ | $0.8816$ | $0.8701$ | $0.8779$ | $0.8615$ | $\mathbf{0.8831}$ | $0.8750$
pc1 | $\mathbf{0.8919}$ | $0.8782$ | $0.8709$ | $0.8864$ | $0.8753$ | $0.8888$ | $0.8767$ | $0.8903$
banknote-authentication | $\mathbf{1.0000}$ | $\mathbf{1.0000}$ | $0.9997$ | $\mathbf{1.0000}$ | $\mathbf{1.0000}$ | $\mathbf{1.0000}$ | $\mathbf{1.0000}$ | $\mathbf{1.0000}$
blood-transfusion-service-center | $0.7590$ | $0.7277$ | $0.7444$ | $0.7599$ | $0.7601$ | $0.7622$ | $\mathbf{0.7687}$ | $0.7423$
qsar-biodeg | $0.9306$ | $0.9242$ | $0.9291$ | $0.9220$ | $0.9242$ | $0.9214$ | $0.9287$ | $\mathbf{0.9293}$
wdbc | $0.9904$ | $0.9909$ | $0.9897$ | $0.9883$ | $0.9922$ | $0.9896$ | $\mathbf{0.9933}$ | $0.9892$
steel-plates-fault | $0.9634$ | $0.9623$ | $0.9596$ | $0.9613$ | $0.9623$ | $0.9631$ | $0.9627$ | $\textbf{}{0.9636}$
climate-model-simulation-crashes | $0.9625$ | $0.9563$ | $0.9354$ | $0.9343$ | $0.8915$ | $0.9546$ | $0.8946$ | $\mathbf{0.9649}$

In Table 7, 8, 9 and 10, we show the performance of TabPFGen and baseline
methods on each of the 18 datasets individually. We display the average values
over three runs, but omit error bars to preserve space. We find that TabPFGen
outperforms the baseline methods on the majority of the datasets. Also
notably, SMOTE performs quite well even when compared against modern deep
learning methods, which is consistent with the recent work from Manousakas and
Aydöre [25].

###  D.5 Experiments with Different Hyperparameters of Downstream Models

Table 11: Synthetic data as augmentation with various downstream model hyperparameters. Bolding is done here just to highlight the best result up to three significant figures – only one run of each setting was conducted. Model | Original | SMOTE | CTGAN | TVAE | NF | RTVAE | TabDDPM | TabPFGen
---|---|---|---|---|---|---|---|---
XGB 0 | $0.903$ | $0.897$ | $0.901$ | $0.897$ | $0.891$ | $0.903$ | $0.909$ | 0.914
RF 0 | $0.902$ | $0.895$ | $0.900$ | $0.895$ | $0.892$ | $0.900$ | $0.903$ | 0.920
LR 0 | $0.888$ | $0.894$ | $0.884$ | $0.886$ | $0.884$ | $0.888$ | $0.871$ | 0.905
XGB 1 | $0.906$ | $0.902$ | $0.903$ | $0.904$ | $0.896$ | $0.906$ | $0.911$ | 0.915
RF 1 | $0.908$ | $0.902$ | $0.897$ | $0.897$ | $0.891$ | $0.904$ | $0.911$ | 0.919
LR 1 | 0.920 | $0.913$ | $0.901$ | $0.908$ | $0.905$ | $0.905$ | $0.885$ | 0.920
XGB 2 | $0.915$ | $0.914$ | $0.906$ | $0.909$ | $0.909$ | $0.911$ | $0.914$ | 0.928
RF 2 | $0.906$ | $0.905$ | $0.902$ | $0.898$ | $0.897$ | $0.905$ | $0.911$ | 0.922
LR 2 | $0.868$ | $0.870$ | $0.863$ | $0.865$ | $0.866$ | $0.869$ | $0.852$ | 0.885
XGB 3 | $0.894$ | $0.897$ | $0.864$ | $0.862$ | $0.853$ | $0.871$ | $0.881$ | 0.912
RF 3 | $0.909$ | $0.909$ | $0.904$ | $0.903$ | $0.900$ | $0.908$ | $0.914$ | 0.921
LR 3 | $0.897$ | $0.894$ | $0.866$ | $0.898$ | $0.882$ | $0.893$ | $0.871$ | 0.922
Table 12: Different downstream model hyperparameters for Table 11. |  XGB |  RF |  LR
---|---|---|---
Set 0 |  alpha: 1.274e-10, colsample_bylevel: 0.960, colsample_bytree: 0.785, gamma: 5.737e-07, lambda: 3.493e-14, learning_rate: 1.235e-06, max_depth: 8, min_child_weight: 3.737e-11, n_estimators: 971, subsample: 0.766 |  max_depth: 39, max_features: sqrt, min_samples_split: 5, n_estimators: 34 |  C: 1.362e-03, fit_intercept: True, max_iter: 320, penalty: l2
Set 1 |  alpha: 2.165e-16, colsample_bylevel: 0.975, colsample_bytree: 0.865, gamma: 2.892e-13, lambda: 9.203e-14, learning_rate: 1.922e-06, max_depth: 6, min_child_weight: 1.312e-16, n_estimators: 956, subsample: 0.432 |  max_depth: 43, max_features: sqrt, min_samples_split: 5, n_estimators: 40 |  C: 0.2776, fit_intercept: True, max_iter: 152, penalty: none
Set 2 |  alpha: 9.415e-07, colsample_bylevel: 0.311, colsample_bytree: 0.433, gamma: 9.377e-11, lambda: 2.719e-09, learning_rate: 0.0313, max_depth: 3, min_child_weight: 2.436e-10, n_estimators: 343, subsample: 0.673 |  max_depth: 39, max_features: sqrt, min_samples_split: 5, n_estimators: 94 |  C: 7.744e-05, fit_intercept: True, max_iter: 137, penalty’: l2
Set 3 |  alpha: 5.717e-16, colsample_bylevel: 0.686, colsample_bytree: 0.336, gamma: 1.149e-15, lambda: 0.293, learning_rate: 0.574, max_depth: 2, min_child_weight: 2.729e-10, n_estimators: 445, subsample: 0.278 |  max_depth: 11, max_features: sqrt, min_samples_split: 5, n_estimators: 119 |  C: 0.0266, fit_intercept: False, max_iter: 180, penalty: none

We also experiment with different hyperparameters of the downstream models to
verify the robustness of our approach to the specific choice of downstream
model, ensuring that the superior performance of TabPFGen over other
generative models is not sensitive to such choices. Table 11 shows the results
of the “synthetic data as augmentation” experiment with 4 different sets of
hyperparameters each for downstream XGBoost (XGB), random forest (RF), and
logistic regression (LR) models. We have found that TabPFGen has the best
result in all of the hyperparameter settings, showing that the results
presented in the main text are robust to such choices. The parameters are
randomly chosen using ParameterSampler from the scikit-learn library to avoid
biasing the algorithm. We list the different hyperparameters we have used in
Table 12.

Table 13: Hyperparameters of baseline models. Model |  Hyperparameters
---|---
TVAE |  n_units_embedding=500, lr=5e-4, weight_decay=1e-5, batch_size=1000, decoder_n_layers_hidden=2, decoder_n_units_hidden=256, decoder_nonlin="leaky_relu", decoder_dropout=0.1, encoder_n_layers_hidden=3, encoder_n_units_hidden=256, encoder_nonlin="leaky_relu", encoder_dropout=0.1, loss_factor=1, data_encoder_max_clusters=10, clipping_value=1, sampling_patience=500
RTVAE |  n_units_embedding=500, lr=0.001, weight_decay=1e-5, batch_size=200, decoder_n_layers_hidden=3, decoder_n_units_hidden=500, decoder_nonlin="leaky_relu", decoder_dropout=0, encoder_n_layers_hidden=3, encoder_n_units_hidden=500, encoder_nonlin="leaky_relu", encoder_dropout=0.1, data_encoder_max_clusters=10, robust_divergence_beta=2, clipping_value=1, sampling_patience=500
CTGAN |  generator_n_layers_hidden=2, generator_n_units_hidden=256, generator_nonlin="relu", generator_dropout=0.1, generator_opt_betas=(0.9, 0.999), discriminator_n_layers_hidden=2, discriminator_n_units_hidden=256, discriminator_nonlin="leaky_relu", discriminator_n_iter=1, discriminator_dropout=0.1, discriminator_opt_betas=(0.9, 0.999), lr=5e-4, weight_decay=1e-3, batch_size=1000, clipping_value=1, lambda_gradient_penalty=10, encoder_max_clusters=10, sampling_patience=500
NF |  n_layers_hidden=2, n_units_hidden=256, batch_size=1000, num_transform_blocks=1, dropout=0.1, batch_norm=False, num_bins=8, tail_bound=3, lr=5e-4, apply_unconditional_transform=True, base_distribution="standard_normal", linear_transform_type="permutation", base_transform_type="rq-autoregressive", encoder_max_clusters=10, n_iter_min=100, sampling_patience=500
TabDDPM |  is_classification=True, lr=0.002, weight_decay=0.0001, batch_size=1024, gaussian_loss_type=’mse’, scheduler=’cosine’, model_type=’mlp’, dim_embed=128, continuous_encoder=’quantile’, sampling_patience=500