BELLE2-CONF-2022-006

August 27, 2024

Belle II Collaboration

#  Measurements of the branching fraction, isospin asymmetry, and lepton-
universality ratio in $B\to J/\psi K$ decays at Belle II

F. Abudinén  I. Adachi  R. Adak  K. Adamczyk  L. Aggarwal  P. Ahlburg  H.
Ahmed  J. K. Ahn  H. Aihara  N. Akopov  A. Aloisio  F. Ameli  L. Andricek  N.
Anh Ky  D. M. Asner  H. Atmacan  V. Aulchenko  T. Aushev  V. Aushev  T. Aziz
V. Babu  S. Bacher  H. Bae  S. Baehr  S. Bahinipati  A. M. Bakich  P. Bambade
Sw. Banerjee  S. Bansal  M. Barrett  G. Batignani  J. Baudot  M. Bauer  A.
Baur  A. Beaubien  A. Beaulieu  J. Becker  P. K. Behera  J. V. Bennett  E.
Bernieri  F. U. Bernlochner  M. Bertemes  E. Bertholet  M. Bessner  S.
Bettarini  V. Bhardwaj  B. Bhuyan  F. Bianchi  T. Bilka  S. Bilokin  D. Biswas
A. Bobrov  D. Bodrov  A. Bolz  A. Bondar  G. Bonvicini  A. Bozek  M. Bračko
P. Branchini  N. Braun  R. A. Briere  T. E. Browder  D. N. Brown  A. Budano
L. Burmistrov  S. Bussino  M. Campajola  L. Cao  G. Casarosa  C. Cecchi  D.
Červenkov  M.-C. Chang  P. Chang  R. Cheaib  P. Cheema  V. Chekelian  C. Chen
Y. Q. Chen  Y.-T. Chen  B. G. Cheon  K. Chilikin  K. Chirapatpimol  H.-E. Cho
K. Cho  S.-J. Cho  S.-K. Choi  S. Choudhury  D. Cinabro  L. Corona  L. M.
Cremaldi  S. Cunliffe  T. Czank  S. Das  N. Dash  F. Dattola  E. De La Cruz-
Burelo  G. de Marino  G. De Nardo  M. De Nuccio  G. De Pietro  R. de Sangro
B. Deschamps  M. Destefanis  S. Dey  A. De Yta-Hernandez  R. Dhamija  A. Di
Canto  F. Di Capua  S. Di Carlo  J. Dingfelder  Z. Doležal  I. Domínguez
Jiménez  T. V. Dong  M. Dorigo  K. Dort  D. Dossett  S. Dreyer  S. Dubey  S.
Duell  G. Dujany  P. Ecker  S. Eidelman  M. Eliachevitch  D. Epifanov  P.
Feichtinger  T. Ferber  D. Ferlewicz  T. Fillinger  C. Finck  G. Finocchiaro
P. Fischer  K. Flood  A. Fodor  F. Forti  A. Frey  M. Friedl  B. G. Fulsom  M.
Gabriel  A. Gabrielli  N. Gabyshev  E. Ganiev  M. Garcia-Hernandez  R. Garg
A. Garmash  V. Gaur  A. Gaz  U. Gebauer  A. Gellrich  J. Gemmler  T. Geßler
D. Getzkow  G. Giakoustidis  R. Giordano  A. Giri  A. Glazov  B. Gobbo  R.
Godang  P. Goldenzweig  B. Golob  P. Gomis  G. Gong  P. Grace  W. Gradl  E.
Graziani  D. Greenwald  T. Gu  Y. Guan  K. Gudkova  J. Guilliams  C.
Hadjivasiliou  S. Halder  K. Hara  T. Hara  O. Hartbrich  K. Hayasaka  H.
Hayashii  S. Hazra  C. Hearty  M. T. Hedges  I. Heredia de la Cruz  M.
Hernández Villanueva  A. Hershenhorn  T. Higuchi  E. C. Hill  H. Hirata  M.
Hoek  M. Hohmann  S. Hollitt  T. Hotta  C.-L. Hsu  Y. Hu  K. Huang  T. Humair
T. Iijima  K. Inami  G. Inguglia  N. Ipsita  J. Irakkathil Jabbar  A. Ishikawa
S. Ito  R. Itoh  M. Iwasaki  Y. Iwasaki  S. Iwata  P. Jackson  W. W. Jacobs
I. Jaegle  D. E. Jaffe  E.-J. Jang  M. Jeandron  H. B. Jeon  Q. P. Ji  S. Jia
Y. Jin  C. Joo  K. K. Joo  H. Junkerkalefeld  I. Kadenko  J. Kahn  H. Kakuno
M. Kaleta  A. B. Kaliyar  J. Kandra  K. H. Kang  P. Kapusta  R. Karl  G.
Karyan  Y. Kato  H. Kawai  T. Kawasaki  C. Ketter  H. Kichimi  C. Kiesling  B.
H. Kim  C.-H. Kim  D. Y. Kim  H. J. Kim  K.-H. Kim  K. Kim  S.-H. Kim  Y.-K.
Kim  Y. Kim  T. D. Kimmel  H. Kindo  K. Kinoshita  C. Kleinwort  B. Knysh  P.
Kodyš  T. Koga  S. Kohani  I. Komarov  T. Konno  A. Korobov  S. Korpar  N.
Kovalchuk  E. Kovalenko  R. Kowalewski  T. M. G. Kraetzschmar  F. Krinner  P.
Križan  R. Kroeger  J. F. Krohn  P. Krokovny  H. Krüger  W. Kuehn  T. Kuhr  J.
Kumar  M. Kumar  R. Kumar  K. Kumara  T. Kumita  T. Kunigo  M. Künzel  S. Kurz
A. Kuzmin  P. Kvasnička  Y.-J. Kwon  S. Lacaprara  Y.-T. Lai  C. La Licata  K.
Lalwani  T. Lam  L. Lanceri  J. S. Lange  M. Laurenza  K. Lautenbach  P. J.
Laycock  R. Leboucher  F. R. Le Diberder  I.-S. Lee  S. C. Lee  P. Leitl  D.
Levit  P. M. Lewis  C. Li  L. K. Li  S. X. Li  Y. B. Li  J. Libby  K. Lieret
J. Lin  Z. Liptak  Q. Y. Liu  Z. A. Liu  D. Liventsev  S. Longo  A. Loos  A.
Lozar  P. Lu  T. Lueck  F. Luetticke  T. Luo  C. Lyu  C. MacQueen  M. Maggiora
R. Maiti  S. Maity  R. Manfredi  E. Manoni  S. Marcello  C. Marinas  L. Martel
A. Martini  L. Massaccesi  M. Masuda  T. Matsuda  K. Matsuoka  D. Matvienko
J. A. McKenna  J. McNeil  F. Meggendorfer  F. Meier  M. Merola  F. Metzner  M.
Milesi  C. Miller  K. Miyabayashi  H. Miyake  H. Miyata  R. Mizuk  K. Azmi  G.
B. Mohanty  N. Molina-Gonzalez  S. Moneta  H. Moon  T. Moon  J. A. Mora
Grimaldo  T. Morii  H.-G. Moser  M. Mrvar  F. J. Müller  Th. Muller  G.
Muroyama  C. Murphy  R. Mussa  I. Nakamura  K. R. Nakamura  E. Nakano  M.
Nakao  H. Nakayama  H. Nakazawa  M. Naruki  Z. Natkaniec  A. Natochii  L.
Nayak  M. Nayak  G. Nazaryan  D. Neverov  C. Niebuhr  M. Niiyama  J. Ninkovic
N. K. Nisar  S. Nishida  K. Nishimura  M. H. A. Nouxman  B. Oberhof  K. Ogawa
S. Ogawa  S. L. Olsen  Y. Onishchuk  H. Ono  Y. Onuki  P. Oskin  F. Otani  E.
R. Oxford  H. Ozaki  P. Pakhlov  G. Pakhlova  A. Paladino  T. Pang  A. Panta
E. Paoloni  S. Pardi  K. Parham  H. Park  S.-H. Park  B. Paschen  A. Passeri
A. Pathak  S. Patra  S. Paul  T. K. Pedlar  I. Peruzzi  R. Peschke  R.
Pestotnik  F. Pham  M. Piccolo  L. E. Piilonen  G. Pinna Angioni  P. L. M.
Podesta-Lerma  T. Podobnik  S. Pokharel  L. Polat  V. Popov  C. Praz  S. Prell
E. Prencipe  M. T. Prim  M. V. Purohit  H. Purwar  N. Rad  P. Rados  S. Raiz
A. Ramirez Morales  R. Rasheed  N. Rauls  M. Reif  S. Reiter  M. Remnev  I.
Ripp-Baudot  M. Ritter  M. Ritzert  G. Rizzo  L. B. Rizzuto  S. H. Robertson
D. Rodríguez Pérez  J. M. Roney  C. Rosenfeld  A. Rostomyan  N. Rout  M.
Rozanska  G. Russo  D. Sahoo  Y. Sakai  D. A. Sanders  S. Sandilya  A. Sangal
L. Santelj  P. Sartori  Y. Sato  V. Savinov  B. Scavino  C. Schmitt  M.
Schnepf  M. Schram  H. Schreeck  J. Schueler  C. Schwanda  A. J. Schwartz  B.
Schwenker  M. Schwickardi  Y. Seino  A. Selce  K. Senyo  I. S. Seong  J.
Serrano  M. E. Sevior  C. Sfienti  V. Shebalin  C. P. Shen  H. Shibuya  T.
Shillington  T. Shimasaki  J.-G. Shiu  B. Shwartz  A. Sibidanov  F. Simon  J.
B. Singh  S. Skambraks  J. Skorupa  K. Smith  R. J. Sobie  A. Soffer  A.
Sokolov  Y. Soloviev  E. Solovieva  S. Spataro  B. Spruck  M. Starič  S.
Stefkova  Z. S. Stottler  R. Stroili  J. Strube  J. Stypula  R. Sugiura  M.
Sumihama  K. Sumisawa  T. Sumiyoshi  D. J. Summers  W. Sutcliffe  S. Y. Suzuki
H. Svidras  M. Tabata  M. Takahashi  M. Takizawa  U. Tamponi  S. Tanaka  K.
Tanida  H. Tanigawa  N. Taniguchi  Y. Tao  P. Taras  F. Tenchini  R. Tiwary
D. Tonelli  E. Torassa  N. Toutounji  K. Trabelsi  I. Tsaklidis  T. Tsuboyama
N. Tsuzuki  M. Uchida  I. Ueda  S. Uehara  Y. Uematsu  T. Ueno  T. Uglov  K.
Unger  Y. Unno  K. Uno  S. Uno  P. Urquijo  Y. Ushiroda  Y. V. Usov  S. E.
Vahsen  R. van Tonder  G. S. Varner  K. E. Varvell  A. Vinokurova  L. Vitale
V. Vobbilisetti  V. Vorobyev  A. Vossen  B. Wach  E. Waheed  H. M. Wakeling
K. Wan  W. Wan Abdullah  B. Wang  C. H. Wang  E. Wang  M.-Z. Wang  X. L. Wang
A. Warburton  M. Watanabe  S. Watanuki  J. Webb  S. Wehle  M. Welsch  C.
Wessel  J. Wiechczynski  P. Wieduwilt  H. Windel  E. Won  L. J. Wu  X. P. Xu
B. D. Yabsley  S. Yamada  W. Yan  S. B. Yang  H. Ye  J. Yelton  I. Yeo  J. H.
Yin  M. Yonenaga  Y. M. Yook  K. Yoshihara  T. Yoshinobu  C. Z. Yuan  Y. Yusa
L. Zani  Y. Zhai  J. Z. Zhang  Y. Zhang  Y. Zhang  Z. Zhang  V. Zhilich  J.
Zhou  Q. D. Zhou  X. Y. Zhou  V. I. Zhukova  V. Zhulanov  R. Žlebčík

###### Abstract

We report a study of $B\to J/\psi(\ell^{+}\ell^{-})K$ decays, where $\ell$
represents an electron or a muon, using $e^{+}e^{-}$ collisions at the
$\Upsilon(4S)$ resonance. The data were collected by the Belle II experiment
at the SuperKEKB asymmetric-energy collider during 2019–2021, corresponding to
an integrated luminosity of $189$ fb-1. The measured quantities are the
branching fractions (${\mathcal{B}}$) of the decay channels $B^{+}\to
J/\psi(e^{+}e^{-})K^{+}$, $B^{+}\to J/\psi(\mu^{+}\mu^{-})K^{+}$, $B^{0}\to
J/\psi(e^{+}e^{-})K^{0}_{S}$, and $B^{0}\to J/\psi(\mu^{+}\mu^{-})K^{0}_{S}$;
the lepton-flavor-dependent isospin asymmetries for the electron
[$A_{I}\left(B\to J/\psi(e^{+}e^{-})K\right)$] and muon [$A_{I}\left(B\to
J/\psi(\mu^{+}\mu^{-})K\right)$] channels; and the ratios of branching
fractions between the muon and electron channels for the charged
[$R_{K^{+}}\left(J/\psi\right)$] and neutral kaon
[$R_{K^{0}}\left(J/\psi\right)$] case. We obtain

$\displaystyle\mathcal{B}\left(B^{+}\to J/\psi(e^{+}e^{-})K^{+}\right)$
$\displaystyle=$ $\displaystyle(6.00\pm 0.10\pm 0.19)\times 10^{-5},$
$\displaystyle\mathcal{B}\left(B^{+}\to J/\psi(\mu^{+}\mu^{-})K^{+}\right)$
$\displaystyle=$ $\displaystyle(6.06\pm 0.09\pm 0.19)\times 10^{-5},$
$\displaystyle\mathcal{B}\left(B^{0}\to J/\psi(e^{+}e^{-})K_{S}^{0}\right)$
$\displaystyle=$ $\displaystyle(2.67\pm 0.08\pm 0.12)\times 10^{-5},$
$\displaystyle\mathcal{B}\left(B^{0}\to
J/\psi(\mu^{+}\mu^{-})K_{S}^{0}\right)$ $\displaystyle=$
$\displaystyle(2.78\pm 0.08\pm 0.12)\times 10^{-5},$ $\displaystyle
A_{I}\left(B\to J/\psi(e^{+}e^{-})K\right)$ $\displaystyle=$
$\displaystyle-0.022\pm 0.016\pm 0.030,$ $\displaystyle A_{I}\left(B\to
J/\psi(\mu^{+}\mu^{-})K\right)$ $\displaystyle=$ $\displaystyle-0.006\pm
0.015\pm 0.030,$ $\displaystyle R_{K^{+}}\left(J/\psi\right)$ $\displaystyle=$
$\displaystyle 1.009\pm 0.022\pm 0.008,\text{ and}$ $\displaystyle
R_{K^{0}}\left(J/\psi\right)$ $\displaystyle=$ $\displaystyle 1.042\pm
0.042\pm 0.008,$

where the first uncertainties are statistical and the second are systematic.
The measurements are consistent with the world-average values.

Belle II, ..

##  1 Introduction

The decays $B\to K\ell^{+}\ell^{-}$, where $\ell$ stands for an electron or a
muon, are flavor-changing-neutral-current processes. Being governed by a $b\to
s$ quark-level transition, these decays are forbidden at tree level in the
standard model (SM) of particle physics GIM , but can proceed through $b\to
s\ell^{+}\ell^{-}$ loop amplitudes at lowest order. Various SM extensions BSM1
; BSM2  predict new particles that contribute to the processes, altering the
values of observables from their SM predictions. These possibilities make
$B\to K\ell^{+}\ell^{-}$ decays a sensitive probe for beyond-the-SM physics.

One of the key predictions of the SM is that the coupling strengths of
electroweak gauge bosons to charged leptons $e$, $\mu$, and $\tau$ are the
same, a property known as lepton-flavor universality (LFU). Accordingly, the
ratio of branching fractions of $B\to K\mu^{+}\mu^{-}$ to $B\to Ke^{+}e^{-}$
decays, called $R_{K}$, is expected to be close to unity rkeq1 ; rkerr . A
recent measurement of $R_{K}$ by the LHCb Collaboration reported a $3.1\sigma$
discrepancy with respect to its SM prediction in the range of dilepton mass
squared $q^{2}\in\left(1.1,6.0\right)$ GeV2/$c^{4}$ lhcb . On the other hand,
the measurements reported by the Belle experiment are consistent with both the
SM and the LHCb result Belle_Rk , albeit with significantly less precision
compared to the latter.

In this report, we describe measurements of branching fractions
$\mathcal{B}\left(B\to J/\psi K\right)$, LFU ratios

$R_{K}{\left(J/\psi\right)}=\frac{\mathcal{B}\left(B\to
J/\psi(\mu^{+}\mu^{-})K\right)}{\mathcal{B}\left(B\to
J/\psi(e^{+}e^{-})K\right)},$ (1)

and isospin asymmetries

$A_{I}=\frac{\Gamma[B^{0}\to J/\psi(\ell^{+}\ell^{-})K^{0}]-\Gamma[B^{+}\to
J/\psi(\ell^{+}\ell^{-})K^{+}]}{\Gamma[B^{0}\to
J/\psi(\ell^{+}\ell^{-})K^{0}]+\Gamma[B^{+}\to
J/\psi(\ell^{+}\ell^{-})K^{+}]},$ (2)

performed using data recorded by the Belle II experiment. In contrast to
suppressed, charmless $B\to K\ell^{+}\ell^{-}$ decays, the $B\to
J/\psi(\ell^{+}\ell^{-})K$ decays involve a favored $b\to c$ tree-level
transition. Hence, contributions from beyond-the-SM physics are expected to
have a negligible impact. Since the branching fraction of $B\to
J/\psi(\ell^{+}\ell^{-})K$ is two orders of magnitude larger than that of
$B\to K\ell^{+}\ell^{-}$ decays and both channels share the same final-state
particles, the former decays constitute an excellent control sample for
studies of the latter. Therefore, the measurement of
$R_{K}\left(J/\psi\right)$ and its consistency with unity would be a strong
validation of the future $R_{K}$ measurement in the charmless counterpart of
$B\to K\ell\ell$ decays. Throughout the report, charge conjugate processes are
implicitly included.

Section 2 gives a brief introduction to the Belle II detector and the data
samples used in the analysis. Sections 3 and 4 describe the event selection
and background suppression methods. Section 5 explains the signal yield
determination and Section 6 presents the measurements of observables. Section
7 discusses systematic uncertainties and a summary is provided in Section 8.

##  2 The Belle II Detector and Data Sample

The Belle II detector is a large-solid-angle magnetic spectrometer designed to
study final-state particles produced in energy-asymmetric $e^{+}e^{-}$
collisions delivered by SuperKEKB SuperKEKB  at a center-of-mass energy
corresponding to the mass of the $\Upsilon(4S)$ resonance. The detector is
composed of several subdetectors arranged in a cylindrical structure around
the beam pipe. The subdetectors are two layers of silicon pixel detectors
(PXD), four layers of double-sided silicon strip detectors (SVD), a 56-layer
central drift chamber (CDC), a time-of-propagation counter (TOP) in the barrel
region, a proximity focusing ring imaging Cherenkov counter (ARICH) in the
forward region, and an electromagnetic calorimeter (ECL) comprising CsI(Tl)
crystals. All these subdetectors are located inside a superconducting solenoid
that provides a 1.5 T magnetic field. The return yoke of the magnet is
instrumented with plastic scintillators and resistive plate chambers to
identify $K^{0}_{L}$ mesons and muons (KLM). Further details about the
detector can be found in Ref. b2tdr .

The data sample used in this analysis amounts to an integrated luminosity of
189 fb-1, which is equivalent to $198\times 10^{6}$ $B\overline{B}$ events. To
study the properties of signal events, to optimize selection criteria, and to
determine the detection efficiency, $2\times 10^{6}$ simulated signal events
are generated for each of the $B^{+}\to J/\psi(e^{+}e^{-})K^{+}$, $B^{+}\to
J/\psi(\mu^{+}\mu^{-})K^{+}$, $B^{0}\to J/\psi(e^{+}e^{-})K_{S}^{0}$ and
$B^{0}\to J/\psi(\mu^{+}\mu^{-})K_{S}^{0}$ decay channels. In addition, we use
a simulated sample equivalent to 1 ab-1 of inclusive $B\overline{B}$ and
$q\overline{q}$ continuum events to study background processes. Here, $q$
denotes a $u$, $d$, $s$, or $c$ quark. The $B$ meson decays are generated
using the EvtGen EVTGEN  package, where the final-state-radiation effects from
charged particles are incorporated using the PHOTOS package PHOTOS . Continuum
background events are simulated using the KKMC KKMC  generator interfaced with
PYTHIA PYTHIA , and Geant4 GEANT4  simulates the interaction of generated
particles with the detector. The Belle II analysis software framework BASF2
is used to process both data and simulation events.

##  3 Event Selection and Reconstruction

The reconstruction follows a hierarchical procedure, where we first
reconstruct neutral and charged particles using information from ECL and
tracking subdetectors. Information from all subdetectors except the PXD and
SVD is used to identify charged particles as kaons, electrons, and muons.
Next, we take pairs of oppositely-charged leptons to reconstruct $J/\psi$
candidates. Pairs of charged particles are used to reconstruct $K^{0}_{S}$
candidates. Finally, we combine $J/\psi$ candidates with kaons, which are
either $K^{+}$ or $K^{0}_{S}$ candidates, to reconstruct $B$ mesons. The
remainder of this section describes each step in detail.

The information from PXD, SVD, and CDC is used to reconstruct charged
particles. We require that the distance of closest approach to the
$e^{+}e^{-}$ interaction point in the plane transverse to the $z$ axis be less
than 2.0 cm and that along the $z$ axis be less than 4.0 cm, to select charged
particles originating from a region around the collision point. The $z$ axis
is collinear with the symmetry axis of the solenoid approximately in the
$e^{-}$ beam direction. To suppress contamination from low-multiplicity
$e^{+}e^{-}\to e^{+}e^{-}\ell^{+}\ell^{-}$, $e^{+}e^{-}\to e^{+}e^{-}\gamma$
and $e^{+}e^{-}\to\mu^{+}\mu^{-}\gamma$ backgrounds, we require that the event
contains at least five charged particles, each with a transverse momentum
greater than 100 MeV/$c$, and that the normalized second Fox–Wolfram moment of
the event is less than 0.7 R2 .

From the set of selected tracks, we distinguish charged kaons from pions using
specific ionization information from the CDC, arrival time of the two-
dimensional information of a Cherenkov ring image from the TOP, and the number
of detected Cherenkov photons from the ARICH. The kaon identification
efficiency is 86% with a pion misidentification rate of 7%, estimated for
kaons within a momentum range $(0.5,4.5)\>\text{GeV}/c$ and polar-angle
acceptance $(0.49,2.62)$ rad. The penetration depth and transverse scattering
pattern in the KLM are used to identify muon candidates. Furthermore, we
require a minimum momentum of 0.8 GeV/$c$ to ensure that the candidate reaches
the KLM. These criteria result in a muon identification efficiency of 87% with
a pion misidentification rate of 7%, calculated using muons within the
momentum range $(0.4,6.5)\>\text{GeV}/c$ and polar-angle acceptance
$(0.40,2.60)$ rad. Electron candidates are identified mainly using the ratio
of the ECL energy to the momentum, the ECL shower shape, and the position
matching of the track with the calorimetric cluster. To ensure that electrons
reach the ECL, a momentum threshold of 0.5 GeV/$c$ is applied. The electron
identification efficiency is 94% and the pion misidentification rate is around
2%, estimated for electrons within the momentum range
$(0.4,7.0)\>\text{GeV}/c$ and polar-angle acceptance $(0.22,2.71)$ rad.
Relativistic electrons tend to lose their energy by radiating photons via the
bremsstrahlung process. For a given electron depositing energy in the ECL, the
four-momenta of all photons detected within a cone of 50 mrad with respect to
the initial momentum direction of the electron and having an energy greater
than 50 MeV are added to that of the electron to recover bremsstrahlung energy
loss.

The $K^{0}_{S}$ candidates are reconstructed from two oppositely charged
particles, assumed to be pions. A kinematic fit is performed on the
$K^{0}_{S}$ vertex to ensure that the pion tracks originate from a common
vertex. We retain candidates with an invariant mass in the range
$(0.487,0.508)\,{\rm GeV}\\!/c^{2}$, which is $\pm 3\sigma$ resolution around
the known $K^{0}_{S}$ mass. In addition, momentum-dependent selections are
applied on the $K^{0}_{S}$ flight length in the transverse plane, the
azimuthal angle between the momentum vector and the vector connecting the
interaction point with the decay vertex of the $K^{0}_{S}$ candidate, and the
difference between the distances of closest approach of the two pion tracks
along the $z$ axis.

We reconstruct $J/\psi$ candidates by combining two oppositely-charged leptons
of the same flavor. The invariant mass of the $J/\psi$ candidate must lie in
the range $(2.91,3.19)\,{\rm GeV}\\!/c^{2}$ and $(2.96,3.19)\,{\rm
GeV}\\!/c^{2}$ for the electron and muon channel, respectively. Imperfect
recovery of the energy lost due to bremsstrahlung motivates a wider asymmetric
interval around the known $J/\psi$ mass in the former case.

We reconstruct $B$ candidates by combining a kaon candidate ($K^{\pm}$ or
$K^{0}_{S}$) with a $J/\psi$ candidate. A vertex fit is performed on the $B$
candidates. We require the fit to converge in order to suppress background
coming from a random combination of charged particles. To distinguish signal
from background, the following two kinematic variables are used:

$M_{\rm bc}=\sqrt{s/4-\vec{p}^{\,\ast 2}_{B}}\text{ and}\\\ $ (3) $\Delta
E=E^{\ast}_{B}-\sqrt{s}/2,$ (4)

where ($E_{B}^{\ast},\vec{p}^{\,\ast}_{B}$) is the four-momentum of the $B$
candidate in the center-of-mass frame and $\sqrt{s}$ is the center-of-mass
energy. For a correctly reconstructed $B$ candidate, $M_{\rm bc}$ and $\Delta
E$ distributions are expected to peak around the known $B$ mass and zero,
respectively. We retain candidate events satisfying $M_{\rm
bc}\in\left(5.20,5.29\right)$ GeV/$c^{2}$ and $|\Delta E|<0.20$ GeV.

##  4 Background suppression

Following the selection, we find a major background contribution coming from
$B\to J/\psi K^{*}(K\pi)$ decays, in which $B$ candidates are reconstructed
without the final-state pion. As a result, the $\Delta E$ distribution for
such events peaks below $-0.10\,\mathrm{GeV}$. A selection of $\Delta
E>-0.10\,\mathrm{GeV}$ thus rejects about 98% of $B\to J/\psi K^{*}(K\pi)$
background events.

The background for the charged $B$ channels is mostly from $B^{+}\to
J/\psi\pi^{+}$ decays where the pion is misidentified as a kaon. These decays
peak inside the $B$ meson mass region of the $M_{\rm bc}$ distribution. Unlike
$M_{\rm bc}$, the $\Delta E$ variable is directly sensitive to the mass
hypotheses of the $B$ candidate decay products. Hence, these events show a
shifted peak in the $\Delta E$ distribution with respect to signal events. The
other background contribution arises from a random combination of $B$ decay
products, which does not peak in either the $M_{\rm bc}$ or $\Delta E$
distribution.

After applying all the selection criteria, the average candidate multiplicity
ranges between 1.005 and 1.010 depending on the decay channel. In case of
multiple candidates, we retain the one with the highest $\chi^{2}$ probability
of the $B$ vertex fit. The efficiency obtained from simulation to select the
correctly reconstructed signal from an event with multiple reconstructed $B$
candidates varies from 78 to 83% depending on the decay channel.

##  5 Signal Extraction

The signal yield is obtained from an extended maximum-likelihood fit to the
$M_{\rm bc}$ and $\Delta E$ distributions. For a data sample containing $N$
candidates, the likelihood function is given as

${\mathcal{L}}(\vec{\alpha},\vec{\beta},\vec{n})=\frac{e^{-\sum_{j}n_{j}}}{N!}\prod^{N}_{i=1}\sum_{j}n_{j}\mathcal{P}_{j}(M_{\rm
bc}^{i};{\vec{\alpha}_{j}})\mathcal{Q}_{j}(\Delta E^{i};{\vec{\beta}_{j}}),$
(5)

where $\mathcal{P}_{j}$ and $\mathcal{Q}_{j}$ are the probability density
functions (PDFs) of the $M_{\rm bc}$ and $\Delta E$ distributions, and $n_{j}$
is the number of events corresponding to the $j\text{th}$ component. We assume
the $M_{\rm bc}$ and $\Delta E$ distributions to be uncorrelated. The fit
performed for the neutral $B$ channels employs two components, one for the
correctly reconstructed signal and the other for misreconstructed background
events. The fit to charged channels has an additional component to model
$B^{+}\to J/\psi\pi^{+}$ background events. The argument $M_{\rm bc}^{i}$ and
$\Delta E^{i}$ denote the $M_{\rm bc}$ and $\Delta E$ values for the
$i\text{th}$ candidate; $\vec{\alpha}_{j}$ and $\vec{\beta}_{j}$ are the set
of shape parameters for $\mathcal{P}_{j}$ and $\mathcal{Q}_{j}$, respectively.
The likelihood function is maximized with respect to $\vec{\alpha}_{j}$,
$\vec{\beta}_{j}$, and $n_{j}$.

The PDF that describes the $\Delta E$ distribution of the signal component is
determined using correctly reconstructed events from the simulated signal
sample. The signal PDF is a sum of an empirical function RooCruijiff  and a
Gaussian. The signal $M_{\rm bc}$ PDF is parametrized by a function introduced
by the Crystal Ball Collaboration CB . This function accounts for radiative
tails in the distribution as well as for the finite photon energy resolution.
Other than the mean and width, the remaining signal shape parameters are fixed
to those obtained from the simulated sample. The $\Delta E$ distribution of
background is modeled by an exponential function, whose shape parameter is
determined from the fit to data. The background $M_{\rm bc}$ distribution is
parameterized by a threshold function introduced by the ARGUS Collaboration
ARGUS . The kinematic endpoint of the ARGUS PDF is fixed to $\sqrt{s}/2\approx
5.291$ GeV$/c^{2}$ with the other shape parameter determined from the fit to
data. The $M_{\rm bc}$ and $\Delta E$ distributions for the $B^{+}\to
J/\psi\pi^{+}$ component are modeled using a Gaussian function, whose shape
parameters are also fixed to those obtained from the simulated sample. The
yield of $B^{+}\to J/\psi\pi^{+}$ events is fixed to a value estimated using
the known branching fraction PDG  and the $\pi\to K$ misidentification rate
obtained in auxiliary data. We estimate around 11 and 14 events coming from
$B^{+}\to J/\psi(e^{+}e^{-})\pi^{+}$ and $B^{+}\to
J/\psi(\mu^{+}\mu^{-})\pi^{+}$ background, respectively. The distributions of
$M_{\rm bc}$ and $\Delta E$ for each $B$ channel are shown in Figs. 1 and 2,
respectively; the fit results are superimposed.

To check for potential fit biases, an ensemble of 1000 simulated data samples
is prepared from 1 ab-1 of fully-simulated events using sampling with
replacement bootstrap . The number of signal and background events in each
data sample are drawn from a Poisson distribution, with the mean corresponding
to the total number of events expected in the data. Each of these simulated
data samples is fit. The estimators are unbiased and have Gaussian
distributions, validating our assumption that in simulated data $M_{\rm bc}$
and $\Delta E$ distributions are uncorrelated.

Figure 1: $M_{\rm bc}$ distributions for each $B\to J/\psi(\ell^{+}\ell^{-})K$
channel with the fit result superimposed (top) and pull distribution with
respect to the fit result (bottom), where the pull is defined as the
difference between the fit result and the value of the distribution in a bin,
divided by the estimated uncertainty in that bin. Black dots with error bars
denote the data, blue curves denote the total fit, dashed red curves are the
signal component, dotted green curves are the background component, and filled
cyan regions in the charged channels are the $B^{+}\to J/\psi\pi^{+}$
component.

Figure 2: $\Delta E$ distributions for each $B\to J/\psi(\ell^{+}\ell^{-})K$
channel with the fit result superimposed (top) and pull distribution with
respect to the fit result (bottom). Black dots with error bars denote the
data, blue curves denote the total fit, dashed red curves are the signal
component, dotted green curves are the background component, and filled cyan
regions in the charged channels are the $B^{+}\to J/\psi\pi^{+}$ component.

##  6 Measurement of observables

The branching fractions are determined using the relation

$\mathcal{B}=\frac{n_{\rm sig}}{2\,N_{B\overline{B}}\,f^{i}\,\epsilon},$ (6)

where ${n}_{\rm sig}$ is the signal yield determined by the fit,
$N_{B\overline{B}}$ is the number of $B\overline{B}$ events, $\epsilon$ is the
signal selection efficiency, $f^{i}$ is $f^{\pm}$ for the charged channels and
$f^{00}$ for the neutral channels; $f^{\pm}(f^{00})$ is the branching fraction
of $\Upsilon(4S)$ to charged (neutral) $B\overline{B}$ pairs assuming
$\mathcal{B}\left(\Upsilon(4S)\to B\overline{B}\right)=1$. We use the values
$f^{00}=0.487\pm 0.013$ and $f^{\pm}=1-f^{00}=0.513\pm 0.013$ reported in Ref.
fpmf00  rather than the world average HFLAV , as the latter is dominated by
measurements that assume isospin symmetry in $B\to J/\psi K$ decays.

We calculate $R_{K}$ using the relation

$R_{K}{\left(J/\psi\right)}=\frac{\mathcal{B}\left(J/\psi(\mu^{+}\mu^{-})K\right)}{\mathcal{B}\left(J/\psi(e^{+}e^{-})K\right)}=\frac{n_{\rm
sig}^{J/\psi(\mu^{+}\mu^{-})K}}{n_{\rm
sig}^{J/\psi(e^{+}e^{-})K}}\times\frac{\epsilon^{J/\psi(e^{+}e^{-})K}}{\epsilon^{J/\psi(\mu^{+}\mu^{-})K}},$
(7)

and $A_{I}$ using the relation

$A_{I}=\frac{2\,(\tau_{B^{+}}/\tau_{B^{0}})(f^{\pm}/f^{00})(n_{\rm
sig}/\epsilon)|_{J/\psi(\ell^{+}\ell^{-})K_{S}^{0}}-(n_{\rm
sig}/\epsilon)|_{J/\psi(\ell^{+}\ell^{-})K^{+}}}{2\,(\tau_{B^{+}}/\tau_{B^{0}})(f^{\pm}/f^{00})(n_{\rm
sig}/\epsilon)|_{J/\psi(\ell^{+}\ell^{-})K_{S}^{0}}+(n_{\rm
sig}/\epsilon)|_{J/\psi(\ell^{+}\ell^{-})K^{+}}},$ (8)

where $(\tau_{B^{+}}/\tau_{B^{0}})=1.076\pm 0.004$ PDG  is the ratio of the
lifetimes of the charged and neutral $B$ meson, and $(f^{\pm}/f^{00})=1.053\pm
0.052$ fpmf00 . The factor of 2 in Eq. (8) arises because a $K^{0}$ forms a
$K^{0}_{S}$ meson half the time. Results are listed in Tables 1 and 2.

Table 1: Reconstruction efficiency, signal yield, and branching fraction for each channel. Where two uncertainties are given, the first is statistical and the second is systematic, and the single uncertainty corresponds to the statistical component. Channel |  $\epsilon$ (%) | $n_{\text{\rm sig}}$ |  |  $\mathcal{B}$ ($10^{-5}$)
---|---|---|---|---
$B^{+}\to J/\psi(e^{+}e^{-})K^{+}$ | 30.4 | $3706\pm 62$ |  | $6.00\pm 0.10\pm 0.19$
$B^{+}\to J/\psi(\mu^{+}\mu^{-})K^{+}$ | 37.2 | $4578\pm 62$ |  | $6.06\pm 0.09\pm 0.19$
$B^{0}\to J/\psi(e^{+}e^{-})K_{S}^{0}$ | 20.4 | $1052\pm 33$ |  | $2.67\pm 0.08\pm 0.12$
$B^{0}\to J/\psi(\mu^{+}\mu^{-})K_{S}^{0}$ | 25.0 | $1343\pm 37$ |  | $2.78\pm 0.08\pm 0.12$
Table 2: Measured $A_{I}$ and $R_{K}\left(J/\psi\right)$, where the first quoted uncertainty is statistical and the second is systematic. Observable | Measured value
---|---
$A_{I}\left(J/\psi(ee)K\right)$ | $-0.022\pm 0.016\pm 0.030$
$A_{I}\left(J/\psi(\mu\mu)K\right)$ | $-0.006\pm 0.015\pm 0.030$
$R_{K^{+}}\left(J/\psi\right)$ | $\phantom{-}1.009\pm 0.022\pm 0.008$
$R_{K_{S}^{0}}\left(J/\psi\right)$ | $\phantom{-}1.042\pm 0.042\pm 0.008$

##  7 Systematic Uncertainties

We consider several sources of systematic uncertainty contributing to the
measurements. The nominal fits are performed with some PDF shape parameters
fixed to values obtained from simulated events. To evaluate the systematic
uncertainties associated with this, the data are refit by varying each fixed
shape parameter by $\pm 1\sigma$, with the obtained variation in signal yield
taken as the systematic uncertainty. Here $\sigma$ is the data driven
uncertainty on each of these shape parameters, obtained by fitting to data. We
assign 0.2% uncertainty for the fixed shape parameters. An overall uncertainty
of 1.5% is assigned as a systematic uncertainty due to the estimate of the
number of $B\overline{B}$ events. From the study of data–simulation
differences in a sample of $e^{+}e^{-}\to\tau^{+}\tau^{-}$, we assign a 0.3%
systematic uncertainty for each charged track in the final state tracking .
For the branching fraction measurements, we assign 0.9% and 1.2% systematic
uncertainties for $B^{+}$ and $B^{0}$ channels, respectively. For the
measurement of $A_{I}$, since the leptons are common in both charged and
neutral channels, their correction factor cancels, and we assign a 0.9%
systematic uncertainty for the charged kaon and two decay product tracks of
the neutral kaon. The 0.1% uncertainty in efficiency due to the limited
simulation sample size is also considered as a source of systematic
uncertainty.

From studies performed with a sample of $D^{*+}\to D^{0}\left(\to
K^{0}_{S}\pi^{+}\pi^{-}\right)\pi^{+}$ decays, we observe that the
data–simulation ratio of the $K^{0}_{S}$ reconstruction efficiency changes
linearly as a function of the flight distance, which corresponds to an
uncertainty of 0.4% per cm of average flight length. The average flight length
of $K^{0}_{S}$ for the signal channel is estimated to be 7.6 cm using
simulated signal events. Hence, we assign a 3.0% systematic uncertainty for
branching fraction and isospin asymmetry measurements involving a $K^{0}_{S}$
meson. A $D^{*+}\to D^{0}\left(\to K^{-}\pi^{+}\right)\pi^{+}$ sample is used
to estimate the difference in the kaon identification efficiency between data
and simulation. The correction factors are calculated as functions of particle
momentum and polar angle. We assign a 0.2% systematic uncertainty for
measurements other than $R_{K}\left(J/\psi\right)$. An inclusive
$J/\psi\to\ell^{+}\ell^{-}$ sample is used to compute the difference in lepton
identification efficiency between data and simulation, similar to the case of
kaon identification. We assign a $0.6\%$ and $0.4\%$ systematic uncertainty
for electrons and muons, respectively.

As we reconstruct $K^{0}_{S}$ candidates from their $\pi^{+}\pi^{-}$ decays,
we use $\mathcal{B}\left(K^{0}_{S}\to\pi^{+}\pi^{-}\right)=(69.20\pm 0.05)\%$
to account for the other $K^{0}_{S}$ decay channels PDG . We assign a 0.1%
systematic uncertainty from this source to the measurement of $\mathcal{B}$ of
neutral channels and $A_{I}$. We use the measured branching fraction fpmf00
of the $\Upsilon(4S)$ decay to a charged (neutral) $B$ meson pair, or
$f^{\pm}$ ($f^{00}$). The associated systematic uncertainty for either of them
is 2.6%. The ratio $f^{\pm}/f^{00}$ contributes a 5.2% systematic uncertainty
for the $A_{I}$ measurement. The assumed value of
$(\tau_{B^{+}}/\tau_{B^{-}})$ leads to a 0.4% uncertainty in the $A_{I}$
measurement PDG .

We summarize the sources of systematic uncertainties for $\mathcal{B}$,
$R_{K}\left(J/\psi\right)$, and $A_{I}$ in Table 3. The individual sources of
uncertainties are assumed to be independent, and the corresponding
contributions are added in quadrature to obtain the total uncertainty. The
uncertainties for branching fractions and $R_{K}$ are relative, while the
uncertainty for $A_{I}$ is absolute.

Table 3:  Relative systematic uncertainties (%) on $\mathcal{B}\left(B\to J/\psi K\right)$, $R_{K}\left(J/\psi\right)$, and absolute uncertainty on $A_{I}\left(B\to J/\psi K\right)$.  Source | $\mathcal{B}\left(B\to KJ/\psi\right)$ | $R_{K}$ | $A_{I}$
---|---|---|---
| $K^{+}$ | $K^{+}$ | $K_{S}^{0}$ | $K_{S}^{0}$ | $K^{+}$ | $K^{0}$ |  |
| $e^{+}e^{-}$ | $\mu^{+}\mu^{-}$ | $e^{+}e^{-}$ | $\mu^{+}\mu^{-}$ |  |  | $e^{+}e^{-}$ | $\mu^{+}\mu^{-}$
Number of $B\overline{B}$ events | 1.5 | 1.5 | 1.5 | 1.5 | – | – | – | –
PDF shape | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 | 0.1 | 0.1
Electron identification | $0.6$ | – | $0.6$ | – | $0.6$ | $0.6$ | – | –
Muon identification | – | $0.4$ | – | $0.4$ | $0.4$ | $0.4$ | – | –
Kaon identification | 0.2 | 0.2 | – | – | – | – | 0.1 | 0.1
$K^{0}_{S}$ reconstruction | – | – | 3.0 | 3.0 | – | – | 1.5 | 1.5
Tracking efficiency | 0.9 | 0.9 | 1.2 | 1.2 | – | – | 0.4 | 0.4
Simulation sample size | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
$\Upsilon(4S)$ branching fraction | 2.6 | 2.6 | 2.6 | 2.6 | – | – | 2.6 | 2.6
$(\tau_{B^{+}}/\tau_{B^{0}})$ | – | – | – | – | – | – | 0.2 | 0.2
Total | $3.2$ | $3.2$ | $4.4$ | $4.4$ | $0.8$ | $0.8$ | $3.0$ | $3.0$

##  8 Summary

We report the measurements of $\mathcal{B}\left(B\to J/\psi K\right)$,
$A_{I}$, and $R_{K}$ of $B\to J/\psi(\ell^{+}\ell^{-})K$ decays performed by
the Belle II experiment. The results

$\displaystyle\mathcal{B}\left(B^{+}\to J/\psi(e^{+}e^{-})K^{+}\right)$
$\displaystyle=$ $\displaystyle(6.00\pm 0.10\pm 0.19)\times 10^{-5},$
$\displaystyle\mathcal{B}\left(B^{+}\to J/\psi(\mu^{+}\mu^{-})K^{+}\right)$
$\displaystyle=$ $\displaystyle(6.06\pm 0.09\pm 0.19)\times 10^{-5},$
$\displaystyle\mathcal{B}\left(B^{0}\to J/\psi(e^{+}e^{-})K_{S}^{0}\right)$
$\displaystyle=$ $\displaystyle(2.67\pm 0.08\pm 0.12)\times 10^{-5},$
$\displaystyle\mathcal{B}\left(B^{0}\to
J/\psi(\mu^{+}\mu^{-})K_{S}^{0}\right)$ $\displaystyle=$
$\displaystyle(2.78\pm 0.08\pm 0.12)\times 10^{-5},$ $\displaystyle
A_{I}\left(B\to J/\psi(e^{+}e^{-})K\right)$ $\displaystyle=$
$\displaystyle-0.022\pm 0.016\pm 0.030,$ $\displaystyle A_{I}\left(B\to
J/\psi(\mu^{+}\mu^{-})K\right)$ $\displaystyle=$ $\displaystyle-0.006\pm
0.015\pm 0.030,$ $\displaystyle R_{K^{+}}\left(J/\psi\right)$ $\displaystyle=$
$\displaystyle 1.009\pm 0.022\pm 0.008,\text{ and}$ $\displaystyle
R_{K^{0}}\left(J/\psi\right)$ $\displaystyle=$ $\displaystyle 1.042\pm
0.042\pm 0.008,$

are consistent with the world average values PDG . Because the signal
selection efficiency of electron channels is comparable to that of muon
channels, we can have uncertainties in the $R_{K}$ result that are almost
equally contributed by the two lepton flavors. The $B\to J/\psi K$ decays will
serve as control channels for the $B\to K\ell^{+}\ell^{-}$ study.

##  9 Acknowledgement

We thank the SuperKEKB group for the excellent operation of the accelerator,
the KEK cryogenics group for the efficient operation of the solenoid, and the
KEK computer group for on-site computing support.

## References

  * (1) S. L. Glashow, J. Iliopoulos, and L. Maiani, Phys. Rev. D 2, 1285 (1970).
  * (2) S. L. Glashow, D. Guadagnoli, and K. Lane, Phys. Rev. Lett. 114, 091801 (2015), arXiv:1411.0565 [hep-ph].
  * (3) M. Bauer and M. Neubert, Phys. Rev. Lett. 116, 141802 (2016),  arXiv:1511.01900 [hep-ph].
  * (4) G. Hiller and F. Krüger, Phys. Rev. D 69, 074020 (2004), arXiv:hep-ph/0310219.
  * (5) S. Descotes-Genon et al., JHEP 06, 092 (2016), arXiv:1510.04239 [hep-ph].
  * (6) R. Aaij et al. (LHCb Collaboration), Nat. Phys. 18, 277 (2022), arXiv:2103.11769 [hep-ex].
  * (7) S. Choudhury et al. (Belle Collaboration), JHEP 03, 105 (2021), arXiv:1908.01848 [hep-ex].
  * (8) K. Akai et al. (SuperKEKB Accelerator Team), Nucl. Instrum. Meth. A 907, 188 (2018), arXiv:1809.01958 [physics.acc-ph].
  * (9) T. Abe et al. (Belle II Collaboration), arXiv:1011.0352 [physics.ins-det].
  * (10) D. J. Lange, Nucl. Instrum. Meth. A 462, 152 (2001).
  * (11) E. Barberio, B. van Eijk, and Z. Was, Comp. Phys. Comm. 66, 115 (1991).
  * (12) S. A. Yost et al., PoS RADCOR2017 023 (2018), arXiv:1801.03560 [hep-ph].
  * (13) T. Sjöstrand, S. Mrenna, and P. Skands, Comp. Phys. Comm. 178, 852 (2008), arXiv:0710.3820 [hep-ph].
  * (14) S. Agostinelli et al., Nucl. Instrum. Meth. A 506, 250 (2003), arXiv:0710.3820 [hep-ph].
  * (15) T. Kuhr et al., Comput. Softw. Big Sci. 3, 1 (2019).
  * (16) G. C. Fox and S. Wolfram,  Phys. Rev. Lett. 41, 1581 (1978).
  * (17) P. del Amo Sanchez et al. (BaBar Collaboration), Phys. Rev.D 82, 051101 (2010).
  * (18) T. Skwarnicki, PhD thesis, INP Krakow, DESY-F31-86-02 (1986).
  * (19) H. Albrecht et al. (ARGUS Collaboration), Phys. Lett. B 340, 217 (1994)
  * (20) P. A. Zyla et al. (Particle Data Group), Prog. Theor. Exp. Phys. 083C01 (2020).
  * (21) B. Efron, Ann. Statist. 7, 1 (1979).
  * (22) B. Aubert et al. (BaBar Collaboration), Phys. Rev. Lett. 95, 042001 (2005), arXiv:hep-ex/0504001.
  * (23) Y. Amhis et al. (Heavy Flavor Averaging Group), Eur. Phys. J. C 81, 226 (2021), arXiv:1909.12524 [hep-ex].
  * (24) V. Bertacchi et al. (Belle II Tracking Group), Comp. Phys. Comm. 259, 107610 (2021), arXiv:2003.12466 [physics.ins-det].


Assessment of Differentially Private Synthetic Data for Utility and Fairness
in End-to-End Machine Learning Pipelines for Tabular Data

Mayana Pereira1,2,*, Meghana Kshirsagar1, Sumit Mukherjee3, Rahul Dodhia1,
Juan Lavista Ferres1, Rafael de Sousa2

1 AI for Good Research Lab, Microsoft, Redmond, Washington, U.S.A.

2 Department of Electrical Engineering, University of Brasilia, Brasilia,
Brazil

3 INSITRO, San Francisco, CA, U.S.A.

*<EMAIL_ADDRESS>
## Abstract

Differentially private (DP) synthetic data sets are a solution for sharing
data while preserving the privacy of individual data providers. Understanding
the effects of utilizing DP synthetic data in end-to-end machine learning
pipelines impacts areas such as health care and humanitarian action, where
data is scarce and regulated by restrictive privacy laws. In this work, we
investigate the extent to which synthetic data can replace real, tabular data
in machine learning pipelines and identify the most effective synthetic data
generation techniques for training and evaluating machine learning models. We
systematically investigate the impacts of differentially private synthetic
data on downstream classification tasks from the point of view of utility as
well as fairness. Our analysis is comprehensive and includes representatives
of the two main types of synthetic data generation algorithms: marginal-based
and GAN-based.

To the best of our knowledge, our work is the first that: (i) proposes a
training and evaluation framework that does not assume that real data is
available for testing the utility and fairness of machine learning models
trained on synthetic data; (ii) presents the most extensive analysis of
synthetic data set generation algorithms in terms of utility and fairness when
used for training machine learning models; and (iii) encompasses several
different definitions of fairness.

Our findings demonstrate that marginal-based synthetic data generators surpass
GAN-based ones regarding model training utility for tabular data. Indeed, we
show that models trained using data generated by marginal-based algorithms can
exhibit similar utility to models trained using real data. Our analysis also
reveals that the marginal-based synthetic data generator MWEM PGM can train
models that simultaneously achieve utility and fairness characteristics close
to those obtained by models trained with real data.

## Introduction

Differential privacy (DP) is the standard for privacy-preserving statistical
summaries [1]. Companies such as Microsoft [2], Google [3], Apple [4], and
government organizations such as the US Census [5], have successfully applied
DP in machine learning and data sharing scenarios. The popularity of DP is due
to its strong mathematical guarantees. Differential Privacy guarantees privacy
by ensuring that the inclusion or exclusion of any particular individual does
not significantly change the output distribution of an algorithm.

In areas ranging from health care, humanitarian action, education, and
socioeconomic studies, the publication and sharing of data is crucial for
informing society and scientific collaboration. However, the disclosure of
such data sets can often reveal private, sensitive information. Privacy-
preserving data publishing aims at enabling such collaborations while
preserving the privacy of individual entries in the data set.
Tabular/categorical data about individuals are relevant in many applications,
from health care to humanitarian action. Privacy-preserving data publishing
for such data can be done in the form of a synthetic data table that has the
same schema and similar distributional properties as the real data. The aim
here is to release a perturbed version of the original information, so that it
can still be used for statistical analysis, but the privacy of individuals in
the database is preserved.

The biggest advantage of synthetic data sets is that, once released, all data
analysis and machine learning tasks are performed in the same way it is done
with real data. As noted by [6], the switch between real and synthetic data in
data analysis and machine learning pipelines is seamless - the same analysis
tools, libraries and algorithms are applied in the same manner in both data
sets. Other privacy-preserving technologies, such as federated learning,
requires expertise and appropriate tools to perform data analysis and model
training.

Due to the all the potential benefits of synthetic data, understanding the
impacts of synthetic data in downstream classification tasks have become of
extreme importance. A trend observed in recent studies is to evaluate
performance of synthetic data generators of two types: marginal-based
synthesizers [7] and generative adversarial networks (GAN) based synthesizers
[8, 9, 6]. Marginal-based synthetic data generators are suitable for tabular
data only, and have gained increased popularity after the algorithm MST won
the NIST competition in 2018 [10]. Marginal-based synthesizers are named as
such due to the fact that they learn approximate data distributions by
querying noisy marginals from the real data. Notable marginal-based algorithms
are MWEM PGM [11] and PrivBayes [12]. GAN-based synthesizers, on the other
hand, are flexible algorithms, and are suitable for tabular, image and other
data formats. GANs learn patterns and relationships from the input data based
on a game, in the sense of game theory, between two machine learning models, a
discriminator model and the generator model. Among popular differentially
private GAN architectures we list DP-GAN [13], DP-CTGAN [14] , PATE-GAN [15]
and PATE-CTGAN [14].

One of the major applications of synthetic data is for training machine
learning models. Therefore, it is paramount to understand how exchanging real
data for synthetic data impacts the performance of the trained machine
learning models. By performance, we mean not only the utility of the model
(its accuracy, for example) but also how well the model performs for different
subgroups of the data set - the fairness of the model. The impact of machine
learning models on minorities subgroups is an active area of research, and
several works have investigated the trade-offs among model accuracy, bias, and
privacy [16, 17, 18, 19]. However, only recently bias caused by the use of
synthetic data in downstream classification received attention [20, 7, 21].
This problem becomes particularly relevant in the context of synthetic data
sets generated with differential privacy guarantees. It is known that
differential privacy can affect fairness in machine learning models [17].
Despite recent work investigating the impact of synthetic data in downstream
model fairness [20, 8], there are important questions that remain unanswered.

  * •

There is no published work that systematically studies the utility and
fairness of machine learning models trained on several GAN based and marginal-
based synthetic tabular data set generation algorithms.

  * •

Previous studies have not evaluated machine learning models trained on
synthetic data set generation algorithms for multiple definitions of fairness.

  * •

In previous studies, it was always assumed that real data was available for
evaluating the fairness of models trained on synthetic data. Here, we propose
and evaluate a pipeline where no such assumption is necessary.

Contributions In this work, we investigate the impacts of differentially
private synthetic data on downstream classification, where we focus on
understanding the impacts on model utility and fairness. Our investigation
focus on two aspects of such impact:

  * •

What is the impact in model utility when utilizing synthetic data for training
machine learning models? Can synthetic data also be used to evaluate utility
of machine learning models?

  * •

What is the impact in model fairness when utilizing synthetic data for
training machine learning models? Can synthetic data be used to evaluate
fairness of machine learning models?

In our investigations we also evaluate if there are clear differences in
performance between marginal-based and GAN-based synthetic data, and if there
is a synthesizer algorithm that produces data that clearly outperform others.

Fig 1: Pipeline for model training and evaluation using synthetic data (1) We
generate Synthetic data sets for model training and model testing utilizing
differentially private synthesizers. (2) We train models utilizing synthetic
data and evaluate on a synthetic test data. Model selection is made during
this phase. (3) Based on the previous phase results, model is trained using
synthetic data and deployed. Model is applied to real (test) data in
production phase.

Our research work evaluates the impact of utilizing synthetic data sets for
both training and testing in machine learning pipelines. We empirically
compare the performance of marginal-based synthesizers and GAN-based
synthesizers within the context of a machine learning pipeline. Our
experiments yield a comprehensive analysis, encompassing utility and fairness
metrics. Our main contributions are:

  * •

We propose a training and evaluation framework that does not assume that real
data is available for testing the utility and fairness of machine learning
models trained on synthetic data.

  * •

We present an extensive analysis of synthetic data set generation algorithms
in terms of utility and fairness when used for training machine learning
models. In particular, this is the first systematic comparison of several
marginal-based and GAN-based algorithms for fairness and utility of the
resulting machine learning models.

  * •

This is the first of such studies that includes several different definitions
of fairness.

Main Findings:

  * 1

Marginal-based synthetic data can accurately train machine learning models for
tabular data. Marginal-based synthetic data can train models with similar
utility to models trained on real data. Our experiments show that for a
privacy-loss parameter $\epsilon>5.0$, models trained with MWEM PGM (AUC =
0.684), MST (AUC = 0.662) and Privbayes (AUC = 0.668) provides utility very
similar to models trained on real data (AUC = 0.684). Additionally, we
evaluated models using synthetic data, and found that marginal-based synthetic
provides a good evaluation, with synthetic data providing an AUC = 0.671
versus AUC = 0.684 (measured using real data).

  * 2

Synthetic data sets trained with MWEM PGM can be used for accurate model
training and fairness evaluation in the case of tabular data. We found that
MWEM PGM synthetic data can train models that achieves very similar utility
and fairness characteristics of models trained with real data. Additionally,
the synthetic data generated by MWEM PGM algorithm showed very similar
behavior to real data when used to evaluate utility an fairness of machine
learning models. This is the first study that (first time that it is showing
that synthetic data can actually present reliable behavior and a potential
substitute for real data sets in end-to-end machine learning pipelines)

This work significantly extends and sub sums a previous version, presented at
the _Machine Learning for Data: Automated Creation, Privacy, Bias Workshop_ at
the _International Conference on Machine Learning (ICML)_ (workshop without
proceedings) [22].

##  1 Related Works

As synthetic data generation becomes standard practice for data sharing and
publishing, understanding the impacts of utilizing synthetic data in machine
learning pipelines is of significant importance. Although previous works have
advised against using synthetic data to train and evaluate any final tools
deployed in the real world [23], in very sensitive scenarios, such as human
trafficking data [24], synthetic data might be the only available data for
training and testing models.

The promises synthetic data brings generated an interest in understanding
impacts of utilizing synthetic in data analysis and machine learning. Some of
these works include analysing the utility of differentially private synthetic
data in different tasks [25], investigating if training models with
differentially private synthetic images can increase subgroup disparities [8],
the impacts different types of synthetic data can have in model fairness [20,
26], utility of synthetic data in downstream health care classification
systems [7], and whether feature importance can be accurately analyzed using
differentially private synthetic data [21]. All these works are ultimately
trying to answer a same question: to which extent can we substitute real data
with synthetic data, and which are the best synthetic data generation
techniques for model training?

However these works still left questions unanswered. First of all, there
hasn’t been a systematic study of impacts of using synthetic data sets in end-
to-end machine learning pipelines, which means evaluating the use of synthetic
data for model training and model evaluation. Additionally, there has been a
lot of focus on image classification tasks [8, 20] where the disparity in
accuracy are largely attributable to the class imbalance in these data sets:
i.e disadvantaged classes are also rare classes in the data set thereby
leading to worse performance on these. In contrast, our work studies these
issues in the context of tabular data sets and in settings where the data has
an intrinsic bias against sub-populations that are not necessarily rare in the
data set. Moreover, our work focus on comparing two types of data
synthetization algorithm families: marginal-based and GAN-based data
synthesizers. While, these two type of data synthetization algorithms have
been previously compared for utility [25], no such extensive comparative
analysis exists for fairness.

We are the first to extensively study the differences of applying data
generated by these two families types of data synthetization algorithms in
end-to-end machine learning pipelines for utility and multiple fairness
metrics.

##  2 Preliminaries

In this section we introduce the concepts of differential privacy and
algorithmic fairness. We refer the reader to [1, 27, 28] for detailed
explanation of these concepts. Additionally, we describe the synthetic data
generation techniques and the data sets used in our experiments.

###  2.1 Differential privacy

Differential privacy is a rigorous privacy notion used to protect an
individual’s data in a data set disclosure. We present in this section
notation and definitions that we will use to describe our privatization
approach. We refer the reader to [29], [30] and [31] for detailed explanations
of these definitions and theorems.

Pure Differential Privacy. A randomized mechanism
$\mathcal{M}:\mathcal{D}\rightarrow\mathcal{A}$ with data base domain
$\mathcal{D}$ and output set $\mathcal{A}$ is $\epsilon$-differentially
private if, for any output $A\subseteq\mathcal{Y}$ and neighboring databases
$D,D^{\prime}\in\mathcal{D}$ (i.e., $D$ and $D^{\prime}$ differ in at most one
entry), we have

$\textnormal{Pr}[\mathcal{M}(D)\in A]\leq
e^{\epsilon}\textnormal{Pr}[\mathcal{M}(D^{\prime})\in A]$

Approximate Differential Privacy. A randomized mechanism
$\mathcal{M}:\mathcal{D}\rightarrow\mathcal{A}$ with data base domain
$\mathcal{D}$ and output set $\mathcal{A}$ is
$(\epsilon,\delta)$-differentially private if, for any output
$A\subseteq\mathcal{Y}$ and neighboring databases $D,D^{\prime}\in\mathcal{D}$
(i.e., $D$ and $D^{\prime}$ differ in at most one entry), we have

$\textnormal{Pr}[\mathcal{M}(D)\in A]\leq
e^{\epsilon}\textnormal{Pr}[\mathcal{M}(D^{\prime})\in A]+\delta$

The privacy loss of the mechanism is defined by the parameter $\epsilon\geq 0$
in the case of ’pure’ differential privacy and parameters $\epsilon,\delta\geq
0$ in the case of ’approximate’ differential privacy.

The definition of neighboring databases used in this paper is user-level
privacy. User-level privacy defines neighboring to be the addition or deletion
of a single user in the data and all possible records of that user.
Informally, the definition above states that the addition or removal of a
single individual in the database does not provoke significant changes in the
probability of any differentially private output. Therefore, differential
privacy limits the amount of information that the output reveals about any
individual.

A function $f$ (also called query) from a data set $D\in\mathcal{D}$ to a
result set $A\subseteq\mathcal{A}$ can be made differentially private by
injecting random noise to its output. The amount of noise depends on the
sensitivity of the query.

###  2.2 Fairness Metrics

In this section we present the definition of two different fairness metrics:
Equal Opportunity [27] and Statistical Disparity[28]. Given a data set
$W=(X,Y^{\prime},C)$ with binary protected attribute $C$ (e.g. race, sex,
religion, etc), remaining decision variables $X$ and predicted outcome
$Y^{\prime}$, we define Equal Opportunity and Statistical Disparity as
follows.

Equal Opportunity/ Equality of Odds requires equal True Positive Rate (TPR)
across subgroups:

$\textnormal{Pr}(Y^{\prime}=1|Y=1,C=0)=\textnormal{Pr}(Y^{\prime}=1|Y=1,C=1)$

where Y’ is the model output.

Statistical Parity requires positive predictions to be unaffected by the value
of the protected attribute, regardless of true label

$\textnormal{Pr}(Y^{\prime}=1|C=0)=\textnormal{Pr}(Y^{\prime}=1|C=1)$

We follow the approach of [32, 33] and utilize difference in Equal Oportunity
(DEO) =
$|\textnormal{Pr}(Y^{\prime}=1|Y=1,C=0)-\textnormal{Pr}(Y^{\prime}=1|Y=1,C=1)|$
and difference in Statistical Parity (DSP) =
$|\textnormal{Pr}(Y^{\prime}=1|C=0)-\textnormal{Pr}(Y^{\prime}=1|C=1)|$ to
measure model fairness.

###  2.3 Differentially Private Synthetic Data Generators.

We use several differentially private (DP) synthetic data generators that have
been specifically tailored for generating tabular data with the goal of
enhancing their utility for learning tasks. We consider two broad categories
of approaches: i) marginal-based methods, ii) and Generative Adversarial
Network (GAN) based models.

####  2.3.1 Marginal-based methods

##### MWEM PGM

Is a variation of the multiplicative weights with exponential mechanism
algorithm (MWEM), which is an algorithm that generated synthetic data based on
linear queries. The algorithm aims to produce a data distribution that
produces query answers similar answers resulted when querying the real data
set. The MWEM PGM variation combines probabilistic graphical models with the
MWEM algorithm. The structure of the graphical model is determined by the
measurements, such that no information is lost relative to a full contingency
table representation.

##### MST

Is a synthetic data generation algorithm that acts selecting 2- and 3-way
marginals for measurement. It combines one principled step, which is to find
the maximum spanning tree (MST) on the graph where edge weights correspond to
mutual information between two attributes, with some additional heuristics to
ensure that certain important attribute pairs are selected, and a final step
to select triples while keeping the graph tree-like.

##### PrivBayes

In order to improve the utility of the generated synthetic data, [12]
approximates the actual distribution of the data by constructing a Bayesian
network using the correlations between the data attributes. This allows them
to factorize the joint distribution of the data into marginal distributions.
Next, to ensure differential privacy, noise is injected into each of the
marginal distributions and the simulated data is sampled from the approximate
joint distribution constructed from these noisy marginals.

####  2.3.2 GAN-based methods

Generative neural networks (GANs) are a type of artificial neural network used
in machine learning for generating new data samples similar to a given
training data set. Generative adversarial networks are based on a game, in the
sense of game theory, between two machine learning models, a discriminator
model $D$ and the generator $G$ model. The goal of the generator is to learn
realistic samples that can fool the discriminator, while the goal of the
discriminator is to be able to tell generator generated samples from real ones
[13].

Conditional Tabular GAN (CTGAN) [34] is an approach for generating tabular
data. CTGAN adapts GANs by addressing issues that are unique to tabular data
that conventional GANs cannot handle, such as the modeling of multivariate
discrete and mixed discrete and continuous distributions. It achieves these
challenges by augmenting the training procedure with mode-specific
normalization, and by employing a conditional generator and training-by-
sampling that allows it to explore discrete values more evenly. When applying
differentially private SGD (DP-SGD) [35] in combination with CTGAN the result
is a DP approach for generating tabular data.

The PATE (Private Aggregation of Teacher Ensembles) framework [36] protects
the privacy of sensitive data during training, by transferring knowledge from
an ensemble of teacher models trained on partitions of the data to a student
model. To achieve DP guarantees, only the student model is published while
keeping the teachers private. The framework adds Laplacian noise to the
aggregated answers from the teachers that are used to train the student
models. CTGAN can provide differential privacy by applying the PATE framework.
We call this combination PATE-CTGAN, which is similar to PATE-GAN [15], for
images. The original data set is partitioned into $k$ subsets and a DP teacher
discriminator is trained on each subset. Further, instead of using one
generator to generate samples, $k$ conditional generators are used for each
subset of the data.

###  2.4 Data sets

##### Adult data set

In the Adult data set (32561 instances), the features were categorized as
protected variable (C): gender (male, female); and response variable (Y):
income (binary); decision variables (X): the remaining variables in the data
set. We map into categorical variables all continuous variables.

##### Prison Recidivism data set

From the COMPAS data set (7214 instances), we select severity of charge,
number of prior crimes, and age category to be the decision variables (X). The
outcome variable (Y) is a binary indicator of whether the individual
recidivated (re-offended), and race is set to be the protected variable (C).
We utilize a reduced set of features as proposed in [18].

##### Fair Prison Recidivism data set

We construct a ”fair” data set based on the COMPAS recidivism data set by
employing a data preprocessing technique for learning non-discriminating
classifiers from [37], which involves changing the class labels in order to
remove discrimination from the data set. This approach selects examples close
to the decision boundary to be either ’promoted’, i.e label flipped to the
desirable class, or ‘demoted’, i.e label flipped to the undesirable class (ex:
the ’recidivate’ label in the COMPAS data set is the undesirable class). By
flipping an equal number of positive and negative class examples, the class
skew in the data set is maintained.

##  3 Experimental Evaluation

One potential outcome of synthetic data sharing is the utilization of
synthetic data for training and evaluating an ML model. The trained model
could be deployed without assessing its performance on real data, due to lack
of data access. However, it is important to acknowledge that these trained
models are ultimately applied to real data. This scenario is illustrated in
Figure 1. In our experiments, we address the concern that there may be
substantial disparities in performance between the evaluation phase (employing
synthetic data) and the deployment phase (utilizing real data). We compare the
performance of logistic regression models trained with differentially private
synthesizers, focusing on two performance dimensions: utility and fairness.
The follow the approach of [20] and use logistic regression for downstream
classification evaluation to avoid another layer of stochasticity.

To assess the utility performance, we employ the AUC-ROC metric, which
quantifies trade-off between the recall and false positive rate. We examine
fairness performance through three different perspectives. Previous research
[17] has indicated that differentially private machine learning models tend to
perform worse on minority groups. To this point we evaluate the decay in
accuracy for the different subgroups in the protected attribute. We also
measure the difference in equality of odds (DEO) and the difference in
statistical parity (DSP). These metrics allow us to assess any disparities or
bias in the model’s predictions across different groups. Furthermore, we also
investigate the extent to which one can accurately assess a model utilizing
synthetic data sets. Again, we evaluate two performance dimensions: utility
and fairness.

Our experiments include two types of synthesizers: marginal-based and GAN-
based synthesizers. We generate synthetic data using three differentially
private marginal-based synthesizers: MST [10], MWEM-PGM [11] and PrivBayes
[38]; and four GAN-based synthesizers: DP-GAN, DP-CTGAN, PATE-GAN and PATE-
CTGAN [14]. For each synthetic data generation technique, we generate data
sets utilizing different four privacy-loss budgets
$\epsilon=\\{0.5,1.0,5.0,10.0\\}$.

We randomly divide the real data set into an 80/20 split, separating the data
into generator and test data sets. We run 10 rounds of synthetic DP data
generation on the 80% split (generator data), where we generate synthetic
train and synthetic test data sets. We utilize the SmartNoise
Library111https://smartnoise.org implementation of the synthesizers, and
approximate-DP approaches use the library’s default value of $\delta$. For
experiments using PrivBayes Synthesizers, we use the DiffPrivLib
implementation 222https://github.com/IBM/differential-privacy-library.

We train Logistic Regression models using the generated DP synthetic data
sets. In experiments where we test the trained models on real data, model
performance is evaluated on the real test data (the 20% test split from the
real data). In experiments where we test the trained models on synthetic data,
models are evaluated using the synthetic test data sets.

We report, for each technique and each value of privacy loss parameter, the
mean across 10 rounds. Our experiments use three data sets: the UCI Adult data
set [39] and ProPublica’s COMPAS recidivism data [40], and a fair COMPAS data
set as defined in Section 2.4. The fair COMPAS data set provides a way to
evaluate synthetic data generation performance in fair and biased versions of
the same data set.

###  3.1 Utility analysis of synthetic data in machine learning pipelines

We evaluate the quality of models trained with synthetic data sets by
measuring AUC and accuracy of the protected class. We consider privacy-loss
budgets of 0.5, 1.0, 5.0 and 10.0 . We compare the AUC obtained in our
experiments with the AUC measured by training models with the real (non-
synthetic) Adult, COMPAS, and fair COMPAS data sets.

Figure 2 (a) shows AUC for different privacy losses and different
synthesizers. The plots show the variation of AUC as a function of $\epsilon$
for marginal-based and GAN-based synhtesizers. The top row refers to marginal-
based synthesizers. Overall, the performance of the models trained on
marginal-based synthetic data is very close to the baseline model, trained on
real data. For all three synthesizers, we see an increase in AUC as we
increase $\epsilon$. For all data sets, Adult, COMPAS and fair COMPAS, the
perfomance of MST and MWEM-PGM are similar across all values of $\epsilon$.
PrivBayes has a slightly lower performance. For $\epsilon>5.0$, all three
synthesizer presented very similar performance. For COMPAS data set (which has
a small dimension) the performance of synthetic data sets as training data is
very close to the performance of the real data. The bottom row of figure 2 (a)
presents the perfomance of GAN-based synthetic data. The overall performance
of this type of synthesizer is worse and the performance of the marginal-based
synthesizer. As noted by [25], models trained on GAN-based synthetic data
perform worse than models trained on marginal-based synthetic data. With AUC
$\approx 0.5$, we can say that they do not do much better than random
guessing. Additionally, we see a much greater variance in results for a same
privacy-loss budget, which is observed by the large error bars. Finally, as
the privacy-loss budget increases, the utility does not necessarily increase.

Although several works have assessed the performance of machine learning
models trained with synthetic data sets [25, 20, 21], this is the first study
to analyze if synthetic data sets can be used for model assessment, and how
close to reality such assessment is. In Figure 2 (b) we present the plots of
variation of AUC for different values of epsilon. The plots in the first line
refer to performance of models trained on marginal-based synthesizers, the the
plots in the second line refer to GAN-based synthesizers. By comparing the
evaluation of models trained with marginal-based data in Figure 2 (a) -
assessment with real data, and in Figure 2 (b) - assessment with synthetic
data, we see that the assessment is very similar in both cases when the
synthesizers are MST and MWEM PGM. When assessing with synthetic data, we
notice that PrivBayes present a large difference in assessment results when
assessing model trained on Adult and fair COMPAS synthetic data. GAN-based
synthetic data present inconsistent behavior when used for model assessment.
When comparing the assessments in Figure 2 (a) - assessment with real data,
and (b) - assessment with synthetic data, we notice that using DP-GAN sythetic
data for model assessment can over estimate model AUC. Overall, GAN-based
synthetic data will make assessments that are as good as random guessing.

(a) AUC variation of models trained on synthetic data and evaluated using real
data. (b) AUC variation of models trained on synthetic data and evaluated
using synthetic data.

Fig 2: Impact in utility caused by the use of differentially private synthetic
data in model training and testing. In (a) we show the decay in model utility
when utilizing marginal-based and GAN-based synthetic data sets for model
training. In (b) we show what is the measured model utility when the
instrument for measuring model performance is a synthetic data set.

##### Marginal-based synthetic data does better at training and assessing
utility of models.

We ranked the utility performance of all synthesizers taking based on two
criteria: ability to generate synthetic data for model training and ability to
generate synthetic data for model assessment. Table 1 shows the ranking of
synthesizers when generating training ans assessment data for the Adult data
and the COMPAS data. The table also shows model AUC metrics when measured with
real data - AUC(R), and model AUC when measured with synthetic data - AUC(S).
All table results accounts for synthetic data generated with privacy-loss
parameter $\epsilon=5.0$.

| Rank | Adult | COMPAS
---|---|---|---
Synthesizer | Adult | COMPAS | AUC (R) | AUC (S) | AUC (R) | AUC (S)
MWEM PGM | 1st | 1st | 0.850 | 0.820 | 0.684 | 0.671
MST | 2nd | 2nd | 0.836 | 0.804 | 0.662 | 0.643
PrivBayes | 3rd | 3rd | 0.846 | 0.544 | 0.668 | 0.629
DP-GAN | 4th | 6th | 0.667 | 0.880 | 0.503 | 0.568
PATE-CTGAN | 5th | 4th | 0.343 | 0.504 | 0.552 | 0.492
DP-CTGAN | 6th | 5th | 0.284 | 0.485 | 0.504 | 0.502
PATE-GAN | 7th | 7th | 0.210 | 0.597 | 0.362 | 0.587
Table 1: Synthesizer utility comparison. We compare and rank all synthesizers
by their ability to generate quality training data and evaluation data for
machine learning pipelines. The comparison presented accounts for synthetic
data generated with privacy-loss parameter $\epsilon=5.0$. In addition to
present a performance ranking for Adult and COMPAS data, we show a comparison
of model AUC measured with real data - AUC(R), and model AUC measured with
synthetic data - AUC(S).

MWEM PGM synthetic data outperforms all other synthetic data for both tasks:
utility as training data for machine learning models and utility as evaluation
data for machine learning models. The performance of synthetic data sets
generated with MWEM PGM and MST perform well and with a small performance
decay when compared to real data, both when using the synthetic data for model
training and model assessment. For model training, when comparing the AUC
achieved by model trained with the real data set (AUC = 0.892 ) to the metrics
achieved by models trained with MWEM PGM data (AUC = 0.850 ) and MST (AUC =
0.836), the decrease in performance is small. The synthetic data sets also
present a good performance as assessment data. The model assessment resulted
when using MST data (AUC = 0.804) and MWEM PGM data (AUC = 0.820) presents
consistent results with a small decay. Although PrivBayes data presents good
performance in model training (AUC = 0.846), there is a significant
discrepancy between assessment utilizing real data and assessment utilizing
synthetic data. We reach similar conclusions when analysing results for COMPAS
data. Using GAN-based data as training data resulted in models with utility
very close to random guess, as already observed in previous analysis, with DP-
GAN synthetic data performing slightly better than the rest of GAN-based data
sets.

###  3.2 Fairness analysis of synthetic data in machine learning pipelines

##### Impacts on subgroup accuracy

In the previous section, we showed that adding privacy by utilizing synthetic
data sets in machine learning pipelines results in a utility decrease. We now
proceed to perform a fairness analysis. In this experiment, presented in Table
2 we analyze model accuracy for different groups in the protected class to
understand whether the addition of privacy to the data pipeline harms model
utility more for the minority class than it does for the privileged class.
Results in Table 2 refer to the Adult data set.

| Accuracy of different subgroups - Adult data
---|---
Synthesizer | minority (R) | minority (S) | privileged (R) | privileged (S)
Real | 0.924 | – | 0.804 | –
MWEM PGM | 0.909 | 0.898 | 0.779 | 0.770
MST | 0.914 | 0.895 | 0.756 | 0.765
PrivBayes | 0.892 | 0.596 | 0.709 | 0.575
DP-GAN | 0.733 | 0.929 | 0.585 | 0.855
PATE-CTGAN | 0.892 | 0.938 | 0.695 | 0.942
DP-CTGAN | 0.889 | 0.999 | 0.693 | 0.999
PATE-GAN | 0.892 | 0.874 | 0.695 | 0.854
Table 2: Accuracy comparison for different groups. The comparison presented
accounts for synthetic data generated with privacy-loss parameter
$\epsilon=5.0$. We show a comparison of model accuracy for the different
groups measured with real data (R), and model accuracy measured with synthetic
data (S).

From a fairness perspective, the overall behavior of all synthesizers is to
have less accuracy decay for the protected class than it does for the
privileged class. As observed on the utility experiments, MWEM PGM and MST are
the best performing synthetic data sets for both pipeline tasks: training and
evaluation. Although MWEM PGM presents good results for minority and
privileged classes, where the model accuracy is very close to the baseline
model - captured by accuracy minority(R) and accuracy privileged(R) in Table
2. Additionally, evaluation with MWEM PGM synthetic data sets captured
accuracy metric for both classes - captured by accuracy minority(S) and
accuracy privileged(S) - that are very close to model evaluation done with
real data.

##### Impacts on statistical parity

A model presents statistical parity if the percentage of positive predictions
are the same for all subgroups. The goal of the experiments in this section is
to measure whether models trained with synthetic data preserve the
characteristics of models trained on real data.

Our experiments measure the difference in statistical parity (DSP) of models.
We measure DSP of models using real data - DSP(R), and using synthetic data -
DSP(S). We present a detailed comparison of DSP for all three data sets and
all synthesizers on Table 3. We notice from our experiments that several
models trained on synthetic data seem to be less biased than the model trained
on real data. MWEM PGM synthesizer presented the best utility overall, based
on the results present in the previous experiments. PATE-CTGAN, however, was
ranked in 5th place in utility.

Data | Synthesizer | DSP(R) | DSP(S) | DSP delta
---|---|---|---|---
Adult | MST | 0.083 | 0.072 | 0.011
| MWEM PGM | 0.168 | 0.159 | 0.009
| PrivBayes | 0.051 | 0.035 | 0.016
| DP-CTGAN | -0.001 | 0.000 | -0.001
| DP-GAN | 0.346 | 0.253 | -0.093
| PATE-CTGAN | 0.000 | 0.000 | 0.000
| PATE-GAN | 0.000 | 0.000 | 0.000
| Real | 0.189 |  |
COMPAS | MST | -0.182 | -0.101 | -0.082
| MWEM PGM | -0.218 | -0.190 | -0.028
| PrivBaeys | -0.211 | -0.166 | -0.046
| DP-CTGAN | -0.034 | 0.001 | -0.034
| DP-GAN | 0.072 | -0.089 | 0.161
| PATE-CTGAN | -0.008 | -0.009 | 0.001
| PATE-GAN | 0.000 | -0.001 | 0.001
| Real | -0.205 |  |
COMPAS | MST | -0.185 | -0.090 | -0.095
(fair) | MWEM PGM | -0.018 | 0.015 | -0.032
| PrivBayes | -0.065 | 0.037 | -0.027
| DP-CTGAN | -0.034 | -0.004 | -0.030
| DP-GAN | 0.066 | 0.096 | -0.030
| PATE-CTGAN | 0.000 | 0.000 | 0.000
| PATE-GAN | 0.000 | 0.000 | 0.000
| Real | -0.025 |  |
Table 3: Difference in statistical parity (DSP) of models trained with
synthetic data. We measure the DSP of models using real test data - DSP(R) and
synthetic test data DSP(S). DEO delta quantifies the difference between DSP(R)
and DSP(S). All synthetic data where generated using privacy-loss parameter
$\epsilon=5.0$.

To understand better what is behind this apparent fairness provided by PAET-
CTGAN, we investigate the percentage of positive labelled samples in the
training data, evaluation data and predictions. We present percentages for
minority and privileged classes for adult data in Table 4.

| Ratio of positive labels - Adult data
---|---
Generation | Generated data | Predictions(R) | Predictions(S)
algorithm | Female | Male | Female | Male | Female | Male
Real | 0.109 | 0.303 | 0.055 | 0.244 |  |
MWEM PGM | 0.120 | 0.307 | 0.042 | 0.209 | 0.043 | 0.202
MST | 0.123 | 0.297 | 0.032 | 0.115 | 0.031 | 0.102
PrivBayes | 0.265 | 0.342 | 0.004 | 0.055 | 0.056 | 0.091
PATE-GAN | 0.125 | 0.144 |  $\approx$ 0 |  $\approx$ 0 |  $\approx$ 0 |  $\approx$ 0
PATE-CTGAN | 0.056 | 0.058 |  $\approx$ 0 |  $\approx$ 0 |  $\approx$ 0 |  $\approx$ 0
DP-GAN | 0.061 | 0.307 | 0.199 | 0.545 | 0.016 | 0.269
DP-CTGAN |  $\approx$ 0 | 0.002 | 0.227 | 0.130 |  $\approx$ 0 |  $\approx$ 0
Table 4: Ratio of samples with positive labels for each subgroup in the
protect class in the Adult data. We compare percentages present in the true
labels of the real data and the predicted labels. Analogously, we measure the
percentage of samples with positive present in the training, testing and
predicted labels for data sets generated from three distinct synthesizer
techniques: MWEM PGM, PATE-CTGAN and DP-GAN. Predictions(data1/data2)
represents prediction labels of an experiment where model was trained with
data1, and predictions were performed on data2.

We observe in Table 4 that synthetic data generated with PATE-CTGAN presents a
very similar percentages of samples with positive labels, of $\approx 5\%$ for
each group that belongs to the protected attribute. At a first sight, this
seems like a data set with promising fairness capabilities. However, when
training models with such data, there are no positive predictions resulting
from the model scoring. The model trained with PATE-CTGAN data acts like a
majority baseline classifier for all groups. The data sets generated with DP-
CTGAN presented an accentuated disparity in positive labels percentages
between minority and privileged classes. In the real data 30% of privileged
class contains positive labels, while only 10% of minority class contains
positive labels. Although DP-GAN synthesizer generates data where 31% of
privileged class with positive labels (a value similar to the one presented in
the real data - 30%), there is a significant decrease in the percentage of
positive class in the minority class, which is $\approx 6\%$. This imbalance
is even further accentuated by the models trained with DP-GAN synthetic data.
Model predictions resulted in over half of samples from the privileged class
being classified with positive labels (versus 20% of minority class).

MWEM PGM once again was the best overall performing model, as it preserves
similar percentages of positive labels for all groups, 11% and 30% (compared
to 11% and 30% in real data). Models trained with MWEM PGM also presented
similar metric to models trained with real data, and even presenting slightly
improvement in fairness.

The DSP delta presented in Table 3 quantifies the difference in DSP observed
during model evalution with real data and model evaluation with synthetic
data. For Adult data set, a positive DSP delta means that evaluation with
synthetic data observed fairer results than evaluation with real data. For
COMPAS and fair COMPAS data, a negative DSP delta means that evaluation with
synthetic data observed fairer results than evaluation with real data.

Across all data sets, models trained with MWEM PGM presented DSP metrics very
similar to models trained with real data, this is captured by the DSP(R)
metric.

##### Impacts on equal opportunity

Equal Opportunity requires equal True Positive Rate (TPR) across subgroups.
Difference in equal opportunity (DEO) measures the difference of privileged
group TPR and minority group TPR.

We perform a thorough analysis to understand two points.First, what is the DEO
of models trained with synthetic data sets, and how does it compare with
models trained with real data? Second, we investigate whether synthetic data
preserves similar true positive rates across all subgroups.

We present in Table 5 experiment results comparing DEO of models trained with
differentially private synthetic data sets ($\epsilon=5.0$). These experiment
are similar to the statistical parity experiments, we use real data - DEO(R) -
to measure DEO of models trained on synthetic data, as well as synthetic data
- DEO(S).

Data | Synthesizer | DEO (R) | DEO (S) | DEO Delta
---|---|---|---|---
Adult | MST | 0.038 | 0.076 | -0.037
| MWEM PGM | 0.206 | 0.200 | 0.006
| PrivBayes | 0.094 | 0.030 | 0.063
| DP-CTGAN | -0.002 |  $\approx$0.00 | -0.002
| DP-GAN | 0.527 | 0.641 | -0.116
| PATE-CTGAN | 0.000 | 0.000 | 0.000
| PATE-GAN | 0.000 | 0.000 | 0.000
| Real | 0.173 |  |
COMPAS | MST | -0.150 | -0.089 | -0.061
| MWEM PGM | -0.215 | -0.224 | 0.009
| PrivBayes | -0.177 | -0.158 | -0.020
| DP-CTGAN | -0.031 | -0.000 | -0.031
| DP-GAN | -0.075 | 0.020 | 0.055
| PATE-CTGAN | -0.011 | -0.009 | -0.002
| PATE-GAN | 0.000 | -0.001 | 0.001
| Real | -0.204 |  |
COMPAS | MST | -0.181 | -0.073 | -0.107
(fair) | MWEM PGM | -0.019 | 0.037 | -0.056
| PrivBayes | -0.057 | 0.003 | -0.054
| DP-CTGAN | -0.030 | -0.005 | -0.026
| DP-GAN | 0.097 | 0.087 | 0.010
| PATE-CTGAN | 0.000 | 0.000 | 0.000
| PATE-GAN | 0.000 | -0.001 | -0.000
| Real | -0.027 |  |
Table 5: Difference in equal opportunity (DEO) of models trained with
synthetic data. We measure the DEO of models using real test data - DEO(R) and
synthetic test data DEO(S). DEO delta quantifies the difference between DEO(R)
and DEO(S). All synthetic data where generated using privacy-loss parameter
$\epsilon=5.0$.

The model trained with MWEM PGM synthetic data was the only one that presented
a similar DEO to the baseline model, outperforming all other models trained
with synthetic data. Note that our comparison, as in the DSP case, focus on
understanding which synthetic data sets can train model that behave as close
as possible to models trained with real data. Models trained with MST, which
presented promising utility metrics and subgroup accuracy, did not capture as
well the difference in equality on odds in experiments with the Adult data.
For experiments with COMPAS and fair COMPAS data, MST performs better, but
still worse than MWEM PGM, as we can see on Table 5.

As we investigate the details of variation in TPR it becomes clear MWEM PGM is
the the best technique for training models that preserve fairness
characteristics of models trained with real data. Experiments with Adult data
(Figure 3) show that the difference between the privileged group TPR and the
minority group TPR of models trained with MWEM PGM data is very similar to the
difference between subgroups TPR of models trained with real data. Experiments
with COMPAS data (Figure 4) are even more compelling. Not only the difference
between the subgroup TPR of the model trained with MWEM PGM data is close to
that of the model trained with real data, but the true positive rates of the
subgroups are also very similar to the TPR of the model trained with real
data. Figures 3 and 4 show that models trained with marginal-based synthetic
data outperforms models trained with GAN-based synthetic data for our tested
data sets.

Fig 3: True positive rate (TPR) variation of different subgroups of the
protected attribute of the Adult data. The top two rows shows TPR variation
for different values of privacy-loss parameter $\epsilon$, of models trained
with synthetic data and evaluated with real data. The bottom two rows shows
TPR variation for different values of privacy-loss parameter $\epsilon$, of
models trained with synthetic data and evaluated with synthetic data.

We make a similar analysis when evaluating how good synthetic data sets are
for assessing TPRs. Figures 3 and 4 also present plots of TPR when synthetic
data is used during model assessment. Models trained with MWEM PGM data
present very similar assessment when using both real and synthetic data as
test data. Models trained on MST and PrivBayes present greater discrepancies.
Models trained on GAN-based data present even greater discrepancies between
assessments made with real and synthetic data as test data.

Fig 4: TPR variation: COMPAS and $\epsilon$, of models trained with synthetic
data and evaluated with real data.We also present TPR variation for different
values of $\epsilon$, of models trained with synthetic data and evaluated with
synthetic data.

##### Marginal-based synthetic data preserves and better assess model fairness

We evaluated the performance of the synthetic data sets based on two key model
fairness tasks: the ability to mirror the behavior of actual data in
downstream model fairness, and the ability to produce synthetic data for
assessing model fairness. Our analysis includes a rigorous assessment of model
fairness, which includes measuring subgroup accuracy, the difference in
statistical parity(DSP) and the difference in equal opportunity (DEO). Beyond
measuring the classical fairness metrics, we also assess the Positive
Predictive Value (PPV) and True Positive Rate (TPR) for each subgroup within
the protected class. The significance of evaluating PPV and TPR lies in
understanding if the model upholds fairness because it accurately represents
PPV and TPR for all subgroups, or if it does so merely by acting as a random
classifier.

Table 6 shows the best synthesizers in end-to-end machine learning pipelines
when evaluating for fairness metrics. All table results accounts for synthetic
data generated with privacy-loss parameter $\epsilon=5.0$.

MWEM PGM synthetic data, once more, outperforms all other synthetic data in
the three fairness metrics. This advanatge is observed when MWEM PGM synthetic
data is used as a training data set as well as when used as a testing data
set.

As we investigate subgroup PPV and TPR metrics to get insights into model
fairness performances. We note that MWEM PGM synthetic data presents a ratio
of positive labels comparable to that obtained with real data (Table 4), for
all subgroups. When evaluating the ratio of positive labels in prediction for
all subgroups (female and male) in Table 4, we see that MWEM PGM also results
is metrics that are the closest to real data.

The evaluation of true positive rate provides more insights into the bias
introduced by synthetic data set in end-to-end machine learning pipelines.
Figures 3 and 4 shows the variation of TPR for different values of $\epsilon$,
in experiments with Adult, COMPAS ans fair COMPAS, respectively. For COMPAS
data set, MWEM PGM provides performance comparable the real data set in an
end-to-end analysis. For Adult data, $\epsilon>5$ provides comparable metrics.
Other algorithms, such as PrivBayes, that presented utility results (AUC
metric) comparable to real data, showed low performance in terms of TPR.
Finally, marginal-based synthesizers presented similar performance from the
point of view of utility and fairness for both biased and fair versions of the
COMPAS data set.

Metric | Best Synthesizer | Runner up
---|---|---
Subgroup accuracy | MWEM PGM | MST
Difference in statistical parity | MWEM PGM | MST
Difference in equality of odds | MWEM PGM | MST
Table 6: Difference in equal opportunity (DEO) of models trained with
synthetic data. We measure the DEO of models using real test data - DEO(R) and
synthetic test data DEO(S). DEO delta quantifies the difference between DEO(R)
and DEO(S). All synthetic data where generated using privacy-loss parameter
$\epsilon=5.0$.

##  4 Limitations and Future Works

Although the data sets utilized in our analysis are commonly employed in
fairness literature, extending the validity of our findings to larger-scale
data sets would provide a more comprehensive understanding of the
generalizability and robustness of marginal-based synthetic data approaches.
Future research should focus on exploring the performance of these frameworks
in real-world scenarios with diverse and extensive data sets. This would
contribute to the broader applicability and reliability of synthetic data
methods in various domains and facilitate a more nuanced understanding of
their limitations and capabilities. Finally, extending our analysis to non-
tabular data would be an interestign sequel to this work.

##  5 Conclusion

Our research comprehensively evaluates the impact of synthetic data sets for
training and testing in machine learning pipelines in the case of tabular data
sets. Specifically, we compare the performance of marginal-based and GAN-based
synthesizers within a machine-learning pipeline and analyze various utility
and fairness metrics for tabular data sets.

Our main findings are as follows: Marginal-based synthetic data demonstrated
comparable utility to real data in end-to-end machine-learning pipelines. MWEM
PGM (AUC = 0.684) provides utility very close to models trained on real data
(AUC = 0.684). Furthermore, we show that model evaluation using synthetic data
also provides similar results to evaluation using real data, for tabular data.
The metrics obtained when utilizing marginal-based synthetic data (AUC=0.671)
are comparable to real data (AUC = 0.684). Synthetic data sets trained with
MWEM PGM do not increase model bias and can provide a realistic fairness
evaluation. Our study reveals that MWEM PGM synthetic data can train models
that achieve similar utility and fairness characteristics as models trained
with real data. Additionally, when used to evaluate the utility and fairness
of machine learning models, the synthetic data generated by the MWEM PGM
algorithm exhibits behavior very similar to real data.

These findings highlight synthetic data’s potential reliability and viability
as a substitute for real data sets in end-to-end machine learning pipelines
for tabular data. Furthermore, our research sheds light on the implications of
model fairness when utilizing differentially private synthetic data for model
training.

One crucial observation is that synthetic data that does well in model
training might perform differently when used as evaluation data. This was the
case with Privbayes and some of the GAN-based synthetic data. This observation
is important as synthetic data techniques gain acceptance as the standard data
publishing approach in domains such as healthcare, humanitarian action,
education, and population studies.

## References

  * 1. Dwork C, McSherry F, Nissim K, Smith A.  Calibrating noise to sensitivity in private data analysis.  In: Theory of cryptography conference. Springer; 2006. p. 265–284.
  * 2. Pereira M, Kim A, Allen J, White K, Ferres JL, Dodhia R.  US Broadband Coverage Data Set: A Differentially Private Data Release.  arXiv preprint arXiv:210314035. 2021;.
  * 3. Aktay A, Bavadekar S, Cossoul G, Davis J, Desfontaines D, Fabrikant A, et al.  Google COVID-19 community mobility reports: Anonymization process description (version 1.0).  arXiv preprint arXiv:200404145. 2020;.
  * 4. Tang J, Korolova A, Bai X, Wang X, Wang X.  Privacy loss in apple’s implementation of differential privacy on macos 10.12.  arXiv preprint arXiv:170902753. 2017;.
  * 5. Abowd JM.  The US Census Bureau adopts differential privacy.  In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; 2018. p. 2867–2867.
  * 6. Qian Z, Callender T, Cebere B, Janes SM, Navani N, van der Schaar M.  Synthetic data for privacy-preserving clinical risk prediction.  medRxiv. 2023; p. 2023–05.
  * 7. Movahedi P, Nieminen V, Perez IM, Pahikkala T, Airola A.  Evaluating Classifiers Trained on Differentially Private Synthetic Health Data.  In: 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS). IEEE; 2023. p. 748–753.
  * 8. Cheng V, Suriyakumar VM, Dullerud N, Joshi S, Ghassemi M.  Can You Fake It Until You Make It? Impacts of Differentially Private Synthetic Data on Downstream Classification Fairness.  In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency; 2021. p. 149–160.
  * 9. Ganev G.  DP-SGD vs PATE: Which Has Less Disparate Impact on GANs?  arXiv preprint arXiv:211113617. 2021;.
  * 10. McKenna R, Miklau G, Sheldon D.  Winning the NIST Contest: A scalable and general approach to differentially private synthetic data.  arXiv preprint arXiv:210804978. 2021;.
  * 11. McKenna R, Sheldon D, Miklau G.  Graphical-model based estimation and inference for differential privacy.  In: International Conference on Machine Learning. PMLR; 2019. p. 4435–4444.
  * 12. Zhang J, Cormode G, Procopiuc CM, Srivastava D, Xiao X.  Privbayes: Private data release via bayesian networks.  ACM Transactions on Database Systems (TODS). 2017;42(4):1–41.
  * 13. Xie L, Lin K, Wang S, Wang F, Zhou J.  Differentially private generative adversarial network.  arXiv preprint arXiv:180206739. 2018;.
  * 14. Rosenblatt L, Liu X, Pouyanfar S, de Leon E, Desai A, Allen J.  Differentially Private Synthetic Data: Applied Evaluations and Enhancements.  arXiv preprint arXiv:201105537. 2020;.
  * 15. Jordon J, Yoon J, Van Der Schaar M.  PATE-GAN: Generating synthetic data with differential privacy guarantees.  International Conference on Learning Representations. 2018;.
  * 16. Wiens J, Saria S, Sendak M, Ghassemi M, Liu VX, Doshi-Velez F, et al.  Do no harm: a roadmap for responsible machine learning for health care.  Nature medicine. 2019;25(9):1337–1340.
  * 17. Bagdasaryan E, Poursaeed O, Shmatikov V.  Differential privacy has disparate impact on model accuracy.  Advances in Neural Information Processing Systems. 2019;32:15479–15488.
  * 18. Calmon FP, Wei D, Vinzamuri B, Ramamurthy KN, Varshney KR.  Optimized pre-processing for discrimination prevention.  In: Proceedings of the 31st International Conference on Neural Information Processing Systems; 2017. p. 3995–4004.
  * 19. Rajotte JF, Mukherjee S, Robinson C, Ortiz A, West C, Ferres JL, et al.  Reducing bias and increasing utility by federated generative modeling of medical images using a centralized adversary.  arXiv preprint arXiv:210107235. 2021;.
  * 20. Ganev G, Oprisanu B, De Cristofaro E.  Robin hood and matthew effects: Differential privacy has disparate impact on synthetic data.  In: International Conference on Machine Learning. PMLR; 2022. p. 6944–6959.
  * 21. Giles O, Hosseini K, Mingas G, Strickson O, Bowler L, Smith CR, et al.  Faking feature importance: A cautionary tale on the use of differentially-private synthetic data.  arXiv preprint arXiv:220301363. 2022;.
  * 22. Pereira M, Kshirsagar M, Mukherjee S, Dodhia R, Ferres JL.  An analysis of the deployment of models trained on private tabular synthetic data: Unexpected surprises.  arXiv preprint arXiv:210610241. 2021;.
  * 23. Jordon J, Szpruch L, Houssiau F, Bottarelli M, Cherubin G, Maple C, et al.  Synthetic Data–what, why and how?  arXiv preprint arXiv:220503257. 2022;.
  * 24. Research M. The global victim-perpetrator synthetic dataset; 2022.  Available from: https://www.ctdatacollaborative.org/global-victim-perpetrator-synthetic-dataset.
  * 25. Tao Y, McKenna R, Hay M, Machanavajjhala A, Miklau G.  Benchmarking differentially private synthetic data generation algorithms.  arXiv preprint arXiv:211209238. 2021;.
  * 26. Bullwinkel B, Grabarz K, Ke L, Gong S, Tanner C, Allen J.  Evaluating the Fairness Impact of Differentially Private Synthetic Data.  arXiv preprint arXiv:220504321. 2022;.
  * 27. Heidari H, Loi M, Gummadi KP, Krause A.  A moral framework for understanding fair ML through economic models of equality of opportunity.  In: Proceedings of the Conference on Fairness, Accountability, and Transparency; 2019. p. 181–190.
  * 28. Barocas S, Hardt M, Narayanan A.  Fairness in machine learning.  Nips tutorial. 2017;1:2.
  * 29. Dwork C, Roth A, et al.  The algorithmic foundations of differential privacy.  Foundations and Trends in Theoretical Computer Science. 2014;9(3-4):211–407.
  * 30. McSherry FD.  Privacy integrated queries: an extensible platform for privacy-preserving data analysis.  In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of data; 2009. p. 19–30.
  * 31. Dwork C, McSherry F, Nissim K, Smith A.  Calibrating noise to sensitivity in private data analysis.  In: Theory of cryptography conference. Springer; 2006. p. 265–284.
  * 32. Xu W, Zhao J, Iannacci F, Wang B.  FFPDG: Fast, Fair and Private Data Generation.  online preprint. 2021;.
  * 33. Perrone V, Donini M, Zafar MB, Schmucker R, Kenthapadi K, Archambeau C.  Fair bayesian optimization.  arXiv preprint arXiv:200605109. 2020;.
  * 34. Xu L, Skoularidou M, Cuesta-Infante A, Veeramachaneni K.  Modeling tabular data using conditional gan.  arXiv preprint arXiv:190700503. 2019;.
  * 35. Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, et al.  Deep learning with differential privacy.  In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security; 2016. p. 308–318.
  * 36. Papernot N, Abadi M, Erlingsson U, Goodfellow I, Talwar K.  Semi-supervised knowledge transfer for deep learning from private training data.  arXiv preprint arXiv:161005755. 2016;.
  * 37. Kamiran F, Calders T.  Data preprocessing techniques for classification without discrimination.  Knowledge and Information Systems. 2012;33(1):1–33.
  * 38. Zhang J, Cormode G, Procopiuc CM, Srivastava D, Xiao X.  Privbayes: Private data release via bayesian networks.  ACM Transactions on Database Systems (TODS). 2017;42(4):1–41.
  * 39. Dua D, Graff C. UCI Machine Learning Repository; 2017.  Available from: http://archive.ics.uci.edu/ml.
  * 40. Barenstein M.  ProPublica’s COMPAS Data Revisited.  arXiv preprint arXiv:190604711. 2019;.