# Differentially Private Normalizing Flows for Density Estimation, Data
Synthesis, and Variational Inference with Application to Electronic Health
Records

Bingyue Su111co-first authors Yu Wang∗ Daniele E. Schiavazzi  Fang Liu

###### Abstract

Electronic health records (EHR) often contain sensitive medical information
about individual patients, posing significant limitations to sharing or
releasing EHR data for downstream learning and inferential tasks. We use
normalizing flows (NF), a family of deep generative models, to estimate the
probability density of a dataset with differential privacy (DP) guarantees,
from which privacy-preserving synthetic data are generated. We apply the
technique to an EHR dataset containing patients with pulmonary hypertension.
We assess the learning and inferential utility of the synthetic data by
comparing the accuracy in the prediction of the hypertension status and
variational posterior distribution of the parameters of a physics-based model.
In addition, we use a simulated dataset from a nonlinear model to compare the
results from variational inference (VI) based on privacy-preserving synthetic
data, and privacy-preserving VI obtained from directly privatizing NFs for VI
with DP guarantees given the original non-private dataset. The results suggest
that synthetic data generated through differentially private density
estimation with NF can yield good utility at a reasonable privacy cost. We
also show that VI obtained from differentially private NF based on the free
energy bound loss may produce variational approximations with significantly
altered correlation structure, and loss formulations based on alternative
dissimilarity metrics between two distributions might provide improved
results.

keyword: differential privacy, privacy-preserving, synthetic data, normalizing
flow, variational inference, free energy bound, KL divergence, density
estimation, electronic health records

##  1 Introduction

###  1.1 Background

In the last decade, data science has gained tremendous popularity as an
interdisciplinary field, thanks to the exponential growth in data volume, fast
development of powerful and efficient data analysis techniques, and
availability of computational facilities and resources. When sharing data
among researchers or releasing information to the public, there are always
privacy concerns and risks for re-identification or disclosure of sensitive
information. Even with key identifiers removed, an adversary may still be able
to identify an individual or learn his or her sensitive information,
leveraging publicly available data and sophisticated attacks. Some notable
privacy attacks includes re-identification based on health records (Sweeney,
1997, 2013), Netflix prize (Narayanan and Shmatikov, 2008), membership attacks
(Homer et al., 2008; Backes et al., 2016; Shokri et al., 2017), among others.

In this work, we use electronic health records (EHR) as an example to present
approaches for privacy-preserving data sharing and analysis. EHR data often
contain sensitive medical information about individual patients. For example,
cardiovascular EHRs, the type of EHR data examined in this work, contain
measurements of vital signs such as heart rate, arterial and venous, systemic
and pulmonary pressures, cardiac output, atrial and ventricular pressures and
volumes, ejection fraction, etc., which can be used to diagnose hypertension.
Due to privacy concerns, accessing large single-center EHR datasets for
research purposes is often problematic (see recent efforts in making large
clinical datasets publicly available, for example, the MIMIC-IV dataset
discussed in Johnson et al., 2020), not to mention merging multiple EHR data
sources to create large multi-center datasets enabling complex predictive
models to be trained (see, in this context, the eICU Collaborative Research
Database discussed in Pollard et al., 2018).

To mitigate privacy concerns when releasing information and sharing data,
various privacy concepts have been proposed over the past two decades, such as
$k$-anonymity (Samarati and Sweeney, 1998), $t$-closeness (Li et al., 2007),
$l$-diversity (Aggarwal and Yu, 2008), and differential privacy (DP) (Dwork et
al., 2006c, b, a), among others. We focus on DP, a popular privacy concept and
framework in contemporary privacy research. Besides providing mathematical
guarantees on privacy and being robust to various privacy attacks (Dwork et
al., 2017), DP has attractive properties, such as privacy loss composition and
immunity to post-processing, which have significantly contributed to its
popularity for both research and practical applications. Privacy-preserving
techniques and algorithms are available for a wide spectrum of statistical and
machine-learning methods with DP guarantees, ranging from releasing counts and
histograms to predictions from regression models or deep neural networks.

In this work, we examine differentially private normalizing flow (NF) to
generate synthetic data via density estimation and to obtain variational
inference (VI) with DP guarantees, respectively, under a pre-defined privacy
budget. NFs are generative models (Rezende and Mohamed, 2015; Dinh et al.,
2016; Kingma et al., 2016; Papamakarios et al., 2017) with major applications
in density estimation and VI.

###  1.2 Our Work and Contributions

Our work comprises two parts. The first part focuses on synthetic data
generation from differentially private density estimation and the utility of
the synthetic data for downstream leaning tasks. To our knowledge, there is
only one other paper (Waites and Cummings, 2021) on incorporating DP in NF for
density estimation. In addition, no prior work has explored the use of
synthetic data generated through DP-NF for downstream learning such as
prediction or inference. Our work intends to fill this gap.

Specifically, we apply DP-NF for density estimation given an EHR dataset that
contains hemodynamic measurements from a cohort of patients including
individuals suffering from elevated pulmonary pressures (group II pulmonary
hypertension, see Simonneau et al., 2013) and heart failure with preserved
ejection fraction (Harrod et al., 2021). The EHR dataset has a small sample
size and a relatively large number of attributes with continuous or discrete
numerical measurements and contains missing values. This provides an
opportunity to examine the feasibility of DP-NF in generating useful privacy-
preserving synthetic data for small-sized datasets, under data type
heterogeneity and a greater degree of realism compared to simulated data or
well-studied benchmark datasets. We run regression and classification on
synthetic data generated by DP-NF and compare their prediction accuracy with
that obtained from the original data. We also perform VI of a computationally
expensive physics-based model given the synthetic data. To reduce the
computation cost, we leverage an offline-trained and fast neural network
surrogate model for the physics-based model.

The second part of this work investigates privacy-preserving VI by directly
privatizing NF for VI with DP guarantees (as opposed to VI from private
synthetic data from differentially private density estimated via NF). We
examine the performance of the procedure in inferring the parameters of a
nonlinear regression model in simulated data. To the best of our knowledge,
this paper is the first to examine the feasibility of differentially
privatizing NF for VI using free energy bound loss and stochastic gradient-
based optimization. We show that this loss formulation may yield
differentially private variational distributions that deviate significantly
from the target density, particularly on the correlations if the the target
distribution is multivariate, when the marginal variances in the
differentially private variational distributions are inflated due to DP
guarantees compared to the target. We also examined several other
dissimilarity metrics as the loss function for NF-VI, similar observations as
in the case of free energy bound were observed in some of them, but not all of
them. This phenomenon suggests the importance of carefully selecting the loss
function when performing VI based on DP-NF.

In both parts, we apply Gaussian DP to achieve privacy guarantees through
stochastic gradient descent optimization, in contrast to the moment account
method (Abadi et al., 2016) for tracking privacy loss used in existing work
(Waites and Cummings, 2021). Gaussian DP yields a tighter bound for the
privacy loss composition and is shown to yield higher prediction accuracy deep
learning (see Bu et al., 2020), compared to the moment account technique that
is based on Rényi DP.

###  1.3 Related Work

A straightforward approach for differentially private density estimation from
which synthetic data can be generated is through histogram sanitization with
DP, such as the perturbed histogram or the smoothed histogram approaches
(Wasserman and Zhou, 2010). Histograms as density estimators suffer from the
curse of dimensionality, a problem that is only exacerbated after factoring in
DP; in addition, continuous attributes need to be discretized in histogram
estimators. Hall et al. (2013) and Alda and Rubinstein (2017) develop DP
mechanisms to privatize functional outputs with applications to kernel density
estimation. Adaptations of local DP to non-parametric density estimation have
also been proposed (Butucea et al., 2020; Kroll, 2021). Waites and Cummings
(2021) developed the privacy-preserving version of NF for density estimation
by privatizing SGD iteratively with the moment accountant method (Abadi et
al., 2016) to track privacy loss.

Work also exists on VI with privacy guarantees. Karwa et al. (2015) directly
model a DP mechanism used for sanitizing a likelihood function and obtain a
variational approximation to the resulting sanitized posterior distribution.
Jälkö et al. (2016) privatize SGDs using the moment accountant method with an
assumed family of variational distributions. Park et al. (2016) develop a
privacy-preserving variational Bayes framework by sanitizing the expected
sufficient statistics over the distribution of latent variables using the
Gaussian mechanism when Bayesian models belong to the conjugate-exponential
family. Sharma et al. (2019) examine differentially private VI in the setting
of federated learning. Jälkö et al. (2022) propose aligned gradients to factor
in the fact that different elements in the gradient vector associated with
multi-dimensional variational parameters are often of different magnitudes,
aiming to improve the utility of differentially private VI without incurring
additional privacy cost. To our knowledge, none of the work above realizes
differentially private VI through NF.

Data synthesis for statistical disclosure limitation was first proposed by
Rubin (1993) and is employed by government agencies for releasing survey data
and by the healthcare industry to share medical data, among other
applications. Synthesized surrogate data sets have the same structure as the
original dataset but comprise pseudo-individuals. Differentially private data
synthesis (DIPS) provides a solution to integrate formal privacy guarantees
into data synthesis. DIPS can be achieved through both model-free and model-
based approaches; interested readers can refer to Bowen and Liu (2020) for an
overview of some of the recent DIPS techniques. The introduction of
variational autoencoders (VAE Kingma and Welling, 2013), generative
adversarial networks (GANs Goodfellow et al., 2014), and NF (Rezende and
Mohamed, 2015; Papamakarios et al., 2017), opened new possibilities for data
synthesis through deep-neural-networks-based generative models with DP
guarantees. Xie et al. (2018) generate privacy-preserving synthetic data via
GANs (DP-GANs), incorporating DP guarantees using the moment accountant
method. Jordon et al. (2018) combine GANs and the Private Aggregation of
Teacher Ensembles (PATE) framework (Papernot et al., 2016, 2018) to generate
differentially private synthetic data and demonstrate its superiority over DP-
GANs in utility in empirical studies. Chen et al. (2018) propose a
differentially private autoencoder-based generative model and a differentially
private variational autoencoder-based generative model and demonstrate both
approaches can yield satisfactory utility but the former is more robust
against some privacy attacks than the latter. Beaulieu-Jones et al. (2019)
apply DP-GANs to release synthetic data to a real clinical dataset with
reasonable DP guarantees and demonstrate the utility of the synthetic data.
Pfitzner and Arnrich (2022) generate privacy-preserving synthetic data by
aggregating only the decoder component of VAEs in federated learning with DP
while leaving the encoders personal at local servers.

The rest of the paper is organized as follows. Section 2 introduces the
definitions of NF for density estimation and VI and the concept of DP. Section
3 presents the procedure for differentially private NF for density estimation
and generation of privacy-preserving synthesis from estimated density, with an
application to a real EHR dataset. Section 4 presents the procedure for
differentially private NF for VI, with an application to repeated measures
data simulated from a nonlinear model and an in-depth analysis of the utility
of variational distributions obtained from the procedure. The paper concludes
in Section 5 with a summary and some final remarks.

##  2 Preliminaries

###  2.1 Normalizing Flow

Normalizing Flow (NF) is defined as a map
$F:\mathbb{R}^{d}\times\Lambda\to\mathbb{R}^{d}$ parameterized by
$\lambda\in\Lambda$ transforming realizations from an easy-to-sample base
distribution such as $\bm{z}_{0}\sim\mathcal{N}(\bm{0},I_{d})$, to
realizations from a desired target density. Specifically, $F$ consists of a
composition of $K$ bijections
$F_{k}:\mathbb{R}^{d}\times\Lambda_{k}\to\mathbb{R}^{d}$, each parameterized
by $\lambda_{k}\in\Lambda_{k}$:
$F_{\lambda}(\bm{z}_{0})=F(\bm{z}_{0};\lambda)=\left[F_{K}(\ \cdot\
;\lambda_{K})\circ F_{K-1}(\ \cdot\ ;\lambda_{K-1})\circ\cdots\circ F_{1}(\
\cdot\ ;\lambda_{1})\right](\bm{z}_{0})$, where
$\bm{z}_{k}=F_{k}(\bm{z}_{k-1};\lambda_{k})$ for $k=1,\ldots,K$. Since
$F_{k}(\ \cdot\ ;\lambda_{k})$ is a bijection from $\bm{z}_{k-1}$ to
$\bm{z}_{k}$, $q_{k}(\bm{z}_{k})$, the distribution of $\bm{z}_{k}$, can be
obtained by the change of variable

$q_{k}(\bm{z}_{k})=q_{k-1}(\bm{z}_{k-1})\left|\det\frac{\partial
F_{k}^{-1}}{\partial\bm{z}_{k-1}}\right|=q_{k-1}(\bm{z}_{k-1})\left|\det\frac{\partial
F_{k}}{\partial\bm{z}_{k-1}}\right|^{-1}.$ (1)

Taking the logarithm and summing over $k$, Eqn. (1) becomes

$\textstyle\log q_{K}(\bm{z}_{K})=\log
q_{0}(\bm{z}_{0})-\sum_{k=1}^{K}\log\left|\det\frac{\partial
F_{k}}{\partial\bm{z}_{k-1}}\right|.$ (2)

Generally, the objective of training a NF is to determine an _optimal_ set of
parameters $\lambda$ so that $q_{K}$ can approximate either a target density
$p$ (VI) or the distribution of the observed data $\bm{x}$ (density
estimation).

For general bijections, the computational cost of computing the Jacobian
determinants in Eqn. (1) may increase significantly with the dimensionality
$d$ and the number of layers $K$. To efficiently compute these determinants,
various NF formulations have been proposed based on coupling layers such as
RealNVP (Dinh et al., 2016) and GLOW (Kingma and Dhariwal, 2018), and
autoregressive transformations such as MAF (Papamakarios et al., 2017) and IAF
(Kingma et al., 2016), both of which are associated with tractable triangular
Jacobian matrices and easy to compute.

In the applications of NF to density estimation and VI in Sections 3 and 4, we
use MAF (Papamakarios et al., 2017). The autoregressive property of MAF is
obtained by setting
$p(z_{i}|z_{1},\dots,z_{i-1})=\phi((z_{i}-\mu_{i})/e^{\alpha_{i}}$, where
$\phi$ is the density function of the standard normal distribution,
$\mu_{i}=f_{\mu_{i}}(z_{1},\dots,z_{i-1})$,
$\alpha_{i}=f_{\alpha_{i}}(z_{1},\dots,z_{i-1})$, and $f_{\mu_{i}}$ and
$f_{\alpha_{i}}$ are masked autoencoder neural networks (MADE, Germain et al.,
2015). Consider $\bm{z}$ and $\hat{\bm{z}}$ as the input and output of a MADE
network having $L$ hidden layers with $d_{l},\,l=1,\cdots,L$ neurons per
layer. The mappings between layers could be represented as

$\left\\{\begin{matrix}[l]\bm{y}_{1}\\!\\!\\!\\!&\\!=\\!h_{1}(b_{1}+(\bm{W}^{1}\odot\bm{M}^{1})\bm{z}),&\text{where
}\bm{M}^{1}\text{ is }d_{1}\times d\text{ and
}\bm{M}^{1}_{u,v}=\mathbbm{1}_{m^{1}(u)\geq v}\\\\[6.99997pt]
\bm{y}_{l}\\!\\!\\!\\!&\\!=\\!h_{l}(b_{l}+(\bm{W}^{l}\odot\bm{M}^{l})\bm{y}_{l-1}),&\text{where
}\bm{M}^{l}\text{ is }d_{l}\times d_{l-1}\text{ and
}\bm{M}^{l}_{u,v}\\!=\\!\mathbbm{1}_{m^{l}(u)\geq m^{l-1}(v)}\\\\[6.99997pt]
\hat{\bm{z}}\\!\\!\\!\\!&\\!\\!=\\!h_{L+1}(b_{L+1}+\\!(\bm{W}^{L+1}\odot\bm{M}^{L+1})\bm{y}_{L})&\text{where
}\bm{M}^{L+1}\text{ is }d\times d_{L}\text{ and
}\bm{M}^{L+1}_{u,v}=\mathbbm{1}_{u>m^{L}(v)},\\\ \end{matrix}\right.$

where $h_{l}$ is the activation function between layer $l-1$ and $l$ for
$l=1,\cdots,L+1$, $m^{l}(k)$ is a pre-set or random integers from $1$ to
$d-1$, $b_{l}$ and $\bm{W}^{l}$ are the bias and weight parameters for layer
$l=1,\dots,L+1$.

The matrix product $\prod_{l=L+1}^{1}\bm{M}^{l}$ is strictly lower triangular,
satisfying the autoregressive property (Germain et al., 2015). The masks
$\bm{M}^{1},\dots,\bm{M}^{L+1}$ enable the computation of all $\mu_{i}$ and
$\alpha_{i}$ in a single forward pass (Papamakarios et al., 2017).
Additionally, the Jacobian of each MAF layer is lower-triangular, with
determinant equal to $|\det\partial
f/\partial\bm{z}|^{-1}=\exp(\sum_{i=1}^{d}\alpha_{i})$. Before each MADE
layer, batch normalization (Ioffe and Szegedy, 2015) can be used to normalize
the outputs from the previous layer. Batch normalization performs elementwise
scaling and shifting operations, is easily invertible, has a tractable
Jacobian, and is found to help stabilize and accelerate training and increase
accuracy (Papamakarios et al., 2017).

When NF is employed for density estimation given observed data $\bm{x}$, the
following log-likelihood is maximized, assuming $n$ independent observations
$\bm{x}_{i},\,i=1,\dots,n$.

$\displaystyle\textstyle\ell(\lambda;\bm{x})$ $\displaystyle=\textstyle\log
q_{K}(\bm{x})=\sum_{i=1}^{n}\log q_{K}(\bm{x}_{i})$
$\displaystyle=\sum_{i=1}^{n}\log
q_{0}(\bm{z}_{i,0})-\sum_{i=1}^{n}\sum_{k=1}^{K}\left|\det\frac{\partial
F_{k}}{\partial\bm{z}_{i,k-1}}\right|,$ (3)

where $\bm{z}_{i,k-1}=F^{-1}_{k}\circ\cdots\circ F^{-1}_{K-1}\circ
F^{-1}_{K}(\bm{x}_{i})$. Once the NF parameters are obtained via the maximum
likelihood approach, samples from $q_{K}$ can be generated by first sampling
from the base distribution and then transforming the samples through a
sequence of bijections. NF has been shown to provide accurate approximations
of complex density functions of various data types (Papamakarios et al., 2017;
Dinh et al., 2016; Kingma and Dhariwal, 2018).

The other major application of NF is VI. Consider the likelihood
$l(\bm{z};\bm{x})$ of parameters or latent variables $\bm{z}$ given observed
data $\bm{x}_{i}$ for $i=1,\ldots,n$. Given prior $p(\bm{z})$, an NF-based
approximation $q_{K}(\bm{z})$ of the posterior distribution $p(\bm{z}|\bm{x})$
can be computed by maximizing the lower bound to the log marginal likelihood
$\log p(\bm{x})$ (the _evidence lower bound_ or ELBO), or, equivalently, by
minimizing the free energy bound (Rezende and Mohamed, 2015), expressed as

$\displaystyle\mathcal{F}(\bm{x})$
$\displaystyle=\mathbb{E}_{q_{K}(\bm{z})}\left[\log q_{K}(\bm{z})-\log
p(\bm{x},\bm{z})\right]=\mathbb{E}_{q_{K}(\bm{z}_{K})}\left[\log
q_{K}(\bm{z}_{K})-\log p(\bm{x},\bm{z}_{K})\right]$
$\displaystyle=\mathbb{E}_{q_{0}(\bm{z}_{0})}[\log
q_{0}(\bm{z}_{0})]-\mathbb{E}_{q_{0}(\bm{z}_{0})}[\log
p(\bm{x},\bm{z}_{K})]-\mathbb{E}_{q_{0}(\bm{z}_{0})}\left[\sum_{k=1}^{K}\log\left|\det\frac{\partial
F_{k}}{\partial\bm{z}_{k-1}}\right|\right]$
$\displaystyle=\sum_{i=1}^{n}\\!\left(\\!\frac{1}{n}\mathbb{E}_{q_{0}(\bm{z}_{0})}[\log
q_{0}(\bm{z}_{0})]-\mathbb{E}_{q_{0}(\bm{z}_{0})}[\log
p(\bm{x}_{i},\bm{z}_{K})]-\\!\frac{1}{n}\mathbb{E}_{q_{0}(\bm{z}_{0})}\\!\left[\sum_{k=1}^{K}\log\left|\det\frac{\partial
F_{k}}{\partial\bm{z}_{k-1}}\right|\right]\right)$
$\displaystyle=\textstyle\sum_{i=1}^{n}l_{i}$ (4)

The expectations in Eqn. (4) can be approximated by their corresponding Monte
Carlo (MC) estimates using samples $\bm{z}_{i,0},\,i=1,\dots,n$ from the basic
distribution $q_{0}$ during implementation.

###  2.2 Differential Privacy

Differential Privacy (DP) is a mathematical framework designed to produce
robust privacy guarantees. It aims at protecting privacy of every individual
in a dataset in the worst-case scenario.

######  Definition 1 ($(\epsilon,\delta)$-DP).

(Dwork et al., 2006c, b, a) A randomized mechanism $\mathcal{M}$ is
$\epsilon$-DP, if $\Pr[\mathcal{M}\left(D_{1}\right)\in S]\leq
e^{\epsilon}\Pr[\mathcal{M}\left(D_{2}\right)\in S]+\delta$ for any $S\in{\rm
range}(\mathcal{M})$ and neighbouring datasets $D_{1},D_{2}$ that differ by
one record, where $\epsilon>0$ and $\delta\in[0,1)$.

Informally, DP implies that the chance of identifying an individual in a
dataset using a randomized mechanism is limited if the released information is
about the same with or without that individual in the data. The quantities
$\epsilon$ and $\delta$ represent privacy budget or privacy loss parameters.
When $\delta=0$, $(\epsilon,\delta)$-DP in Definition 1 reduces to pure
$\epsilon$-DP. The smaller the $\epsilon$, the higher the privacy protection
there is for the individuals in the data.

When we apply DP mechanisms repeatedly to query a dataset, privacy loss will
accumulate during the process. It is therefore necessary to track and account
for the accumulated loss to so that the privacy loss would not be too large.
This has motivated the development of variants and extensions of the original
DP concepts, aiming at achieving tighter bounds for composite privacy loss,
such as the concentrated DP (Dwork and Rothblum, 2016; Bun and Steinke, 2016;
Bun et al., 2018), Rényi DP (Mironov, 2017), and Gaussian DP (Dong et al.,
2019). Abadi et al. (2016) propose a widely popular moment account technique
based on Rényi DP to track privacy loss in stochastic gradient descent-based
optimization. Dong et al. (2019) provide a tighter bound based on Gaussian DP
(GDP), which is the DP paradigm we will use for the reminder of this paper.

######  Definition 2 ($\mu$-GDP).

(Dong et al., 2019) Let $T$ be a trade-off function that maps $[0,1]$ to
$[0,1]$, defined as
$T(P_{0},P_{1})(\alpha)=\inf\\{\beta_{\phi}:\alpha_{\phi}\leq\alpha\\}$ for
two distributions $P_{1}$ from $P_{0}$ to be distinguished through hypothesis
testing ($H_{0}:P_{0}$ vs. $H_{1}:P_{1}$), where $\phi\in[0,1]$ is a rejection
rule with the type I error $\alpha_{\phi}=\mathbb{E}_{P_{0}}[\phi]$ and type
II error $\beta_{\phi}=1-\mathbb{E}_{P_{1}}[\phi]$. A randomized mechanism
$\mathcal{M}$ is of $\mu$-GDP if

$T(\mathcal{M}(D_{1}),\mathcal{M}(D_{2}))\geq
T(\mathcal{N}(0,1),\mathcal{N}(\mu,1))$

for any neighboring datasets $D_{1},D_{2}$ differing by one record,

The Gaussian mechanism can be applied to achieve $\mu$-GDP guarantees when
releasing a statistic or a query result from a dataset.

######  Definition 3 (Gaussian Mechanism of $\mu$-GDP).

(Dong et al., 2019) For a given query $f$ and a dataset $D$, a Gaussian
mechanism $\mathcal{M}(D)$ satisfying $\mu$-GDP is defined as

$f^{*}_{\mathcal{M}}(D)\sim\mathcal{N}(f(D),\Delta_{f}^{2}/\mu^{2}),$

where $\Delta_{f}=\max\limits_{D_{1},D_{2}}\|f(D_{1})-f(D_{2})\|_{1}$ for any
pair of neighboring datasets $(D_{1},D_{2})$.

Dong et al. (2019) also presents a noisy stochastic gradient descent (SGD)
algorithm as a composition of repeated Gaussian mechanisms application with
sub-sampling in If we denote the number of iterations of the SGD algorithm by
$T$, the sub-sampling rate by $r$, and the scale of noise added to the
gradient at each iteration by $\sigma$, then the total privacy loss under GDP
over $T$ iterations is $\mu=r\sqrt{T(e^{1/\sigma^{2}}-1)}$.

There is a duality between $(\epsilon,\delta)$-DP and $\mu$-GDP (Dong et al.,
2019), which helps to translate the privacy loss between the two DP
frameworks. If a mechanism is of $\mu$-GDP, then it also satisfies
$(\epsilon,\delta(\epsilon))$-DP, where
$\delta(\epsilon)=\Phi(-\frac{\epsilon}{\mu}+\frac{\mu}{2})-e^{\epsilon}\Phi(-\frac{\epsilon}{\mu}-\frac{\mu}{2}))$,
where $\Phi$ is the CDF of the standard normal distribution.

##  3 Private-preserving Density Estimation and Synthetic Data Generation
through DP-NF

In this section, we present differentially private density estimation via NF,
which can be used to synthesize privacy-preserving surrogate datasets before
their public release. Users can then conduct statistical analysis or train
machine learning algorithms on the synthetic data in the same way as if they
had the original data. In what follows, we first present the EHR data that
motivates the work, then provide an algorithm to incorporate DP in NF for
density estimation, apply the algorithm to generate private EHR data, and
finally examine the utility of the surrogate data via downstream learning
tasks including classification, regression, and VI.

###  3.1 Description of the EHR Dataset

The EHR dataset contains anonymized clinical measurements from 82 adult
patients. The data were collected in the context of a research project funded
by Google through its ATAP initiative, focusing on modeling noninvasive
measurements of cardiovascular dynamics. Table 1 lists the 26 clinical
attributes present in the dataset with the summary statistics. Figure 1 shows
the missing data pattern. Each row of the heat map was normalized to a zero to
one range, to highlight the relative magnitude of each clinical target. Each
patient has zero to nineteen clinical attributes. While there is not a single
patient that has all 26 measurements, at least three clinical attributes
(heart rate, diastolic blood pressure, and systolic blood pressure) are
available for all but one patient.

Table 1: Summary Statistics on the EHR dataset containing 26 clinical
measurements.

| Attribute |  Description |  Unit |  SD∗ |  n# |  Min |  Max
---|---|---|---|---|---|---|---
1 | heart_rate2 |  Heart rate |  bpm |  3.0 |  81 |  69.0 |  105.0
2 | systolic_bp_2 |  Systolic blood pressure |  mmHg |  1.5 |  81 |  83.0 |  180.0
3 | diastolic_bp_2 |  Diastolic blood pressure |  mmHg |  1.5 |  81 |  49.0 |  102.0
4 | cardiac_output |  Cardiac output |  L/min |  0.2 |  65 |  3.3 |  10.9
5 | systemic_vascular_resistan |  Systemic vascular resistance |  dynes$\cdot$s$\cdot$cm-5 |  50.0 |  64 |  668.6 |  2046.8
6 | pulmonary_vascular_resista |  Pulmonary vascular resistance |  dynes$\cdot$s$\cdot$cm-5 |  5.0 |  50 |  5.0 |  245.8
7 | cvp |  Central venous pressure |  mmHg |  0.5 |  30 |  2.0 |  28.0
8 | right_ventricle_diastole |  Right ventricle diastolic pressure |  mmHg |  1.0 |  11 |  2.0 |  10.0
9 | right_ventricle_systole |  Right ventricle systolic pressure |  mmHg |  1.0 |  46 |  12.0 |  45.0
10 | rvedp |  Right ventricle EDP |  mmHg |  1.0 |  46 |  1.0 |  20.0
11 | aov_mean_pg |  Average PG across aortic valve |  mmHg |  0.5 |  2 |  5.0 |  10.0
12 | aov_peak_pg |  Peak PG across aortic valve |  mmHg |  0.5 |  37 |  3.0 |  18.0
13 | mv_decel_time |  Mitral valve deceleration time |  ms |  6.0 |  41 |  88.0 |  289.0
14 | mv_e_a_ratio |  Mitral valve E/A ratio |  - |  0.2 |  39 |  0.7 |  20.9
15 | pv_at |  Pulmonary valve acceleration time |  ms |  6.0 |  18 |  53.0 |  190.0
16 | pv_max_pg |  Peak PG across pulmonary valve |  mmHg |  0.5 |  31 |  1.0 |  18.0
17 | ra_pressure |  Mean right atrial pressure |  mmHg |  0.5 |  50 |  3.0 |  15.0
18 | ra_vol_a4c |  Right atrial volume |  mL |  3.0 |  4 |  57.0 |  173.0
19 | la_vol_a4c |  Left atrial volume |  mL |  3.0 |  7 |  69.0 |  179.0
20 | lv_esv |  Left ventricular end-systolic volume |  mL |  10.0 |  1 |  32.0 |  32.0
21 | lv_vol_a4c |  Left ventricular volume |  mL |  20.0 |  5 |  59.0 |  642.0
22 | lvef |  Left ventricular ejection fraction |  - |  2.0 |  53 |  12.5 |  75.0
23 | lvot_max_flow |  Peak flow velocity across LVOT |  cm/s |  - |  0 |  - |  -
24 | pap_diastolic |  Diastolic PAP |  mmHg |  1.0 |  65 |  5.0 |  36.0
25 | pap_systolic |  Systolic PAP |  mmHg |  1.0 |  65 |  10.0 |  60.0
26 | wedge_pressure |  Pulmonary wedge pressure |  mmHg |  1.0 |  50 |  2.0 |  26.0

∗standard deviation; determined based on a preliminary literature review
(Yared et al., 2011; Gordon et al., 1983; Maceira et al., 2006).
---
#Number of patients with observed measurements on the respective attribute.

Figure 1: Missing data pattern in EHR dataset. Each row in the above heat map
is normalized to a $[0,1]$ range to highlight the relative magnitude of the
clinical target across patients.

This dataset focuses on cases of group II pulmonary hypertension (i.e.,
hypertensive/not hypertensive), where a reversible increase of PVR follows an
increase in left ventricular filling pressures222We do not consider pulmonary
arterial hypertension (group I pulmonary hypertension) nor make any claim of
differentiating group I and II pulmonary hypertension. Other criteria have
been proposed to classify hypertension due to left ventricular diastolic
dysfunction (see, e.g., Simonneau et al., 2013).. A often used binary
classification in practice is a criterion based on two clinical measurements,
the diastolic pulmonary artery pressure ($p_{p,d}$) and the systolic artery
pressure ($p_{p,s}$), i.e.

${\rm hypertensive}=\begin{cases}1,&\text{if}\quad p_{p,m}=\frac{2}{3}\cdot
p_{p,d}+\frac{1}{3}\cdot p_{p,s}>20\text{ mmHg}\\\ 0,&\text{if}\quad
p_{p,m}=\frac{2}{3}\cdot p_{p,d}+\frac{1}{3}\cdot p_{p,s}\leq 20\text{
mmHg.}\end{cases}$ (5)

Medical information, including EHR data, is highly sensitive and may reveal
patients’ medical conditions and history, among others, if not properly
protected for privacy. Sharing the data is not an option due to privacy
concerns. Generating and releasing synthetic data is proposed as an
alternative and is actively explored by researchers and in practice (the
SHARED team, 2023; Rankin et al., 2020; Dash et al., 2019; Yale et al., 2020;
Benaim et al., 2020).

###  3.2 Privacy-preserving Synthetic Data Generation through DP-NF

Figure 1 shows that the EHR dataset contains a considerable portion of missing
values. Before synthetic data generation, we first imputed the missing values
using the MICE package (v3.13.0) in R. The MICE package implements
multivariate imputation by chained equations (Van Buuren and Groothuis-
Oudshoorn, 2011), which draws, in an alternated fashion (as in a Gibbs’
sampler), imputed values for the missing attribute
$X_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bullet$}}}}},j}$
from the conditional distribution
$p(X_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bullet$}}}}},j}|\bm{\theta}_{j},\bm{X}_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bullet$}}}}},-j})$
and parameters $\bm{\theta}_{j}$ from its posterior distribution
$p(\bm{\theta}_{j}|\bm{X})$ for $j=1,\ldots,p$.
$\bm{X}_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bullet$}}}}},-j}$
contains all variables in the data except for
$X_{\mathchoice{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.6}{$\scriptscriptstyle\bullet$}}}}},j}$.

In the EHR data, seven attributes (cvp, aov_mean_pg, ra_vol_a4c, la_vol_a4c,
lv_esc, lv_vol_a4c, and lvot_max_flow) have large fractions of missing values
that range from 63% to 100%. These large fractions of missing data turned out
to be problematic for MICE. For that reason, these seven attributes were not
considered in the imputation and any subsequent analysis. We obtained 5 sets
of imputed data and then split each of the 5 imputed datasets into a training
set and a testing set with 64 and 20 patients, respectively.

We used DP-NF for density estimation in each of the 5 imputed datasets using
Algorithm 1. After computing the privacy-preserving density $p^{*}(\bm{x})$
(we use ∗ to represent private or sanitized quantities), we can generate
synthetic data $\bm{x}^{*}$ by first sampling from the base distribution
$\bm{z}_{0}\sim q_{0}$ and then use the learned NF map
$F_{\lambda^{*}}(\bm{z}_{0})$ to generate realizations from $p^{*}$. Though
the algorithm is presented using the SGD framework, it can easily be extended
for other gradient-based optimization such as RMSprop (Tieleman et al., 2012)
and Adam (Kingma and Ba, 2014), by replacing the update
($\lambda^{*(t+1)}=\lambda^{*(t)}-\eta\,g^{*(t)}$) with their respective
parameter update paradigms.

Algorithm 1 DP-NF for Density Estimation

0: data set $\bm{x}=\\{\bm{x}_{1},\ldots,\bm{x}_{n}\\}$, initial parameter
values $\lambda^{(0)}$ of NF $F_{\lambda}(\cdot)$ with base distribution
$q_{0}$, learning rate $\eta$, DP noise scale $\sigma$, subsampling rate $r$,
clipping constant on gradient $C$, number of iterations $T$.

0: privacy-preserving estimated density $p^{*}(\bm{x})$

for $t=0,\cdots,T$ do

sub-sample $\bm{x}^{(t)}$ from $\bm{x}$ with rate $r$; let $b^{(t)}$ be the
size of $\bm{x}^{(t)}$

for $\bm{x}_{i}\in\bm{x}^{(t)}$ do

let $\bm{z}_{i}\\!=\\!F_{\lambda}^{-1}(\bm{x}_{i})$

obtain $l_{i}=-\log q_{0}(\bm{z}_{i})-\log|\partial
F_{\lambda}/\partial\bm{z}_{i}|$

compute $\bm{g}_{i}^{(t)}=\nabla_{\lambda}l_{i}$

$\bm{g}_{i}^{(t)}\leftarrow\bm{g}_{i}^{(t)}/\max(1,\|\bm{g}_{i}^{(t)}\|_{2}/C)$

end for

obtain sanitized gradient
$\bm{g}^{*(t)}=\left(\sum_{i=1}^{b^{(t)}}\bm{g}_{i}^{(t)}+\mathcal{N}(\bm{0},\sigma^{2}C^{2}\bm{I})\right)/b^{(t)}$

$\lambda^{*(t+1)}=\lambda^{*(t)}-\eta\,g^{*(t)}$

end for

In the application of Algorithm 1 to the EHR data, we set $r=0.5,\eta=2\times
10^{-5},T=8000,C=10$. We examined 4 privacy budgets $\mu$ in the setting of
$\mu$-GDP, i.e., $\mu=(6.10,3.92,2.45,1.49$)333We also attempted smaller $\mu$
(e.g. $\mu\leq 1$), but the DP-NF algorithm did not converge due to a large
amount of noise injected to the gradients., corresponding to
$\sigma=(7.36,11.44,18.28,29.93)$ and $\epsilon=(32,16,8,4)$ if $\delta=0.01$
in the $(\epsilon,\delta)$-DP setting. The NF architecture we used has 15 MADE
layers, each consisting of a fully connected neural network having 1 hidden
layer with 200 neurons, and one batch normalization layer following each MADE.
For each of the 5 imputed training sets, we run DP-NF 10 times to obtain 10
privacy-preserving densities; from each estimated density, 10 synthetic
datasets were generated, leading to a total of 500 synthetic data sets444We
generated multiple data sets to examine the stability of the results in the
downstream learning tasks in Sections 3.3 and 3.4. In practice, one may
release multiple synthetic data to take into account the uncertainty around
sanitization and synthesis, and then combine the inferences from multiple sets
using the rule in Liu (2022). Correspondingly, the overall privacy budget
needs to split across multiple sets when generating privacy-preserving
synthetic data. For machine learning tasks based on synthetic data, releasing
a single synthetic dataset may be sufficient.. Each synthetic data contains
the same number of subjects (82) as in the original dataset.

We use the generated synthetic data for three subsequent learning and
inferential tasks. The first two focus on detecting pulmonary hypertension and
predicting the mean pulmonary arterial pressure from the testing data (Section
3.3), while the last conducts VI for the parameters of a physics-based model
(Section 3.4).

###  3.3 Classification and Regression based on Privacy-preserving Synthetic
EHR Data

The goal of this learning task is to detect pulmonary hypertension, defined as
a mean pulmonary artery pressure $p_{p,m}$ greater than 20 mmHg in Eqn. (5).
Toward that end, we train a binary classifier (hypertensive/not hypertensive)
using support vector machine (SVM, Cortes and Vapnik, 1995) and a numerical
estimate of the mean pulmonary artery pressure using a random forest (see,
e.g., Breiman, 2001). We evaluated the classification and regression testing
accuracy of these two procedures trained on the privacy-preserving synthetic
data. SVM outputs a probability of having pulmonary hypertension for each
case. Instead of using a cutoff of 0.5, we adopted the Fowlkes–Mallows (FM)
index (Fowlkes and Mallows, 1983) to find an optimal cutoff. The FM index is
defined as
$\sqrt{\frac{\text{TP}}{\text{TP+FP}}\cdot\frac{\text{TP}}{\text{TP+FN}}}$,
where TP is the number of true positives, FP is the number of false positives,
and FN is the number of false negatives. The cutoff that produces the maximum
FM index was used.

Tables 2 and 3 summarize the prediction results over the 500 synthetic
datasets. Both tables present two sets of results, one based on SVM or random
forest trained on the raw generated synthetic data, the other based on SVM or
random forest trained on post-processed synthetic data by projecting an out-
of-bound synthetic data point back to either its lower or upper bound. The
reason for the post-processing is that the attributes in the EHR data are
clinical attributes and they are naturally bounded. For example, the attribute
“peak pressure gradient across pulmonary valve” relates to pulmonary valve
stenosis; values outside $[0,50]$ would be regarded as physiologically
impossible for our EHR dataset which does not contain patients with pulmonary
valve stenosis. Therefore, if a synthetic value on “peak PG across pulmonary
valve” is $<0$ mmHg, it is set at 0 and, is set at 50 mmHg if found $>50$
mmHg.

Table 2: Privacy-preserving mean classification accuracy rate (SD) via SVM
based on synthetic data generated via DP-NF

Truncation∗ | Imputation | Original | $\mu=\infty$ | $\mu=6.10$ | $\mu=3.92$ | $\mu=2.45$ | $\mu=1.49$
---|---|---|---|---|---|---|---
| 1 | 1.00 | 0.95 (0.07) | 0.91 (0.08) | 0.82 (0.24) | 0.81 (0.15) | 0.87 (0.09)
| 2 | 1.00 | 0.88 (0.24) | 0.93 (0.06) | 0.86 (0.10) | 0.88 (0.12) | 0.83 (0.11)
No | 3 | 1.00 | 0.92 (0.09) | 0.91 (0.06) | 0.90 (0.08) | 0.88 (0.10) | 0.72 (0.38)
| 4 | 0.95 | 0.89 (0.10) | 0.88 (0.05) | 0.80 (0.09) | 0.76 (0.22) | 0.86 (0.12)
| 5 | 0.95 | 0.85 (0.20) | 0.90 (0.07) | 0.88 (0.17) | 0.92 (0.17) | 0.80 (0.10)
| mean | 0.98 | 0.90 (0.16) | 0.91 (0.07) | 0.85 (0.15) | 0.85 (0.20) | 0.81 (0.20)
| 1 | 1.00 | 0.92 (0.08) | 0.87 (0.09) | 0.79 (0.24) | 0.80 (0.15) | 0.85 (0.09)
| 2 | 1.00 | 0.83 (0.26) | 0.89 (0.07) | 0.86 (0.09) | 0.86 (0.11) | 0.83 (0.10)
Yes | 3 | 1.00 | 0.90 (0.11) | 0.90 (0.07) | 0.90 (0.08) | 0.88 (0.10) | 0.70 (0.37)
| 4 | 0.95 | 0.85 (0.16) | 0.87 (0.06) | 0.80 (0.08) | 0.74 (0.33) | 0.82 (0.12)
| 5 | 0.95 | 0.77 (0.28) | 0.85 (0.08) | 0.85 (0.17) | 0.88 (0.17) | 0.80 (0.10)
| mean | 0.98 | 0.85 (0.20) | 0.88 (0.08) | 0.84 (0.15) | 0.83 (0.20) | 0.80 (0.20)

∗ Imputed and synthetic data on an attribute may exceed the physiologically
feasible bounds. The results in
---
“Truncation = Yes” are obtained based on post-processed imputed and synthetic
data by setting the out-of-
bound values at the closest bound.

Table 3: Privacy-preserving prediction mean squared error (SD) via random
forest based on synthetic data generated via DP-NF

Truncation∗ | Imputation | Original | $\mu=\infty$ | $\mu=6.10$ | $\mu=3.92$ | $\mu=2.45$ | $\mu=1.49$
---|---|---|---|---|---|---|---
| 1 | 20.63 | 40.30 (23.37) | 54.71 (20.20) | 62.45 (20.55) | 83.44 (42.54) | 70.39 (30.52)
| 2 | 18.02 | 35.70 (19.87) | 48.08 (18.10) | 60.06 (22.89) | 62.44 (33.44) | 80.30 (44.03)
No | 3 | 11.50 | 71.76 (62.20) | 36.53 (7.39) | 47.02 (23.01) | 72.22 (28.51) | 71.36 (30.02)
| 4 | 18.87 | 32.92 (27.36) | 44.15 (12.19) | 63.09 (19.02) | 46.81 (26.87) | 60.52 (17.59)
| 5 | 20.66 | 81.45 (59.57) | 53.98 (17.75) | 59.53 (29.41) | 41.09 (18.28) | 122.91 (41.70)
| mean | 17.94 | 52.52 (47.18) | 47.40 (17.17) | 58.41 (23.86) | 61.39 (34.49) | 80.10 (39.92)
| 1 | 21.49 | 47.50 (27.68) | 72.30 (23.98) | 79.00 (26.64) | 95.76 (43.68) | 75.85 (28.33)
| 2 | 18.54 | 47.08 (24.01) | 56.77 (15.78) | 68.91 (22.45) | 73.43 (36.21) | 88.21 (42.44)
Yes | 3 | 12.12 | 68.25 (55.06) | 47.09 (12.74) | 55.26 (22.01) | 80.08 (29.44) | 85.45 (29.94)
| 4 | 18.06 | 35.55 (30.28) | 55.24 (13.12) | 71.30 (24.08) | 51.26 (26.73) | 66.17 (22.38)
| 5 | 19.79 | 78.79 (53.02) | 63.31 (18.31) | 68.13 (31.79) | 53.75 (21.02) | 112.54 (40.61)
| mean | 18.00 | 55.91 (42.61) | 58.94 (19.23) | 68.52 (26.77) | 71.91 (35.70) | 84.91 (36.52)

∗ Imputed and synthetic data on an attribute may exceed the physiologically
feasible bounds. The results in
---
“Truncation = Yes” are obtained based on post-processed imputed and synthetic
data by setting the out-of-
bound values at the closest bound.

Overall, as expected, the higher the privacy loss (larger $\mu$), the higher
the accuracy rates (Table 2) and the smaller the prediction MSE (Table 3). The
post-processing truncation needed to enforce bounds for each clinical
measurement satisfies the practical data constraints but negatively affects
the prediction results; the accuracy rates decrease in Table 2 and the
prediction MSE increases in Table 3 with truncation vs. without truncation,
regardless of the preferred privacy budget; but the trend over $\mu$ is
similar to that before the truncation.

###  3.4 VI of Parameters in Physics-based model using Privacy-preserving
Synthetic EHR data

####  3.4.1 the CVSim-6 physiologic model

The evolution of blood pressure, flow, and volume in the circulatory system of
an adult subject is simulated, in this study, using CVSim-6 (Davis, 1991;
Heldt et al., 2010), a lumped parameter hemodynamic model with 6-compartments
available through the PhysioNet repository (Goldberger et al., 2000). A custom
Python/Cython implementation can be found at
https://github.com/desResLab/supplMatHarrod20. The model includes compartments
for the left heart, right heart, systemic arteries, systemic veins, pulmonary
arteries and pulmonary veins. Arterial and venous compartments consist of RC
circuits, while heart chambers are simulated using a pressure generator, two
unidirectional diodes and a resistor to account for the pressure loss produced
at the valves. All elements are considered linear, disregarding collapsibility
in veins associated with negative pressures. Inertial effects are also assumed
negligible (the model does not contain inductors) and a two-chamber heart is
considered, disregarding any contribution from the atria. In addition,
interaction is only considered between adjacent compartments. Thus, unlike
other model formulations in the literature (see, e.g., CircAdapt, see Arts et
al., 2005) no mechanical interaction is considered between the two ventricular
chambers through the septal wall. CVSim-6 consists of a system of six
differential equations (one per compartment). The flows between the
compartments, under the assumption of nonlinear unidirectional valves (without
regurgitation), are expressed as (see Davis, 1991, Sections 4.2.1 and 4.2.2)

$\begin{tabular}[]{cc}$q_{li}=\begin{cases}(P_{pv}-P_{l})/R_{li}&\text{if}\,\,P_{pv}>P_{l}\\\
0&\text{otherwise}\end{cases}$&$q_{lo}=\begin{cases}(P_{l}-P_{a})/R_{lo}&\text{if}\,\,P_{l}>P_{a}\\\
0&\text{otherwise}\end{cases}$\\\ \\\
$q_{ri}=\begin{cases}(P_{v}-P_{r})/R_{ri}&\text{if}\,\,P_{v}>P_{r}\\\
0&\text{otherwise}\end{cases}$&$q_{ro}=\begin{cases}(P_{r}-P_{pa})/R_{ro}&\text{if}\,\,P_{r}>P_{pa}\\\
0&\text{otherwise}\end{cases}$\\\ \\\
$q_{a}=(P_{a}-P_{v})/R_{a}$&$q_{pv}=(P_{pa}-P_{pv})/R_{pv}$\\\ \end{tabular},$
(6)

leading to the following differential equations for RC elements with fixed and
time-varying capacitance

$\begin{split}\dfrac{dP_{l}}{dt}&=\dfrac{q_{li}-q_{lo}-(P_{l}-P_{th})\,dC_{l}(t)/dt}{C_{l}(t)},\quad\dfrac{dP_{a}}{dt}=\dfrac{q_{lo}-q_{a}}{C_{a}},\\\
\dfrac{dP_{v}}{dt}&=\dfrac{q_{a}-q_{ri}}{C_{v}},\quad\dfrac{dP_{r}}{dt}=\dfrac{q_{ri}-q_{ro}-(P_{r}-P_{th})\,dC_{r}(t)/dt}{C_{r}(t)},\\\
\dfrac{dP_{pa}}{dt}&=\dfrac{q_{ro}-q_{pv}}{C_{pa}},\quad\,\,\dfrac{dP_{pv}}{dt}=\dfrac{q_{pv}-q_{li}}{C_{pv}}.\\\
\end{split}$ (7)

Initial conditions are determined from a system of linear equilibrium
equations on the stressed volume,

$\displaystyle C_{l,dias}\,(P_{l,dias}-P_{th})-C_{l,sys}\,(P_{l,sys}-P_{th})=$
$\displaystyle\,C_{r,dias}\,(P_{r,dias}-P_{th})-C_{r,sys}\,(P_{r,sys}-P_{th})$
$\displaystyle=$
$\displaystyle\,T_{sys}\,\dfrac{P_{l,sys}-P_{a}}{R_{lo}}=T_{tot}\,\dfrac{P_{a}-P_{v}}{R_{a}}$
(8) $\displaystyle=$
$\displaystyle\,T_{dias}\,\dfrac{P_{v}-P_{r,dias}}{R_{v}}=T_{sys}\,\dfrac{P_{r,sys}-P_{pa}}{R_{ro}}$
$\displaystyle=$
$\displaystyle\,T_{tot}\,\dfrac{P_{pa}-P_{pv}}{R_{pv}}=T_{dias}\,\dfrac{P_{pv}-P_{l,dias}}{R_{li}}$
$\displaystyle V_{tot}-V_{0,tot}=$
$\displaystyle\,C_{l,dias}(P_{l,dias}-P_{th})+C_{a}(P_{a}-\frac{1}{3}P_{th})+$
$\displaystyle\,C_{v}\,P_{v}+C_{r}(P_{r}-P_{th})+C_{pa}(P_{pa}-P_{th})+$
$\displaystyle\,C_{pv}(P_{pv}-P_{th}).$

The acronyms used in the equations above are defined in Table 8 in the
Appendix. Default model parameters are reported in Table 4 and pressure,
volume, and flow rate time histories produced by CVSim-6, when such parameters
are used as inputs, are shown in Figure 2.

Table 4: Default parameters for CVSim-6 model (see Table 4.1 in Davis (1991))

Compartment |  $R_{i}$ (mmHg s/mL) |  $R_{o}$ (mmHg s/mL) | C (mL/mmHg) |  $V_{0}$ (mL)
---|---|---|---|---
Left Heart | 0.01 | 0.006 | 0.4-10 | 15
Systemic Arteries | 0.006 | 1.00 | 1.6 | 715
Systemic Veins | 1.00 | 0.05 | 100.0 | 2500.0
Right Heart | 0.05 | 0.003 | 1.2-20 | 15
Pulmonary Arteries | 0.003 | 0.08 | 4.3 | 90
Pulmonary Veins | 0.08 | 0.01 | 8.4 | 490

Systemic parameters: total blood volume = 5000 mL, heart rate = 72 bpm;
thransthoracic pressure = -4.0 mmHg.
---

(a) LV pressures.

(b) LV volumes.

(c) LV flows.

(d) RV pressures.

(e) RV volumes.

(f) RV flows.

Figure 2: Time histories of state and auxiliary variables from CVSim-6 model,
when using the default input parameters listed in Table 4 (see Figure 4-4 in
Davis, 1991).

The CVSim-6 model is compactly represented as $\bm{x}=\bm{f}(\bm{z})$, where
$\bm{z}$ and $\bm{x}$ represent the model inputs (parameters, either unknown
or fixed at constants) and outputs (observed), respectively. A subset
consisting of 8 outputs was in the construction of the likelihood function
given the EHR data, including heart rate, pulmonary vascular resistance,
central venous pressure, right ventricular diastolic pressure, right
ventricular systolic pressure, right ventricle end-diastolic pressure, average
pressure gradient across the aortic valve, and peak pressure gradient across
the aortic valve. Such quantities are obtained by post-processing the time
histories for the state and auxiliary variables shown in Figure 2. In this
study, we regard right ventricular resistance $R_{ro}$ and arterial
capacitance $C_{a}$ as unknown parameters (inputs to the model), whereas the
other parameters are fixed to their default values in Table 4.

####  3.4.2 VI based on privacy-preserving synthetic EHR data

To infer the CVSim-6 model parameters $R_{ro}$ and $C_{a}$ for hypertensive
patients, we construct the likelihood based on a subset of 43 synthetic
hypertensive patients. We assume a uniform prior of the transformed parameters
$(R^{\prime},C^{\prime})$ on $[-7,7]^{2}$,

$(R_{ro},C_{a})=(\tanh(3/7\cdot
R^{\prime})\cdot(H_{R}-L_{R})/2+R_{0},\tanh(3/7\cdot
C^{\prime})\cdot(H_{C}-L_{C})/2+C_{0}),$ (9)

where $R_{ro}\in[L_{R},H_{R}]$, $C_{a}\in[L_{C},H_{C}]$ are the original
parameters with respective bounds, $R_{0},C_{0}$ are default values, and
$(R^{\prime},C^{\prime})$ are their transformed unbounded counterpart.

We assume the 8-dimensional output $\bm{x}_{i}=(x_{i,1},\ldots,x_{i,8})$,
given the parameters $(R_{ro},C_{a})$, follows a multivariate Gaussian
distribution. Assuming independence across the 8 outputs with known marginal
variance for each output, the he log-likelihood of $(R_{ro},C_{a})$ is

$\ell(R_{ro},C_{a};\bm{x})=-\frac{1}{2}\sum_{i=1}^{n}\,\left[\bm{x}_{i}-\bm{f}(R_{ro},C_{a})\right]^{T}\,\bm{\Sigma}^{-1}\,\left[\bm{x}_{i}-\bm{f}(R_{ro},C_{a})\right]^{2}+\mbox{constant},$
(10)

where $\bm{f}(R_{ro},C_{a})$ represents the output of the CVSim-6 model with
all but two input variables fixed at their default values and $\bm{\Sigma}$ is
a diagonal matrix with marginal variances equal to (444.84, 145.60, 2.21,
45034.41, 475.59, 24.77, 84.06, 15.48) for the 8-dimensional output.

The CVSim-6 model is computationally expensive. To reduce the computational
cost of VI with input parameters $R_{ro}$ and $C_{a}$, we trained offline a
fully connected neural network surrogate. The network contains 2 hidden layers
with 64 and 32 neurons, respectively, and Tanh activations. We generated 100
realizations using Sobol’ sampling in the ($R_{ro}$,$C_{a}$) plane and
computed the corresponding outputs from the CVSim-6 model. We then trained the
surrogate model based on 100 sets of inputs (2-dimensional) and outputs
(8-dimensional) in 120,000 iterations with the RMSprop optimizer (learning
rate 0.01, learning rate scheduler with exponential decay factor 0.9999) using
a $\ell_{2}$ loss with $\ell_{2}$ regularization (penalty 0.0001).

We train the surrogate model and run NF to obtain posterior distribution on
the transformed $(R^{\prime},C^{\prime})$ first, which are then transformed
back to obtain the posterior distribution and VI of $R_{ro}$,$C_{a}$. VI was
performed using a MAF architecture with 5 alternated MADE and batch
normalization layers. Each MADE uses a fully connected neural network
characterized by 1 hidden layer with 100 neurons. The activation functions
within and between successive MADE layers are ReLU. The network is trained for
15,000 iterations, generating batches of 500 samples at each iteration. We use
RMSProp as the optimizer with a learning rate of 0.01 and an exponential
scheduler with a decay rate of 0.9999.

VI results for $R_{ro}$ and $C_{a}$ based on the privacy-preserving synthetic
data are presented in Table 5, showing the posterior mean and posterior
standard deviations of the two parameters, and the correlations between them
across 5 different imputations. In summary, the estimates based on privacy-
preserving synthetic data are somewhat different from those based on the
original data and there does not appear to be an obvious trend over $\mu$ in
terms of how the former differs from the latter, except for the posterior
correlation between $R_{ro}$ and $C_{a}$, which appears to get more negative
as $\mu$, the privacy loss, decreases.

Table 5: Privacy-preserving VI for CVSim-6 model parameters $R_{ro}$ and
$C_{a}$ based on synthetic data generated via DP-NF

parameter |  posterior estimate | original | $\mu=6.10$ | $\mu=3.92$ | $\mu=2.45$ | $\mu=1.49$
---|---|---|---|---|---|---
$R_{ro}$ | mean |  35.70 ( 1.918) |  43.83 ( 2.798) |  45.90 ( 2.788) |  47.52 ( 4.400) |  45.61 ( 3.748)
| SD |  1.452 ( 0.038) |  1.907 ( 0.032) |  2.168 ( 0.341) |  2.189 ( 0.383) |  2.144 ( 0.165)
$C_{a}$ | mean ($\times 10^{-4}$) |  6.520 ( 0.130) |  8.569 ( 1.169) |  8.345 ( 1.647) |  7.972 ( 0.967) |  8.917 ( 1.023)
| SD ($\times 10^{-4}$) |  0.347 ( 0.013) |  0.609 ( 0.135) |  0.666 ( 0.362) |  0.594 ( 0.200) |  0.724 ( 0.192)
$R_{ro},C_{a}$ correlation | -0.120 ( 0.042) | -0.123 ( 0.192) | -0.169 ( 0.275) | -0.213 ( 0.244) | -0.193 ( 0.164)

Figure 3 presents the scatter plots of the posterior samples on $R_{ro}$ and
$C_{a}$ from the NF run on one private synthetic dataset. The observations are
consistent with those in Table 5. In general, the posterior samples generated
from the NF constructed from the privacy-preserving synthetic data are located
near the high-density regions of the posterior distribution based on the
original data and true model, though not quite around the mode, and the mild
negative correlation between the two parameters is also roughly retained for
$\mu\geq 2.45$. There is more dispersion among the posterior samples at all
$\mu$, especially at $\mu=1.49$, due to the noise injected to achieve DP
guarantees, as expected.

Figure 3: Example of privacy-preserving posterior samples of $R_{ro}$ and
$C_{a}$ through MAF based on one synthetic dataset. The contours are the
densities of the true posterior distribution of $R_{ro}$ and $C_{a}$ given the
true model and the original data.

##  4 Privacy-Preserving VI via DP-NF

In this section, we examine the feasibility of privacy-preserving VI through
sanitization of the optimization procedure employed by NF for VI, referred as
the DP-NF-VI Procedure for short. Though NF can be used for both density
estimation and VI and gradient-based optimization can be used to obtain the
respective solutions in both cases, the loss functions in the two cases are
different (likelihood in Eqn. (2.1) vs. free energy bound in Eqn. (4)). In
what follows, we first provide an algorithm for DP-NF-VI with DP guarantees
for NF-VI and then examine the inferential utility of privacy-preserving VI
via DP-NF in simulated data.

###  4.1 The DP-NF-VI Procedure

The steps of the DP-NF-VI Procedure is given in Algorithm 2 Similar to
Algorithm 1, Algorithm 2 is obtained from an SGD optimizer by clipping and
perturbing the gradient with additive noise; and it can easily be extended to
other gradient-based optimization procedures such as RMSprop or Adam by
replacing the parameter updating step
($\lambda^{*(t+1)}=\lambda^{*(t)}-\eta\,g^{*(t)}$) with their respective
parameter update paradigms. The differences between the two algorithms are in
the loss function and the privacy-preserving output which are the free energy
and the privacy-preserving variational density approximation $p^{*}(\bm{z})$,
respectively, in Algorithm 1. We can draw samples from privacy-preserving
samples of $\bm{z}$ from $p^{*}(\bm{z})$ by first generating $\bm{z}_{0}\sim
q_{0}$ and then applying the transformation $F_{\lambda^{*}}(\bm{z}_{0})$.

Algorithm 2 The DP-NF-VI Procedure

0: observed data $\bm{x}=\\{\bm{x}_{1},\ldots,\bm{x}_{n}\\}$, target posterior
distribution $p(\bm{z}|\bm{x})$, NF $F_{\lambda}(\bm{z})=f_{K}\circ
f_{K-1}\circ\cdots f_{1}(\bm{z})$ with base distribution $q_{0}$, initial
values $\lambda^{(0)}$, learning rate $\eta$, DP noise scale $\sigma$, sub-
sampling rate $r$, gradient clipping constant $C$, number of iterations $T$,
batch size $m$.

0: privacy-preserving variational distribution $p^{*}(\bm{z})$.

for $t=0,\cdots,T$ do

sub-sample $\bm{x}^{(t)}$ from $\bm{x}$ with rate $r$; let $b^{(t)}$ denote
the size of $\bm{x}^{(t)}$

draw $m$ samples from $\bm{z}_{0}\sim q_{0}$

for $\bm{x}_{i}\in\bm{x}^{(t)}$ do

$l_{i}(\lambda)=m^{-1}\sum_{j=1}^{m}\left\\{n^{-1}\log
q_{0}(\bm{z}_{0,j})-\log
p(\bm{x}_{i},F_{\lambda}(\bm{z}_{0,j}))-n^{-1}\log|\partial
F_{\lambda}/\partial\bm{z}_{0,j}|\right\\}$

obtain $\bm{g}_{i}^{(t)}=\nabla_{\lambda}l_{i}(\lambda)$ and
$\bm{g}_{i}^{(t)}\leftarrow\bm{g}_{i}^{(t)}/\max(1,\|\bm{g}_{i}^{(t)}\|_{2}/C)$

end for

obtain sanitized gradient
$\bm{g}^{*(t)}=\left(\sum_{i=1}^{b^{(t)}}\,\bm{g}_{i}^{(t)}+\mathcal{N}(\bm{0},\sigma^{2}C^{2}\bm{I})\right)/b^{(t)}$

$\lambda^{*(t+1)}=\lambda^{*(t)}-\eta\,g^{*(t)}$

end for

###  4.2 Application of DP-NF-VI to Simulated Data

We used a simulated dataset555We also ran the algorithm on the EHR dataset but
it failed to converge. to evaluate the utility of VI obtained via Algorithm 2.
Specifically, we simulated outputs $\bm{x}$ as given covariates $\bm{w}$ and
model parameters $\bm{\beta}$ from the following nonlinear model

$x_{i,k}=f(\bm{w}_{i},\bm{\beta})+r_{i}+e_{i,k},$ (11)

where

$f(\bm{w}_{i},\bm{\beta})=\beta_{0}+\exp[\beta_{1}\,w_{i,1}]+\log[w_{i,2}+\exp(\beta_{2})]+\exp[\beta_{2}\,w_{i,3}]+\log[w_{i,4}+\exp(\beta_{4})],$
(12)

and, additionally,

$r_{i}\sim\mathcal{N}(0,\sigma_{0}^{2}),\quad
e_{i,k}\sim\mathcal{N}(0,\sigma^{2}),$ (13)

for $i=1,\ldots,6000$ and $k=1,\ldots 5$. The covariates $\bm{w}$ were
generated using $w_{i,j}\sim\mathcal{U}(0,a_{j})$ independently with
$i=1,\cdots,n$ and $a_{1}=1$, $a_{2}=3$, $a_{3}=0.5$, $a_{4}=2$. We set
$\sigma_{0}=\sigma=0.2$ and $\bm{\beta}=(0.2,1.0,0.8,-1.2,0.6)^{T}$. The
random effect $r_{i}$ introduces correlation among 5 repeated measurements
observed for the $i$-th subject, where the correlation is
$\sigma^{2}/(\sigma^{2}+\sigma_{0}^{2})=1/2$.

Our goal is to estimate a privacy-preserving posterior distribution of
$\bm{\beta}$ and obtain its privacy-preserving posterior inference given a
non-informative uniform prior $p(\bm{\beta})$ and the likelihood
$l(\bm{\beta}|\bm{x})$, where, WLOG, $\sigma^{2}_{0},\sigma^{2}$ are assumed
known. We applied Algorithm 2 to the simulated data at various levels of
$\mu$-GDP guarantees ($\mu=6.58,1.12,0.5$ and 0.27, which correspond to
$\epsilon=50,5,2,1$ when $\delta=1\times 10^{-5}$). We used a MAF architecture
with one MADE (1 hidden layer with 10 neurons) and one batch norm layer. We
also used a batch of size $m=1,000$ to compute the Monte Carlo estimates of
the loss function in Algorithm 2, a sub-sampling rate $r=500/6,000$, learning
rate $\eta=0.01$ with exponential scheduler decay factor 0.999, number of
iterations $T=8,000$, and clipping constant $C=10.0$.

The results on the privacy-preserving posterior inference for $\bm{\beta}$ are
reported in Table 6. In summary, the privacy-preserving posterior means remain
roughly constant regardless of the selected privacy loss parameter $\mu$,
whereas the SD values in the parentheses are greater than the corresponding
non-private values (column “original VI NF”), and become increasingly larger
when $\mu$ decreases. Additionally, the private correlations among different
elements of $\bm{\beta}$ computed using DP-NF-VI are in general more
pronounced with respect to the corresponding original values and are
associated with large SDs from the 5 different DP-NF-VI runs, indicating a
general lack of stability for these estimates.

Table 6: Differentially private VI on the nonlinear regression model
parameters $\bm{\beta}$ from DP-NF-VI (Algorithm 2).

parameter |  original nonlinear model |  original NF-VI |  ${\mu=6.68}$ ${\sigma=1.30}$ |  ${\mu=1.12}$ ${\sigma=6.68}$ |  ${\mu=0.50}$ ${\sigma=14.88}$ |  ${\mu=0.27}$ ${\sigma=27.82}$
---|---|---|---|---|---|---
Posterior Mean |
$\beta_{0}$ ( 0.2) |  0.1581 |  0.1593 ( 0.0020) |  0.1532 ( 0.0035) |  0.1381 ( 0.0151) |  0.1231 ( 0.0486) |  0.1419 ( 0.0701)
$\beta_{1}$ ( 1.0) |  1.0022 |  1.0016 ( 0.0015) |  0.9952 ( 0.0040) |  0.9957 ( 0.0036) |  0.9922 ( 0.0059) |  0.9943 ( 0.0026)
$\beta_{2}$ ( 0.8) |  0.8262 |  0.8265 ( 0.0013) |  0.8386 ( 0.0034) |  0.8404 ( 0.0150) |  0.8378 ( 0.0208) |  0.8584 ( 0.0278)
$\beta_{3}$ (-1.2) | -1.1909 | -1.1920 ( 0.0008) | -1.1928 ( 0.0019) | -1.1891 ( 0.0079) | -1.1822 ( 0.0201) | -1.1945 ( 0.0246)
$\beta_{4}$ ( 0.6) |  0.6088 |  0.6092 ( 0.0006) |  0.6100 ( 0.0010) |  0.6215 ( 0.0057) |  0.6311 ( 0.0390) |  0.6057 ( 0.0479)
Posterior SD |
$\beta_{0}$ |  0.0265 |  0.0255 ( 0.0011) |  0.1341 ( 0.0147) |  0.2576 ( 0.0241) |  0.2573 ( 0.0490) |  0.2791 ( 0.0825)
$\beta_{1}$ |  0.0036 |  0.0036 ( 0.0001) |  0.1089 ( 0.0884) |  0.0687 ( 0.0587) |  0.1344 ( 0.1247) |  0.0554 ( 0.0058)
$\beta_{2}$ |  0.0179 |  0.0174 ( 0.0008) |  0.0755 ( 0.0222) |  0.1267 ( 0.0250) |  0.1729 ( 0.0309) |  0.1655 ( 0.0305)
$\beta_{3}$ |  0.0151 |  0.0151 ( 0.0004) |  0.0812 ( 0.0056) |  0.1341 ( 0.0141) |  0.1644 ( 0.0203) |  0.1581 ( 0.0217)
$\beta_{4}$ |  0.0192 |  0.0191 ( 0.0006) |  0.0889 ( 0.0163) |  0.1197 ( 0.0089) |  0.1595 ( 0.0384) |  0.2012 ( 0.0399)
Posterior Correlation |
$\beta_{0},\beta_{1}$ | -0.1200 | -0.0319 ( 0.0499) | -0.1147 ( 0.2240) | -0.1230 ( 0.2623) |  0.0237 ( 0.1028) | -0.1619 ( 0.2592)
$\beta_{0},\beta_{2}$ | -0.4340 | -0.3415 ( 0.1899) | -0.1180 ( 0.5120) | -0.3923 ( 0.4756) |  0.0396 ( 0.3675) | -0.3269 ( 0.7675)
$\beta_{0},\beta_{3}$ | -0.7370 | -0.7550 ( 0.0127) | -0.9108 ( 0.0851) | -0.8923 ( 0.0353) | -0.8629 ( 0.0407) | -0.8622 ( 0.0420)
$\beta_{0},\beta_{4}$ | -0.4880 | -0.4883 ( 0.0356) | -0.5299 ( 0.4673) | -0.8545 ( 0.0839) | -0.6826 ( 0.1586) | -0.4920 ( 0.5249)
$\beta_{1},\beta_{2}$ | -0.0030 | -0.0082 ( 0.0417) | -0.0310 ( 0.5503) |  0.0285 ( 0.0406) |  0.0916 ( 0.1965) | -0.0395 ( 0.1101)
$\beta_{1},\beta_{3}$ | -0.0160 | -0.0892 ( 0.0264) |  0.1255 ( 0.2115) | -0.0381 ( 0.2270) | -0.1815 ( 0.1485) | -0.0691 ( 0.2162)
$\beta_{1},\beta_{4}$ | -0.0080 | -0.0680 ( 0.0252) | -0.2880 ( 0.4993) |  0.1445 ( 0.2869) | -0.0457 ( 0.0595) |  0.0784 ( 0.2416)
$\beta_{2},\beta_{3}$ |  0.0040 | -0.0538 ( 0.1526) | -0.0947 ( 0.6286) |  0.1131 ( 0.4442) | -0.2520 ( 0.4011) |  0.1453 ( 0.7946)
$\beta_{2},\beta_{4}$ | -0.0020 | -0.0836 ( 0.0936) | -0.2267 ( 0.4445) |  0.3336 ( 0.5772) | -0.2036 ( 0.2303) | -0.0271 ( 0.5412)
$\beta_{3},\beta_{4}$ | -0.0020 |  0.0318 ( 0.0443) |  0.3368 ( 0.3616) |  0.6469 ( 0.1602) |  0.4218 ( 0.0884) |  0.2300 ( 0.5860)
The results were averaged over 5 runs of DP-NF-VI; the SD values over the 5
runs in the parentheses.

As a comparison, we also performed VI for $\bm{\beta}$ in the nonlinear model
based on privacy-preserving synthetic data generated using DP-NF for density
estimation (Algorithm 1). For this task, we use a MAF architecture consisting
of one MADE (1 hidden layer, 10 neurons) and one batch norm layer. We also
used RMSprop with a batch size of 500, a learning rate of 0.01, and an
exponential learning rate scheduler with a decay factor equal to 0.999. The
total number of iterations was set to 8,000 and the gradient clipping constant
$C$ to 10.0.

The results are presented in Table 7. In general, regardless of $\mu$, the
posterior estimates of $\bm{\beta}$ from NF-VI based on privacy-preserving
synthetic data show superior accuracy and are more similar to the results from
non-private inference than those obtained directly from DP-NF-VI (Table 6). In
addition, the SD values in the parentheses across multiple sets of synthetic
data are relatively small, while the posterior correlations $\bm{\beta}$ are
reasonably close to their original non-private values.

Table 7: Privacy-preserving VI based on synthetic data generated via DP-NF for
density estimation (Algorithm 1).

parameter |  original nonlinear model |  original NF |  ${\mu=6.68}$ ${\sigma=1.30}$ |  ${\mu=1.12}$ ${\sigma=6.68}$ |  ${\mu=0.50}$ ${\sigma=14.88}$ |  ${\mu=0.27}$ ${\sigma=27.82}$
---|---|---|---|---|---|---
Posterior Mean |
$\beta_{0}$ ( 0.2) |  0.1581 |  0.1593 ( 0.0020) |  0.1632 ( 0.0146) |  0.1062 ( 0.0505) |  0.0866 ( 0.0127) |  0.0448 ( 0.0600)
$\beta_{1}$ ( 1.0) |  1.0022 |  1.0016 ( 0.0015) |  0.9891 ( 0.0060) |  0.9943 ( 0.0060) |  0.9917 ( 0.0097) |  0.9775 ( 0.0126)
$\beta_{2}$ ( 0.8) |  0.8262 |  0.8265 ( 0.0013) |  0.8235 ( 0.0078) |  0.8357 ( 0.0286) |  0.8246 ( 0.0230) |  0.8294 ( 0.0515)
$\beta_{3}$ (-1.2) | -1.1909 | -1.1920 ( 0.0008) | -1.1807 ( 0.0081) | -1.1630 ( 0.0251) | -1.1406 ( 0.0138) | -1.1034 ( 0.0225)
$\beta_{4}$ ( 0.6) |  0.6088 |  0.6092 ( 0.0006) |  0.6011 ( 0.0230) |  0.6261 ( 0.0237) |  0.6197 ( 0.0145) |  0.6214 ( 0.0233)
Posterior SD |
$\beta_{0}$ |  0.0265 |  0.0255 ( 0.0011) |  0.0280 ( 0.0004) |  0.0279 ( 0.0003) |  0.0277 ( 0.0006) |  0.0274 ( 0.0004)
$\beta_{1}$ |  0.0036 |  0.0036 ( 0.0001) |  0.0040 ( 0.0000) |  0.0040 ( 0.0000) |  0.0039 ( 0.0001) |  0.0038 ( 0.0001)
$\beta_{2}$ |  0.0179 |  0.0174 ( 0.0008) |  0.0201 ( 0.0003) |  0.0210 ( 0.0005) |  0.0207 ( 0.0013) |  0.0203 ( 0.0017)
$\beta_{3}$ |  0.0151 |  0.0151 ( 0.0004) |  0.0159 ( 0.0003) |  0.0153 ( 0.0002) |  0.0144 ( 0.0007) |  0.0142 ( 0.0006)
$\beta_{4}$ |  0.0192 |  0.0191 ( 0.0006) |  0.0208 ( 0.0002) |  0.0229 ( 0.0003) |  0.0247 ( 0.0020) |  0.0250 ( 0.0020)
Posterior Correlation |
$\beta_{0},\beta_{1}$ | -0.1200 | -0.0319 ( 0.0499) | -0.0590 ( 0.0262) | -0.0568 ( 0.0130) | -0.0647 ( 0.0338) | -0.0359 ( 0.0185)
$\beta_{0},\beta_{2}$ | -0.4340 | -0.3415 ( 0.1899) | -0.4184 ( 0.0159) | -0.4155 ( 0.0378) | -0.4588 ( 0.0606) | -0.4774 ( 0.0296)
$\beta_{0},\beta_{3}$ | -0.7370 | -0.7550 ( 0.0127) | -0.7293 ( 0.0050) | -0.6919 ( 0.0092) | -0.6951 ( 0.0200) | -0.7049 ( 0.0326)
$\beta_{0},\beta_{4}$ | -0.4880 | -0.4883 ( 0.0356) | -0.5206 ( 0.0103) | -0.5418 ( 0.0122) | -0.5168 ( 0.0315) | -0.4835 ( 0.0388)
$\beta_{1},\beta_{2}$ | -0.0030 | -0.0082 ( 0.0417) | -0.0289 ( 0.0211) | -0.0244 ( 0.0196) | -0.0315 ( 0.0297) | -0.0774 ( 0.0207)
$\beta_{1},\beta_{3}$ | -0.0160 | -0.0892 ( 0.0264) | -0.0723 ( 0.0285) | -0.0674 ( 0.0176) | -0.0607 ( 0.0249) | -0.0557 ( 0.0267)
$\beta_{1},\beta_{4}$ | -0.0080 | -0.0680 ( 0.0252) | -0.0269 ( 0.0204) | -0.0318 ( 0.0166) | -0.0229 ( 0.0198) | -0.0428 ( 0.0171)
$\beta_{2},\beta_{3}$ |  0.0040 | -0.0538 ( 0.1526) | -0.0202 ( 0.0194) | -0.0409 ( 0.0257) |  0.1278 ( 0.2343) |  0.2144 ( 0.2168)
$\beta_{2},\beta_{4}$ | -0.0020 | -0.0836 ( 0.0936) | -0.0346 ( 0.0156) | -0.0518 ( 0.0347) | -0.1289 ( 0.0761) | -0.1792 ( 0.0979)
$\beta_{3},\beta_{4}$ | -0.0020 |  0.0318 ( 0.0443) |  0.0484 ( 0.0110) |  0.0316 ( 0.0170) | -0.0257 ( 0.0677) | -0.0676 ( 0.0510)
The results were averaged over 5 sets of synthetic data generated via DP-NF
for density estimation; the SD values
in the parentheses are computed over the 5 sets.

###  4.3 Further Utility Analysis of DP-NF-VI

The unsatisfactory results on the posterior correlations among the parameters,
as displayed in Table 6, are not accidental. We tested the DP-NF-VI algorithm
in multiple experiments on different datasets, and similar results were
observed. We conjecture that this might be due to the usage of the free energy
bound as the loss function in DP-NF-VI. Minimizing the free energy bound in
NF-VI is equivalent to minimizing the KL divergence between the true posterior
and its variational approximation. As a result of noise injection and gradient
clipping in differentially private gradient-based optimization to achieve DP
guarantees, the posterior variances of the parameters are expected to increase
in the privacy-preserving variational distribution relative to those in the
non-private target distribution. Thus, the DP-NF-VI procedure looks for
densities in the variational family that are the most similar to the true
posterior measured by the KL divergence but have larger variances. This may
lead to an optimal variational density with a different correlation structure
than the original.

To further demonstrate this phenomenon and test our conjecture, we run the
following experiment. Despite the simplicity of the experiment, the results
are interesting and insightful. Suppose the _target_ distribution is a
bivariate Gaussian distribution

$p=\mathcal{N}(\bm{0},\bm{\Sigma}_{0}),\,\,\text{with}\,\,\bm{\Sigma}_{0}=\sigma_{0}^{2}\,\left[(1-\rho_{0})\bm{I}_{2\times
2}+\rho_{0}\bm{1}_{2\times 2}\right].$

We aim to approximate the distribution by another bivariate Gaussian
distribution

$q=\mathcal{N}(\bm{0},\Sigma),\,\,\text{with}\,\,\bm{\Sigma}=\sigma^{2}\,\left[(1-\rho)\,\bm{I}_{2\times
2}+\rho\bm{1}_{2\times 2}\right],$

which can be regarded as the _private_ distribution, expected to have a larger
marginal variance in each dimension compared to those in $p$; i.e.,
$\sigma>\sigma_{0}$. We examine the following four dissimilarity (or
divergence) metrics between $p$ and $q$

$L(p,q)=\begin{cases}\mbox{KL divergence}&D(q\|p),\\\ \mbox{reversed KL
divergence}&D(p\|q),\\\ \mbox{$\ell_{1}$
distance}&\|q-p\|_{1}=\int_{\mathbb{R}^{2}}|p(\bm{x})-q(\bm{x})|\,d\bm{x},\\\
\mbox{Wasserstein
distance}&W_{\alpha}(p,q)=(\inf_{\gamma\in\Gamma(p,q)}\mathbb{E}_{\gamma}|p-q|^{\alpha})^{1/\alpha}.\end{cases}.$
55footnotetext: Equivalent to twice the total variation norm
$\|q-p\|_{TV}=\sup_{A\subset\mathbb{R}^{2}}\,|p(A)-q(A)|$

In this experiment, the Wasserstein distance is computed as

$W_{2}(p,q)=\text{trace}(\bm{\Sigma}_{0}+\bm{\Sigma}-2[\bm{\Sigma}^{1/2}\,\bm{\Sigma}_{0}\,\bm{\Sigma}^{1/2}]^{1/2}).$

We computed the dissimilarity between $p$ and $q$ (loss function) for each of
the 4 metrics above as a function of $\rho$ and $\sigma$ and generated the
corresponding contour plots in Figure 4, for three pairs
$(\rho_{0},\sigma_{0})$ at $(0,1),(1/2,1)$ and $(1/2,4)$, respectively. We
then identified the value of $\rho$ that minimizes the dissimilarity given an
inflated standard deviation $\sigma=\widehat{\sigma}$; that is,

$\rho^{*}(\widehat{\sigma})={\arg\min}_{\rho}L(\rho_{0},\sigma_{0},\rho,\widehat{\sigma}),$

The results of the experiment in Figure 4 suggest that the correlation $\rho$
of the private distribution with _inflated_ variance ($\sigma>\sigma_{0}$)
that optimally approximates the target distribution (in the sense of minimal
dissimilarity) depends, in general, on both $\rho_{0}$ and the selected
dissimilarity metric.

When $\rho_{0}=0$, the optimal private approximation determined through both
the KL divergence and the Wasserstein distance correctly matches the target
correlation, i.e., $\rho=0$; if the reverse KL divergence or the $\ell_{1}$
distance is used, the optimal private approximation is not even unique, until
its marginal variance $\sigma$ is sufficiently close or smaller than
$\sigma_{0}$.

When $\rho_{0}$ is not 0, the optimal private approximation with inflated
variance is characterized by a correlation $\rho$ that rapidly tends to 1.0 as
the inflation ratio $\sigma/\sigma_{0}$ increases. This behavior is observed
in the cases of the KL divergence, the reverse KL divergence, and the
$\ell_{1}$ distance. However, the Wasserstein distance is still able to
produce a private approximation with $\rho=\rho_{0}$ independently of the
inflated variance $\sigma$.

In conclusion, the Wasserstein distance appears to be the best metric to use
if one aims to preserve the correlation structure among the estimated
parameters in private variational distributions, even with inflated marginal
variance resulting from noise injection to achieve privacy guarantees.

Figure 4: KL Divergence, reversed KL divergence, $\ell_{1}$ and Wasserstain
distance between _target_ distribution $p=\mathcal{N}(\bm{0},\Sigma_{0})$ and
its _private_ approximation $q=\mathcal{N}(\bm{0},\Sigma)$. The red line in
contour plots is the trajectory of
$\rho^{*}(\widehat{\sigma})={\arg\min}_{\rho}\,L(\rho_{0},\sigma_{0},\rho,\widehat{\sigma})$
that minimizes loss function $L$ given $\widehat{\sigma}$.

##  5 Discussion

We examined the feasibility and utility of generating synthetic data via
privacy-preserving NF for density estimation, with an application to an EHR
dataset. Overall, the results suggest that the downstream learning
(prediction) and inferential procedure (VI) based on the privacy-preserving
synthetic data can yield results of satisfactory and acceptable utility at
practically reasonable choices of privacy loss parameters for the dataset.
Though more empirical studies will be needed, this study on a real dataset
with a relatively small sample size and a relatively large number of
attributes of mixed types and missing values, suggests generating synthetic
data via privacy-preserving NF for density estimation may be a practically
feasible option for releasing and sharing sensitive information.

We also examined the feasibility of directly sanitizing the NF-VI procedure to
generate privacy-preserving posterior inference. We show that the free energy
bound as the loss function for NF-VI may yield variational distribution
deviating significantly from the target distribution. This implies that
alternative loss functions may better preserve the information on certain
aspects of the target distribution, e.g., its correlation structure. In this
context, we plan to further explore how the use of different distance metrics
affects the performance of algorithms for DP-NF-VI, as a way to develop more
effective strategies leveraging, for example, Wasserstein distance-based loss
formulations.

The code for both the density estimation and VI through DP-NF is available on
Github at https://github.com/cedricwangyu/DPNF.

## Acknowledgements

This work is supported by an NSF Big Data Science & Engineering grant #1918692
(PI DES), an NSF CAREER grant #1942662 (PI DES), an NSF OAC CDS&E grant
#2104831 (Notre Dame PI DES) and used computational resources provided through
the Center for Research Computing at the University of Notre Dame.

## Appendix

A. Compliance With Ethical Standards

The EHR dataset utilized in this work contains data from external studies that
involved human participants. The medical procedures that occurred in these
studies were performed in accordance with both the ethical standards of the
institutional and national research committee and with the 1964 Helsinki
declaration and its later amendments. This study is classified as research not
involving human subjects and was approved on June 13th, 2019, by the Office of
Research Compliance and Institutional Review Board at the University of Notre
Dame under IRB#19-05-5371.

B. Acronyms in CVSim-6 model

Table 8: Acronyms in CVSim-6 model

Flows
---
$q_{li}$ | Left ventricular inflow | $q_{lo}$ | Left ventricular outflow
$q_{ri}$ | Right ventricular inflow | $q_{a}$ | Systemic aortic flow
$q_{ro}$ | Right ventricular outflow | $q_{pv}$ | Pulmonary venous flow
Pressures
$P_{pv}$ | Pulmonary venous pressure | $P_{l}$ | Left ventricular pressure
$P_{a}$ | Systemic arterial pressure | $P_{v}$ | Systemic venous pressure
$P_{r}$ | Right ventricular pressure | $P_{pa}$ | Pulmonary arterial pressure
$P_{th}$ | Thransthoracic pressure | $P_{r,dias}$ | Right ventricular diastolic pressure
$P_{r,sys}$ | Right ventricular systolic pressure | $P_{l,dias}$ | Left ventricular diastolic pressure
$P_{l,sys}$ | Left ventricular systolic pressure |  |
Resistances
$R_{a}$ | Systemic arterial resistance | $R_{v}=R_{ri}$ | Systemic venous resistance
$R_{lo}$ | Aortic valve resistance | $R_{v}$ | Systemic venous resistance
$R_{ro}$ | Pulmonary valve resistance | $R_{pv}=R_{li}$ | Pulmonary venous resistance
Capacitances
$C_{l}(t)$ | Left ventricular capacitance | $C_{r}(t)$ | Right ventricular capacitance
$C_{a}$ | Systemic arterial capacitance | $C_{v}$ | Systemic venous capacitance
$C_{pa}$ | Pulmonary arterial capacitance | $C_{pv}$ | Pulmonary venous capacitance
$C_{l,dias}$ | Diastolic left ventricular capacitance | $C_{l,sys}$ | Systolic left ventricular capacitance
$C_{r,dias}$ | Diastolic right ventricular capacitance | $C_{r,sys}$ | Systolic
Times
$T_{tot}$ | Total heart cycle time | $T_{dias}$ | Duration of diastolic phase
$T_{sys}$ | Duration of systolic phase |  |
Volumes
$V_{tot}$ | Total blood volume | $V_{0,tot}$ | Unstressed blood volume

C. Hyper-parameters

We list the hyper-parameters used in all of our experiments in Table 9. EHR
DP-DE indicates the differentially private density estimation in the EHR data
(see Sec 3.2); EHR VI means VI with KL divergence in the privacy-preserving
synthetic EHR data (see Sec 3.4.2); Reg DP-VI represents differentially
private VI in the simulated data via the nonlinear regression model (see Sec
4.2); Reg DP-DE means differentially private density estimation of the
simulated data (see Sec 4.2); Reg VI means VI with KL divergence in the
privacy-preserving simulated data l (See Sec 4.2). Except for the surrogate
model (a fully connected neural network) for the CVSim-6 model, where the
activation function tanh was used, ReLU was the activation function in all
other NF and neural networks.

Table 9: Hyper-parameters used in all the experiments

parameter | EHR DP-DE | EHR VI | Reg DP-VI | Reg DP-DE | Reg VI
---|---|---|---|---|---
block No. | 15 | 5 | 1 | 18 | 1
hidden No. | 1 | 1 | 1 | 1 | 1
hidden size | 200 | 100 | 10 | 100 | 10
input size | 19 | 2 | 5 | 9 | 5
iter No. | 8000 | 15000 | 8000 | 8000 | 8000
batch size | 100 | 500 | 1000 | 100 | 1000
optimizer | SGD | RMSProp | RMSProp | RMSProp | RMSProp
learn rate | 0.00002 | 0.01 | 0.01 | 0.002 | 0.01
scheduler | - | Exp | Exp | Exp | Exp
decay factor | - | 0.9999 | 0.999 | 0.9995 | 0.999
clipping | 10 | - | 10 | 5 | -
poisson rate | 50% | - | 8.33% | 8.33% | -

## References

  * Abadi et al. (2016) M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , pages 308–318, 2016.
  * Aggarwal and Yu (2008) C. C. Aggarwal and P. S. Yu.  A general survey of privacy-preserving data mining models and algorithms.  In _Privacy-preserving data mining_ , pages 11–52. Springer, 2008\.
  * Alda and Rubinstein (2017) F. Alda and B. I. Rubinstein.  The Bernstein mechanism: Function release under differential privacy.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Arts et al. (2005) T. Arts, T. Delhaas, P. Bovendeerd, X. Verbeek, and F. W. Prinzen.  Adaptation to mechanical load determines shape and properties of heart and circulation: the CircAdapt model.  _American Journal of Physiology-Heart and Circulatory Physiology_ , 288(4):H1943–H1954, 2005.
  * Backes et al. (2016) M. Backes, P. Berrang, M. Humbert, and P. Manoharan.  Membership privacy in microrna-based studies.  In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , pages 319–330, 2016.
  * Beaulieu-Jones et al. (2019) B. K. Beaulieu-Jones, Z. S. Wu, C. Williams, R. Lee, S. P. Bhavnani, J. B. Byrd, and C. S. Greene.  Privacy-preserving generative deep neural networks support clinical data sharing.  _Circulation: Cardiovascular Quality and Outcomes_ , 12(7):e005122, 2019.
  * Benaim et al. (2020) A. R. Benaim, R. Almog, Y. Gorelik, I. Hochberg, L. Nassar, T. Mashiach, M. Khamaisi, Y. Lurie, Z. S. Azzam, J. Khoury, et al.  Analyzing medical research results based on synthetic data and their relation to real data results: systematic comparison from five observational studies.  _JMIR medical informatics_ , 8(2):e16492, 2020\.
  * Bowen and Liu (2020) C. M. Bowen and F. Liu.  Comparative study of differentially private data synthesis methods.  _Statistical Science_ , 35(2):280–307, 2020.
  * Breiman (2001) L. Breiman.  Random forests.  _Machine learning_ , 45(1):5–32, 2001.
  * Bu et al. (2020) Z. Bu, J. Dong, Q. Long, and W. J. Su.  Deep learning with Gaussian differential privacy.  _Harvard data science review_ , 2020(23), 2020.
  * Bun and Steinke (2016) M. Bun and T. Steinke.  Concentrated differential privacy: Simplifications, extensions, and lower bounds.  In _Theory of Cryptography Conference_ , pages 635–658. Springer, 2016.
  * Bun et al. (2018) M. Bun, C. Dwork, G. N. Rothblum, and T. Steinke.  Composable and versatile privacy via truncated CDP.  In _Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing_ , pages 74–86, 2018.
  * Butucea et al. (2020) C. Butucea, A. Dubois, M. Kroll, and A. Saumard.  Local differential privacy: Elbow effect in optimal density estimation and adaptation over Besov ellipsoids.  _Bernoulli_ , 26(3):1727–1764, 2020.
  * Chen et al. (2018) Q. Chen, C. Xiang, M. Xue, B. Li, N. Borisov, D. Kaarfar, and H. Zhu.  Differentially private data generative models.  _arXiv preprint arXiv:1812.02274_ , 2018.
  * Cortes and Vapnik (1995) C. Cortes and V. Vapnik.  Support-vector networks.  _Machine learning_ , 20(3):273–297, 1995.
  * Dash et al. (2019) S. Dash, R. Dutta, I. Guyon, A. Pavao, K. P. Bennett, et al.  Privacy preserving synthetic health data.  In _ESANN 2019-European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning_ , 2019.
  * Davis (1991) T. L. Davis.  _Teaching physiology through interactive simulation of hemodynamics_.  PhD thesis, Massachusetts Institute of Technology, 1991.
  * Dinh et al. (2016) L. Dinh, J. Sohl-Dickstein, and S. Bengio.  Density estimation using real NVP.  _arXiv preprint arXiv:1605.08803_ , 2016.
  * Dong et al. (2019) J. Dong, A. Roth, and W. J. Su.  Gaussian differential privacy.  _arXiv preprint arXiv:1905.02383_ , 2019.
  * Dwork and Rothblum (2016) C. Dwork and G. N. Rothblum.  Concentrated differential privacy.  _arXiv:1603.01887v2_ , 2016.
  * Dwork et al. (2006a) C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor.  Our data, ourselves: privacy via distributed noise generation.  In _Advances in Cyptology: Proceedings of EUROCRYPT_ , pages 485–503. Springer Berlin Heidelberg, 2006a.
  * Dwork et al. (2006b) C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor.  Our data, ourselves: Privacy via distributed noise generation.  In _Annual International Conference on the Theory and Applications of Cryptographic Techniques_ , pages 486–503. Springer, 2006b.
  * Dwork et al. (2006c) C. Dwork, F. McSherry, K. Nissim, and A. Smith.  Calibrating noise to sensitivity in private data analysis.  In _Theory of cryptography conference_ , pages 265–284. Springer, 2006c.
  * Dwork et al. (2017) C. Dwork, A. Smith, T. Steinke, and J. Ullman.  Exposed! a survey of attacks on private data.  _Annu. Rev. Stat. Appl_ , 4(1):61–84, 2017.
  * Fowlkes and Mallows (1983) E. B. Fowlkes and C. L. Mallows.  A method for comparing two hierarchical clusterings.  _Journal of the American statistical association_ , 78(383):553–569, 1983.
  * Germain et al. (2015) M. Germain, K. Gregor, I. Murray, and H. Larochelle.  Made: Masked autoencoder for distribution estimation.  In _International Conference on Machine Learning_ , pages 881–889. PMLR, 2015.
  * Goldberger et al. (2000) A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley.  PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals.  _circulation_ , 101(23):e215–e220, 2000.
  * Goodfellow et al. (2014) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gordon et al. (1983) E. P. Gordon, I. Schnittger, P. J. Fitzgerald, P. Williams, and R. L. Popp.  Reproducibility of left ventricular volumes by two-dimensional echocardiography.  _Journal of the American College of Cardiology_ , 2(3):506–513, 1983.
  * Hall et al. (2013) R. Hall, A. Rinaldo, and L. Wasserman.  Differential privacy for functions and functional data.  _The Journal of Machine Learning Research_ , 14(1):703–727, 2013.
  * Harrod et al. (2021) K. K. Harrod, J. L. Rogers, J. A. Feinstein, A. L. Marsden, and D. E. Schiavazzi.  Predictive modeling of secondary pulmonary hypertension in left ventricular diastolic dysfunction.  _Frontiers in Physiology_ , 12, 2021.  URL https://www.frontiersin.org/article/10.3389/fphys.2021.666915.
  * Heldt et al. (2010) T. Heldt, R. Mukkamala, G. B. Moody, and R. G. Mark.  CVSim: an open-source cardiovascular simulator for teaching and research.  _The open pacing, electrophysiology & therapy journal_, 3:45, 2010.
  * Homer et al. (2008) N. Homer, S. Szelinger, M. Redmann, D. Duggan, W. Tembe, J. Muehling, J. Pearson, D. Stephan, S. Nelson, and D. Craig.  Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays.  _PLoS Genet_ , 4:e1000167, 2008.
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Jälkö et al. (2016) J. Jälkö, O. Dikmen, and A. Honkela.  Differentially private variational inference for non-conjugate models.  _arXiv preprint arXiv:1610.08749_ , 2016.
  * Jälkö et al. (2022) J. Jälkö, L. Prediger, A. Honkela, and S. Kaski.  Dpvim: Differentially private variational inference improved.  _arXiv preprint arXiv:2210.15961_ , 2022.
  * Johnson et al. (2020) A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, L. A. Celi, and R. Mark.  Mimic-IV.  _PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021)_ , 2020.
  * Jordon et al. (2018) J. Jordon, J. Yoon, and M. Van Der Schaar.  Pate-gan: Generating synthetic data with differential privacy guarantees.  In _International conference on learning representations_ , 2018.
  * Karwa et al. (2015) V. Karwa, D. Kifer, and A. B. Slavković.  Private posterior distributions from variational approximations.  _arXiv preprint arXiv:1511.07896_ , 2015.
  * Kingma and Ba (2014) D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma and Dhariwal (2018) D. P. Kingma and P. Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  _Advances in neural information processing systems_ , 31, 2018.
  * Kingma and Welling (2013) D. P. Kingma and M. Welling.  Auto-encoding variational Bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma et al. (2016) D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling.  Improved variational inference with inverse autoregressive flow.  _Advances in neural information processing systems_ , 29:4743–4751, 2016.
  * Kroll (2021) M. Kroll.  On density estimation at a fixed point under local differential privacy.  _Electronic Journal of Statistics_ , 15(1):1783–1813, 2021.
  * Li et al. (2007) N. Li, T. Li, and S. Venkatasubramanian.  t-closeness: Privacy beyond k-anonymity and l-diversity.  In _2007 IEEE 23rd international conference on data engineering_ , pages 106–115. IEEE, 2007.
  * Liu (2022) F. Liu.  Model-based differentially private data synthesis and statistical inference in multiply synthetic differentially private data.  _Transactions on Data Privacy_ , 15(3):141–175, 2022.
  * Maceira et al. (2006) A. Maceira, S. Prasad, M. Khan, and D. Pennell.  Normalized left ventricular systolic and diastolic function by steady state free precession cardiovascular magnetic resonance.  _Journal of Cardiovascular Magnetic Resonance_ , 8(3):417–426, 2006.
  * Mironov (2017) I. Mironov.  Rényi differential privacy.  In _2017 IEEE 30th Computer Security Foundations Symposium (CSF)_ , pages 263–275. IEEE, 2017.
  * Narayanan and Shmatikov (2008) A. Narayanan and V. Shmatikov.  Robust de anonymization of large sparse datasets.  _IEEE Symposium on Security and Privacy_ , pages 111–125, 2008.
  * Papamakarios et al. (2017) G. Papamakarios, T. Pavlakou, and I. Murray.  Masked autoregressive flow for density estimation.  _Advances in neural information processing systems_ , 30, 2017.
  * Papernot et al. (2016) N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar.  Semi-supervised knowledge transfer for deep learning from private training data.  _arXiv preprint arXiv:1610.05755_ , 2016.
  * Papernot et al. (2018) N. Papernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and Ú. Erlingsson.  Scalable private learning with pate.  _arXiv preprint arXiv:1802.08908_ , 2018.
  * Park et al. (2016) M. Park, J. Foulds, K. Chaudhuri, and M. Welling.  Variational Bayes in private settings (VIPS).  _arXiv preprint arXiv:1611.00340_ , 2016.
  * Pfitzner and Arnrich (2022) B. Pfitzner and B. Arnrich.  Dpd-fvae: Synthetic data generation using federated variational autoencoders with differentially-private decoder.  _arXiv preprint arXiv:2211.11591_ , 2022.
  * Pollard et al. (2018) T. J. Pollard, A. E. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi.  The eICU collaborative research database, a freely available multi-center database for critical care research.  _Scientific data_ , 5(1):1–13, 2018.
  * Rankin et al. (2020) D. Rankin, M. Black, R. Bond, J. Wallace, M. Mulvenna, G. Epelde, et al.  Reliability of supervised machine learning using synthetic data in health care: Model to preserve privacy for data sharing.  _JMIR Medical Informatics_ , 8(7):e18910, 2020\.
  * Rezende and Mohamed (2015) D. Rezende and S. Mohamed.  Variational inference with normalizing flows.  In _International conference on machine learning_ , pages 1530–1538. PMLR, 2015.
  * Rubin (1993) D. B. Rubin.  Statistical disclosure limitation.  _Journal of official Statistics_ , 9(2):461–468, 1993.
  * Samarati and Sweeney (1998) P. Samarati and L. Sweeney.  Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression.  _technical report, SRI International_ , 1998.
  * Sharma et al. (2019) M. Sharma, M. Hutchinson, S. Swaroop, A. Honkela, and R. E. Turner.  Differentially private federated variational inference.  _arXiv preprint arXiv:1911.10563_ , 2019.
  * Shokri et al. (2017) R. Shokri, M. Stronati, C. Song, and V. Shmatikov.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , pages 3–18. IEEE, 2017.
  * Simonneau et al. (2013) G. Simonneau, M. Gatzoulis, I. Adatia, C. Celermajer, D.and Denton, A. Ghofrani, M. Sanchez Gomez, R. Kumar, M. Landzberg, R. Machado, H. Olschewski, I. Robbins, and S. R.  Updated clinical classification of pulmonary hypertension.  _Journal of the American College of Cardiology_ , 62(25 Supplement):D34–D41, 2013.
  * Sweeney (1997) L. Sweeney.  Maintaining patient confidentiality when sharing medical data requires a symbiotic relationship between technology and policy.  _Artificial Intelligence Laboratory, Massachusetts Institute of Technology_ , AIWP-WP344, 1997.
  * Sweeney (2013) L. Sweeney.  Matching Known Patients to Health Records in Washington State Data.  _Social Science Research Network_ , 2013.  id=2289850.
  * the SHARED team (2023) the SHARED team.  The synthetic health and research data (shared) project, 2023.  URL {https://shared.landen.co/}.  accessed on Jan 8, 2023.
  * Tieleman et al. (2012) T. Tieleman, G. Hinton, et al.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ , 4(2):26–31, 2012.
  * Van Buuren and Groothuis-Oudshoorn (2011) S. Van Buuren and K. Groothuis-Oudshoorn.  mice: Multivariate imputation by chained equations in R.  _Journal of statistical software_ , 45:1–67, 2011.
  * Waites and Cummings (2021) C. Waites and R. Cummings.  Differentially private normalizing flows for privacy-preserving density estimation.  _arXiv preprint arXiv:2103.14068_ , 2021.
  * Wasserman and Zhou (2010) L. Wasserman and S. Zhou.  A statistical framework for differential privacy.  _Journal of the American Statistical Association_ , 105(489):375–389, 2010.
  * Xie et al. (2018) L. Xie, K. Lin, S. Wang, F. Wang, and J. Zhou.  Differentially private generative adversarial network.  _arXiv preprint arXiv:1802.06739_ , 2018.
  * Yale et al. (2020) A. Yale, S. Dash, R. Dutta, I. Guyon, A. Pavao, and K. P. Bennett.  Generation and evaluation of privacy preserving synthetic health data.  _Neurocomputing_ , 416:244–255, 2020.
  * Yared et al. (2011) K. Yared, P. Noseworthy, A. Weyman, E. McCabe, M. Picard, and A. Baggish.  Pulmonary artery acceleration time provides an accurate estimate of systolic pulmonary arterial pressure during transthoracic echocardiography.  _Journal of the American Society of Echocardiography_ , 24(6):687–692, 2011.