# Generating Synthetic Mixed-type Longitudinal Electronic Health Records for
Artificial Intelligent Applications

Jin Li†‡, Benjamin J. Cairns‡, Jingsong Li†∗, Tingting Zhu‡
†Zhejiang University, ‡University of Oxford
{jin.li<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding authors.

###### Abstract

The recent availability of electronic health records (EHRs) have provided
enormous opportunities to develop artificial intelligence (AI) algorithms.
However, patient privacy has become a major concern that limits data sharing
across hospital settings and subsequently hinders the advances in AI.
Synthetic data, which benefits from the development and proliferation of
generative models, has served as a promising substitute for real patient EHR
data. However, the current generative models are limited as they only generate
single type of clinical data, i.e., either continuous-valued or discrete-
valued. In this paper, we propose a generative adversarial network (GAN)
entitled EHR-M-GAN which synthesizes mixed-type timeseries EHR data. EHR-M-GAN
is capable of capturing the multidimensional, heterogeneous, and correlated
temporal dynamics in patient trajectories. We have validated EHR-M-GAN on
three publicly-available intensive care unit databases with records from a
total of 141,488 unique patients, and performed privacy risk evaluation of the
proposed model. EHR-M-GAN has demonstrated its superiority in performance over
state-of-the-art benchmarks for synthesizing clinical timeseries with high
fidelity. Notably, prediction models for outcomes of intensive care performed
significantly better when training data was augmented with the addition of
EHR-M-GAN-generated timeseries. EHR-M-GAN may have use in developing AI
algorithms in resource-limited settings, lowering the barrier for data
acquisition while preserving patient privacy.

##  1 Introduction

The past decade has witnessed ground-breaking advancements been made in
computational health, owing to the explosion of medical data, such as
electronic health records (EHRs) (Artzi et al.,, 2020; Raket et al.,, 2020;
Menger et al.,, 2019). The secondary uses of EHRs give rise to research in a
wide range of varieties, especially machine learning (ML)-based digital health
solutions for improving the delivery of care (Wilkinson et al.,, 2020; Watson
et al.,, 2019; Futoma et al.,, 2020; Esteva et al.,, 2021; Rajkomar et al.,,
2019). However, in practice, the benefits of data-driven research are limited
to healthcare organizations (HCOs) who possess the data (Wirth et al.,, 2021;
Dinov,, 2016). Due to concerns about patient privacy, HCO stakeholders are
reluctant to share patient data (Miotto et al.,, 2018; Kim et al.,, 2021;
Simon et al.,, 2019). Access to clinical data is often restricted, or can be
prohibitively expensive to obtain, meaning that ML in biomedical research lags
behind other areas in AI.

To accelerate the progress of developing AI methods in medicine, one promising
alternative is for the data holder to create synthetic yet realistic data
(Jordon et al.,, 2018; Frid-Adar et al.,, 2018). By avoiding “one-to-one”
mapping to the genuine data compared with data anonymization, synthetic data
offers a solution to circumvent the issue of privacy, while the correlations
in the original data distributions are preserved for downstream AI
applications. There have been successes in the literature using synthetic data
to improve AI models where otherwise not possible due to limited availability
of resources (Jordon et al.,, 2020; Chen et al.,, 2021; Tucker et al.,, 2020;
El Emam et al.,, 2021). For example, large-scale data sharing programs have
been demanded for advancing studies related to COVID-19, such as in National
COVID Cohort Collaborative (N3C) (N3C.,, 2021), and Clinical Practice Research
Datalink (CPRD) database in the UK (CPRD.,, 2021).

The availability of data synthesis facilitates the development of data-driven
clinical models in research community for healthcare. However, some of the
methods for generating synthetic patient EHRs rely heavily upon hand-crafted
rules or clinical expertise (McLachlan et al.,, 2016; Lombardo & Moniz,,
2008). For example, McLachlan et al. (McLachlan et al.,, 2016) proposed to
generate synthetic EHRs by formalizing the clinical practice guidelines into a
state transition machine (STM), with the whole process being domain knowledge-
intensive. In addition, since these methods can only handle data with low
dimensionality (and are typically disease-specific), the general utility of
these methods is severely limited. Recent advances in generative adversarial
networks (GANs) (Goodfellow et al.,, 2014) and their variants offer more
efficacious means to generate multidimensional, high-fidelity data with
complex correlations for many different applications (Kearney et al.,, 2020;
Yang et al.,, 2018; Marouf et al.,, 2020). Furthermore, owing to their
flexible tuning techniques, GAN models can better generate samples of
anomalous or sparse events, mitigating the issue of the rarity which
potentially leads to bias during training downstream ML models (Zhou et al.,,
2020; Schlegl et al.,, 2019).

EHRs are sets of digital patient-centered records collected over time, thereby
recording patient health status and care trajectories during the
hospitalization. Recent research has aimed to use this information to generate
clinical timeseries which capture the character of these trajectories (Esteban
et al.,, 2017; Lee et al.,, 2020; Zhang et al.,, 2021; Kiyasseh et al.,,
2020). In this study, we focus on generating timeseries in critical care,
specifically in intensive care units (ICUs) where patient information are
fully digitized. In the high-paced ICUs, various measurements and complex
signals are recorded as mixed-type data for critically ill patients undergoing
intensive monitoring (Sanchez-Pinto et al.,, 2018; Shillan et al.,, 2019) (see
Fig. 1 for data extraction). Typically, two types of data are observed: (1)
the continuous-valued physiological signals, such as heart rate (HR), oxygen
saturation (SpO2), and measurements from blood gas tests; (2) the discretized-
valued medical intervention data that change over time, including the usage of
therapeutic devices or intravenous medications. Also, these mixed-type
clinical timeseries are highly correlated (Yu et al.,, 2019; Ghassemi et al.,,
2017; Wang et al.,, 2019). For example, the medications prescribed to patients
rely heavily on measurements of their physiological status in the ICU.
Meanwhile, the efficacy of the medical treatments, in turn, can critically
affect the patient’s physiological condition.

While mixed-type data can provide a rich environment for developing ML
algorithms for assisting decision-making in critical care, they also impose
complexity in creating generative models. As a result, most recent research
has investigated how to synthesize the longitudinal data of different types
separately, i.e., with a focus only on continuous or only on discrete
timeseries (Yoon et al.,, 2019; Lee et al.,, 2020; Yu et al.,, 2017). GANs
have not previously been applied to address the issue in the mixed-data
setting. Therefore for the first time, we propose a GAN for synthesizing
mixed-type longitudinal EHR data (denoted as EHR-M-GAN thereafter). Patient
trajectories with high-dimensionality and heterogeneous data type (both
continuous-valued and discrete-valued timeseries) are generated while the
underlying temporal dependencies are captured.

The main contributions of our work are as follows:

  1. 1.

A novel GAN model entitled EHR-M-GAN is proposed for generating mixed-type
multivariate EHR timeseries with high fidelity. EHR-M-GAN first maps data from
different observational spaces into a reversible, lower-dimensional, shared
latent space through a dual variational autoencoder (dual-VAE). Then, a
sequentially coupled generator that is built upon a coupled recurrent network
(CRN) is employed to capture the correlated temporal dynamics of the mixed-
type data. In addition, a conditional version of our model — EHR-M-
GAN${}_{\texttt{cond}}$ — is also implemented, which is capable of
synthesizing condition-specific EHR patient data, such as those result in ICU
mortality or hospital readmission. The code of our proposed work is publicly
available on GitHub111https://github.com/jli0117/ehrMGAN.

  2. 2.

Evaluations are performed based on three publicly available ICU datasets:
MIMIC-III (Johnson et al.,, 2016), eICU (Pollard et al.,, 2018) and HiRID
(Yèche et al.,, 2021) from a total of 141,488 patients. Standardized
preprocessing pipelines are applied for the three ICU datasets to provide
generalizable machine learning benchmarks. The code for the end-to-end
preprocessing pipelines is also available on
GitHub222https://github.com/jli0117/preprocessing_physionet.

  3. 3.

Our EHR-M-GAN outperforms the state-of-the-art benchmarks on a diverse
spectrum of evaluation metrics. When compared to real EHR data, both
qualitative and quantitative metrics are used to assess the representativeness
of the mixed-type data and their inter-dependencies. We further demonstrate
the advantages offered by EHR-M-GAN in augmenting clinical timeseries for
downstream tasks under various clinical scenarios.

  4. 4.

In the evaluation of privacy risks, we perform an empirical analysis on EHR-M-
GAN based on membership inference attack (Shokri et al.,, 2017). We then
further evaluate the performance of EHR-M-GAN under the framework of
differential privacy for its application in downstream task (Dwork,, 2006).

##  2 Methods

In this section, we first formulate the problem based on the mixed-type
temporal EHR data and its corresponding mathematical notation. Then the
proposed EHR-M-GAN model is introduced in details.

###  2.1 Problem formulation

The longitudinal patient EHR dataset is denoted as
$\smash{\mathscr{D}=\\{(\mathbf{x}_{i,1:T_{i}})\\}_{i=1}^{N}}$, with each
record (e.g., individual patient) being indexed by
$\smash{i\in\\{1,2,...N\\}}$. Here we consider the i-th instance tuple
$\smash{\mathbf{x}_{i,1:T_{i}}=\\{\mathbf{x}_{i,1:T_{i}}^{\mathcal{C}},\mathbf{x}_{i,1:T_{i}}^{\mathcal{D}}\\}}$
consists of two components (i.e., two types of data). Let
$\smash{\mathbf{x}_{i,1:T_{i}}^{\mathcal{C}}}\in\mathbb{R}^{|J|}$ denote the
$|J|$-dimensional continuous-valued timeseries, such as physiological signals
from real-time bedside monitors. And
$\smash{\mathbf{x}_{i,1:T_{i}}^{\mathcal{D}}}\in\mathbb{Z}^{|K|}$ denotes the
$|K|$-dimensional discrete-valued timeseries, such as life-support
interventions with the categorical value indicate its status (presence or
absence).

###  2.2 Proposed model

The preliminary for GANs and conditional GANs can be found in Supplementary
materials (See Section S.1.A). We factorize EHR-M-GAN into two key components
(see Network architecture in Fig. 1): (1) a dual-VAE framework for learning
the shared latent space representations; (2) an RNN-based sequentially coupled
generator and its corresponding sequence discriminators. As shown in Fig. 1,
during the pretrain stage, two types of temporal trajectories are first
jointly mapped into a shared latent space (first component). Then, the
sequentially coupled generator in EHR-M-GAN (second component) produces the
synthetic latent representations, which further can be recovered into features
in the observational space by the pretrained decoders in the dual-VAE.
Finally, the adversarial loss is provided based on discriminative results and
backpropagated to update the network. The following sections discuss them in
turn.

Figure 1: Overall schematics. a. Data extraction. Electronic health records
(EHRs) data are routinely collected for patients in intensive care units
(ICUs). Intensively monitored vital signals and laboratory measurements are
recorded as continuous-valued timeseries, while the presence or absence of
medical interventions is collected as discrete-valued timeseries during the
ICU admission. b. Network architecture. EHR-M-GAN contains two key components.
Dual-VAE is first pretrained for mapping heterogeneous data into shared latent
representations (Step 1). Multiple objective loss constraints are used to
bridge the domain gap, including ELBO loss, matching loss, contrastive loss,
and semantic loss (for EHR-M-GAN${}_{\texttt{cond}}$ only). Then, a coupled
recurrent network (CRN) is established as the generator based on the parallel
bilateral LSTM block (Step 2). After decoding the generated representations
provided by CRN (Step 3), the adversarial loss is derived from the
discriminators and backpropagated to update the network (Step 4).

####  2.2.1 Dual-VAE pretraining for shared latent space representations

One prior of successfully training EHR-M-GAN to generate reversible latent
codes is to meet the assumption that both
$\smash{\mathbf{x}_{i,1:T_{i}}^{\mathcal{C}}}$ and
$\smash{\mathbf{x}_{i,1:T_{i}}^{\mathcal{D}}}$ can be encoded into the same
latent space $\smash{\mathcal{H}^{\mathcal{S}}\subset\mathbb{R}^{|S|}}$ for
the patient indexed with $i$, where $|S|$ denotes its spatial dimension. For
the sake of simplicity, the subscripts $i$ are omitted throughout most of the
paper. To achieve this, we propose to use a dual-VAE framework to encode both
continuous and discrete multivariate timeseries into dense representations
within $\smash{\mathcal{H}^{\mathcal{S}}}$ based on multiple constraints. We
found VAE preferable to vanilla autoencoder in our case, considering that (1)
the KL regularization in VAE strengthens the learning of the compressed latent
representations, which further narrow the domain gap for mixed-type features
(Wan et al.,, 2020); (2) VAE can be easily extended to the conditional
learning scenario in EHR-M-GANcond.

Fig. S1 (see Section S.1.B in Supplementary materials) diagrams the details of
the proposed dual-VAE framework for learning the shared latent
representations. We start with training two encoders, i.e.,
$\smash{\mathit{Enc}^{\mathcal{C}}}$:
$\smash{\phi_{\mathcal{T}\times\mathcal{X}^{\mathcal{C}}}\to\phi_{\mathcal{T}\times\mathcal{H}^{\mathcal{S}}}}$
and $\smash{\mathit{Enc}^{\mathcal{D}}}$:
$\smash{\phi_{\mathcal{T}\times\mathcal{X}^{\mathcal{D}}}\to\phi_{\mathcal{T}\times\mathcal{H}^{\mathcal{S}}}}$,
with the embedding functions:

$\mathbf{z}_{1:T}^{\mathcal{C}}=\mathit{Enc}^{\mathcal{C}}(\mathbf{x}_{1:T}^{\mathcal{C}})\quad\quad\mathbf{z}_{1:T}^{\mathcal{D}}=\mathit{Enc}^{\mathcal{D}}(\mathbf{x}_{1:T}^{\mathcal{D}})$
(1)

After passing data from $\smash{\mathcal{X}^{\mathcal{C}}}$ and
$\smash{\mathcal{X}^{\mathcal{D}}}$ through two encoders, a pair of embedding
vectors
$\smash{(\mathbf{z}^{\mathcal{C}}_{1:T},\mathbf{z}^{\mathcal{D}}_{1:T})}$ in
the shared latent space $\smash{\mathcal{H}^{\mathcal{S}}}$ can be obtained.
Then the decoders for both domains
$\smash{\mathit{Dec}^{\mathcal{C}}:\psi_{\mathcal{T}\times\mathcal{H}^{\mathcal{S}}}\to\psi_{\mathcal{T}\times\mathcal{X}^{\mathcal{C}}}}$
and
$\smash{\mathit{Dec}^{\mathcal{D}}:\psi_{\mathcal{T}\times\mathcal{H}^{\mathcal{S}}}\to\psi_{\mathcal{T}\times\mathcal{X}^{\mathcal{D}}}}$
further reconstruct features based on the latent embeddings using mapping
functions that operate in the opposite direction:

$\tilde{\mathbf{x}}_{1:T}^{\mathcal{C}}=\mathit{Dec}^{\mathcal{C}}(\mathbf{z}_{1:T}^{\mathcal{C}})\quad\quad\tilde{\mathbf{x}}_{1:T}^{\mathcal{D}}=\mathit{Dec}^{\mathcal{D}}(\mathbf{z}_{1:T}^{\mathcal{D}})$
(2)

For the conditional extension in EHR-M-GAN${}_{\texttt{cond}}$, both encoders
and decoders in the dual-VAE condition on the auxiliary (one-hot) labels from
$\mathcal{L}$, to make the model better adapted to particular contexts. Also,
to incentivize dual-VAE to better bridge the gap between domains of
continuous-valued and discrete-valued timeseries, we enforce a weight-sharing
constraint (Liu et al.,, 2017; Liu & Tuzel,, 2016) within specific layers of
both the encoders pairs
$\smash{(\mathit{Enc}^{\mathcal{C}},\mathit{Enc}^{\mathcal{D}})}$, and the
decoders pairs
$\smash{(\mathit{Dec}^{\mathcal{C}},\mathit{Dec}^{\mathcal{D}})}$. The weight-
sharing constraint can extract and broadcast high-level concepts of the input
features across domains of mixed-type data. The implementation details can be
found in Supplementary materials (See Section S.1.B).

In the following subsections, we define multiple loss constraints for the
optimization of dual-VAE, including ELBO loss, Matching loss, Contrastive
loss, as well as Semantic loss for EHR-M-GAN${}_{\texttt{cond}}$. Intuitions
and descriptions behind the objectives are discussed in turn.

Evidence Lower Bound (ELBO).

We first incorporate the standard VAE loss, with the optimization objective as
the evidence lower bound (ELBO). VAE holds the assumption of spherical
Gaussian prior for the distribution of latent embeddings, where features can
then be reconstructed by sampling from that space. We leverage the re-
parameterization tricks to enable differentiable stochastic sampling and
network optimization. For encoder and decoder in the dual-VAE for domain
$d\in\\{\mathcal{C},\mathcal{D}\\}$, the objective function is defined as:

$\begin{split}\mathcal{L}^{\mathrm{ELBO}}_{d}=-\mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})}[\log
p_{\psi}&(\mathbf{x}|\mathbf{z})]+\\\
\beta_{\mathrm{KL}}D_{\mathrm{KL}}(q_{\phi}&(\mathbf{z}|\mathbf{x})\|p_{\psi}(\mathbf{z}))\end{split}$
(3)

where $\mathbf{z}\sim\mathit{Enc}(\mathbf{x})\triangleq
q_{\phi}(\mathbf{z}|\mathbf{x}),\tilde{\mathbf{x}}\sim\mathit{Dec}(\mathbf{z})\triangleq
p_{\psi}(\mathbf{x}|\mathbf{z})$, and $D_{\mathrm{KL}}$ is the Kullback-
Leibler divergence. The first term in Eq. (3) is the expected log likelihood
term that penalizes the deviations in reconstructing the inputs, while the
second term of KL-divergence is the regularization imposed over the latent
distribution from its Gaussian prior (normally chosen to be
$\mathcal{N}(\mathbf{0},\boldsymbol{I})$). $\beta_{\mathrm{KL}}$ is the
hyperparameter for balancing the weights between two terms.

Matching loss.

Typically, representations derived from the same patient are assumed to
capture the shared context. Therefore, embedding vectors
$\smash{(\mathbf{z}^{\mathcal{C}}_{i,1:T_{i}},\mathbf{z}^{\mathcal{D}}_{i,1:T_{i}})}$
in the shared latent space projected from the same patient $i$, are supposed
to be positioned in close proximity (See Fig. S1 in Supplementary materials).
Hence, the pairwise matching loss is incorporated to motivate the encoders to
minimize the distance within the corresponding representation pairs. In the
low-dimensional Euclidean space, we optimize the network by using the
following objective:

$\mathcal{L}^{\mathrm{Match}}=\mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}}[\sum_{t\in\mathcal{T}}||\mathbf{z}^{\mathcal{C}}_{t}-\mathbf{z}^{\mathcal{D}}_{t}||^{2}]$
(4)

The pairwise matching loss achieve its optimal when the distance proxy
$\mathcal{L}^{\mathrm{Match}}$ becomes zero.

Contrastive loss.

On the flip side, pairwise reconstruction error (i.e., intra-correlations
within one instance) measured by matching loss neglects the commonalities
present across patients (inter-correlations of data) (Kiyasseh et al.,, 2021).
In order to guarantee sufficient bound for representation learning, we
incorporate contrastive loss as another distance metric to capture the inter-
correlations among the population.

Contrastive learning is a concept that has recently been popularized in self-
supervised learning (SSL) and representation learning, which aims to capture
intrinsic patterns from input data without human annotations. The core of
contrastive learning is to encourage networks to attract positive pairs closer
and repulse negative pairs apart in the latent space. In this study, we
instantiate the contrastive loss via NT-Xent, which is proposed by Chen et al.
in their work SimCLR (Chen et al.,, 2020b). We define the corresponding
auxiliary tasks for calculating the contrastive loss as — whether a set of
representations transformed from the observational space belong to the same
patient. And this leads to the corresponding positive pairs (true) and
negative pairs (false).

For patient data of $N$ records, we can obtain $N$ pairs of latent
representations from the encoders in dual-VAE. For patient indexed with $i$,
$\mathbf{h}_{i}^{\mathcal{C}}$ and $\mathbf{h}_{i}^{\mathcal{D}}$ denotes the
embeddings derived from the continuous-valued and discrete-valued
observational space, respectively. Due to the symmetric architecture of dual-
VAE, here we use $d$ and $d^{\prime}$ to represent one of each different
domain, i.e., $d,d^{\prime}\in\\{\mathcal{C},\mathcal{D}\\}$ and $d\neq
d^{\prime}$. Therefore, the positive pairs for patient $i$ can be referred as
$(i^{d},i^{d^{\prime}})$, while the other $2(N-1)$ samples are regarded as
negative pairs. Then the contrastive loss for a positive pair
$(i^{d},i^{d^{\prime}})$ is defined as:

$\mathcal{L}_{i^{d},i^{d^{\prime}}}^{\mathrm{Contra}}=-\log\frac{\exp\left(\operatorname{sim}\left(\mathbf{h}_{i^{d}},\mathbf{h}_{i^{d^{\prime}}}\right)/\tau\right)}{\sum_{i^{dd^{\prime}}=1}^{2N}\mathbb{1}_{[{i^{dd^{\prime}}}\neq
i^{d}]}\exp\left(\operatorname{sim}\left(\mathbf{h}_{i^{d}},\mathbf{h}_{{i^{dd^{\prime}}}}\right)/\tau\right)}$
(5)

where $\operatorname{sim}(u,v)=u^{T}v/\|u\|\|v\|$ denotes the cosine
similarity between two vectors. $\tau>0$ denotes a temperature hyperparameter.
$\mathbb{1}_{[n\neq m]}\in\\{0,1\\}$ is an indicator evaluating to 1 iff
$n\neq m$. And $i^{dd^{\prime}}\in\\{1,2,...,2N\\}$ represents the index of
latent embeddings from both data types. The final contrastive loss is computed
across the total number of $|i^{d}-i^{d^{\prime}}|=N$ positive pairs for both
$(i^{d},i^{d^{\prime}})$ and $(i^{d^{\prime}},i^{d})$, and is defined as:

$\mathcal{L}^{\mathrm{Contra}}=\frac{1}{2N}\sum^{N}_{i^{d}=1}\sum^{N}_{i^{d^{\prime}}=1}[\mathcal{L}_{i^{d},i^{d^{\prime}}}^{\mathrm{Contra}}+\mathcal{L}_{i^{d^{\prime}},i^{d}}^{\mathrm{Contra}}]$
(6)

Semantic loss.

In EHR-M-GANcond, semantic loss is imposed to better align patients with same
labels (conditions) into the same latent space clusters. For example, if the
label of severe clinical deterioration in the ICU is given for conditional
data generation, the corresponding synthetic continuous-valued timeseries
(e.g., severely deranged vital signs) is supposed to be strongly associated
with the discrete-valued timeseries (e.g., intensive medical interventions)
under the same label. For both domains, additional linear classifiers are
trained to classify the latent embeddings based on their corresponding
conditions in the observational space. We implement logistic regression as the
linear classifiers and calculate the cross entropy as the semantic losses for
both domains. For $d\in\\{\mathcal{C},\mathcal{D}\\}$, given the latent
embedding vector $\mathbf{z}^{d}$ and the conditional information vector
$\mathbf{y}$:

$\mathcal{L}_{d}^{\mathrm{Class}}=\mathbb{E}_{\mathbf{z}^{d}\in\mathcal{H}^{\mathcal{S}}}\mathrm{CE}\left(f^{d}_{\mathtt{linear}}(\mathbf{z}^{d}),\mathbf{y}\right)$
(7)

where $f^{d}_{\mathtt{linear}}$ denotes the linear classifier for the
corresponding domain. And
$\mathrm{CE}=-\sum_{j}y_{j}\log\left(\widehat{y}_{j}\right),\;(j=1,2,...,|L|)$
denotes the cross entropy loss, where $\hat{y}_{j}$ is the output of the
linear classifier, and $y_{j}$ is the ground truth label for class $j$ in
condition vector $\mathbf{y}$.

In summary, to train the dual-VAE for learning the shared latent space
representation, the total objective function for
$d\in\\{\mathcal{C},\mathcal{D}\\}$ is:

$\mathcal{L}_{d}=\beta_{0}\mathcal{L}^{\mathrm{ELBO}}_{d}+\beta_{1}\mathcal{L}^{\mathrm{Match}}+\beta_{2}\mathcal{L}^{\mathrm{Contra}}$
(8)

Under the conditional learning scenario of EHR-M-GANcond, the total loss
becomes:

$\mathcal{L}_{d}=\beta_{0}\mathcal{L}^{\mathrm{ELBO}}_{d}+\beta_{1}\mathcal{L}^{\mathrm{Match}}+\beta_{2}\mathcal{L}^{\mathrm{Contra}}+\beta_{3}\mathcal{L}_{d}^{\mathrm{Class}}$
(9)

where $\beta_{0}$, $\beta_{1}$, $\beta_{2}$, and $\beta_{3}$ are scalar loss
weights used to balance the loss terms.

####  2.2.2 Sequentially coupled generator based on CRN

The objective of EHR-M-GAN is to capture the dynamics in the mixed-type
timeseries, which can be regarded as the composition of the spatial and
temporal correlations. To this end, we propose the sequentially coupled
generator, which is built based on the coupled recurrent network (CRN).
Specifically, a CRN comprises two single recurrent networks (SRN) (one for
each type of the multivariate spatial timeseries, i.e., continuous or
discrete-valued). In each SRN, LSTM network is introduced as the recurrent
layer to preserve the temporal dependencies in the sequences. The description
for the basic structure of SRN can be found in the Supplementary materials
(see Section S.1.C). In the following section, we first discuss the structure
of BLSTM in detail as an essential part of CRN, and then build the
sequentially coupled generator based on CRN.

Bilateral long short-term memory.

As shown in Fig. 1, two streams of BLSTM entangles with each other in a CRN,
and are jointly trained to sequentially exploit the correlations between two
types of trajectories. Given $d,d^{\prime}\in\\{\mathcal{C},\mathcal{D}\\}$ in
CRN, since two branches of BLSTM associated with the corresponding domains are
symmetric, we present the mathematical expressions from the perspective of
branch $d$ as an example to illustrate the algorithm of BLSTM in details.
Similar as the formulations in SRN (see Section S.1.C for the detailed
description), $\bm{\upsilon}^{d}_{t}$ and $\mathbf{h}^{d}_{t}$ denotes the
input vector and hidden state vector for domain $d$ at time step $t$,
respectively. An additional set of weights for introducing hidden states
representations $\mathbf{h}^{d^{\prime}}_{t}$ from domain $d^{\prime}$ is
included when updating the input gate $\mathbf{i}^{d}_{t}$, forget gate
$\mathbf{f}^{d}_{t}$, output gate $\mathbf{o}^{d}_{t}$, and cell memory
$\tilde{\mathbf{c}}^{d}_{t}$. The state transition functions for BLSTM are:

$\displaystyle\mathbf{i}^{d}_{t}$
$\displaystyle=\sigma\left(\mathbf{W}_{id{\upsilon}}\bm{\upsilon}^{d}_{t}+\mathbf{W}_{idh^{d^{\prime}}}\mathbf{h}^{d^{\prime}}_{t-1}+\mathbf{W}_{idh^{d}}\mathbf{h}^{d}_{t-1}+\mathbf{b}_{id}\right)$
(10) $\displaystyle\mathbf{f}^{d}_{t}$
$\displaystyle=\sigma\left(\mathbf{W}_{fd{\upsilon}}\bm{\upsilon}^{d}_{t}+\mathbf{W}_{fdh^{d^{\prime}}}\mathbf{h}^{d^{\prime}}_{t-1}+\mathbf{W}_{fdh^{d}}\mathbf{h}^{d}_{t-1}+\mathbf{b}_{fd}\right)$
$\displaystyle\mathbf{o}^{d}_{t}$
$\displaystyle=\sigma\left(\mathbf{W}_{od{\upsilon}}\bm{\upsilon}^{d}_{t}+\mathbf{W}_{odh^{d^{\prime}}}\mathbf{h}^{d^{\prime}}_{t-1}+\mathbf{W}_{odh^{d}}\mathbf{h}^{d}_{t-1}+\mathbf{b}_{od}\right)$
$\displaystyle\tilde{\mathbf{c}}^{d}_{t}$
$\displaystyle=\tanh\left(\mathbf{W}_{cd{\upsilon}}\bm{\upsilon}^{d}_{t}+\mathbf{W}_{cdh^{d^{\prime}}}\mathbf{h}^{d^{\prime}}_{t-1}+\mathbf{W}_{cdh^{d}}\mathbf{h}^{d}_{t-1}+\mathbf{b}_{cd}\right)$
$\displaystyle\mathbf{c}^{d}_{t}$
$\displaystyle=\mathbf{f}^{d}_{t}\odot\mathbf{c}^{d}_{t-1}+\mathbf{i}^{d}_{t}\odot\tilde{\mathbf{c}}^{d}_{t}$
$\displaystyle\mathbf{h}^{d}_{t}$
$\displaystyle=\mathbf{o}^{d}_{t}\odot\tanh\left(\mathbf{c}^{d}_{t}\right)$

The additional set of weights in BLSTM facilitates the model to intrinsically
decide how much information it should pass through the gates from its
counterpart. An illustration of the BLSTM cell can be found in the
Supplementary materials (see Fig. S2).

Coupled recurrent network.

The architecture of CRN consists of three layers: the input layers, the
recurrent layers, and the fully connected layers. First, the random noise
vectors $\smash{\bm{\upsilon}_{t}^{d}}$ and
$\smash{\bm{\upsilon}_{t}^{d^{\prime}}}$ for two domains are separately fed
into the input layers. Then the recurrent layers $\smash{f_{\mathtt{rec}}}$,
which are built based on BLSTM as suggested in Eq. (10), are used to
recursively iterate hidden states from both branches. Finally, the fully
connected layers $f_{\mathtt{conn}}^{d}$ and $f_{\mathtt{conn}}^{d^{\prime}}$
produce the generated latent vectors $\smash{\hat{\mathbf{z}}_{t}^{d}}$ and
$\smash{\hat{\mathbf{z}}_{t}^{d^{\prime}}}$ for the decoding stage in dual-
VAE. At time step $t$, CRN can be formulated as:

$\displaystyle(\mathbf{h}_{t}^{d},\mathbf{h}_{t}^{d^{\prime}})$
$\displaystyle=f_{\mathtt{rec}}((\bm{\upsilon}_{t}^{d},\bm{\upsilon}_{t}^{d^{\prime}}),(\mathbf{h}_{t-1}^{d},\mathbf{h}_{t-1}^{d^{\prime}}))$
(11) $\displaystyle\hat{\mathbf{z}}_{t}^{d}$
$\displaystyle=f^{d}_{\mathtt{conn}}(\mathbf{h}_{t}^{d})$
$\displaystyle\hat{\mathbf{z}}_{t}^{d^{\prime}}$
$\displaystyle=f^{d^{\prime}}_{\mathtt{conn}}(\mathbf{h}_{t}^{d^{\prime}})$

Note that the parameters are initialized independently within the two branches
of the recurrent layers $f_{\mathtt{rec}}$ regardless of the coupled
architecture of the network. With respect to EHR-M-GAN${}_{\texttt{cond}}$,
the generator $G^{\mathrm{CRN}}$ generates synthetic latent embeddings
conditioned on the auxiliary information $\mathbf{y}$ (conditional labels
$\mathbf{y}$ are also incorporated during the pretrain stage for dual-VAE). In
summary, heterogeneous timeseries that exhibits mutual influence are
integrated into the parallelly coupled network architecture to leverage the
shared information. By introducing BLSTM, two streams of the inputs in the
generator are encouraged to “communicate” with each other. CRN is therefore
capable of exploiting the interplay between mixed-type data that correlates
over time.

####  2.2.3 Joint training and optimization

The overall architecture of EHR-M-GAN is shown in Fig. 1. In this section, we
give a detailed description of the entire network’s structure and the
optimization objective of the model. The steps for the training and
optimization of EHR-M-GAN are as follows:

  1. 1.

The pretraining of dual-VAE: First, a dual-VAE network which consists of a
pair of encoders ($\mathit{Enc}^{\mathcal{C}},\mathit{Enc}^{\mathcal{D}}$) and
decoders ($\mathit{Dec}^{\mathcal{C}},\mathit{Dec}^{\mathcal{D}}$) is
pretrained. Based on multiple objective constraints in Eq. 8 (for EHR-M-
GAN${}_{\texttt{cond}}$ the objective function can be referred in Eq. 9), a
shared latent space is learnt using dual-VAE, where the gap between the
embedding representations
$(\mathbf{z}_{1:T}^{\mathcal{C}},\mathbf{z}_{1:T}^{\mathcal{D}})$ from both
domains is minimized.

  2. 2.

The generation of latent representations based on CRN: Then, during the joint
training stage, the sequentially coupled generator is built based on CRN with
the internal transition functions iterating across the timesteps
$t\in\\{1,2,...,T\\}$. Therefore, the synthetic latent embedding
representations
$(\hat{\mathbf{z}}_{1:T}^{\mathcal{C}},\hat{\mathbf{z}}_{1:T}^{\mathcal{D}})$
for both types of data can be obtained.

  3. 3.

The decoding for the mixed-type timeseries: Next, the generated latent
embeddings
$(\hat{\mathbf{z}}_{1:T}^{\mathcal{C}},\hat{\mathbf{z}}_{1:T}^{\mathcal{D}})$
are further fed into the pretrained decoders
($\mathit{Dec}^{\mathcal{C}},\mathit{Dec}^{\mathcal{D}}$) and decoded into the
corresponding synthetic patient trajectories
$(\hat{\mathbf{x}}_{1:T}^{\mathcal{C}},\hat{\mathbf{x}}_{1:T}^{\mathcal{D}})$
in the observational space.

  4. 4.

The adversarial loss update based on the discriminators: Finally, the
adversarial loss can be calculated from the LSTM network-based discriminators
$D^{\mathcal{C}}$ and $D^{\mathcal{D}}$ by distinguishing between the real
samples and synthetic timeseries.

Note that in the conditional version of EHR-M-GAN${}_{\texttt{cond}}$, the
discriminators are also conditioned on the same auxiliary labels information
as in the generator. The mathematical expressions for the min-max objectives
in EHR-M-GAN and EHR-M-GAN${}_{\texttt{cond}}$ are provided as follows:

$\begin{split}\min_{G}\max_{D}V_{\textrm{EHR-M-
GAN}}=&\mathbb{E}_{\mathbf{x}\sim p_{{\mathbf{x}}}}[\log
D^{\mathcal{C}}(\mathbf{x}^{\mathcal{C}})+\log
D^{\mathcal{D}}(\mathbf{x}^{\mathcal{D}})]+\\\ &\mathbb{E}_{\bm{\upsilon}\sim
p_{\bm{\upsilon}}}[\log(1-D^{\mathcal{C}}(\hat{\mathbf{x}}^{\mathcal{C}}))+\log(1-D^{\mathcal{D}}(\hat{\mathbf{x}}^{\mathcal{D}}))]\end{split}$
(12)

$\begin{split}\min_{G}\max_{D}V_{\textrm{EHR-M-
GAN${}_{\texttt{cond}}$}}=\mathbb{E}_{\mathbf{y},\mathbf{x}\sim
p_{\mathbf{y},\mathbf{x}}}&[\log
D^{\mathcal{C}}(\mathbf{x}^{\mathcal{C}}|\mathbf{y})+\log
D^{\mathcal{D}}(\mathbf{x}^{\mathcal{D}}|\mathbf{y})]+\\\
\mathbb{E}_{\mathbf{y}\sim p_{\mathbf{y}},\bm{\upsilon}\sim
p_{{\bm{\upsilon}}}}&[\log(1-D^{\mathcal{C}}(\hat{\mathbf{x}}^{{\mathcal{C}}}|\mathbf{y}))+\log(1-D^{\mathcal{D}}(\hat{\mathbf{x}}^{{\mathcal{D}}}|\mathbf{y}))]\end{split}$
(13)

The pseudocodes for dual-VAE and EHR-M-GAN are provided in the Supplementary
materials (see Section S.1.C).

##  3 Data and Evaluation

###  3.1 Dataset Description

The following three publicly accessible ICU datasets are used for evaluating
the performance of EHR-M-GAN in generating the longitudinal EHR data:

  * •

MIMIC-III (Medical Information Mart for Intensive Care) (Johnson et al.,,
2016) — a freely accessible database that comprises de-identified EHRs
associated with approximately 60,000 ICU admitted patients and 312 million
observations to Beth Israel Deaconess Medical Center.

  * •

eICU (eICU Collaborative Research Database) (Pollard et al.,, 2018) — a multi-
center critical care database containing data for over 200,000 admissions and
827 million observations to ICUs from 208 hospitals located throughout the
United States.

  * •

HiRID (High time-resolution ICU dataset) (Yèche et al.,, 2021) — a high-
resolution ICU dataset relating to more than 3 billion observations from
almost 34,000 ICU patient admissions, monitored at the Department of Intensive
Care Medicine, Bern University Hospital, Switzerland. .

All these critical care databases include vital sign measurements, laboratory
tests, treatment information, survival records, and other routinely collected
data from hospital EHR systems. From these clinical observations, we featurize
the patient trajectories as the combination of continuous-valued physiological
timeseries and discrete-valued medical intervention timeseries. Data are
preprocessed following an open-source framework — MIMIC-Extract (Wang et al.,,
2020). Details on data curation, including the cohort selection criteria, full
list of features, and imputation method, are explained in Supplementary
Materials (see S.2 Datasets). Overall, the summarising statistics of the
finalised cohorts for three databases are shown in Table 1.

Table 1:  Summary of the cohorts after preprocessing on three critical care
databases. Number of patients and ICU admissions are provided for each
dataset. Note that only the first ICU admission is selected for each patient.
The dimension of the continuous- and discrete-valued data are provided. The
conditional labels for training EHR-M-GAN${}_{\texttt{cond}}$ and the
corresponding counts for each class are also listed.

Number of patients Number of ICU admissions Dimension of continuous -valued
variables Dimension of discrete -valued variables Conditional labels Counts
MIMIC-III $28,344$ $28,344$ $78$ $20$ ICU mortality $1,870\,(6.59\%)$ Hospital
mortality $\;\;\;911\,(3.21\%)$ 30-day readmission $1,122\,(3.95\%)$ No 30-day
readmission $24,441\,(86.22\%)$ eICU $99,015$ $99,015$ $55$ $19$ ICU mortality
$4,500\,(4.54\%)$ Hospital mortality $3,291\,(3.32\%)$ Hospital discharge
$91,224\,(92.13\%)$ HiRID $14,129$ $14,129$ $50$ $39$ ICU mortality
$1,266\,(8.96\%)$ ICU discharge $12,963\,(91.74\%)$

###  3.2 Baselines models

We compare the performance of EHR-M-GAN with six state-of-the-art generative
methods: For continuous-valued timeseries generation, benchmark GAN models
include C-RNN-GAN (Mogren,, 2016), R(C)GAN (Esteban et al.,, 2017) and TimeGAN
(Yoon et al.,, 2019). For discrete-valued timeseries generation, medGAN (Choi
et al.,, 2017), seqGAN (Yu et al.,, 2017) and DualAAE (Lee et al.,, 2020) are
used for comparison. For medGAN, we feed the flattened temporal sequence as
the input since the model cannot produce timeseries data. Also, an ablation
study is performed to assess the effectiveness of distinct modules integrated
in the proposed work. More specifically, two main components in our model are
considered: dual-VAE for learning the shared latent representations, and
sequentially coupled generator for capturing the temporal dynamics in mixed-
type data. As a result, multiple variants of EHR-M-GAN are tested:

  * •

GANVAE: It contains the simplified version of EHR-M-GAN, which combines a pair
of pretrained VAE networks and GANs (one for each type of inputs). The
architectures of loss-constrained dual-VAE and sequentially coupled generator
are not included.

  * •

GANSL: In addition to GANVAE, it learns the shared latent space
representations through dual-VAE by adding the corresponding loss functions.

  * •

EHR-M-GAN: In addition to GANSL, it incorporates the sequentially coupled
generator for the learning the correlated temporal dynamics in timeseries of
different data types. This is the proposed full model.

  * •

EHR-M-GANcond: This version is implemented on the basis of conditional GAN
(Mirza & Osindero,, 2014), where the conditional inputs are fed into EHR-M-GAN
to generate patients under specific labels.

Temporal trajectories 24 hours prior to patients’ ICU endpoints (discharge or
death) are extracted. For training EHR-M-GANcond, auxiliary information from
the patient status is used as conditional input. For MIMIC-III dataset, the
classes are (1) ICU mortality: patient died within the ICU; (2) Hospital
mortality: patient discharged alive from the ICU, and died within the
hospital; (3) 30-day readmission: patient discharged alive from the hospital,
and readmitted to the hospital within 30 days; (4) No 30-day readmission:
patient discharged alive from the hospital, and had no readmission record to
the hospital within 30 days. For eICU and HiRID datasets, the corresponding
labels are also extracted based on the availability of the patient outcomes
(see Table 1).

###  3.3 Evaluation metrics

Evaluating GAN models is a notoriously challenging task. Advantages and
pitfalls of commonly used evaluation metrics for GANs are discussed in
(Borji,, 2021). In this work, we first individually assess the
representativeness of the synthetic continuous-valued and discrete-valued
timeseries. This includes measuring the distance between underlying data
distributions (such as Maximum mean discrepancy and Dimension-wise
probability), comparing the indistinguishability of the synthetic data to the
true data (i.e., Discriminative score). Secondly, we assess our model by using
a set of qualitative metrics (such as Embedding visualisation, Patient
trajectory plot and Pearson pairwise correlations) for evaluating to which
extent our model can reconstruct the interdependency between two types of
data. Thirdly, we introduce data augmentation by incorporating synthetized EHR
timeseries under various settings, and quantitatively assess the improvement
provided by EHR-M-GAN in the Downstream tasks for medical intervention
prediction in the ICU. Lastly, we measure the attribute of patient privacy-
preserving of EHR-M-GAN under Membership inference attack. We also evaluate
the performance of the same downstream tasks under Differential privacy
guarantees.

##  4 Results

###  4.1 Maximum mean discrepancy

To measure the similarity between the continuous-valued synthetic data and the
real data, maximum mean discrepancy (MMD) is used. MMD can assess whether two
samples, in our case, true data $x$ and synthetic data $x^{\prime}$, are from
the same distributions. It can be expressed as the first-order moments, i.e.,
mean embeddings, of the two samples in a reproducing kernel Hilbert space
(RKHS):
$\mathrm{k}(\mathbf{x},\mathbf{x^{\prime}})=\sum_{i}\exp\left({\|\mathbf{x}-\mathbf{x^{\prime}}\|^{2}}/{\sigma_{i}^{2}}\right)$.
We follow the implementations in (Sutherland et al.,, 2016) by using a sum of
Gaussian kernel sets.

Table 2: Maximum mean discrepancy (MMD) of continuous-valued synthetic data.
Lower values of MMD indicate models which can better learn the distribution of
the real data.

C-RNN-GAN R(C)GAN TimeGAN GANVAE GANSL EHR-M-GAN EHR-M-GANcond MIMIC-III
$1.038\pm 0.013$ $0.971\pm 0.029$ $0.694\pm 0.025$ $0.926\pm 0.038$ $0.745\pm
0.040$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.692}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.034}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.604}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.027}}$
eICU $1.139\pm 0.023$ $1.106\pm 0.042$ $0.672\pm 0.038$ $0.842\pm 0.029$
$0.670\pm 0.034$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.651}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.026}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.540}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.018}}$
HiRID $0.982\pm 0.017$ $0.865\pm 0.020$ $0.470\pm 0.024$ $0.532\pm 0.035$
$0.508\pm 0.028$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.428}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.015}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.389}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.024}}$

As indicated in Table 2, EHR-M-GAN outperforms the state-of-the-art benchmarks
among all three datasets in synthesizing continuous-valued timeseries. The
conditional version — EHR-M-GAN${}_{\texttt{cond}}$ further boosts the
performance of the model by leveraging the information of the condition-
specific inputs. Furthermore, as shown in the ablation study, EHR-M-GAN and
EHR-M-GAN${}_{\texttt{cond}}$ produce smaller MMD values when compared to
their variants. Using MIMIC-III as an example, compared with the basic model
GAN${}_{\texttt{VAE}}$, by integrating the shared latent space learning using
dual-VAE under multiple loss constraints, the performance of
GAN${}_{\texttt{SL}}$ significantly improves (GAN${}_{\texttt{SL}}$ vs.
GAN${}_{\texttt{VAE}}$, 0.745 to 0.926, p-value < 0.05 from t-test333Unpaired
(two-sample) t-test with a significance level of 0.05 is used throughout the
paper unless specified otherwise.). By further building the sequentially
coupled generator based on BLSTMs and exploiting the information within mixed-
type data, the MMD of EHR-M-GAN shows a nearly 24% improvement over
GAN${}_{\texttt{VAE}}$.

###  4.2 Dimension-wise probability

To evaluate the representativeness of the synthetic discrete-valued
timeseries, the dimension-wise probability test is employed. As a sanity
check, it investigates if the distribution across the spatial and temporal
dimensions is matched between the real and synthetic data. Therefore, the
Bernoulli success probability $p\in[0,1]$ is calculated over all feature
dimensions for the discrete-valued timeseries, and is visualized through
scatterplot.

Figure 2: Scatterplot of the dimension-wise probability test on MIMIC-III
dataset. The x-axis and y-axis represents the probability distribution for the
real data and synthetic data with same sample size, respectively. Each dot
represents a physiological measurement or treatment status at a particular
time in the patient EHR data. Same color indicates same measurement or status
(but with varying timestamps). The optimal performance appears along the
diagonal line. Dimension-wise probability plot for eICU and HiRID dataset can
be found in Supplementary materials (see S.3.A).

As shown in Fig. 2 (see Fig. S3 and S4 for more results on eICU and HiRID
datasets), the optimal results are provided by EHR-M-GAN and EHR-M-GANcond.
The close-to-real probability distributions that appear along the diagonal
line indicate the remarkable similarity between the real data and the
synthetic data provided by our models. Similar to the results in MMD, the
dimensional-wise distributions are better captured when modules such as dual-
VAE and sequentially coupled generator are introduced in EHR-M-GAN.

Among all state-of-the-art benchmark models, DualAAE shows comparable results
to EHR-M-GAN. While in contrast, skewed distributions are observed in medGAN
versus the real data, as it is incapable to model the temporal correlations
within timeseries. On the other hand, despite the well-known performance of
SeqGAN in natural language generation (Yu et al.,, 2017), it is not applicable
in synthesizing sequential clinical EHRs.

Generating discrete-valued features are known to be problematic for
traditional GANs. Due to their limitation in passing the gradients from the
critic models, vanilla GANs cannot update their generators efficiently based
on the adversarial loss (Yu et al.,, 2017; Choi et al.,, 2017). However, the
result of EHR-M-GAN shows its superiority in explicitly capturing each
dimension of the discrete-valued sequences. EHR-M-GAN mitigates this problem
by learning the shared latent representations using dual-VAE. Discrete-valued
timeseries are encoded into a gradient-differentiable space for further
optimizing the generators and thus solving the problem.

###  4.3 Discriminative score

For both continuous-valued and discrete-valued data, the discriminative score
is measured as the accuracy of a discriminator trained post-hoc to separate
real from generated samples. Synthetic data are generated with the same amount
of the hold-out test set from the original data, and are labeled as synthetic
and real correspondingly to train the binary classifier. We opt to implement
the classifier as a single-layered Bi-directional Long Short-Term Memory (Bi-
LSTM) model (i.e., many-to-one), to characterize the temporal correlations
within the patient EHR timeseries in this supervised task.

Table 3: Discriminative score of synthetic data. A discriminative model is
trained post-hoc to discriminate between synthetic samples and real samples.
The accuracy from the discriminative classifier is used as the discriminative
score, where the lower value indicates better performance. The result is
bounded by 0.5 when the classifier cannot distinguish between two
distributions.

Method MIMIC-III eICU HiRID Continuous-valuad synthetic data C-RNN-GAN
$0.825\pm 0.013$ $0.876\pm 0.010$ $0.774\pm 0.022$ R(C)GAN $0.833\pm 0.028$
$0.850\pm 0.021$ $0.742\pm 0.016$ TimeGAN $0.763\pm 0.018$ $0.790\pm 0.013$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.716}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.024}}$
GANVAE $0.842\pm 0.020$ $0.871\pm 0.014$ $0.802\pm 0.017$ GANSL $0.786\pm
0.016$ $0.813\pm 0.023$ $0.752\pm 0.021$ EHR-M-GAN
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.746}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.018}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.776}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.015}}$
$0.724\pm 0.015$ EHR-M-GANcond
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.729}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.025}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.745}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.017}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.693}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.012}}$
Discrete-valuad synthetic data medGAN $0.903\pm 0.027$ $0.915\pm 0.034$
$0.896\pm 0.021$ seqGAN $0.937\pm 0.025$ $0.924\pm 0.023$ $0.913\pm 0.027$
DualAAE $0.847\pm 0.029$ $0.860\pm 0.033$ $0.829\pm 0.024$ GANVAE $0.862\pm
0.024$ $0.881\pm 0.029$ $0.824\pm 0.018$ GANSL $0.829\pm 0.032$ $0.844\pm
0.028$ $0.816\pm 0.025$ EHR-M-GAN
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.813}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.026}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.831}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.024}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.802}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.020}}$
EHR-M-GANcond
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.784}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.024}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.803}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.022}}$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.778}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.019}}$

Results in Table 3 indicate that synthetic data that are highly
indistinguishable from the original data are produced by EHR-M-GAN and EHR-M-
GAN${}_{\texttt{cond}}$. Especially for EHR-M-GAN${}_{\texttt{cond}}$, it
achieves the optimal discriminative scores consistently against other
benchmarks for both continuous-valued and discrete-valued timeseries. For
discrete-valued data generation, EHR-M-GAN-generated samples achieve the
discriminative score of 0.813 on the MIMIC-III dataset, which is a 4%
statistically significant improvement than the best performance among state-
of-the-art benchmarks (EHR-M-GAN vs. DualAAE: 0.802 to 0.829, $p<0.05$). For
continuous-valued timeseries generation, the discriminative score of TimeGAN
on HiRID dataset outperforms the other models as well as EHR-M-GAN, though not
statistically significant (EHR-M-GAN vs. TimeGAN: 0.724 to 0.716, $p=0.4374$).
By leveraging the additional information from the conditional inputs, EHR-M-
GAN${}_{\texttt{cond}}$ can provide significantly better result than TimeGAN
(EHR-M-GAN${}_{\texttt{cond}}$ vs. TimeGAN: 0.693 to 0.716, $p<0.05$).

The ablation study has proved the effectiveness of EHR-M-GAN for generating
high quality EHR timeseries. The shared latent space representation learning
in the dual-VAE (i.e., GANSL) have shown remarkable success as making the
synthetic data more realistic than separately generating the latent embeddings
based on VAEs (as in GANVAE). The sequentially coupled generator further
improves the model by capturing the dynamics between mixed-type data and
iterating over time, therefore enabling the synthetic timeseries to become
more indistinguishable from the original.

###  4.4 Pearson pairwise correlation

To evaluate the inter- and intra- correlations between continuous-valued and
discrete-valued timeseries, we employ Pearson pairwise correlation plot to
produce a qualitative comparison between the real and synthetic data. The
positive and negative correlation among each feature pair is measured by
Pearson correlation coefficient (PCC), which ranges from -1 to 1.

Figure 3: Pearson pairwise correlation (PPC) between continuous-valued and
discrete-valued timeseries. The plots contrast the PPC calculated within the
real data (left column) and the synthetic data generated by EHR-M-GAN (right
column).

Five commonly measured vital sign and laboratory measurement features — Oxygen
Saturation (SpO2), Systolic Blood Pressure (SBP), Respiratory Rate (RR), Heart
Rate (HR), Temperature (Temp), as well as two medical intervention features —
Mechanical Ventilation (Vent.) and Vasopressor (Vaso.) are considered and
compared as an exemplar in Fig. 3. Timestamps are extracted with every 3 hours
interval in a total of 24 hours ICU stay, to explore the temporal dependencies
within different variables (ticks of the timestamps are omitted).

As observed, correlation trends over distinctive features are closely
reflected by the synthetic data. For example, negative correlations are shown
between Vasopressor and Systolic Blood Pressure in all three datasets. Similar
correlations are also well-captured in the synthetic data provided by EHR-M-
GAN. Such correlations can be interpreted as the time-delay effect of
prescribing the treatment of vasopressors to elevate mean arterial pressure
(MAP) once noticing patients’ abnormal blood pressure.

In addition, temporal dependencies revealed in the original EHR data are also
preserved in the synthetic timeseries. For example, synchronized correlations
across timestamps are observed between Respiratory Rate and Heart Rate in
MIMIC-III dataset. This can be explained by the common regulation of these two
features by the autonomic nervous system and their synchronized increase in
cases of physiological stress, such as hypoxemia. Similar trends in the
synthetic data suggest that EHR-M-GAN can successfully recover temporal
dependencies with a high granularity from real patient trajectories. It is
also worth noticing that these three ICU datasets exhibit similar yet not
identical correlations over different feature pairs. This demonstrates the
data heterogeneity originating from the distribution discrepancy of the
underlying patient populations.

###  4.5 Patient trajectory visualisation

Sample trajectories generated by EHR-M-GAN and EHR-M-GANcond are shown in Fig.
4 for MIMIC-III dataset (see Fig. S6, S7, S8 in Supplementary Materials for
more results). By visual inspection, we can see that the proposed models can
capture patterns within substantially different time-varying signals. Temporal
dynamics in both rapidly fluctuating vital signs (e.g., Oxygen Saturation) and
infrequently changing intervention (e.g., Mechanical Ventilation) can be well-
preserved. The variability of the synthetic signals suggests that rather than
just intelligently “memorizing” the training data, EHR-M-GAN is capable of
modelling the underlying true data distributions in order to produce genuine
samples.

Furthermore, EHR-M-GANcond shows even more superior performance as it can
generate trajectories with predefined patient conditions. From the instance
generated by EHR-M-GANcond under the conditional information of ICU mortality
in Fig. 4, it is possible to observe the variations in physiological signals
which suggest that the patient suffers from severe deterioration towards the
end of the clinical endpoint. Meanwhile, temporal dependencies are observed
within different physiological signals along with medical intervention.
Synchronized correlations are shown between Oxygen Saturation, Systolic Blood
Pressure, and Heart Rate from approximately 18 hr to 21 hr, followed by the
presence of Mechanical Ventilation for providing the respiratory support for
the patient at 22 hr.

(a) Sampled patient trajectories on MIMIC-III dataset using EHR-M-GAN.

(b) Sampled patient trajectories on MIMIC-III dataset using EHR-M-GANcond
(under the condition of ICU mortality).

Figure 4: Examples of synthetic patient trajectories. Multivariate timeseries
are generated by EHR-M-GAN and EHR-M-GANcond for the 24 hours before patients’
discharge or death in the ICU, including Oxygen Saturation, Systolic Blood
Pressure, Respiratory Rate, Heart Rate, Temperature, Vasopressor and
Mechanical Ventilation. For interventions (vasopressor use and mechanical
ventilation), the y-axis indicates whether they are being applied ("On") or
not ("Off") at a given time. For the conditional model, EHR-M-GANcond, patient
trajectories under 4 conditions — ICU mortality, Hospital mortality, 30-day
readmission, No 30-day readmission — are generated individually (only the
condition of ICU mortality is shown here). Synthetic patient trajectories
under the rest of the conditions in MIMIC-III, and results on eICU and HiRID
dataset can be found in Supplementary materials.

###  4.6 Embedding visualisation

Conditional GAN helps to generate patient data with a set of specific medical
conditions (Mirza & Osindero,, 2014), therefore improving the diversity of the
data when synthesizing clinical timeseries. We apply t-SNE to qualitatively
visualise the latent embedding generated by EHR-M-GANcond. The distributions
of the latent embeddings are induced by the encoders in the dual-VAE during
learning the shared latent space representations. As shown in Fig. S5, good
separability is shown in the latent representations recovered from EHR-M-
GANcond. On the other hand, the result also indicates that raw EHR timeseries
do not have distinctly different modes in the lower dimensional space. By
leveraging the conditional information and placing the semantic loss
constraint to learn the shared latent representations, the conditional version
of the proposed model can further yield benefits by synthesizing condition-
specific EHR timeseries with respect to distinctive patient health status.

###  4.7 Downstream tasks

As previously discussed, one of the most prominent goals for GANs is to
benefit the future downstream analyses in the real clinical application. A
relevant question in the ICU is whether specialized medical treatments, such
as therapeutic interventions or organ support, are required for critically ill
patients during the admission. Accurate predictions on such tasks can help
clinicians to provide actionable, in-time interventions in the resource-
intensive ICU. Therefore in this section, clinical intervention prediction
tasks are implemented to evaluate the potential of EHR-M-GAN and EHR-M-GANcond
in synthesizing high-fidelity synthetic data to further boost the performance
of ML classifiers. In line with prior work (Wang et al.,, 2020; Wu et al.,,
2017; Suresh et al.,, 2017), we establish LSTM-based classifiers to predict
the status of mechanical ventilation and vasopressors using continuous-valued
multivariate physiological signals as the predictors. A fixed duration of 12
hours is used for both observation window and prediction window (see Fig. 5).
Four outcomes of medical intervention status are defined as: Stay on, Onset,
Switch off, Stay off (detailed descriptions can be found in Fig. 5).

Figure 5: Downstream prediction example. Data within 24-hours prior to the
patient’s endpoints in the ICU (discharge or mortality) is extracted. Both the
observation window and prediction window are fixed as 12 hours. The
classification task is to use patients’ continuous-valued physiological
measurements within the observation window as input, to predict the
forthcoming discrete-valued medical intervention status in the prediction
window. The four outcomes of the intervention status can be categorized as
follows: Stay On: The intervention begins with on and stays on within the
prediction window; Onset: The intervention begins with off and is turned on
within the prediction window; Switch off: The intervention begins with on and
is stopped within the prediction window; Stay Off: The intervention begins
with off and stays off within the prediction window.

We partition the dataset as illustrated in Figure 6(a), and the performances
are assessed from two aspects (see Figure 6(b)): (i) Traditional approach: To
explore whether the synthetic data can represent the real data accurately, we
compare Train on Real, Test on Real (TRTR) with Train on Synthetic, Test on
Real (TSTR), to show whether the performance of a classifier trained on
synthetic data from EHR-M-GAN or EHR-M-GANcond can be generalized to real
data; and (ii) Data augmentation approach: As data augmentation is employed as
a means of circumventing the issue caused by the under-resourced EHR data,
here we explore whether synthetic data can used to improve the existing ML
algorithms through data augmentation. Therefore, Train on Synthetic and Real,
Test on Real (TSRTR) is compared with TRTR to measure the improvement of the
classifier’s performance when trained on the augmented data (Esteban et al.,,
2017; Kiyasseh et al.,, 2020). The augmentation ratio $\alpha$ or $\beta$ is
applied on sub-train data $A^{\prime}_{Tr}$ or synthetic data $B$, in two
different scenarios of TSRTR, respectively. Details are explained as follows
(also see Figure 6(b) for illustration).

Firstly, as the dearth of data potentially degrades the performance of
downstream classifiers, given that the real data has a limited and fixed
sample size, we investigate whether adding synthetic EHR data provided by EHR-
M-GAN and EHR-M-GAN${}_{\texttt{cond}}$ can improve the training of downstream
classifiers. Ratio ${\bm{\alpha}}$ indicates the portion of synthetic data
(see Figure 6(b)) being used to augment the real data to improve the quality
and robustness of the downstream classifiers. $\alpha$ is set to be 10%, 25%,
and 50%, representing the availability of synthetic samples provided for
augmentation.

Secondly, the acquisition of healthcare data is generally time-consuming and
expensive, therefore another overarching goal for the generative model is to
minimize the efforts in collecting data. In this section, we investigate
whether high-fidelity synthetic data can offer a viable solution for boosting
the downstream classifiers’ performance when the availability of real data is
limited. This allows us to understand if the sample size required for real
data collection can be reduced while maintaining sufficient predictive power
through the use of synthetic data. During the experiment, the synthetic data
$B$ is given (to emulate the scenario where synthetic datasets are available
for a particular clinical research purpose), which further is combined with
limited real data (collected during clinical trial), to train the downstream
classifiers (i.e., augment synthetic data with limited real data). Then by
implementing EHR-M-GAN or EHR-M-GAN${}_{\texttt{cond}}$ in TSRTR, we
investigate the proportion of the real data $A^{\prime}_{Tr}$ (ratio
${\bm{\beta}}$) required to maintain the same performance as in TRTR based on
the entire synthetic dataset $B$ (see Figure 6(b)).

(a) Data splitting.

(b) Data augmentation scenarios.

Figure 6: Downstream intervention prediction experimental setup. a. Data
splitting. During training stage, the real data is split into two sets with
70% training data $A$ and 30% test data $A^{\prime}$. The test data
$A^{\prime}$ is further split into sub-train data $A^{\prime}_{Tr}$ and sub-
test data $A^{\prime}_{Te}$ with equal size. Then, the synthetic data $B$,
with size equal to the sub-train data $A^{\prime}_{Tr}$, is synthesized by
EHR-M-GAN (or EHR-M-GANcond) trained on the real training data $A$. b. Data
augmentation scenarios. Subsequent experiments are trained on set
$A^{\prime}_{Tr}$, or $B$, or $A^{\prime}_{Tr}\cup B$ and then tested on
$A^{\prime}_{Te}$. In traditional approach, results based on Train on Real,
Test on Real (TRTR) and Train on Synthetic, Test on Real (TSTR) are compared
to assess the generalisability of the synthetic data. In data augmentation
approach, i.e., Train on Synthetic and Real, Test on Real (TSRTR), we either
augment real data $A^{\prime}_{Tr}$ with $\alpha$ (augmentation ratio, 0 to
50%) of the synthetic samples $B$, or augment synthetic samples $B$ with
$\beta$ (0 to 50%) of the real data $A^{\prime}_{Tr}$.

Traditional approach.

Table 4(a) compares the classification performances of predicting forthcoming
medical interventions in the ICUs under the experimental setting of TRTR and
TSTR. It is expected that the optimal AUROCs are achieved by the classifiers
that are trained on real data. In comparison, the classifiers trained on the
synthetic data provided by proposed models can achieve similar performances.
More specifically, synthetic data generated by EHR-M-GAN${}_{\texttt{cond}}$
demonstrates better generalisability when compared with EHR-M-GAN in the
downstream application, such as the task of predicting mechanical ventilation
on the HiRID dataset (TRTR vs. TSTR from EHR-M-GAN${}_{\texttt{cond}}$: 0.867
to 0.856, with p=0.3906).

Data augmentation approach (with ratio ${\bm{\alpha}}$).

The results in Table 4(b) demonstrate that classifiers boosted by EHR-M-GAN
can consistently outperform TRTR (see Table 4(a)) at the augmentation ratio of
50%. In comparison, only 25% of augmentation ratio is needed to achieve
improved results for EHR-M-GAN${}_{\texttt{cond}}$. For example, the
classifier trained on MIMIC-III to predict the status of Vasopressor with
augmentation ratio $\alpha$ set as 50%, significantly increase the AUROC by 6%
when compared to the classifier trained using only the real data (EHR-M-
GAN${}_{\texttt{cond}}$ vs. TRTR: 0.896 to 0.841, $p<0.05$). Our experiment
results have demonstrated that the proposed models can be used for data
augmentation to overcome the issue of data scarcity and subsequently improve
the classifiers’ performance.

Data augmentation approach (with ratio ${\bm{\beta}}$).

On the other hand, as shown in Table 4(c), by augmenting with the synthetic
data provided by EHR-M-GAN, only approximately 50% of the real data is
required to keep the classification AUROCs on par with, or even significantly
better than fully exploiting the real data under TRTR. For EHR-M-GANcond, the
ratio needed for real data to maintain the comparable predictive power is
further reduced to 25%, which equivalently indicates a 75% reduction of sample
size required in real data collection. Overall, results presented in Table
4(c) demonstrate that by exploiting only a limited ratio of the real data,
EHR-M-GAN and EHR-M-GAN${}_{\texttt{cond}}$ can robustly maintain the level of
prediction performance, therefore alleviating the necessity for acquiring
clinical data at scale.

Table 4: Downstream task evaluation. Downstream tasks are evaluated under the
training scenarios of Train on Real, Test on Real (TRTR), Train on Synthetic,
Test on Real (TSTR), or Train on Synthetic and Real, Test on Real (TSRTR).
Prediction of two outcomes of interest – intervention by Mechanical
ventilation (Vent.) and Vasopressors (Vaso.) are selected as exemplary tasks.
Macro-AUROC is used to score the performance of the LSTM-based classifiers on
the mutli-class prediction tasks (labeled as Stay on, Onset, Switch off, Stay
off).

(a) Performance of the LSTM-based classifier under TRTR and TSTR.

Dataset Treatments Real data EHR-M-GAN EHR-M-GANcond MIMIC-III Vent.
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.894}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.016}}$
$0.740\pm 0.009$ $0.823\pm 0.020$ Vaso.
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.841}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.009}}$
$0.725\pm 0.015$ $0.810\pm 0.019$ eICU Vent.
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.868}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.015}}$
$0.745\pm 0.008$ $0.795\pm 0.015$ Vaso.
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.813}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.018}}$
$0.706\pm 0.014$ $0.748\pm 0.017$ HiRID Vent.
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.867}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.012}}$
$0.825\pm 0.019$ $0.856\pm 0.033$ Vaso.
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.878}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.010}}$
$0.814\pm 0.015$ $0.844\pm 0.018$

(b) Performance of the downstream LSTM-based classifier under TSRTR with data
augmentation. All data from sub-train data $A^{\prime}_{Tr}$ concated with
$\alpha$ of the synthetic data $B$ (augmentation ratio $\alpha$ = 10%, 25% or
50%) is used as the training set. The upper arrow ($\uparrow$) indicates that
the AUROC value under TSRTR is higher than TRTR for the corresponding task,
while the bold arrow (black$\uparrow$) indicates that the value is
significantly improved using t-test (p$\leq$0.05).

EHR-M-GAN EHR-M-GANcond Dataset Treatments $\alpha=10\%$ $\alpha=25\%$
$\alpha=50\%$ $\alpha=10\%$ $\alpha=25\%$ $\alpha=50\%$ MIMIC-III Vent.
$0.828\pm 0.013$ $0.877\pm 0.014$ $0.912\pm 0.015$ (black$\uparrow$) $0.845\pm
0.022$ $0.896\pm 0.013$ ($\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.923}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.018}}$
(black$\uparrow$) Vaso. $0.816\pm 0.015$ $0.834\pm 0.023$ $0.859\pm 0.013$
(black$\uparrow$) $0.848\pm 0.012$ ($\uparrow$) $0.876\pm 0.017$
(black$\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.896}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.015}}$
(black$\uparrow$) eICU Vent. $0.858\pm 0.008$ $0.862\pm 0.012$ $0.873\pm
0.014$ ($\uparrow$) $0.865\pm 0.009$ $0.879\pm 0.014$ ($\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.883}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.016}}$
($\uparrow$) Vaso. $0.798\pm 0.015$ $0.805\pm 0.020$ $0.821\pm 0.028$
($\uparrow$) $0.813\pm 0.016$ ($\uparrow$) $0.834\pm 0.019$ (black$\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.839}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.014}}$
(black$\uparrow$) HiRID Vent. $0.871\pm 0.025$ ($\uparrow$) $0.882\pm 0.021$
($\uparrow$) $0.913\pm 0.019$ (black$\uparrow$) $0.894\pm 0.015$
(black$\uparrow$) $0.906\pm 0.018$ (black$\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.923}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.021}}$
(black$\uparrow$) Vaso. $0.850\pm 0.016$ $0.874\pm 0.022$ $0.894\pm 0.018$
(black$\uparrow$) $0.883\pm 0.017$ ($\uparrow$) $0.908\pm 0.024$ ($\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.913}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.019}}$
(black$\uparrow$)

(c) Performance of the downstream LSTM-based classifier under TSRTR with data
augmentation. All data from synthetic data $B$ concated with $\beta$ of the
sub-train data $A^{\prime}_{Tr}$ (augmentation ratio $\beta$ = 10%, 25% or
50%) is used as the training set. The upper arrow ($\uparrow$) indicates that
the AUROC value under TSRTR is higher than TRTR for the corresponding task,
while the bold arrow (black$\uparrow$) indicates that the value is
significantly improved using t-test (p$\leq$0.05).

EHR-M-GAN EHR-M-GANcond Dataset Treatments $\beta=10\%$ $\beta=25\%$
$\beta=50\%$ $\beta=10\%$ $\beta=25\%$ $\beta=50\%$ MIMIC-III Vent. $0.757\pm
0.016$ $0.824\pm 0.010$ $0.885\pm 0.009$ $0.847\pm 0.017$ $0.903\pm 0.014$
($\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.915}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.009}}$
(black$\uparrow$) Vaso. $0.786\pm 0.019$ $0.810\pm 0.020$ $0.849\pm 0.017$
($\uparrow$) $0.823\pm 0.014$ $0.851\pm 0.019$ ($\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.873}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.017}}$
(black$\uparrow$) eICU Vent. $0.761\pm 0.011$ $0.822\pm 0.012$ $0.870\pm
0.019$ ($\uparrow$) $0.816\pm 0.016$ $0.845\pm 0.018$
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.872}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.020}}$
($\uparrow$) Vaso. $0.742\pm 0.014$ $0.797\pm 0.013$ $0.846\pm 0.018$
(black$\uparrow$) $0.785\pm 0.022$ $0.819\pm 0.021$ ($\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.834}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.013}}$
(black$\uparrow$) HiRID Vent. $0.856\pm 0.012$ $0.879\pm 0.019$ ($\uparrow$)
$0.895\pm 0.021$ (black$\uparrow$) $0.874\pm 0.016$ ($\uparrow$) $0.896\pm
0.018$ (black$\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.904}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.012}}$
(black$\uparrow$) Vaso. $0.826\pm 0.024$ $0.859\pm 0.013$ $0.893\pm 0.018$
($\uparrow$) $0.865\pm 0.025$ $0.897\pm 0.021$ (black$\uparrow$)
$\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.914}}\pm\leavevmode\resizebox{22.77785pt}{6.44444pt}{{0.018}}$
(black$\uparrow$)

###  4.8 Privacy risk evaluation

Patient privacy is a major concern with regards to sharing electronic health
records in any means. Although generative models overcome the explicit one-to-
one mapping towards the underlying original data (in contrast to data
anonymisation), GAN could potentially raise privacy concerns of information
leakage if they simply “memorise” the training data. In that case, sensitive
medical information (e.g. national insurance number) belonging to a specific
patient used in training GANs can be retrieved during the generative stage,
thus posing challenges for preserving privacy in downstream applications.

In this section, we first quantify the vulnerability of EHR-M-GAN to
adversary’s membership inference attacks, also known as presence disclosure
(Hayes et al.,, 2019; Chen et al.,, 2020a). The threat model is implemented
under the membership inference for GANs in the black-box settings (Hayes et
al.,, 2019). The attacker is assumed to possess complete knowledge of all the
patient records set $P$, where a subset from $P$ further is used to train
GANs. During the experiment, the number of samples in the subset for training
EHR-M-GAN are varied to investigate the impact of the availability of training
data on the success of the attacker (see Figure 7(a)). By observing the
synthetic patient records from EHR-M-GAN, the adversary’s goal is to determine
whether a single known record $x$ in the patient record set $P$ is from the
data used in training EHR-M-GAN. Determined by whether the attacker can
correctly infer a given record is in or not in GAN’s training, the accuracy
and recall can be calculated.

As shown in Figure 7(a), when 90% of the training data is used for developing
EHR-M-GAN, the attacker had a recall of 0.533 and accuracy of 0.527 to recover
which training data are considered. This is eminently close to flipping a coin
with random guess (i.e., 0.5), indicating EHR-M-GAN is sufficiently robust
against the membership inference attack. On the other hand, as the percentage
of the training data reduces, both accuracy and recall for membership
inference attacks rise. An accuracy of 0.624 and recall of 0.732 are reached
with 20% of training data. This offers a guideline for future application in
developing GANs that incorporating more training data can make the generator
less susceptible to such attack. This is also consistent with the conclusion
derived from the experiment on membership inference attacks in the prior
research (Lin et al.,, 2020).

(a) Membership inference attack.

(b) Differential privacy.

Figure 7: Privacy risk evaluation of EHR-M-GAN on MIMIC-III dataset. a.
Membership inference attack. Membership inference attack against EHR-M-GAN vs.
the percentage of the training data. Accuracy and recall are used to evaluate
the success rate of such attacks. Lower accuracy or recall indicates less
privacy information disclosed by the attacker from the generative model (0.5
can be seen as the random guess baseline where strong privacy guarantees are
provided by GANs). Recall indicates the ratio of samples that are successfully
claimed by the attacker among all the real data that are used in training GAN
models. b. Differential privacy. Performance of medical intervention
prediction tasks, under various differential privacy (DP) budgets, measured by
Macro-AUROC.

The concept differential privacy (DP) (Dwork,, 2008), which appears to be a
rigorous mathematical definition of privacy, has emerged to be the prevailing
notion in the context of statistically analyzing data privacy. The
$(\epsilon,\delta){\text{-}}$differential privacy is guaranteed for model $M$,
if given any pair of adjacent datasets $D$ and $D^{\prime}$ (differing on a
single patient record), it holds: $P[\mathcal{M}(D)\in S]\leq
e^{\epsilon}P\left[\mathcal{M}\left(D^{\prime}\right)\in S\right]+\delta$. In
our case, $\mathcal{M}(\cdot)$ is the GAN model trained based on $D$ or
$D^{\prime}$, and $S$ is the subset of any possible outcomes of the generative
process. By perturbing the underlying data distribution, DP bounds the maximum
variations of the algorithm when any single individual is included or excluded
from the dataset. In practice, recent works on developing differentially
private deep learning models has benefited from differential private
stochastic gradient descent (DP-SGD) algorithm. DP-SGD operates DP by gradient
clipping and noise adding during SGD, thereby ensuring that the impact of
single record in the training dataset on algorithm parameters is limited
within DP’s extend. In this section, $(\epsilon,\delta){\text{-}}$differential
privacy is implemented in EHR-M-GAN using TensorFlow
Privacy444https://github.com/tensorflow/privacy. We then perform the same
downstream tasks on medical intervention prediction using synthetic data
generated from DP-guaranteed EHR-M-GAN, and compare its performance with TSTR
(as shown in Table 4(a)).

Figure 7(b) shows the TSTR performance of EHR-M-GAN under differential privacy
guarantee with varying budgets $\epsilon$ ($\delta$ fixed at $\leq 0.001$).
The value $\epsilon$ determines how strict the privacy is, where the smaller
value indicates a stronger privacy restriction. As suggested in Figure 7(b),
the performance of the downstream tasks operated based on the synthetic data
generated by EHR-M-GAN improves as the DP budget relaxes ($\epsilon$
increases). We observe that the AUROC of DP-bounded EHR-M-GAN can maintain at
an acceptable level even under strict privacy setting. For example, the AUROC
for predicting the treatment of Vasopressor can maintain at 0.714 (AUROC =
0.725 under TRTR) even when the $\epsilon$ decrease to 4, which is an
empirically reasonable value for implementing DP in practice (Differential
Privacy Team,, 2017). Future work that focuses on privacy-preserving GAN under
DP-guarantee is expected, where the fidelity of the synthetic data can be
restored without compromising its privacy.

##  5 Discussion and conclusions

In this study, we propose a generative adversarial network entitled EHR-M-GAN,
aiming at mitigating the challenge of synthesizing longitudinal EHR with mixed
data types. To better capture the correlations between the continuous-valued
and discrete-valued timeseries, shared latent representations are learnt based
on the proposed dual-VAE, where the dimensionality for the subsequent
adversarial learning is reduced. Then the proposed sequentially coupled
generator built based on the architecture of RNNs enables EHR-M-GAN to model
the temporal dependencies between heterogeneous data. During the quantitative
and qualitative evaluations of the proposed model, both EHR-M-GAN and its
conditional version, EHR-M-GAN${}_{\texttt{cond}}$, demonstrate consistent
improvements against the state-of-the-art benchmark GANs in synthesizing
timeseries data with high-fidelity. Notably, as opposed to previous models
which were confined to synthesizing only one specific type of data, EHR-M-GAN
can produce mixed-type timeseries and successfully capture the temporal
dynamics within. During downstream task evaluation, given the prediction of
medical interventions in fast-paced critical care environments as an exemplar,
the results demonstrate the broad applicability of our model in developing ML
algorithm-based decision support tools through data augmentation.

There are limitations in the current work. First, our model is built upon the
smoothed timeseries signals, thus making no attempt at modelling the time gaps
between signals. As during the data preprocessing, we hourly aggregate
patients physiological and medical intervention signals based on their mean
statistics, as suggested in MIMIC-Extract (Wang et al.,, 2020). However, in
real clinical data, dealing with irregularity of the timestamps in the sampled
physiological timeseries data is still an area of active research, not only
for data synthesis, but also for downstream secondary EHR data analysis
(Shamout et al.,, 2019). Modelling such time intervals is non-trivial as the
determinative perspectives can sometimes go beyond the scope of inferring
patients physiological status, including resource allocations within
hospitals, or economic considerations from patients. This could be
investigated in future work by incorporating additional inputs from clinical
information systems or socioeconomic backgrounds.

Second, although our models successfully extend the scope of synthetic data
generation into mixed-type timeseries, they are still limited in single
modality (i.e., structured EHR data). However, the adoption of digitized
healthcare information systems provide a vast range of other — heterogeneous
databases with mixed modalities (Huang et al.,, 2020). Open-access data with
various modalities are provided across the computational health community,
including medical image data (such as magnetic resonance imaging (MRI) scans
from BraTS (Menze et al.,, 2014)), physiological waveform signals (such as ECG
and PPG data from MIMIC-III Waveform (Moody et al.,, 2020)), as well as
unstructured natural language information (such as free-text clinical notes
from i2b2 platform (n2c2.,, 2021)). Future work can be investigated in the
topic of generating synthetic data with multi-modalities, as integrating such
data will increase the value of data-driven ML models in the real world
clinical applications.

Synthetic data provides an alternative to sharing real patient data while
preserving patient privacy. Results in our study demonstrate that the proposed
EHR-M-GAN and EHR-M-GAN${}_{\texttt{cond}}$ can generate realistic
longitudinal EHR timeseries with mixed data types. By providing synthetic EHR
data with higher fidelity and more variety, the proposed model can therefore
enable faster development in AI-driven clinical tools with increased
robustness and adaptability. In addition to the improved performance against
the existing state-of-the-art benchmark models, augmentation provided by
synthetic data during training boosts the predictive performance in downstream
clinical tasks. EHR-M-GAN can help eliminate the barriers to data acquisition
for healthcare studies, therefore overcoming the challenges posed by the
paucity of medical data available and approved for research use.

## References

  * Artzi et al., (2020) Artzi, Nitzan Shalom, Shilo, Smadar, Hadar, Eran, Rossman, Hagai, Barbash-Hazan, Shiri, Ben-Haroush, Avi, Balicer, Ran D, Feldman, Becca, Wiznitzer, Arnon, & Segal, Eran. 2020.  Prediction of gestational diabetes based on nationwide electronic health records.  Nature medicine, 26(1), 71–76.
  * Borji, (2021) Borji, Ali. 2021.  Pros and Cons of GAN Evaluation Measures: New Developments.  arXiv preprint arXiv:2103.09396.
  * Chen et al., (2020a) Chen, Dingfan, Yu, Ning, Zhang, Yang, & Fritz, Mario. 2020a.  Gan-leaks: A taxonomy of membership inference attacks against generative models.  Pages 343–362 of: Proceedings of the 2020 ACM SIGSAC conference on computer and communications security.
  * Chen et al., (2021) Chen, Richard J, Lu, Ming Y, Chen, Tiffany Y, Williamson, Drew FK, & Mahmood, Faisal. 2021.  Synthetic data in machine learning for medicine and healthcare.  Nature Biomedical Engineering, 1–5.
  * Chen et al., (2020b) Chen, Ting, Kornblith, Simon, Norouzi, Mohammad, & Hinton, Geoffrey. 2020b.  A simple framework for contrastive learning of visual representations.  Pages 1597–1607 of: International conference on machine learning.  PMLR.
  * Choi et al., (2017) Choi, Edward, Biswal, Siddharth, Malin, Bradley, Duke, Jon, Stewart, Walter F, & Sun, Jimeng. 2017.  Generating multi-label discrete patient records using generative adversarial networks.  Pages 286–305 of: Machine learning for healthcare conference.  PMLR.
  * CPRD., (2021) CPRD. 2021.  Synthetic data at CPRD.  https://www.cprd.com/content/synthetic-data.  Accessed: 2021-12-02.
  * Differential Privacy Team, (2017) Differential Privacy Team, Apple. 2017.  Learning with Privacy at Scale.
  * Dinov, (2016) Dinov, Ivo D. 2016.  Methodological challenges and analytic opportunities for modeling and interpreting Big Healthcare Data.  Gigascience, 5(1), s13742–016.
  * Dwork, (2006) Dwork, Cynthia. 2006.  Differential privacy.  Pages 1–12 of: International Colloquium on Automata, Languages, and Programming.  Springer.
  * Dwork, (2008) Dwork, Cynthia. 2008.  Differential privacy: A survey of results.  Pages 1–19 of: International conference on theory and applications of models of computation.  Springer.
  * El Emam et al., (2021) El Emam, Khaled, Mosquera, Lucy, Jonker, Elizabeth, & Sood, Harpreet. 2021.  Evaluating the utility of synthetic COVID-19 case data.  JAMIA open, 4(1), ooab012.
  * Esteban et al., (2017) Esteban, Cristóbal, Hyland, Stephanie L, & Rätsch, Gunnar. 2017.  Real-valued (medical) time series generation with recurrent conditional gans.  arXiv preprint arXiv:1706.02633.
  * Esteva et al., (2021) Esteva, Andre, Chou, Katherine, Yeung, Serena, Naik, Nikhil, Madani, Ali, Mottaghi, Ali, Liu, Yun, Topol, Eric, Dean, Jeff, & Socher, Richard. 2021.  Deep learning-enabled medical computer vision.  NPJ digital medicine, 4(1), 1–9.
  * Frid-Adar et al., (2018) Frid-Adar, Maayan, Diamant, Idit, Klang, Eyal, Amitai, Michal, Goldberger, Jacob, & Greenspan, Hayit. 2018.  GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification.  Neurocomputing, 321, 321–331.
  * Futoma et al., (2020) Futoma, Joseph, Simons, Morgan, Panch, Trishan, Doshi-Velez, Finale, & Celi, Leo Anthony. 2020.  The myth of generalisability in clinical research and machine learning in health care.  The Lancet Digital Health, 2(9), e489–e492.
  * Ghassemi et al., (2017) Ghassemi, Marzyeh, Wu, Mike, Hughes, Michael C, Szolovits, Peter, & Doshi-Velez, Finale. 2017.  Predicting intervention onset in the ICU with switching state space models.  AMIA Summits on Translational Science Proceedings, 2017, 82\.
  * Goodfellow et al., (2014) Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, & Bengio, Yoshua. 2014.  Generative adversarial nets.  Advances in neural information processing systems, 27.
  * Hayes et al., (2019) Hayes, Jamie, Melis, Luca, Danezis, George, & De Cristofaro, Emiliano. 2019.  Logan: Membership inference attacks against generative models.  Pages 133–152 of: Proceedings on Privacy Enhancing Technologies (PoPETs), vol. 2019.  De Gruyter.
  * Huang et al., (2020) Huang, Shih-Cheng, Pareek, Anuj, Seyyedi, Saeed, Banerjee, Imon, & Lungren, Matthew P. 2020.  Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines.  NPJ digital medicine, 3(1), 1–9.
  * Johnson et al., (2016) Johnson, Alistair EW, Pollard, Tom J, Shen, Lu, Li-Wei, H Lehman, Feng, Mengling, Ghassemi, Mohammad, Moody, Benjamin, Szolovits, Peter, Celi, Leo Anthony, & Mark, Roger G. 2016.  MIMIC-III, a freely accessible critical care database.  Scientific data, 3(1), 1–9.
  * Jordon et al., (2018) Jordon, James, Yoon, Jinsung, & Van Der Schaar, Mihaela. 2018.  PATE-GAN: Generating synthetic data with differential privacy guarantees.  In: International conference on learning representations.
  * Jordon et al., (2020) Jordon, James, Wilson, Alan, & van der Schaar, Mihaela. 2020.  Synthetic Data: Opening the data floodgates to enable faster, more directed development of machine learning methods.  arXiv preprint arXiv:2012.04580.
  * Kearney et al., (2020) Kearney, Vasant, Chan, Jason W, Wang, Tianqi, Perry, Alan, Descovich, Martina, Morin, Olivier, Yom, Sue S, & Solberg, Timothy D. 2020.  DoseGAN: a generative adversarial network for synthetic dose prediction using attention-gated discrimination and generation.  Scientific reports, 10(1), 1–8.
  * Kim et al., (2021) Kim, Jihoon, Neumann, Larissa, Paul, Paulina, Day, Michele E, Aratow, Michael, Bell, Douglas S, Doctor, Jason N, Hinske, Ludwig C, Jiang, Xiaoqian, Kim, Katherine K, et al. 2021\.  Privacy-protecting, reliable response data discovery using COVID-19 patient observations.  Journal of the American Medical Informatics Association, 28(8), 1765–1776.
  * Kiyasseh et al., (2020) Kiyasseh, Dani, Tadesse, Girmaw Abebe, Thwaites, Louise, Zhu, Tingting, Clifton, David, et al. 2020\.  Plethaugment: Gan-based ppg augmentation for medical diagnosis in low-resource settings.  IEEE journal of biomedical and health informatics, 24(11), 3226–3235.
  * Kiyasseh et al., (2021) Kiyasseh, Dani, Zhu, Tingting, & Clifton, David A. 2021.  Clocs: Contrastive learning of cardiac signals across space, time, and patients.  Pages 5606–5615 of: International Conference on Machine Learning.  PMLR.
  * Lee et al., (2020) Lee, Dongha, Yu, Hwanjo, Jiang, Xiaoqian, Rogith, Deevakar, Gudala, Meghana, Tejani, Mubeen, Zhang, Qiuchen, & Xiong, Li. 2020.  Generating sequential electronic health records using dual adversarial autoencoder.  Journal of the American Medical Informatics Association, 27(9), 1411–1419.
  * Lin et al., (2020) Lin, Zinan, Jain, Alankar, Wang, Chen, Fanti, Giulia, & Sekar, Vyas. 2020.  Using GANs for sharing networked time series data: Challenges, initial promise, and open questions.  Pages 464–483 of: Proceedings of the ACM Internet Measurement Conference.
  * Liu & Tuzel, (2016) Liu, Ming-Yu, & Tuzel, Oncel. 2016.  Coupled generative adversarial networks.  Advances in neural information processing systems, 29, 469–477.
  * Liu et al., (2017) Liu, Ming-Yu, Breuel, Thomas, & Kautz, Jan. 2017.  Unsupervised image-to-image translation networks.  Pages 700–708 of: Advances in neural information processing systems.
  * Lombardo & Moniz, (2008) Lombardo, Joseph S, & Moniz, Linda J. 2008.  Ta method for generation and distribution.  Johns Hopkins APL Technical Digest, 27(4), 356.
  * Marouf et al., (2020) Marouf, Mohamed, Machart, Pierre, Bansal, Vikas, Kilian, Christoph, Magruder, Daniel S, Krebs, Christian F, & Bonn, Stefan. 2020.  Realistic in silico generation and augmentation of single-cell RNA-seq data using generative adversarial networks.  Nature communications, 11(1), 1–12.
  * McLachlan et al., (2016) McLachlan, Scott, Dube, Kudakwashe, & Gallagher, Thomas. 2016.  Using the caremap with health incidents statistics for generating the realistic synthetic electronic healthcare record.  Pages 439–448 of: 2016 IEEE International Conference on Healthcare Informatics (ICHI).  IEEE.
  * Menger et al., (2019) Menger, Vincent, Spruit, Marco, Van Est, Roel, Nap, Eline, & Scheepers, Floor. 2019\.  Machine learning approach to inpatient violence risk assessment using routinely collected clinical notes in electronic health records.  JAMA network open, 2(7), e196709–e196709.
  * Menze et al., (2014) Menze, Bjoern H, Jakab, Andras, Bauer, Stefan, Kalpathy-Cramer, Jayashree, Farahani, Keyvan, Kirby, Justin, Burren, Yuliya, Porz, Nicole, Slotboom, Johannes, Wiest, Roland, et al. 2014\.  The multimodal brain tumor image segmentation benchmark (BRATS).  IEEE transactions on medical imaging, 34(10), 1993–2024.
  * Miotto et al., (2018) Miotto, Riccardo, Wang, Fei, Wang, Shuang, Jiang, Xiaoqian, & Dudley, Joel T. 2018\.  Deep learning for healthcare: review, opportunities and challenges.  Briefings in bioinformatics, 19(6), 1236–1246.
  * Mirza & Osindero, (2014) Mirza, Mehdi, & Osindero, Simon. 2014.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784.
  * Mogren, (2016) Mogren, Olof. 2016.  C-RNN-GAN: Continuous recurrent neural networks with adversarial training.  arXiv preprint arXiv:1611.09904.
  * Moody et al., (2020) Moody, B, Moody, G, Villarroel, M, Clifford, G, & Silva III, I. 2020.  MIMIC-III Waveform Database (version 1.0).
  * n2c2., (2021) n2c2. 2021.  National NLP Clinical Challenges (n2c2). 2022 n2c2 Shared Task and Workshop. Boston, MA: Harvard Medical School.  https://n2c2.dbmi.hms.harvard.edu/2022-challenge.  Accessed: 2021-12-08.
  * N3C., (2021) N3C. 2021.  N3C. Synthetic Data Workstream.  https://covid.cd2h.org/N3C_synthetic_data.  Accessed: 2021-12-02.
  * Pollard et al., (2018) Pollard, Tom J, Johnson, Alistair EW, Raffa, Jesse D, Celi, Leo A, Mark, Roger G, & Badawi, Omar. 2018.  The eICU Collaborative Research Database, a freely available multi-center database for critical care research.  Scientific data, 5(1), 1–13.
  * Rajkomar et al., (2019) Rajkomar, Alvin, Dean, Jeffrey, & Kohane, Isaac. 2019.  Machine learning in medicine.  New England Journal of Medicine, 380(14), 1347–1358.
  * Raket et al., (2020) Raket, Lars Lau, Jaskolowski, Jörn, Kinon, Bruce J, Brasen, Jens Christian, Jönsson, Linus, Wehnert, Allan, & Fusar-Poli, Paolo. 2020.  Dynamic ElecTronic hEalth reCord deTection (DETECT) of individuals at risk of a first episode of psychosis: a case-control development and validation study.  The Lancet Digital Health, 2(5), e229–e239.
  * Sanchez-Pinto et al., (2018) Sanchez-Pinto, L Nelson, Luo, Yuan, & Churpek, Matthew M. 2018.  Big data and data science in critical care.  Chest, 154(5), 1239–1248.
  * Schlegl et al., (2019) Schlegl, Thomas, Seeböck, Philipp, Waldstein, Sebastian M, Langs, Georg, & Schmidt-Erfurth, Ursula. 2019.  f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks.  Medical image analysis, 54, 30–44.
  * Shamout et al., (2019) Shamout, Farah E, Zhu, Tingting, Sharma, Pulkit, Watkinson, Peter J, & Clifton, David A. 2019.  Deep interpretable early warning system for the detection of clinical deterioration.  IEEE journal of biomedical and health informatics, 24(2), 437–446.
  * Shillan et al., (2019) Shillan, Duncan, Sterne, Jonathan AC, Champneys, Alan, & Gibbison, Ben. 2019.  Use of machine learning to analyse routinely collected intensive care unit data: a systematic review.  Critical care, 23(1), 1–11.
  * Shokri et al., (2017) Shokri, Reza, Stronati, Marco, Song, Congzheng, & Shmatikov, Vitaly. 2017.  Membership inference attacks against machine learning models.  Pages 3–18 of: 2017 IEEE Symposium on Security and Privacy (SP).  IEEE.
  * Simon et al., (2019) Simon, Gregory E, Shortreed, Susan M, Coley, R Yates, Penfold, Robert B, Rossom, Rebecca C, Waitzfelder, Beth E, Sanchez, Katherine, & Lynch, Frances L. 2019.  Assessing and minimizing re-identification risk in research data derived from health care records.  eGEMs, 7(1).
  * Suresh et al., (2017) Suresh, Harini, Hunt, Nathan, Johnson, Alistair, Celi, Leo Anthony, Szolovits, Peter, & Ghassemi, Marzyeh. 2017.  Clinical intervention prediction and understanding with deep neural networks.  Pages 322–337 of: Machine Learning for Healthcare Conference.  PMLR.
  * Sutherland et al., (2016) Sutherland, Danica J, Tung, Hsiao-Yu, Strathmann, Heiko, De, Soumyajit, Ramdas, Aaditya, Smola, Alex, & Gretton, Arthur. 2016.  Generative models and model criticism via optimized maximum mean discrepancy.  arXiv preprint arXiv:1611.04488.
  * Tucker et al., (2020) Tucker, Allan, Wang, Zhenchen, Rotalinti, Ylenia, & Myles, Puja. 2020.  Generating high-fidelity synthetic patient data for assessing machine learning healthcare software.  NPJ digital medicine, 3(1), 1–13.
  * Wan et al., (2020) Wan, Ziyu, Zhang, Bo, Chen, Dongdong, Zhang, Pan, Chen, Dong, Liao, Jing, & Wen, Fang. 2020.  Old photo restoration via deep latent space translation.  arXiv preprint arXiv:2009.07047.
  * Wang et al., (2019) Wang, Lu, Zhang, Wei, & He, Xiaofeng. 2019.  Continuous patient-centric sequence generation via sequentially coupled adversarial learning.  Pages 36–52 of: International Conference on Database Systems for Advanced Applications.  Springer.
  * Wang et al., (2020) Wang, Shirly, McDermott, Matthew BA, Chauhan, Geeticka, Ghassemi, Marzyeh, Hughes, Michael C, & Naumann, Tristan. 2020.  Mimiextract: A data extraction, preprocessing, and representation pipeline for mimic-iii.  Pages 222–235 of: Proceedings of the ACM Conference on Health, Inference, and Learning.
  * Watson et al., (2019) Watson, David S, Krutzinna, Jenny, Bruce, Ian N, Griffiths, Christopher EM, McInnes, Iain B, Barnes, Michael R, & Floridi, Luciano. 2019.  Clinical applications of machine learning algorithms: beyond the black box.  Bmj, 364.
  * Wilkinson et al., (2020) Wilkinson, Jack, Arnold, Kellyn F, Murray, Eleanor J, van Smeden, Maarten, Carr, Kareem, Sippy, Rachel, de Kamps, Marc, Beam, Andrew, Konigorski, Stefan, Lippert, Christoph, et al. 2020\.  Time to reality check the promises of machine learning-powered precision medicine.  The Lancet Digital Health.
  * Wirth et al., (2021) Wirth, Felix Nikolaus, Meurers, Thierry, Johns, Marco, & Prasser, Fabian. 2021\.  Privacy-preserving data sharing infrastructures for medical research: systematization and comparison.  BMC Medical Informatics and Decision Making, 21(1), 1–13.
  * Wu et al., (2017) Wu, Mike, Ghassemi, Marzyeh, Feng, Mengling, Celi, Leo A, Szolovits, Peter, & Doshi-Velez, Finale. 2017.  Understanding vasopressor intervention and weaning: risk prediction in a public heterogeneous clinical time series database.  Journal of the American Medical Informatics Association, 24(3), 488–495.
  * Yang et al., (2018) Yang, Qingsong, Yan, Pingkun, Zhang, Yanbo, Yu, Hengyong, Shi, Yongyi, Mou, Xuanqin, Kalra, Mannudeep K, Zhang, Yi, Sun, Ling, & Wang, Ge. 2018.  Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss.  IEEE transactions on medical imaging, 37(6), 1348–1357.
  * Yèche et al., (2021) Yèche, Hugo, Kuznetsova, Rita, Zimmermann, Marc, Hüser, Matthias, Lyu, Xinrui, Faltys, Martin, & Ratsch, Gunnar. 2021.  HiRID-ICU-Benchmark—A Comprehensive Machine Learning Benchmark on High-resolution ICU Data.
  * Yoon et al., (2019) Yoon, Jinsung, Jarrett, Daniel, & Van der Schaar, Mihaela. 2019.  Time-series generative adversarial networks.
  * Yu et al., (2019) Yu, Chao, Liu, Jiming, & Zhao, Hongyi. 2019.  Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units.  BMC medical informatics and decision making, 19(2), 111–120.
  * Yu et al., (2017) Yu, Lantao, Zhang, Weinan, Wang, Jun, & Yu, Yong. 2017.  Seqgan: Sequence generative adversarial nets with policy gradient.  In: Proceedings of the AAAI conference on artificial intelligence, vol. 31.
  * Zhang et al., (2021) Zhang, Ziqi, Yan, Chao, Lasko, Thomas A, Sun, Jimeng, & Malin, Bradley A. 2021\.  SynTEG: a framework for temporal structured electronic health data simulation.  Journal of the American Medical Informatics Association, 28(3), 596–604.
  * Zhou et al., (2020) Zhou, Kang, Gao, Shenghua, Cheng, Jun, Gu, Zaiwang, Fu, Huazhu, Tu, Zhi, Yang, Jianlong, Zhao, Yitian, & Liu, Jiang. 2020.  Sparse-gan: Sparsity-constrained generative adversarial network for anomaly detection in retinal oct image.  Pages 1227–1231 of: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).  IEEE.