# Synthesising Electronic Health Records: Cystic Fibrosis Patient Group

Emily Muller,1 Xu Zheng, 2 Jer Hayes 2

###### Abstract

Class imbalance can often degrade predictive performance of supervised
learning algorithms. Balanced classes can be obtained by oversampling exact
copies, with noise, or interpolation between nearest neighbours (as in
traditional SMOTE methods). Oversampling using augmentation, as is typical in
computer vision tasks, can be achieved with deep generative models. Deep
generative models are effective data synthesisers due to their ability to
capture complex underlying distributions. Synthetic data in healthcare can
enhance interoperability between healthcare providers by ensuring patient
privacy. Equipped with large synthetic datasets which do well to represent
small patient groups, machine learning in healthcare can address the current
challenges of bias and generalisability. This paper evaluates synthetic data
generators ability to synthesise patient electronic health records. We test
the utility of synthetic data for patient outcome classification, observing
increased predictive performance when augmenting imbalanced datasets with
synthetic data.

## Introduction

Massive amounts of data are generated in the field of medicine from sources
such as biosensors, physiological measurements, genome sequencing and
electronic health records (EHR). Machine learning is uniquely placed to
address pattern recognition in the medical field, whether helping to interpret
medical scans, predicting disease trajectory, improving health system
workflows or promoting patient data sovereignty (Topol 2019). Large datasets
that are representative and diverse are needed to mitigate bias. However,
aggregating data in this way can be; expensive, if the data is proprietary;
inaccessible, due to interoperability standards in the sharing of health data;
or illegal as it has the potential to violate privacy. To overcome this,
synthetic data are increasingly being used in the healthcare setting (Chen et
al. 2021). Synthetic data generation of EHRs for underrepresented patient
groups or small patient datasets can enable better predictive modelling while
preserving patient privacy. In this work, we collect Cystic Fibrosis EHRs and
synthesise new patient records. We examine the utility of four synthetic data
generators and augment the original data with synthetic data to enhance
predictability in patient outcomes.

Synthetic data generators are algorithms used generate new samples that
preserve the original data distribution while adhering to numerous properties,
namely: utility and privacy. Utility measures the effectiveness of the
synthetic data on another task, risk prediction for example. Privacy ensures
that the resulting synthetic data cannot be used to infer anything about any
single individual. Differential privacy provides an algorithmic formulation of
privacy (Dwork, Roth et al. 2014). Since our original data is anonymised EHRs
with non-sensitive features, we consider instead of privacy the requirement of
uniqueness: to not simply copy the input data.

Data augmentation has shown increased performance for image related deep
learning tasks(Shorten and Khoshgoftaar 2019). Traditionally, tabular
augmentation methods, such as Synthetic Minority Over-sampling Technique
(SMOTE) and its variants, synthesise tabular data as midpoints to k-nearest
neighbours (Chawla et al. 2002). Another simple and effective method of
synthetic data generation is through perturbation of the original data by
adding noise (winner of Hide-and-seek challenge, 2020 (Alaa et al. 2021)).
Deep learning methods for tabular data synthesis involve generative models
such as generative adversarial networks (GANs) and variational autoencoders
(VAEs) (Xu and Veeramachaneni 2018; Choi et al. 2017; Xu et al. 2019; Xie et
al. 2018; Yoon, Drumright, and Van Der Schaar 2020). These models implicitly
parameterise the multivariate distribution of the original data using deep
neural networks. In (Fiore et al. 2019), authors use a GAN to synthesise the
underrepresented class of fraudulent credit card cases, observing a maximum
increase in classification sensitivity of $3.4$ percentage points when
augmenting the small class 3 times its size ($0.55\%$ of the training set). A
conditional GAN (cGAN) has shown optimal performance on a set of 22 tabular
datasets (Douzas and Bacao 2018). However, conditioning on the outcome class
is not always desirable or available in practice. In an application of
augmenting classes of thermal comfort, the authors find 2 experiments where
synthetic data alone has a higher F1 score than the original training data.
Similar to (Fiore et al. 2019), the author synthesises the underrepresented
class hence making known the class label. Other work has shown the efficacy of
generative networks above traditional methods (Liu et al. 2019; Ngwenduna and
Mbuvha 2021; Engelmann and Lessmann 2020). However, we do not find any studies
reporting the similarity of the synthesisers to the original dataset. This is
especially relevant for deep generative models which can suffer mode collapse
and learn to copy the training data, thus, violating privacy assurances. The
high capacity of generative models makes them a good candidate for capturing
complex non-linear distributions in the original data, while their intractable
likelihood functions make evaluation difficult. We employ model-agnostic
sample-wise similarity metrics to evaluate model performance at each training
epoch. Similarity and uniqueness are calculated for each synthetic dataset and
we compare this to utility in a classification task. We find that synthetic
augmentation improves accuracy in two out of four models, while there are
trade-offs in similarity. In the next section, we detail the feature
extraction procedure, model selection, and similarity and uniqueness metrics.

## Method

The complex trajectory for Cystic Fibrosis patients requires close monitoring
of care and high-risk decision making, such as lung transplant referral.
Opportunities for individualised patient care using machine learning have
recently been put forward for the case of Cystic Fibrosis (CF) patients
(Abroshan et al. 2020). The authors highlight risk prediction and personalised
treatment recommendation, amongst others, as areas where data can be leveraged
by machine learning. Traditionally, research on risk prediction has used
regression models, see review article (Breuer et al. 2018). More recently,
machine learning method AutoPrognosis was able to outperform 3 regression
models and the clinical standard of measured forced expiratory volume
$FEV_{1}\%<30$ by taking a feature-agnostic approach and including a set of
115 patient registry variables (Alaa and Schaar 2018). We explore predictive
modeling for CF patients using synthetically augmented EHR records.

### Feature extraction.

Cystic Fibrosis patients are extracted from the IBM Explorys database. A total
of $10074$ patients are extracted, representing about 1/3 ($31199$) of all CF
patients in the US (USCFF 2020). Patients belong to two subgroups: having died
or having received a lung transplant, or having survived. We predict outcome
based on a set of features including: demographic, comorbidity, lung infection
and therapy variables. We find large discrepancies in the prevalance of
comorbidities, lung infections and therapy variables between our patient group
and those reported in the UK CF Registry (Alaa and Schaar 2018) and the US CF
Foundation (USCFF 2020) (see Appendix Table 1). Since the IBM Explorys
database are EHRs based on medical health insurance it is plausible that
entire diagnosis codes go unrecorded if they are irrelevant for a medical
claim. A lung transplant is an expensive procedure, hence we assume there is
no missingness in this outcome class. Missingness of features in this dataset
should be treated carefully for clinical research. However, since we are
evaluating synthetic data generators, we continue to use this CF patient
dataset with its caveats.

### Pre-processing.

We remove all demographic or measurement variables with greater than 50%
missingness and uncertain encoding, resulting in a binary matrix of 41
variables (see Appendix Table 1). In order to enhance synthetic diversity, we
remove all samples with no diagnosis codes and all duplicates. Our final
dataset has $3184$ patients, with $\sim 80\%$ belonging to the survived
subgroup.

### Synthetic Data Generators.

We create 4 synthetic datasets using VAE, Differentially Private GAN (DPGAN)
with two noise levels (Xie et al. 2018) and Conditional GAN (CTGAN) (Xu et al.
2019). Our VAE is trained with 2 hidden layers in both the encoder and decoder
and ELBO loss minimisation. DPGAN is trained as a typical minimax game between
the discriminator and generator where noise is added to the gradient during
training. We employ the cross-entropy derived loss. The authors show addition
of noise in this way guarantees a level or differential privacy. CTGAN
addresses sparsity and imbalanced categorical columns by sampling vectors
during training and introducing a condition for which the generator learns the
conditional distribution. We optimise each model using model-agnostic
similarity metrics (see Appendix B: Experimental Details).

### Similarity.

Similarity of the synthetic dataset with the original dataset is measured in
two ways. Fidelity measures the degree to which generated samples resemble the
original data and diversity measures whether generated samples cover the full
variability of the original data. The latter is particularly useful for
evaluating generative models which are prone to mode collapse. Precision and
recall metrics for evaluating generative models were originally proposed in
(Sajjadi et al. 2018). Precision is defined as the proportion of the synthetic
probability distribution that can be generated by the original probability
distribution, thus measuring fidelity, and recall symmetrically defines
diversity. Precision and recall are effectively calculated as the proportion
of samples which lie in the support of the comparative distribution. This
assumes uniform density across the support and therefore alternative metrics
for precision and recall have been proposed to ameliorate this (Kynkäänniemi
et al. 2019; Naeem et al. 2020; Alaa et al. 2021). We employ the original
precision and recall metrics along with density and coverage metrics from
(Naeem et al. 2020). Density and coverage addresses the lack of robustness to
outliers, failure to detect matching distributions and inability to diagnose
different types of distribution failure.

(a) Authenticity proportion

(b) Similarity metrics

(c) Synthetic data AUC-ROC

Figure 1: (A) Authenticity of $100k$ samples from each generator. Grey are
proportion of samples which appear in the original training data, with darker
grey (far left) representing those which are unique. Light purple are samples
which do not appear in the original training data, and darker purple represent
those which are unique. (B) Similarity metrics for each model. For each fold,
a dataset matching the size of the original fold with the equivalent
proportion of classes is sampled from the unique synthetic dataset (dark blue
only). This is repeated 10 times and similarity metrics show mean and standard
deviation over folds and repetitions.

### Uniqueness.

A synthetic data generator can achieve perfect fidelity and diversity by
copying the original training data. Privacy assurances are essential to
prevent leakage of personal information. Differential privacy is one such well
known and commonly researched assurance (Dwork, Roth et al. 2014).
Differentially private algorithms limit how much; the output can differ based
on whether input is included or not; one can learn about a person because
their data was included; and confidence about whether someone’s data was
included. In practice, there are various distance based metrics to gaurantee
such assurance. In (Alaa et al. 2021), the authors quantify a generated sample
as authentic if its closest real sample is at least closer to any other real
sample than the generated sample. Extending this to the case of binary
variables, we could consider hamming distance, however, since the IBM Explorys
data is de-personalised and non-identifiable, we assess our generators based
on how many exact copies are made. We calculate the uniqueness of each model
by generating a large finite number of samples, and reporting the percentage
overlap with the original training data, and the uniqueness of samples. In
post-processing, we remove all generated samples which copy the original
training data and remove all duplicates.

### Utility.

To empirically validate the utility of the generated dataset we introduce two
different training testing settings. Setting A: train the predictive models on
the synthetic training set, test the performance of the models on the real
testing set. Setting B: train on the synthetically augmented balanced-class
training set, test on the real testing set. We performance 5-fold cross
validation, sampling each fold with proportional representation of each class.
After training the synthesisers (described below), we obtain unique synthetic
datasets for each fold and for each model. Setting A was implemented as
follows: for each fold, we sampled a synthetic dataset the size of the
original dataset with equivalent class proportions to calculate utility. We
repeated this 10 times resulting in 50 measurements for each model for which
we report the mean and standard deviation. For each synthetic dataset we
trained the following classification models: Support Vector Machine, Linear
Regression, Naive Bayes, K-Nearest Neighbours and a Random Forest. We chose
the classifier which maximised area under the receiver operating curve (AUC-
ROC) for the hold out test set and compared this to training with the original
data. As well as testing utility on the synthetic dataset alone, we also
augmented the original dataset to obtain class balance (Setting B). Not all
synthesisers provided enough unique samples of the small class, and we
therefore attempted to upsample as far as possible, otherwise downsampling the
larger class. With a balanced class augmented training set, we performed the
same utility experiment as outlined above reporting both area under the
receiver operating curve (AUC-ROC) and accuracy.

### Synthesisers.

We optimised each synthesiser on the sum of precision, recall, density and
coverage for a random 5th of the dataset stratified by outcome class. After
hyperparameter tuning, we trained each model for each of the 5 separate
training folds and generated $100K$ synthetic samples from each. We post-
processed each synthetic dataset to obtain only unique samples. For each fold,
we sampled a synthetic dataset the size of the original dataset with
equivalent class proportions to calculate similarity. We repeated this 10
times resulting in 50 measurements for each metric and model for which we
report the mean and standard deviation (see Appendix Appendix B: Experimental
Details).

### Implementation.

Synthesisers were implemented on a nvidia-gpu. Utility models were implemented
on a CPU using python and the sklearn.classification module. Similarity
metrics were calculuted using the python module prdc (Naeem et al. 2020) with
$k=5$.

(a) AUC-ROC

(b) Accuracy

(c) Precision

(d) Recall

Figure 2: Performance of original data augmented with synthetic samples. For
each fold, synthetic samples are randomly drawn to augment the small class.
Where there are not enough of the small class, the large class is randomly
downsampled achieveing class balance. Gaussian Naive Bayes, Random Forest,
Neural Network, SVM and Logistic Regression classifiers are each trained to
predict the hold out test set outcome and results which maximise performance
metrics are reported, averaged over each fold and repeated 10 times.The
average number of samples for each model and class is $2128$, $419$, $1029$,
$1908$, $1974$ and $2128$ for Upsampling, Downsampling, VAE, DPGAN050,
DPGAN001, CTGAN respectively.

## Results

### Uniqueness.

Figure 1(a) shows the average per fold proportion of duplicated data from
generation of $100,000$ samples. Of $100,000$ generated samples, VAE has the
highest duplication rate of the original training data ($67\%$), followed by
the CTGAN ($34\%$), DPGAN001 ($32\%$) and DPGAN050 ($18\%$), as shown in dark
grey. Although DPGAN050 has the lowest duplication rate of the original
training data, its novel synthesised samples are largely duplicated (approx
$71$k duplicates shown in light purple), compared to $26$k, $55$k and $31$k
for VAE, DPGAN001 and CTGAN, respectively. CTGAN generates the largest number
of unique samples at an average of $33,689$ and VAE has the fewest with
$6,647$ (dark purple). Both DPGAN models generate a similar number of unique
samples ($12,481$ for DPGAN001 and $11,288$ for DPGAN050).

### Similarity.

Random subsets from each unique synthetic dataset are sampled and similarity
with the original data is computed (shown in Figure 1(b)). Batches are sampled
to match the size of the training data while preserving the outcome class
ratio. Synthesiser VAE has the highest precision ($0.95\pm 0.01$) and density
($0.63\pm 0.03$). Given that this model has the largest copying rate of the
original data, it follows that its unique data also lies closest to the
original data, preserving fidelity. Synthesiser CTGAN has the lowest fidelity
(both precision $0.37\pm 0.04$ and density $0.13\pm 0.02$) with the original
training data, however, attains the highest recall ($0.67\pm 0.02$). When
measuring diversity using coverage, CTGAN displays the largest degredation
($0.31\pm 0.02$) placing last with DPGAN050 ($0.31\pm 0.02$). This suggests
that many of the diverse samples generated by CTGAN are considered outliers
(do not match the density of the original distribution). VAE also displays a
marked reduction in diversity, compared to both DPGAN models, when measured
with recall ($0.58\pm 0.03$) versus coverage ($0.43\pm 0.01$). DPGAN001 scores
consistently higher than DPGAN050 across all similarity metrics ($0.64\pm
0.09$ vs $0.54\pm 0.05$ for precision, $0.46\pm 0.12$ vs $0.31\pm 0.02$ for
recall, $0.28\pm 0.05$ vs $0.22\pm 0.02$ for density and $0.44\pm 0.10$ vs
$0.31\pm 0.12$ for coverage).

### Classification.

None of the synthetic datasets reach an AUC-ROC that is higher than the
original training data. VAE has the highest AUC of all synthetic datasets,
which is in line with it ranking the highest across similarity metrics. Model
rank across the sum of similarity metrics is VAE, DPGAN001, CTGAN and finally
DPGAN050 which does not follow the ranking of AUC-ROC scores.

### Synthetic augmentation to balance classes.

We compared performance measures of a balanced dataset by both upsampling and
downsampling, to a balanced dataset augmented with synthetic samples (Figure
2). The ordering of AUC-ROC scores is preserved from the synthetic data only
classification task (Figure 2(a) versus Figure 1(c)) with a smaller difference
to the original baseline performance. In contrast, synthetically augmented
data from VAE and CTGAN outperform the baseline models when measured using
accuracy (Figure 2(b)). This increased performance is largely driven by recall
(true positive rate) (Figure 2(d)).

(a) Oversampled

(b) VAE

(c) DPGAN050

(d) DPGAN001

(e) CTGAN

(f) Oversampled

(g) VAE

(h) DPGAN050

(i) DPGAN001

(j) CTGAN

Figure 3: Top row shows heatmap of binary feature matrices for each
synthesised dataset. Class 0 is top half of matrix and Class 1 is bottom half.
Note that bottom half is from original data while top half is augmented with
synthetic data. This balanced dataset serves as input for the classification
algorithms as per Results section. The bottom row shows 2D PCA embeddings of
the same data.

## Discussion

Synthetic data augmented with the original data from models VAE and CTGAN
outperform the original data accuracy when tested on the hold out set. This is
driven by recall, indicating that these models are better able to identify the
positive class with little degradation to precision. On average VAE and CTGAN
make $511$ and $494$ predictions for the positive class while there are on
average $532$ out of $637$ positive samples in the test set. This compared to
DPGAN001 and DPGAN050 which make $300$ and $341$ predictions for the positive
class. These classification models have learnt a set of features which predict
each class with greater accuracy than both DPGAN models and simply resampling
(baseline). We observe that CTGAN has the greatest number of unique samples
(Figure 1(a)) and has the greatest diversity when measured using recall
(Figure 1(b)). This diversity of CTGAN lends itself to extracting
generalisable features which predict each class. While VAE produces far fewer
unique samples, it retains diversity in its samples. In order to better
understand the nature of the signal identified by these models, we have
plotted the heatmap of the binary feature matrix for one fold (see Figure 3).
In the top row of figures, the lower half of the heatmap is coming from the
original distribution, while the upper half is synthetic data augmented with
the original data for class balance. Here we observe that synthesiser VAE has
largely exaggerated the signal for features in the class 0. This has resulted
in greater separability in the input space, as shown by the 2D PCA reduction
in Figure 3(g). Reduced separability is displayed by both DPGAN models (Figure
3(i) and 3(h)) with DPGAN001 reporting a lower accuracy. The heatmaps go some
way in explaining this since there does not appear to be a marked difference
in the upper and lower halves of the DPGAN001 heatmap (Figure 3(d)). DPGAN050
however, does show some exaggerated features for class 0 (Figure 3(c)).
Visually, CTGAN appears to have the most similar heatmap as the original data
upsampled, while obtaining greater separability (Figure 3(j)). This visual
heuristic of similarity appears to be in contrast to the similarity metrics
reported in the previous section. This discrepancy could be on account of two
things. Since our similarity metrics measure both the positive and negative
outcome class in proportions with which they appear in the original training
set, the similarity of the larger positive class could have far outweighed
that of the smaller negative class. On the other hand, it may be that our
similarity metrics are simply not robust at capturing the true similarity of
our data. Another method for measuring precision and recall has been proposed
in (Alaa et al. 2021) displaying greater adherence to utility ranking than
those measures described in this paper. Unfortunately, we did not obtain code
for their metrics, called $\alpha$-precision and $\beta$-precision. These
metrics essentially limit the support of each distribution by defining an
$\alpha$ or $\beta$ radius on embedded hypersphere of data.

## Conclusion

We examine the utility, similarity and uniqueness of four synthetic data
generators and augment the original data with synthetic data to enhance
predictability in patient outcome. We observed increased accuracy in
performance for both VAE and CTGAN synthetic data generators. Similarity
metrics appear to go a little way in explaining the performance of synthetic
data generators. Furthermore, while the amplification of a signal in the
synthetic dataset may do poorly to preserve faithfulness of the original data,
it can provide greater separability hence predictive performance. Considering
also the uniqueness of each synthetic data generator, synthesiser CTGAN offers
both high predictive performance and uniqueness of samples. This is beneficial
for considering stricter conditions on privacy. These trade-offs are problem
specific and conclusions are to be arrived at based on clinical relevance. For
example, do exaggerated signals corroborate clinical evidence for co-occurence
of diseases? Given the caveats of our dataset, we have not shown the clinical
relevance of individual features. We have compared the efficacy of generative
models to augment imbalanced data for increased predictive performance.

## References

  * Abroshan et al. (2020) Abroshan, M.; Alaa, A. M.; Rayner, O.; and van der Schaar, M. 2020.  Opportunities for machine learning to transform care for people with cystic fibrosis.  _Journal of Cystic Fibrosis_ 19(1): 6–8.
  * Alaa and Schaar (2018) Alaa, A.; and Schaar, M. 2018.  Autoprognosis: Automated clinical prognostic modeling via bayesian optimization with structured kernel learning.  In _International conference on machine learning_ , 139–148. PMLR.
  * Alaa et al. (2021) Alaa, A. M.; van Breugel, B.; Saveliev, E.; and van der Schaar, M. 2021.  How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models.  _arXiv preprint arXiv:2102.08921_ .
  * Breuer et al. (2018) Breuer, O.; Caudri, D.; Stick, S.; and Turkovic, L. 2018.  Predicting disease progression in cystic fibrosis.  _Expert review of respiratory medicine_ 12(11): 905–917.
  * Chawla et al. (2002) Chawla, N. V.; Bowyer, K. W.; Hall, L. O.; and Kegelmeyer, W. P. 2002.  SMOTE: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ 16: 321–357.
  * Chen et al. (2021) Chen, R. J.; Lu, M. Y.; Chen, T. Y.; Williamson, D. F.; and Mahmood, F. 2021.  Synthetic data in machine learning for medicine and healthcare.  _Nature Biomedical Engineering_ 1–5.
  * Choi et al. (2017) Choi, E.; Biswal, S.; Malin, B.; Duke, J.; Stewart, W. F.; and Sun, J. 2017.  Generating multi-label discrete patient records using generative adversarial networks.  In _Machine learning for healthcare conference_ , 286–305. PMLR.
  * Douzas and Bacao (2018) Douzas, G.; and Bacao, F. 2018.  Effective data generation for imbalanced learning using conditional generative adversarial networks.  _Expert Systems with applications_ 91: 464–471.
  * Dwork, Roth et al. (2014) Dwork, C.; Roth, A.; et al. 2014.  The algorithmic foundations of differential privacy.  _Found. Trends Theor. Comput. Sci._ 9(3-4): 211–407.
  * Engelmann and Lessmann (2020) Engelmann, J.; and Lessmann, S. 2020.  Conditional wasserstein gan-based oversampling of tabular data for imbalanced learning.  _arXiv preprint arXiv:2008.09202_ .
  * Fiore et al. (2019) Fiore, U.; De Santis, A.; Perla, F.; Zanetti, P.; and Palmieri, F. 2019.  Using generative adversarial networks for improving classification effectiveness in credit card fraud detection.  _Information Sciences_ 479: 448–455.
  * Kynkäänniemi et al. (2019) Kynkäänniemi, T.; Karras, T.; Laine, S.; Lehtinen, J.; and Aila, T. 2019\.  Improved precision/recall metric for assessing generative models.  _arXiv preprint arXiv:1904.06991_ .
  * Liu et al. (2019) Liu, Y.; Zhou, Y.; Liu, X.; Dong, F.; Wang, C.; and Wang, Z. 2019.  Wasserstein GAN-based small-sample augmentation for new-generation artificial intelligence: a case study of cancer-staging data in biology.  _Engineering_ 5(1): 156–163.
  * Naeem et al. (2020) Naeem, M. F.; Oh, S. J.; Uh, Y.; Choi, Y.; and Yoo, J. 2020.  Reliable fidelity and diversity metrics for generative models.  In _International Conference on Machine Learning_ , 7176–7185. PMLR.
  * Ngwenduna and Mbuvha (2021) Ngwenduna, K. S.; and Mbuvha, R. 2021.  Alleviating Class Imbalance in Actuarial Applications Using Generative Adversarial Networks.  _Risks_ 9(3): 49.
  * Sajjadi et al. (2018) Sajjadi, M. S.; Bachem, O.; Lucic, M.; Bousquet, O.; and Gelly, S. 2018.  Assessing generative models via precision and recall.  _arXiv preprint arXiv:1806.00035_ .
  * Shorten and Khoshgoftaar (2019) Shorten, C.; and Khoshgoftaar, T. M. 2019.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ 6(1): 1–48.
  * Topol (2019) Topol, E. J. 2019.  High-performance medicine: the convergence of human and artificial intelligence.  _Nature medicine_ 25(1): 44–56.
  * USCFF (2020) USCFF. 2020.  Patient Registry Annual Data Report.  https://www.cff.org/media/23476/download.  Accessed: 07-12-2021.
  * Xie et al. (2018) Xie, L.; Lin, K.; Wang, S.; Wang, F.; and Zhou, J. 2018.  Differentially private generative adversarial network.  _arXiv preprint arXiv:1802.06739_ .
  * Xu et al. (2019) Xu, L.; Skoularidou, M.; Cuesta-Infante, A.; and Veeramachaneni, K. 2019.  Modeling tabular data using conditional gan.  _arXiv preprint arXiv:1907.00503_ .
  * Xu and Veeramachaneni (2018) Xu, L.; and Veeramachaneni, K. 2018.  Synthesizing tabular data using generative adversarial networks.  _arXiv preprint arXiv:1811.11264_ .
  * Yoon, Drumright, and Van Der Schaar (2020) Yoon, J.; Drumright, L. N.; and Van Der Schaar, M. 2020.  Anonymization through data synthesis using generative adversarial networks (ads-gan).  _IEEE journal of biomedical and health informatics_ 24(8): 2378–2388.
  * Yousefpour et al. (2021) Yousefpour, A.; Shilov, I.; Sablayrolles, A.; Testuggine, D.; Prasad, K.; Malek, M.; Nguyen, J.; Ghosh, S.; Bharadwaj, A.; Zhao, J.; Cormode, G.; and Mironov, I. 2021.  Opacus: User-Friendly Differential Privacy Library in PyTorch.  _arXiv preprint arXiv:2109.12298_ .

##  Appendix A Appendix A: Cystic Fibrosis Patient Descriptives

Table 1: Baseline characteristics of patients with Cystic Fibrosis in the IBM
Explorys EHR dataset. $\Delta$ columns show IBM statistics less statistics
from UK CF Registry from (Alaa and Schaar 2018) and the US CF Registry (USCFF
2020). Variables with strikethrough are not included in analysis either
because there is a high proportion of missingness or because the encoding is
uncertain (see footnotes). Codes used are available on request.

Variable |  Alive & no LTn = 8781 (%) |  Death/LT n = 1293 (%) |  UKCF$\Delta$ |  USCF$\Delta$ |  Variable |  Alive & no LTn = 8781 (%) |  Death/LT n = 1293 (%) |  UKCF$\Delta$ |  USCF$\Delta$
---|---|---|---|---|---|---|---|---|---
Gender(%male) |  4,925 (56.1) |  622 (48.1%) |  |  |  Pancreatic |  |  |  |
Age (years)§ | 55.6% missing |  |  |  Cirrhosis |  125 (1.4) |  32 (2.5) |  (1.2) |  (1.6)
Height (cm)§ | 81.0% missing |  |  |  Liver Disease |  309 (3.5) |  68 (5.3) |  (12.5) |  (-0.1)
Weight (kg)§ | 80.0% missing |  |  |  Pancreatitisc |  11 (0.1) |  2 (0.2) |  (1.3) |  (0.9)
BMI (kg/m2)§ | 81.0% missing |  |  |  Liver Enzymesd |  5 (0.05) |  0 (0) |  (15.1) |
Genotype |  |  |  |  |  Gall Bladder |  96 (1.1) |  17 (1.3) |  (-0.6) |
Homozygous | Not EHR |  |  |  GI Bleed (variceal) | Not found |  |
Heterozygous | Not EHR |  |  |  Gastrointestinal |  |  |  |
$\Delta$F508 | Not EHR |  |  |  GERD |  1675 (19.1) |  269 (20.8) |  (1.6) |  (17.4)
G551D | Not EHR |  |  |  GIB (no variceal) |  168 (2.0) |  43 (3.3) |  (-2.0) |
Class I | Not EHR |  |  |  Intestinal Obstruction |  358 (4.1) |  125 (9.7) |  (3.6) |
Class II | Not EHR |  |  |  Musculoskeletal |  |  |  |
Class III | Not EHR |  |  |  Arthropathy |  440 (5.0) |  62 (4.8) |  (4.6) |  (-1.8)
Class IV | Not EHR |  |  |  Bone Fracture |  131 (1.5) |  24 (1.9) |  (-0.4) |  (-1.3)
Class V | Not EHR |  |  |  Osteopenia |  4 (0.05) |  1 (0.1) |  (20.6) |  (9.9)
Class VI | Not EHR |  |  |  Other |  |  |  |
Spirometry § |  |  |  |  |  Cancer |  17 (0.2) |  6 (0.5) |  (0.1) |  (-0.1)
FEV1 (L) | 99.3% missing |  |  |  Diabetes |  1592 (18.1) |  245 (18.9) |  (9.0) |
FEV1% | 97.6% missing |  |  |  CFRD |  789 (9.0) |  153 (11.8) |  (23.2) |  (9.3)
Best FEV1 (L) | 99.3% missing |  |  |  Pulmonary Abscess |  32 (0.4) |  15 (1.2) |  (-0.4) |
Best FEV1% | 98.7% missing |  |  |  Chr. Pseudomonas |  549 (6.3) |  148 (11.4) |  (49.5) |
FEV1% (2017) | 99.5% missing |  |  |  Osteoporosis |  497 (5.7) |  129 (10.0) |  (3.4) |  (-2.1)
FEV1% (2016) | 99.5% missing |  |  |  AICU | Not found |  |
FEV1% (2015) | 99.5% missing |  |  |  Kidney Stones |  439 (5.0) |  79 (6.0) |  (-3.6) |  (-4.4)
FEV1% (2014) | 99.5% missing |  |  |  Cough Fracture |  53 (0.6) |  10 (5.5) |  (-0.5) |
Lung Infections |  |  |  |  |  Hypertension |  1784 (20.3) |  285 (0.22) |  (-14.9) |  (-14.6)
B. Cepaciaa |  7 (0.01) |  2 (0.1) |  (5.2) |  (1.6) |  A.Mycobacteria |  132 (1.5) |  18 (1.4) |  (2.0) |  (8.5)
P. Aeruginosa |  686 (7.8) |  172 (13.3) |  (52.8) |  (23.7) |  Hearing Loss |  255 (2.9) |  59 (4.6) |  (-0.4) |  (-0.5)
MRSA |  831 (9.5) |  172 (13.3) |  (-5.6) |  (9.7) |  Depression |  1203 (13.7) |  161 (12.5) |  (-5.8) |  (3.4)
Aspergillus |  132 (1.5) |  62 (4.8) |  (10.5) |  |  Inhaled Antibiotics | Not found |  |
NTM |  80 (1.0) |  6 (0.5) |  (4.2) |  (9.0) |  Muco-active Therapy |  |  |  |
H. Influenza |  72 (0.8) |  25 (1.9) |  (4.1) |  (9.1) |  DNase |  252 (2.9) |  17 (1.3) |  (55.2) |  (88.8)
E. Coli |  203 (2.3) |  50 (3.9) |  (-2.0) |  |  Hypertonic Saline |  1038 (11.8) |  209 (16.1) |  (11.6) |  (61.3)
K. Pneumoniae |  72 (0.8) |  26 (2.0) |  (-0.6) |  |  Promixine |  9 (0.1) |  2 (0.2) |  (20.5) |
Gram-negative |  0 |  0 |  (0.5) |  |  Tobramycin |  365 (4.2) |  70 (5.4) |  (-0.9) |  (61.0)
ALCA |  37 (0.4) |  4 (0.3) |  (2.3) |  |  iBuprofen |  433 (5.0) |  32 (2.5) |  (-4.5) |  (-3.8)
Staph. Aureus |  323 (3.7) |  89 (6.9) |  (19.9) |  (52.9) |  Oral Corticosteroids | Not found |  |
Xanthomonasb |  60 (0.7) |  9 (0.7) |  (3.2) |  |  IV Antibiotics | Not found |  |
B. Multivorans | This is a UK concept |  |  |  IV Antibiotic Courses |  |  |  |
B. Cenocepacia | This is a UK concept |  |  |  Days at Home | Not found |  |
Pandoravirus | Not found |  |  |  Days at Hospital | Not found |  |
Comorbidities |  |  |  |  |  Non-IV Hospitalization | Not found |  |
Respiratory |  |  |  |  |  Non-IV Ventilation | Not found |  |
ABPA |  138 (1.6) |  17 (1.3) |  (10.8) |  (3.4) |  Oxygen Therapy | Not found |  |
Nasal Polyps |  269 (3.1) |  35 (2.7) |  (0.0) |  (6.8) |  Continuous | Not found |  |
Asthma |  1604 (18.3) |  151 (11.7) |  (-2.0) |  (13.4) |  Nocturnal | Not found |  |
Sinus Disease |  750 (8.5) |  110 (8.5) |  (4.5) |  (1.4) |  Exacerbation | Not found |  |
Hemoptysis |  398 (4.5) |  85 (6.6) |  (-3.2) |  (-2.8) |  Pro re nata | Not found |  |

a LOINC code (44800-1) refers to genus Burkholderia and does not specify
Cepacia. b Includes Stenotrophomonas maltophilia and not exhaustive of
Xanthomonas species. c d e Finding not diagnosis.

##  Appendix B Appendix B: Experimental Details

### Model Architectures.

We implement a differentially private GAN using Opacus pytorch library
(Yousefpour et al. 2021). This adds gaussian noise directly to the parameter
gradients as detailed in (Xie et al. 2018). Our generator is made up of 4
blocks followed by a fully connected layer and tanh activation function (as
shown in Figure 4, discriminator architecture shown in Figure 5). We train the
model with a binary cross entropy loss.

$z_{\mathcal{N}(0,1)}$$\text{B1}_{z,128}$$\text{B2}_{128,256}$$\text{B3}_{256,512}$$\text{B4}_{512,512}$$\text{FC}_{512,41}$tanh$\tilde{x}$Block$\text{FC}_{in,out}$dropoutbatch
normReLu Figure 4: Generator of DPGAN synthesiser takes a $z$-dimensional
noise vector as input.
$\tilde{x}$$\text{FC}_{\tilde{x},512}$ReLu$\text{FC}_{512,256}$dropoutReLu$\text{FC}_{256,1}$sigmoid
Figure 5: Discriminator architecture of DPGAN synthesiser.
$x$$\text{FC}_{41,h}$ReLudropout$\text{FC}_{h,h}$ReLu$\text{fc}_{h,z}$$z$sigmoid$\text{FC}_{h,41}$ReLu$\text{FC}_{h,h}$dropoutReLu$\text{FC}_{z,h}$$\tilde{x}$
Figure 6: Variational Autoencoder Synthesiser architecture.

We implement CTGAN by adopting the model architecture provided in (Xu et al.
2019) and the corresponding code repo. Our variational autoencoder model is
shown in Figure 6 trained with a binary cross entropy reconstruction loss and
Kullback Liebler (KL) regularisation.

### Hyperparameter Optimisation.

We perform hyperparameter optimisation (HO) for the models using grid search
with optimisation objective to maximise the sum of precision, recall, density
and coverage. We performed optimisation a stratified (by outcome class) sample
of $80\%$ of the data. The final parameters for DPGAN001 are batch size of
$150$, latent dimensions $32$, sigma $0.01$, gradient clip $0.1$, lr $0.002$
with decay $0.3$ and $0.999$. For DPGAN050 we used batch size of $50$, latent
dimension $10$, sigma $0.5$, gradient clip $0.05$, lr $0.002$ with decay $0.8$
and $0.8$. For the VAE we have batch size $150$, latent dimension $16$, hidden
dimension $500$ and lr $0.001$. CTGAN has batch size $300$, generator latent
dimension $50$, discriminator latent dimension $512$ and embedding dimension
$z=32$. For each generator update we perform $7$ discriminator updates and
group $20$ samples together when applying discriminator. We use discriminator
learning rate of $0.0002$ and decay $1e-6$, and generator learning rate of
$0.002$ and decay of $1e-7$.

In Table 2 we observe that the maximum achieved from HO is very similar as
that from each fold for both DPGAN models. VAE obtains slightly higher than
HO, whereas CTGAN is considerably lower. That CTGAN has not obtained the
maximum PRDC from HO could be due to loss of generalisability across folds
since unstable combinations of parameters achieve maximum. We have not
investigated this further since the training curved across folds appear stable
(see Figure 7).

Model | PRDC (HO) | PRDC folds
---|---|---
DPGAN001 | [1.35,2.75] | $2.80\pm 0.03$
DPGAN050 | [0.84,2.04] | $2.00\pm 0.20$
CTGAN | [1.18,2.72] | $2.07\pm 0.04$
VAE | [2.12,2.63] | $2.78\pm 0.05$
Table 2: Range of PRDC values from hyperparametisation optimisation (HO).
Final mean and std deviation reported in the final column. Figure 7: PRDC
metrics per training epoch for each model and fold. VAE is shown in pink,
DPGAN001 in green, DPGAN050 in blue and CTGAN in orange.