segmentation and classification. Using a PGGAN, [235] augments data for
segmenting prostate and organs in 3D pelvic CT data. Building on this, [236]
employs PGGANs for unsupervised synthesis of high-resolution CT scans,
yielding outputs that are often indistinguishable from real CT scans, as
demonstrated in a comprehensive Visual Turing test study. However, challenges
persist in accurately generating smaller anatomical structures, indicating
limitations in current methods due to homogeneous data training. Addressing
this, [65] introduces a hybrid model using a DDPM with a Swin-transformer-
based network for abdomen and pelvic CT scan generation, achieving state-of-
the-art performance, in terms of synthetic image quality and diversity,
compared to existing generative models such as WGAN, PGGAN, SGAN and DDPM.
Challenges exist in extending this approach to 3D synthesis due to complexity
and resource-intensive requirements.

Numerous applications in CT imaging revolve around generating synthetic lung
and chest CT images, such as [237] and [238], utilizing a 3D CT-GAN, enabling
a gradual generation of CT volumes by generating sub-component slices and
slabs, reducing the GPU memory requirements. Interest in clinical applications
of unsupervised and/or unconditional generative methods has expanded to
include COVID-19 detection [239, 240, 241]. [239] presents a 3D cycleGAN
framework tailored for generating non-contrast CT images from contrast-
enhanced ones, leveraging a sizable dataset of more than 1000 COVID-19 scans.
In [241], the authors explore COVID-19 detection in CT scans using vision
transformers, augmented with synthetic images from SAGAN-ResNet. [240] delves
into the challenges of employing large diffusion models, proposing a
lightweight variant with a U-Net architecture and step information injection,
stressing the importance of specific operations in encoder-decoder models to
maintain synthetic image quality.

Recent advancements in exploring conditional models have emphasized the
injection of greater control into the generation process, resulting in the
production of higher quality synthetic images. [242] leverages a VAE to
compress image space, and then conditions a latent diffusion model on its
output to synthesize CT images for the denoising of LDCT images. [243]
introduces MedGen3D, a novel diffusion-based generative framework, which
utilizes conditioning on mask sequences generated by a Multi-Condition
Diffusion Probabilistic Model (MC-DPM), considering both the position or
direction in which sequences are generates (forward or backward), as well as
the sequence content. These conditions guide the synthesis of realistic
medical images aligned with masks while maintaining spatial consistency across
adjacent slices.

Conditioning on label maps has been thoroughly explored to guide the accurate
anatomical synthesis of tissues in chest and brain [244], liver [245], as well
as lung CT [246, 247, 248, 249, 67]. [244] benchmarks variations of PGGAN and
StyleGAN for synthetic medical image generation, revealing insights into their
performance under variable conditions, such as the variation in label
combinations, sample size and spatial resolution. Reducing class numbers
improves synthetic data quality, but caution is needed to avoid overfitting on
rare classes, while refining label conditioning mechanisms is crucial for
enhancing overall predictive performance. [245] presents a 3D GAN framework
for liver CT image synthesis using vascular segmentation masks and region-
based weight-balancing with a stable Multiple Gradient Descent Algorithm
(MGDA). While promising, challenges remain in accurately labeling vascular
structures, underscoring the need for more explicit information for precise 3D
liver image synthesis. Variations of the Pix2Pix model are commonly used for
synthesizing lung images with diverse pathological cases [246], generating
free-form lesion images from tumor sketches using StyleGAN concepts for
accurate style translation [67], and lung nodule synthesis [247] for data
augmentation, employing window-guided semantic learning to control the
generation of smaller semantic features. [248] investigates InfoGAN for
generating shape-controlled tumor images and evaluates their potential for
histological classification of lung cancer with CT scans. InfoGAN outperforms
WGAN in controlling tumor size and chest wall presence, improving image
features. However, both GANs still fall short of fully capturing the entire
distribution of real images, likely due to differences in generated tumor
shapes. [249] explores Semantic Diffusion Models (SDM) for generating high-
quality pulmonary CT images compared to other SOTA approaches, such as SPADE
GANs, while prioritizing lung nodule detection.

Text-to-image synthesis is gaining traction for diversifying data
augmentation, offering text-controlled generation of synthetic images to
address challenges like rare case simulation and privacy concerns by creating
images without the need for sharing sensitive or identifiable data, typically
sourced from real images. [250] utilizes radiology reports for high-resolution
3D medical image synthesis, starting with low-resolution synthesis guided by
tokenized reports and up-scaled using a super-resolution module. [251]
introduces a text-conditional CT generation method, combining a pre-trained
language model, transformer-based 3D chest CT generation, and a super-
resolution diffusion model. Limitations include the lack of benchmarks,
reliance on 2D super-resolution, and considerations for model generalization,
applicability to non-chest tissues, and clinical relevance evaluation.

Application | Type | Modality | Technology | Paper | DT | Evaluation | Code
---|---|---|---|---|---|---|---
U | F | D | Q | C | P
Train | Test
Unconditional | GAN | Pelvic CT | PGGAN | [235] | Seg. | R,S | R |  |  |  | ✓ |  |
Whole body CT | PGGAN | [236] | None |  |  |  |  | ✓ |  |  |
Lung CT | CT-SGAN with BiLSTM | [237] | Det. | R,S,A | R | ✓ |  |  |  |  |
3D CT-GAN | [238] | Class. | R,A | R | ✓ |  | ✓ |  |  |
COVID19 | SA-GAN ResNet | [241] | Det. | R,A | R |  |  |  |  |  | ✓
3D patch-based cycle-GAN | [239] | Class. | R | R, S | ✓ |  | ✓ |  |  |
DM | Abd. & pelvic CT | MT-DDPM | [65] | Class. | R,S,A | R | ✓ | ✓ | ✓ |  |  |
COVID19 | U-Net DM | [240] | None |  |  | ✓ |  |  |  |  |
Class or Label Cond. | GAN | Brain CT | PGGAN, cpd-GAN | [244] | Class. | R,S | R | ✓ |  | ✓ |  | ✓ | ✓
Liver CT | U-Net patch-GAN | [245] | None |  |  | ✓ | ✓ |  | ✓ |  |
Lung CT | Pix2Pix | [246] | None |  |  | ✓ |  | ✓ |  |  |
Info GAN | [248] | Class.* | R,S | R |  |  |  |  |  |
Style Pix2Pix | [67] | Class. | R,S,A | R | ✓ | ✓ |  |  |  |
MGGAN + Pix2Pix + WGSLN | [247] | Class. | R,A | R | ✓ |  |  | ✓ |  |
DM | SDM | [249] | Det. | R,A | R | ✓ |  |  |  |  |
Attribute Cond. | DM | Abd. & head CT | CLDM | [242] | None |  |  | ✓ |  |  |  |  |
Thoracic CT | Cond. DPM + Semantic DM | [243] | Seg.* | R,S,A | R | ✓ | ✓ |  |  |  |
Text-guided | DM | Lung CT | BERT + cond. DDPM | [250] | None |  |  | ✓ |  |  | ✓ |  | ✓
Hybrid | Chest CT | LLM, ViT, transf.-MaskGT | [251] | Class. | R,S,A | R | ✓ |  |  |  |  | ✓
Intra-modal trans. | GAN |  CBCT $\rightarrow$ CT | R2ACNN | [252] | None |  |  | ✓ |  |  | ✓ |  |
CT $\rightarrow$CTA | DCT-GAN | [253] | None |  |  | ✓ |  |  |  |  | ✓
NDCT $\rightarrow$ LDCT | GAN-NETL | [254] | None |  |  | ✓ |  |  | ✓ |  | ✓
dCT $\rightarrow$ CT | Pix2Pix | [255] | None |  |  | ✓ |  |  | ✓ |  | ✓
CBCT $\rightarrow$ CT | Attention GAN | [256] | None |  |  | ✓ |  |  | ✓ |  |
COVID19 | StarGANv2 | [257] | Seg. | R,A | R |  |  |  |  |  |
sRD-GAN | [66] | None |  |  | ✓ | ✓ | ✓ | ✓ |  |
Cycle GAN | [258] | Seg. | R,S | R |  |  |  | ✓ |  |
DM |  CBCT $\rightarrow$ CT | Cond. DDPM | [259] | None |  |  | ✓ |  |  | ✓ |  |
Transf. |  insp. $\rightarrow$ exp. CT | SWIN | [260] | None |  |  | ✓ | ✓ |  | ✓ |  |
Inter-modal trans. | GAN | MR $\rightarrow$ CT | CTF GAN (Pix2Pix) | [261] | Seg. | A | R | ✓ |  |  |  |  |
3D Pix2Pix | [262] | None |  |  |  |  |  | ✓ |  |
Pix2Pix + StyleGAN | [263] | None |  |  | ✓ |  |  |  |  |
Multi-cycle GAN | [264] | None |  |  | ✓ |  |  |  |  |
U-Net cycleGAN | [265] | None |  |  | ✓ |  |  | ✓ |  |
Cycle GAN + attention | [266] | None |  |  | ✓ |  | ✓ |  |  |
Cycle GAN | [267] | None |  |  | ✓ |  | ✓ |  |  |
3D Cycle GAN | [268] | None |  |  |  |  |  | ✓ |  |
Pix2Pix | [269] | None |  |  | ✓ |  |  | ✓ |  |
Cycle GAN, structGAN, Flow CGAN | [270] | None |  |  |  |  |  | ✓ |  |
U-Net-based WGAN | [271] | None |  |  | ✓ |  |  | ✓ |  |
PET/MRI $\rightarrow$ CT | Patch GAN | [272] | None |  |  | ✓ | ✓ |  |  |  |
US $\rightarrow$ CT | cGAN + SRGAN | [273] | None |  |  | ✓ |  |  | ✓ |  |
DM | MR $\rightarrow$ CT | SynDiff | [274] | None |  |  | ✓ |  |  |  |  | ✓
DDPM + SDE | [275] | None |  |  | ✓ |  |  |  |  |
DDMM | [276] | None |  |  | ✓ |  |  |  |  |
Hybrid | Swin-VNET | [277] | None |  |  | ✓ |  |  | ✓ |  |
RTCGAN-CNN/transf. | [278] | None |  |  | ✓ |  | ✓ | ✓ |  |
Table 10: Overview of all summarized and discussed studies on CT synthesis
with information about the technology utilized, the downstream task (DT), the
evaluation procedure and code availability. Evaluation is split according to
the evaluation metrics reported, covering utility (U), whereby automated
machine learning models are trained with real (R), synthetic (S) or augmented
with synthetic data (A), and tested on real or synthetic data, fidelity (F),
diversity (D), visual quality (Q), clinical utility (C) and privacy (P)
evaluation. * stands for studies that have utilized synthesized images for
pre-training.

Applications of generative models for LDCT synthesis include [254], which
introduces a GAN with Noise Encoding Transfer Learning (GAN-NETL) to generate
paired Normal-dose CT (NDCT) and LDCT datasets, addressing variations in LDCT
noise across scanners. For MRI-only radiotherapy, [261] proposes joint
synthesis and segmentation using a Pix2Pix GAN, transferring semantic
information for anatomical structure correction in synthetic CT images.
Additionally, [262] extends the Pix2Pix generator for 3D CT synthesis from
T1-weighted MR images within a patch-based conditional GAN, showing high
correlation with real CT.

To tackle the lack of paired data, unpaired synthesis with cycle GANs has
become popular across various tasks, including generating synthetic abdominal
and pelvic CT [235, 65], prostate [267], lumbar spine [268] in the realm of
inter-modal translation, and COVID-19 [67, 257, 258] in intra-modal
translation applications. To address structural consistency issues between MRI
and CT images inherent in cycle GAN-based synthesis for inter-modal
translation, [264] propose a multi-cycle GAN with a pseudo-cycle consistent
module and domain control module. This ensures high-quality and realistic CT
image generation. Similarly, [270] introduce augmented cycle consistent GANs,
injecting structural information and optical flow consistency constraints.
[266] present a paired-unpaired unsupervised attention-guided GAN, combining
Wasserstein GAN adversarial loss with content and L1 losses, capturing fine
structures and improving global consistency. Additionally, to mitigate spatial
inconsistencies often generated by 2D cycle GANs in synthetic 3D CT images,
[265] employ a double U-Net cycle GAN for 2.5D synthesis.

Cone beam CT (CBCT) images are crucial for image-guided therapy but often
suffer from low contrast and artifacts, impacting adaptive radiotherapy dose
calculations. Synthetic CT addresses these challenges, with [256] using an
attention-guided GAN for unpaired low-dose CBCT to CT synthesis, emphasizing
artifact correction due to X-ray scatter and respiratory movements. Similarly,
[252] introduces a sequence-aware contrastive generative network to enhance
CBCT image quality. Other approaches include [253] using a cascaded GAN for
synthesizing plain CT images into CTA images, [279] applying styleGAN2 with a
tailored loss function for breast CT images, and [263] manipulating
styleGAN2’s latent space for diverse CT image generation.

Intra- and inter-modality translation models, like [274], propose a cycle-
consistent architecture for diffusion-based synthesis between CT and MRI
scans. Similarly, [275] explores diffusion and score-matching models for CT-
MRI image conversion, showcasing superior image quality with faithful
anatomical details and avoiding over-smoothing and artifacts. Additionally,
[276] introduces a denoising diffusion model for cross-modal medical image
synthesis, allowing for adjustment of CT projection numbers and enhancing CT
image fidelity. Meanwhile, [277] proposes a method for generating CT scans
from MRI data, employing a transformer-based DDPM with a shifter-window
transformer network for the diffusion process, facilitating the synthesis of
synthetic CT from MRI.

While many studies use fidelity metrics, comprehensive evaluations of newly
proposed models, especially those employing diffusion or transformer-based
approaches, are lacking [240, 242, 243, 249, 275, 277]. The practical utility
of generated images in downstream tasks is often overlooked, raising concerns
about clinical applicability. A crucial need for realistic synthetic medical
images is highlighted, emphasizing the importance of a visual Turing test with
radiologists [65, 246, 266]. However, these tests often involve few
participants and limited samples, introducing biases, and challenges,
particularly in assessing 2D slices representing 3D data.

Examining model limitations, studies like [238, 246] reveal the presence of
artificial features and challenges in the realistic synthesis of certain
anatomical regions, such as the thoracoabdominal junction. [238] suggests
modifications to the visual Turing test and stresses involving multiple
radiologists for robust evaluations. Insights from [244] indicate challenges
in discriminating between real and synthetic images at lower spatial
resolutions, underlining the importance of image quality and label accuracy in
realistic synthesis. Interestingly, classification accuracy improves with
higher spatial resolution due to detailed features and visual artifacts,
aligning with similar findings. These considerations emphasize ongoing
challenges in evaluating synthetic medical images, highlighting the need for a
standardized approach in assessing generative models in clinical contexts.

Metrics like FID and MMD, though effective for natural images, may overlook
crucial anatomical details in medical images, as evidenced by the small gap
between methods. A clinically relevant evaluation is thus pivotal in
introducing synthetic images into the medical field. CT synthetic images are
primarily assessed by calculating differences in Hounsfield units per organ
[235, 247, 252, 256, 259, 272, 273, 267, 277, 278], crucial for reflecting
tissue density on CT scans. Given CT’s significance in radiation oncology,
dose distribution comparison is common [252, 255, 256, 262, 267, 269, 271,
273]. Additional studies evaluate organ contour quality and target volumes for
planning and dose calculation [245, 256, 269, 273, 278]. A subset of studies
delves into clinically-relevant anatomical measurements from synthetic images,
showcasing diverse applications. [260] calculates biomarkers for air trapping
and emphysema, [262] compares skull properties for transcranial ultrasound
procedures, and [268] assesses errors in lumbar spine synthetic CT images.
[250] conducts an in-depth analysis of diffusion-based text-guided image
synthesis, evaluating anatomical preservation in lung vessels and airways.
This method, utilizing specific input prompts from real radiology reports,
enhances the evaluation of text-based models, considering clinical indices
like pleural effusion and cardiothoracic ratio. Additionally, [272] examines
attenuation correction in PET images from synthetic CT, compared to current
MRI methods.

These methods offer insights into generative model effectiveness for clinical
use, surpassing traditional metrics. However, integrating clinical information
into the generation process remains uncommon, and evaluating privacy
preservation with synthetic data is seldom done. Notably, [244] stands out as
one of the few studies employing nearest neighbor matching and cosine distance
to demonstrate that GAN models may inherently offer privacy through stochastic
gradient descent. Nevertheless, the study emphasizes the imperative need to
formulate and quantify privacy guarantees.

Lastly, a crucial approach in [280] uniquely advocates for employing synthetic
data in AI model validation, utilizing it as a part of the training process to
prevent overfitting and facilitate model selection. The study introduces
synthetic tumors in an extensive validation dataset, incorporated into a
continual learning framework. Models trained and validated on dynamically
expanding synthetic data consistently outperform those relying solely on real-
world data for tumor segmentation. The findings highlight the efficacy of
synthetic data in mitigating overfitting and supporting early cancer
detection, suggesting its potential for large-scale testing, especially in
scenarios where obtaining real data is challenging, as is often the case in
clinical settings.

####  3.3.6 X-Ray

Common applications of X-ray image synthesis (Table 11) involve generating
images with pathological features like lesions in Chest X-rays (CXR). Lesions
vary in shape and size, posing challenges for detection, especially in deep
learning methods reliant on extensive training data. Generating X-ray images
of COVID-19 patients has also attracted significant interest, with various
approaches using different conditional and unconditional methods to enhance
the diversity of the COVID-19 patient population, leading to more accurate
detection. Consequently, translating between X-rays of normal cases and those
showing indications of pneumonia or tumors is frequently performed to improve
the performance of anomaly detection algorithms.

Common methods for unconditional synthesis of X-ray images involve WGAN-GPs
[281, 282], DCGANs [282, 283], and PGGANs [283, 284, 285], known for high-
resolution synthetic image generation. [281] pioneers WGANs for realistic knee
joint X-rays, validated by 15 medical experts, showing notable realism and
improved osteoarthritis severity classification accuracy. [282] notes DCGAN’s
instability during COVID-19 CXR generation. [286] stabilizes training with
multi-scale gradient flow and self-attention mechanisms. [287] explores co-
evolutionary learning, and [288] leverages a PGGAN for chest X-ray synthesis,
useful in cardiomegaly [289] and abnormality detection [285]. While [281] and
[283] investigate the utility of synthetic images for osteoarthritis severity
and abnormality classification across various training scenarios, other
approaches mainly focus on augmenting with synthetic data alone [282, 285].
[289] suggests synthetic data for cardiothoracic ratio calculation in
cardiomegaly. Despite using FID scores [282, 283, 286, 287], studies often
lack detailed downstream task analysis [281, 282, 285, 288, 289]. [283]
attempts downstream task analysis, but the evaluation lacks detail, raising
concerns about practical applicability in clinical settings.

Recent studies exploring conditional approaches for data generation include
employing a DDPM-based model with shared latent noise for semantic consistency
in synthesizing image/label pairs [290], as well as latent class optimization
for synthesis of diverse pathologies [284]. In parallel, [291] synthesizes
pathology by translating disease-containing medical images into disease-free
ones using a cycleGAN and a domain label as a conditional input. In the
evaluation process, both [290] and [284]extensively assess synthetic images,
including a thorough examination in downstream segmentation tasks, such as
semantic and instance-aware segmentation, for both in-domain and out-of-domain
samples. Furthermore, [284] stands out as one of the first studies to delve
into a comprehensive analysis of the applicability of FID for quality
assessment, showcasing its effectiveness in distinguishing not only between
various pathologies and normal X-rays but also among different pathologies
themselves.

Some studies extend the use of label-conditioned synthetic methods beyond
basic label utilization, broadening data generation capabilities. For
instance, [292] uses a DCGAN-based shape generator with size modulation for
precise control of lung nodule shape and diameter. Incorporating PGGAN
enhances visual plausibility, adopting a coarse-to-fine generation approach.
Notably, they demonstrate that deliberate selection of challenging examples
improves augmentation significantly. In a different approach, [293] employs
StyleGAN2 and Bayesian image reconstruction for ROI-conditioned synthesis of
CXR images. It aims to generate synthetic data while preserving clinically
relevant features, proposing a potential data sharing strategy for privacy
protection. Additionally, [294] merges PGGAN with Pix2PixHD GAN to facilitate
multi-stage generation from dots, representing various anatomical parts and
serving as seeds for subsequent steps that produce high-resolution images.
These methods, tailored for specific augmentation tasks, highlight the
importance of designing synthesis procedures around downstream challenges,
leading to better-performing models and higher-quality generated images.

In the realm of medical text-to-image synthesis, latent diffusion models, as
described in [62], employ strategies like reduced image sizes conditioned on
textual inputs and incorporate reverse diffusion steps to enhance denoising.
Notably, [295] introduces semantic diffusion, which involves translating
latent variables into image space using a decoder and refining/upscaling
through a super-resolution diffusion process. Models aiming to align generated
reports with X-ray images, as discussed in [296, 297, 298], encounter
challenges such as false positives [62, 299]. Joint learning of language
embeddings and image generation reveals limitations, prompting [295] to
propose a cascaded latent diffusion model that includes an autoencoder
component. Despite the advancements, comprehensive evaluations of image
quality and clinical applicability, as highlighted in studies like [295, 296,
297], are still lacking. Furthermore, the potential utility of these models in
data augmentation remains largely unexplored, and there is a paucity of
comparative analyses against other state-of-the-art methodologies.

Application | Type | Modality | Technology | Paper | DT | Evaluation | Code
---|---|---|---|---|---|---|---
U | F | D | Q | C | P
Train | Test
Unconditional | GAN | Knee joint XR | WGAN | [281] | Class.* | R,S,A | R | ✓ |  | ✓ |  | ✓ |
Pneumonia | DCGAN + PGGAN | [283] | Class.* | A | R | ✓ |  |  |  |  |
COVID19 | DCGAN + WGAN-GP | [282] | Class. | A | R | ✓ |  |  |  |  |
MSG-SAGAN | [286] | None |  |  | ✓ |  |  |  |  |
MLPs, CNNs | [287] | None |  |  | ✓ |  |  |  |  | ✓
CXR | PGGAN | [288] | None |  |  | ✓ | ✓ |  |  |  |
[285] | Class. | R,A | R |  |  | ✓ |  |  |
[289] | Class. | R | S |  |  |  | ✓ |  |
Class or Label Cond. | GAN | CXR | DCGAN + patchGAN | [292] | Det. | A | R | ✓ |  |  |  |  | ✓
StyleGAN2 + Bayesian recon. | [293] | Class.* | R,A | R | ✓ |  |  |  |  |
PGGAN + Pix2PixHD | [294] | Seg.* | R, S | R,S |  |  |  |  |  |
DM | DDPMs | [290] | Seg. | A | R | ✓ | ✓ |  |  |  |
Attribute conditioning | GAN | CXR | PGGAN with latent space optim. | [284] | Class. | R,S | R | ✓ |  | ✓ |  |  |
COVID19 | SD-GAN | [291] | Class. | A | R | ✓ | ✓ |  |  |  | ✓
Text-guided | DM |  CXR $\&$ Reports | Semantic DM & SR diffusion | [295] | None |  |  | ✓ | ✓ |  |  |  | ✓
CXR $\&$ Impressions | Stable diffusion v2 | [62] | Det. |  | S | ✓ | ✓ |  |  |  | ✓
Hybrid | CXR $\&$ Reports | VQ-GAN + GPT-Neox LLM | [297] | None |  |  | ✓ | ✓ |  | ✓ |  |
SDM-VAE + CLIP text-enc. | [299] | Class. | R,S,A | R | ✓ | ✓ |  | ✓ |  |
VQ-GAN + bidirec. transf. | [298] | Class. |  | S | ✓ |  | ✓ | ✓ |  | ✓
Cycle GAN + LSTM | [296] | None |  |  | ✓ | ✓ | ✓ |  |  |
Intra-modal trans. | GAN | Normal $\rightarrow$ Abn. CXR | cond. GAN + spatial transf. | [300] | Det. | A | R |  |  |  |  |  |
Cycle GAN + classifier | [301] | Class. | R,A | R |  |  |  |  |  |
Normal $\rightarrow$ COVID19 | Adaptive Cycle GAN | [302] | None |  |  | ✓ |  | ✓ |  |  |
Cycle GAN | [303] | Det.* | R,A | R |  |  |  |  |  |
Urinary stones | U-Net cond. GAN | [304] | Seg. | R,A | R | ✓ |  |  |  |  |
Normal $\rightarrow$ Pneumonia | Cond. + Cycle GAN | [305] | Class. |  | S |  |  |  |  |  |
Inter-modal trans. | Hybrid |  Spine US $\rightarrow$ XR | DDPM with ViT | [306] | None |  |  | ✓ |  |  | ✓ |  |
Table 11: Overview of all summarized and discussed studies on X-Ray (XR)
synthesis with information about the technology utilized, the downstream task
(DT), the evaluation procedure and code availability. Evaluation is split
according to the evaluation metrics reported, covering utility (U), whereby
automated machine learning models are trained with real (R), synthetic (S) or
augmented with synthetic data (A), and tested on real or synthetic data,
fidelity (F), diversity (D), visual quality (Q), clinical utility (C) and
privacy (P) evaluation. * stands for studies that have utilized synthesized
images for pre-training. CXR stands for Chest X-Ray. Papers that specify the
downstream task without training data, signify instances where a pre-existing
model was directly applied or tested on synthetic data.

In evaluating the quality of generated X-ray images, most studies use
downstream tasks and generic fidelity and diversity metrics that potentially
overlook nuances specific to medical imaging. However, a few studies, such as
[281], conduct human perception quality and clinical usability evaluations.
[281] performed a Visual Turing test with specialists, revealing that
synthetic X-ray images are often mistaken for real ones, suggesting sufficient
realism for deceiving experts. Another study, [285], conducted an extensive
Turing test with 400 images, demonstrating that radiologists struggled to
distinguish between synthetic and real CXRs, especially in abnormal cases,
containing lung lesions. In the context of CXRs, [284] finds that synthetic
data is often mistaken for real, but improved replication of fine anatomical
details is required for truly indistinguishable samples. The study also
evaluates FID’s applicability to synthetic CXRs, noting that small-scale
details may not be reproduced at the highest quality. [298] conducts human
evaluation with board-certified clinicians, revealing that the model generates
realistic CXRs but lacks precise alignment with reports. The proposed method
excels in generating view-specific X-rays but occasionally struggles with fine
details. Furthermore, [302] employs the Universal Image Quality Index (UIQI)
and Visual Information Fidelity (VIF) to compare synthetic and real images.
The quantitative metrics suggest challenges for GANs in translating high-
quality synthetic images, possibly due to insufficient training samples. A few
studies offer insights into the use of synthetic images for clinical tasks.
For instance, [298] employs synthetic images for a 14-class diagnosis of
anomalies in X-ray images, [289] calculates ratios to classify cardiomegaly in
CXR images, and [297] evaluates findings generated from synthetic images,
identifying false positives and misdiagnoses. [306] measures the Cobb angle
difference between synthetic and original X-ray images, demonstrating
effective use in evaluating scoliosis. Additionally, [281] explores privacy
concerns by assessing the nearest real image neighbors to synthetic images,
used as criteria for selecting training data to ensure privacy in downstream
tasks; however, a proper evaluation of preserved privacy is lacking.

####  3.3.7 OCT

Application | Type | Modality | Technology | Paper | Downstream Task | Evaluation | Code
---|---|---|---|---|---|---|---
U | F | D | Q | C | P
Train | Test
Unconditional | GAN | AS OCT | PGGAN | [307] | Class. | R,S | R |  |  | $\checkmark$ |  |  |
AS OCT | Cycle + patch GAN | [308] | Seg. | R,S | R | $\checkmark$ |  |  | $\checkmark$ |  |
SD DM, NOR, AMD | Vanilla GAN | [309] | Class. | R,S,A | R | $\checkmark$ |  |  |  |  |
Retinal OCT | StyleGAN2-ADA | [310] | Class.* | A | R |  |  |  |  |  |
FH OCT | StyleGAN2-ADA | [311] | Class. | R,A | R |  |  | $\checkmark$ | $\checkmark$ |  | $\checkmark$
DM | AS OCT | DDPM | [312] | Seg./Det. | R | S | $\checkmark$ |  |  | $\checkmark$ |  |
Hybrid | Retinal OCT | DDPM + transf. | [313] | Seg. | A | R | $\checkmark$ |  |  |  |  |
Class or Label cond. | GAN | Retinal OCT | U-Net + patch GAN | [314] | Seg. | R | S | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | $\checkmark$
Modified Pix2Pix | [315] | Class. | R,S,A | R |  |  |  |  |  |
Chorio-retinal OCT | Modified DCGAB | [316] | Seg. | R,S,A | R | $\checkmark$ |  |  |  |  |
SD OCT | StyleGAN2 | [317] | Seg./Class. | R,S,A | R | $\checkmark$ |  |  |  |  |
Coronary atrial plaques | cond. GAN | [318] | Class. | A | R |  |  |  |  |  |
VAE | Retinal (diseased) OCT | Modified cond. VAE | [319] | None |  |  |  |  |  |  |  | $\checkmark$
Attribute cond. | GAN | SD OCT | BAABGAN | [320] | None |  |  | $\checkmark$ |  | $\checkmark$ | $\checkmark$ |  |
DME OCT | DDFA-GAN | [69] | Class. | A | R | $\checkmark$ | $\checkmark$ |  |  |  | $\checkmark$
AMD OCT and CFP | Pix2PixHD | [321] | Class.* | A | R |  |  |  |  |  | $\checkmark$
Retinal OCT | Contrastive CNN-based | [322] | None |  |  | ✓ |  | $\checkmark$ | $\checkmark$ |  |
FOF-GAN | [323] | Class. | R | S | $\checkmark$ |  |  |  |  |
StarGAN | [324] | Seg. | R | S |  |  | $\checkmark$ | $\checkmark$ |  |
Intra-modal trans. | GAN | Pre-T - Post-T OCT | Pix2PixHD | [325] | None |  |  |  |  | $\checkmark$ | $\checkmark$ |  |
OCT - PS-OCT | Pix2PixHD | [326] | Class. | R,S | R | $\checkmark$ |  | $\checkmark$ |  |  |
Inter-device SD-OCT | CycleGAN | [327] | Seg. | R | S |  |  |  | $\checkmark$ |  |
Inter-modal trans. | GAN | FA - OCT | Pix2Pix | [68] | None |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |
Fundus - OCT | U-net-based | [328] | Class.* | R,S,A | R | $\checkmark$ |  |  |  |  |
Table 12: Overview of all summarized and discussed studies on OCT synthesis
with information about the technology utilized, the downstream task (DT), the
evaluation procedure and code availability. Evaluation is split according to
the evaluation metrics reported, covering utility (U), whereby automated
machine learning models are trained with real (R), synthetic (S) or augmented
with synthetic data (A), and tested on real or synthetic data, fidelity (F),
diversity (D), visual quality (Q), clinical utility (C) and privacy (P)
evaluation. AS, SD, DM, NOR, AMD, FH, DME, CFP, Pre-T, Post-T, PS, FA stand
for anterior segment, spectral domain, diabetic maculopathy, normal, age-
related macular degeneration, foveal hypoplasia, diabetic macular edema, color
fundus photography, pre-therapy, post-therapy, polarization sensitive OCT and
fluorescein angiography, respectively. * stands for studies that have utilized
synthesized images for pre-training.

Non-invasive retinal OCT is crucial for diagnosing eye diseases, as features
like retinal layer shapes and thicknesses strongly correlate with conditions
such as glaucoma, age-related macular degeneration, and diabetic macular
edema. Thus, in recent years, there has been an increase in the development of
automated methods for identifying macular pathologies [329]. However, these
approaches face challenges due to limited data availability and imbalanced
datasets, given the rarity of pathological findings. Therefore, the primary
goal of OCT image synthesis (Table 12) is to augment data for algorithms,
enabling not just disease detection but also enhancing the comprehension of
retinal diseases.

The array of unconditional generative methods, as exemplified by [307, 308,
309, 310, 311, 312, 313], underscores a notable improvement in downstream
tasks when the generation process is meticulously tailored to the specific
task and the modality of interest, such as various types of OCT images. These
studies reveal that the careful consideration of generative architectures
significantly enhances the performance of subsequent analyses. Addressing
challenges like limited dataset availability, artifact removal, and domain
shift issues through the adoption of techniques like PGGANs, structure-
consistency GANs, and StyleGAN2-ADA contributes significantly to the
generation of diverse and realistic synthetic images. The effectiveness of
these unconditional methods is further emphasized in tasks like
classification, despeckling, and segmentation, showcasing their versatility in
diverse medical image analyses. The integration of diffusion-based methods,
exemplified by [313], further accentuates this point, showcasing the
adaptability of generative architectures on different tasks. These findings
collectively emphasize the importance of thoughtfully tailoring generative
approaches to the intricacies of medical image analysis, promoting their
adoption in the domain of OCT image generation and analysis.

Highlighting the importance of conditioning for accurate representation of
retinal pathologies in OCT images, [319] employs a conditional VAE with
contrastive learning to discriminate between diseases in the embedding space.
Similarly, [323] introduces a FastGAN with a Fourier Attention Block (FAB),
improving feature map rescaling based on frequency contributions. Inspired by
Spectral Domain OCT (SD-OCT) that utilizes Fourier/spectral detection for
improving OCT imaging resolution, the generator uses the Fast Fourier
Transform (FFT) and inverse-FFT algorithms to eliminate noise and enhance
details. Results demonstrate FAB’s effectiveness in noise reduction, but lower
accuracies in classifying certain pathologies indicate the need for
incorporating more distinctive features. [324] underscores GANs’ importance in
evaluating subject-specific eye development and tracking subtle retinal
changes over time. This study introduces a counterfactual GAN based on
starGAN, synthesizing high-resolution counterfactual OCT images and
longitudinal time series from retrospective data. These images allow
exploration of hypothetical scenarios by altering certain subject
characteristics while preserving identity and image acquisition settings.
Despite clinical evaluations aligning with previous studies, limitations
include potential correlation of disease features with age or sex, which may
result in the GAN altering them when generating counterfactual images, as well
as the inability to learn relationships for unseen subject groups. Expert
evaluations reveal challenges in distinguishing real from synthetic images,
highlighting the algorithm’s realism in mimicking retinal changes and
emphasizing the need for further research into counterfactual approaches in
medical imaging for a deeper understanding of subject-specific changes over
time.

[320] presents a biomarkers-aware asymmetric bibranch GAN for post-therapeutic
(PT) OCT image generation. Using Adaptive Memory Batch Normalization (AMBN),
the method transfers knowledge from large-scale data to the target branch,
trained on small-scale paired data. The generated images closely align with
clinical requirements, aiding in treatment adjustment and monitoring of
neovascular age-related macular degeneration (nAMD) progression, whereby over
95% of synthetic SD-OCT images effectively convey clinically relevant
information and predict biomarker presence accurately. Additionally, the study
shows significant advancements in predicting therapeutic effects over longer
periods, up to 12 months after initial treatment, compared to existing
methods. [69] employs a dual-discriminator Fourier acquisitive GAN to generate
realistic OCT images, leveraging Fourier domain similarity. Meanwhile, [321]
uses a Pix2PixHD GAN for multi-modal age-related macular degeneration
categorization with CAM manipulation to enhance image diversity. Recognizing
the limitations of 2D OCT methods, [321] underscores the importance of 3D
volumes aligned with routine clinical examinations. Additionally, [322]
enhances cross-subsectional resolution using a fully convolutional approach,
validated for realism by medical experts. [322] also introduces BRISQUE, a
perceptual image quality indicator, effectively evaluating image quality
without a reference. These studies contribute nuanced perspectives to OCT
image synthesis, emphasizing practical applications and diagnostic
advancements.

In the absence of paired data for training, unpaired architectures, like
cycle-consistent GAN variations, are employed for image harmonization,
ensuring consistency of features across different acquisition sites. However,
challenges like instability and contrast inversion limit their reliability in
real-world medical imaging. Methods in [314, 315, 316, 317, 318] preserve
anatomical layout during generation, assuming consistent morphological shape
across imaging sites. [314] introduces a segmentation-renormalized image
translation approach using a residual U-Net, enhancing image harmonization and
robustness against perturbations. It explores Kernel Inception Distance (KID)
alongside FID for evaluating visual fidelity, suggesting a multitask
autoencoder trained on medical images for feature extraction and FID/KID
calculation. Several studies have made significant advancements in medical
imaging using GANs. [315] achieved a 94.83% accuracy in detecting DME fluid
with synthetic data using a Pix2Pix U-net-based generator for OCT image
synthesis. [316] demonstrated comparable performance between synthetic and
real data for OCT patch generation using a DCGAN. [317] enhanced chorio-
retinal segmentation by optimizing generator choice with an improved
StyleGAN2, leading to substantial improvements in patch classification.
Lastly, [318] improved plaque detection accuracy by 15.8% using conditional
GANs with class encoding, aligning well with clinical findings.

Translation-based approaches in OCT imaging address critical clinical needs,
like predicting treatment responses through generating post-therapeutic OCT
images from pre-therapeutic ones. For example, [325] adapts Pix2PixHD to
forecast responses to anti-vascular endothelial growth factors (anti-VEGF)
therapy in retinal vein occlusion patients. The generated synthetic OCT images
closely match real ones, aiding in treatment response prediction and patient
follow-up. Moreover, [326] introduces a method for synthesizing polarization-
sensitive (PS) OCT images using a modified Pix2Pix GAN, offering an
alternative to complex hardware. Validation demonstrates interchangeability
between synthetic and real PS-OCT images, promising applications in cancer
diagnosis. Additionally, [327] tackles differences in image intensities and
signal-to-noise ratios between instruments using an unpaired cycleGAN-based
domain adaptation network. This enables cross-instrumental image analysis,
crucial for deep learning model training and clinical segmentation tasks.

[68] presents a method using inter-modality translation to synthesize OCT
color-coded macular thickness maps from fluorescein angiography (FA) data and
vice versa, aiding in diagnosing diabetic macular edema. By integrating FA’s
physiological data with OCT’s structural data, a more precise assessment of
treatment response becomes possible, despite limitations in training data and
generator architecture. Similarly, [328] employs GANs, specifically the U-Net
generator, for generating OCT images from fundus images to diagnose glaucoma.
[328] demonstrates that incremental training improves model accuracy and
adaptability across different hospital settings, showcasing the potential of
generated images in glaucoma identification.

Synthesized OCT image quality evaluation typically mirrors assessments in
other modalities, relying on objective fidelity and diversity metrics. Visual
Turing tests, conducted in a limited number of studies [307, 322, 325, 68]
with 2-3 experts, indicate that artifacts around crucial regions and unusual
noise patterns often hint at synthetic samples. In [307], specialists noted
lower visibility of scleral spurs and anterior segment structures in synthetic
AS-OCT ACA images, critical for imaging the anterior chamber angle, aiding
identification. [68] revealed the model’s ability to represent leakage areas
in FA frames with color codes but also generated inconsistent background
vessels, signaling synthetic images. In [324], despite difficulty
distinguishing real from fake images in many cases, features like choroid and
vitreous areas, pathologic features, and shadowing artifacts hinted at
synthetic images. Several studies have assessed the clinical utility of
synthetic OCT images, providing insights into image quality and applicability.
In [320], analysis by retinal clinicians of generated post-therapeutic images
unveiled predictive accuracy for short and long-term treatment effects,
facilitating effective doctor-patient communication and adjustments in
treatment strategy. The proposed BAABGAN also demonstrated effective
prediction of biomarkers. [324] introduced a neural network generating high-
resolution counterfactual OCT images, enabling exploration of hypothetical
scenarios for retinal aging research. [311] estimated foveal hypoplasia grade
and visual acuity using deep learning, incorporating real and GAN-generated
images. [325] focused on calculating central macular thickness, while[327]
compared retinal volumes.Additionally, [312] addressed despeckling in AS-OCT
images. Despite these advances, the evaluation of synthetic images is lacking,
especially in terms of diversity, quality and clinical utility, with privacy
preservation mostly unaddressed in current methods.

####  3.3.8 Multiple Modalities

Application | Type | Modalities | Technology | Paper | DT | Evaluation | Code
---|---|---|---|---|---|---|---
U | F | D | Q | C | P
Train | Test
Unconditional | GAN,DM | MR-brain,Chest x-ray | StyleGAN, DDPM | [330] | None |  |  | ✓ |  |  |  | ✓ |
|  H:VAE,
---
DDPM
| 3D:CT-lung, MR-brain,
---
MR-breast, MR-knee
| VQ-GAN followed
---
by DDPM
[61] | Seg | A | R |  | ✓ | ✓(2) | ✓ |  | ✓
DM |  | CT-abdomen, CT-pelvic,
---
Chest x-ray, MR-heart
MT-DDPM | [63] | Class | S,A | R | ✓ | ✓ | ✓(3) |  |  |
Class or Label cond. | GAN | CT-lung, MR-brain | HA-GAN | [64] | Class | ✓ | ✓ | ✓ |  | ✓ |  | ✓
GAN |  | Conditioned on seg
---
mask: cardiac cine-MRI,
liver CT, and Fundus images
Multiple GANs | [331] | Seg | ✓ | ✓ |  | ✓(4) |  |  |
DM |  | Chest x-ray, Histopathology,
---
Opthalmology Fundus images
| Medfusion:
---
Conditional LDM
[332] | Class | S | R | ✓ | ✓ |  |  |  | ✓
Attribute cond. | DM |  | Chest x-ray, Dermatoscopy,
---
Histopathology
DDPM | [333] | Class | ✓ |  |  |  |  |  |
Text-guided | DM |  | breast Ultrasound, spleen CT,
---
prostate MR
EMIT-Diff | [334] | Seg | ✓ | ✓ |  |  |  |  | **
| Chest x-ray, Histopathology,
---
MR-prostate
| FineTuned
---
StableDiffusion
[335] | Class | ✓ | ✓ |  |  |  |  |
Other cond. | DM | prostate-MRI, chest x-ray |  | Med-cDiff:
---
Conditional DDPM
[336] | Seg,Class | ✓ | ✓ |  |  |  |  |
| Conditioned on 2perpendicular
---
2d DMs as a 3d prior: MRI, CT
TPDM | [337] | None |  |  | ✓ |  |  |  |  | ✓
GAN |  | Combining Chest x-ray
---
images and Tabular Data
$\alpha$GAN and CTGAN | [338] | Class,Reg | ✓ |  |  | * |  |  |
Table 13: Overview of all summarized and discussed studies on multiple
modalities synthesis with information about the technology utilized, the
downstream task (DT), the evaluation procedure and code availability.
Evaluation is split according to the evaluation metrics reported, covering
utility (U), fidelity (F), diversity (D), qualitative assessment (Q), clinical
utility (C) and privacy (P) evaluation.In the Type column, H stands for hybrid
approach combining two different models. In evaluation of utility (U), R
stands for real data, S stands for synthetic data, and A stands for augmented
data. In qualitative evaluation, the number between parenthesis indicate the
number of specialists employed, while * stands for visual inspection of
visualization plots.** stands for the availability of code of the original
method. Acronyms: Classification (class), Segmentation (seg), regression
(reg), conditioning (cond),Denoising Diffusion Probabilistic Model (DDPM),
Transformer-based Denoising Diffusion Probabilistic Model (MT-
DDPM),Hierarchical Amortized GAN (HA-GAN),Two Perpendicular 2D DMs (TPDM)

In preceding sections, the focus was primarily on papers centered around
generative models targeting a single modality, such as exclusive generation of
MRIs or CT scans. This section delves into papers exploring models capable of
generating multiple modalities using the same architecture, thereby avoiding
significant modifications. For instance, [61] demonstrated that diffusion
models are capable of generating realistic 3D synthetic data across four
different anatomical regions, two modalities (CT and MRI), and three different
resolutions. Remarkably, the diffusion model achieved this without the need
for fine-tuning any hyper-parameters to adapt to the different datasets,
despite the small training datasets. In addition to the diffusion model, a VQ-
GAN architecture was used to compress the 3D CT and MRI images into a latent
space, where the DDPM process was performed. In [330], a comparison was made
between the performance of a GAN and a diffusion model in generating synthetic
brain MRI and chest X-rays, with a focus on memorization of training data.[63]
utilize Swin-transformer in the design of a diffusion model to enhance the
image synthesis quality, applied on multiple imaging modalities. [334] adopts
a holistic approach to medical image synthesis, utilizing controllable
diffusion with DDPMs incorporating edge information to generate realistic and
diverse synthetic ultrasounds, spleen CTs, or prostate MRI images while
preserving essential characteristics. Initial pretraining on RadImageNet
involves triplets for diverse prompts, while edge information from
RadImageNet, guided by a HED algorithm (Holistically-Nested Edge Detection)
[339], enhances realistic image generation. Subsequent fine-tuning on a task-
specific dataset adapts the diffusion model to unique characteristics.

[331] employed multiple GANs conditioned on segmentation label masks for
different MRI, CT, and retinal images, while [332] utilized a class
conditional LDM on various modalities including chest x-ray, histopathology,
and fundus images. Notably, a DDPM was employed in [333] conditioned on either
the diagnostic label alone or in conjunction with a property such as hospital
or a sensitive attribute (e.g.: ethnicity,sex). Augmenting the training data
with synthetic data improved both the accuracy and fairness of downstream
classification models, particularly for under-represented groups in the
training data.

Training diffusion-based models for text-to-image generation from the scratch
demands access to extensive datasets with image-caption pairs and significant
computational resources. However, in the context of medical image generation,
the availability of large, publicly accessible datasets containing text
reports is limited. [335] proposes a solution by demonstrating that pre-
trained Stable Diffusion models, initially trained on natural images, can be
adapted to diverse medical imaging modalities through textual inversion [340].
To introduce a medical modality as a new concept to a pre-trained diffusion
model, textual inversion finds a vector in the text embedding space that best
represents the new concept. This is achieved by freezing the entire
architecture except for this specific embedding vector, and then performing
backpropagation with a similarity loss using a small set of example images.

[338] is the only paper in our survey that generates synthetic hybrid data
that combines both images and non-image data, whereby two GANs are used to
create synthetic records containing both CXRs and non-image clinical data.
This is done by reducing the dimensionality of the images using the pretrained
encoder of the GAN.

While the majority of papers focus on utility and fidelity evaluation of the
synthetic data, only few papers considered measures of diversity [61, 63, 332,
330]. [63] used a diversity score (DS) to measure how diverse the synthetic
images were. The DS, or the nearest SSIM difference, is the KL divergence
between the distributions of nearest SSIMs in real and synthetic images. To
find the nearest SSIM for the synthetic images, the authors calculated the
SSIM of each synthetic image with all the others and found the closest match.
[330] argues that if the SSIM was calculated between each synthetic image and
all training images, then this score would quantify the memorization of the
training data by the model. [61] also used SSIM to measure diversity of
synthetic images, by calculating the SSIM between pairs of synthetic data to
check how similar they are, and thus investigate model’s ability to generate
diverse images.

Few studies consider qualitative assessment of the generated data, with a
limitation of considering small number of experts. For instance, [61] relied
on the evaluation of image quality by only two human experts in terms of
realistic image appearance, anatomical correctness, and consistency between
slices, while [63] involved three experts, and [331] considered four experts.

Finally, the evaluation of memorization i.e. that generative models can simply
generate samples that are copies of the training data, is the focus of [330].
It involved calculating pairwise image correlation between each synthetic
image (from a selection of only 100 synthetic images for analysis) and the
entire set of training images, and the findings indicated that diffusion
models exhibit a higher susceptibility to memorizing the training images than
GANs.

###  3.4 Medical text

Application | Type | Modality | Technology | Paper |  Downstream Task | Evaluation |  Comparison | Code
---|---|---|---|---|---|---|---|---
U | F | D | Q | C | P
| NLP Enhancement
---
LLM |  | Discharge summary
---
| LLaMa
---
GPT-3.5, 4
[341] | NLP Tasks | ✓ |  |  | ✓ | ✓ |  | ✓ | ✓
| NLP Enhancement
---
LLM |  | Diabetic ketoacidosis
---
guideline
GPT-4 | [342] |  | Question
---
Answering
✓ |  |  |  |  |  |  |
| NLP Enhancement
---
| RNN,
---
GAN,
PLM
| Patient history,
---
present Illness
| CharRNN, GAN,
---
GPT-2, CTRL
[343] |  | Name entity
---
recognition
✓ |  |  | ✓ |  |  |  | ✓
| Text augmentation,
---
de-identification
PLM |  | Injury records
---
BERT | [344] |  | Replace sensitive
---
data
✓ |  |  |  |  | ✓ |  | ✓
| Text augmentation
---
PLM |  | Discharge summary
---
| Transformer,
---
GPT-2
[345] | Classification | ✓ | ✓ |  |  |  | ✓ | ✓ | ✓
| De-identification,
---
Text augmentation
PLM |  | Discharge summary
---
| LSTM, GPT-2
---
[346] |  | Name entity
---
recognition
✓ |  |  |  |  | ✓ |  | ✓
Table 14: Reviewed generative models for medical text. Acronyms: Large
Language Model (LLM),Large Language Model Meta AI (LLaMa), Generative Pre-
Trained Transformer (GPT), Pre-trained Language Models (PLM), Natural Language
Processing (NLP), Recurrent Neural Netwrok (RNN), Character Recurrent Neural
Network (CharRNN), Conditional Transformer Language (CTRL), Bidirectional
Encoder Representations from Transformers (BERT)

Different from synthesizing medical imaging or signal data, medical text
generation focuses on language models and transformers. As we mentioned
before, the main purposes of generating medical text are categorized into (i)
enhancing NLP tasks such as Name entity recognition, concept extraction,
relation extraction, and question answering, (ii) augmenting medical text data
where the high-quality clinical notes are limited and privacy sensitive to use
for research, (iii) replacing sensitive information in the real data with
generated synthetic data. A variety of medical text data types have been
utilized to train the models including

  1. 1.

discharge summary, a clinical document summarizing the patient’s
hospitalization from admission to discharge to provide a comprehensive
overview of the patient’s diagnosis, treatments, condition at discharge, and
recommendations for follow-up care;

  2. 2.

disease guideline (a.k.a clinical guideline or practice guideline), a
systematically developed documentation to assist medical professionals and
patient decisions about appropriate healthcare for specific clinical
situations;

  3. 3.

case report, a detailed report of the symptoms, diagnosis, treatment, and
follow-up of an individual patient;

  4. 4.

history of present illness which is a detailed documentation of the
development of the patient’s illness, including the onset of symptoms,
duration, intensity, and any factors that aggravate or relieve them;

  5. 5.

chief complaint which describes the primary reason or symptoms that brought
the patient to seek medical attention

Some additional input can also be used for medical text generation such as EHR
data to generate clinical notes and predict outcomes [345], diagnostic reports
where studies take radiology, pathology, and other diagnostic images as input
to generate imaging reports with detailed descriptions and diagnosis, clinical
trial reports where studies take extensive medical documents as input to
generate summarization of the reports.

In medical text generation, there are no standards or well-accepted evaluation
metrics to assess the fidelity, accuracy, and clinical utility for the
generators. Moreover, an additional crucial aspect in evaluating synthetic
medical text is the truthfulness and correctness of the generated data,
ensuring that the information presented adheres to current medical guidelines
and practices. If a medical text generator were to generate inaccurate or
misleading information (such as hallucinating facts), the implications could
be far more serious than with general-purpose text generators. Such errors
could mislead healthcare professionals, impacting their decision-making
processes and potentially affecting patient care outcomes. Another important
aspect is the understand-ability and readability by humans, mainly medical
professionals and patients

Pre-trained and large language models such as BERT [48] and GPT [49] are
widely applied in medical text generation applications. Similarly, there are
commonly used training text datasets such as MIMIC clinical notes data [347,
348], i2b2 (Informatics for Integrating Biology and the Bedside) data [349],
and n2c2 (National NLP Clinical Challenges) data [350]. The i2b2 and n2c2 are
well-known initiatives that provide datasets for NLP research in the
biomedical and clinical domains.

[341] developed a generator based on LLaMA [52] but particularly focusing on
clinical note generation using case reports, discharge summaries, and clinical
notes from multiple sources including MIMIC-III [347], MIMIC-IV [348], i2b2
[349], and CASI [351]. [341] first generated 158K synthetic clinical notes
from case reports from the PubMed central patients dataset [352] using
GPT-3.5. Then, incorporating the generated clinical notes, the proposed
generator was fine-tuned based on LLaMA using discharge summaries from MIMIC-
III database to capture the peculiarities in clinical notes such as capturing
the relations between two medical terminologies, recognizing the reasoning
process in the notes. The proposed model is fully open-source and trained on
open-access datasets. The study evaluated the utility of the generator by
conducting the NLP tasks using generated text (e.g., summarization, concept
extraction, relationship extraction, name entity recognition, question
answering) focusing on accuracy, relevancy, and completeness. They applied
GPT-4 to design specific evaluation prompts and also validated by four real
clinicians. The study demonstrated their model outperformed other open-source
instruction fine-tuned LLMs such as Alpaca [353] and medical domain-specific
models such as MedAlpaca [354] and ChatDoctor [355]. The study demonstrated
there is no significant difference between models trained on actual and
synthetic clinical notes. However, in the training aspect, the proposed model
is limited to dealing with only discharge summaries in one-turn instruction
following tasks which means the user can ask one question or give one prompt
then generator will provide an answer or generated text based on this single
input.

In the evaluation, the study did not analyze whether and to what extent the
model would hallucinate the outcome, which is one of the major concerns for
the language model use in the clinical practice.

[342] utilized ChatGPT-4 and a Link Retrievel Plug-In to retrieve the medical
information, integrate and analyze the collected information, and compile them
into a human-understandable answer using three international guidelines on
treating diabetic ketoacidosis. The study designed two prompts to 1) retrieve
and integrate information focusing on diagnostic criteria, risk factors, signs
and symptoms, investigations, and treatments of diabetic ketoacidosis and 2)
generate comprehensive and evidence-based answers to the questions. The study
aims to enhance the traceability and retrieval accuracy of ChatGPT-4 and
claims their work maintained the accuracy, reliability, and quality of the
generated answers. However, there are not sufficient systematic and
quantitative experiments and evaluations being conducted in the study to
support this conclusion. Furthermore, the truthfulness which assesses if the
language models hallucinate the facts is not considered and evaluated
sufficiently. This reflects many other use cases in using generative models in
practical applications in medicine and healthcare domain.

[343] used a text dataset of 570 patient History and Present Illness (HPI)
description from i2b2 and n2c2 challenges to train four different generative
models - Character Recurrent Neural Network (CharRNN), Sub-sequence Generative
Adversarial Network (SegGAN), GPT-2, and Conditional Transformer Language
(CTRL). The four models were constructed based on different architectures and
with different focuses. CharRNN is a character-level RNN which has shown its
capability to tackle the out-of-vocabulary problem. Out-of-vocabulary
challenge occurs when a word or phrase is not in the training data’s
vocabulary or dictionary used by a language model. The language models learn
to understand and generate text only based on the training data, which means
they can only recognize and interpret words in their training process and
unable to process or understand the out-of-vocabulary words directly. SegGAN
follows the traditional GAN framework, adapted to deal with discrete text
output and able to adversarially learn on both the entire sequences (i.e.
sentences, paragraphs, documents) and sub-sequences (i.e., words, phrases).
GPT-2, as a large transformer-based pre-trained language model, has the
capability to generate diverse and massive synthetic text data, while CTRL is
a parameter conditional transformer language model which is conditioned on
control codes that govern the content and task-specific behavior. Control
words are specific words or phrases that specify domain, subdomain, entities,
relationships between entities to steer or guide the generation process of the
model towards a desired outcome. The study fine-tuned the GPT-2 and CTRL
models to generate synthetic clinical HPI corpus. The models’ performance was
assessed using the Bilingual Evaluation Understudy (BLEU) metrics [356], to
select the best model to create a synthetic corpus of 500 HPI subsections.
BLEU examines the similarity of the machine translation and human translation
in n-gram representation. An n-gram is a contiguous sequence of n items from a
text, where these items can be words for text generation. After manually
annotating clinical entities such as problems, treatments, and tests in real
and synthetic text, NER models were developed on both real and synthetic
datasets to conduct a utility evaluation. The study showed NER models that
were trained on synthetic corpus achieved slightly higher performance than
that of the real corpus. The NER models that were trained on the augmented
corpus (real + synthetic) achieved better performance than that trained on the
real corpus only. The study concluded the generated synthetic HPI text can be
used to enhanced the development of clinical NLP models.

[344] proposed a model generating partially synthetic clinical text to protect
against re-identification for individual patient and preserve the
characteristics information from the population. The proposed model has two
key processes - the artificial masking and the content substitution. Masking
process utilized the tokenization function in BERT model to break down the
sentence into tokens, added special tokens to indicate the beginning and end
of sentences, then converted all the tokens into corresponding IDs in the pre-
trained BERT vocabulary. Then, the tokenized data was fed into BERT for
training. To mask the personally identifiable information in an efficient
manner, the study embedded a blacklist and a white list to direct the masking.
The blacklist and whitelist are customisable for different use cases. For
instance, the blacklist can include words without meaning such as standard
punctuation, stop words, while the whitelist contains sensitive attributes
such as personally identifiable information. The model will avoid masking
words in the blacklist, while the words in the whitelist will be masked with
higher probabilities than words that are not in the lists. Masking sensitive
words may result in obfuscation in some records. Therefore, the second step in
the generative model is to replace defined sensitive information (i.e. words
or phrases in the whitelist) of each document with text in similar documents
in the dataset using Rapid Automatic Keyword Extraction [357] and TextRank
[358] methods. The model was trained on 153K injury reports including textual
injury descriptions with gender, age, and injury and illness labels and MIMIC
III discharge summaries. The study evaluated data privacy by comparing the
similarity between original and generated synthetic texts in word choice, word
order, and word frequency. The satisfactory privacy-preserved synthetic text,
which is well-obfuscated text, should be significantly different from the
original text in word choice, word order, and word frequency. The privacy
evaluation utilized BLEU scores, ranging from 0 to 1, measuring the
similarity, with lower scores indicating better obfuscation. The data utility
of the generated synthetic text was assessed by training a classification
model on the original text and applying the trained model to the synthetic
text to calculate the data utility loss. Data utility loss was measured by
comparing the classification performances between the original and obfuscated
text (partially synthetic text). Finally, readability of the generated text
was evaluated by a machine evaluator which is a fine-tuned text classifier
based on a BERT model to determine if the text input is a real clinical note
or generated from a machine.

[345] proposed a differentially private and self-attention generative neural
network model to generate synthetic medical texts. The generated text does not
contain any identifying information from any individual if the input datasets
were private. The study proposed a model as an autoregressive variation of the
transformer model introduced by [359]. In the proposed model, each of the
decoder blocks is composed of a multi-headed masked self-attention layer and a
simple feed-forward network. The autoregressive feature means when masked
self-attention stops the model from seeing tokens that are at the right of the
current position which allows the model to take into account the last token
when generating the next token. Furthermore, the study applied a
differentially private Gaussian mechanism technique by modifying the
optimization algorithms in their generator training process. This means in
each training iteration, the original weights are adjusted by adding noise
values derived from the Gaussian mechanism. This process generates
differentially private tokens, resulting in an input layer distinct from the
original and ensuring a certain level of privacy guarantee. Similar to [341]
and [343], this study also used discharge summaries from MIMIC-III and i2b2
datasets. The model performance is evaluated by measuring the data utility on
the word-level, document-level, and corpus-level. On the word-level, the study
used BLEU score, Jaccard similarity, and G2 Test respectively to analyze the
word co-occurrence or similarity, assessing whether the generated text has
similar word distributions as the original text. On the document-level, data
utility was analyzed by performing a classification task on original and
synthetic data to determine whether the underlying document contains a certain
disease or not. The study trained classifier on original text then tested on
synthetic text, and trained on synthetic text then tested on original. The
corpus-level utility, containing all documents, was assessed by an adversarial
classification task. The study trained multiple classifiers on combined
datasets of original and generated text and predicted whether a document is
original or synthetic. The purpose of the classifier is to distinguish between
original and synthetic text which is analogous to the Turing test, where a
human evaluator predicts the data to be original or synthetic. Regarding data
privacy, the study applied different privacy in the model and emphasized on
the privacy-preserving feature that proposed model has, but the study did not
provide sufficient privacy evaluation on generated text.

[346] developed two language models - LSTM-based model and GPT-2 (Transformer)
based model. The study sampled medical documents from the EHRs of 39
healthcare organizations in the Netherlands. Three domains of healthcare are
represented within this sample: elderly care, mental care and disabled care.
The models were evaluated in terms of the utility and privacy of the synthetic
text. The utility was assessed by using synthetic text as a replacement of the
real data to protect sensitive personal information and using synthetic text
as a data augmentation approach to enrich the real medical documents. To
evaluate data privacy, a user study was conducted with participants who were
presented with the synthetic documents with high risks of privacy disclosure
to judge if the documents contain sensitive information. The study observed
the distribution of annotations in the training text and the diversity of the
generated text have significant impact on the downstream task performance. The
study reported that non-English language training faces additional challenges
due to the lack of clear structure and conformity of the languages. Regarding
data privacy and utility balance, the study argued that synthetic text does
not need to be realistic for utility in downstream tasks so that privacy
protection can be priotized. However, it is necessary to provide a
mathematical privacy preservation in synthetic text generators such as using
differential privacy methods.

##  4 Discussion

The survey of the different papers across different data types and modalities
has yielded notable findings that we categorize into goal of synthesis,
generation methods, and evaluation methods. For each insight or conclusion, we
provide a non-exhaustive list of supporting references, previously discussed
in section 3. For further details, readers are directed to this section.

###  4.1 Synthesis applications and purpose of synthesis

Diversity of clinically valid synthesis applications: Conditional generative
models have demonstrated impressive capabilities in synthesizing missing or
corrupted modalities across a diverse range of applications. As depicted in
Fig.4, these applications vary depending on the data types involved. For
example, in signals and images, both intra and inter modal translation between
different modalities are common for various clinically valid reasons, such as
synthesizing missing or corrupted MRI contrast when acquiring such images is
not feasible [194, 203, 221], translating low-to-high density mammograms [158]
or DFM to FFDM mammograms [163], or B-mode to Elastography US [180, 181]. In
the context of EHR, the synthesis and the choice of generative model depends
on the specific format and requirements of the EHR data. For instance, the
capability to manage high dimensionality is pivotal when handling longitudinal
[71, 74] and aggregated [8, 81] EHR data, but its significance diminishes in
the context of time-dependent [79, 77] and snapshot [88, 85] formats.
Conversely, the preservation of temporal dependencies among clinical features
remains crucial for both longitudinal and time-dependent formats. In medical
text generation, the generated data serve multiple purposes beyond augmenting
training data [344, 345] . They can be also used for NLP enhancement, such as
entity extraction [343] and question-answering systems [342] , as well as text
replacement for privacy reasons [346], as depicted in Table 14.

Limited utilization of synthetic data beyond augmentation: Despite its
potential, synthetic data is underutilized beyond augmentation. This valuable
resource, capable of generating diverse datasets varying in quality and
anatomy, could serve as a comprehensive validation and testing set during
model development [273, 280, 62, 323, 324, 360]. It has the potential to
uncover algorithm limitations early on, particularly for challenging cases
that may not be accessible due to data sharing policies or rarity,
facilitating smoother translation of algorithms into clinical practice.

Most generative approaches are designed to handle the lack of training data
but thorough analysis of how this is achieved is missing: While numerous
methodologies have been proposed to address the scarcity of training data, the
specific elements of the generated data that contribute to improved outcomes
remain largely unexplored. This is particularly true when the use of synthetic
data alone does not yield any enhancement in performance [361, 237, 257, 362,
363, 364]. Only a handful of studies have indicated that the type of synthetic
data incorporated into the training process can significantly impact results
[263, 292]. For instance, incorporating challenging cases such as dense
mammograms [158], severe examples of lung nodules in X-ray images or
overlapping benign and malignant nodules in breast ultrasound [178], or
underrepresented groups [148, 147], has been shown to enhance performance.
This underscores the potential benefits of active learning strategies, either
during the data generation process or in selecting synthetic samples to be
included in the training set.

###  4.2 Generation techniques

The leap of textual conditioning opens future directions: Recent advancements
in text-conditional generative models offer an innovative approach to
integrating descriptive data into synthetic medical data generation. This can
involve structured text input specifying medical conditions and demographic
data or utilizing medical reports as a source of clinically relevant guidance
for the model [117, 145, 157, 250, 295, 334]. Readers are directed to the
”Text-guided” row in the tables for more information. Text-guided conditioning
has been already established across various modalities, although some
modalities are lagging behind and have not yet implemented this approach .

Conditional Models- Limited leverage of clinical knowledge, patient context,
and text: A significant gap exists in the utilization of prior clinical
knowledge, the availability of complimentary information accompanying the
acquisition, as well as pathological processes, biomarkers and radiology
reports to inform the generation process [252, 258, 279, 320, 68, 365]. Most
models fail to take advantage of these valuable resources, leading to
potentially sub-optimal results. Similarly, conditioning based on subject
characteristics patient and demographics is rarely utilized, except in
isolated instances [54, 76, 147, 157]. This highlights the need to focus on
more personalized synthesis tailored to specific subject characteristics and
patient demographics, prioritizing fairness and inclusivity in medical data
generation. Strong evidence suggests that conditional models leveraging such
textual information, clinical knowledge, or other images/data inputs
consistently produce higher quality and more realistic data compared to their
non-conditional counterparts.

Limited tailoring of generative approaches to the specifics of medical data:
Most synthetic methods proposed for medical image generation still largely
rely on loss functions and architectures initially designed for natural
images, without considering the intricate aspects of medical image
acquisition, physics and well defined statistics that can guide the generative
process [250, 255, 279, 365]. Moreover, a significant number of models are
designed to generate small patches (especially 3D) and lead to the generation
of artificial features that can lead to large errors in anomaly detection and
other tasks [236, 249, 259, 272]. Computational limitations also restrict
recent transformer-based approaches to slice-based training [236, 251, 366]
and diffusion models to low-resolution image generation. In contrast, EHR
generative models draw from established deep learning architectures like
CTGANs [88, 89] but address unique challenges such as synthesizing mixed-type
data with static (e.g., demographics data) and sequential features (e.g.,
vital sign time-series) [73, 74], numerical features (e.g., blood pressure),
and categorical features with many categories (e.g., medical codes) [58, 71].
Some of these models also consider modeling missing data patterns for
informative analyses [76, 77]. See Table 3 for more information. Similarly,
generative models for physiological signals leverage techniques for time-
series data, some customizing to signal specifics like morphology or inter-
lead [105, 122]. Regarding medical text generation, the application of pre-
trained models and large language models has recently become prominent in the
field. These models are based on general foundation language models such as
BERT and GPT and then adapted for medical applications through training or
fine-tuning with specialized datasets, such as PubMed abstracts or clinical
notes from the MIMIC database [342, 341]. See Table 14 for more information.
However, this approach has limitations in that these advanced medical text
generators may not fully grasp the underlying principles or logic of medical
knowledge.

###  4.3 Evaluation

Limited benchmarking, comparative studies and the use of small private
datasets The absence of comparative studies between different methods and the
lack of a well-defined benchmark for comparison leaves a gap in understanding
the relative strengths and weaknesses of various synthetic data generation
approaches and therefore highlights the need for a more systematic approach
[237, 242, 281, 299, 307, 325]. Moreover, the reliance on small, often private
datasets for training and testing generative models raises concerns about the
generalizability and usability of the generated data in clinical settings
[255, 271, 362, 367]. Most studies report both training and validation of the
proposed synthesis approaches on significantly limited datasets - typically up
to several hundreds of samples are used for training with a substantially
smaller set for validation/testing. In addition, there is a severe lack of
investigation of the effects of training data itself [239].

Limited evaluation frameworks: Most approaches focus on quantitative
evaluation using fidelity metrics that are not designed for the specific kind
of medical data. For example, in the case of medical images, the fidelity
metrics employed are tailored for diverse data types such as natural images,
overlooking the distinctive characteristics and statistical properties
inherent to typical medical imaging modalities [240, 67, 254, 286, 293, 316].
Similarly, the evaluation metrics used for time-dependent EHR and
physiological signals may overlook critical temporal features essential for an
accurate evaluation [93, 110, 111, 115]. In medical text generation, there are
no standards or well-accepted evaluation metrics to assess the fidelity of the
generated data. Sole reliance on such generic metrics, without incorporating
additional measures of clinical validation, utility, and human visual
perception studies can lead to misleading conclusions about the true quality
and suitability of synthetic medical data.

However, despite the crucial role of clinical validation in making sure that
crucial anatomical details are maintained in the generated data, there is a
notable lack in its proper implementation, hindering the translation of
synthetic data into practical clinical settings. Studies that do perform
clinical validation, typically perform it on a small subset of the generated
data, signifying a need for evaluation on large internal and external
validation datasets [256, 260, 262, 271, 272, 273].

Moreover, human assessments are still limited and face significant design
shortcomings. Typically, these studies involve a limited number of experts,
often ranging from 1 to 3, which raises concerns about the statistical
significance of the results [65, 244, 246, 247, 325, 368]. Furthermore, the
design of these studies is often unclear, especially regarding the selection
criteria of synthetic samples for evaluation [236, 238, 239, 66]. To enhance
the effectiveness of these studies, they should be designed to gather reports
from experts on indications that suggest certain samples are synthetic,
providing the ability to correct for these artifacts in future approaches
[238]. In other instances, human visual assessment is reduced to a mere check
of a few image samples for realism or a visual comparison of images from
different methods, or merely inspecting visualizations such as PCA or t-SNE
plots.

A more comprehensive evaluation of synthetic data should include measures of
diversity, ensuring that the synthetic data encompasses the full range of
anatomical and pathological variations encountered in clinical practice [255,
271, 362, 367]. However, many current methods for generating synthetic data
focus on reproducing specific image characteristics rather than generating a
truly diverse set of instances. The lack of well-defined diversity metrics
adds to the challenge, with researchers employing their own individual
definitions [330, 61, 63, 332].

Limited evaluation of recent approaches based on diffusion and transformers:
Current research on diffusion models for data generation indicates they are
more stable compared to GANs, with the ability to avoid mode collapse and
vanishing gradients and add more control over the sampling process. However, a
lot of diffusion models presented so far are yet to be properly tested and
evaluated - utility studies are largely missing, as well as clinical
validation or detailed qualitative and fidelity analysis of the generated
data. Moreover, training is still performed on limited data that may lack
adequate variability [240, 242, 246, 260, 259, 274].

Privacy disparities in synthetic medical data: The assessment of privacy
protection in generated medical data, encompassing images, EHR, and signals,
faces notable disparities in standardized methodologies. Despite assertions of
privacy capabilities and the potential for widespread data sharing, the
absence of well-defined evaluation methods for medical images, signals, and
text compared to EHR raises concerns. EHR privacy assessments commonly involve
testing synthetic data for vulnerabilities against potential adversary attacks
and demonstrating resilience against various threats. In contrast, privacy
discussions in medical images primarily focus on the memorization of training
data, leaving questions about identity disclosure unexplored. The extent to
which generators and diffusion models memorize or replicate data, and their
impact on tracing back learned distributions to original training data,
remains unclear. While EHR benefits from established evaluation practices,
addressing the significant gaps in privacy protection evaluation for other
data types is crucial for ensuring the responsible deployment of synthetic
data in medical contexts.

##  5 Recommendations

Based on the insights collected from the surveyed papers, we propose the
following recommendations to be taken into account for future research:

Generative approaches should be tailored to the medical data and downstream
task or their intended use: As specified earlier, generative approaches should
be more tailored to the specifics of the medical data and also to the specific
task at hand, as there are different requirements for each tasks [242, 257,
258, 254, 261, 366]. For instance, requirements for
segmentation/classification are substantial different compared to registration
or denoising tasks .

Synthetic data should be utilized beyond data augmentation to train and
evaluate medical AI software: Synthetic data should not only serve as a tool
for data augmentation and training downstream models but also for validation
and testing purposes [298, 294, 289, 312, 327]. Moreover, it can be used for
the development of explainability mechanisms to improve not only the
confidence of the proposed analysis models, but also the confidence of
clinicians in utilizing automated methods in clinical practice.

Prioritizing fairness and inclusivity in medical data generation: Due to its
scarcity, there is a need to focus on more personalized synthesis tailored to
specific subject characteristics and patient demographics, prioritizing
fairness and inclusivity in medical data generation.

Adapting language models to medical tasks should be further approached with
more care: A thorough analysis is needed on the interpretability capabilities
of diffusion models when it comes to the understanding and translation of
radiological reports to images.

A call for benchmarking and comparative studies to promote openness and
collaboration:  The use of established and well defined benchmarks and open-
source datasets would not only accelerate progress in the field but also
provide valuable insights into the strengths and weaknesses of various
techniques, guiding researchers towards the optimal tools for specific
clinical applications.

A need for more comprehensive evaluation: There is a need for holistic
assessment covering more specific fidelity metrics, proper clinical validation
and human assessment, and well-defined diversity metrics. Moreover, there is a
need for well established privacy evaluation practices, especially for data
types where there is a significant gap. This is crucial for ensuring the
responsible deployment of synthetic data in medical contexts.

##  6 Conclusion

The advent of AI and ML has a great potential to revolutionize healthcare,
with synthetic data generation emerging as a promising solution to the
challenges of data availability and diversity. Despite the growing interest in
generative models for medical data synthesis, existing reviews often have
limited scope, focusing on specific modalities or data types, or overlooking
certain types of generative models beyond basic data augmentation techniques.
This survey paper aims to fulfill this need by providing a comprehensive
examination of the use of generative models, such as GANs, VAEs, LLMs, and
DMs, in synthesizing medical data across various modalities, including
imaging, text, time-series, and tabular data. Through a meticulous search
strategy involving databases such as Scopus, PubMed, and arXiv, this survey
paper identifies and analyzes 249 relevant studies published from January 2021
to November 2023, excluding review and perspective papers. The survey aimed to
provide a holistic understanding of the applications of these generative
models in generating synthetic medical data,shedding light on the current
practices, potential, and challenges in this field, delving into three key
aspects: the purpose of synthesis, generation techniques, and evaluation
methods.

Our findings reveal a diversity of clinically valid synthesis applications,
yet also highlight a limited exploration of synthetic data’s potential and its
utilization beyond augmentation. Most generative approaches are designed to
handle the lack of training data, but a thorough analysis of how this is
achieved is often missing.

In terms of generation techniques, the advent of textual conditioning opens up
exciting future directions. However, there is limited leverage of clinical
knowledge, patient context, and text in conditional models, and generative
approaches are often not tailored to the specifics of medical data.

Evaluation methods also present challenges, with limited benchmarking,
comparative studies, and the use of small private datasets. Evaluation
frameworks are often limited, and recent approaches based on diffusion and
transformers are not thoroughly evaluated. Privacy disparities in synthetic
medical data also pose a significant concern.

In essence, while synthetic data generation holds immense potential in
transforming healthcare research and practice, there are notable gaps and
challenges that need to be addressed. There is a need for more personalized
synthesis approaches, standardized evaluation methodologies, and in-depth
evaluation approaches relevant to clinical applications. The field would also
benefit from more benchmarking and comparative studies to promote openness and
collaboration.

This survey paper serves as a valuable resource for researchers and
practitioners interested in leveraging generative models for synthesizing
medical data. By providing a holistic understanding of the field, we hope to
spur further research and innovation in this critical area of healthcare AI
and ML. The insights gained from this survey underscore the importance of
continued exploration and innovation in synthetic medical data generation,
with the ultimate goal of advancing patient care and clinical practice.

##  7 Acknowledgements

This work was funded by the European Union under the Horizon Europe grant
101095435. Views and opinions expressed are however those of the author(s)
only and do not necessarily reflect those of the European Union. Neither the
European Union nor the granting authority can be held responsible for them.

##  8 Statements and Declarations

The authors have no competing interests to declare that are relevant to the
content of this article.

##  9 CRediT author statement

Mahmoud Ibrahim: Conceptualization, Methodology, Investigation, Data Curation,
Writing, Visualization, Project Administration.Yasmina Al Khalil: Methodology,
Investigation, Writing, Visualization. Sina Amirrajab: Methodology,
Investigation, Writing. Chang Sun: Methodology, Investigation, Writing,
Funding Acquisition. Bart Elen: Writing - Review and Editing, Funding
Acquisition, Supervision. Gokhan Ertaylan: Funding Acquisition, Supervision.
Michel Dumontier: Funding Acquisition, Supervision.

## References

  * [1] Rajotte JF, Bergen R, Buckeridge DL, El Emam K, Ng R, Strome E.  Synthetic Data as an Enabler for Machine Learning Applications in Medicine.  iScience. 2022;25(11):105331.
  * [2] Liu Y, Dwivedi G, Boussaid F, Bennamoun M.  3D Brain and Heart Volume Generative Models: A Survey.  arxiv. 2022.
  * [3] Murtaza H, Ahmed M, Khan NF, Murtaza G, Zafar S, Bano A.  Synthetic Data Generation: State of the Art in Health Care Domain.  Computer Science Review. 2023;48:100546.
  * [4] Kazerouni A, Aghdam EK, Heidari M, Azad R, Fayyaz M, Hacihaliloglu I, et al.  Diffusion Models in Medical Imaging: A Comprehensive Survey.  Medical Image Analysis. 2023;88:102846.
  * [5] Koo H, Kim TE.  A Comprehensive Survey on Generative Diffusion Models for Structured Data.  arxiv. 2023.
  * [6] Berger L, Haberbusch M, Moscato F.  Generative Adversarial Networks in Electrocardiogram Synthesis: Recent Developments and Challenges.  Artificial Intelligence in Medicine. 2023;143:102632.
  * [7] Kebaili A, Lapuyade-Lahorgue J, Ruan S.  Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review.  Journal of Imaging. 2023;9(4):81.
  * [8] Yuan H, Zhou S, Yu S.  EHRDiff: Exploring Realistic EHR Synthesis with Diffusion Models.  arxiv. 2023.
  * [9] Chen Y, Yang XH, Wei Z, Heidari AA, Zheng N, Li Z, et al.  Generative Adversarial Networks in Medical Image Augmentation: A Review.  Computers in Biology and Medicine. 2022;144:105382.
  * [10] Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al.  PRISMA 2020 statement: systematic review and meta-analysis protocols (PRISMA-P) 2021 extension.  BMJ. 2021;374:n71.
  * [11] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al.  Generative adversarial nets.  Advances in neural information processing systems. 2014;27.
  * [12] Kingma DP, Welling M.  Auto-Encoding Variational Bayes.  arxiv. 2022.
  * [13] Sohl-Dickstein J, Weiss EA, Maheswaranathan N, Ganguli S.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics.  arxiv. 2015.
  * [14] Jordan MI, Ghahramani Z, Jaakkola TS, Saul LK.  An Introduction to Variational Methods for Graphical Models.  In: Learning in Graphical Models. Springer; 1999. p. 156-71.
  * [15] Ganguly A, Earp SWF.  An Introduction to Variational Inference.  arxiv. 2021.
  * [16] Kotelnikov A, Baranchuk D, Rubachev I, Babenko A.  TabDDPM: Modelling Tabular Data with Diffusion Models.  arxiv. 2022.
  * [17] Tian M, Chen B, Guo A, Jiang S, Zhang AR.  Fast and Reliable Generation of EHR Time Series via Diffusion Models.  arxiv. 2023.
  * [18] Ceritli, T , Ghosheh, G O , Chauhan, V K , et al.  Synthesizing Mixed-type Electronic Health Records Using Diffusion Models.  arxiv. 2023.
  * [19] Hoogeboom E, Nielsen D, Jaini P, Forré P, Welling M.  Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions.  arxiv. 2021.
  * [20] Donahue C, McAuley J, Puckette M.  Synthesizing audio with GANs.  arXiv. 2018.
  * [21] Raoof I, Gupta MK.  A Study of Semi Supervised Based Approaches for Motor Imagery Signal Generation.  In: 2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA). IEEE; 2021. p. 1312-6.
  * [22] Radford A, Metz L, Chintala S.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:151106434. 2015.
  * [23] Arjovsky M, Chintala S, Bottou L.  Wasserstein generative adversarial networks.  In: International conference on machine learning. PMLR; 2017. p. 214-23.
  * [24] Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S.  Least squares generative adversarial networks.  In: Proceedings of the IEEE international conference on computer vision; 2017. p. 2794-802.
  * [25] Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC.  Improved training of wasserstein gans.  Advances in neural information processing systems. 2017;30.
  * [26] Karras T, Aila T, Laine S, Lehtinen J.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:171010196. 2017.
  * [27] Alqahtani H, Kavakli-Thorne M, Kumar G.  Applications of generative adversarial networks (gans): An updated review.  Archives of Computational Methods in Engineering. 2021;28:525-52.
  * [28] Mirza M, Osindero S.  Conditional generative adversarial nets.  arXiv preprint arXiv:14111784. 2014.
  * [29] Isola P, Zhu JY, Zhou T, Efros AA.  Image-to-Image Translation with Conditional Adversarial Networks.  arxiv. 2018.
  * [30] Wang TC, Liu MY, Zhu JY, Tao A, Kautz J, Catanzaro B.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs.  arxiv. 2018.
  * [31] Zhu JY, Park T, Isola P, Efros AA.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.  arxiv. 2020.
  * [32] Wang T, Lei Y, Fu Y, Wynne JF, Curran WJ, Liu T, et al.  A review on medical imaging synthesis using deep learning and its clinical applications.  Journal of applied clinical medical physics. 2021;22(1):11-36.
  * [33] Karras T, Laine S, Aila T.  A style-based generator architecture for generative adversarial networks.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2019. p. 4401-10.
  * [34] Park T, Liu MY, Wang TC, Zhu JY.  Semantic image synthesis with spatially-adaptive normalization.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2019. p. 2337-46.
  * [35] Esser P, Rombach R, Ommer B.  Taming transformers for high-resolution image synthesis.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2021. p. 12873-83.
  * [36] Oord Avd, Vinyals O, Kavukcuoglu K.  Neural discrete representation learning.  arXiv preprint arXiv:171100937. 2017.
  * [37] Ho J, Jain A, Abbeel P.  Denoising Diffusion Probabilistic Models.  arxiv. 2020.
  * [38] Ronneberger O, Fischer P, Brox T.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  arxiv. 2015.
  * [39] Song J, Meng C, Ermon S.  Denoising Diffusion Implicit Models.  arxiv. 2022.
  * [40] Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B.  High-Resolution Image Synthesis with Latent Diffusion Models.  arxiv. 2022.
  * [41] Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al.  Learning Transferable Visual Models From Natural Language Supervision.  arxiv. 2021.
  * [42] Roberts A, Chung HW, Levskaya A, Mishra G, Bradbury J, Andor D, et al.  Scaling Up Models and Data with t5x and seqio.  arxiv. 2022.
  * [43] Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D.  Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction.  arxiv. 2020.
  * [44] Sennrich R, Haddow B, Birch A.  Neural Machine Translation of Rare Words with Subword Units.  arxiv. 2016.
  * [45] Liu F, Shareghi E, Meng Z, Basaldella M, Collier N.  Self-Alignment Pretraining for Biomedical Entity Representations.  arxiv. 2021.
  * [46] Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al.  A survey of large language models.  arXiv preprint arXiv:230318223. 2023.
  * [47] Lee SH.  Natural language generation for electronic health records.  NPJ digital medicine. 2018;1(1):63.
  * [48] Devlin J, Chang MW, Lee K, Toutanova K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:181004805. 2018.
  * [49] Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I, et al.  Language models are unsupervised multitask learners.  OpenAI blog. 2019;1(8):9.
  * [50] Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics. 2020;36(4):1234-40.
  * [51] Alsentzer E, Murphy JR, Boag W, Weng WH, Jin D, Naumann T, et al.  Publicly available clinical BERT embeddings.  arXiv preprint arXiv:190403323. 2019.
  * [52] Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:230213971. 2023.
  * [53] Chung HW, Hou L, Longpre S, Zoph B, Tay Y, Fedus W, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:221011416. 2022.
  * [54] Yale A, Dash S, Dutta R, Guyon I, Pavao A, Bennett KP.  Generation and Evaluation of Privacy Preserving Synthetic Health Data.  Neurocomputing. 2020;416:244-55.
  * [55] Dwork C.  Differential Privacy: A Survey of Results.  In: Agrawal M, Du D, Duan Z, Li A, editors. Theory and Applications of Models of Computation. vol. 4978. Springer Berlin Heidelberg; 2008. p. 1-19.
  * [56] Alaa AM, van Breugel B, Saveliev E, van der Schaar M.  How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models.  arxiv. 2022.
  * [57] Kuo NIH, Jorm L, Barbieri S.  Synthetic Health-related Longitudinal Data with Mixed-type Variables Generated Using Diffusion Models.  arxiv. 2023.
  * [58] Lu C, Reddy CK, Wang P, Nie D, Ning Y.  Multi-Label Clinical Time-Series Generation via Conditional GAN.  IEEE Transactions on Knowledge and Data Engineering. 2023:1-13.
  * [59] Alzantot, M , Garcia, L , Srivastava, M .  PhysioGAN: Training High Fidelity Generative Model for Physiological Sensor Readings.  arXiv. 2022.
  * [60] Pinaya WHL, Tudosiu PD, Dafflon J, da Costa PF, Fernandez V, Nachev P, et al.  Brain Imaging Generation with Latent Diffusion Models.  arxiv. 2022.
  * [61] Khader F, Müller-Franzes G, Tayebi Arasteh S, Han T, Haarburger C, Schulze-Hagen M, et al.  Denoising Diffusion Probabilistic Models for 3D Medical Image Generation.  Scientific Reports. 2023;13(1):7303.
  * [62] Dombrowski M, Reynaud H, Müller JP, Baugh M, Kainz B.  Trade-offs in Fine-tuned Diffusion Models Between Accuracy and Interpretability.  arxiv. 2023.
  * [63] Pan S, Wang T, Qiu RLJ, Axente M, Chang CW, Peng J, et al.  2D Medical Image Synthesis Using Transformer-Based Denoising Diffusion Probabilistic Model.  Physics in Medicine and Biology. 2023;68(10).
  * [64] Sun L, Chen J, Xu Y, Gong M, Yu K, Batmanghelich K.  Hierarchical Amortized GAN for 3D High Resolution Medical Image Synthesis.  IEEE journal of biomedical and health informatics. 2022;26(8):3966-75.
  * [65] Pan S, Wang T, Qiu RL, Axente M, Chang CW, Peng J, et al.  2D medical image synthesis using transformer-based denoising diffusion probabilistic model.  Physics in Medicine & Biology. 2023;68(10):105004.
  * [66] Lee KW, Chin RKY.  Diverse covid-19 CT image-to-image translation with stacked residual dropout.  Bioengineering. 2022;9(11):698.
  * [67] Toda R, Teramoto A, Kondo M, Imaizumi K, Saito K, Fujita H.  Lung cancer CT image generation from a free-form sketch using style-based pix2pix for data augmentation.  Scientific reports. 2022;12(1):12867.
  * [68] Abdelmotaal H, Sharaf M, Soliman W, Wasfi E, Kedwany SM.  Bridging the resources gap: deep learning for fluorescein angiography and optical coherence tomography macular thickness map image translation.  BMC ophthalmology. 2022;22(1):355.
  * [69] Tajmirriahi M, Kafieh R, Amini Z, Lakshminarayanan V.  A dual-discriminator Fourier acquisitive GAN for generating retinal optical coherence tomography images.  IEEE Transactions on Instrumentation and Measurement. 2022;71:1-8.
  * [70] Zhang Z, Yan C, Lasko TA, Sun J, Malin BA.  SynTEG: A Framework for Temporal Structured Electronic Health Data Simulation.  Journal of the American Medical Informatics Association : JAMIA. 2021;28(3):596-604.
  * [71] Biswal S, Ghosh S, Duke J, Malin B, Stewart W, Xiao C, et al.  EVA: Generating Longitudinal Electronic Health Records Using Conditional Variational Autoencoders.  In: Proceedings of Machine Learning Research. vol. 149; 2021. p. 260-82.
  * [72] Zhong Y, Cui S, Wang J, Wang X, Yin Z, Wang Y, et al.  MedDiffusion: Boosting Health Risk Prediction via Diffusion-based Data Augmentation.  arxiv. 2023.
  * [73] Wang Z, Sun J.  PromptEHR: Conditional Electronic Healthcare Records Generation with Prompt Learning.  arxiv. 2022.
  * [74] Theodorou B, Xiao C, Sun J.  Synthesize High-Dimensional Longitudinal Electronic Health Records via Hierarchical Autoregressive Language Model.  Nature Communications. 2023;14(1):5305.
  * [75] Dash S, Yale A, Guyon I, Bennett KP.  Medical Time-Series Data Generation Using Generative Adversarial Networks.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12299 LNAI; 2020. p. 382-91.
  * [76] Bing S, Dittadi A, Bauer S, Schwab P.  Conditional Generation of Medical Time Series for Extrapolation to Underrepresented Populations.  PLOS digital health. 2022;1(7):e0000074.
  * [77] Yoon J, Mizrahi M, Ghalaty NF, Jarvinen T, Ravi AS, Brune P, et al.  EHR-Safe: Generating High-Fidelity and Privacy-Preserving Synthetic Electronic Health Records.  NPJ digital medicine. 2023;6(1):141.
  * [78] Sun S, Wang F, Rashidian S, Kurc T, Abell-Hart K, Hajagos J, et al.  Generating Longitudinal Synthetic EHR Data with Recurrent Autoencoders and Generative Adversarial Networks.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12921 LNCS; 2021. p. 153-65.
  * [79] Li J, Cairns BJ, Li J, Zhu T.  Generating Synthetic Mixed-Type Longitudinal Electronic Health Records for Artificial Intelligent Applications.  NPJ digital medicine. 2023;6(1):98.
  * [80] van Hoorn R, Bakkes T, Tokoutsi Z, de Jong Y, Bouwman RA, Pechenizkiy M.  Generating Privacy-Preserving Longitudinal Synthetic Data.  In: NeurIPS 2023 Workshop on Synthetic Data Generation with Generative AI; 2023. p. .  Available from: https://openreview.net/forum?id=Xr13v66xxT.
  * [81] Torfi A, Fox EA, Reddy CK.  Differentially Private Synthetic Medical Data Generation Using Convolutional GANs.  Information Sciences. 2022;586:485-500.
  * [82] Yan C, Zhang Z, Nyemba S, Malin BA.  Generating Electronic Health Records with Multiple Data Types and Constraints.  In: AMIA Annual Symposium Proceedings. arXiv; 2020. p. 1335–1344.
  * [83] Naseer AA, Walker B, Landon C, Ambrosy A, Fudim M, Wysham N, et al.  ScoEHR: Generating Synthetic Electronic Health Records Using Continuous-time Diffusion Models.  In: Proceedings of Machine Learning Research 219:1–22, 2023; 2023. p. 489-508.
  * [84] He H, Zhao S, Xi Y, Ho JC.  MedDiff: Generating Electronic Health Records Using Accelerated Denoising Diffusion Model.  arXiv. 2023.
  * [85] Yoon J, Drumright LN, van der Schaar M.  Anonymization Through Data Synthesis Using Generative Adversarial Networks (ADS-GAN).  IEEE journal of biomedical and health informatics. 2020;24(8):2378-88.
  * [86] Venugopal R, Shafqat N, Venugopal I, Tillbury BMJ, Stafford HD, Bourazeri A.  Privacy Preserving Generative Adversarial Networks to Model Electronic Health Records.  Neural networks : the official journal of the International Neural Network Society. 2022;153:339-48.
  * [87] Fang ML, Dhami DS, Kersting K.  DP-CTGAN: Differentially Private Medical Data Generation Using CTGANs.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 13263 LNAI; 2022. p. 178-88.
  * [88] Rodriguez-Almeida AJ, Fabelo H, Ortega S, Deniz A, Balea-Fernandez FJ, Quevedo E, et al.  Synthetic Patient Data Generation and Evaluation in Disease Prediction Using Small and Imbalanced Datasets.  IEEE journal of biomedical and health informatics. 2023;27(6):2670-80.
  * [89] Perets, O , Rappoport, N .  Ensemble Synthetic EHR Generation for Increasing Subpopulation Model’s Performance.  arxiv. 2023.
  * [90] Bietsch D, Stahlbock R, Voß S.  Synthetic Data as a Proxy for Real-World Electronic Health Records in the Patient Length of Stay Prediction.  Sustainability (Switzerland). 2023;15(18).
  * [91] Zhang Z, Yan C, Mesa DA, Sun J, Malin BA.  Ensuring Electronic Medical Record Simulation through Better Training, Modeling, and Evaluation.  Journal of the American Medical Informatics Association : JAMIA. 2019;27(1):99-108.
  * [92] Yoon J, Jarrett D, van der Schaar M.  Time-Series Generative Adversarial Networks.  In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc.; 2019. p. .
  * [93] Adib, E , Afghah, F , Prevost, J J .  Synthetic ECG Signal Generation Using Generative Neural Networks.  arXiv. 2021.
  * [94] Choi E, Biswal S, Malin B, Duke J, Stewart WF, Sun J.  Generating Multi-label Discrete Patient Records Using Generative Adversarial Networks.  In: Proceedings of the 2nd Machine Learning for Healthcare Conference. PMLR; 2017. p. 286-305.
  * [95] Baowaly MK, Lin CC, Liu CL, Chen KT.  Synthesizing Electronic Health Records Using Improved Generative Adversarial Networks.  Journal of the American Medical Informatics Association : JAMIA. 2019;26(3):228-41.
  * [96] Hjelm RD, Jacob AP, Che T, Trischler A, Cho K, Bengio Y.  Boundary-Seeking Generative Adversarial Networks.  arxiv. 2018.
  * [97] Goncalves A, Ray P, Soper B, Stevens J, Coyle L, Sales AP.  Generation and Evaluation of Synthetic Patient Data.  BMC medical research methodology. 2020;20(1):108.
  * [98] Ho J, Salimans T.  Classifier-Free Diffusion Guidance.  arxiv. 2022.
  * [99] Mironov I.  Rényi Differential Privacy.  In: 2017 IEEE 30th Computer Security Foundations Symposium (CSF). IEEE; 2017. p. .
  * [100] Xu L, Skoularidou M, Cuesta-Infante A, Veeramachaneni K.  Modeling Tabular data using Conditional GAN.  arxiv. 2019.
  * [101] Bhanot, K , Baldini, I , Wei, D , et al.  Downstream Fairness Caveats with Synthetic Healthcare Data.  arxiv. 2022.
  * [102] Weldon, J , Ward, T , Brophy, E .  Generation of Synthetic Electronic Health Records Using a Federated GAN.  arxiv. 2021.
  * [103] Nik AHZ, Riegler MA, Halvorsen P, Storås AM.  Generation of Synthetic Tabular Healthcare Data Using Generative Adversarial Networks.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 13833 LNCS; 2023. p. 434-46.
  * [104] Yoo H, Moon J, Kim JH, Joo HJ.  Design and Technical Validation to Generate a Synthetic 12-Lead Electrocardiogram Dataset to Promote Artificial Intelligence Research.  Health Information Science and Systems. 2023;11(1).
  * [105] Banerjee R, Ghose A.  Synthesis of Realistic ECG Waveforms Using a Composite Generative Adversarial Network for Classification of Atrial Fibrillation.  In: European Signal Processing Conference. vol. 2021-August; 2021. p. 1145-9.
  * [106] Yang Z, Li Y, Zhou G.  TS-GAN: Time-series GAN for Sensor-based Health Data Augmentation.  ACM Transactions on Computing for Healthcare. 2023;4(2).
  * [107] Zhang, Y -H , Babaeizadeh, S .  Synthesis of Standard 12-Lead Electrocardiograms Using Two Dimensional Generative Adversarial Network.  Journal of Electrocardiology. 2021.
  * [108] Thambawita V, Isaksen JL, Hicks SA, Ghouse J, Ahlberg G, Linneberg A, et al.  DeepFake Electrocardiograms Using Generative Adversarial Networks Are the Beginning of the End for Privacy Issues in Medicine.  Scientific Reports. 2021;11(1).
  * [109] Li X, Metsis V, Wang H, Ngu AHH.  TTS-GAN: A Transformer-Based Time-Series Generative Adversarial Network.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 13263 LNAI; 2022. p. 133-43.
  * [110] Adib, E , Fernandez, A , Afghah, F , et al.  Synthetic ECG Signal Generation Using Probabilistic Diffusion Models.  arxiv. 2023.
  * [111] Zama MH, Schwenker F.  ECG Synthesis via Diffusion-Based State Space Augmented Transformer.  Sensors. 2023;23(19).
  * [112] Neifar, N , Ben-Hamadou, A , Mdhaffar, A , et al.  DiffECG: A Generalized Probabilistic Diffusion Model for ECG Signals Synthesis.  arxiv. 2023.
  * [113] Xia Y, Wang W, Wang K.  ECG Signal Generation Based on Conditional Generative Models.  Biomedical Signal Processing and Control. 2023;82:104587.
  * [114] Adib, E , Afghah, F , Prevost, J J . Arrhythmia Classification Using CGAN-Augmented ECG Signals; 2022.
  * [115] Nankani D, Dutta Baruah R.  Improved Diagnostic Performance of Arrhythmia Classification Using Conditional GAN Augmented Heartbeats.  In: Intelligent Systems Reference Library. vol. 217; 2022. p. 275-304.
  * [116] Sang Y, Beetz M, Grau V.  Generation of 12-Lead Electrocardiogram with Subject-Specific, Image-Derived Characteristics Using a Conditional Variational Autoencoder.  In: Proceedings - International Symposium on Biomedical Imaging. vol. 2022-March; 2022. .
  * [117] Chung, H , Kim, J , Kwon, J -M , et al.. TEXT-TO-ECG: 12-LEAD ELECTROCARDIOGRAM SYNTHESIS CONDITIONED ON CLINICAL TEXT REPORTS; 2023.
  * [118] Bagga, M , Jeon, H , Issokson, A . ECGNet: A Generative Adversarial Network (GAN) Approach to the Synthesis of 12-Lead ECG Signals from Single Lead Inputs; 2023.
  * [119] Jo, Y -Y , Choi, Y S , Jang, J -H , et al.  ECGT2T: Towards Synthesizing Twelve-Lead Electrocardiograms from Two Asynchronous Leads.  arXiv. 2021.
  * [120] Shome, D , Sarkar, P , Etemad, A .  Region-Disentangled Diffusion Model for High-Fidelity PPG-to-ECG Translation.  arXiv. 2023.
  * [121] Dissanayake T, Fernando T, Denman S, Sridharan S, Fookes C.  Generalized Generative Deep Learning Models for Biosignal Synthesis and Modality Transfer.  IEEE Journal of Biomedical and Health Informatics. 2023;27(2):968-79.
  * [122] Agarwal E.  CardiacGen: A Hierarchical Deep Generative Model for Cardiac Signals.  In: Machine Learning for Health; 2022. .
  * [123] Neifar N, Mdhaffar A, Ben-Hamadou A, Jmaiel M, Freisleben B.  Disentangling Temporal and Amplitude Variations in ECG Synthesis Using Anchored GANs.  In: Proceedings of the ACM Symposium on Applied Computing; 2022. p. 645-52.
  * [124] Alcaraz JML, Strodthoff N.  Diffusion-Based Conditional ECG Generation with Structured State Space Models.  Computers in Biology and Medicine. 2023;163.
  * [125] Mogren O. C-RNN-GAN: Continuous recurrent neural networks with adversarial training; 2016.
  * [126] Gu A, Goel K, Ré C. Efficiently Modeling Long Sequences with Structured State Spaces; 2022.
  * [127] Alcaraz JML, Strodthoff N. Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models; 2023.
  * [128] Aristimunha B, de Camargo RY, Chevallier S, Thomas AG, Lucena O, Cardoso MJ, et al.  Synthetic Sleep EEG Signal Generation Using Latent Diffusion Models.  arXiv. 2023.
  * [129] Li H, Yu S, Principe J.  Causal Recurrent Variational Autoencoder for Medical Time Series Generation.  In: Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023. vol. 37; 2023. p. 8562-70.
  * [130] Granger CWJ.  Investigating Causal Relations by Econometric Models and Cross-spectral Methods.  Econometrica. 1969;37(3):424-38.  Available from: http://www.jstor.org/stable/1912791.
  * [131] Lee JS, Hsiang J. Patent Claim Generation by Fine-Tuning OpenAI GPT-2; 2019.
  * [132] Nishi Y, Suge A, Takahashi H.  Text analysis on the stock market in the automotive industry through fake news generated by GPT-2.  Proceedings of the Artificial Intelligence of and for Business. 2019.
  * [133] Bird JJ, Pritchard M, Fratini A, Ekart A, Faria DR.  Synthetic Biological Signals Machine-Generated by GPT-2 Improve the Classification of EEG and EMG through Data Augmentation.  IEEE Robotics and Automation Letters. 2021;6(2):3498-504.
  * [134] Kalashami MP, Pedram MM, Sadr H.  EEG Feature Extraction and Data Augmentation in Emotion Recognition.  Computational Intelligence and Neuroscience. 2022;2022.
  * [135] Vetter, J , Macke, J H , Gao, R .  Generating Realistic Neurophysiological Time Series with Denoising Diffusion Probabilistic Models.  bioRxiv. 2023.
  * [136] Shu, K , Zhao, Y , Wu, L , et al.  Data Augmentation for Seizure Prediction with Generative Diffusion Model.  arXiv. 2023.
  * [137] Ganti B, Chaitanya G, Balamurugan RS, Nagaraj N, Balasubramanian K, Pati S.  Time-Series Generative Adversarial Network Approach of Deep Learning Improves Seizure Detection From the Human Thalamic SEEG.  Frontiers in Neurology. 2022;13.
  * [138] Pérez E, Ventura S.  Progressive Growing of Generative Adversarial Networks for Improving Data Augmentation and Skin Cancer Diagnosis.  Artificial intelligence in medicine. 2023;141:102556.
  * [139] Mohanty, A , Sutherland, A , Bezbradica, M , et al.  High Fidelity Synthetic Face Generation for Rosacea Skin Condition from Limited Data.  Research Square. 2023.
  * [140] Teodoro AAM, Silva DH, Rosa RL, Saadi M, Wuttisittikulkij L, Mumtaz RA, et al.  A Skin Cancer Classification Approach Using GAN and RoI-Based Attention Mechanism.  Journal of Signal Processing Systems. 2023;95(2-3):211-24.
  * [141] Zhao C, Shuai R, Ma L, Liu W, Hu D, Wu M.  Dermoscopy Image Classification Based on StyleGAN and DenseNet201.  IEEE Access. 2021;9:8659-79.
  * [142] Mikołajczyk A, Majchrowska S, Carrasco Limeros S.  The (de)Biasing Effect of GAN-Based Augmentation Methods on Skin Lesion Images.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 13438 LNCS; 2022. p. 437-47.
  * [143] Limeros, S C , Majchrowska, S , Zoubi, M K , et al.  GAN-based Generative Modelling for Dermatological Applications - Comparative Study.  arXiv. 2022.
  * [144] Kaur R, Gholamhosseini H, Sinha R.  Synthetic Images Generation Using Conditional Generative Adversarial Network for Skin Cancer Classification.  In: IEEE Region 10 Annual International Conference, Proceedings/TENCON. vol. 2021-December; 2021. p. 381-6.
  * [145] Sagers, L W , Diao, J A , Melas-Kyriazi, L , et al.  AUGMENTING MEDICAL IMAGE CLASSIFIERS WITH SYNTHETIC DATA FROM LATENT DIFFUSION MODELS.  arXiv. 2023.
  * [146] Akrout, M , Gyepesi, B , Holló, P , et al.  Diffusion-Based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images.  arXiv. 2023.
  * [147] Sagers, L W , Groh, M , Adamson, A S , et al.  Improving Dermatology Classifiers across Populations Using Images Generated by Large Diffusion Models.  arXiv. 2022.
  * [148] Wang H, Qi Q, Sun W, Li X, Dong B, Yao C.  Classification of Skin Lesions with Generative Adversarial Networks and Improved MobileNetV2.  International Journal of Imaging Systems and Technology. 2023;33(5):1561-76.
  * [149] TB; F. The validity and practicality of sun-reactive skin types I through VI. U.S. National Library of Medicine;.  Available from: https://pubmed.ncbi.nlm.nih.gov/3377516/.
  * [150] Xie Q, Dai Z, Hovy E, Luong MT, Le QV. Unsupervised Data Augmentation for Consistency Training; 2020.
  * [151] Zhong Z, Zheng L, Kang G, Li S, Yang Y. Random Erasing Data Augmentation; 2017.
  * [152] Zhao S, Liu Z, Lin J, Zhu JY, Han S. Differentiable Augmentation for Data-Efficient GAN Training; 2020.
  * [153] Ruiz N, Li Y, Jampani V, Pritch Y, Rubinstein M, Aberman K. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation; 2023.
  * [154] Szafranowska Z, Osuala R, Breier B, Kushibar K, Lekadir K, Diaz O.  Sharing Generative Models Instead of Private Data: A Simulation Study on Mammography Patch Classification.  In: Proceedings of SPIE - The International Society for Optical Engineering. vol. 12286; 2022. .
  * [155] Joseph AJ, Dwivedi P, Joseph J, Francis S, Pournami PN, Jayaraj PB, et al.  Prior-Guided Generative Adversarial Network for Mammogram Synthesis.  Biomedical Signal Processing and Control. 2024;87.
  * [156] Oyelade ON, Ezugwu AE, Almutairi MS, Saha AK, Abualigah L, Chiroma H.  A Generative Adversarial Network for Synthetization of Regions of Interest Based on Digital Mammograms.  Scientific Reports. 2022;12(1).
  * [157] Montoya-del Angel R, Sam-Millan K, Vilanova JC, Martí R. MAM-E: Mammographic Synthetic Image Generation with Diffusion Models; 2023.
  * [158] Garrucho L, Kushibar K, Osuala R, Diaz O, Catanese A, del Riego J, et al.  High-Resolution Synthesis of High-Density Breast Mammograms: Application to Improved Fairness in Deep Learning Based Mass Detection.  Frontiers in Oncology. 2023;12.
  * [159] Yamazaki A, Ishida T.  Two-View Mammogram Synthesis from Single-View Data Using Generative Adversarial Networks.  Applied Sciences (Switzerland). 2022;12(23).
  * [160] Aizaz Z, Khare K, Khursheed A, Tirmizi A.  Pix2Pix Generative Adversarial Networks (GAN) for Breast Cancer Detection.  In: 2022 5th International Conference on Multimedia, Signal Processing and Communication Technologies, IMPACT 2022; 2022. .
  * [161] Lee J, Nishikawa RM.  Identifying Women with Mammographically- Occult Breast Cancer Leveraging GAN-Simulated Mammograms.  IEEE Transactions on Medical Imaging. 2022;41(1):225-36.
  * [162] Guan Q, Chen Y, Wei Z, Heidari AA, Hu H, Yang XH, et al.  Medical Image Augmentation for Lesion Detection Using a Texture-Constrained Multichannel Progressive GAN.  Computers in Biology and Medicine. 2022;145.
  * [163] Zhou Y, Wei J, Wu D, Zhang Y.  Generating Full-Field Digital Mammogram From Digitized Screen-Film Mammogram for Breast Cancer Screening With High-Resolution Generative Adversarial Network.  Frontiers in Oncology. 2022;12.
  * [164] Shen T, Hao K, Gou C, Wang FY.  Mass Image Synthesis in Mammogram with Contextual Information Based on GANs.  Computer Methods and Programs in Biomedicine. 2021;202.
  * [165] Giess CS, Frost EP, Birdwell RL.  Interpreting one-view mammographic findings: Minimizing callbacks while maximizing cancer detection.  Radiographics. 2014;34(4):928-40.
  * [166] Kerlikowske K, Zhu W, Tosteson AN, Sprague BL, Tice JA, Lehman CD, et al.  Identifying women with dense breasts at high risk for interval cancer: a cohort study.  Breast Cancer Surveillance Consortium (BCSC). 2015;19(2):236-48.
  * [167] Lee RS, Gimenez F, Hoogi A, Miyake KK, Gorovoy M, Rubin DL.  A curated mammography data set for use in computer-aided detection and diagnosis research.  Scientific Data. 2017 Dec;4(1).
  * [168] Bowyer KW.  Digital Image Database with gold standard and performance metrics for Mammographic Image Analysis Research. 1999 Aug.
  * [169] Halling-Brown MD, Warren LM, Ward D, Lewis E, Mackenzie A, Wallis MG, et al.. OPTIMAM Mammography Image Database: a large scale resource of mammography images and clinical data; 2020.
  * [170] Moreira IC, Amaral I, Domingues I, Cardoso A, Cardoso MJ, Cardoso JS.  INbreast: Toward a Full-Field Digital Mammographic Database.  Academic Radiology. 2011;19(2):236-48.
  * [171] Iskandar, M , Mannering, H , Sun, Z , et al.  Towards Realistic Ultrasound Fetal Brain Imaging Synthesis.  arXiv. 2023.
  * [172] Liang J, Chen J.  Data Augmentation of Thyroid Ultrasound Images Using Generative Adversarial Network.  In: IEEE International Ultrasonics Symposium, IUS; 2021. .
  * [173] Montero A, Bonet-Carne E, Burgos-Artizzu XP.  Generative Adversarial Networks to Improve Fetal Brain Fine-Grained Plane Classification.  Sensors (Basel, Switzerland). 2021;21(23).
  * [174] Maack L, Holstein L, Schlaefer A.  GANs for Generation of Synthetic Ultrasound Images from Small Datasets.  Current Directions in Biomedical Engineering. 2022;8(1):17-20.
  * [175] Che H, Ramanathan S, Foran DJ, Nosher JL, Patel VM, Hacihaliloglu I.  Realistic Ultrasound Image Synthesis for Improved Classification of Liver Disease.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12967 LNCS; 2021. p. 179-88.
  * [176] Pang T, Wong JHD, Ng WL, Chan CS.  Semi-Supervised GAN-based Radiomics Model for Data Augmentation in Breast Ultrasound Mass Classification.  Computer Methods and Programs in Biomedicine. 2021;203.
  * [177] Tan H, Lang X, He B, Lu Y, Zhang Y.  GAN-based Medical Image Augmentation for Improving CNN Performance in Myositis Ultrasound Image Classification.  In: 2023 6th International Conference on Electronics Technology, ICET 2023; 2023. p. 1329-33.
  * [178] Gong X, Zhou H, Gu Y, Guo Y.  Breast Ultrasound Image Classification with Hard Sample Generation and Semi-Supervised Learning.  Biomedical Signal Processing and Control. 2023;86.
  * [179] Stojanovski D, Hermida U, Lamata P, Beqiri A, Gomez A.  Echo from Noise: Synthetic Ultrasound Image Generation Using Diffusion Models for Real Image Segmentation.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 14337 LNCS; 2023. p. 34-43.
  * [180] Yao Z, Luo T, Dong YJ, Jia XH, Deng YH, Wu GQ, et al.  Virtual Elastography Ultrasound via Generative Adversarial Network for Breast Cancer Diagnosis.  Nature Communications. 2023;14(1).
  * [181] Zhang Q, Zhao J, Long X, Luo Q, Wang R, Ding X, et al.  AUE-Net: Automated Generation of Ultrasound Elastography Using Generative Adversarial Network.  Diagnostics. 2022;12(2).
  * [182] Qiao S, Pan S, Luo G, Pang S, Chen T, Singh AK, et al.  A Pseudo-Siamese Feature Fusion Generative Adversarial Network for Synthesizing High-Quality Fetal Four-Chamber Views.  IEEE Journal of Biomedical and Health Informatics. 2023;27(3):1193-204.
  * [183] Liang J, Yang X, Huang Y, Li H, He S, Hu X, et al.  Sketch Guided and Progressive Growing GAN for Realistic and Editable Ultrasound Image Synthesis.  Medical Image Analysis. 2022;79.
  * [184] Dorent R, Haouchine N, Kogl F, Joutard S, Juvekar P, Torio E, et al.  Unified Brain MR-Ultrasound Synthesis Using Multi-modal Hierarchical Representations.  In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 14229 LNCS; 2023. p. 448-58.
  * [185] Chen L, Liao H, Kong W, Zhang D, Chen F.  Anatomy Preserving GAN for Realistic Simulation of Intraoperative Liver Ultrasound Images.  Computer Methods and Programs in Biomedicine. 2023;240.
  * [186] Jin S, Monkam P, Lu W.  Echocardiography Segmentation Based on Cross-modal Data Augmentation Method.  In: IEEE International Ultrasonics Symposium, IUS. vol. 2022-October; 2022. .
  * [187] Song Y, Zheng J, Lei L, Ni Z, Zhao B, Hu Y.  CT2US: Cross-modal Transfer Learning for Kidney Segmentation in Ultrasound Images with Synthesized Data.  Ultrasonics. 2022;122.
  * [188] Tiago C, Gilbert A, Beela AS, Aase SA, Snare SR, Šprem J, et al.  A Data Augmentation Pipeline to Generate Synthetic Labeled Datasets of 3D Echocardiography Images Using a GAN.  IEEE Access. 2022;10:98803-15.
  * [189] Sigrist RM, Liau J, El Kaffas A, Chammas MC, Willmann JK.  Ultrasound elastography: review of techniques and clinical applications.  Theranostics. 2017;7(5):1303.
  * [190] Isola P, Zhu JY, Zhou T, Efros AA. Image-to-Image Translation with Conditional Adversarial Networks; 2018.
  * [191] Gilbert A, Marciniak M, Rodero C, Lamata P, Samset E, McLeod K.  Generating Synthetic Labeled Data from Existing Anatomical Models: An Example with Echocardiography Segmentation.  IEEE Transactions on Medical Imaging. 2021;40(10):2783-94.
  * [192] Chen X, Duan Y, Houthooft R, Schulman J, Sutskever I, Abbeel P. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets; 2016.
  * [193] La Rosa F, Yu T, Barquero G, Thiran JP, Granziera C, Bach Cuadra M.  MPRAGE to MP2RAGE UNI translation via generative adversarial network improves the automatic tissue and lesion segmentation in multiple sclerosis patients.  Comput Biol Med. 2021;132:104297.  Place: United States.
  * [194] Qin Z, Liu Z, Zhu P, Ling W.  Style transfer in conditional GANs for cross-modality synthesis of brain magnetic resonance images.  Comput Biol Med. 2022;148:105928.  Place: United States.
  * [195] Zhu L, He Q, Huang Y, Zhang Z, Zeng J, Lu L, et al.  DualMMP-GAN: Dual-scale multi-modality perceptual generative adversarial network for medical image segmentation.  Comput Biol Med. 2022;144:105387.  Place: United States.
  * [196] Fei Y, Zhan B, Hong M, Wu X, Zhou J, Wang Y.  Deep learning-based multi-modal computing with feature disentanglement for MRI image synthesis.  Med Phys. 2021;48(7):3778-89.  Place: United States.
  * [197] Huo J, Vakharia V, Wu C, Sharan A, Ko A, Ourselin S, et al.  Brain Lesion Synthesis via Progressive Adversarial Variational Auto-Encoder.  In: Zhao C, Svoboda D, Wolterink JM, Escobar M, editors. Simulation and Synthesis in Medical Imaging. Lecture Notes in Computer Science. Springer International Publishing; 2022. p. 101-11.
  * [198] Dhaene AP, Loecher M, Wilson AJ, Ennis DB.  Myocardial Segmentation of Tagged Magnetic Resonance Images with Transfer Learning Using Generative Cine-To-Tagged Dataset Transformation.  Bioengineering. 2023;10(2).
  * [199] Chen X, Lian C, Wang L, Deng H, Kuang T, Fung SH, et al.  Diverse data augmentation for learning image segmentation with cross-modality annotations.  Med Image Anal. 2021;71:102060.  Place: Netherlands.
  * [200] Yurt M, Dar SU, Erdem A, Erdem E, Oguz KK, Çukur T.  mustGAN: multi-stream Generative Adversarial Networks for MR Image Synthesis.  Med Image Anal. 2021;70:101944.  Place: Netherlands.
  * [201] Moshe YH, Buchsweiler Y, Teicher M, Artzi M.  Handling Missing MRI Data in Brain Tumors Classification Tasks: Usage of Synthetic Images vs. Duplicate Images and Empty Images.  J Magn Reson Imaging. 2023.  Place: United States.
  * [202] Fan CC, Peng L, Wang T, Yang H, Zhou XH, Ni ZL, et al.  TR-GAN: Multi-Session Future MRI Prediction With Temporal Recurrent Generative Adversarial Network.  IEEE Trans Med Imaging. 2022;41(8):1925-37.  Place: United States.
  * [203] Huang P, Li D, Jiao Z, Wei D, Cao B, Mo Z, et al.  Common feature learning for brain tumor MRI synthesis by context-aware generative adversarial network.  Med Image Anal. 2022;79:102472.  Place: Netherlands.
  * [204] Amirrajab S, Al Khalil Y, Lorenz C, Weese J, Pluim J, Breeuwer M.  Label-informed cardiac magnetic resonance image synthesis through conditional generative adversarial networks.  Comput Med Imaging Graph. 2022;101:102123.  Place: United States.
  * [205] Al Khalil Y, Amirrajab S, Lorenz C, Weese J, Pluim J, Breeuwer M.  On the usability of synthetic data for improving the robustness of deep learning-based segmentation of cardiac magnetic resonance images.  Medical Image Analysis. 2023;84.
  * [206] Lustermans DRPRM, Amirrajab S, Veta M, Breeuwer M, Scannell CM.  Optimized automated cardiac MR scar quantification with GAN-based data augmentation.  Comput Methods Programs Biomed. 2022;226:107116.  Place: Ireland.
  * [207] Thermos S, Liu X, O’Neil A, Tsaftaris SA.  Controllable Cardiac Synthesis via Disentangled Anatomy Arithmetic.  In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 12903 LNCS; 2021. p. 160-70.
  * [208] Amirrajab S, Lorenz C, Weese J, Pluim J, Breeuwer M.  Pathology Synthesis of 3D Consistent Cardiac MR Images Using 2D VAEs and GANs.  In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 13570 LNCS; 2022. p. 34-42.