11institutetext: Department of Statistics, LMU Munich
22institutetext: Department of Radiology, University Hospital, LMU Munich
33institutetext: Munich Center for Machine Learning (MCML)
33email<EMAIL_ADDRESS>
# Cascaded Latent Diffusion Models for High-Resolution Chest X-ray Synthesis

Tobias Weber 112233 0000-0002-5430-2595 Michael Ingrisch 2233
0000-0003-0268-9078 Bernd Bischl 1133 0000-0001-6002-6980 David Rügamer 1133
0000-0002-8772-9202

###### Abstract

While recent advances in large-scale foundational models show promising
results, their application to the medical domain has not yet been explored in
detail. In this paper, we progress into the realms of large-scale modeling in
medical synthesis by proposing Cheff \- a foundational cascaded latent
diffusion model, which generates highly-realistic chest radiographs providing
state-of-the-art quality on a 1-megapixel scale. We further propose MaCheX,
which is a unified interface for public chest datasets and forms the largest
open collection of chest X-rays up to date. With Cheff conditioned on
radiological reports, we further guide the synthesis process over text prompts
and unveil the research area of report-to-chest-X-ray generation.

###### Keywords:

latent diffusion model chest radiograph image synthesis.

##  1 Introduction

Chest X-ray examinations are one of the most common, if not the most common,
procedures in everyday clinical practice. This not only enables the collection
of large databases but paves the way for lots of great opportunities to
advance medical AI assistance in clinical workflows. Automated lung disease
diagnosis [21], for example, can help to accelerate an examination and assist
radiologists by reducing human error in a high-stress environment. Despite the
availability of public datasets, many challenges in practical usefulness
remain, partly induced due to inherent class imbalances and noisy labels. One
possible approach to tackle this problem is by synthesizing underrepresented
classes via generative modeling [31]. Generative models provide various other
opportunities to improve clinical routines by, e.g., modeling the
characteristics of diseases in different degrees using the patient’s original
thoracic scan [35] or suppressing bones to enhance soft tissue within the lung
[8].

The basis for these methods is a stable and high-quality synthesis process.
While previous evaluations in clinical practice focus on generative
adversarial networks (GANs; see, e.g., [28]), the trend of generative
architectures moves away from an adversarial setting. Current state-of-the-art
approaches tend to employ large-scale autoencoders with a focus on generating
a prior in their latent space [23, 5, 25]. Moreover, diffusion models [30, 10]
have shown immense potential in image synthesization [4, 17] resulting in a
variety of proposed large-scale models including GLIDE [18], DALL·E 2 [22],
Imagen [26] or Stable Diffusion [25].

Recently, GoogleAI released an API CXR Foundation Tool offering chest
radiograph embeddings based on a contrastive neural network [29]. The
underlying training dataset contains a significant proportion of non-disclosed
proprietary data, though. The imminent risk of biases in foundational chest
X-ray models has been recently discussed in [7] warning about the consequences
in real-world applications. While these biases are inherent in the existing
data, closed-source analyses make it difficult for the research community to
counteract these problems.

Figure 1: High-resolution synthetic chest X-rays generated with our cascaded
latent diffusion pipeline Cheff.

##### Our Contributions

In this paper, we adapt large-scale learning concepts and models of general
computer vision synthesis to chest radiography. To make this possible, our
first contribution is to provide a unified interface to a massive collection
of public chest X-ray datasets (MaCheX) with over 650,000 scans.111MaChex:
https://github.com/saiboxx/machex This dataset allows the training of Cheff, a
cascaded chest X-ray latent diffusion pipeline, our second
contribution.222Cheff: https://github.com/saiboxx/chexray-diffusion The basis
for Cheff are two foundational chest X-ray models: (i) an autoencoder for
obtaining chest X-ray embeddings and (ii) a super-resolution diffusion model
for refining low-resolution scans. A task-specific diffusion process in the
latent space of the autoencoder leads to high-fidelity and diverse synthesis
(see, e.g., Figure 1). This further enables variable conditioning mechanisms
while reducing computational training costs due to the usage of shared
foundational models in the full pipeline. Finally, our model provides
realistic report-to-chest-X-ray conversion by applying our model stack in a
text-to-image setting by conditioning on radiologists’ reports. Overall, our
approach pushes the limits in state-of-the-art synthesis of radiological
images and fosters development of downstream clinical AI assistance tools by
making both our model and dataset publicly accessible.

##  2 Related Work

In recent years, the rising popularity of generative models resulted in
various approaches focusing on chest X-ray synthesis. [28] investigate the
class-guided synthesis of targeted pathologies over classifier gradients.
Their training is, however, only based on a single-center dataset while using
a progressive-growing GAN ([13]; PGAN) known to be less capable than other
more recently proposed algorithms. [35] obtain chest scan embeddings by
applying GAN inversion to the generator of [28] and investigate
characteristics of pathologies in this latent space. [9] utilize privacy-free
synthetic chest X-rays for federated learning to ensure data protection across
hospitals. [32] proposes a decomposition of diseases, ultimately producing
anomaly saliency maps, by adding a second generative network exclusive for
abnormal scans to produce a residual for a healthy version of this scan. Other
use cases of generative methods include bone suppression [8] or improving
classifier performance on underrepresented classes [31].

Apart from the classic adversarial literature, there is a surge in diffusion
model research that also deals with chest X-rays. [36] propose a routine
training on normal data and then utilizes a partial denoising process to
detect anomalies. [3] fine-tune the foundational Stable Diffusion on a few
thousand scans. While the synthesized scans lack a realistic style, it allows
for a first glimpse into how report-to-scan generation can be used in medical
imaging. [20] promote the usage of diffusion models over PGANs by fitting a
latent diffusion model (LDM; [25]) on chest X-rays and observing a performance
boost in a synthetically aided classifier. Additionally, the synthesizing
process ensures anonymity with respect to the patient retrieval problem by
employing a privacy-enhancing sampling algorithm [19]. A different strategy
involves prompting DALL·E 2 to investigate zero-shot capabilities for medical
imaging synthesis, however with limited success [1]. Generative modeling for
automated radiological reports from chest X-rays is another highly active
research area (cf. [15, 14]), whereas the topic of creating images from
reports is still an underrepresented, if not an unexplored, subject.

##  3 Methods

In the general framework of diffusion models [30, 10], a Markov chain
$(\bm{X}_{t})_{t=1,\ldots,T}$ with joint distribution defined by the density
$q(\bm{x}_{1:T})\coloneqq\prod^{T}_{t=1}q(\bm{x}_{t}|\bm{x}_{t-1})$ is used to
model a diffusion process of an uncorrupted image $\bm{x}_{0}\in\mathcal{X}$.
This is done by assuming
$\bm{X}_{t}\mid\bm{X}_{t-1}\\!=\\!\bm{x}_{t-1}\sim\mathcal{N}(\sqrt{1-\beta_{t}}\bm{x}_{t-1},\beta_{t}\bm{I})$
with time-varying variance $\beta_{t}\in(0,1)$, which is chosen by a pre-
defined variance schedule. In other words, this autoregressive process
incrementally applies Gaussian noise to an input $\bm{x}_{0}$ for a number of
timesteps $T$. The learning task then is to revert this so-called forward
process, i.e., inferring $\bm{x}_{t-1}$ from its corrupted version
$\bm{x}_{t}$ for $t=1,...,T$. For large $T$ the result $\bm{x}_{T}$ will
approximate isotropic Gaussian noise and the learned reverse process can be
utilized as a powerful iterative generative model to model structured
information from noise.

A simplified training objective (c.f. [25, 10]) can be formulated by
minimizing

$\mathbb{E}_{t\sim
U(1,T),\bm{x}_{0}\in\mathcal{X},\bm{\epsilon}_{t}\sim\mathcal{N}(\bm{0},\bm{1})}\left[\|\bm{\epsilon}_{t}-\bm{\epsilon}_{\theta}(\bm{x}_{t},t)\|^{2}_{2}\right]\,\text{,}$
(1)

where $\bm{\epsilon}_{\theta}$ usually is a time-conditional U-net [6]
parameterized with $\theta$ to predict the noise $\bm{\epsilon}_{t}$ in
$\bm{x}_{t}$ as a function of $\bm{x}_{0}$ for uniformly drawn $t$ from
{1,…,T} using the fact that
$\bm{x}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{t}$
with $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$.

##### Cheff

We now present our method Cheff to iteratively generate high-resolution
images. Our multi-stage synthesis approach can be decomposed into three
cascading phases: (i) Modeling a diffusion process in latent space, (ii)
translating the latent variables into image space with a decoder, (iii)
refinement and upscaling using a super-resolution diffusion process. Phase (i)
and (ii) together define an LDM. Figure 2 summarizes this multi-stage approach
again.

Every model can be trained in an encapsulated fashion as described in the
subsequent subsections. In the following, we denote $\bm{x}_{\text{\tiny HR}}$
as a high-resolution image and $\bm{x}_{\text{\tiny LR}}$ as its lower-
resolution variant (in our model stack these are 1024 and 256 pixels,
respectively). We start by explaining the second (reconstruction) part of our
model first.

Figure 2: Flow of our cascaded synthesis pipeline. A first diffusion model
generates a latent sample $\bm{z}\in\mathcal{Z}$ from
$\bm{z}^{T}\sim\mathcal{N}(\bm{0},\bm{I})$ through its reverse process.
Optionally, an embedded conditioning $\tau_{\phi}(\bm{y})$ can be added, where
$\bm{y}$ in our application could, e.g., be pathology labels, radiological
reports, or radiologist’s annotations. The decoded image $\bm{x}_{\text{\tiny
LR}}=D(\bm{z})$ is subsequently refined by a second diffusion model to a high-
resolution image $\bm{x}_{\text{\tiny HR}}$. Foundational models are marked in
red, whereas blue components are task-specific.

##### Autoencoder training

It has been shown that applying diffusion in a semantic latent space instead
of a high-dimensional data space leads to a notable reduction in computational
costs due to reduced input size with a neglectable loss in synthesis quality
[25]. We construct a latent space by training an autoencoder with an encoder
$E$ and decoder $D$. The autoencoder’s task is to reconstruct
$\bm{x}_{\text{\tiny LR}}$, where the reconstruction is
$D(E(\bm{x}_{\text{\tiny LR}}))$ and $\bm{z}=E(\bm{x}_{\text{\tiny
LR}})\in\mathcal{Z}$ represents a latent sample. We follow [25, 5] for the
autoencoder training and use a pixel-wise reconstruction loss next to a
perceptual loss [37] and an adversarial objective.

##### Semantic Generation

The actual synthesis diffusion model is trained by first obtaining
$\bm{z}=E(\bm{x}_{\text{\tiny LR}})$ and modifying the optimization objective
for the semantic generation as

$\mathbb{E}_{t\sim
U(1,T),\bm{z}_{0}\sim\mathcal{Z},\bm{\epsilon}_{t}\sim\mathcal{N}(\bm{0},\bm{1})}\left[\|\bm{\epsilon}_{t}-\bm{\epsilon}_{\theta}(\bm{z}_{t},\tau_{\phi}(\bm{y}),t)\|^{2}_{2}\right]\,\text{.}$
(2)

where $\bm{\epsilon}_{\theta}$ again is a time-conditional U-net. The
diffusion process in (2) is the same as described before, but operates only on
samples in the latent space. Additionally, we allow the conditioning on some
modality $\bm{y}$ to guide the synthesis process. $\bm{y}$ can represent
information of various kinds, e.g., radiological reports, class conditioning,
or annotations. This information is fed into the model using a conditioning
embedding, which is obtained by $\tau_{\phi}(\bm{y})$, where $\tau_{\phi}$ is
a neural network tasked with creating an embedding for $\bm{y}$.

##### Super-resolution

After synthesizing $\bm{z}$ and reconstructing $\bm{x}_{\text{\tiny LR}}$ from
it, we apply an additional iterative refinement procedure to not only
counteract blurriness, which is induced by the autoencoder but also to advance
to realistic resolution domains in clinical practice. Inspired by SR3 [27], we
condition a diffusion model with $\bm{x}_{\text{\tiny LR}}$ to infer an
optimal high-resolution output $\bm{x}_{\text{\tiny HR}}$. Thus, the training
objective is:

$\mathbb{E}_{t\sim U(1,T),\bm{x}_{\text{\tiny HR}}\sim\mathcal{X}_{\text{\tiny
HR}},\bm{\epsilon}_{t}\sim\mathcal{N}(\bm{0},\bm{1})}\left[\|\bm{\epsilon}_{t}-\bm{\epsilon}_{\psi}(\bm{x}_{\text{\tiny
HR},t},\bm{x}_{\text{\tiny LR}},t)\|^{2}_{2}\right]\,\text{,}$ (3)

where $\bm{\epsilon}_{\psi}$ is a denoising time-conditional U-net with
conditioning on $\bm{x}_{\text{\tiny LR}}$.

##  4 MaCheX: Massive Chest X-ray Dataset

We present a large-scale, open-source, diverse collection of chest X-ray
images using a common interface for a major selection of open datasets and
unify them as Massive Chest X-Ray Dataset(MaCheX). With MaCheX we provide the
largest available openly accessible composition of chest X-ray data and hope
to support fair and unbiased future research in the area.The multi-centric
setup of the MaCheX collection encourages diversity with sources from across
the world and aims to be less prone to local biases.

Table 1: Overview of the components of MaCheX.

|  | No. samples |  |
---|---|---|---|---
Dataset | No. patients | Train | Test | $\in$ MaCheX | % | Origin
ChestX-ray14 [34] | 23,152 | 86,523 | 25,595 | 86,523 | 13,28% | Bethesda, MD, USA
CheXpert [11] | 65,240 | 223,414 | 235 | 191,027 | 29,32% | Palo Alto, CA, USA
MIMIC-CXR [12] | 65,379 | 377,110 | - | 243,334 | 37,35% | Boston, MA, USA
PadChest [2] | 67,625 | 160,868 | - | 96,278 | 14,78% | Alicante, Spain
BRAX [24] | 18,529 | 40,967 | - | 19,309 | 2,96% | São Paulo, Brazil
VinDr-CXR [16] | - | 15,000 | 3,000 | 15,000 | 2,30% | Hanoi, Vietnam

Fostering a global data collection and increasing data fidelity is an
important step toward offering a generalized solution without gender or racial
prejudice. Our multi-centric collection includes various datasets described in
detail in the Supplementary Material and summarized in Table 1.

In the current version of MaCheX, only frontal AP/PA scans are considered. By
inclusion of the lateral position, another 250,000 chest X-rays could be added
to the collection, totaling nearly a million samples. The present work,
however, focuses on the analysis and synthesis of a frontal viewing position.
All scans are rescaled so that the shortest edge meets a 1024px resolution and
are then center-cropped to 1024 $\times$ 1024px. We do not apply histogram
equalization or other standardization techniques. By combining the designated
train subsets of the respective datasets, the full MaCheX dataset amounts to
651,471 samples. This includes over 440,000 labels, over 220,000 free-text
radiological reports, and 15,000 coordinate bounding boxes for radiologist
annotations.

We open-source our implementation of the pre-processing setup and provide an
easy-to-use interface to access MaCheX in a deep learning setting with
PyTorch. The structure of MaCheX is clear and simple, allowing to
straightforwardly adapt the code to different frameworks and use cases.

##  5 Experiments

Our model stack effectively contains three models: The semantic diffusion
model (SDM), the autoencoder (AE), and the super-resolution diffusion model
(SR). AE and SR form a foundational basis that is trained on full MaCheX with
a separate test set of size 25,000. A full description of our training routine
and hyperparameter setup can be found in the supplementary material. For AE we
use a KL-regularized continuous variational AE (VAE) with a downsampling
factor 4 as in [25]. In SR, we found that conditioning on bicubic downsampled
versions of $\bm{x}_{\text{\tiny HR}}$ as $\bm{x}_{\text{\tiny LR}}$ as in
[27] does not perfectly reflect the structure of recovered latent samples
$D(\bm{z})$ and results in artifacts and slight blurriness. To align SR with
the synthesis pipeline, we fine-tune SR on $D(\bm{z})$.

SDM in the lower-dimensional (latent sample) domain is trained with a task-
specific data subset of MaCheX. To guide the synthesis process for chosen
conditionings, we induce the respective conditioning embedding
$\tau_{\phi}(\bm{y})$ into the model over cross-attention [33] in the
attention layers of the time-conditional U-net.

##### Reconstruction Quality

Using our AE and SR, a high-resolution image can be converted to the latent
space and successfully reconstructed (cf. Figure 3). Despite a compression
factor of $16$, even small details are recovered in fine-grained quality and
limited blurriness, some structures even gaining sharpness. Reconstruction of
text annotations and small numbers, however, still proves to be difficult.
Table 2 additionally provides a quantitative assessment, i.e., the efficacy of
the various transitions in Cheff as well as the effect of fine-tuning SR, by
analyzing the mean squared error (MSE), the structural similarity index
measure (SSIM), and the peak signal-to-noise ratio (PSNR) on the test data
set. The fine-tuned SR performs better than the base SR in almost every
aspect, even in naive upscaling, while all transitions show high-quality
reconstructions in general.

|  |  |
---|---|---|---
Figure 3: Original images (the respective left image) compared to their reconstructions (the respective right image) from the latent space $\mathcal{Z}$.  Table 2: Test set performance for different reconstruction workflows. \raisebox{-0.9pt}{1}⃝ symbolizes the encoding of $\bm{x}_{\text{\tiny HR}}$ and retrieval from $\bm{z}$ in the full Cheff pipeline, whereas \raisebox{-0.9pt}{2}⃝ and \raisebox{-0.9pt}{3}⃝ analyze the reconstruction capacity of AE and SR separately.  Reconstruction workflow |  MSE $\bm{\downarrow}$ |  PSNR $\bm{\uparrow}$ |  SSIM $\bm{\uparrow}$
---|---|---|---
\raisebox{-0.9pt}{1}⃝ | $\bm{x}_{\text{\tiny HR}}\xrightarrow[]{\text{bicubic}}\bm{x}_{\text{\tiny LR}}\xrightarrow[]{E}\bm{z}\xrightarrow[]{D}\hat{\bm{x}}_{\text{\tiny LR}}\xrightarrow[]{\text{{\color[rgb]{0,0.57421875,0.57421875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.57421875,0.57421875}SR}}_{\text{{\color[rgb]{0,0.57421875,0.57421875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.57421875,0.57421875}base}}}}\hat{\bm{x}}_{\text{\tiny HR}}$ | 0.0039 | 24.05 | 0.9512
$\bm{x}_{\text{\tiny HR}}\xrightarrow[]{\text{bicubic}}\bm{x}_{\text{\tiny LR}}\xrightarrow[]{E}\bm{z}\xrightarrow[]{D}\hat{\bm{x}}_{\text{\tiny LR}}\xrightarrow[]{\text{{\color[rgb]{0.57421875,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.57421875,0,0}SR}}_{\text{{\color[rgb]{0.57421875,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.57421875,0,0}fine}}}\,\,}\hat{\bm{x}}_{\text{\tiny HR}}$ | 0.0026 | 25.77 | 0.9510
\raisebox{-0.9pt}{2}⃝ |  $\bm{x}_{\text{\tiny HR}}\xrightarrow[]{\text{bicubic}}$ $\bm{x}_{\text{\tiny LR}}\xrightarrow[]{E}\bm{z}\xrightarrow[]{D}\hat{\bm{x}}_{\text{\tiny LR}}$ | 0.0005 | 36.80 | 0.9926
\raisebox{-0.9pt}{3}⃝ | $\bm{x}_{\text{\tiny HR}}\xrightarrow[]{\text{bicubic}}\bm{x}_{\text{\tiny LR}}\xrightarrow[\hskip 79.66771pt]{\text{{\color[rgb]{0,0.57421875,0.57421875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.57421875,0.57421875}SR}}_{\text{{\color[rgb]{0,0.57421875,0.57421875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.57421875,0.57421875}base}}}}\hat{\bm{x}}_{\text{\tiny HR}}$ | 0.0035 | 24.59 | 0.9540
$\bm{x}_{\text{\tiny HR}}\xrightarrow[]{\text{bicubic}}\bm{x}_{\text{\tiny LR}}\xrightarrow[\hskip 79.66771pt]{\text{{\color[rgb]{0.57421875,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.57421875,0,0}SR}}_{\text{{\color[rgb]{0.57421875,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.57421875,0,0}fine}}}\,\,}\hat{\bm{x}}_{\text{\tiny HR}}$ | 0.0025 | 26.09 | 0.9559

##### Unconditional Synthesis

Scaling up the data and model stack leads to a realistic synthesis with high
fidelity and realism (Figure 4(e)) in a 1-megapixel resolution setting
including a variety of medical devices, e.g., chest tubes, pacemakers, ECG
leads, etc. This is a major improvement over previously proposed approaches
for chest X-ray synthesis (Figure 4). The PGAN of [28] misses the capacity of
medical devices (cf. [35]). While synthetic samples of [1] by prompting DALL·E
2 are comparatively realistic for a non-medical zero-shot setting, they
involve color jittering. In contrast, Stable Diffusion is not able to generate
clinically realistic chest X-rays, clearly showing its artistic style even
when fine-tuned [3]. LDM training of [20] involves only a 256-pixel resolution
of a single data source.

In comparison to our latent diffusion-based pipeline, we investigate a fully
cascaded stack of diffusion models in the style of Imagen [26] and DALL·E 2
[22] with U-net configurations from [27] (see Figure 4(d)). The cascade
involves three models in an upscaling chain: 64px $\rightarrow$ 256px
$\rightarrow$ 1024px, where the first model synthesizes samples from noise.
This uncovers a potential downside of utilizing models with a progressive
growing backend (same as in [28]). While the samples maintain high quality,
there is also an abundance of medical devices and minor foreign objects. This
fact also reflects in the Fréchet Inception Distance (FID $\downarrow$) and
Kernel Inception Distance (KID $\downarrow$), which is 46.54 and 0.0530 for
the full cascade but reaches 11.58 and 0.0099, respectively, using Cheff. We
hypothesize that details of this granularity are lost when synthesizing on the
low-resolution levels of $\leq$ 64 pixels. The subsequent upsampling models
cannot recover these already disregarded elements. Our approach works on
semantic features instead of low-resolution images and thus is able to
compress necessary information in a meaningful way, successfully circumventing
this issue (Figure 4(e)).

|
---|---
(a) PGAN [28]

|
---|---
(b) Finetuned Stable Diff. [3]

|
---|---
(c) Zero-shot DALL·E 2 [1]

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
(d) Triple cascaded diffusion model.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
(e) Cheff (Ours).

Figure 4: Synthesized chest X-rays using different methods.

##### In- and Outpainting

The iterative nature of diffusion models allows for simple adaption of
inpainting tasks, where the model is used to fill a designated marked area.
This method can be utilized to, e.g., remove occluding and distracting
elements like medical devices from a thoracic scan (Figure 5). While Cheff has
the ability to synthesize removed parts such as devices, object removal works
well since the rest of the image is provided and acts as a prior. Thus Cheff
fills the removed parts with the most likely content – a clear lung – and not
the original devices. In Cheff or LDMs in general, this kind of masking is
possible as the commonly used autoencoder implementation produces a spatially-
aware convolutional embedding, which, despite living in a latent space,
maintains structural information of the input data (Figure 6). In another
formulation, masking as outpainting serves to interpolate the remaining parts
of an image. As seen in Figure 7, a variety of chest X-rays can be synthesized
sharing the same initial provided area. Notably, Cheff does not collapse to
one solution but is able to explore various in-distribution options.

|  |  |
---|---|---|---
Figure 5: Removal of support devices via image inpainting.

|
---|---
|
|

Figure 6: Comparison of original image (left) with latent embedding (right) shows spatial coherency. |  |  |
---|---|---|---
|  |  |

Figure 7: Outpainting variations of the highlighted area in the upper left
image.

##### Radiological Report-to-Image Synthesis

MIMIC-CXR provides a range of radiological text reports for every study (over
220,000), which can be utilized as conditioning $\bm{y}$. The sections
Findings and Impressions are extracted from the raw reports and concatenated
before applying a tokenizer. Following [25], $\tau_{\phi}$ is a trainable
BERT-style encoder-only transformer. Examples in Figure 8 show that the model
has learned concepts of various pathologies and allows for a customized
synthesis of a patient’s condition. Conditioning on both Findings and
Impressions allows controlling not only pathologies but also the creation of
external devices like pacemakers. Furthermore, the freedom of text inputs
allows for specifying the localization of the targeted disease or item.

|  |  |  |
---|---|---|---|---
(a) “Large pleural effusion is in the right lower lung. A pacemaker is in the
left upper chest.”

(b) “Acute
cardiomegaly.”

(c) “Prominent
left-sided atelectasis.”

(d) “Mass filling is in the upper zone of the right lung.”

(e) “Pneumonia is in the right lung.”

Figure 8: Prompting a text-conditioned Cheff via radiological findings.

##  6 Conclusion

In this paper, we advance the state-of-the-art in Chest X-ray synthesis by
proposing a multi-stage foundational cascaded latent diffusion model called
Cheff. A success factor for the quality of Cheff is MaCheX \- our large-scale
multi-centric Chest X-ray collection from numerous, publicly available
datasets with high diversity in phenotypes, medical conditions, diseases, and
medical devices.

Approaching the terrain of indistinguishable synthetic patient samples, Cheff
requires an increased awareness of potential harms and responsible
utilization. Further, despite being able to generate realistic-looking
1-megapixel radiographs, modern X-ray scanners produce images with an up to
7-megapixel resolution and 14-bit depth, which is a requirement for
radiological reading on dedicated monitors and is still out of reach for
current chest X-ray synthesis. When developing downstream applications with
Cheff for a clinical context, this should be kept in mind and potential biases
should be addressed, e.g., through rigorous evaluation on real-world data.

By proposing Cheff, we offer a high-capacity generator for chest radiographs
that forms the basis for a variety of use cases, e.g., image inpainting for
removal of distracting medical devices or aiding existing efforts to increase
reliability in classifiers by generating underrepresented classes. Our method
does not only offer traditional synthesis but also enters the exciting area of
radiological report-to-chest-X-ray generation, which allows fine-grained
control over the diffusion process via text prompts.

## Acknowledgments

The authors gratefully acknowledge LMU Klinikum for providing computing
resources on their Clinical Open Research Engine (CORE). This work has been
partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) as part of BERD@NFDI - grant number 460037581.

## References

  * [1] Ali, H., Murad, S., Shah, Z.: Spot the fake lungs: Generating synthetic medical images using neural diffusion models. arXiv:2211.00902 [cs, eess] (2022)
  * [2] Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: PadChest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis 66 (2020)
  * [3] Chambon, P., Bluethgen, C., Langlotz, C.P., Chaudhari, A.: Adapting pretrained vision-language foundational models to medical imaging domains. arxiv:2210.04133 [cs] (2022)
  * [4] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. In: Advances in Neural Information Processing Systems. vol. 34 (2021)
  * [5] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021)
  * [6] Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K., et al.: U-net: deep learning for cell counting, detection, and morphometry. Nature methods 16 (2019)
  * [7] Glocker, B., Jones, C., Bernhardt, M., Winzeck, S.: Risk of bias in chest x-ray foundation models. arXiv:2209.02965 (2022)
  * [8] Han, L., Lyu, Y., Peng, C., Zhou, S.K.: Gan-based disentanglement learning for chest x-ray rib suppression. Medical Image Analysis 77 (2022)
  * [9] Han, T., Nebelung, S., Haarburger, C., Horst, N., Reinartz, S., Merhof, D., Kiessling, F., Schulz, V., Truhn, D.: Breaking medical data sharing boundaries by using synthesized radiographs. Science advances 6 (2020)
  * [10] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems. vol. 33 (2020)
  * [11] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33 (2019)
  * [12] Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 6 (2019)
  * [13] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive Growing of GANs for Improved Quality, Stability, and Variation. In: 6th International Conference on Learning Representations, ICLR (2018)
  * [14] Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
  * [15] Liu, G., Hsu, T.M.H., McDermott, M., Boag, W., Weng, W.H., Szolovits, P., Ghassemi, M.: Clinically accurate chest x-ray report generation. In: Machine Learning for Healthcare Conference (2019)
  * [16] Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., et al.: VinDr-CXR: An open dataset of chest x-rays with radiologist’s annotations. arXiv:2012.15029 [eess] (2022)
  * [17] Nichol, A., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International Conference on Machine Learning (2021)
  * [18] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In: International Conference on Machine Learning (2022)
  * [19] Packhäuser, K., Gündel, S., Münster, N., Syben, C., Christlein, V., Maier, A.: Deep learning-based patient re-identification is able to exploit the biometric nature of medical chest x-ray data. Scientific reports 12 (2022)
  * [20] Packhäuser, K., Folle, L., Thamm, F., Maier, A.: Generation of anonymous chest radiographs using latent diffusion models for training thoracic abnormality classification systems. arxiv:2211.01323 [cs, eess] (2022)
  * [21] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al.: CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. arXiv:1711.05225 [cs, stat] (2017)
  * [22] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with CLIP latents. arxiv:2204.06125 [cs] (2022)
  * [23] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning (2021)
  * [24] Reis, E.P., Paiva, J., Bueno da Silva, M.C., Sousa Ribeiro, G.A., Fornasiero Paiva, V., et al.: BRAX, a brazilian labeled chest x-ray dataset. Scientific Data 9 (2022)
  * [25] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
  * [26] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., et al.: Photorealistic text-to-image diffusion models with deep language understanding. arXiv:2205.11487 (2022)
  * [27] Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., Norouzi, M.: Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [28] Segal, B., Rubin, D.M., Rubin, G., Pantanowitz, A.: Evaluating the Clinical Realism of Synthetic Chest X-Rays Generated Using Progressively Growing GANs. SN Computer Science 2(4) (2021)
  * [29] Sellergren, A.B., Chen, C., Nabulsi, Z., Li, Y., Maschinot, A., Sarna, A., Huang, J., Lau, C., Kalidindi, S.R., Etemadi, M., et al.: Simplified transfer learning for chest radiography models using less data. Radiology 305(2) (2022)
  * [30] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International Conference on Machine Learning (2015)
  * [31] Sundaram, S., Hulkund, N.: Gan-based data augmentation for chest x-ray classification. arXiv:2107.02970 (2021)
  * [32] Tang, Y., Tang, Y., Zhu, Y., Xiao, J., Summers, R.M.: A disentangled generative model for disease decomposition in chest x-rays via normal image synthesis. Medical Image Analysis 67 (2021)
  * [33] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [34] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2017)
  * [35] Weber, T., Ingrisch, M., Bischl, B., Rügamer, D.: Implicit embeddings via GAN inversion for high resolution chest radiographs. In: Medical Image Computing and Computer Assisted Intervention (MICCAI), Medical Applications with Disentanglement (2022)
  * [36] Wolleb, J., Bieder, F., Sandkühler, R., Cattin, P.C.: Diffusion models for medical anomaly detection. In: Medical Image Computing and Computer Assisted Intervention (MICCAI). vol. 13438 (2022)
  * [37] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2018)

##  7 Supplementary Material

###  7.1 Training and sampling with diffusion models.

The following two algorithms show the training and sampling strategy for a
diffusion model adapted from [10].

Required: Denoising network $\bm{\epsilon}_{\theta}$ with parameters $\theta$,
forward variances $\beta_{i}$ with $i\in\\{1,...,T\\}$

repeat

// Sample uncorrupted image from data distribution

$\bm{x}_{0}\sim q(\bm{x}_{o})$

// Sample uniformly distributed timestep

$t\sim U({1,...,T})$

// Sample normal distributed noise

$\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})$

// Compute alpha products

$\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$

// Obtain corrupted $\bm{x}_{0}$ at timestep $t$

$\bm{x}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$

// Update weights with loss gradients

$\theta\leftarrow\theta-\nabla_{\theta}\|\bm{\epsilon}_{t}-\bm{\epsilon}_{\theta}(\bm{x}_{t},t)\|^{2}_{2}$

until  _converged_ ;

Algorithm 1 Training of a vanilla diffusion model

Required: Denoising network $\bm{\epsilon}_{\theta}$, forward variances
$\beta_{i}$ with $i\in\\{1,...,T\\}$, reverse variances $\sigma_{i}$ with
$i\in\\{1,...,T\\}$

// Sample start as isotropic noise

$\bm{x}_{T}\sim\mathcal{N}(\bm{0},\bm{I})$

for  _$t\leftarrow T$ to $1$_ do

// Compute alpha products

$\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$

// Do not set any noising for the last step

if  _$t >1$_ then $\bm{\epsilon}\sim\mathcal{N}(\bm{0},\bm{I})$ else
$\epsilon=\bm{0}$ ;

// Compute state of $\bm{x}$ on previous timestep

$\bm{x}_{t-1}\leftarrow\frac{1}{\sqrt{1-\beta_{t}}}\left(\bm{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}(\bm{x}_{t},t)\right)+\sigma_{t}\epsilon$

end for

return $\bm{x}_{0}$

Algorithm 2 Sampling from a vanilla diffusion model

Here, $\sigma_{t}$ is the variance of the reverse process at timestep $t$.
While this parameter is learnable in general, [10] presents two static
versions of $\sigma_{t}$: Setting it directly to $\beta_{t}$ or
$\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}-}\cdot\beta_{t}$. We chose the
latter option in our experiments. We refer to [10] for extended derivations
and proofs for the above algorithms and a deeper understanding of diffusion
modeling.

###  7.2 MaCheX Collection Details

In the following, more details about the components of MaCheX are given.

  * •

ChestX-ray14 [34] contains 108,948 frontal-view X-ray images from the NIH
clinical center in Bethesda between 1992 to 2015. Labels for over 14
pathologies are sourced via NLP methods from radiological reports.

  * •

CheXpert [11] consists of 224,316 frontal- and lateral-view scans from the
Stanford Hospital in Palo Alto. The studies were performed between 2002 and
2017. While text reports are also the basis for the provided labels, the
extraction process offers the option of marking a pathology as not mentioned
or uncertain. The so-called CheXpert labeler is also open-sourced, but the
text reports are not.

  * •

MIMIC-CXR [12] provides 377,110 images from the Beth Israel Deaconess Medical
Center Emergency Department recorded between 2011 and 2016. In addition to
labels provided in the same style as in CheXpert, the dataset also releases
full-text radiological reports.

  * •

PadChest [2] contains studies with 160,868 images from the San Juan hospital
in Alicante from 2009 to 2017. The dataset promotes radiological texts in
Spanish and inferred labels from the reports.

  * •

BRAX [24] is a Brazilian dataset from the Hospital Israelita Albert Einstein,
housing 40,967 chest x-rays with supplied labels in the style of CheXpert.

  * •

VinDr-CXR [16] consists of 18,000 thoracic scans from 2018 to 2020 recorded at
Hospital 108 and Hanoi Medical University Hospital in Vietnam. It provides
hand-labeled annotations with coordinate bounding boxes for various
pathologies.

###  7.3 Training details of Cheff

All model training was conducted on an NVIDIA DGX system with 8 A100 @ 40GB
VRAM. To reduce computational complexity, we do not conduct a hyperparameter
search but rely on the recommended and subsequently elaborated settings
proposed in [25, 27]. To accelerate the sampling procedure in the diffusion
models, we utilize DDIM sampling with 150 steps. All models were trained with
the Adam optimizer.

####  7.3.1 Autoencoder (AE)

For AE we use the KL-regularized continuous variational AE (VAE) with a
downsampling factor 4 as in [25] and train it 3 epochs with a batch size of 64
on MaCheX. We did not see an improvement in validation errors in longer
training sessions. The convolutional embedding has a dimension of $3\times
64\times 64=12288$. The architecture consists of 3 blocks with 128, 256, and
512 channels respectively. Each block contains two residual layers. No self-
attention or dropout was applied. The decoder is the mirrored version of the
encoder. To limit the normal prior in the latent space and focus on
reconstruction quality, a KL-divergence weighting of $1\text{\times}{10}^{-6}$
was set. We stick to the default learning rate of $4.5\text{\times}{10}^{-6}$.
We experimented with larger downsampling factors as well as discretized
embeddings and found that factor 4 offers the highest, most fine-grained
reconstruction quality.

####  7.3.2 Semantic diffusion model (SDM)

##### Unconditional SDM

The model is trained on the full MaCheX collection for 500,000 steps with a
batch size of 256. The time-conditional U-net has 4 downsampling blocks with a
base filter number of 224 and the multipliers $(1,2,4,4)$ respectively. Each
downsampling block has 2 residual layers. Attention is applied to the spatial
resolution of 8, 16, and 32. The learning rate is set to
$5.0\text{\times}{10}^{-05}$. The timestep is encoded over sinusoidal position
embeddings.

We noticed that the default linear variance schedule with a maximum variance
of 0.0195 does not induce enough noise into the process to guarantee an
approximate isotropic Gaussian in $\bm{x}_{T}$, which proves to be extremely
destructive for the synthesis process. The number of used timesteps is 1000.
Raising the maximum variance to 0.0295 eliminates this issue.

##### Report-conditioned SDM

The structure and training procedure of the base U-net stays the same, but we
train for a longer period of time, namely for 750,000 steps. We apply the
BERT-tokenizer to preprocess the text sequences and limit the length of the
tokenized input to a maximum of 150. The conditioning $\bm{y}$ is fed to the
model over cross-attention, where $\tau_{\phi}$ is a transformer with a depth
of 32 and an embedding dimension of 1280. As it is described in [25, 33], an
embedding $\tau_{\phi}(\bm{y})\in\mathbb{R}^{d_{\tau}}$ is induced to the
model over cross-attention. Thus, an attention layer with
$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V$
has keys $K$, queries $Q$ and values $V$ defined as

$\displaystyle Q$ $\displaystyle=W_{Q}\cdot\varepsilon$ $\displaystyle K$
$\displaystyle=W_{K}\cdot\tau_{\phi}(\bm{y})$ $\displaystyle V$
$\displaystyle=W_{V}\cdot\tau_{\phi}(\bm{y})\,\text{.}$

$W_{Q}\in\mathbb{R}^{d_{q}\times
d_{\varepsilon}},W_{K}\in\mathbb{R}^{d_{k}\times
d_{\tau}},W_{V}\in\mathbb{R}^{d_{v}\times d_{\tau}}$. $\varepsilon$ is an
intermediate vectorized representation in $\bm{\epsilon}_{\theta}$, i.e., the
input to the respective attention layer in the time-conditioned denoising
U-net. In other words, cross-attention works by providing keys $K$ and values
$V$ with information over the targeted conditioning and querying (via $Q$)
with the concept that is being processed, i.e., the image to be denoised.

####  7.3.3 Super-resolution (SR)

SR training amounts to 1 million steps on MaCheX with a batch size of 32. The
conditioning is done by simply concatenating the low-resolution image to the
target input on the channel dimension. To obtain $\bm{x}_{\text{\tiny LR}}$
for training, we resize $\bm{x}_{\text{\tiny HR}}$ to the low resolution with
bicubic downsampling. To align SR with the synthesis pipeline, we fine-tune SR
for another 500,000 steps on $D(\bm{z})$. The learning rate for normal
training was $5.0\text{\times}{10}^{-05}$ and $2.0\text{\times}{10}^{-05}$
during finetuning. A cosine variance schedule with 2000 timesteps is used
instead of the piece-wise distribution in [27].

The model itself is a U-net with sinusoidal position embeddings for the
timestep. It has a base number of filters of 16 and 8 downsampling blocks
following the filter multiplication schedule of $(1,2,4,8,16,32,32,32)$. Each
block has 2 residual layers. Self-attention is applied in the lowest 5
downsampling levels.

###  7.4 Sparse latent space

While the basic formulation of a VAE often promises the existence of a
generative model in itself, we observe that this is not the case in the AE
component here. As the AE is trained with an extremely low weighting of KL-
divergence, the influence of the Gaussian prior is minimal in the structure of
the posterior latent space $\mathcal{Z}$. As a result, $\mathcal{Z}$ is not
smooth, and sampling $\bm{z}\sim\mathcal{N}(\bm{0},\sigma\bm{I})$ does not
yield any valid samples of the dataspace even for small values of $\sigma$.
This can also be easily explained, as the convolutional AE fosters a spatial
embedding, and isotropic sampled noise does not resemble anything lung shape-
related. Yet, simple linear interpolation between two valid samples in latent
space still seems rather smooth, but when investigated in detail, the
transition is more a blending of the two images instead of merging on a
conceptual level.

Figure 9: Reconstructions for samples $\bm{z}\sim\mathcal{N}(\bm{0},\bm{I})$
showing missing generative capacity in a non-smooth latent space for the AE
without a functioning prior like a diffusion model.
---
Figure 10: Linear interpolation in the latent space of the AE.

###  7.5 Experiments with VQ-VAE

We also experimented with a vector-quantized version of AE, but found a dip in
reconstruction quality and especially in annotations, and favored the classic
VAE formulation.

(a) Original images

(b) Reconstructed images

Figure 11: Comparison of original and reconstructed chest x-ray samples using
VQ-VAE.

###  7.6 Effects of insufficient noising

When training with the full default settings of LDM, we were only able to
produce corrupted and solarized synthetic samples (Figure 12). As it turns
out, this was due to a too-small maximal $\beta$ value of $0.0195$ in the
linear variance schedule.

Figure 12: Insufficient noise in the forward process leads to corrupted
synthesis.

By investigating the forward diffusion process, we observed that the maximal
corrupted latent space sample $\bm{z}_{T}$ still encodes some structural
information of its original image $\bm{x}_{0}$ (see Figure 13(a) right column)
when decoding $\bm{z}_{T}$ with $D$. During synthesis, where
$\bm{z}_{T}\sim\mathcal{N}(\bm{0},\bm{I})$ is chosen as a starting point, the
model does not know what to infer from this unseen unconditional information,
which results in low-quality samples. Increasing the maximal $\beta$ to
$0.0295$ resolves this issue.

(a) Low maximal $\beta$

(b) Increased maximal $\beta$

Figure 13: Comparison the forward diffusion process for the default $\beta$
(right) and our increased value (left).

###  7.7 More inpainting examples using Cheff

The below figure displays two more examples of inpainting in chest
radiographs. The first image shows the removal of a chest port, the second
image shows the successful completion of separated edges.

|  |  |
---|---|---|---
Figure 14: More examples of image inpainting with Cheff.

###  7.8 More synthetic samples of Cheff

On the following pages more random synthetic Chest X-rays, which are generated
by Cheff, are showcased.

---
Figure 15: Synthetic Chest X-rays generated by Cheff (I)
---
Figure 16: Synthetic Chest X-rays generated by Cheff (I) Figure 17: Synthetic
Chest X-rays generated by Cheff (III) Figure 18: Synthetic Chest X-rays
generated by Cheff (IV) Figure 19: Synthetic Chest X-rays generated by Cheff
(V)