# Diffusion Models for Medical Image Analysis: A Comprehensive Survey

Amirhossein Kazerouni
School of Electrical Engineering
Iran University of Science and Technology
Tehran, Iran
<EMAIL_ADDRESS>
&Ehsan Khodapanah Aghdam
Department of Electrical Engineering
Shahid Beheshti University
Tehran, Iran
<EMAIL_ADDRESS>
&Moein Heidari
School of Electrical Engineering
Iran University of Science and Technology
Tehran, Iran
<EMAIL_ADDRESS>
&Reza Azad
Institute of Imaging and Computer Vision
RWTH Aachen University
Aachen, Germany
<EMAIL_ADDRESS>Fayyaz
Microsoft
Berlin, Germany
<EMAIL_ADDRESS>
&Ilker Hacihaliloglu
Department of Radiology
Department of Medicine
University of British Columbia
British Columbia, Canada
<EMAIL_ADDRESS>
&Dorit Merhof
Faculty of Informatics and Data Science
University of Regensburg
Regensburg, Germany
<EMAIL_ADDRESS>

###### Abstract

Denoising diffusion models, a class of generative models, have garnered
immense interest lately in various deep-learning problems. A diffusion
probabilistic model defines a forward diffusion stage where the input data is
gradually perturbed over several steps by adding Gaussian noise and then
learns to reverse the diffusion process to retrieve the desired noise-free
data from noisy data samples. Diffusion models are widely appreciated for
their strong mode coverage and quality of the generated samples in spite of
their known computational burdens. Capitalizing on the advances in computer
vision, the field of medical imaging has also observed a growing interest in
diffusion models. With the aim of helping the researcher navigate this
profusion, this survey intends to provide a comprehensive overview of
diffusion models in the discipline of medical image analysis. Specifically, we
start with an introduction to the solid theoretical foundation and fundamental
concepts behind diffusion models and the three generic diffusion modeling
frameworks, namely, diffusion probabilistic models, noise-conditioned score
networks, and stochastic differential equations. Then, we provide a systematic
taxonomy of diffusion models in the medical domain and propose a multi-
perspective categorization based on their application, imaging modality, organ
of interest, and algorithms. To this end, we cover extensive applications of
diffusion models in the medical domain, including segmentation, anomaly
detection, image-to-image translation, 2/3D generation, reconstruction,
denoising, and other medically-related challenges. Furthermore, we emphasize
the practical use case of some selected approaches, and then we discuss the
limitations of the diffusion models in the medical domain and propose several
directions to fulfill the demands of this field. Finally, we gather the
overviewed studies with their available open-source implementations at our
GitHub111https://github.com/amirhossein-kz/Awesome-Diffusion-Models-in-
Medical-Imaging. We aim to update the relevant latest papers within it
regularly.

_K_ eywords Generative models $\cdot$ Diffusion models $\cdot$ Denoising
diffusion models $\cdot$ Noise conditioned score networks $\cdot$ Score-based
models $\cdot$ Medical imaging $\cdot$ Medical applications $\cdot$ Survey

##  1 Introduction

][c]0.46

(a) GAN

][c]0.46

(b) Energy-based Models

][c]0.46

(c) VAE

][c]0.46

(d) Flow-based models

][c]

(e) Diffusion models

Figure 1: In this figure, We display the revolution of generative models and
the insights behind them. (1(a)) General Adversarial Network (GAN) [1] is an
end-to-end pipeline that trains the generator in an adversarial manner to
generate samples that the discriminator is capable of distinguishing from the
real data sample. (1(b)) Energy-based Model (EBM) [2], also known as non-
normalized probabilistic models, trains in the same way as GANs with two major
modifications. First, the discriminator learns a proper energy-based function
that maps the data sample to a distribution space. Second, the generator
utilizes a prior input to enhance the sample generation performance. (1(c))
Variational AutoEncoder (VAE) [3] is a standalone network that follows a
projection from a data sample to a low-dimensional latent space by the encoder
and generates by sampling from it via a decoder path. (1(d)) Normalizing flow
(NF) [4] utilizes an invertible flow function to transform input to latent
space and generate samples with the inverse flow function. (1(e)) Diffusion
Models intermingle the noise with the input in successive steps until it
becomes a noise distribution before applying a reverse process to neutralize
the noise addition in each step in the sampling procedure.

Generative modeling using neural networks has been a formidable force in the
past decade of deep learning. Since their emergence, generative models have
made a tremendous impact in various domains ranging from images [5, 6], audio
[7, 8], to text, [9] and point clouds [10]. From a probabilistic modeling
viewpoint, the key defining characteristic of a generative model is that it is
trained in such a way so that its samples $\tilde{\boldsymbol{x}}\sim
p_{\theta}(\tilde{\boldsymbol{x}})$ come from the same distribution as the
training data distribution, $\boldsymbol{x}\sim p_{d}(\boldsymbol{x})$. The
pioneering energy-based models achieve this by defining an unnormalized
probability density over a state space; however, these methods require Markov
Chain Monte Carlo (MCMC) sampling during both training and inference, which is
a slow iterative process [11]. Following the unprecedented surge of available
datasets, as well as advances in general deep learning architectures, there
has been a revolutionary paradigm shift in generative modeling. Specifically,
the three mainstream generative frameworks include, namely, Generative
adversarial networks (GANs) [1], variational autoencoders (VAEs) [12, 3], and
normalizing flows [13] (see Figure 1). Generative models typically entail key
requirements to be adopted in real-world problems. These requirements include
(i) high-quality sampling, (ii) mode coverage and sample diversity, and (iii)
fast execution time and computationally inexpensive sampling [14] (see Figure
2).

Figure 2: Generative learning trial [14]. Despite the ability of GANs to
quickly generate high-fidelity samples, their mode coverage is limited. In
addition, VAEs and normalizing flows have been revealed to have a great deal
of diversity; however, they generally have poor sampling quality. Diffusion
models have emerged to compensate for the deficiency of VAEs and GANs by
showing adequate mode coverage and high-quality sampling. Nevertheless, due to
their iterative nature, which causes a slow sampling process, they are
practically expensive and require more improvement.

Generative models often make accommodations between these criteria.
Specifically, GANs are capable of generating high-quality samples rapidly, but
they have poor mode coverage and are prone to lack sampling diversity.
Conversely, VAEs and normalizing flows suffer from the intrinsic property of
low sample quality despite being witnessed in covering data modes. GANs
consist of two models: a generator and a critic (discriminator), which compete
with each other while simultaneously making each other stronger. The generator
tries to capture the distribution of true examples, while the discriminator,
which is typically a binary classifier, estimates the probability of a given
sample coming from the real dataset. It works as a critic and is optimized to
recognize the synthetic samples from the real ones. A common concern with GANs
is their training dynamics which have been recognized as being unstable,
resulting in deficiencies such as mode collapse, vanishing gradients, and
convergence [15]. Therefore, an immense interest has also influenced the
research direction of GANs to propose more efficient variants [16, 17].
Variational auto-encoders (VAEs) optimize the log-likelihood of the data by
maximizing the evidence lower bound (ELBO). Despite the remarkable
achievements, the behavior of Variational Autoencoders is still far from
satisfactory due to some theoretical and practical challenges such as
balancing issue [18] and variable collapse phenomenon [19]. A flow-based
generative model is constructed by a sequence of invertible transformations.
Specifically, a normalizing flow transforms a simple distribution into a
complex one by applying a sequence of invertible transformation functions
where one can obtain the desired probability distribution for the final target
variable using a change of variables theorem. Unlike GANs and VAEs, these
models explicitly learn the data distribution; therefore, their loss function
is simply the negative log-likelihood [20]. Despite being feasibly designed,
these generative models have their specific drawbacks. Since the Likelihood-
based method has to construct a normalized probability model, a specific type
of architecture must be used (Autoregressive Model, Flow Model), or in the
case of VAE, an alternative Loss such as ELBO is not calculated directly for
the generated probability distribution. In contrast, the learning process of
GANs is inherently unstable due to the nature of the adversarial loss of the
GAN. Recently, diffusion models [21, 22] have emerged as powerful generative
models, showcasing one of the leading topics in computer vision so that
researchers and practitioners alike may find it challenging to keep pace with
the rate of innovation. Fundamentally, diffusion models work by destroying
training data through the successive addition of Gaussian noise and then
learning to recover the data by reversing this noising process.

To date, diffusion models have been found to be useful in a wide variety of
areas, ranging from generative modeling tasks such as image generation [23],
image super-resolution [24], image inpainting [25] to discriminative tasks
such as image segmentation [26], classification [27], and anomaly detection
[28]. Recently, the medical imaging community has witnessed exponential growth
in the number of diffusion-based techniques (see Figure 4). As shown in Figure
4, a wealth of research is dedicated to the applications of diffusion models
in diverse medical imaging scenarios. Thus, a survey of the existing
literature is both beneficial for the community and quite timely. To this end,
this survey sets out to provide a comprehensive overview of the recent
advances made and provides a holistic overview of this class of models in
medical imaging. A thorough search of the relevant literature revealed that we
are the first to cover the diffusion-based models exploited in the medical
domain. We hope this work will point out new paths, provide a road map for
researchers, and inspire further interest in the vision community to leverage
the potential of diffusion models in the medical domain. Our major
contributions include:

$\bullet$ This is the first survey paper that comprehensively covers
applications of diffusion models in the medical imaging domain. Specifically,
we present a comprehensive overview of all available relevant papers (until
October 2022).

$\bullet$ We devise a multi-perspective categorization of diffusion models in
the medical community, providing a systematical taxonomy of research in
diffusion models and their applications. We divide the existing diffusion
models into three categories: denoising diffusion probabilistic models, noise-
conditioned score networks, and stochastic differential equations. Moreover,
we group the applications of diffusion models into seven categories: anomaly
detection, denoising, reconstruction, segmentation, image-to-image
translation, image generation, and other applications.

$\bullet$ We do not restrict our attention to application and provide a new
taxonomy (see Figure 4) where each paper is broadly classified according to
the proposed algorithm along with the organ concerned and imaging modality,
respectively.

$\bullet$ Finally, we discuss the challenges and open issues and identify the
new trends raising open questions about the future development of diffusion
models in the medical domain in both algorithms and applications.

_Paper Organization._ In Section 2.1, we present a detailed overview of the
concepts and theoretical foundations behind diffusion models, covering three
sub-categories with a similar baseline definition. Sections 3.1, 3.2, 3.3,
3.4, 3.5, 3.6 and 3.7 comprehensively cover the applications of diffusion
models in several medical imaging tasks, as shown in Figure 3, whereas Section
3.8 provides a comparative task-specific overview of different literature
work. We conclude this survey by pinpointing future directions and open
challenges facing diffusion models in the medical imaging domain in Section 4.

][c]0.33

(a)

][c]0.33

(b)

][c]0.33

(c)

Figure 3: The diagram (a) shows the relative proportion of published papers
categorized according to their application and (b) according to their imaging
modalities. (c) indicates the number of diffusion-based research papers
published in the medical domain. The growth rate per year reveals the
importance of diffusion models for future work.

##  2 Taxonomy

Generative approaches have undergone significant advances in medical imaging
over the past few decades. Therefore, there have been numerous survey papers
published on deep generative models for medical imaging [29, 30, 31]. Some of
these papers focus on a specific application only, while others concentrate on
a specific image modality. There is, however, a lack of comprehensive surveys
on the applications of diffusion models in medical imaging. To this end, in
this survey, we devise a multi-perspective vision of diffusion models in which
we discuss existing literature based on their applications in the medical
domain. Nonetheless, we do not restrict our interest to the applications but
describe the underlying working principles, the organ, and the imaging
modality of the proposed method. We further discuss how this additional
information can help researchers attempt to consolidate the literature across
the spectrum. A brief outlook of our paper is depicted in Figure 4.

Figure 4: The proposed taxonomy for diffusion-based medical image analysis
research is built on seven sub-fields: 1) Image-to-Image Translation, 2)
Medical Image Reconstruction, 3) Image Segmentation, 4) Medical Image
Denoising, 5) Image Generation, 6) Anomaly Detection and 7) multi-disciplinary
applications, named Other Applications. For the sake of brevity, we utilize
the prefix numbers in the paper’s name in ascending order and denote the
reference for each study as follows: 1\. [32], 2\. [33], 3\. [34], 4\. [35],
5\. [36], 6\. [37], 7\. [38], 8\. [39], 9\. [40], 10\. [41], 11\. [42], 12\.
[43], 13\. [44], 14\. [45], 15\. [46], 16\. [47], 17\. [48], 18\. [49], 19\.
[50], 20\. [51], 21\. [52], 22\. [53], 23\. [54], 24\. [55], 25\. [28], 26\.
[56], 27\. [57], 28\. [58], 29\. [59].

###  2.1 Algorithm

There are at least three sub-categories of diffusion models that comply with
the baseline definition of diffusion models [60]. First, there are denoising
diffusion probabilistic models (DDPMs) [21, 22], which are a class of latent
variable models inspired by considerations from nonequilibrium thermodynamics.
The second sub-category is represented by noise-conditioned score networks
(NCSNs) [61], which is fundamentally based on estimating the derivative of the
log density function of the perturbed data distribution at different noise
levels. Stochastic differential equations (SDEs) [62] form the third sub-
category, which encapsulates previous approaches and can be viewed as a
generalization over DDPMs and NCSNs. We hereinafter elaborate on the details
of each category.

####  2.1.1 Denoising Diffusion Probabilistic Models (DDPMs)

Forward Process. DDPM defines the forward diffusion process as a Markov Chain
where Gaussian noise is added in successive steps to obtain a set of noisy
samples. Consider $q\left(x_{0}\right)$ as the uncorrupted (original) data
distribution. Given a data sample $x_{0}\sim q\left(x_{0}\right)$, a forward
noising process $p$ which produces latent $x_{1}$ through $x_{T}$ by adding
Gaussian noise at time $t$ is defined as follows:

$q\left(x_{t}\mid
x_{t-1}\right)=\mathcal{N}\left(x_{t};\sqrt{1-\beta_{t}}\cdot
x_{t-1},\beta_{t}\cdot\mathbf{I}\right),\forall t\in\\{1,\ldots,T\\}$ (1)

where $T$ and $\beta_{1},\ldots,\beta_{T}\in[0,1)$ represent the number of
diffusion steps and the variance schedule across diffusion steps respectively.
$\mathbf{I}$ is the identity matrix and $\mathcal{N}(x;\mu,\sigma)$ represents
the normal distribution of mean $\mu$ and covariance $\sigma$. Considering
$\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$, one
can directly sample an arbitrary step of the noised latent conditioned on the
input $x_{0}$ as follows:

$q\left(\mathbf{x}_{t}\mid\mathbf{x}_{0}\right)=N\left(\mathbf{x}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)$
(2)

$\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\overline{\alpha_{l}}}\epsilon$
(3)

Reverse Process. Leveraging the above definitions, we can approximate a
reverse process to get a sample from $q\left(x_{0}\right)$. To this end, we
can parameterize this reverse process by starting at
$p\left(\mathbf{x}_{T}\right)=\mathcal{N}\left(\mathbf{x}_{T};\mathbf{0},\mathbf{I}\right)$
as follows:

$p_{\theta}\left(\mathbf{x}_{0:T}\right)=p\left(\mathbf{x}_{T}\right)\prod_{t=1}^{T}p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}\right)$
(4)
$p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}\right)=\mathcal{N}\left(\mathbf{x}_{t-1};\mu_{\theta}\left(\mathbf{x}_{t},t\right),\Sigma_{\theta}\left(\mathbf{x}_{t},t\right)\right)$
(5)

To train this model such that $p\left(x_{0}\right)$ learns the true data
distribution $q\left(x_{0}\right)$, we can optimize the following variational
bound on negative log-likelihood:

$\displaystyle\mathbb{E}\left[-\log
p_{\theta}\left(\mathbf{x}_{0}\right)\right]$
$\displaystyle\leq\mathbb{B}_{q}\left[-\log\frac{p_{\theta}\left(\mathbf{x}_{0:T}\right)}{q\left(\mathbf{x}_{1:T}\mid\mathbf{x}_{0}\right)}\right]$
(6) $\displaystyle=\mathbb{E}_{q}\left[-\log
p\left(\mathbf{x}_{T}\right)-\sum_{t\geq
1}\log\frac{p_{\theta}\left(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}\right)}{q\left(\mathbf{x}_{t}\mid\mathbf{x}_{t-1}\right)}\right]$
$\displaystyle=-L_{\mathrm{VL}.\mathrm{B}}$

Ho et al. [22] found it better not to directly parameterize
$\mu_{\theta}\left(x_{t},t\right)$ as a neural network, but instead to train a
model $\epsilon_{\theta}\left(x_{t},t\right)$ to predict $\epsilon$. Hence, by
reparameterizing Equation 6, they proposed a simplified objective as follows:

$L_{\text{simple
}}=E_{t,x_{0},\epsilon}\left[\left\|\epsilon-\epsilon_{\theta}\left(x_{t},t\right)\right\|^{2}\right]$
(7)

where the authors draw a connection between the loss in Equation 6 to
generative score networks in Song et al. [61].

####  2.1.2 Noise Conditioned Score Networks (NCSNs)

The score function of some data distribution $p(x)$ is defined as the gradient
of the log density with respect to the input, $\nabla_{x}\log p(x)$. To
estimate this score function, one can train a shared neural network with score
matching. Specifically, the score network $\mathbf{s}_{\boldsymbol{\theta}}$
is a neural network parameterized by $\boldsymbol{\theta}$, which is trained
to approximate the score of $p(x)$ ($s_{\theta}(x)\approx\nabla_{x}\log p(x)$)
by minimizing the following objective:

$\mathbb{E}_{x\sim p(x)}\left\|s_{\theta}(x)-\nabla_{x}\log
p(x)\right\|_{2}^{2}$ (8)

However, due to the computational burden of calculating $\nabla_{x}\log p(x)$,
score matching is not scalable to deep networks and high dimensional data. To
mitigate this problem, the authors of [61] propose to exploit denoising score
matching [63] and sliced score matching [64]. Moreover, Song et al. [61]
highlight major challenges that prevent a naive application of score-based
generative modeling in real data. The key challenge is the fact that the
estimated score functions are inaccurate in low-density regions since data in
the real world tend to concentrate on low-dimensional manifolds embedded in a
high-dimensional space (the manifold hypothesis). The authors demonstrate that
these problems can be addressed by perturbing the data with Gaussian noise at
different scales, as it makes the data distribution more amenable to score-
based generative modeling. They propose to estimate the score corresponding to
all noise levels by training a single noise-conditioned score network (NCSN).
They derive $\nabla_{x}\log\left(p_{\sigma_{t}}(x)\right)$ as
$\nabla_{x_{t}}\log p_{\sigma_{t}}\left(x_{t}\mid
x\right)=-\frac{x_{t}-x}{\sigma_{t}}$ by choosing the noise distribution to be
$p_{\sigma_{t}}\left(x_{t}\mid
x\right)=\mathcal{N}\left(x_{t};x,\sigma_{t}^{2}\cdot\mathbf{I}\right)$ where
$x_{t}$ is a noised version of $x$. Thus, for a given sequence of Gaussian
noise scales $\sigma_{1}<\sigma_{2}<\cdots<\sigma_{T}$, Equation 8 can be
written as:

$\frac{1}{T}\sum_{t=1}^{T}\lambda\left(\sigma_{t}\right)\mathbb{E}_{p(x)}\mathbb{E}_{x_{t}\sim
p_{\sigma_{t}}\left(x_{t}\mid
x\right)}\left\|s_{\theta}\left(x_{t},\sigma_{t}\right)+\frac{x_{t}-x}{\sigma_{t}}\right\|_{2}^{2}$
(9)

where $\lambda\left(\sigma_{t}\right)$ is a weighting function. The inference
is done using an iterative procedure called "Langevin dynamics" [65, 66].
Langevin dynamics design an MCMC procedure to sample from a distribution
$p(\mathbf{x})$ using only its score function $\nabla_{\mathbf{x}}\log
p(\mathbf{x})$. Specifically, to move from a random sample
$\mathbf{x}_{0}\sim\pi(\mathbf{x})$ towards samples from $p(\mathbf{x})$, it
iterates the following:

$x_{i}=x_{i-1}+\frac{\gamma}{2}\nabla_{x}\log
p(x)+\sqrt{\gamma}\cdot\omega_{i}$ (10)

where $\omega_{i}\sim\mathcal{N}(0,\mathbf{I})$, and $i\in\\{1,\ldots,N\\}$.
When $\gamma\rightarrow 0$ and $N\rightarrow\infty,\mathbf{x}_{i}$ samples
obtained from this procedure converge to a sample from $p(\mathbf{x})$. The
authors of [61] propose a modification of this algorithm nomenclature as the
annealed Langevin dynamics algorithm since the noise scale $\sigma_{i}$
decreases (anneals) gradually over time to mitigate some pitfalls and failure
modes of score matching [67].

####  2.1.3 Stochastic Differential Equations (SDEs)

Similar to the aforementioned two approaches, score-based generative models
(SGMs) [62] transform the data distribution $q\left(x_{0}\right)$ into noise.
However, by generalizing the number of noise scales to infinity, one can view
the previous probabilistic models as a discretization of an SGM. We know that
many stochastic processes, such as the diffusion process, are the solution to
a stochastic differential equation (SDE) in the following form:

$\mathrm{d}\mathbf{x}=\mathbf{f}(\mathbf{x},t)\mathrm{d}t+g(t)\mathrm{d}\mathbf{w}$
(11)

where $\mathbf{f}(.,t)$ is the drift coefficient of the SDE,
$\mathrm{g}(\mathrm{t})$ is the diffusion coefficient, and w represents the
standard Brownian motion. Let $\mathbf{x}_{0}$ be the uncorrupted data sample,
and $\mathbf{x}_{T}$ denote the perturbed data approximating standard Gaussian
distribution. For the given forward SDE, there exists a reverse time SDE
running backward where, by starting out with a sample from $p_{T}$ and
reversing this diffusion SDE, we can obtain samples from our data distribution
$p_{0}$. The reverse-time SDE is given as:

$d\mathbf{x}=\left[\mathbf{f}(\mathbf{x},t)-g^{2}(t){\color[rgb]{1,0,0}\nabla_{x}\log
p_{t}(x)}\right]dt+g(t)d\bar{\mathbf{w}}$ (12)

where $dt$ is the infinitesimal negative time step, and $\bar{w}$ is the
Brownian motion running backward. In order to numerically solve the reverse-
time SDE, one can train a neural network to approximate the actual score
function via score matching [61, 62] to estimate
$\boldsymbol{s}_{\theta}(\boldsymbol{x},t)\simeq\nabla_{\boldsymbol{x}}\log
p_{t}(\boldsymbol{x})$ (denoted red in Equation 12). This score model is
trained using the following objective:

$\mathcal{L}(\theta)=\mathbb{E}_{\mathbf{x}(t)\sim
p(\mathbf{x}(t)\mid\mathbf{x}(0)),\mathbf{x}(0)\sim p_{\text{data
}}}\left[\frac{\lambda(t)}{2}\left\|s_{\theta}(\mathbf{x}(t),t)-\nabla_{\mathbf{x}(t)}\log
p_{t}(\mathbf{x}(t)\mid\mathbf{x}(0))\right\|_{2}^{2}\right]$ (13)

where $\lambda$ is a weighting function, and $t\sim\mathcal{U}([0,T])$.
Notably, $\nabla_{\boldsymbol{x}}\log p_{t}(\boldsymbol{x})$ is replaced with
$\nabla_{\boldsymbol{x}}\log p_{0t}(\boldsymbol{x}(t)\mid\boldsymbol{x}(0))$
to circumvent technical difficulties.

Figure 5: Ten synthetic histopathology images generated by MFDPM [49].

###  2.2 Clinical Importance

Generative models have significantly impacted the field of medical imaging,
where there is a strong need for tools to improve the routines of clinicians
and patients. Concretely, the complexity of data collection procedures, the
lack of experts, privacy concerns, and the compulsory requirement of
authorization from patients create a major bottleneck in the annotation
process in medical imaging. This is where generative models become
advantageous [68]. Several perspectives have driven our interest in generative
diffusion models for medical imaging. In the medical field, many datasets
suffer from severe class imbalance due to the rare nature of some pathologies.
Diffusion models can alleviate this restriction by generating diverse
realistic-looking images to leverage in the medical field. Furthermore,
generating synthetic medical images has substantial educational value. With
its ability to produce a limitless source of unique instances of different
medical imaging modalities, diffusion models can satisfy educational demands
by constructing distinct synthetic samples for teaching and practice.
Additionally, these artificial images can mitigate data security concerns
associated with using patient data in public settings. These artificial images
can also solve a particular significant difficulty in training deep neural
networks for medical applications. Generally, the annotation of medical images
is a lengthy and costly process that necessitates the assistance of an expert.
Hence, using diffusion models to generate synthetic samples can alleviate the
problem of medical data scarcity to a great extent. A case study using [49] to
generate histopathology images with rare cancer subtypes is described in
Figure 5.

##  3 Diffusion Models in Action

Providing a taxonomy for diffusion models more or less follows the same route
as other techniques for medical imaging analysis. We provide, however,
detailed additional information for each sub-category paper in Figure 4. In
this section, we explore diffusion-based methods, which are proposed to solve
any disentanglement from the medical imaging analysis in seven application
categories, as in Figure 4: (1) Image-to-Image Translation, (2) Image
Segmentation, (3) Anomaly Detection, (4) Medical Image Reconstruction, (5)
Image Generation, (6) Medical Image Denoising and (7) multi-disciplinary
applications, named Other Applications. Figure 4 represents a collection of
numerous studies for each category with extensive information on each study,
such as the modality of study, the organ of interest, and the specific
algorithm utilized in the reverse process of the diffusion model for study.
Finally, in Section 3.8, we discuss the overall algorithms used in the studies
and try to shed light on the main novelty and contribution of the papers in
Table 1.

###  3.1 Anomaly Detection

Medical anomaly detection is an important topic in computer vision, aiming to
highlight the anomalous regions of the image [69, 70, 71]. Generative models
have dramatically shaped queries on anomaly detection in recent years and have
shown promising results. Accordingly, we explore diffusion models as dominant
generative models in anomaly detection in the following section.

Wolleb et al. [28] introduce a weakly supervised learning method based on
Denoising Diffusion Implicit Models (DDIMs) [72] for medical anomaly
detection. Given an input image of a healthy or diseased subject, image-to-
image translation first performs such that the objective is to translate the
input image into the healthy one. Then, the anomaly regions are identified by
subtracting the output image from the input. This process begins by encoding
an input image into a noisy image with reversed DDIM sampling. Then, the
denoising process is guided through a binary classifier trained beforehand on
the healthy and diseased images to produce the healthy image. Finally, the
anomaly map is calculated by taking the difference between the output and
input. Results on BRATS2020 [73, 74, 75] and CheXpert [76] datasets
demonstrate the superiority of the proposed approach compared to both VAE [77]
and GAN [78] models.

Wyatt et al. [53] in AnoDDPM train a DDPM only on healthy medical samples. The
anomaly image is then rendered by computing the difference between the output
and input images. They also show that leveraging Simplex noise over Gaussian
noise significantly enhances the performance.

In contrast, CDPM [54] demonstrates that training the diffusion probabilistic
models only on healthy data generates poor segmentation performance. Thus,
CDPM presents a counterfactual diffusion probabilistic model for generating
healthy counterfactuals from factual input images. As illustrated in Figure 6,
the input image is initially encoded into a latent space by iteratively
applying diffusion models using an unconditional model. Then, the decoding
step is accomplished by reversing the diffusion process. Using implicit
guidance [79], the latent is decoded into a counterfactual by conditioning it
on a healthy state and $\emptyset$. Inspired by [80, 81, 82], Sanchez et al.
[54] then enhance the conditioning process by incorporating conditional
attention into the U-Net backbone. As a final step, a dynamic normalization
technique is applied during inference to avoid saturation in latent space
pixels, caused by the guided iterative process that may change the image
statistics. Eventually, the location of abnormality is determined by
subtracting the input image from the generated healthy counterfactual.

Figure 6: An overview of CDPM [54]. Iteratively applying diffusion models
using an unconditional model $(c=\emptyset)$ encodes the input image into a
latent space. Then, reversing the diffusion process from the latent space
decodes a healthy state image. The decoding process is guided by conditioning
it on the healthy state and $\emptyset$. The anomaly heatmap is generated by
subtracting the input image from the generated counterfactual.

###  3.2 Denoising

The major challenge in medical imaging is obtaining an image without losing
important information. The images obtained may be corrupted by noise or
artifacts during the acquisition and/or further processing stages [83, 84].
Noise reduces the image quality and is especially significant when the imaged
objects are small and have relatively low contrast [85]. Due to the nature of
generative models, diffusion models are convenient for diverse denoising
problems. In this section, we will explore the contribution of diffusion
models to this field.

Hu et al. [47] utilized a DDPM [22] to despeckle Optical Coherence Tomography
(OCT) volumetric retina data in an unsupervised manner, denoted as DenoOCT-
DDPM. OCT imaging utility suffers from limited spatial-frequency bandwidth,
which leads to the resulting images containing speckle noise. Speckle noise
hinders the ophthalmologist’s diagnosis and can severely affect the visibility
of the tissue. The classic methods, such as averaging multiple b-scans at the
same location, have extreme drawbacks, such as prolonged acquisition time and
registration artifacts. Due to the multiplicative properties of speckle noise,
these methods enrich rather than reduce the noise. Deep-based models perform
outstandingly. This performance, however, depends on the availability of
noise-free images, which is a rare and costly process to acquire. To this end,
DenoOCT-DDPM [47] utilizes DDPM’s feasibility in noise patterns rather than
real-data pattern. Therefore, they use a self-fusion [86] as a preprocessing
step to feed the DDPM with a clear reference image and train the parameterized
Markov chain (see Figure 7(a)). Their investigation demonstrated the SOTA
results over the Pseudo-Modality Fusion Network (PMFN) [87], which uses
information from the single-frame noisy b-scan and a pseudo-modality that is
created with the aid of the self-fusion [86] method, regarding Signal-to-Noise
Ratio (SNR) metric. The qualitative results over PMNF depicted in Figure 7(b)
(represented in multiple acquisition SNRs) endorse the ability of diffusion
models in removing the speckle noise while preserving the fine-grained
features like small vessels.

][c]0.58

(a)

][c]0.4  SNR=92dB SNR=96dB SNR=101dB 5-mean PMFN Method

(b)

Figure 7: (Left) General pipeline of DenoOCT-DDPM [47]. To feed the diffusion
model more low-noise reference images, DenoOCT-DDPM applies a self-fusion [86]
on the red canvas b-scan in (a) with neighboring b-scans for higher SNR
b-scan. (b) indicates a straightforward DDPM scheme in an unsupervised manner
to learn speckle noise distribution in the red arrow sampling stream. (Right)
Ultimate visual comparison of DenoOCT-DDPM [47] for denoising b-scans of
volumetric OCT images demolished by speckle noise. DenoOCT-DDPM compared
itself with PMFN [87] and the average of 5 successive b-scans as the ground
truth. PMFN results in good feature preservation in diverse retinal layers,
but it can easily over-smooth small vessel regions. In contrast, the DenoOCT-
DDPM retinal layers are more homogenous than PMFN in diverse acquisition SNRs.
Figure 8: Comparison by different methods of surface error mapping of the left
hemisphere from 20 18F-MK-6240 test dataset. The results confirm that DDPM-MR-
PETCon has the lowest error, followed by the DDPM-PETMR method. DDPM-MR-PETCon
is short for using the MR image as network input while using the PET image as
a data consistency item, and DDPM-PETMR denotes that PET and MR images are
used as network input.

Positron Emission Tomography (PET) is a non-invasive imaging utility that
plays a crucial role in cancer screening and diagnosis. However, as with OCT
devices, PETs suffer from low SNR and resolution due to the low beam count
radiation to patients. Deep learning methods over PET image denoising have
advanced, but over-smoothing is a prominent drawback of CNN-based approaches.
Therefore, Conditional Generative Adversarial Networks (CGANs) [88, 89]
neutralize the mentioned deficiency but still depend on training and test sets
distributions. Gong et al. [46] proposed the DDPM-based framework for PET
denoising in collaboration with an assistive modality embedding as prior
information to DDPM formulation, namely PET-DDPM. Gong et al. used 18F-FDG and
18F-MK-6240 datasets for PET and MR modalities, respectively. PET-DDM is a
multi-disciplinary study investigating the excessive modalities of
collaboration in learning noise distribution through PET images. This
intuition follows the original paper in a generative paradigm [23] with a
guided classifier to converge the learned distribution to desired
distribution. PET-DDM produced SOTA results compared with U-Net [90] based
denoising network in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural
Similarity Index Measure (SSIM).

###  3.3 Reconstruction

Medical imaging modalities such as Computed Tomography (CT) and Magnetic
Resonance Imaging (MRI) are the most popular imaging tools in medicine.
However, due to their functionality, they can harm patients; therefore, their
radiation exposure is reduced from the standard dose, or the imaging process
is done in an under-sampled or sparse-view manner [91, 92, 93, 94]. To
diminish these drawbacks, e.g., low Signal-to-Noise Ratio (SNR) and Contrast-
to-Noise Ratio (CNR), medical image reconstruction must overcome the
challenges mentioned and solve this ill-posed inversion problem [95]. This
section overviews the diffusion-based paradigms for medical image
reconstruction and enhancement.

Magnetic Resonance Imaging (MRI) is a popular non-invasive imaging utility in
medical diagnosis treatment, but due to its innate physics, it is a time-
consuming process of imaging sessions in which the movement of patients
results in various artifacts in images. Therefore, to decrease bedtime and to
accelerate the reverse problem-solving from the spatial domain (or k-space) to
image level, miscellaneous solutions are provided in the supervised-learning
concept. However, these methods are not robust to distribution changes or
drifts in their train/test sets. Jalal et al. [96] proposed the first study in
the MRI reconstruction domain via Compressed Sensing with Generative Models
(CSGM). To this end, CSGM trains the score-based generative models [67] on MRI
images to utilize as prior information for the inversion pathway in
reconstructing realistic MRI data from under-sampled MRI in posterior sampling
scheme with Langevin dynamics [97]. CSGM [96] demonstrated its better
performance over fastMRI [98] and Stanford MRI [99] datasets with Structural
Similarity Index Measure (SSIM) and PSNR metrics in comparison with end-to-end
supervised-learning paradigms.

Cheng et al. [37] propose a score-based diffusion framework that solves the
inverse problem for image reconstruction from accelerated MRI scans, as
depicted in Figure 9. In the first step, a single continuous time-dependent
score function with denoising score matching is trained only with magnitude
images. Then, in the reverse SDE process, the Variance Exploding (VE)-SDE [62]
is exploited to sample from the pretrained score model distribution,
conditioned on the measurement. Afterward, the image is first split into real
and imaginary components at each step. Each part is fed into the predictor,
followed by data consistency mapping to reconstruct the image. The obtained
image is split again, and the correcter and the data consistency mapping are
applied to each part to compensate for errors during the diffusion and
reconstruct the improved image, respectively. Results demonstrate that the
proposed model outperforms the previous SOTA methods [100, 101, 98] and can
even reconstruct the data, which is considerably outside the training
distribution with high fidelity, e.g., reconstructing anatomy not seen during
training. In addition, the proposed framework has shown to be very effective
for reconstructing the image when multiple coils exist. For the aforementioned
problem, they present two approaches: (1) they reconstruct each coil image in
parallel; (2) they take into account the correlation between the coil images
by injecting the dependency between them at each given step during reverse
SDE. Then, the final image is acquired by taking the sum-of-root-sum-of-
squares (SSoS) [102] of each reconstructed coil image. Although these two
techniques have shown great results qualitatively and practically, they are
time-consuming.

Figure 9: An overview of Score-MRI [54]. $x_{T}$ is derived from the pre-
trained prior distribution, and $x_{0}$ is retrieved by sequentially applying
the predictor, data consistency, corrector, and data consistency steps, given
the measurement through the reverse SDE procedure.

###  3.4 Segmentation

Image segmentation is a vital task in computer vision, which investigates
simplifying the complexity of the image by decomposing an image into multiple
meaningful image segments [103, 104]. Specifically, it facilitates medical
analysis by providing beneficial information about anatomy-related areas.
However, deep learning models often require vast amounts of diverse pixel-
annotated training data in order to produce generalizable results [105, 106].
Nonetheless, the number of images and labels accessible for medical image
segmentation is restricted due to the time, cost, and expertise required [107,
108, 109]. To this end, diffusion models have been propelled to the forefront
in investigations of image segmentation by synthesizing the labeled data and
obviating the necessity for pixel-annotated data.

brainSPADE [43] proposes a generative model for synthesizing labeled brain MRI
images that can be used for training segmentation models. brainSPADE is
composed of a label generator and an image generator sub-model. The former is
responsible for creating synthetic segmentation maps, and the latter for
synthesizing images based on generated labels. In the label generator, the
input segmentation map is first encoded during training using a spatial VAE
encoder and builds a latent space. The compressed latent code is then diffused
and denoised via LDMs [82] and produces an efficient latent space in which
imperceptible details are ignored, and semantic information is highlighted
more. A spatial VAE decoder then constructs the artificial segmentation map
via the latent space. In the image generator, Fernandez et al. [43] take
advantage of SPADE [110], a VAE-GAN model, to build a style latent space from
the input arbitrary style and use it with the artificial segmentation map to
decode the output image. nnU-Net [111] was leveraged to examine the
performance. Findings show that the model achieves comparable results when
trained on synthetic data compared to that trained on factual data, and their
combination significantly improves the model result.

Kim et al. [44] propose a novel diffusion adversarial representation learning
(DARL) model for self-supervised vessel segmentation, aiming to diagnose
vascular diseases. There are two main modules in the proposed DARL model: a
diffusion module, which learns background image distribution, and a generation
module, which generates vessel segmentation masks or synthetic angiograms
using a switchable SPADE algorithm [110]. Figure 10 illustrates two ways in
which this method can be applied. In path (A), a real noisy angiography image
$x^{a}_{t_{a}}$ is input into the model to produce a segmentation mask
$\hat{s}^{v}$, and the SPADE switch is off. In path (B), a real noisy
background image $x^{b}_{t_{b}}$ is fed into the model, and the SPADE becomes
active and receives a vessel-like fractal mask, generating a synthetic
angiography image $\hat{x}^{a}$. Then, by giving the generated synthetic
angiography images into the path (A), a cycle is formed, which helps in
learning the vessel information. In addition, during inference, path (A) is
performed at one step, where the model produces the mask by only inputting the
noisy angiography image into the model. Results verify the generalization,
robustness, and superiority of the proposed method compared to SOTA un/self-
supervised learning approaches.

Figure 10: An overview of DARL [44]. Path (A) involves feeding a real noisy
angiography image through the model to generate a segmentation map. Path (B)
incorporates passing a noisy background image alongside a vessel-like fractal
mask through the model to synthesize a synthetic angiography image.

###  3.5 Image-to-Image Translation

Acquiring multi-modality images for diagnosis and therapy is often crucial.
Also, we may miss modalities in some conditions. Diffusion models have shown
favorable results for generating missing modalities utilizing cross-modalities
and producing ones using other modality types, e.g., translating from MRI to
CT.

CT and MRI are two of the most prevalent imaging types. CT, however, is
limited in displaying the intricacies of the images for soft tissue injuries.
Hence, a subsequent MRI may be needed for a conclusive diagnosis after
receiving the initial CT results. Nevertheless, in addition to being time-
consuming and costly, this process may also cause misalignment between MRI and
CT images. To this end, Lyu et al. [32] take advantage of the recently
introduced DDPMs [112, 22] and score-based diffusion models [62] in solving
the translation problem between two modalities, i.e., from MRI to CT. In
particular, they present conditional DDPM and conditional SDE, in which their
reverse process is conditioned on T2w MRI images and conducts comprehensive
experiments. The authors adopt the DDPM and SDE with three different sampling
methods and compare them with the existing GAN-based [113] and CNN-based [90]
methods. These three methods, which aim to solve the reverse-time SDE Equation
12, can be summarized as follows:

  1. 1.

Euler-Maruyama (EM) method: Using a simple discretization technique in which
$dt$ is replaced with $\Delta t$ and $d\bar{w}$ with Gaussian noise
$z\sim\mathcal{N}(0,\Delta t\cdot I)$, Equation 12 can be solved.

  2. 2.

Prediction-Correction (PC) method: In this method, the prediction and
correction process takes place in a nested loop, in which the prior data is
first predicted and then corrected in several steps. The predictor can be
solved using EM. Since the corrector can be any score-based Markov Chain Monte
Carlo (MCMC) method, including annealed Langevin dynamics, it can be solved
utilizing Langevin dynamics in Equation 10.

  3. 3.

Probability flow ODE method: SDE equations in Equation 11 can be written as
ODE equations as follows:

$d\mathbf{x}=\left[\mathbf{f}(\mathbf{x},t)-\frac{1}{2}g^{2}(t)\nabla_{x}\log
p_{t}(x)\right]dt$ (14)

Hence, by solving the ODE problem, $x_{0}$ can be found. However, while ODE is
a quick solver, it lacks a stochastic term to correct errors, resulting in
slightly diminished performance.

Their extensive experiments on the Gold Atlas male pelvis dataset [114]
demonstrate that diffusion models outperform both CNN and GAN-based methods in
terms of SSIM and PNSR. Additionally, they employ the Monte Carlo (MC) method
to investigate the uncertainties of diffusion models; in this technique, the
model outputs ten times, and the average yields the final result. Qualitative
results are depicted in Figure 11.

Figure 11: Visual and quantitative comparison of different methods of
translating an MR image to a CT image conducted by [32]. The second and third
rows indicate zoomed regions in the original image, delineated by red and
green boxes. Results after applying the Monte Carlo method (taking an average
from ten outputs) using DDPM, ODE, EM, and ODE sampling methods are displayed
with DDPM-M, ODE-M, EM-M, and PC-M, respectively.

To cope with the missing modality issue, Meng et al. [33] propose a unified
multi-modal conditional score-based generative approach (UMM-CSGM), which
synthesizes the missing modality based on all remaining modalities as
conditions. The proposed model is a conditional format of SDE [62], in which
it employs only a score-based network to learn different cross-modal
conditional distributions. Experiments on the BraTS19 dataset [73, 74, 75],
which contains four MRI modalities for each subject, show that the UMM-CSGM is
capable of generating missing-modality images with higher fidelity and
structural information of the brain tissue compared to SOTA methods [115, 116,
117, 118, 119].

###  3.6 Image Generation

Image generation is one of the primary objectives of diffusion models, which
has been widely applied in a variety of styles, including generating synthetic
2D/3D medical images [48, 49, 51, 52], reconstructing 3D cell from 2D cell
images [50], etc. This section will outline the diffusion-based approaches for
medical image generation.

Using 4D imaging to follow anatomical changes is one of the methods used in
medicine to track 3D volumes over time to detect anomalies and disease
progression. Such 4D images are primarily obtained with MRI, but this is a
relatively time-consuming process. Kim et al. [52] recently proposed the
Diffusion Deformable Model (DDM), which takes source and target images and
generates intermediate temporal frames along the continuous trajectory. This
approach comprises two main modules: (i) a denoising diffusion probabilistic
model (DDPM) module and (ii) a deformation module. In the DDPM module, a
latent code is constructed by learning the source and target images, and in
the deformation module, the acquired latent code and the source image are used
to render the deformed image. In the training phase, as shown in Figure 12(a),
the diffusion model, derived from [22], takes source, target, and perturbed
target images and outputs a latent code. The learned latent code along the
source image is fed into the deformation module, adopted from [120], and
creates deformation fields. Then, the spatial transformation layer (STL)
[121], with tri-linear interpolation, is employed to warp the source volume
using the deformation fields in order to build the deformed source image.
Afterward, inference begins with the diffusion module providing the latent
code, which contains spatial information from the source toward the target
(see Figure 12(b)). Then, deformed intermediate frames are generated using the
deformation module by scaling the latent code with a factor, which is an
element of [0, 1].

(a)

(b)

Figure 12: (a) demonstrates the DDM [52] training phase and (b) the inference
phase. Figure 13: Visual comparison of DDM [52], VM [120], and VM-Diff [122]
for generating temporal cardiac images. The deformed intermediate frames
$S(\tilde{\phi})$ (right) are constructed using the source and target, and
produced the deformation fields $\tilde{\phi}$ (left).

Histopathology involves the study of tissues and cells at the microscopic
level in order to diagnose diseases and cancer [123]. Histology images,
however, are rare for some cancer subtypes, thereby increasing the
significance of generative models to fill the void. To this end, Moghadam et
al. [49] investigate adopting DDPMs for generating histopathology images for
the first time. Specifically, they exploit the DDPMs with genotype guidance to
synthesize images containing various morphological and genomic information. To
tackle this data consistency problem and enforce the model to focus more on
morphological patterns, they first feed input images into a color
normalization module [124] to unify the domain of all images. In addition,
they apply a morphology levels prioritization module [125] that designates
higher weight values to the loss at earlier levels to emphasize perceptual
information and lower weights to the loss at later levels, resulting in higher
fidelity samples. Experiments on the Cancer Genome Atlas (TCGA) dataset [126]
exhibit the superiority of the proposed method compared to GAN-based
approaches [127].

###  3.7 Other Applications and Multi-tasks

Based on Figure 4, there are still studies that could not be assigned to a
particular category, and the use of diffusion is not limited to those six
categories. Multi-scale DDAM [56] utilizes DDPM [22] for designing a
neutralizing concept for adversarial attacks by Gaussian noise. Diffusion
models are not limited to vision-related tasks in the medical domain and can
be an innovation in biology. The platforms presented in [57, 128] can be used
for designing drugs and vaccines. In the following paragraphs, we explore some
of the recent diffusion-based approaches used in multi-task learning and
distinctive uses in the medical domain.

Un/self-supervised learning is an ideal alternate approach in medical image
denoising, where accessing paired clean and noisy images is difficult to
achieve [129, 130]. Conventional deep-based networks utilize Minimum Mean
Square Error (MMSE) estimates, which lead to unsatisfactory and blurred images
due to the distribution change in train/test data or the preliminary
assumption of Gaussian noise is at odds with the data’s actual distribution.
Chung et al. [58] proposed a multi-successive paradigm for MRI image denoising
and super-resolution, namely R2D2+, with the SDE [62] algorithm to tackle the
mentioned deficiencies. Diffusion generative models are robust to any
distribution change over the data and produce more realistic data [23].
Despite the advantages of diffusion models, they are very time-consuming. To
this end, Chung et al. [58] do not start the reverse diffusion process from
the pure noise but start from the initial noisy image. R2D2+ [58] solves a
reverse time SDE procedure with a non-parametric estimation method based on
eigenvalue analysis of covariance matrix rather than the conventional
numerical methods [62]. To restrain structure alleviation through the process,
R2D2+ uses a low-frequency regularization to hamper any change in the low-
frequency counterparts of the image. R2D2+ utilizes the same network for the
super-resolution task after the denoising step. The overall results over the
single coiled fastMRI [98] knee dataset and private liver MRI dataset indicate
the superiority of this approach over conventional SOTA un/self-supervised
learning schemes in terms of SNR and Contrast-to-Noise Ratio (CNR) metrics.

Figure 14: An overview of BAnoDDPM [59]. An autoencoder compresses the input
image into a latent code, further enhanced by applying diffusion and reverse
processes, and decodes into the pixel space.

Pinaya et al. [59] propose a fast DDPM-based approach for detecting and
segmenting anomalous regions in the brain MR images (see Figure 14). This
method follows the strategy of generating a healthy sample and delineating the
anomaly segmentation map by subtracting it from the input image. To this end,
VQ-VAE [131] is first adopted following [82], which encodes the input images
into a compact latent representation and provides the quantized latent
representation from input images utilizing a codebook. The DDPM then uses the
acquired latent space and learns the distribution of the latent representation
of the healthy samples. A binary mask indicating the location of the anomaly
is constructed by applying a pre-calculated threshold on the average of
intermediate samples of the reverse process, which contain less noisy and more
distinct values. Using the middle step as the starting point for the reverse
process, they denoise the anomalous areas of the image and preserve the rest
using the obtained mask, thereby removing the lesion from the sample.
Eventually, upon decoding the sample, a healthy image is produced.

###  3.8 Comparative Overview

Table 1 comprehensively categorizes the reviewed diffusion model papers
according to which algorithm they directly used or inspired: (1) DDPMs, (2)
NCSNs, and (3) SDEs. In addition, Table 1 highlights the key concepts and
objectives of each algorithm and represents the practical use cases that can
be investigated and utilized in future research based on reviewed papers.

It is evident that conditioning the reverse diffusion process is one of the
most studied methods for obtaining the desired output. This guiding process
can be done using different constraint types. In [32, 36, 50], they control
the reverse process by applying conditions using images; in particular, Lyu et
al. [32] condition the DDPM and SDE utilizing T2w MR images to obtain CT
images, Xie et al. [36] propose measurement-conditioned DDPM constituted for
under-sampled medical image reconstruction, and Waibel et al. [50] constrain
the 3D model using 2D microscopy images for generating 3D single cell shapes.
Moreover, BrainGen [39] produces realistic examples of brain scans, which are
conditioned on meta-data such as age, gender, ventricular volume, and brain
volume relative to intracranial volume. In addition, the use of a classifier
and implicit guidance methods in [28] and [54] have been investigated
thoroughly. In this way, the distribution is shifted in a manner that is more
likely to reach the expected outcome.

Some of the primary concerns and limitations of diffusion models are their
slow speed and required computational cost. Several methods have been
developed to address these drawbacks. Training-free Denoising Diffusion
Implicit Model (DDIM) [72] is one of the advancements designed to accelerate
the sampling process. DDIM extends the DDPM by substituting the Markovian
process with the non-Markovian one, resulting in a faster sampling procedure
with negligible quality degradation. [132, 133] propose a suitable
initialization instead of a random Gaussian noise in the reverse process,
causing a significant acceleration. Specifically, Chung et al. [133] prove
that, based on stochastic contraction theory, beginning the reversion, for
example, after one step prediction of the pre-trained neural network, fastens
the reverse diffusion and reduces the number of reverse samplings. Dar et al.
[39] also verify that adversarial learning can boost reverse diffusion speed
by two orders of magnitude.

Several methods have also been worked on to enhance the output quality of
diffusion models. AnoDDPM [53] substantiates that generalization to other
types of noise distributions can enhance task-specific quality. They ascertain
that in the case of anomaly detection, Simplex noise shows superiority over
Gaussian noise. Furthermore, Cao et al. [132] corroborate that operating the
diffusion process only in the high-frequency part of the image improves the
stability and quality of the MRI reconstruction.

With their remarkable results, diffusion models have proven that they can be a
powerful competitor against other generative models, as an examination of
Figure 15 reveals the promising future of this field in the academic
environment.

Table 1: Overview of the reviewed diffusion models for medical image analysis
based on their algorithm choice presented in our taxonomy, Figure 4. The
symbol * indicates that the mentioned paper explores both DDPM and SDE
algorithms.

Algorithm |  Networks |  Core Ideas |  Practical Use Cases
---|---|---|---
|  |  |
Denoising Diffusion Probabilistic Models (DDPMs) |  1AnoDDPM [53]
2CDPM [54]
3IITM-Diffusion [55]
4AnoDDIM [28]
5PET-DDPM [46]
6DenoOCT-DDPM [47]
7brainSPADE [43]
8DARL [52]
9IISE [45]
10*MT-Diffusion [32]
11SynDiff [34]
12MC-DDPM [36]
13DiffuseRecon [38]
14AdaDiff [48]
15BrainGen [39]
16MFDPM [49]
17DISPR [50]
183D-DDPM [51]
19DDM [52]
20Multi-scale DDAM [56]
21SMCDiff [57]
22BAnoDDPM [59]
23DiffuseMorph [134]
2420x-DenoDDPM [135] |  In DDPMs [22], the forward diffusion process is represented as a Markov chain in which Gaussian noise is gradually added to the data. Data generation is then accomplished using the attained pure random noise and begins iterative denoising through a parametrized reverse process. Unlike VAE, where both the encoder and decoder are trained, only a single network is trained during the reverse process, and the forward process is considered fixed. An objective function of DDPMs is to simulate noise, which means that given a noisy input image, the neural network will produce the distribution modeled as normal distribution and indicate where the noise originates. |  $\bullet$ Generalization to other types of noise distributions [53]
$\bullet$ Facilitating diffusion process via implicit guidance [54]
$\bullet$ Acceleration improvement using training-free DDIM sampling [55, 28]
$\bullet$ Guiding diffusion process via classifier guidance [28]
$\bullet$ Conditional DDPMs [46, 32, 36, 50, 135, 57]
$\bullet$ Generating synthetic segmentation datasets [43, 44]
$\bullet$ Cross-modality translation [34]
$\bullet$ Multi-modal conversion [32, 34]
$\bullet$ Exploiting K-space parameter-free guidance [38]
$\bullet$ Accelerate MC sampling using coarse-to-fine sampling [38]
$\bullet$ Adversarial learning in the reverse diffusion process [39, 34]
$\bullet$ 3D reconstruction from 2D images [50]
$\bullet$ Conditioning on medical meta-data [48]
$\bullet$ Using LDM to enhance the training and sampling efficiency [48, 59]
$\bullet$ DDPMs for histopathology images [49]
$\bullet$ 3D medical image generation [51]
$\bullet$ Using deformation fields for medical image generation [134, 52]
$\bullet$ DDPMs in skin image adversarial attacks [56]
|  |  |
Noise Conditioned Score Networks (NCSNs) |  1CSGM-MRI-Langevin [96] 2Self-Score [41] |  In this algorithm [61], creating samples requires solving the Langevin dynamics equation. However, this equation mandates the solution of the gradient of the log density w.r.t. the input, $\nabla_{x}\log p(x)$, which is unknown and intractable. NCSN formulates the forward diffusion process by disturbing the data with Gaussian noise at different scales. Through this approach, this equation can be solved by training a single score network conditioned on the noise level and modeling the scores at all noise levels. Therefore, using the annealed Langevin dynamics algorithm and estimated score function at each scale, data can be generated. |  $\bullet$ Using Langevin dynamics with different random initializations to get multiple reconstructions [96]
$\bullet$ Using conditioned Langevin Markov chain Monte Carlo (MCMC) sampling
[41]
$\bullet$ Utilizing K-space data [41]
|  |  |
Stochastic Differential Equations (SDEs) |  1*MT-Diffusion [32]
2UMM-CSGM [33]
3SIM-SGM [35]
4Score-MRI [37]
5MRI-DDMC [40]
6MCG [42]
7HFS-SDE [132]
8Diffuse-Faster [133]
9HKGM [136]
10WKGM [137]
11R2D2+ [58] |  As in the cases of DDPMs and NCSNs, SDEs [62] follow a similar approach in the forward diffusion process, in which the input data is perturbed consecutively utilizing Gaussian noise. In contrast, previous probabilistic models can be viewed as discretizations of score-based models by extending the number of noise scales to infinity and treating them as continuous.
Having operated backward in time, the reverted SDE Process solves the reverse-
time Stochastic Differential Equation and recovers the data. Even so, each
time step of this process requires the score function of the density. SDEs
are, therefore, intended to learn the actual score function via a neural
network and construct samples using numerical SDE solvers.
Any numerical method can be adopted to solve the reverse-time SDE equation. In particular, three notable sampling methods are named and described in Section 3.5: Euler-Maruyama (EM) method, Prediction-Correction (PC), and Probability Flow ODE. EM follows a simple strategy, PC generates more high-fidelity samples, and Probability Flow ODE is fast and efficient. |  $\bullet$ SDEs achieve better results in multi-modal conversion [32]
$\bullet$ Cross-modality translation [33]
$\bullet$ Conditional SDEs [33, 38, 42]
$\bullet$ Solving linear inverse problems [35]
$\bullet$ Using K-space data [136, 137]
$\bullet$ Using manifold constraints [42]
$\bullet$ Good initialization in the reverse process instead of random
Gaussian noise leads to a faster convergence [133, 132]
$\bullet$ Diffusion only in high-frequency space increases the stability and
quality [132]

|  |  |

Figure 15: Diffusion models timeline from 2021 to 2022-Q3 through the first
paper in the medical field. The superscripts in ascending order represent
1.[96], 2.[37], 3.[35], 4.[133], 5.[45], 6.[134], 7.[47], 8.[40], 9.[28],
10.[38], 11.[36], 12.[58], 13.[51], 14.[55], 15.[137], 16.[52], 17.[59],
18.[53], 19.[57], 20.[42], 21.[39], 22.[34], 23.[33], 24.[54], 25.[136],
26.[50], 27.[56], 28.[132], 29.[46], respectively. Figure 16: Simple
diffusion-based ecosystem in the medical domain. Some of the image elements
are designed by Freepik© [138].

##  4 Future Direction and Open Challenges

Diffusion models are becoming a popular topic in medical vision and medical-
biology fields; evidence shows an upward trend in Figure 3. Due to the lack of
such data in this field, diffusion processes do not need a pair of labeled or
not labeled data, making them a strong candidate in medical image analysis.
This paper gathered the latest medical-related papers and investigated
numerous of them based on the taxonomy proposed in Figure 4 to demonstrate the
power of diffusion models. We hope this review sheds light on the emergence of
diffusion models in the medical field and serves as a guide for readers. Thus,
in this section, we highlight the open directions for future investigations.
In addition, due to the nature of diffusion models, they are a strong
candidate for denoising medical images in diverse modalities. Still, a limited
number of studies address this concern (see Section 3.2), rendering it
suitable for further research. For clarification, sonography uses a high
frequency to view inside a body, thus the transmitted ultrasound beams weaken
as they penetrate the tissue. Moreover, Radio-Frequency (RF) data contains an
additive Gaussian noise which is brought about by the acquisition card and the
sensor noise [139]. Therefore, diffusion models could compensate for the
drawbacks of sonography imaging technology practically instead of utilizing
the conventional methods, e.g., increasing the power of transmitted beams that
control by safety threshold, averaging successive frames to hinder the noise
effect that causes a low frame rate and decreasing the transmitting frequency
that slumps the spatial resolution.

Diffusion models generally suffer from deficiencies that are probably
transferred to early-stage medical-based studies. First, diffusion models are
time-consuming processes that originate from performing iterative steps at
inference time to generate a sample. Second, most of the works paid attention
to Gaussian noise distribution through the diffusion models and neglected
other perturbations, e.g., Poisson noise which is the most probable
distribution in imaging modalities such as PET [140]. Efforts in facilitating
the mentioned limitations could be a direction of research in the medical
domain with diffusion models.

Due to privacy concerns in medical imaging, which limit data integration,
diffusion models along with federated learning can create a profound and
robust learning platform in the medical domain. In addition, due to the same
privacy concerns, diffusion models can consolidate their steps in generating
synthetic medical data for educational purposes. The inverse problem-solving
of the diffusion models could be performed by the reinforcement learning
paradigm to estimate the best inversion path rather than solid mathematical
solutions. Diffusion models become a cornerstone in text-to-image tasks [80,
141]. However, the reverse problem, image-to-text, is still in its early
stages [142], which is a promising direction for research to produce
comprehensive texts such as transcripts from medical images as assistive
information for physicians. Exploiting conditional diffusion models for using
meta-data like CT images in the analysis of MRI, and PET could be another
deviation of the diffusion research in the medical field. Finally, due to the
wide use of diffusion models in diverse medical tasks (as we studied during
this survey), it would be remarkable to create software ecosystems that
contain diffusion models for these tasks analogous to primitive studies such
as nnU-Net [111] and Ivadomed [143]. In Figure 16, we depict the simple
proposal ecosystem on a small scale within a specific healthcare
establishment, i.e., a hospital. The first step of this pipeline starts with
data acquisition within the diverse imaging systems in the organization.
Afterward, by applying preprocessing steps, the data is stored in the data
centers to share with diffusion-based software for desired purposes.

##  5 Conclusion

In this paper, we provided a survey of the literature on diffusion models with
a focus on applications in the medical imaging field. Specifically, we
investigated the applications of diffusers in anomaly detection, medical image
segmentation, denoising, classification, reconstruction, synthesis,
generation, and other tasks. In particular, for each of these applications, we
provided a taxonomy and high-level abstraction of the core techniques from
various angles. Moreover, we characterized the existing models based on
techniques where we identified three primary formulations of diffusion
modeling based on: DDPMs, NCSNs, and SDEs. Finally, we outlined possible
avenues for future research.

## References

  * [1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [2] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang.  A tutorial on energy-based learning.  Predicting structured data, 1(0), 2006.
  * [3] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [4] George Papamakarios, Eric T Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  J. Mach. Learn. Res., 22(57):1–64, 2021.
  * [5] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua.  Cvae-gan: fine-grained image generation through asymmetric training.  In Proceedings of the IEEE international conference on computer vision, pages 2745–2754, 2017.
  * [6] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  Advances in neural information processing systems, 32, 2019.
  * [7] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [8] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  arXiv preprint arXiv:1609.03499, 2016.
  * [9] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  arXiv preprint arXiv:2205.14217, 2022.
  * [10] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541–4550, 2019.
  * [11] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks.  Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models.  arXiv preprint arXiv:2103.04922, 2021.
  * [12] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In International conference on machine learning, pages 1278–1286. PMLR, 2014.
  * [13] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  arXiv preprint arXiv:1605.08803, 2016.
  * [14] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the generative learning trilemma with denoising diffusion gans.  arXiv preprint arXiv:2112.07804, 2021.
  * [15] Maciej Wiatrak and Stefano V Albrecht.  Stabilizing generative adversarial network training: A survey.  2019\.
  * [16] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  arXiv preprint arXiv:1802.05957, 2018.
  * [17] Tanya Motwani and Manojkumar Parmar.  A novel framework for selection of gans for an application.  arXiv preprint arXiv:2002.08641, 2020.
  * [18] Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak.  Hyperspherical variational auto-encoders.  arXiv preprint arXiv:1804.00891, 2018.
  * [19] Andrea Asperti.  Variational autoencoders and the variable collapse phenomenon.  Sensors & Transducers, 234(6):1–8, 2019.
  * [20] Lilian Weng.  Flow-based deep generative models.  lilianweng. github. io/lil-log, 2019.
  * [21] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [22] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [23] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [24] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen.  Srdiff: Single image super-resolution with diffusion probabilistic models.  Neurocomputing, 479:47–59, 2022.
  * [25] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  * [26] Tomer Amit, Eliya Nachmani, Tal Shaharbany, and Lior Wolf.  Segdiff: Image segmentation with diffusion probabilistic models.  arXiv preprint arXiv:2112.00390, 2021.
  * [27] Roland S Zimmermann, Lukas Schott, Yang Song, Benjamin A Dunn, and David A Klindt.  Score-based generative classifiers.  arXiv preprint arXiv:2110.00473, 2021.
  * [28] Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C Cattin.  Diffusion models for medical anomaly detection.  arXiv preprint arXiv:2203.04306, 2022.
  * [29] Manal AlAmir and Manal AlGhamdi.  The role of generative adversarial network in medical image analysis: an in-depth survey.  ACM Computing Surveys (CSUR), 2022.
  * [30] Hazrat Ali, Rafiul Biswas, Farida Ali, Uzair Shah, Asma Alamgir, Osama Mousa, and Zubair Shah.  The role of generative adversarial networks in brain mri: a scoping review.  Insights into Imaging, 13(1):1–15, 2022.
  * [31] Yizhou Chen, Xu-Hua Yang, Zihan Wei, Ali Asghar Heidari, Nenggan Zheng, Zhicheng Li, Huiling Chen, Haigen Hu, Qianwei Zhou, and Qiu Guan.  Generative adversarial networks in medical image augmentation: a review.  Computers in Biology and Medicine, page 105382, 2022.
  * [32] Qing Lyu and Ge Wang.  Conversion between ct and mri images using diffusion and score-matching models.  arXiv preprint arXiv:2209.12104, 2022.
  * [33] Xiangxi Meng, Yuning Gu, Yongsheng Pan, Nizhuan Wang, Peng Xue, Mengkang Lu, Xuming He, Yiqiang Zhan, and Dinggang Shen.  A novel unified conditional score-based generative framework for multi-modal medical image completion.  arXiv preprint arXiv:2207.03430, 2022.
  * [34] Muzaffer Özbey, Salman UH Dar, Hasan A Bedel, Onat Dalmaz, Şaban Özturk, Alper Güngör, and Tolga Çukur.  Unsupervised medical image translation with adversarial diffusion models.  arXiv preprint arXiv:2207.08208, 2022.
  * [35] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon.  Solving inverse problems in medical imaging with score-based generative models.  arXiv preprint arXiv:2111.08005, 2021.
  * [36] Yutong Xie and Quanzheng Li.  Measurement-conditioned denoising diffusion probabilistic model for under-sampled medical image reconstruction.  arXiv preprint arXiv:2203.03623, 2022.
  * [37] Hyungjin Chung and Jong Chul Ye.  Score-based diffusion models for accelerated mri.  Medical Image Analysis, page 102479, 2022.
  * [38] Cheng Peng, Pengfei Guo, S Kevin Zhou, Vishal M Patel, and Rama Chellappa.  Towards performant and reliable undersampled mr reconstruction via diffusion model sampling.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 623–633. Springer, 2022.
  * [39] Salman UH Dar, Şaban Öztürk, Yilmaz Korkmaz, Gokberk Elmas, Muzaffer Özbey, Alper Güngör, and Tolga Çukur.  Adaptive diffusion priors for accelerated mri reconstruction.  arXiv preprint arXiv:2207.05876, 2022.
  * [40] Guanxiong Luo, Martin Heide, and Martin Uecker.  Mri reconstruction via data driven markov chain with joint uncertainty estimation.  arXiv preprint arXiv:2202.01479, 2022.
  * [41] Zhuo-Xu Cui, Chentao Cao, Shaonan Liu, Qingyong Zhu, Jing Cheng, Haifeng Wang, Yanjie Zhu, and Dong Liang.  Self-score: Self-supervised learning on score-based models for mri reconstruction.  arXiv preprint arXiv:2209.00835, 2022.
  * [42] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye.  Improving diffusion models for inverse problems using manifold constraints.  arXiv preprint arXiv:2206.00941, 2022.
  * [43] Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Petru-Daniel Tudosiu, Mark S Graham, Tom Vercauteren, and M Jorge Cardoso.  Can segmentation models be trained with fully synthetically generated data?  In International Workshop on Simulation and Synthesis in Medical Imaging, pages 79–90. Springer, 2022.
  * [44] Boah Kim, Yujin Oh, and Jong Chul Ye.  Diffusion adversarial representation learning for self-supervised vessel segmentation.  arXiv preprint arXiv:2209.14566, 2022.
  * [45] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin.  Diffusion models for implicit image segmentation ensembles.  arXiv preprint arXiv:2112.03145, 2021.
  * [46] Kuang Gong, Keith A Johnson, Georges El Fakhri, Quanzheng Li, and Tinsu Pan.  Pet image denoising based on denoising diffusion probabilistic models.  arXiv preprint arXiv:2209.06167, 2022.
  * [47] Dewei Hu, Yuankai K Tao, and Ipek Oguz.  Unsupervised denoising of retinal oct with diffusion probabilistic model.  In Medical Imaging 2022: Image Processing, volume 12032, pages 25–34. SPIE, 2022.
  * [48] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso.  Brain imaging generation with latent diffusion models.  In MICCAI Workshop on Deep Generative Models, pages 117–126. Springer, 2022.
  * [49] Puria Azadi Moghadam, Sanne Van Dalen, Karina C Martin, Jochen Lennerz, Stephen Yip, Hossein Farahani, and Ali Bashashati.  A morphology focused diffusion probabilistic model for synthesis of histopathology images.  arXiv preprint arXiv:2209.13167, 2022.
  * [50] Dominik JE Waibel, Ernst Röoell, Bastian Rieck, Raja Giryes, and Carsten Marr.  A diffusion model predicts 3d shapes from 2d microscopy images.  arXiv preprint arXiv:2208.14125, 2022.
  * [51] Zolnamar Dorjsembe, Sodtavilan Odonchimed, and Furen Xiao.  Three-dimensional medical image synthesis with denoising diffusion probabilistic models.  In Medical Imaging with Deep Learning, 2022.
  * [52] Boah Kim and Jong Chul Ye.  Diffusion deformable model for 4d temporal medical image generation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 539–548. Springer, 2022.
  * [53] Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willcocks.  Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 650–656, 2022.
  * [54] Pedro Sanchez, Antanas Kascenas, Xiao Liu, Alison Q O’Neil, and Sotirios A Tsaftaris.  What is healthy? generative counterfactual diffusion for lesion localization.  arXiv preprint arXiv:2207.12268, 2022.
  * [55] Julia Wolleb, Robin Sandkühler, Florentin Bieder, and Philippe C Cattin.  The swiss army knife for image-to-image translation: Multi-task diffusion models.  arXiv preprint arXiv:2204.02641, 2022.
  * [56] Yongwei Wang, Yuan Li, and Zhiqi Shen.  Fight fire with fire: Reversing skin adversarial examples by multiscale diffusive and denoising aggregation mechanism.  arXiv preprint arXiv:2208.10373, 2022.
  * [57] Brian L Trippe, Jason Yim, Doug Tischer, Tamara Broderick, David Baker, Regina Barzilay, and Tommi Jaakkola.  Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem.  arXiv preprint arXiv:2206.04119, 2022.
  * [58] Hyungjin Chung, Eun Sun Lee, and Jong Chul Ye.  Mr image denoising and super-resolution using regularized reverse diffusion.  arXiv preprint arXiv:2203.12621, 2022.
  * [59] Walter HL Pinaya, Mark S Graham, Robert Gray, Pedro F Da Costa, Petru-Daniel Tudosiu, Paul Wright, Yee H Mah, Andrew D MacKinnon, James T Teo, Rolf Jager, et al.  Fast unsupervised brain anomaly detection and segmentation with diffusion models.  arXiv preprint arXiv:2206.03461, 2022.
  * [60] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  arXiv preprint arXiv:2209.04747, 2022.
  * [61] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in Neural Information Processing Systems, 32, 2019.
  * [62] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [63] Pascal Vincent.  A connection between score matching and denoising autoencoders.  Neural computation, 23(7):1661–1674, 2011.
  * [64] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.  Sliced score matching: A scalable approach to density and score estimation.  In Uncertainty in Artificial Intelligence, pages 574–584. PMLR, 2020.
  * [65] Giorgio Parisi.  Correlation functions and computer simulations.  Nuclear Physics B, 180(3):378–384, 1981.
  * [66] Ulf Grenander and Michael I Miller.  Representations of knowledge in complex systems.  Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549–581, 1994.
  * [67] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  Advances in neural information processing systems, 33:12438–12448, 2020.
  * [68] Yanbin Liu, Girish Dwivedi, Farid Boussaid, and Mohammed Bennamoun.  3d brain and heart volume generative models: A survey.  arXiv preprint arXiv:2210.05952, 2022.
  * [69] Liyang Chen, Zhiyuan You, Nian Zhang, Juntong Xi, and Xinyi Le.  Utrad: Anomaly detection and localization with u-transformer.  Neural Networks, 147:53–62, 2022.
  * [70] Maximilian E Tschuchnig and Michael Gadermayr.  Anomaly detection in medical imaging-a mini review.  Data Science–Analytics and Applications, pages 33–38, 2022.
  * [71] Tharindu Fernando, Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes.  Deep learning for medical anomaly detection–a survey.  ACM Computing Surveys (CSUR), 54(7):1–37, 2021.
  * [72] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [73] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  Scientific data, 4(1):1–13, 2017.
  * [74] Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al.  Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge.  arXiv preprint arXiv:1811.02629, 2018.
  * [75] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  * [76] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019.
  * [77] David Zimmerer, Simon AA Kohl, Jens Petersen, Fabian Isensee, and Klaus H Maier-Hein.  Context-encoding variational autoencoder for unsupervised anomaly detection.  arXiv preprint arXiv:1812.05941, 2018.
  * [78] Md Mahfuzur Rahman Siddiquee, Zongwei Zhou, Nima Tajbakhsh, Ruibin Feng, Michael B Gotway, Yoshua Bengio, and Jianming Liang.  Learning fixed points in generative adversarial networks: From image-to-image translation to disease detection and localization.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 191–200, 2019.
  * [79] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [80] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [81] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [82] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [83] Zhicheng Zhang, Lequan Yu, Xiaokun Liang, Wei Zhao, and Lei Xing.  Transct: dual-path transformer for low dose computed tomography.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 55–64. Springer, 2021.
  * [84] Achleshwar Luthra, Harsh Sulakhe, Tanish Mittal, Abhishek Iyer, and Santosh Yadav.  Eformer: Edge enhancement based transformer for medical image denoising.  arXiv preprint arXiv:2109.08044, 2021.
  * [85] Salome Kazeminia, Christoph Baur, Arjan Kuijper, Bram van Ginneken, Nassir Navab, Shadi Albarqouni, and Anirban Mukhopadhyay.  Gans for medical image analysis.  Artificial Intelligence in Medicine, 109:101938, 2020.
  * [86] Ipek Oguz, Joseph D Malone, Yigit Atay, and Yuankai K Tao.  Self-fusion for oct noise reduction.  In Medical Imaging 2020: Image Processing, volume 11313, pages 45–50. SPIE, 2020.
  * [87] Dewei Hu, Joseph D Malone, Yigit Atay, Yuankai K Tao, and Ipek Oguz.  Retinal oct denoising with pseudo-multimodal fusion network.  In International Workshop on Ophthalmic Medical Image Analysis, pages 125–135. Springer, 2020.
  * [88] Long Zhou, Joshua D Schaefferkoetter, Ivan WK Tham, Gang Huang, and Jianhua Yan.  Supervised learning with cyclegan for low-dose fdg pet image denoising.  Medical image analysis, 65:101770, 2020.
  * [89] Tzu-An Song, Samadrita Roy Chowdhury, Fan Yang, and Joyita Dutta.  Pet image super-resolution using generative adversarial networks.  Neural Networks, 125:83–91, 2020.
  * [90] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [91] Chang Min Hyun, Hwa Pyung Kim, Sung Min Lee, Sungchul Lee, and Jin Keun Seo.  Deep learning for undersampled mri reconstruction.  Physics in Medicine & Biology, 63(13):135007, 2018.
  * [92] Yilmaz Korkmaz, Salman UH Dar, Mahmut Yurt, Muzaffer Özbey, and Tolga Cukur.  Unsupervised mri reconstruction via zero-shot learned adversarial transformers.  IEEE Transactions on Medical Imaging, 2022.
  * [93] Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, and Yong Xu.  Task transformer network for joint mri reconstruction and super-resolution.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 307–317. Springer, 2021.
  * [94] Yilmaz Korkmaz, Mahmut Yurt, Salman Ul Hassan Dar, Muzaffer Özbey, and Tolga Cukur.  Deep mri reconstruction with generative vision transformers.  In International Workshop on Machine Learning for Medical Image Reconstruction, pages 54–64. Springer, 2021.
  * [95] Ritu Gothwal, Shailendra Tiwari, and Shivendra Shivani.  Computational medical image reconstruction techniques: A comprehensive review.  Archives of Computational Methods in Engineering, pages 1–28, 2022\.
  * [96] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir.  Robust compressed sensing mri with deep generative priors.  Advances in Neural Information Processing Systems, 34:14938–14954, 2021.
  * [97] Dominique Bakry and Michel Émery.  Diffusions hypercontractives.  In Seminaire de probabilités XIX 1983/84, pages 177–206. Springer, 1985.
  * [98] Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, et al.  fastmri: An open dataset and benchmarks for accelerated mri.  arXiv preprint arXiv:1811.08839, 2018.
  * [99] Stanford University.  Stanford mri.  http://mridata.org/.  Online; accessed 6 October 2022.
  * [100] Kai Tobias Block, Martin Uecker, and Jens Frahm.  Undersampled radial mri with multiple coils. iterative image reconstruction using a total variation constraint.  Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 57(6):1086–1098, 2007\.
  * [101] Bo Zhou and S Kevin Zhou.  Dudornet: learning a dual-domain recurrent network for fast mri reconstruction with deep t1 prior.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4273–4282, 2020.
  * [102] Peter B Roemer, William A Edelstein, Cecil E Hayes, Steven P Souza, and Otward M Mueller.  The nmr phased array.  Magnetic resonance in medicine, 16(2):192–225, 1990.
  * [103] Reza Azad, Moein Heidari, Yuli Wu, and Dorit Merhof.  Contextual attention network: Transformer meets u-net.  arXiv preprint arXiv:2203.01932, 2022.
  * [104] Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, and Dorit Merhof.  Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation.  arXiv preprint arXiv:2207.08518, 2022.
  * [105] Reza Azad, Mohammad T Al-Antary, Moein Heidari, and Dorit Merhof.  Transnorm: Transformer provides a strong spatial normalization mechanism for a deep segmentation model.  IEEE Access, 10:108205–108215, 2022.
  * [106] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [107] Reza Azad, Moein Heidari, Moein Shariatnia, Ehsan Khodapanah Aghdam, Sanaz Karimijafarbigloo, Ehsan Adeli, and Dorit Merhof.  Transdeeplab: Convolution-free transformer-based deeplab v3+ for medical image segmentation.  In International Workshop on PRedictive Intelligence In MEdicine, pages 91–102. Springer, 2022.
  * [108] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  arXiv preprint arXiv:2105.05537, 2021.
  * [109] Ehsan Khodapanah Aghdam, Reza Azad, Maral Zarvani, and Dorit Merhof.  Attention swin u-net: Cross-contextual attention mechanism for skin lesion segmentation.  arXiv preprint arXiv:2210.16898, 2022.
  * [110] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337–2346, 2019.
  * [111] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature methods, 18(2):203–211, 2021.
  * [112] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [113] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville.  Improved training of wasserstein gans.  Advances in neural information processing systems, 30, 2017.
  * [114] Tufve Nyholm, Stina Svensson, Sebastian Andersson, Joakim Jonsson, Maja Sohlin, Christian Gustafsson, Elisabeth Kjellén, Karin Söderström, Per Albertsson, Lennart Blomqvist, et al.  Mr and ct data with multiobserver delineations of organs in the pelvic area—part of the gold atlas project.  Medical physics, 45(3):1295–1300, 2018.
  * [115] Pu Huang, Dengwang Li, Zhicheng Jiao, Dongming Wei, Guoshi Li, Qian Wang, Han Zhang, and Dinggang Shen.  Coca-gan: common-feature-learning-based context-aware generative adversarial network for glioma grading.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 155–163. Springer, 2019.
  * [116] Agisilaos Chartsias, Thomas Joyce, Mario Valerio Giuffrida, and Sotirios A Tsaftaris.  Multimodal mr synthesis via modality-invariant latent representation.  IEEE transactions on medical imaging, 37(3):803–814, 2017.
  * [117] Xiaofeng Liu, Fangxu Xing, Georges El Fakhri, and Jonghye Woo.  A unified conditional disentanglement framework for multimodal brain mr image translation.  In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 10–14. IEEE, 2021.
  * [118] Anmol Sharma and Ghassan Hamarneh.  Missing mri pulse sequence synthesis using multi-modal generative adversarial network.  IEEE transactions on medical imaging, 39(4):1170–1183, 2019.
  * [119] Tao Zhou, Huazhu Fu, Geng Chen, Jianbing Shen, and Ling Shao.  Hi-net: hybrid-fusion network for multi-modal mr image synthesis.  IEEE transactions on medical imaging, 39(9):2772–2781, 2020.
  * [120] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  An unsupervised learning model for deformable medical image registration.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9252–9260, 2018.
  * [121] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al.  Spatial transformer networks.  Advances in neural information processing systems, 28, 2015.
  * [122] Adrian V Dalca, Guha Balakrishnan, John Guttag, and Mert R Sabuncu.  Unsupervised learning for fast probabilistic diffeomorphic registration.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 729–738. Springer, 2018.
  * [123] Cecilia Di Ruberto, Lorenzo Putzu, HR Arabnia, and T Quoc-Nam.  A feature learning framework for histology images classification.  Emerging trends in applications and infrastructures for computational biology, bioinformatics, and systems biology: systems and applications, pages 37–48, 2016.
  * [124] Abhishek Vahadane, Tingying Peng, Amit Sethi, Shadi Albarqouni, Lichao Wang, Maximilian Baust, Katja Steiger, Anna Melissa Schlitter, Irene Esposito, and Nassir Navab.  Structure-preserving color normalization and sparse stain separation for histological images.  IEEE transactions on medical imaging, 35(8):1962–1971, 2016.
  * [125] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon.  Perception prioritized training of diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11472–11481, 2022.
  * [126] Robert L Grossman, Allison P Heath, Vincent Ferretti, Harold E Varmus, Douglas R Lowy, Warren A Kibbe, and Louis M Staudt.  Toward a shared vision for cancer genomic data.  New England Journal of Medicine, 375(12):1109–1112, 2016.
  * [127] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [128] Namrata Anand and Tudor Achim.  Protein structure and sequence generation with equivariant denoising diffusion probabilistic models.  arXiv preprint arXiv:2205.15019, 2022.
  * [129] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  arXiv preprint arXiv:1803.04189, 2018.
  * [130] Kwanyoung Kim and Jong Chul Ye.  Noise2score: tweedie’s approach to self-supervised image denoising without clean images.  Advances in Neural Information Processing Systems, 34:864–874, 2021\.
  * [131] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [132] Chentao Cao, Zhuo-Xu Cui, Shaonan Liu, Dong Liang, and Yanjie Zhu.  High-frequency space diffusion models for accelerated mri.  arXiv preprint arXiv:2208.05481, 2022.
  * [133] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye.  Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12413–12422, 2022.
  * [134] Boah Kim, Inhwa Han, and Jong Chul Ye.  Diffusemorph: Unsupervised deformable image registration along continuous trajectory using diffusion models.  arXiv preprint arXiv:2112.05149, 2021.
  * [135] Wenjun Xia, Qing Lyu, and Ge Wang.  Low-dose ct using denoising diffusion probabilistic model for 20$\times$ speedup.  arXiv preprint arXiv:2209.15136, 2022.
  * [136] Hong Peng, Chen Jiang, Yu Guan, Jing Cheng, Minghui Zhang, Dong Liang, and Qiegen Liu.  One-shot generative prior learned from hankel-k-space for parallel imaging reconstruction.  arXiv preprint arXiv:2208.07181, 2022.
  * [137] Zongjiang Tu, Die Liu, Xiaoqing Wang, Chen Jiang, Minghui Zhang, Qiegen Liu, and Dong Liang.  Wkgm: Weight-k-space generative model for parallel imaging reconstruction.  arXiv preprint arXiv:2205.03883, 2022.
  * [138] freepik.  Freepick.  https://www.freepik.com/.  Online; accessed 25 September 2022.
  * [139] Sobhan Goudarzi and Hassan Rivaz.  Deep ultrasound denoising without clean data.  arXiv preprint arXiv:2201.02604, 2022.
  * [140] A Teymurazyan, Terence Riauka, H-S Jans, and D Robinson.  Properties of noise in positron emission tomography images reconstructed with filtered-backprojection and row-action maximum likelihood algorithm.  Journal of digital imaging, 26(3):447–456, 2013.
  * [141] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [142] Pierre Chambon, Christian Bluethgen, Curtis P Langlotz, and Akshay Chaudhari.  Adapting pretrained vision-language foundational models to medical imaging domains.  arXiv preprint arXiv:2210.04133, 2022.
  * [143] Charley Gros, Andreanne Lemay, Olivier Vincent, Lucas Rouhier, Marie-Helene Bourget, Anthime Bucquet, Joseph Paul Cohen, and Julien Cohen-Adad.  ivadomed: A medical imaging deep learning toolbox.  Journal of Open Source Software, 6(58):2868, 2021.