#  PIE: Simulating Disease Progression via Progressive Image Editing

Kaizhao Liang*†, Xu Cao*⊕, Kuei-Da Liao, Tianren Gao, Zhengyu Chen, Tejas Nama
† University of Texas, Austin
⊕ University of Illionis, Urbana Champaign
<EMAIL_ADDRESS>
###### Abstract

The trajectories of disease progression could greatly affect the quality and
efficacy of clinical diagnosis, prognosis, and treatment. However, one major
challenge is the lack of longitudinal medical imaging monitoring of individual
patients over time. To address this issue, we develop a novel framework termed
Progressive Image Editing (PIE) that enables controlled manipulation of
disease-related image features, facilitating precise and realistic disease
progression simulation in imaging space. Specifically, we leverage recent
advancements in text-to-image generative models to simulate disease
progression accurately and personalize it for each patient. We theoretically
analyze the iterative refining process in our framework as a gradient descent
with an exponentially decayed learning rate. To validate our framework, we
conduct experiments in three medical imaging domains. Our results demonstrate
the superiority of PIE over existing methods such as Stable Diffusion Video
and Style-Based Manifold Extrapolation based on CLIP score (Realism) and
Disease Classification Confidence (Alignment). Our user study collected
feedback from 35 veteran physicians to assess the generated progressions.
Remarkably, 76.2% of the feedback agrees with the fidelity of the generated
progressions. To our best knowledge, PIE is the first method to generate
disease progression images without sequential training data. PIE can allow
healthcare providers to model disease imaging trajectories over time, predict
future treatment responses, and improve patient outcomes.

Figure 1: Illustrative examples of disease progression simulation using PIE.
The top progression sequence depicts a patient’s heart increasing in size
(red), indicating Cardiomegaly. The bottom sequence demonstrates the expanding
mass areas (blue) in a patient’s lung, indicating Edema.

##  1 Introduction

Understanding disease progression is of utmost importance in the field of
healthcare and medical research. Disease progression refers to the way a
particular illness evolves and changes over time within an individual’s body.
By studying the progression of diseases, healthcare professionals can develop
effective treatment strategies and interventions. It enables them to predict
the course of the disease, identify potential complications, and adjust
treatment plans accordingly. Additionally, understanding disease progression
helps in the development of new drugs and therapies, as researchers can target
specific stages or mechanisms of the disease. Furthermore, monitoring disease
progression allows healthcare providers to assess the efficacy of treatments,
measure the impact of interventions, and make informed decisions about patient
care. Ultimately, a comprehensive understanding of disease progression is
essential for improving patient outcomes, advancing medical knowledge, and
finding innovative approaches to prevent and treat diseases.

However, simulating disease progression poses a formidable challenge in the
realm of medical research, primarily due to the lack of continuous monitoring
of individual patients over time and high cost to collect longitudinal data
(Sukkar et al., 2012; Wang et al., 2014; Liu et al., 2015; Cook and Bies,
2016; Severson et al., 2020). The intricate and multifaceted dynamics of
disease progression, combined with the lack of comprehensive and continuous
monitoring of individual patients, result in the absence of established
methodologies. Disease progression encompasses intricate spatial changes,
morphological transformations, and functional alterations over time, all of
which are influenced by an array of factors including genetics, environment,
and behavior (Hinrichs et al., 2011; Ray, 2011; Lee et al., 2019). The
integration of multiple data sources and modalities, such as imaging and
clinical reports, is imperative to holistically depict the intricate and
evolving nature of disease progression. Moreover, disease progression exhibits
significant variability and heterogeneity across patients and disease
subtypes, rendering a uniform approach impracticable. Despite the considerable
challenges, simulating disease progression carries substantial implications
for clinical diagnosis, prognostication, and therapeutic intervention (Barrett
et al., 2022). Accurate simulation facilitates a deeper understanding of the
underlying mechanisms driving disease pathogenesis, enables the identification
of optimal treatment strategies, and facilitates the prediction of patient
outcomes. As such, this area of research assumes pivotal importance in
advancing medical science and elevating patient care standards. Consequently,
our primary focus lies in the development of a novel framework for accurate
and individualized simulation of disease progression by effectively
integrating text and image modalities without continuous monitoring of
individual patients over time.

Past disease progression research is grounded on variations of the Hidden
Markov Model (HMM) (Liu et al., 2015) and deep probabilistic models (Alaa and
van der Schaar, 2019). While the disease trajectories in these models are
interpretable thanks to their focus on transitional probabilities between
disease states, they fail to simulate visually comprehensible symptoms.
Moreover, these studies have limitations in terms of their ability to
incorporate textual information and generate individualized predictions based
on individualized conditions, and they can not handle imaging data. This
highlights the need for more advanced and flexible simulation frameworks to
accurately capture the complex and dynamic nature of disease progression
visually. To incorporate textural information into a conditioned simulation of
disease progression, we propose a progressive framework PIE, for disease
progression simulation that combines text and image modalities. Specifically,
we aim to progressively add and subtract disease-related features, controlled
by text, to conditionally progress the disease without significantly altering
the original base image features (see Figure 1). Since our framework is built
based on the invertibility of DDIM (Song et al., 2020a), our theoretical
analysis shows it can be seen as a gradient descent toward the objective
maximum log-likelihood of given text conditioning. The learning rate in this
iterative process is decaying exponentially with each iteration forward, which
means that the algorithm is effectively exploring the solution space while
maintaining a balance between convergence speed and stability. This
theoretical analysis guarantees that our framework is moving the instance
toward the targeted manifold and ensures modification is bounded.

Moreover, we empirically evaluate PIE on three distinct medical imaging
datasets, including Chexpert (Irvin et al., 2019), Diabetic Retinopathy
Detection (CHF, 2015) and ISIC 2018 (Codella et al., 2019). We demonstrate
that our framework leads to more accurate and individualized predictions of
disease progression on these datasets, which can potentially improve clinical
diagnosis, treatment planning, and patient outcomes. We also conduct a user
study with veteran physicians to evaluate the effectiveness of our proposed
framework for disease progression simulation. The study aimed to assess the
accuracy and quality of the simulated disease progressions generated by our
framework and to determine how well they matched real-world standards. The
study involves presenting the physicians with a set of simulated disease
images and progressions and then asking them to assess the accuracy and
quality of each generated image and progression. The highlight from the study
shows $76.2\%$ of the physicians agree with the simulated disease progressions
generated by our framework, indicating they are closely matching physicians’
expectations.

Overall, our framework has the potential to be a valuable tool for medical
research and clinical practice, having the potential to contribute to the
development of more effective treatments and better patient outcomes. Our main
contributions are listed as follows:

  * •

We propose a novel framework PIE, which allows for more precise and
controllable manipulation of disease-related image features and leads to more
accurate and individualized disease progression simulation.

  * •

We provide theoretical evidence that our iterative refinement process is
equivalent to gradient descent with an exponentially decaying learning rate,
which helps to establish a deeper understanding of the underlying mechanism of
our method and provides a basis for further optimization and improvement of
the approach.

  * •

We demonstrate the superior performance of PIE over baselines in disease
progression prediction with three medical domains. The results show that PIE
produces more accurate and high-quality disease progression prediction.

  * •

We also conduct a user study with veteran physicians to evaluate the
effectiveness of our proposed framework for disease progression simulation.
Remarkably, $76.2\%$ of the physicians agree with the simulated disease
progressions generated by our framework, indicating they are closely matching
real-world expectations.

##  2 Background

Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020) are a class
of generative models that use a diffusion process to transform a simple
initial distribution, such as a Gaussian, into the target data distribution.
The model assumes that the data points are generated by iteratively applying a
diffusion process to a set of latent variables $x_{1},\ldots,x_{T}$ in a
sample space $\chi$. At each time step $t$, Gaussian noise is added to the
latent variables $x_{t}$, and the variables are then transformed back to the
original space using a learned invertible transformation function. This
process is repeated for a fixed number of steps to generate a final output.
The latent variable models can be expressed in the following form,

$p_{\theta}(x_{0})=\int p_{\theta}(x_{0:T})dx_{1:T},\ \ \ \ where\ \ \
p_{\theta}(x_{0:T})\prod_{t=1}^{T}p_{\theta}^{(t)}(x_{t-1}|x_{t})$ (1)

Because of a special property of the forward process,

$q(x_{t}|x_{0})=\int
q(x_{1:t}|x_{0})dx_{1:(t-1)}=\mathcal{N}(x_{t};\sqrt{\alpha_{t}}x_{0},(1-\alpha_{t})\cdot\mathbf{I})$
(2)

we can express $x_{t}$ as a linear combination of $x_{0}$ and a noise variable
$\epsilon$, which is the key to enabling the image editing process.

$x_{t}=\sqrt{\alpha_{t}}\cdot x_{0}+\sqrt{1-\alpha_{t}}\cdot\epsilon,\ \ \ \
where\ \ \ \ \epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (3)

Denoising Diffusion Implicit Models (DDIM) (Song et al., 2020a) that uses a
non-Markovian forward process to generate data. Unlike Denoising Diffusion
Probabilistic Models (DDPM), DDIM does not require explicit modeling of the
latent variables. Instead, the model generates samples by solving a non-linear
differential equation, which defines a continuous-time evolution of the data
distribution. We can express its forward process as follows,

$q_{\sigma}(x_{1:T}|x_{0}):=q_{\sigma}(x_{T}|x_{0})\prod_{t=2}^{T}q_{\sigma}(x_{t-1}|x_{t},x_{0})$
(4)

where
$q_{\sigma}(x_{t-1}|x_{t},x_{0})=\mathcal{N}(\sqrt{\alpha_{T}}x_{0},(1-\alpha_{T}),\mathbf{I})$
and for all $t>1$

$q_{\sigma}(x_{t-1}|x_{t},x_{0})=\mathcal{N}(\sqrt{\alpha_{t-1}x_{0}}+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\cdot\frac{x_{t}-\sqrt{\alpha_{t}}x_{0}}{\sqrt{1-\alpha_{t}}},\sigma_{t}^{2}\mathbf{I})$
(5)

Setting $\sigma_{t}=0$, it defines a generation process going from $x_{t}$ to
$x_{t-1}$ as follows

$x_{t-1}=\sqrt{\alpha_{t-1}}(\frac{x_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}^{(t)}(x_{t})}{\sqrt{\alpha_{t}}})+\sqrt{1-\alpha_{t-1}}\cdot\epsilon^{(t)}_{\theta}(x_{t})$
(6)

where the $\epsilon_{\theta}^{(t)}(x_{t})$ is a model that attempts to predict
$\epsilon_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ from $x_{t}$

Guided Image Editing via Diffusion (Meng et al., 2021; Lugmayr et al., 2022;
Couairon et al., 2022) are general techniques that aim to modify either the
entirety or certain parts of images through diffusion. The process commences
by introducing noise to the genuine base image and subsequently applying the
reverse diffusion process to remove the noise, resulting in the production of
photo-realistic images. Specifically, we employ the DDIM forward process 3 for
the addition of noise and the reverse process 6 for one cycle of image
modification, as demonstrated in 1.

Figure 2: Overview of the PIE pipeline. PIE is illustrated using an example of
disease progression editing X-ray from a healthy state to cardiomegaly. For
any given step $n$ in PIE, we first utilize DDIM inversion to procure an
inverted noise map. Subsequently, we denoise it using clinical reports imbued
with progressive cardiomegaly information. The output of DDIM denoising serves
as the input for step $n+1$, thus ensuring a gradual and controllable disease
progression simulation. After simulating $N$ steps, the image is converged to
the final state.

##  3 Progressive Image Editing

Progressive image editing (PIE) is a novel framework proposed to refine and
enhance images in an iterative and discrete manner, allowing the use of
additional prompts for small and precise adjustments to simulate semantic
modification while keeping realism. Unlike traditional image editing
techniques, PIE involves a multi-stage process where each step builds upon the
previous one, with the aim of achieving a final result that is more refined
and smooth than if all changes were made at once. The approach also enables
precise control over specific semantic features of the image to be adjusted
without significant impacts on other regions. The main purpose of PIE is to
simulate disease progression from multi-modal input data.

Procedure. The inputs to PIE are a medical image $x_{0}^{(0)}$ depicting any
start or middle stage of a disease and a corresponded clinical report $y$ as
the text conditioning (Rombach et al., 2022), which could either be real or
simulated, providing the potential trajectory of the patient’s disease
progression. The output generated is a sequence of images,
$\\{x_{0}^{(0)},x_{0}^{(1)},...,x_{0}^{(N)}\\}$, illustrating the progression
of the disease as per the input report. The iterative PIE procedure is defined
as follows:

###### Proposition 1

Let $x_{0}^{(N)}\sim\chi$, where $\chi$ is distribution of photo-realistic
images , $y$ be the text conditioning, running $PIE^{(n)}(\cdot,\cdot)$
recursively is denoted as following, where $N\geq n\geq 1$,

$x_{0}^{(n)}=PIE^{(n)}(x_{0}^{(n-1)},y)$ (7)

Then, the resulting output $x_{0}^{(N)}$ maximizes the posterior probability
$p(x_{0}^{(N)}|\ y)$.

With each round of editing as shown in Figure 2, the image gets closer to the
objective by moving in the direction of $-\nabla\log p(x|y)$. Due to the
properties of DDIM, the step size would gradually decrease with a constant
factor. Additional and more detailed proofs will be available in Supplementary
A.

Detailed algorithm. We present the single-step PIE in Algorithm 1. For
$\textit{PIE}^{(n)}$, we define $\alpha_{k}$ according to the DDIM case. Line
8 in Algorithm 1 ensures progressive and limited modifications between the
original input image $x_{0}^{(0)}$, the single-step edited output
$x^{\prime}$, and the region guide selector $M_{ROI}$ through the utilization
of interpolation average parameters $\beta_{1}$ and $\beta_{2}$. These
parameters dictate the modification ratio between the ROI mask-guided space
and the original input space. As $\beta_{1}$ increases, the multi-step editing
process becomes smoother, though it may sacrifice some degree of realism.

Medical heuristic guidance. PIE can be particularly useful for disease
progressions, where medical images such as X-rays or Retinopathy need to be
edited to track the progression of a disease over time. During the simulation,
the region guide masks play a big role as prior information. Unlike other
randomly inpainting tasks (Lugmayr et al., 2022), ROI mask for medical imaging
can be extracted from real or synthetic clinical reports Boag et al. (2020);
Lovelace and Mortazavi (2020) using domain-specific Segment Anything models
(Kirillov et al., 2023; Ma and Wang, 2023). It helps keep unrelated regions
consistent through the progressive changes using PIE or baseline models. In
order to generate sequential disease imaging data, PIE uses noise strength
$\gamma$ to control the influence from the patient’s clinically reported and
expected treatment regimen at time $n$. $N$ is used to control the duration of
the disease occurrence or treatment regimen. PIE allows the user to make such
controls over the iterative process, and running $\textit{PIE}^{(n)}$ multiple
times can improve the accuracy of disease imaging tracking and reduce the
likelihood of missed or misinterpreted changes. Related ablation study for
$M_{ROI}$, $\gamma$, $N$, $\beta_{1}$, $\beta_{2}$ will be available in
Supplementary D.

Baselines To our knowledge, there are no existing image editing models
specifically designed for simulating disease progression. To underscore the
unique strengths of PIE, we compare it against two of the most promising
state-of-the-art methods Stable Diffusion Walk (SD Walk) (Raw, 2022) and
Style-Based Manifold Extrapolation (Extrapolation) (Han et al., 2022) for
generating progressive images as our baselines. During the comparison, all
baseline methods are using the same Stable Diffusion weights and also applied
$M_{ROI}$ for region guided. More details about the baseline models are
available in Supplementary B.

Input: Original input image $x_{0}^{(0)}$ at the start point, input image
$x_{0}^{(n-1)}$ at stage n, number of diffusion steps $T$, text conditional
vector $y$, noise strength $\gamma$, stable diffusion parameterized denoiser
$\epsilon_{\theta}$, a ROI mask $M_{ROI}$, $M_{ROI}^{i,j}\in[0,1]$

1

Output: Modified image $x^{\prime}$ as $x_{0}^{n}$

2 $x^{\prime}\leftarrow x_{0}^{(n-1)}$

3$k\leftarrow\gamma\cdot T$

4$\epsilon\sim\mathcal{N}(0,\mathcal{I})$

5$x^{\prime}\leftarrow\sqrt{\alpha_{k}}\cdot
x^{\prime}+\sqrt{1-\alpha_{k}}\cdot\epsilon$

6for  _$t=k$ to $1$_ do

7
$x^{\prime}\leftarrow\sqrt{\alpha_{t-1}}(\frac{x^{\prime}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}^{(t)}(x^{\prime},y)}{\sqrt{\alpha_{t}}})+\sqrt{1-\alpha_{t-1}}\cdot\epsilon_{\theta}^{(t)}(x^{\prime},y)$

8 end for

9

10$x^{\prime}\leftarrow(\beta_{1}\cdot(x^{\prime}-x_{0}^{(0)})+x_{0}^{(0)})\cdot(1-M_{ROI})+(\beta_{2}\cdot(x^{\prime}-x_{0}^{(0)})+x_{0}^{(0)})\cdot
M_{ROI}$

return $x^{\prime}$ as $x^{(n)}_{0}$

Algorithm 1 Progressive Image Editing $n$-th step ($\textit{PIE}^{(n)}$)

##  4 Experiment

In this section, we present experiments on various disease progression tasks.
Experiments results demonstrate that PIE can simulate the disease-changing
trajectory that is influenced by different medical conditions. Notably, PIE
also preserves unrelated visual features from the original medical imaging
report, even as it progressively edits the disease representation. Figure 3
showcases a set of disease progression simulation examples across three
distinct types of medical imaging. Details for Stable Diffusion fine-tuning,
pretraining model for confidence metrics settings will be available in
Supplementary C.

Table 1: Comparisons with multi-step editing simulations. The backbone of PIE and baseline approaches are Stable Diffusion with the same pre-trained weight. Method | Chest X-ray | Retinopathy | Skin Lesion Image
---|---|---|---
Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$
PIE | 0.690 | 0.96 | 0.807 | 0.99 | 0.453 | 0.95
SD Walk | 0.389 | 0.92 | 0.121 | 0.89 | 0.201 | 0.88
Extrapolation | 0.0543 | 0.97 | 0.0742 | 0.99 | 0.226 | 0.95

###  4.1 Experimental Setups

|
---
 Chest X-ray
 Retinopathy
 Skin Input Image Step 1 Step 4 Step 10
Figure 3: Disease Progression Simulation of PIE. The top progression is for
Cardiomegarly. The middle progression is for Diabetic Retinopathy. The bottom
progression is for Melanocytic Nevus.

Datasets for Disease Progression. We validate the disease progression analysis
through end-to-end medical domain-specific image inference. Specifically, we
evaluate the pretrained domain-specific stable diffusion model on three
different types of disease datasets in classification tasks: CheXpert for
chest X-ray classification (Irvin et al., 2019), ISIC 2018 / HAM10000 (Codella
et al., 2019; Tschandl et al., 2018) for skin cancer prediction, and Kaggle
Diabetic Retinopathy Detection Challenge (CHF, 2015). Each of these datasets
presents unique challenges and differ in scale, making them suitable for
testing the robustness and versatility of PIE. We also collected over 30
healthy data among the test set from these datasets. These data were used for
disease progression simulation. Three groups of progression visualization
results can be found in Figure 3.

Evaluation Metrics. The assessment of generated disease progression images
relies on two crucial aspects: alignment to edited disease feature and subject
fidelity. To measure these characteristics, we utilize two primary metrics:
the CLIP-I score and the classification confidence score. The CLIP-I score
represents the average pairwise cosine similarity between the CLIP embeddings
of generated and real images (Radford et al., 2021; Ruiz et al., 2022). The
classification confidence score is determined using supervised train deep
networks for binary classification between negative (healthy) and positive
(disease) samples. It is denoted as $\textbf{Conf}=Sigmoid(f_{\theta}(x))$ and
represent whether the simulation results are aligned to target disease. In our
experiments, we train the DeepAUC maximization method (Yuan et al., 2021)
(state-of-the-art of Chexpert and ISIC 2018 task 3) using DenseNet121 (Huang
et al., 2017) as the backbone to compute the classification confidence score.

Input Image Input Prompt PIE SD Walk Extrapolation
Step 1 Step 2 Step 4 Step 10
---
Figure 4: Using PIE, SD Walk, Extrapolation to simulate Edema progression with
clinical reports as input prompt.

###  4.2 Progression Simulation Comparison

Table 2: In order to quantitatively analyze the responses of experienced physicians, we consider each pathology class as independent and calculate the precision, recall, and F1 score across all pathologies and physicians. Data | Precision | Recall | F1
---|---|---|---
Real | 0.505 | 0.415 | 0.455
PIE | 0.468 | 0.662 | 0.549

In order to demonstrate the superior performance of PIE in disease progression
simulation over other single-step editing methods, we perform experiments on
six different diseases using the datasets previously mentioned. For each
disease, we used 10 healthy samples in the test set as simulation start point
and run PIE, SD Walk, Extrapolation with 5 random seeds. We obtain at least 50
disease imaging trajectories for each patient. Table 1 showcases that PIE
consistently surpasses both SD Walk and Extrapolation in terms of disease
confidence scores while maintaining high CLIP-I scores. For Chexpert dataset,
the 0.690 final confidence score is the average score among 5 classes. For
Diabetic Retinopathy and HAM10000 datasets, we compare PIE with SD Walk,
Extrapolation for editing image to the most common seen class since these
datasets are highly imbalanced. Figure 5 illustrates the evolution of disease
confidence scores during the progression simulation in each step. We observe
that PIE is able to produce more faithful and realistic progressive editing
compared to the other two baselines. Interestingly, while the CLIP-I score of
Extrapolation is comparable to that of PIE, it fails to effectively edit the
key disease features of the input images as its confidence scores are low
throughout and at the end of the progression.

Figure 4 showcases a group of progression simulation results for Edema in
chest X-rays with the input prompt, "Edema lung finding is shown in the chest
cavity." It is evident from our observations that while SD Walk can
significantly alter the input image in the initial step, it fails to identify
the proper direction of progression in the manifold after a few steps and
would easily create uncontrollable noise. Conversely, Extrapolation only
brightens the Chest X-ray without making substantial modifications. PIE, on
the other hand, not only convincingly simulates the disease trajectory but
also manages to preserve unrelated visual features from the original medical
imaging. Further visual comparisons and analysis are presented in
Supplementary D.

Cardiomegaly

Edema

Pleural Effusion

Consolidation

Diabetic Retinopathy

Melanocytic Nevi

Figure 5: PIE excels in comparison to all the baseline methods across six
different disease progression simulations. The inputs utilized are genuine
healthy images from the test sets. For each image, we apply five random seeds
to simulate disease progression over ten steps. The confidence score, a value
that ranges from 0 to 1, signifies the classification confidence for a
specific disease. Figure 6: Distribution of probability of agreement to the
generated progressions among 35 veteran physicians.

###  4.3 User Study

To further assess the quality of our generated images, we ask 35 veteran
physicians with _$14.4$_ years of experience on average to answer a
questionnaire on chest X-rays. The questionnaire includes disease
classifications on the generated and real X-ray images as well as evaluations
of the realism of generated disease progression sequences of Cardiomegaly,
Edema, and Pleural Effusion. More details of the questionnaire are presented
in Supplementary E.1. The participating physicians have agreed with a
probability of $\mathbf{76.2}\%$ that the simulated progressions on the
targeted diseases fit their expectations.

Table 2 provides an interesting insight into experienced physicians’
performance in predicting the pathology on real and generated X-rays. Details
of the calculations of the statistics could be found in Supplementary E.2.
Surprisingly, we find users’ performance on generated X-rays is superior to
their performance on real images, with substantially higher recall and F1. In
addition, the statistical test suggests that the F1 scores of generated scans
are significantly higher (p-value of $0.0038$) than the real scans. One
plausible explanation is due to the nature of PIE, the result of running
progressive image editing makes pathological features more evident. The
superiority of PIE is consistent with the $76\%$ probability that the
participating physicians agree the generated sequences correspond to the
disease progressions as shown in Figure 6. The aggregated results from the
user study demonstrate our framework’s ability to simulate disease progression
to meet real-world standards.

##  5 Related Works

Disease Progression Simulation Longitudinal disease progression data derived
from individual electronic health records offer an exciting avenue to
investigate the nuanced differences in the progression of diseases over time
(Schulam and Arora, 2016; Stankeviciute et al., 2021; Chen et al., 2022;
Mikhael et al., 2023). Most of the previous works are based on HMM (Wang et
al., 2014; Liu et al., 2015; Alaa et al., 2017) and deep probabilistic models
(Alaa and van der Schaar, 2019). Some recent works start to resolve disease
progression simulation by using diffusion probabilistic models. (Raj et al.,
2015) utilized a diffusion model to predict the progression of Alzheimer’s
disease based on changes in brain metabolism over time, demonstrating high
accuracy in predicting future disease status. (Wu et al., 2017) used a similar
approach to predict the progression of Huntington’s disease based on patterns
of brain atrophy observed in MRI scans. However, all these methods struggle to
handle complex modalities such as imaging data and fail to address
personalized healthcare. Moreover, disease progression modeling methods often
require plenty of time-series data in treatment effect estimation to train,
and the lack of such time-series data, in reality, poses a significant
challenge for disease progression simulation (Xue et al., 2020; Chen, 2022;
Berrevoets et al., 2023).

Generative Adversarial Networks Generative models like Variational
Autoencoders (VAEs) (Kingma and Welling, 2013) and Generative Adversarial
Networks (GANs) (Goodfellow et al., 2020) have been widely employed in medical
imaging applications (Nie et al., 2017; Isola et al., 2017; Cao et al., 2020).
GAN-based image editing models, where they edit the image by manipulating
latent vectors, typically focus on a single-step editing task, such as style
transfer (Fetty et al., 2020) based on StyleGAN (Karras et al., 2019, 2020).
Recent GAN models (Kang et al., 2023; Patashnik et al., 2021) have harnessed
the power of CLIP (Radford et al., 2021) embedding to guide image editing
based on contextual prompts. However, GAN-based models are unstable and
difficult to optimize in general.

Denoising Diffusion Models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song
et al., 2020a; Rombach et al., 2022; Karras et al., 2022) have become
increasingly popular in recent years due to their ability to create photo-
realistic images from textual descriptions. These models can be used for a
wide range of applications, from generating artwork (Rombach et al., 2022;
Saharia et al., 2022) to creating simulations of the physical world (Luo and
Hu, 2021; Poole et al., 2022; Tevet et al., 2022). One major advantage of
these models is their ability to learn from large-scale datasets. Among the
various text-to-image models, Stable Diffusion (Rombach et al., 2022) has
received considerable attention because of its impressive performance in
generating high-quality images and its relatively low cost to fine-tune. Its
denoising process works similarly to the diffusion models but in a latent
space, and this process results in a final image that is highly consistent
with the input text, making it an excellent tool for text-guided image
editing. Diffusion models can also be effortlessly incorporated into an image-
to-image editing pipeline (Brooks et al., 2022; Parmar et al., 2023; Orgad et
al., 2023), thus providing users the ability to edit scenarios across multiple
modalities and assess potential imaging progressive editing paths. However,
existing image-to-image methods can only be used for single-step editing,
which is difficult to simulate time-series progression data.

##  6 Conclusion

In conclusion, our proposed framework, Progressive Image Editing (PIE), holds
great potentials as a tool for medical research and clinical practice in
simulating disease progression. By leveraging recent advancements in text-to-
image generative models and fine-tuned Stable Diffusion, PIE achieves high
fidelity and personalized disease progression simulations. The theoretical
analysis shows that the iterative refining process is equivalent to gradient
descent with an exponentially decayed learning rate, and practical experiments
on three medical imaging datasets demonstrate that PIE surpasses baseline
methods, such as SD Walk and Extrapolation, in several quantitative metrics.
Furthermore, a user study conducted with veteran physicians confirms that the
simulated disease progressions generated by PIE meet real-world standards. To
the best of our knowledge, our approach is the first of its kind to offer
clinically meaningful disease progression simulations using medical imaging
data in an iterative image editing context. Despite current limitations due to
the lack of continuous monitoring data and detailed medical reports, our
framework has vast potential in modeling disease trajectories over time,
restoring missing data from previous treatments, predicting future treatment
responses, and ultimately improving patient outcomes. Moving forward, we aim
to incorporate more data with richer descriptions and different monitoring
modalities, such as chemical biomarkers and physiological recordings, into
fine-tuning Stable Diffusion, enabling our framework to more precise control
over disease simulation through text conditioning.

## References

  * CHF [2015] Diabetic Retinopathy Detection, howpublished= https://www.kaggle.com/c/diabetic-retinopathy-detection, 2015.
  * Alaa and van der Schaar [2019] Ahmed M Alaa and Mihaela van der Schaar.  Attentive state-space modeling of disease progression.  _Advances in neural information processing systems_ , 32, 2019.
  * Alaa et al. [2017] Ahmed M Alaa, Scott Hu, and Mihaela Schaar.  Learning from clinical judgments: Semi-markov-modulated marked hawkes processes for risk prognosis.  In _International Conference on Machine Learning_ , pages 60–69. PMLR, 2017.
  * Barrett et al. [2022] Jeffrey S Barrett, Tim Nicholas, Karim Azer, and Brian W Corrigan.  Role of disease progression models in drug development.  _Pharmaceutical Research_ , 39(8):1803–1815, 2022.
  * Berrevoets et al. [2023] Jeroen Berrevoets, Fergus Imrie, Trent Kyono, James Jordon, and Mihaela van der Schaar.  To impute or not to impute? missing data in treatment effect estimation.  In _International Conference on Artificial Intelligence and Statistics_ , pages 3568–3590. PMLR, 2023.
  * Boag et al. [2020] William Boag, Tzu-Ming Harry Hsu, Matthew McDermott, Gabriela Berner, Emily Alesentzer, and Peter Szolovits.  Baselines for chest x-ray report generation.  In _Machine learning for health workshop_ , pages 126–140. PMLR, 2020.
  * Brooks et al. [2022] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  _arXiv preprint arXiv:2211.09800_ , 2022.
  * Cao et al. [2020] Bing Cao, Han Zhang, Nannan Wang, Xinbo Gao, and Dinggang Shen.  Auto-gan: self-supervised collaborative learning for medical image synthesis.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 10486–10493, 2020.
  * Chen [2022] Irene Y Chen.  _Machine Learning Approaches for Equitable Healthcare_.  PhD thesis, Massachusetts Institute of Technology, 2022.
  * Chen et al. [2022] Irene Y Chen, Rahul G Krishnan, and David Sontag.  Clustering interval-censored time-series for disease phenotyping.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 6211–6221, 2022.
  * Codella et al. [2019] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  _arXiv preprint arXiv:1902.03368_ , 2019.
  * Cook and Bies [2016] Sarah F Cook and Robert R Bies.  Disease progression modeling: key concepts and recent developments.  _Current pharmacology reports_ , 2:221–230, 2016.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _arXiv preprint arXiv:2210.11427_ , 2022.
  * Fetty et al. [2020] Lukas Fetty, Mikael Bylund, Peter Kuess, Gerd Heilemann, Tufve Nyholm, Dietmar Georg, and Tommy Löfstedt.  Latent space manipulation for high-resolution medical image synthesis via the stylegan.  _Zeitschrift für Medizinische Physik_ , 30(4):305–314, 2020.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Han et al. [2022] Tianyu Han, Jakob Nikolas Kather, Federico Pedersoli, Markus Zimmermann, Sebastian Keil, Maximilian Schulze-Hagen, Marc Terwoelbeck, Peter Isfort, Christoph Haarburger, Fabian Kiessling, et al.  Image prediction of disease progression for osteoarthritis by style-based manifold extrapolation.  _Nature Machine Intelligence_ , pages 1–11, 2022.
  * Hinrichs et al. [2011] Chris Hinrichs, Vikas Singh, Guofan Xu, Sterling C Johnson, Alzheimers Disease Neuroimaging Initiative, et al.  Predictive markers for ad in a multi-modality framework: an analysis of mci progression in the adni population.  _Neuroimage_ , 55(2):574–589, 2011.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Irvin et al. [2019] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 590–597, 2019.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1125–1134, 2017.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  _arXiv preprint arXiv:2303.05511_ , 2023.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _arXiv preprint arXiv:2206.00364_ , 2022.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Lee et al. [2019] Garam Lee, Kwangsik Nho, Byungkon Kang, Kyung-Ah Sohn, and Dokyoon Kim.  Predicting alzheimer’s disease progression using multi-modal deep learning approach.  _Scientific reports_ , 9(1):1952, 2019.
  * Liu et al. [2015] Yu-Ying Liu, Shuang Li, Fuxin Li, Le Song, and James M Rehg.  Efficient learning of continuous-time hidden markov models for disease progression.  _Advances in neural information processing systems_ , 28, 2015.
  * Loshchilov and Hutter [2016] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lovelace and Mortazavi [2020] Justin Lovelace and Bobak Mortazavi.  Learning to generate clinically coherent chest x-ray reports.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1235–1243, 2020.
  * Lugmayr et al. [2022] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11461–11471, 2022.
  * Luo and Hu [2021] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2837–2845, 2021.
  * Ma and Wang [2023] Jun Ma and Bo Wang.  Segment anything in medical images.  _arXiv preprint arXiv:2304.12306_ , 2023.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.
  * Mikhael et al. [2023] Peter G Mikhael, Jeremy Wohlwend, Adam Yala, Ludvig Karstens, Justin Xiang, Angelo K Takigami, Patrick P Bourgouin, PuiYee Chan, Sofiane Mrah, Wael Amayri, et al.  Sybil: a validated deep learning model to predict future lung cancer risk from a single low-dose chest computed tomography.  _Journal of Clinical Oncology_ , pages JCO–22, 2023.
  * Nie et al. [2017] Dong Nie, Roger Trullo, Jun Lian, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen.  Medical image synthesis with context-aware generative adversarial networks.  In _Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20_ , pages 417–425. Springer, 2017.
  * Orgad et al. [2023] Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov.  Editing implicit assumptions in text-to-image diffusion models.  _arXiv preprint arXiv:2303.08084_ , 2023.
  * Parmar et al. [2023] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  _arXiv preprint arXiv:2302.03027_ , 2023.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2085–2094, October 2021.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raj et al. [2015] Ashish Raj, Eve LoCastro, Amy Kuceyeski, Duygu Tosun, Norman Relkin, Michael Weiner, Alzheimer’s Disease Neuroimaging Initiative (ADNI, et al.  Network diffusion model of progression predicts longitudinal patterns of atrophy and metabolism in alzheimer’s disease.  _Cell reports_ , 10(3):359–369, 2015.
  * Raw [2022] Nathan Raw.  Stable diffusion videos.  https://github.com/nateraw/stable-diffusion-videos, 2022.
  * Ray [2011] Pritha Ray.  Multimodality molecular imaging of disease progression in living subjects.  _Journal of biosciences_ , 36:499–504, 2011.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _arXiv preprint arXiv:2208.12242_ , 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schulam and Arora [2016] Peter Schulam and Raman Arora.  Disease trajectory maps.  _Advances in neural information processing systems_ , 29, 2016.
  * Severson et al. [2020] Kristen A Severson, Lana M Chahine, Luba Smolensky, Kenney Ng, Jianying Hu, and Soumya Ghosh.  Personalized input-output hidden markov models for disease progression modeling.  In _Machine Learning for Healthcare Conference_ , pages 309–330. PMLR, 2020.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Stankeviciute et al. [2021] Kamile Stankeviciute, Ahmed M Alaa, and Mihaela van der Schaar.  Conformal time-series forecasting.  _Advances in Neural Information Processing Systems_ , 34:6216–6228, 2021.
  * Sukkar et al. [2012] Rafid Sukkar, Elyse Katz, Yanwei Zhang, David Raunig, and Bradley T Wyman.  Disease progression modeling using hidden markov models.  In _2012 annual international conference of the IEEE engineering in medicine and biology society_ , pages 2845–2848. IEEE, 2012.
  * Tevet et al. [2022] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022.
  * Tschandl et al. [2018] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  _Scientific data_ , 5(1):1–9, 2018.
  * Wang et al. [2014] Xiang Wang, David Sontag, and Fei Wang.  Unsupervised learning of disease progression models.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 85–94, 2014.
  * Wu et al. [2017] Dan Wu, Andreia V Faria, Laurent Younes, Susumu Mori, Timothy Brown, Hans Johnson, Jane S Paulsen, Christopher A Ross, Michael I Miller, PREDICT-HD Investigators, and Coordinators of the Huntington Study Group.  Mapping the order and pattern of brain structural mri changes using change-point analysis in premanifest huntington’s disease.  _Human brain mapping_ , 38(10):5035–5050, 2017.
  * Xue et al. [2020] Yuan Xue, Nan Du, Anne Mottram, Martin Seneviratne, and Andrew M Dai.  Learning to select best forecast tasks for clinical outcome prediction.  _Advances in Neural Information Processing Systems_ , 33:15031–15041, 2020.
  * Yuan et al. [2021] Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang.  Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3040–3049, 2021.

##  Appendix A Theoretical Analysis

###  A.1 Proof of Proposition 1

In this proof, we follow the conventions and definitions in Song et al.
[2020a]

$x_{t-1}=\sqrt{\alpha_{t-1}}(\frac{x_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}^{(t)}(x_{t},y)}{\sqrt{\alpha_{t}}})+\sqrt{1-\alpha_{t-1}}\cdot\epsilon_{\theta}^{(t)}(x_{t},y)$
(8)

Now given a base image denoted as $\mathbf{x^{(0)}_{0}}$, we wish to perform
diffusion-based editing recursively for $N$ times. The roll-back (to the $k$
th-steps, where $k\geq 1$) according to (3):

$x^{(n)}_{k}=\sqrt{\alpha_{k}}\cdot
x^{(n-1)}_{0}+\sqrt{1-\alpha_{k}}\cdot\epsilon$ (9)

where $\epsilon\sim\mathcal{N}(0,\mathcal{I})$. Plugging (9) into (8),

$\begin{split}x_{k-1}^{(n)}&=\sqrt{\alpha_{k-1}}(\frac{x_{k}^{(n)}-\sqrt{1-\alpha_{k}}\epsilon_{\theta}^{(k)}(x_{k}^{(n)},y)}{\sqrt{\alpha_{k}}})+\sqrt{1-\alpha_{k-1}}\cdot\epsilon_{\theta}^{(k)}(x_{k}^{(n)},y)\\\
&=\sqrt{\alpha_{k-1}}\cdot
x_{0}^{(n-1)}+\sqrt{\frac{\alpha_{k-1}(1-\alpha_{k})}{\alpha_{k}}}\cdot(\epsilon-\epsilon_{\theta}^{(k)}(x_{k}^{(n)},y))+\sqrt{1-\alpha_{k-1}}\cdot\epsilon_{\theta}^{(k)}(x_{k}^{(n)},y)\end{split}$
(10)

Setting $k=1$ to perform the revert diffusion process

$\begin{split}x_{0}^{(n)}&=\sqrt{\alpha_{0}}\cdot
x_{0}^{(n-1)}+\sqrt{\frac{\alpha_{0}(1-\alpha_{1})}{\alpha_{1}}}\cdot(\epsilon-\epsilon_{\theta}^{(1)}(x_{1}^{(n)},y))+\sqrt{1-\alpha_{0}}\cdot\epsilon_{\theta}^{(1)}(x_{1}^{(n)},y)\end{split}$
(11)

Unrolling this recursion in (11), we have

$\begin{gathered}\begin{aligned} x_{0}^{(N)}=(\sqrt{\alpha_{0}})^{N}\cdot
x_{0}^{(0)}+\sqrt{\frac{\alpha_{0}(1-\alpha_{1})}{\alpha_{1}}}\cdot\sum_{i}^{N}(\sqrt{\alpha_{0}})^{i}\cdot\epsilon\\\
+\frac{\sqrt{\alpha_{1}-\alpha_{0}\alpha_{1}}-\sqrt{\alpha_{0}-\alpha_{0}\alpha_{1}}}{\sqrt{\alpha_{1}}}\cdot\sum_{i}^{N}(\sqrt{\alpha_{0}})^{i}\cdot\epsilon^{(i)}_{\theta}(x_{1}^{{(i)}},y)\end{aligned}\end{gathered}$
(12)

Typically, $\alpha_{0}$ is set to a number close to but less than 1, where in
the case of stable diffusion $0.9999$, $\alpha_{1}$ is $0.9995$, assuming $50$
step schedule.

The second term in (12) is a sampling from Gaussian distributions with
geometrically decreasing variances.

$\begin{split}\lim_{N\rightarrow\infty}\sum_{i}^{N}(\sqrt{\alpha_{0}})^{i}\cdot\epsilon=0\end{split}$
(13)

Given a large enough N,

$\begin{split}x_{0}^{(N)}\approx(\sqrt{\alpha_{0}})^{N}\cdot
x_{0}^{(0)}+\frac{\sqrt{\alpha_{1}-\alpha_{0}\alpha_{1}}-\sqrt{\alpha_{0}-\alpha_{0}\alpha_{1}}}{\sqrt{\alpha_{1}}}\cdot\sum_{i}^{N}(\sqrt{\alpha_{0}})^{i}\cdot\epsilon^{(i)}_{\theta}(x_{1}^{{(i)}},y)\end{split}$
(14)

Proposition 1 in Song et al. [2020a] declares "optimal
$\epsilon^{(i)}_{\theta}$ has an equivalent probability flow ODE corresponding
to the “Variance-Exploding” SDE in Song et al. [2020b]". Hence,
$\epsilon^{(i)}_{\theta}(x_{1}^{{(i)}},y):=\nabla_{x}\log p(x|y)$. This can be
seen as gradient descent with a geometrically decaying learning rate with a
factor of $\sqrt{\alpha_{0}}$, with "base learning rate"
$\frac{\sqrt{\alpha_{1}-\alpha_{0}\alpha_{1}}-\sqrt{\alpha_{0}-\alpha_{0}\alpha_{1}}}{\sqrt{\alpha_{1}}}$.

Notice that in (14), there is a decaying factor on the initial image
$x_{0}^{(0)}$, as N grows the original image will surely diminish. Therefore,
some other empirical measures are required to preserve the structure of the
image, such as segmentation masking, edit strength scheduling and etc.

###  A.2 Additional Theoretical Analysis

###### Proposition 2

Assuming $\|x_{0}^{(0)}\|\leq C_{1}$ and $\|\epsilon_{\theta}(x,y)\|\leq
C_{2}$, $(x,y)\in(\chi,\Gamma)$, for any $\delta>0$, if

$n>\frac{2}{\log(\alpha_{0})}\cdot(log(\delta)-C)$ (15)

then,

$\|x^{(n+1)}_{0}-x^{(n)}_{0}\|<\delta$ (16)

where,
$\lambda=\frac{\sqrt{\alpha_{0}-\alpha_{0}\alpha_{1}}-\sqrt{\alpha_{1}-\alpha_{0}\alpha_{1}}}{\sqrt{\alpha_{1}}}$,
$\chi$ is the image distribution, $\Gamma$ is the text condition distribution
, and $C=\log((\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot C_{1}+\lambda\cdot C_{2})$

Proof Continuing on 14, for any $n>1$, we have

$\begin{split}\|x_{0}^{n}-x_{0}^{n-1}\|=\|(\sqrt{\alpha_{0}})^{n}\cdot[(1-\frac{1}{\sqrt{\alpha_{0}}})\cdot
x_{0}^{(0)}-\lambda\cdot\epsilon_{\theta}^{(n)}(x_{1}^{(n)},y)]\|\\\
\leq(\sqrt{\alpha_{0}})^{n}\cdot[\|(1-\frac{1}{\sqrt{\alpha_{0}}})\cdot
x_{0}^{(0)}\|+\lambda\cdot\|\epsilon_{\theta}^{(n)}(x_{1}^{(n)},y)\|]\\\
\leq(\sqrt{\alpha_{0}})^{n}[(\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2}]\end{split}$ (17)

to guarantee $\|x_{0}^{n}-x_{0}^{n-1}\|\leq\delta$, we just need to set,

$\begin{split}(\sqrt{\alpha_{0}})^{n}[(\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2}]<\delta\\\
\frac{n}{2}\log(\alpha_{0})+\log((\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2})<log(\delta)\end{split}$ (18)

hence, given that $\alpha_{0}<1$ and $\log(\alpha_{0})<0$

$\begin{split}n>\frac{2}{\log(\alpha_{0})}\cdot(log(\delta)-\log((\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2}))\end{split}$ (19)

Let $C=\log((\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot C_{1}+\lambda\cdot C_{2})$,
we have

$n>\frac{2}{\log(\alpha_{0})}\cdot(log(\delta)-C)$ (20)

From above, we can conclude that as n grows bigger the changes between steps
would grow smaller. The difference between steps will get arbitrarily small.

###### Proposition 3

For all $N>1$, $\|x_{0}^{(N)}-x_{0}^{(0)}\|$ is bounded by a constant
$\kappa$. where $\kappa>0$

Proof From 17 and applying triangle inequality, we observe that the difference
is a sum of a geometric sequence scaled by a constant factor,

$\begin{split}\|x_{0}^{(N)}-x_{0}^{(0)}\|\leq\sum_{n=1}^{N}\|x_{0}^{(n)}-x_{0}^{(n-1)}\|\\\
\leq\sum_{n=1}^{N}(\sqrt{\alpha_{0}})^{n}[(\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2}]\\\
=\frac{1-(\sqrt{\alpha_{0}})^{N}}{1-\sqrt{\alpha_{0}}}\cdot[(\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2}]\end{split}$ (21)

As N goes to infinity,

$\begin{split}\lim_{N\rightarrow\infty}\|x_{0}^{(N)}-x_{0}^{(0)}\|\leq\frac{1}{1-\sqrt{\alpha_{0}}}\cdot[(\frac{1}{\sqrt{\alpha_{0}}}-1)\cdot
C_{1}+\lambda\cdot C_{2}]=\kappa\end{split}$ (22)

##  Appendix B Baselines

Stable Diffusion Walk (SD Walk) and Style-Based Manifold Extrapolation
(Extrapolation) are two leading techniques in the field of progressive video
generation / medical image editing, each displaying promising results within
specific domains. However, their applicability remains confined to these
particular domains and poses a challenge in extending to the broader scope of
different medical imaging data. To illustrate this, Figure 7 provides a
comparative visualization of single-step editing using these three techniques.

PIE

SD Walk

Extrapolation

Figure 7: Editing path of PIE, SD Walk, and Extrapolation.

Stable Diffusion Walk (SD Walk) [Raw, 2022] control the multi-step denoising
process in the Stable Diffusion Videos pipeline. By smoothly and randomly
traversing through the sampled latent space, SD Walk demonstrates its
capability to generate a series of images that progressively align with a
given text prompt (see Figure B). As the state-of-the-art publicly available
pipeline, SD Walk can generate sequential imaging data by interpolating the
latent space via multi-step Stable Diffusion. Though SD Walk is useful for
general domain [Raw, 2022], it is not controllable for medical prompts.

Style-Based Manifold Extrapolation (Extrapolation) [Han et al., 2022],
involves iteratively modifying images by extrapolating between two latent
manifolds. To determine the directions of latent extrapolation, the nearest
neighbors algorithm is employed on distributions of known trajectories.
However, in cases where progression data is not readily available, as in the
study at hand, the directions are obtained by randomly sampling and computing
the mean of each manifold.

The actual interpolation of Extrapolation for each step can be defined as:

$\Delta=\frac{1}{m}\sum_{m}^{i=1}\frac{\Delta t^{i}}{\Delta
T}[G^{-1}(x_{(0)}^{(n+1)})-G^{-1}(x_{(0)}^{(n)})]$ (23)

where $G^{-1}(x_{(0)}^{(n)})$ is the corresponding latent vector of image at
stage n.

##  Appendix C Experimental Settings

In the following sections, we detail the experimental setup used for training
the Stable Diffusion model with domain-specific medical data. We also outline
the design of our disease progression simulation experiment across three
datasets, as well as provide an evaluation of the time costs.

###  C.1 Implementation Details

Both _PIE_ and the baselines use publicly available Stable Diffusion
checkpoints (CompVis/stable-diffusion-v1-4) that we further fine-tune on the
training sets of each of the target datasets. Our code and checkpoints will be
publicly available upon publication.

Stable Diffusion Training. To fine-tune the Stable Diffusion model, we center-
crop and resize the input images to $512\times 512$ resolution. We utilize the
AdamW optimizer [Loshchilov and Hutter, 2017] with a weight decay set at 0.01.
Additionally, we employ a cosine learning rate scheduler [Loshchilov and
Hutter, 2016], with the base learning rate set at $5\times 10^{-5}$. All
models undergo fine-tuning for 20,000 steps on eight NVIDIA A100 GPUs, with
each GPU handling a batch size of 8.

DeepAUC DenseNet121 Training. We have previously outlined the concept of a
classification confidence score. In order to pre-train the DeepAUC DenseNet121
model, we utilize the original code repository provided by the authors. The
model training process involves the use of an exponential learning rate
scheduler, with the base learning rate set to $1\times 10^{-4}$. Initially,
the model is pre-trained on a multi-class task for each respective dataset.
Subsequently, we employ the AUC loss proposed in the study by [Yuan et al.,
2021] to finetune the binary classification task, distinguishing between
negative (healthy) and positive (disease) samples. The AUC loss finetuning
process involves the use of an exponential learning rate scheduler, with the
base learning rate set to $1\times 10^{-2}$. All models used for calculating
classification confidence score are finetuned for 10 epochs on one NVIDIA A100
GPU, with a batch size of 128. For each class, the final classification
accuracy on the validation set is 95.0 % using finetuned DenseNet121 model.

PIE progression simulation. The PIE progression simulation is tested with one
NVIDIA A100 GPU. In the following sections, we will show the hyper-parameter
search experiment for PIE and explain the insight to adjust them.

Figure 8: ROI masks for three different domains. Note, the white part in the
mask is the disease-related regions.

###  C.2 ROI Mask Generation

The ROI Mask used in experiments are generated by Segment Anything Model (SAM)
Kirillov et al. [2023] and modified to smooth the edge. For different domains,
since the region guide prior is different, the mask shape and size is also
different. Figure 8 showcases examples of ROI masks utilized for simulation in
the PIE and baseline models.

###  C.3 Time Cost Analysis

Given that both PIE and the baseline methods utilize the same Stable Diffusion
backbone Rombach et al. [2022], a comparison of latency among each method is
unnecessary. For simulating disease progression on an image of size $512\times
512$, per step PIE requires approximately 0.078s to generate the subsequent
stage when the strength parameter, $\gamma$, is set to 0.5, batch size is set
to 1 and using one NVIDIA A100 (80GB).

##  Appendix D Ablation Study

To investigate the influence of each hyper-parameter in PIE and analyse the
progression visualization, we conduct several ablation studies for
visualization, failure cases analysis and hyperparameter searching.

 PIE

 SD Walk

Input Image

Step 1

Step 2

Step 4

Step 10

 Extrapolation

Figure 9: Visualization of PIE, SD Walk, Extrapolation to generate disease
progression from Cardiomegaly clinical reports.

 PIE

 Random Walk

Input Image

Step 1

Step 2

Step 4

Step 10

 Interpolation

Figure 10: Visualization of PIE, SD Walk, Extrapolation to generate disease
progression from Diabetic Retinopathy clinical reports.

 PIE

 SD Walk

Input Image

Step 1

Step 2

Step 4

Step 10

 Extrapolation

Figure 11: Visualization of PIE, SD Walk, Extrapolation to generate disease
progression from Melanocytic Nevi clinical reports.

###  D.1 Visualization for Three Medical Imaging Domains

To provide an in-depth understanding of how PIE performs during different
disease progression inference scenarios compared to baseline models, we
present detailed visualizations demonstrating PIE’s advantages. PIE
consistently maintains the realism of the input image even after 10 steps of
progression, excelling in most scenarios. Figure 9 displays a comparison among
three methods simulating Cardiomegaly progression. PIE outperforms both SD
Walk and Extrapolation by expanding the heart without introducing noise after
2 steps. Figure 10 displays a comparison among three methods simulating
Diabetic Retinopathy progression. PIE outperforms both SD Walk and
Extrapolation by adding more bleeding (red) and small blind (white) regions
without introducing noise after 2 steps. Figure 11 displays a comparison among
three methods simulating Melanocytic Nevi progression. PIE outperforms the
other two methods as it keeps the color and shape of the patient’s skin but
continually enhance the feature for Melanocytic Nevi.

SD Walk, while it can interpolate within the prompt latent space, tends to
generate noise. Although it can generate convincing video sequences in the
general domain, it struggles to simulate authentic disease progression. On the
other hand, Extrapolation, despite being effective for bone X-rays, faces
challenges with more complex medical imaging domains like chest X-rays.
Extrapolation’s editing process is considerably slower than the other two
methods, and it fails to be controlled effectively by clinical reports.

Co-occurring diseases. PIE is capable of generating images for co-occurring
diseases, although the performance slightly trails behind that of single
disease generation. Figure 12 illustrates an example of disease progression
simulation for co-occurring Cardiomegaly, Edema, and Pleural Effusion. The
progression for Cardiomegaly and Pleural Effusion escalates rapidly at the
initial stages, whereas Edema develops more slowly. However, after 10 steps,
Edema achieves a high confidence score, indicating successful simulation
despite its slower growth.

Input image

Step 4

Step 10

Confidence Score

Figure 12: PIE can successfully simulate co-occurring diseases progression
(Patent’s clinical report shows high probability to be Cardiomegaly, Edema,
Pleural Effusion at the same time).

###  D.2 Failure Case Analysis

Step n-1

Step n

Step n+1

Figure 13: A failure case of the PIE model in preserving the features of a
pacemaker during the simulation of Cardiomegaly disease progression. Pacemaker
is usually used by patient with severe Cardiomegaly. At Step n-1, the X-ray
displays an electronic line. At Step n, both the electronic line and the
pacemaker are visible. However, by Step n+1, all the medical device features,
including the pacemaker, have vanished from the simulation. It’s important to
note that the input clinical prompt did not contain any information regarding
the pacemaker, making it difficult for the model to retain this crucial
feature. This illustrates the challenges faced by models like PIE in dealing
with significant but unmentioned clinical features in the input data. It
underscores the need for incorporating comprehensive and detailed clinical
data to ensure accurate and realistic disease progression simulations.

Input image

Step 4

Step 10

Confidence Score

Figure 14: Two failure cases of the PIE model in simulating co-occurring
diseases progression for Cardiomegaly, Edema, and Pleural Effusion, only the
features for Pleural Effusion are captured. These failure cases arise from the
issue related to imbalanced label distribution in the training data.
Specifically, the prevalence of Pleural Effusion is significantly higher than
the other four classes, leading to an inherent bias in the model’s simulations
for co-occurring diseases. This imbalance emphasizes the need for a more
diversified and balanced training dataset for more accurate simulation of co-
occurring diseases.

Despite outperforming baseline models, PIE still faces limitations tied to
data sensitivity issues. For instance, imbalances in the distribution of
training data for Stable Diffusion can limit PIE’s capability to edit rare
diseases. In some cases, PIE might generate essential medical device features
but fail to preserve them in subsequent stages of progression simulation, as
observed with features like pacemakers. Figure 13 shows a good example of
pacemaker disappearance during simulation. Besides, the co-occurring diseases
simulation also has some failure cases (see Figure 14).

Fundamentally, these shortcomings could be addressed with a larger and label-
equal distribution dataset. However, given that the volume of medical data is
often smaller than in other domains, it is also important to explore fine-
tuning PIE’s diffusion backbone through few-shot learning under extremely
imbalanced label distribution.

 with ROI mask

 w/o ROI mask

 with ROI mask

 w/o ROI mask

 with ROI mask

 w/o ROI mask

Figure 15: Visualization comparison between ROI mask influence for three
medical imaging domains.

###  D.3 Hyperparameter Search & Analysis

While the Fréchet Inception Distance (FID) and Kernel Inception Distance (KID)
metrics are widely utilized in the evaluation of generative models, they do
not necessarily align with the perceptual quality of images, making them
unsuitable for evaluating progression simulation tasks. Nevertheless, for the
subsequent hyperparameter search and ablation study, we added the results of
KID score, given its ability to provide insight into the diversity and
distributional closeness of the generated images in relation to the real data.

To demonstrate the significance of the ROI mask as a key control factor in the
PIE, an experiment was conducted to compare the performance of models using
and not using ROI masks across three domains, with each model tested using 5
different random seeds. The evaluation focused on the classification
confidence score and CLIP score. Results, as shown in Table 3, revealed that
while removing the ROI mask could sometimes increase the confidence score
(observed for chest X-ray and skin imaging), it consistently led to a decrease
in the CLIP score. Further visual analysis depicted in Figure 15 shows that
the ROI mask crucially helps in preserving the basic shape of the medical
imaging during the PIE process. Consequently, these findings suggest that the
ROI mask, alongside clinical reports, serves as a critical medical prior for
simulating disease progression. It helps the PIE to concentrate on disease-
related regions while maintaining the realism of the input image.

We present a comprehensive examination of various hyperparameters, namely
strength ($\gamma$ in Algorithm 1), step ($N$), $\beta_{1}$, and $\beta_{2}$,
and their respective tradeoffs as demonstrated in tables 4, 5, and 6. Notably,
a discernible tradeoff exists between classification confidence score and
CLIP-I/KID, wherein an increase in classification confidence score results in
a decrease in CLIP-I. In the subsequent section, we offer an intricate
analysis of each hyperparameter.

Table 4 illustrates a positive correlation between the strength of PIE and
classification confidence score, while revealing a negative relationship
between strength and CLIP-I/KID. From an intuitive standpoint, as the strength
of PIE increases, more features are directed to align with the pathologies
within the original images. Consequently, the classifier exhibits greater
confidence in accurately predicting the specific disease class, leading to a
more significant deviation from the initial starting point. As a result, the
classification confidence score value increases while the CLIP-I value
decreases, reflecting the inverse relationship between the two metrics.

Table 5 presents a similar pattern initially. However, both classification
confidence score and CLIP-I/KID reach a state of stability after a certain
number of steps. This observation aligns with our theoretical analysis as
outlined in Proposition 2. Moreover, it can be interpreted as the convergence
of a Cauchy geometric sequence, wherein the discrepancy between successive
steps gradually diminishes as the value of $N$ tends towards infinity.

Lastly, in Table 6, we explore the interplay between $\beta_{1}$ and
$\beta_{2}$, which serve as parameters governing the rate of progression
within and outside the region of interest (ROI) respectively. Our findings
reveal that $\beta_{2}$, responsible for regulating the pace of progression
within the ROI, exerts a more pronounced influence on classification
confidence score, while $\beta_{1}$ exhibits a stronger impact on CLIP-I/KID.
This outcome can be intuitively comprehended, considering that the ROI
typically encompasses a smaller area of paramount importance for aligning with
disease-specific features. Conversely, the areas outside the ROI exert a
significantly greater influence on the realism captured by CLIP-I/KID.

In summary, our study elucidates the inherent trade-offs and offers valuable
practical insights, thereby furnishing meaningful guidance for effectively
utilizing PIE in practice.

Table 3: Mask, w/o mask guidance comparisons. Method | Chest X-ray | Retinopathy | Skin Lesion Image
---|---|---|---
Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$
w/o mask | 0.729 | 0.93 | 0.163 | 0.96 | 0.666 | 0.85
with mask | 0.690 | 0.96 | 0.807 | 0.99 | 0.453 | 0.95
Table 4: Strength $\gamma$ selection for $N=10$. Strength | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | KID $(\downarrow)$
---|---|---|---
0.05 | 0.038 | 0.966 | 0.0685
0.1 | 0.120 | 0.969 | 0.0638
0.2 | 0.273 | 0.969 | 0.0885
0.3 | 0.455 | 0.967 | 0.1033
0.4 | 0.746 | 0.965 | 0.1142
0.5 | 0.977 | 0.962 | 0.1389
0.6 | 0.995 | 0.956 | 0.1549
0.7 | 0.998 | 0.955 | 0.1533
0.8 | 0.999 | 0.951 | 0.1629
Table 5: Simulation steps $N$ selection with $\gamma=0.5$. Step ($N$) | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | KID $(\downarrow)$
---|---|---|---
1 | 0.491 | 0.965 | 0.094
2 | 0.731 | 0.964 | 0.098
5 | 0.881 | 0.963 | 0.121
10 | 0.978 | 0.962 | 0.142
20 | 0.989 | 0.961 | 0.111
50 | 0.975 | 0.962 | 0.130
100 | 0.959 | 0.962 | 0.115
Table 6: Beta selection. $\beta_{1}$ | $\beta_{2}$ | Conf $(\uparrow)$ | CLIP-I $(\uparrow)$ | KID $(\downarrow)$
---|---|---|---|---
0.01 | 1.0 | 0.954 | 0.946 | 0.133
0.01 | 0.75 | 0.977 | 0.948 | 0.140
0.01 | 0.5 | 0.554 | 0.965 | 0.090
0.1 | 1.0 | 0.960 | 0.965 | 0.126
0.1 | 0.75 | 0.976 | 0.962 | 0.140
0.1 | 0.5 | 0.554 | 0.962 | 0.089
0.2 | 1.0 | 0.963 | 0.947 | 0.134
0.2 | 0.75 | 0.977 | 0.964 | 0.137
0.2 | 0.5 | 0.556 | 0.962 | 0.089

##  Appendix E User Study

###  E.1 Questionnair

The questionnaire includes 2 parts. Part one consists of $20$ multiple choices
of single image classifications, $10$ single-step generations, and $10$ real
X-ray images sampled from the training set. Part two consists of $3$ generated
disease progressions consisting of Cardiomegaly, Edema as well Pleural
Effusion. Each progression runs 10 steps. For each single image
classification, we ask "Please determine the pathologies of the following
patient" and let the user pick from 6 options {No findings, Cardiomegaly,
Consolidation, Edema, Effusion, Atelectasis} with possible co-occurrence,
while for each of the $10$-step progressions, we ask "Does the below disease
progression fit your expectation?" and let the user input a binary answer of
yes or no.

Below we include the full instructions we gave for our user study both in
English and Chinese. Here we only include the English version.

  1. 1.

Please read the instruction and inspect the images carefully before answering.

  2. 2.

Please provide your years of experience

  3. 3.

For the first 20 questions, please determine the pathologies from the X-ray
images (you can choose more than one answer). For the last 3 questions, please
answer if the disease progression shown fits your expectation.

Figure 16: The distribution of the years of experience from the group of
physicians who participated in the user study. The average number of years of
experience is $14.4$ years. Over half of the group have more than 10 years of
experience. This data attest that our surveyees are highly professional and
experienced.

Figure 17: Example of User Study I: we ask the physicians to pick from the 6
options and it’s possible to pick more than one option.

Figure 18: Example of User Study II: we ask the physicians to decide if the
generated progression of the disease is credible or not.

###  E.2 Statistics

To show the significance of our findings, we perform the paired t-test on the
F1 scores of real and generated scans over the 35 users. Our finding is
significant with a p-value of $0.0038$.

To quantitatively analyze their responses, we treat each class of pathologies
as an independent class and compute precision, recall, and F1 over all the
pathologies and physicians.

##  Appendix F Discussion

###  F.1 Limitations and Societal Impacts

The proposed framework is subject to several limitations, with one of the
primary constraints being the limited scope of Stable Diffusion. Due to the
model’s pre-training on general domain data, the absence of detailed medical
reports poses a significant challenge to the model’s ability to accurately and
reliably edit medical images based on precise text conditioning. Moreover, the
framework’s overall performance may be influenced by the quality and quantity
of available data, which can limit the model’s accuracy and generalizability.
Furthermore, the absence of surgical or drug intervention data further
restricts the framework’s ability to simulate medical interventions. Moving
forward, it would be beneficial to explore ways of integrating more detailed
descriptions of medical data in the fine-tuning process of Stable Diffusion to
improve the framework’s performance and precision in disease simulation
through text conditioning. Additional details on the framework’s limitations,
including an analysis of failure cases, can be found in Supplementary D.

Progressive Image Editing (PIE) holds promise as a technology for simulating
disease progression, but it also raises concerns regarding potential negative
social impacts. One crucial concern revolves around the ethical use of medical
imaging data, which may give rise to privacy and security issues. To address
this, healthcare providers must take measures to safeguard patient privacy and
data security when utilizing PIE. An effective mitigation strategy involves
employing anonymized or de-identified medical imaging data, while also
adhering to ethical guidelines and regulations like those outlined by HIPAA
(Health Insurance Portability and Accountability Act).

Another concern relates to the accuracy of the simulations generated by the
framework, as errors could lead to misdiagnosis or incorrect treatment
decisions. To alleviate this concern, rigorous testing and evaluation of the
technology must be conducted before its implementation in a clinical setting.
Enhancing the accuracy of simulations can be achieved by incorporating
additional data sources, such as patient history, clinical notes, and
laboratory test results. Additionally, healthcare providers should receive
adequate training on effectively utilizing the technology and interpreting its
results.

Discrimination against certain groups, based on factors such as race, gender,
or age, poses yet another potential concern. Healthcare providers must ensure
the fair and unbiased use of the technology. This can be accomplished by
integrating diversity and inclusion considerations into the technology’s
development and training processes, as well as regular monitoring and auditing
its usage to identify any signs of bias or discrimination.

The cost and accessibility of the technology present further concerns,
potentially restricting its availability to specific groups or geographic
regions. To tackle this issue, healthcare providers should strive to make the
technology accessible and affordable to all patients, regardless of
socioeconomic status or geographic location. This can be achieved through the
creation of cost-effective models, partnerships with healthcare providers, and
government funding initiatives.

Lastly, there is a risk of excessive reliance on technology, leading to a
diminished reliance on clinical judgment and expertise. To mitigate this
concern, healthcare providers must be trained to view the technology as a tool
that complements their clinical judgment and expertise, rather than relying
solely on it for diagnostic or treatment decisions. The technology should be
used in conjunction with other data sources and clinical expertise to ensure a
comprehensive understanding of disease progression.

In conclusion, while the use of PIE comes with potential negative social
impacts, there are viable mitigations that can address these concerns.
Healthcare providers must be aware of the ethical implications associated with
this technology and take appropriate measures to ensure its safe and
responsible utilization.