#  ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the
Chain of Diffusion

Youngseok Yoon
UCSB
California, USA
<EMAIL_ADDRESS>
&Dainong Hu
UCSB
California, USA
<EMAIL_ADDRESS>
&Iain Weissburg
UCSB
California, USA
<EMAIL_ADDRESS>
Yao Qin
UCSB, Google
California, USA
<EMAIL_ADDRESS>
&Haewon Jeong
UCSB
California, USA
<EMAIL_ADDRESS>

###### Abstract

Diffusion models have achieved tremendous improvements in generative modeling
for images, enabling high-quality generation that is indistinguishable by
humans from real images. The qualities of images have reached a threshold at
which we can reuse synthetic images for training machine learning models
again. This attracts the area as it can relieve the high cost of data
collection and fundamentally solve many problems in data-limited areas. In
this paper, we focus on a practical scenario in which pretrained text-to-image
diffusion models are iteratively finetuned using a set of synthetic images,
which we call the Chain of Diffusion. Finetuned models generate images that
are used for the next iteration of finetuning. We first demonstrate how these
iterative processes result in severe degradation in image qualities. Thorough
investigations reveal the most impactful factor for the degradation, and we
propose finetuning and generation strategies that can effectively resolve the
degradation. Our method, Reusable Diffusion Finetuning (ReDiFine), combines
condition drop finetuning and CFG scheduling to maintain the qualities of
generated images throughout iterations. ReDiFine works effectively for
multiple datasets and models without further hyperparameter search, making
synthetic images reusable to finetune future generative models.

##  1 Introduction

_Can state-of-the-art AI models learn from their own outputs and improve
themselves?_ As generative AI (e.g., GPT [1], Diffusion [2, 3]) can now churn
out uncountable synthetic texts and images, this question piqued curiosity
from many researchers in the past couple of years [4, 5, 6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18]. While some show positive results of self-
improving [4, 15], most report an undesirable _“model collapse”_ [5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 16, 17, 18]—a phenomenon where a model’s performance
degrades when it goes through multiple cycles of training with the self-
generated data. When large language models (LLMs) are trained with their own
outputs, it begins to produce low-quality text that has a lot of repetitions
[19], and its linguistic diversity declines rapidly [8, 16]; image models also
show quality degradation [7, 10] and loss of diversity [12, 20]. This
phenomenon has even been compared to mad cow disease, a fatal
neurodegenerative disease caused by feeding cattle with the carcasses of other
cattle [12].

Our work examines model collapse in a common use case of generative AI:
customizing pretrained text-to-image generative models through finetuning.
While previous works [5, 12, 14, 19] suggest including more real images in the
training set to avoid model collapse, this is challenging for several reasons.
First, the number of artworks generated by a single artist is often limited.
Even highly productive artists create at most a few thousands of pieces during
their lifetime (e.g., Monet produced 1,983 paintings over 61 years). Secondly,
during image collection, it is easy to be deceived by realistic AI-generated
images and accidentally include them in the training set. Therefore, we need
to understand mode collapse behavior when the proportion of real data is
limited and seek solutions beyond simply increasing the number of real images.
In this paper, we propose a novel concept of generating “reusable” images,
which refers to their potential for finetuning without causing catastrophic
model collapse.

(a)  Self-Consuming Chain of Diffusion.

(b)  Image degradation with the Chain of Diffusion.

Figure 1:  (a) Overall pipeline of the Chain of Diffusion. Given a pretrained
text-to-image diffusion model $M_{0}$ and a prompt set $P$, a finetuned model
$M_{k}$ is trained using $D_{k-1}$ generated at the previous iteration $k-1$.
Then, $M_{k}$ generates $D_{k}$ using the same prompt set $P$. Chain of
diffusion begins with the original dataset $D_{0}$. (b) Image degradation
universally occurs during Chain of Diffusion across four datasets. As the
Chain of Diffusion progresses, the severity of image degradation intensifies,
exhibiting universal patterns of highly saturated images containing repetitive
high-frequency patterns. This consistently holds across four datasets and $10$
scenarios that we comprehensively investigate in Section 3.2.

Our goal in this paper is two-fold: (1) to conduct an in-depth empirical
analysis of model collapse behavior in the self-consuming chain of diffusion
finetuning (referred to as Chain of Diffusion)—a setting already widely used
by artists—and to examine which factors in finetuning and sampling affect the
reusability of diffusion-generated images; (2) to develop a strategy that can
improve the reusability of synthetic images. Our contributions toward these
goals are summarized below:

  * •

We consider a setting widely used for style transfer in diffusion models:
finetuning Stable Diffusion [3] with low-rank adaptation (LoRA) finetuning
[21] using a moderately-sized training set($\sim$$1000$ images). We examine
the evolution of images in the Chain of Diffusion across four datasets: two
digital art datasets (Pokemon [22] and Kumapi [23]) and two natural image
datasets (Butterfly [24] and CelebA-1k [25]). Through extensive analysis, we
show that model collapse universally occurs in all datasets regardless of
training set size, real image mixing (up to 90%), and a wide range of
scenarios with different configurations (see Figure 11(b) and Table 1).

  * •

We discover that among all factors in the Chain of Diffusion, only one
substantially impacts the speed of model collapse: classifier-free guidance
(CFG) scale. With low CFG, we observe low-frequency degradation where images
gradually become unrecognizably blurry. With high CFG scales, high-frequency
degradation occurs, where some features of the images become amplified beyond
the normal range with disturbing color saturation. There exists a medium CFG
range that can significantly slow down model collapse (see Figure 22(b)). To
the best of our knowledge, we are the first to demonstrate such a role of CFG
in the Chain of Diffusion.

  * •

Further, we reveal that the default CFG scale $7.5$ widely used for Stable
Diffusion performs very poorly in terms of reusability metric (defined in Eqn
(4)) despite its excellent image quality in the first iteration (measured by
FID). This highlights the trade-off between first-iteration image quality and
future reusability (see Figure 23).

  * •

We develop a new finetuning strategy, Reusable Diffusion Finetuning
(ReDiFine), to mitigate degradation inspired by our analysis of how CFG
modulates the model collapse pattern in latent diffusion models. We combine
condition drop (for finetuning) and CFG scheduling (for generation) in
ReDiFine. While an optimal CFG scale can sufficiently slow down model
collapse, it is impractical to sweep through CFG scales over multiple
iterations for each dataset to find an optimal trade-off. With our proposed
ReDiFine approach, we show that even the default CFG scale (CFG=7.5) can
generate reusable images across all four datasets without the need for
parameter tuning (see Figure 5 and 6).

##  2 Related Work

The self-consuming training loop and the associated phenomenon known as “model
collapse” have become significant areas of study in the past two years [5, 6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]. Model collapse, defined as “a
degenerative process affecting generations of learned generative models, where
generated data end up polluting the training set of the next generation of
models” in [11], has been observed in both language and image generative
models. Empirical studies on LLMs [8, 15, 16] reveal that linguistic diversity
collapses, especially in high-entropy tasks [8], although this can be
mitigated with data accumulation [15]. In image generation, several works [5,
7, 10, 12, 18, 20] note image degradation when diffusion models are
recursively trained with self-generated data. The authors of [5] and [12]
provide theoretical insights into diversity reduction, demonstrating that the
covariance in a toy Gaussian model converges to zero as iterations increase in
the self-consuming training loop. Among theoretical studies on this topic [13,
14, 17, 19], [17] is particularly relevant. They use dynamical system tools to
show that generative distributions either collapse to a small set of outputs
or become uniform over a large set, depending on the temperature setting. This
finding is analogous to our observations with modulating CFG scales.
Connecting our empirical analysis on finetuning Stable Diffusion with their
theoretical framework could be an interesting future direction.

Regarding mitigation strategies, existing works echo the importance of
incorporating a large proportion of real data throughout the training loop [5,
12, 14] or accumulating more data [15]. The only mitigation strategy beyond
altering the training data composition is proposed by [6], who suggests a
self-correcting self-consuming loop using an expert model to correct synthetic
outputs. While this approach is demonstrated in human motion generation with a
physics simulator, having an expert model may not always be feasible for
finetuning customized image generative models. In our work, we explore an
alternative solution through _reusable image generation_. By making small
improvements in finetuning and sampling steps, we can substantially enhance
the reusability of images, allowing them to be used for many more iterations
before the model starts to deteriorate. In a tangentially related work [9],
the authors discuss bias amplification in self-consuming chains, noting
minimal amplification over iterations.

##  3 Model Collapse in Self-Consuming Chain of Diffusion Finetuning

Table 1:  Description of $10$ potential factors (excluding CFG) that we
examined as candidate sources for model collapse. Dash ($-$) in table entries
indicates that the default value was used. All experiments were conducted on
Pokemon dataset except for the dataset size. For dataset size, we use CelebA
since its original dataset is bigger than Pokemon and we can subsample $500$,
$1000$, $2000$ images. More details in Appendix A.

Candidates | Default | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Learning Rate | 1e-4 | 1e-5 | 1e-3 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Finetune | Both | $-$ | $-$ | Unet | Text | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Noise | No | $-$ | $-$ | $-$ | $-$ | Yes | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Prompt111C, W, S, and L for Combine, Waifu, Short, and Long, respectively. We concatenate BLIP and Waifu captions as default setting, referred as Combine. Short and Long prompts are BLIP captions generated with limitations in the lengths of captions. More details can be found in Appendix C.4. | C | $-$ | $-$ | $-$ | $-$ | $-$ | W | S | L | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
CLIP skip | $2$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $1$ | $3$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Diffusion Steps | $30$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $100$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Epochs | $100$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $50$ | $200$ | $-$ | $-$ | $-$ | $-$ | $-$
Real mixing | $0.0$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $0.5$ | $0.9$ | $-$ | $-$ | $-$
Img / Prompt | $1$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $5$ | $-$ | $-$
Dataset Size | $1000$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $500$ | $2000$

###  3.1 Problem setting & Experimental Setup

##### Chain of Diffusion

We begin with formally defining the self-consuming chain of diffusion
finetuning. Given a pretrained generative model $M_{0}$, an original training
image set $D_{0}=\\{x_{0,i}|i\in[0,N-1]\\}$, and a prompt set
$P=\\{y_{i}|i\in[0,N-1]\\}$, where $N$ is the number of total images in the
dataset, each image $x_{0,i}$ is paired with a corresponding text prompt
$y_{i}$. $M_{k+1}$ is a model finetuned from $M_{0}$ using the generated image
set $D_{k}=\\{x_{k,i}|i\in[0,N-1]\\}$ and the prompt set $P$, which simulates
a common adaptation case of a pretrained generative model with a personally
collected dataset. Then, $M_{k+1}$ generates a set of images $D_{k+1}$ for the
next iteration using the prompt set $P$:

$M_{k+1}=\text{Finetune}(M_{0},D_{k},P),$ (1)
$D_{k+1}=\text{Generate}(M_{k+1},P).$ (2)

During the Chain of Diffusion, $M_{0}$ and $P$ are fixed across all the
iterations. To maintain the same size of training dataset, we generate one
image per text prompt for all iterations. The overall pipeline of the Chain of
Diffusion is shown in Figure 11(a).

##### Model and datasets.

We use Stable Diffusion v1.5 [3] as the pretrained model $M_{0}$ and apply
LoRA [21] to finetune the Stable Diffusion at each iteration. We build our
implementation on [26] and perform experiments on four datasets: Pokemon [22],
Kumapi [23], Butterfly [24], and CelebA-1k [25] to investigate various domains
including animation, handwriting, and real pictures. During each iteration, we
finetune the pretrained Stable Diffusion $M_{0}$ for 100 epochs. More
implementation details can be found in Appendix A.1 and A.2.

(a)  Reusability-FID trade-off333We do not display scenarios that change the
training dataset size, such as Img/Prompt and Dataset Size, as varying sizes
result in FID scores on different scales. The related results are presented in
the Appendix C, where we observe similar image degradation..

(b)  Image degradation across different CFGs.

Figure 2:  (a) Reusability-FID trade-off plot (Pokemon) shows that all
scenarios other than changing CFG scales show poor reusability, amplifying the
FID score $\sim$3x in 6 iterations of the chain. The x-axis represents
$\text{FID}_{1}$, quantifying the generation performance at the first
iteration and the y-axis represents the _reusability_ defined in equation (4).
For both $\text{FID}_{1}$ and _reusability_ , lower is better. _reusability_
(500 $\sim$ 1100) is about two times larger than $\text{FID}_{1}$ (300 $\sim$
600), indicating $\text{FID}_{6}$ three times larger than $\text{FID}_{1}$.
(b) Low CFG leads to blurry images and high CFG leads to high-frequency
degradation in the Chains of Diffusion. CFG $2.5$ for Pokemon and $1.5$ for
CelebA exhibit an ideal middle ground where both types of degradation slow
down. More images in Appendix B.

##### Evaluation metrics.

We use CLIP score [27] to quantify text-to-image alignment and Frechet
Inception Distance (FID) score [28] to measure image fidelity. Furthermore, we
compute the average sample-wise feature distance (SFD) between a pair of
images corresponding to the text prompt as the fidelity metric applicable for
text-to-image generation,

${\rm SFD}_{k}={\frac{1}{N}}\sum d(f(x_{k,i}),f(x_{0,i}))$ (3)

to evaluate each iteration $k$. SFD overcomes the problem of FID being
sensitive to the number of images to compare. Following [29], we use DiNOv2
[30] as a feature extractor for both FID and SFD since it is more consistent
with our visual inspection than Inception-V3 network [31]. With a slight abuse
of terms, we will still refer to the Frechet distance with DiNOv2 features as
the FID score.

##### New evaluation metric: Reusability.

In addition to evaluation metrics for generative models presented above, we
propose a new metric to quantify the reusability of images generated in the
Chain of Diffusion. We define a reusability metric as the performance
discrepancy between the first iteration and the $K$-th iteration in the Chain
of Diffusion:

$K\text{-reusability}={\rm FID}_{K}-{\rm FID}_{1},$ (4)

where ${\rm FID}_{k}$ stands for the FID between $k$-th iteration set and the
original training set. We use FID as a performance metric here, but this can
be CLIP, SFD, or any other performance metric of interest. Note that a low
$K$-reusability indicates more reusable images since it means that the model
does not degrade much in $K$ iterations. In the rest of the paper, $K=6$, and
we simply call it _reusability_.

(a) t-SNE of images.

(b) CLIP.

(c) FID.

(d) SFD.

Figure 3:  (a) t-SNE plot visualizes how the distribution of generated images
shifts from the original distribution (black) in the Chain of Diffusion and
shows distinct paths for three different CFG scales for Pokemon dataset.
Different CFG scales and iterations are differentiated with different colors
and transparency. Arrows indicate how the distributions of generated images of
chains move for different CFG scales. While CFG 2.5 (blue) stays near the
original images (black), high and low CFG scales (1.0 and 7.5) deviate fast,
indicating image degradation. (b-d) Qualitative comparison for different
Chains using CLIP $\uparrow$, FID $\downarrow$, and SFD $\downarrow$ for
Pokemon dataset. CFG $2.5$ achieves the most robust performance for all three
scores. CFG $1.0$ begins to gradually degrade from the beginning of the chain
while CFG $7.5$ begins to degrade in the third iteration, which aligns with
the visual inspection in Figure 22(b). Additional results for different
datasets can be found in Appendix D

###  3.2 Model Collapse in the Chain of Diffusion

In this section, we make a series of observations regarding the model collapse
behavior in the Chain of Diffusion. We conduct extensive investigations to
reveal the most impactful factor in the model collapse and analyze how this
factor contributes to the Chain of Diffusion.

##### Observation 1: Model collapse is universal in the Chain of Diffusion.

We observe significant image degradation in all four datasets (see Figure
11(b)) in the Chain of Diffusion. The quality of generated images begins to
clearly deteriorate in the third iteration and it drops even more rapidly once
the visible degradation emerges, reaching an unrecognizable level in two or
three additional iterations. We investigated a variety of different scenarios
(summarized in Table 1) to see if this degradation is an anomaly of specific
hyperparameter settings or if it is a ubiquitous phenomenon. We tested various
dataset sizes for $D_{0}$ and $D_{k}$ (Appendix C.1), increasing the size of
synthetic datasets $D_{k}$ by generating more than one image per prompt
(Appendix C.2), mixing real images from $D_{0}$ to $D_{k}$ (Appendix C.3),
changing the descriptiveness of prompts (Appendix C.4), freezing U-Net or text
encoder during finetuning (Appendix C.5), and various other hyperparameters (#
sampling steps, # epochs, learning rate, and CLIP skip) (Appendix C.6, C.7,
C.8, and C.9). We also tested adding a small Gaussian noise in each image in
the original set $D_{0}$ to see if having small random perturbations can
improve reusability (Appendix C.10), and investigated if the degradation
happens for larger Stable Diffusion (Appendix C.11). _In all settings we
tested, image degradation was universally present._ We plot the results for
Pokemon dataset on Figure 23 where y-axis is reusability and x-axis is the
${\rm FID}_{1}$. While adding noise and mixing $90\%$ real images show the
best reusability (the lower, the better), they still exhibit significant
degradation (FID score has been doubled in $6$ iterations). More visual
inspections and metric plots can be found in Appendix C.

##### Observation 2: CFG is the only factor that impacts the model collapse.

Throughout all our experiments, classifier-free guidance (CFG) had the biggest
impact in the speed of model collapse. CFG scale was first introduced in [32]
to modulate the strength between unconditional and conditional scores at each
diffusion step as follows:

$\text{Total Score}=\text{Unconditional
Score}+\text{CFG}\cdot(\text{Conditional Score}-\text{Unconditional Score}).$
(5)

High CFG means that we emphasize the conditional score for the given prompt
more, which pushes the generation to align better with the prompt and often
leads to higher-fidelity images. On the other hand, lower CFG places less
weight on the conditional score and provides more diversity in generated
images. For those familiar with temperature sampling [33], CFG plays a similar
role as temperature, which adjusts the trade-off between fidelity and
diversity.

In Figure 23, we observe that as we increase the CFG scale, the image quality
in the first iteration improves (smaller ${\rm FID}_{1}$), which is expected
from our understanding of CFG. A more surprising part is that this comes at
the cost of a worse reusability metric (increase in ${\rm FID}_{6}-{\rm
FID}_{1}$). Also, when the CFG scale is as high as $7.5$ or $10.0$, the
improvement in ${\rm FID}_{1}$ plateaus, and increased CFG worsens both ${\rm
FID}_{1}$ and reusability. Similarly, when the CFG scale is too low—below
$2.0$—the improvement in reusability plateaus and both ${\rm FID}_{1}$ and
reusability begin to decline. Interestingly, there is an optimal region of CFG
values (near $2.5$, specific to Pokemon dataset), where we achieve good
reusability while maintaining a good quality in the first iteration as well.
Figure 3 presents quantitative results for Pokemon dataset to demonstrate how
different CFG scales affect the performance in CLIP, FID, and SFD. CFG scale
$2.5$ achieves the most robust performance for all metrics.

(a) Probability density function (PDF) of values of latent vectors at the last
diffusion step.

(b) Norm of differences between conditional and unconditional scores during
diffusion steps.

Figure 4:  Latent analysis to investigate how different CFG scales affect the
latent of Stable Diffusion (Pokemon dataset). (a) Latent distribution shrinks
over iteration for low CFG and expands with high CFG. Larger values in latent
vectors are more likely to occur with high CFG, gradually increasing the tail
of the distribution. (b) Differences between conditional and unconditional
scores increase as the training set is more degraded. Especially, high
differences in the later diffusion steps can be a cause of high-frequency
degradation.

##### Observation 3: High CFG scales cause high-frequency degradation and low
CFG scales cause low-frequency degradation.

CFG scale does not only affect the speed of model collapse, but also the
pattern of model collapse. As shown in Figure 22(b), CFG $1.0$ makes the
images progressively more blurry in the Chain of Diffusion, eventually
collapsing to images without any structure, which we refer to low-frequency
degradation. On the other hand, for CFG $7.5$ how images degrade looks
completely different: some features start to be emphasized excessively,
repetitive patterns begin to appear, and the overall color distribution
becomes saturated. The t-SNE plot in Figure 33(a) clearly demonstrates that
the distribution shift over iterations follows distinct paths for high, low,
and medium CFG scales. These patterns were consistent in all four datasets and
detailed results can be found in Appendix B.

We further analyze this phenomenon by examining the latent space. First, we
show a histogram of the final latent vectors before decoding in Figure 44(a).
For a CFG of $1.0$, the latent distribution quickly evolves into a Gaussian-
like shape, with its variance decreasing over iterations. This behavior aligns
with previous work [5, 12, 19], which theoretically predicted that the self-
consuming loop would crop out the tails of the distribution, reducing output
diversity until it collapses to a single mode. We believe that these narrowing
distributions in the latent space lead to blurrier and more homogeneous
outputs in the image space. For a CFG of $7.5$, the evolution of the latent
distribution is completely opposite, creating longer tails and approaching a
more uniform distribution over space. For a CFG scale of $2.5$, which has the
best reusability among the three, the latent distribution is better preserved
over iterations.

We then plot how the average norm of Diff ($=\text{Cond Score}-\text{Uncond
Score}$) evolves over diffusion steps in different iterations (Figure 44(b)).
In the first iteration, CFG $1.0$ has the highest Diff value, followed by CFG
$2.5$ and CFG $7.5$. It can be understood as the models’ adaptive behavior to
preserve the values added to the latent vectors, Diff multiplied by CFG scale,
at each step. This trend changes in later iterations. The Diff value for CFG
$7.5$ continues to grow with each iteration, and by iteration $6$, we observe
high Diff values during the entire diffusion steps, creating a significant gap
compared to CFG $2.5$ and CFG $1.0$. We conjecture that this accumulation is
the cause of high-frequency degradation of images for CFG $7.5$. In contrast,
the Diff value for CFG $1.0$ remains relatively stable or even decreases over
iterations.

##### Implications of our observations.

Through extensive investigations, we gained a comprehensive understanding of
CFG scales’ effects on the Chain of Diffusion. Even though a high CFG scale of
$7.5$ is a common choice to generate visually appealing images, it
significantly sacrifices reusability to achieve slightly better FID. Moreover,
it is noteworthy that CLIP score increases for the early three iterations and
drops rapidly in the latter when the CFG scale is $7.5$ (Figure 33(b)). This
shows that Stable Diffusion tends to generate images favored by CLIP encoder,
which can contain artifacts easily amplified by further finetuning. Sampling
for maximizing the perceptual quality was coined as ‘sampling bias’ in [12]
and they provide FID plots with varying levels of sampling bias by changing
the CFG scale. While they reported a monotonic decrease in reusability as CFG
increased from $1.0$ to $2.0$, we show that the holistic picture is not
entirely monotonic when we look at a wider range of CFG scales from $1.0$ to
$10.0$. It shows an intriguing trade-off between perceptual quality and
reusability. This suggests that if developers of diffusion models are mindful
of the reusability metric, can improve future generations of images
substantially by carefully choosing CFG.

##  4 ReDiFine

In the previous section, we showed that choosing a good CFG scale can
considerably improve reusability in the Chain of Diffusion. However, such
optimal CFG scales vary from dataset to dataset. To mitigate model collapse
solely with CFG tuning, we have to do multiple rounds of finetuning and
sampling to evaluate the reusability-FID trade-off. This is too
computationally burdensome to be a practical solution. In this section, we
propose ReDiFine, Reusable Diffusion Finetuning, which can mitigate model
collapse without any parameter tuning. ReDiFine combines two ideas—condition
drop finetuning and CFG scheduling—and effectively improves the reusability-
FID trade-off in all four datasets (comparable to optimal CFG scales).

###  4.1 Condition drop finetuning.

We propose condition drop finetuning, which randomly drops the text condition
during finetuning to update both conditional and unconditional scores. Even
though condition drop was suggested in the original CFG paper [32], it is not
commonly done during finetuning since the first-iteration image qualities are
outstanding without updating unconditional scores. However, the small
difference between conditional and unconditional scores can accumulate over
iterations and create a large gap as the iteration progresses in the Chain of
Diffusion if we do not explicitly finetune the unconditional scores as shown
in Figure 44(b). Inspired by this observation, we drop text conditions with
probability $0.2$ during finetuning. We illustrate the effect of condition
drop with CFG $7.5$ in Figure 44(b). Note that the norm of Diff does not grow
as fast even with a high CFG of $7.5$. While this prevents a potential cause
of the high-frequency degradation, the Diff value still spikes up in iteration
6. We conjecture that this is because the $D_{5}$ already contains severe
artifacts that conditional scores have to grow substantially to fit the data.

###  4.2 CFG Scheduling.

Based on the analysis showing that a high CFG scale amplifies the discrepancy
between the condition and uncondition scores during iterative fine-tuning (see
Figure 44(b)), we propose gradually reducing the CFG scale during diffusion
steps. This CFG scheduling mechanism mitigates the negative impacts of
overemphasizing the conditional score in later stages while preserving the
initial conditional information in the early stages. Specifically, we
exponentially decrease the CFG scale $s$ during $T$ diffusion steps as

$s=s_{0}\times e^{-\alpha\times t/T}$ (6)

where $T$ is $30$ and $s_{0}$ is $7.5$. $\alpha$ denotes the degree of
decrease and we use $2$ throughout this paper. Comparison to different
$\alpha$ and linear decreasing can be found in Appendix E.

Our method gradually decreases the CFG scale during diffusion steps,
leveraging the benefits of high CFG scale in the early stage and low CFG scale
in the later stage. This adaptive approach aligns with observations from [34]
that different steps contribute uniquely to the generation process. In
addition, our CFG scheduling takes advantage of using a small CFG scale
without the need for extensive hyperparameter searches for each dataset,
providing a flexible and robust choice for the Chain of Diffusion.

##  5 Reusable Image Generation

Figure 5:  ReDiFine effectively mitigates image degradation in the Chain of
Diffusion. ReDiFine successfully preserves the characteristics and features
without further dataset-specific hyperparameter search. Artifacts observed in
high-frequency degradation do not exist for all four datasets.

(a) Pokemon.

(b) CelebA-1k.

(c) Kumapi.

(d) Butterfly.

Figure 6:  ReDiFine effectively mitigates Reusability-FID trade-off across all
four datasets. While the optimal CFG scale varies for different datasets,
ReDiFine consistently achieves low reusability and FID at the same time (lower
is better). While CFG scheduling alone works well for Pokemon dataset, it is
still beneficial to have condition drop finetuning for CelebA-1k and Kumapi
datasets. Note that the differences in FID are relatively smaller than those
in reusability, supporting the necessity to evaluate reusability in the Chain
of Diffusion.

In this section, we present generated images and qualitative metrics for
ReDiFine. We use the CFG scale $s_{0}=7.5$ for all datasets and experiments
except for the ablation study. Note that ReDiFine can be further improved with
hyperparameter tuning, e.g., CFG scale, condition drop probability, and CFG
scheduling function. However, our focus in this work is to demonstrate that
ReDiFine can robustly stabilize iterative finetuning regardless of datasets
and hyperparameters. This robustness provides a great advantage when applying
our ReDiFine to finetune Stable Diffusion and generate images using any
personally collected image sets.

##### Visual comparison.

Comparing Figure 5 with Figure 11(b), ReDiFine effectively slows down and
mitigates the image degradation in the Chain of Diffusion compared to using
the default CFG scale $7.5$ across four datasets. In addition, ReDiFine
successfully achieves the same level of high-quality image generation in the
Chain of Diffusion compared to using the optimal CFG scales for each dataset,
which requires extensive hyperparameter tuning. This is demonstrated by
visually comparing the generated images in Figure 5 with the ones generated
using the optimal CFG scales for Pokemon and CelebA dataset in Figure 22(b).
Additional comparisons for Kumapi and Butterfly datasets can be found in
Appendix B.

##### Mitigate Reusability-FID trade-off.

In addition to the visual comparison, we also provide quantitative results in
Figure 6. We can see that ReDiFine consistently helps mitigate Reusability-FID
trade-off in the Chain of Diffusion across all four datasets. In contrast, a
fixed CFG scale that works best on one dataset fails on others. For example,
the optimal CFG scale $2.5$ for Pokemon dataset achieves high (poor) ${\rm
Reusability}$ for CelebA-1k and Butterfly datasets. Moreover, the optimal CFG
scale $1.5$ for CelebA-1k and Butterfly datasets results in the high (poor)
${\rm FID}_{1}$ for Pokemon dataset. In addition to its ability to generalize
across different datasets, ReDiFine demonstrates robustness when initialized
with different CFG scales. Illustrated in Figure 66(a), ReDiFine achieves
comparable performance with CFG scales of $5.0$ and $10.0$ as CFG $=7.5$.

##### Ablation study.

We provide an ablation study to understand the contributions of condition drop
finetuning and CFG scheduling to the success of ReDiFine. As shown in Figure
6, CFG scheduling alone achieves comparable reusability in the Pokemon
dataset, but condition drop finetuning is essential for CelebA-1k and Kumapi.
Combining these two, ReDiFine consistently achieves the best trade-off between
${\rm Reusability}$ and ${\rm FID}_{1}$ across all four datasets.
Additionally, the differences in ${\rm FID}_{1}$ are relatively small—using
different CFG scales can generate high-quality images in the first iteration
except for very low CFG scales, e.g., CFG = $1.0$. In contrast, the variance
of ${\rm Reusability}$ is much larger. This supports the necessity of
evaluating our proposed ${\rm Reusability}$ metric for the Chain of Diffusion.
Additional results can be found in Appendix E.

##### More iterations.

In addition to the comparison over 6 iterations, Figure 7 illustrates the
evolution of image degradation over 12 iterations. Notably, as fine-tuning
iterations increase, high-frequency degradation emerges with the optimal CFG,
highlighting the necessity for additional hyperparameter search to sustain
image quality across longer diffusion lengths. Conversely, ReDiFine preserves
image quality without high-frequency degradation.

Figure 7:  Comparison of the optimal CFG scale and ReDiFine on 12 iterations
for Pokemon dataset.

(a) Histogram of latent values.

(b) Norm of differences.

(c) Power spectrum of latents.

Figure 8:  Comparison of latent analysis at the 6th iteration for ReDiFine and
baselines.  (a) ReDiFine preserves the latent distribution. (b) Conditional
and unconditional score differences do not surge in the $6$-th iteration,
showing dramatic improvements from baselines. (c) ReDiFine maintains similar
power spectrum to CFG scale $2.5$ in the $6$-th iteration.

##### Latent analysis.

Figure 8 demonstrates that ReDiFine can successfully alleviate the symptoms we
observed in the latent space in Section 3. When we plot latent histograms, we
can notice that ReDiFine preserves the original latent distribution at the 6th
iteration as compared to other baselines with different CFG values. More
importantly, the conditional-unconditional score differences (Diff) are kept
at bay all the way until the 6th iteration, as opposed to the baseline CFG
$7.5$, showing a severe discrepancy between the unconditional and conditional
scores. Additionally, the power spectrum of latents in the $6$th iteration
increases or decreases for CFG scales $1.0$ and $7.5$ while ReDiFine and CFG
scale $2.5$ preserve density distribution. More analysis and comparison for
all iterations are provided in Appendix F.1.

(a) Autocorrelation.

(b) Average power spectra.

Figure 9:  The fingerprints of the original training set and synthetic sets
show clear differences, and ReDiFine produces fingerprints similar to CFG
scale 2.5. (a) Autocorrelation of image fingerprints. Horizontal and vertical
lines gradually disappear for CFG scales $1.0$ and $7.5$ while they are
maintained for CFG scale $2.5$ and ReDiFine. (b) Average power spectra of
images. Central regions are amplified or diminished for CFG scales $1.0$ and
$7.5$, demonstrating low and high-frequency degradation.

##### Frequency fingerprint.

Several works [35, 36] aim to identify fingerprints of synthetic images. High-
quality synthetic images from different generative models have clearly
distinct fingerprints, showing the potential to be used for synthetic image
detection. We analyze fingerprints of synthetic images for different CFG
scales and iterations, and compare them to fingerprints of the original
training set. Both power spectra and autocorrelation show clear differences
between the original training set and synthetic images as shown in Figure 9.
Moreover, fingerprints of synthetic images from ReDiFine are similar to those
of images from CFG scale $2.5$, while other CFG scales ($1.0$ and $7.5$) make
fingerprints different from the first iteration as iterations proceed.
Horizontal and vertical lines in autocorrelation gradually disappear and
central regions in power spectra vary for further iterations. Also, the
varying central regions in power spectra imply that low frequency features
increase and decrease for CFG scale $1.0$ and $7.5$, respectively, aligning
with visual inspections. Generating images with fingerprints similar to those
of the original real images can be an interesting future direction to reduce
the degradation in the Chain of Diffusion. Additional analysis with power
spectrum density of images is provided in Appendix F.2.

##  6 Conclusion and Discussion

Motivated by the prominent usage of diffusion models by artists and the trend
of the Internet being filled with synthetic data, we investigate how image
quality degrades as we feed diffusion-generated synthetic images to finetune
another diffusion model. In our self-consuming chain of diffusion finetuning
setting, we observe image degradation in all four datasets we tested. We
reveal that CFG plays a crucial role in the trade-off between image
reusability and its perceptual quality in the first iteration. We propose a
novel idea of _generating reusable iamges_ and propose the ReDiFine strategy
that combines two small techniques (condition drop finetuning and CFG
scheduling) that can be easily implemented in the diffusion finetuning
pipeline with no requirement for additional parameter tuning.

We started this paper with a question: can current AI models learn from their
own output and improve themselves? This is not only an imminent question as
synthetic data rapidly outnumbers human-generated data [37, 38], but also a
long-term existential question as it relates to reaching AI singularity—the
point where AI learns from its own output to continuously enhance its
intelligence and far surpasses human intelligence. Our paper shows a glimpse
that widely-used text-to-image models are not ready to improve from their own
creation quite yet. However, we believe that there are many open questions on
how we develop AI in a world dominated by AI data. We present one angle by
emphasizing the importance of generating _reusable data_ for future AI
developers. Another pathway could be developing a new learning algorithm that
can discern real and synthetic data and take a different training technique
for each.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [4] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.  Large language models can self-improve.  arXiv preprint arXiv:2210.11610, 2022.
  * [5] Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel.  On the stability of iterative retraining of generative models on their own data.  arXiv preprint arXiv:2310.00429, 2023.
  * [6] Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yonglong Tian, and Chen Sun.  Self-Correcting Self-Consuming Loops for Generative Model Training, April 2024.  arXiv:2402.07087 [cs, stat].
  * [7] Matyas Bohacek and Hany Farid.  Nepotistically trained generative-ai models collapse.  arXiv preprint arXiv:2311.12202, 2023.
  * [8] Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel.  The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text, April 2024.  arXiv:2311.09807 [cs].
  * [9] Rohan Taori and Tatsunori Hashimoto.  Data feedback loops: Model-driven amplification of dataset biases.  In International Conference on Machine Learning, pages 33883–33920. PMLR, 2023.
  * [10] Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar.  Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation?  arXiv preprint arXiv:2303.01255, 2023.
  * [11] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson.  The curse of recursion: Training on generated data makes models forget.  arXiv preprint arXiv:2305.17493, 2023.
  * [12] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk.  Self-consuming generative models go mad.  arXiv preprint arXiv:2307.01850, 2023.
  * [13] Elvis Dohmatob, Yunzhen Feng, and Julia Kempe.  Model Collapse Demystified: The Case of Regression, April 2024.  arXiv:2402.07712 [cs, stat].
  * [14] Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao.  Towards theoretical understandings of self-consuming generative models.  arXiv preprint arXiv:2402.11778, 2024.
  * [15] Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al.  Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data.  arXiv preprint arXiv:2404.01413, 2024.
  * [16] Martin Briesch, Dominik Sobania, and Franz Rothlauf.  Large language models suffer from their own output: An analysis of the self-consuming training loop.  arXiv preprint arXiv:2311.16822, 2023.
  * [17] Matteo Marchi, Stefano Soatto, Pratik Chaudhari, and Paulo Tabuada.  Heat death of generative models in closed-loop learning.  arXiv preprint arXiv:2404.02325, 2024.
  * [18] Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar.  Towards understanding the interplay of generative artificial intelligence and the internet.  In International Workshop on Epistemic Uncertainty in Artificial Intelligence, pages 59–73. Springer, 2023.
  * [19] Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe.  A tale of tails: Model collapse as a change of scaling laws.  arXiv preprint arXiv:2402.07043, 2024.
  * [20] Ryuichiro Hataya, Han Bao, and Hiromi Arai.  Will Large-scale Generative Models Corrupt Future Datasets?  In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20498–20508, Paris, France, October 2023. IEEE.
  * [21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [22] Pokémon.  Pokédex.  https://www.pokemon.com/us/pokedex, 2023.  Accessed: 2023-04-19.
  * [23] Ihelon.  Illustrations kumapi390, 2022.  Accessed: 2023-04-19.
  * [24] Veeralakrishna.  Butterfly dataset.  https://www.kaggle.com/datasets/veeralakrishna/butterfly-dataset, 2020.  Accessed: 2024-01-03.
  * [25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [26] kohya-ss.  kohya-ss trainer.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * [28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [29] George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganem.  Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [30] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [31] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [32] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [33] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski.  A learning algorithm for boltzmann machines.  Cognitive science, 9(1):147–169, 1985.
  * [34] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al.  ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [35] Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva.  On the detection of synthetic images generated by diffusion models.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [36] Riccardo Corvi, Davide Cozzolino, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva.  Intriguing properties of synthetic images: from generative adversarial networks to diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 973–982, 2023.
  * [37] Brian Thompson, Mehak Preet Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico.  A shocking amount of the web is machine translated: Insights from multi-way parallelism.  arXiv preprint arXiv:2401.05749, 2024.
  * [38] Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho.  Will we run out of data? an analysis of the limits of scaling datasets in machine learning.  arXiv preprint arXiv:2211.04325, 2022.
  * [39] StabilityAI.  Sd vae ft mse original, 2022.  Accessed: 2023-04-19.
  * [40] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala.  PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.  In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24). ACM, April 2024.
  * [41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022.
  * [42] Hakurei.  Waifu diffusion v1.4.  https://huggingface.co/hakurei/waifu-diffusion-v1-4, 2022.  Accessed: 2023-04-18.

## Supplementary Material

##  Appendix A Experimental setup

###  A.1 Hyperparameters

We use Stable Diffusion v1.5 [3] as a pretrained text-to-image diffusion model
and apply LoRA [21] to fine-tune it at each iteration. We utilize ft-MSE [39]
as a fixed VAE to project images into latent space. We augment images with
horizontal flip, and no additional augmentation is used. Our implementation is
based on [26], with the environment built on PyTorch v2.2.2 [40] and its
accompanying torchvision 0.17.2, utilizing CUDA 12.4 on NVIDIA A-100 and L40S
GPUs. All default hyperparameters used throughout our paper can be found in
Table 2.

Table 2:  Default hyperparameters used for the Chain of Diffusion.  Hyperparameter | Value
---|---
Optimizer | AdamW
Learning Rate - Unet | $0.0001$
Learning Rate - CLIP | $0.00005$
LoRA Weight Scaling | $8$
LoRA Rank | $32$
Batch Size | $6$
Max Epochs | $100$
CLIP Skip | $2$
Noise Offset | $0.0$
Mixed Precision | fp16
Loss Function | MSE
Min SNR gamma | $5.0$
Max Gradient Norm Clipping | $1.0$
Caption Dropout Rate | $0.0$
Sampler | Euler A
Classifier-Free Guidance Scale | $7.5$
Number of Diffusion Steps | $30$
Number of Images per Prompt | $1$

###  A.2 Datasets

We use four image datasets to demonstrate that the degradation occurs
universally: Pokemon [22], Kumapi [23], Butterfly [24], and CelebA-1k [25] to
investigate various domains including animation, handwriting, and real
pictures. All images are resized to $512\times 512$ pixels. BLIP captioner
[41] and Waifu Diffusion v1.4 tagger [42] are used to generate text prompts.
Samples images and prompts can be found in Table 3.

####  A.2.1 Pokemon

The Pokemon dataset [22] contains $1008$ images corresponding to the Pokemon
indexing from No.1 to No.1008. Prompts are constructed by concatenating
captions from Waifu Diffusion tagger and BLIP captioner. We restrict the
length of BLIP captions between $50$ and $75$.

Table 3: Samples images and prompts for Pokemon, CelebA-1k, Kumapi, and
Butterfly datasets.

Pokemon
---
a green pokemon with red eyes and a leaf on the back of its head and tail, an image of the pokemon character with a red eye and big green tail, all set up to look like it is holding a lea, ultra-detailed, high-definition, high quality, masterpiece, sugimori ken (style), solo, smile, open mouth, simple background, red eyes, white background, standing, full body, pokemon (creature), no humans, fangs, transparent background, claws, Bulbasaur |
a very cute looking pokemon with a big leaf on its back and a big leaf on its head, a very cute little pokemon character with leaves in the back ground around his chest and hea, white background, ultra-detailed, high-definition, high quality, masterpiece, sugimori ken (style), solo, red eyes, closed mouth, standing, full body, pokemon (creature), no humans, fangs, transparent background, claws, outline, white outline, animal focus, fangs out |
CelebA-1k
a woman with brown hair smiling and posing for a picture in front of a mirror and gold and white stripes |
a woman with a very long red hair smiles and laughs on a city street and other people in the background |
Kumapi
solo, simple background, food, donut, grey background, no humans, food focus, still life, Kumapi style |
solo, looking at viewer, cute yellow figure, two tiny hands and feet, simple background, black dot eyes, white background, grey background, no humans, Kumapi style |
Butterfly
a crimson patched longwing butterfly with a red and black stripe on its wings, wings are long, narrow, rounded, black, crossed on fore wing by broad crimson patch, and on hind wing by narrow yellow line |
a Common Buckeye butterfly is sitting on a flower in the sun, wings scalloped and rounded except at drawn-out fore wing tip, on hind wing, 1 large eyespot near upper margin and 1 small eyespot below it. Eyespots are black, yellow-rimmed, with iridescent blue and lilac irises, on fore wing, 1 very small near tip and 1 large eyespot in white fore wing bar. |

####  A.2.2 CelebA-1k

We subsample the first $1000$ images from CelebA dataset [25] to construct
CelebA-1k dataset. Prompts are generated using BLIP captioner with length
between $25$ and $50$.

####  A.2.3 Kumapi

The Kumapi dataset [23] consists of $391$ images with handwriting styles.
Prompts are generated using Waifu tagger, and we manually modify some prompts
that do not reflect the images well.

####  A.2.4 Butterfly

Butterfly dataset [24] consists of $832$ images with $8$ species categories.
We combine captions generated using the BLIP captioner and descriptions for
species from the dataset. The length of captions generated by the BLIP
captioner is between $5$ and $75$.

##  Appendix B Different CFG Scales

This section presents images from Chains of Diffusion for different CFG scales
on four datasets outlined above. Figure 10, 11, 12, and 13 are for Pokemon,
CelebA-1k, Kumapi, and Butterfly datasets, respectively. For each dataset,
results for $5$ different CFG scales are provided to cover high, low, and
medium CFG scales. We found $2.5$ to be the optimal medium CFG scale for
Pokemon and Kumapi datasets, and $1.5$ for CelebA-1k and Butterfly datasets.
With CFG scales lower than the optimal value, low-frequency degradations are
observed throughout the four datasets we demonstrate. On the other hand, high-
frequency degradations with saturated color and repetitive patterns are
universally observed with CFG scales higher than the optimal value. Notably,
the optimal value for Pokemon and Kumapi causes the Chain of Diffusion for
CelebA-1k and Butterfly datasets to degrade severely.

Figure 10:  Chain of Diffusion with different CFG scales for Pokemon dataset.
CFG scale $2.5$ is the optimal value we found.  Figure 11:  Chain of Diffusion
with different CFG scales for CelebA-1k dataset. CFG scale $1.5$ is the
optimal value we found.  Figure 12:  Chain of Diffusion with different CFG
scales for Kumapi dataset. CFG scale $2.5$ is the optimal value we found.
Figure 13:  Chain of Diffusion with different CFG scales for Butterfly
dataset. CFG scale $1.5$ is the optimal value we found.

##  Appendix C Hyperparameter investigations to unveil the most impactful
factor of degradation

In this section, we present experiment results to reveal the most impactful
factor of degradation observed in the Chain of Diffusion. We conduct extensive
investigations to check the effects of each hyperparameter outlined in Table
1. Based on the default hyperparameter setting in Table 2, we change each
candidate hyperparameter and investigate whether any of them can meaningfully
mitigate or affect the degradation. CFG scale is $7.5$ for all cases in this
section.

###  C.1 Training set size

We used CelebA-1k dataset to investigate how the size of the training set
(both $D_{0}$ and $D_{k}$) affects the degradation observed in the Chain of
Diffusion. We chose this dataset as we can increase or decrease the training
set size easily by subsampling from a larger original dataset. We changed the
size of the training set to $100$, $250$, $500$, and $2000$. Degradations are
observed regardless of the size of the training set, as shown in Figure 14.
Degradations occur at earlier iterations when a smaller number of images are
used, but it is hardly distinguished in $6$th iteration. We fix the number of
parameter updates.

Figure 14:  Chain of Diffusion with varying training set size on CelebA-1k
dataset. Degradations begin to occur faster with smaller sizes, but all of
them results in unrecognizable degree of degradations in $6$th iteration.

###  C.2 Number of images per prompt

Generating more than one image per prompt to increase the diversity of the
training set is a straightforward method to mitigate the degradation in the
Chain of Diffusion. We conducted an experiment to investigate how increasing
the training set size by generating more images affects the degradation.
Figure 15 demonstrates that $5$ times larger images cannot mitigate the
degradation. It seems the degradation is slowed down by one iteration ($6$th
iteration for the default case and $5$th iteration for the target case look
similar), but at the cost of high computations, which makes it an infeasible
solution.

Figure 15:  Chain of Diffusion when more than one images are generated per
prompt on Pokemon dataset. Increasing the training set size by generating more
than one image per prompt cannot mitigate the degradation.

###  C.3 Mixing real images to synthetic sets

Many previous works suggested augmenting the training set with real images to
mitigate the degradation observed in iterative training. We investigated if
mixing images from the original training set to a synthetic set at each
iteration can relieve the degradation. Images are randomly sampled at each
iteration and replaced by corresponding real images in the original training
set. Figure 16 and 17 demonstrate how the degradation in the Chain of
Diffusion differs when $50\%$ and $90\%$ images at each iteration are replaced
for Pokemon and CelebA-1k datasets, respectively. It is noteworthy that $10\%$
is far enough to cause the degradation, and $50\%$ rarely slows down the
degradation. It is interesting that CelebA-1k dataset is much more vulnerable
to degradation.

Figure 16:  Chain of Diffusion when real images are randomly sampled and
replace synthetic images in every iteration on Pokemon dataset. $50\%$ of real
images rarely slow down the degradation, and $10\%$ is enough to commence the
degradation.  Figure 17:  Chain of Diffusion when real images are randomly
sampled and replace synthetic images in every iteration on CelebA-10k dataset.
The degradation is rarely slowed down even when only $10\%$ of synthetic
images are used for finetuning. Natural images undergo more serious
degradation compared to animation images in Figure 16.

###  C.4 Prompt Set

We hypothesized that the descriptiveness of prompts can affect the degradation
of the Chain of Diffusion. Accordingly, we examined different sets of prompts
for Pokemon and CelebA-1k datasets. The default prompt set used for Pokemon
dataset is the concatenations of Waifu and BLIP captions. BLIP captions are
generated to be between $50$ and $75$. We investigated how Waifu prompt set
alone and different lengths of BLIP captions can affect the degradation in
Figure 18. It is interesting that slightly different styles of high-frequency
degradation are observed and short prompt set can relieve repetitive patterns
at the cost of decreased diversity. Different lengths of BLIP captions in
CelebA-1k dataset demonstrate similar degrees of degradation, but insufficient
or excessive descriptiveness of prompt makes images to be different from the
original ones as shown in Figure 19.

Figure 18:  Chain of Diffusion when different prompts are used on Pokemon
dataset. The default prompts are concatenations of BLIP caption of length
between $50$ and $75$, and Waifu caption. We compare the Chain of Diffusion
with Waifu captions, BLIP captions (used for default prompts), and short BLIP
captions. Short captions are generated to have lengths smaller than $25$.
Figure 19:  Chain of Diffusion when different prompts are used on CelebA-1k
dataset. The default prompts have lengths between $25$ and $50$. We compare
the Chain of Diffusion with the lengths of prompts larger than $50$ (Long) and
smaller than $25$ (Short).

###  C.5 U-Net and Text-Encoder

Figure 20 shows the Chains of Diffusion when one of U-Net or text encoder is
finetuned. Similar degradation happens when the text encoder is not updated
(second row), and the degradation pattern is different when the U-Net of
Stable Diffusion is not updated since image qualities that models can generate
do not change in this case. However the contents of images are not preserved
as text encoder is updated.

Figure 20:  Chain of Diffusion when one of U-net and text encoder is finetuned
on Pokemon dataset.

###  C.6 Number of diffusion steps

We examined if the number of diffusion steps during generation is insufficient
and causes the degradation. Figure 21 demonstrates that the number of
diffusion steps cannot improve the Chain.

###  C.7 Number of Epochs

We also tested if the training is insufficient or excessive for our default
setting. However, images from the first iterations look similar in quality, as
shown in Figure 21, and the Chains result in similar degradations. We finetune
models for $100$ epochs as the default setting since loss values still
decrease in $50$ epochs.

Figure 21:  Chain of Diffusion when the number of diffusion steps and training
epochs differ on Pokemon dataset. Both more diffusion steps and different
training epochs cannot mitigate the degradation, and similar patterns are
observed.

###  C.8 Learning Rate

Similarly, we examined in Figure 22 if finetuning is properly done with our
default setting. We multiplied the learning rate for U-Net and text encoder by
$10$ and $0.1$. Images of the first iterations indicate that the default
values are appropriate for finetuning, and the degradation constantly occurs
for different learning rate.

Figure 22:  Chain of Diffusion when learning rate differs and Gaussian noise
is added to the original training set on Pokemon dataset.

###  C.9 CLIP Skip

We examined if CLIP skip hyperparameter can affect the degradation. CLIP skip
determines what intermediate feature from CLIP text encoder is used as text
embedding for conditional generation. Smaller value means the feature closer
to the output of CLIP text encoder is used, and larger values for features
closer to the input text. Figure 23 shows that this hyperparameter rarely
changes the degradation patterns.

Figure 23:  Chain of Diffusion when different CLIP skip hyperparameter is used
on Pokemon dataset. CLIP skip hyperparameter has negligible effect to the
degradation.

###  C.10 Adding Gaussian noise to the original training set

We inspected if there are any differences between real and synthetic images,
and if they can be cause the degradation. Random Gaussian noise is added to
the original training set $D_{0}$. Figure 22 shows that the characteristics of
the original training set have some effects, but the degradation still
happens. This aligns with our observations that the degradations are universal
to real, animation, and hand-written images.

###  C.11 Stable Diffusion XL

We investigated how image degradation occurs for different Stable Diffusion
models. We note here that optimal hyperparameters for finetuning SDXL using
LoRA have not been investigated enough yet, so we use the same hyperparameters
to Stable Diffusion v1.5 and it can be suboptimal. Due to the space
complexity, we reduce batch size to $2$. We also use the same resolution
$512\times 512$ for this model as the first iteration images are impressively
high-quality.

Figure 24:  Chain of Diffusion of SDXL on Pokemon dataset. Optimal
hyperparameters for finetuning SDXL have not been investigated enough yet, so
our experiments are largely based on hyperparameters for Stable Diffusion
v1.5. We note that images for Pikachu are provided to align with other cases,
but SDXL has a huge bias in generating Pikachu well while it fails for other
indices. Nonetheless, there are clear changes with different CFG scales.

##  Appendix D ReDiFine

Figure 25, 26, 27, and 28 show how robust ReDiFine is to different CFG scales.
It successfully mitigates the high-frequency degradations for an extensive
range of CFG scales.

Figure 25:  Chain of Diffusion of ReDiFine with different CFG scales on
Pokemon dataset. ReDiFine successfully achieves robust image qualities for
varying CFG scales.  Figure 26:  Chain of Diffusion of ReDiFine with different
CFG scales on CelebA-1k dataset. ReDiFine successfully achieves robust image
qualities for varying CFG scales.  Figure 27:  Chain of Diffusion of ReDiFine
with different CFG scales on Kumapi dataset. ReDiFine successfully achieves
robust image qualities for varying CFG scales.  Figure 28:  Chain of Diffusion
of ReDiFine with different CFG scales on Butterfly dataset. ReDiFine
successfully achieves robust image qualities for varying CFG scales.

In addition to visual inspections for images generated using our ReDiFine, we
also compare the quantitative results of ReDiFine to baselines with different
CFG scales on FID, CLIP, and SFD. Results are shown in Figure 29. For
different datasets and metrics, ReDiFine achieves comparable performance to
the optimal CFG scale ($2.5$ for Pokemon and Kumapi, $1.5$ for CelebA-1k and
Butterfly).

(a) Pokemon, CLIP.

(b) CelebA-1k, CLIP.

(c) Kumapi, CLIP.

(d) Butterfly, CLIP.

(e) Pokemon, FID.

(f) CelebA-1k, FID.

(g) Kumapi, FID.

(h) Butterfly, FID.

(i) Pokemon, SFD.

(j) CelebA-1k, SFD.

(k) Kumapi, SFD.

(l) Butterfly, SFD.

Figure 29:  Quantitative results of ReDiFine and baselines (different CFG
scales).

##  Appendix E Ablation study

This section provides an ablation study to understand how condition drop
finetuning and CFG scheduling contribute to the success of ReDiFine.

###  E.1 Condition drop finetuning

We conducted an ablation study to understand how the probability of dropping
text embedding during finetuning affects the image quality in the Chain of
Diffusion. Based on the probability used for training Stable Diffusion ($0.1$
or $0.2$), we tested $0.1$, $0.2$, and $0.4$. For both Pokemon and CelebA-1k
datasets, a probability of 0.2 seems to work the best, as shown in Figure 30
and Figure 31, respectively. Interestingly, condition drop finetuning helps to
mitigate the color saturation problem, but its effect decreases with a higher
probability. For both of these datasets, condition drop finetuning can
mitigate image degradation to some degree, but still, there is a large quality
degradation that needs to be improved.

Figure 30:  Chain of Diffusion with condition drop finetuning on Pokemon
dataset.  Figure 31:  Chain of Diffusion with condition drop finetuning on
CelebA-1k dataset.

###  E.2 CFG scheduling

We also tested how different decreasing strategies for CFG scale affect image
degradation in the Chain of Diffusion. We used two different exponential
orders for exponential decreasing, and also compared a linear decreasing
strategy. Figure 32 shows that CFG scheduling is effective for Pokemon
dataset, generating high-quality images comparable to those of ReDiFine.
However, it fails to improve image quality on CelebA-1k dataset, as shown in
Figure 33. This demonstrates the necessity of condition drop finetuning to
improve the Chain of Diffusion universally for various datasets.

Figure 32:  Chain of Diffusion with CFG scheduling on Pokemon dataset.  Figure
33:  Chain of Diffusion with CFG scheduling on CelebA-1k dataset.

##  Appendix F Analysis

In this section, we provide various analyses of images generated through the
Chain of Diffusion. Specifically, we provide frequency analysis of latents
during diffusion steps and fingerprints of synthetic images for forensic
analysis.

###  F.1 Latent analysis

(a) Histogram of latent vectors.

(b) Diffs.

(c) Power spectrum density.

Figure 34:  Fingerprints of the original training set and synthetic sets for
Pokemon dataset. There are clear differences between fingerprints of natural
images (Original) and synthetic images. ReDiFine makes similar fingerprints to
CFG scale $2.5$. (a) Autocorrelation. Horizontal and vertical lines gradually
disappear for CFG scales $1.0$ and $7.5$ while they are maintained for CFG
scale $2.5$ and ReDiFine. (b) Average power spectra. Central regions are
amplified or diminished for CFG scales $1.0$ and $7.5$, demonstrating low and
high-frequency degradation.

Figure 34 demonstrates how latent distribution evolves for different
iterations and diffusion steps on Pokemon dataset. First, the histogram of the
final latent vectors before decoding is shown in Figure 3434(a). For a CFG of
$1.0$, the latent distribution quickly evolves into a Gaussian-like shape,
with its variance decreasing over iterations. This behavior aligns with
previous work [5, 12, 19], which theoretically predicted that the self-
consuming loop would crop out the tails of the distribution, reducing output
diversity until it collapses to a single mode. We believe that these narrowing
distributions in the latent space lead to blurrier and more homogeneous
outputs in the image space. The evolution of the latent distribution is
completely opposite for $7.5$, creating longer tails and approaching a more
uniform distribution over space. For a CFG scale of $2.5$, which has the best
reusability among the three, the latent distribution is better preserved over
iterations. ReDiFine better preserves the histogram from the first to the last
iteration, achieving both first iteration fidelity and reusability.

We then plot in Figure 3434(b) how the average norm of Diff ($=\text{Cond
Score}-\text{Uncond Score}$) evolves over diffusion steps in different
iterations. In the first iteration, CFG $1.0$ has the highest Diff value,
followed by CFG $2.5$ and CFG $7.5$. It can be understood as the models’
adaptive behavior to preserve the values added to the latent vectors, Diff
multiplied by CFG scale, at each step. This trend changes in later iterations.
The Diff value for CFG $7.5$ continues to grow with each iteration, and by
iteration $6$, we observe high Diff values during the entire diffusion steps,
creating a significant gap compared to CFG $2.5$ and CFG $1.0$. We conjecture
that this accumulation is the cause of high-frequency degradation of images
for CFG $7.5$. In contrast, the Diff value for CFG $1.0$ remains relatively
stable or even decreases over iterations. The deviation of Diff among
different iterations is minimized by ReDiFine, explaining how it can preserve
image qualities for later iterations. It is noticeable that ReDiFine has much
smaller Diffs compared to the baseline with CFG scale $2.5$, comparable to CFG
scale $1.0$.

Figure 3434(c) demonstrates how the power spectrum density is maintained by
ReDiFine. While high and low CFG scales cannot preserve the power spectrum of
the first iteration as the Chain of Diffusion proceeds, ReDiFine can maintain
the density distribution during iterations. This prevents images from being
saturated to high or low-frequency degradation. Moreover, the power spectrum
is more similar to the original training set compared to CFG scale $2.5$.

In summary, it is shown that ReDiFine can successfully alleviate the symptoms
we observed in the latent space for baselines in Section 3. We can notice that
ReDiFine preserves the original latent distribution at the $6$th iteration as
compared to other baselines with different CFG values. More importantly, the
conditional-unconditional score differences (Diff) are kept at bay all the way
until the 6th iteration, as opposed to the baseline CFG $7.5$, showing a
severe discrepancy between the unconditional and conditional scores.

###  F.2 Fingerprints for forensic analysis

(a) Radial spectrum power density.

(b) Angular spectrum power density.

Figure 35:  Power spectrum density of the original training set and synthetic
sets for Pokemon dataset. Images generated by ReDiFine maintain power density
distribution during Chain of Diffusion while baselines fail. Even CFG scale
$2.5$ cannot maintain the distribution for the last iteration. (a) Radial
spectrum power density. ReDiFine shows a density distribution similar to that
of the original training set. (b) Angular spectrum power density. Power
density of generated images by ReDiFine remains during the iterations while
baselines cannot maintain angular distribution.

Fingerprints of synthetic images have distinct characteristics that differ
according to the types of generative models, architectures, and training
images. This provides unique patterns that can be used for synthetic image
detection. Figure 9 shows how fingerprints of the original training set and
synthetic images differ and how fingerprints of synthetic images change in the
Chain of Diffusion. We use $1008$ images from Pokemon dataset. Following
visual inspection, ReDiFine makes similar fingerprints to CFG scale $2.5$.
Moreover, fingerprints for ReDiFine and baseline with CFG scale $2.5$ are
maintained through the Chain of Diffusion. On the other hand, low or high CFG
scales remain different patterns to the synthetic images as iterations
proceed. Fingerprints gradually change for low or high CFG scales, showing the
effects of high or low-frequency degradation.

Figure 35 demonstrates the radial and angular spectrum power density of the
original and synthetic images. It is clearly observed that ReDiFine maintains
a similar radial spectrum power density compared to the original training set,
while even CFG scale $2.5$ fails. Moreover, it shows stable angular spectrums
during the Chain of Diffusion despite it being distinct from that of the
original training set.