# Tackling the Singularities at the Endpoints of Time Intervals in Diffusion
Models

Pengze Zhang1111Equal contribution. This work was done when Pengze Zhang was
an intern at WeChat. Hubery Yin2111Equal contribution. This work was done when
Pengze Zhang was an intern at WeChat. Chen Li2 Xiaohua Xie1222Corresponding
Author.
1Sun Yat-sen University 2Wechat, Tencent Inc.
<EMAIL_ADDRESS>{hubery<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://pangzecheung.github.io/SingDiffusion/

###### Abstract

Most diffusion models assume that the reverse process adheres to a Gaussian
distribution. However, this approximation has not been rigorously validated,
especially at singularities, where $t=0$ and $t=1$. Improperly dealing with
such singularities leads to an average brightness issue in applications, and
limits the generation of images with extreme brightness or darkness. We
primarily focus on tackling singularities from both theoretical and practical
perspectives. Initially, we establish the error bounds for the reverse process
approximation, and showcase its Gaussian characteristics at singularity time
steps. Based on this theoretical insight, we confirm the singularity at $t=1$
is conditionally removable while it at $t=0$ is an inherent property. Upon
these significant conclusions, we propose a novel plug-and-play method
SingDiffusion to address the initial singular time step sampling, which not
only effectively resolves the average brightness issue for a wide range of
diffusion models without extra training efforts, but also enhances their
generation capability in achieving notable lower FID scores.

##  1 Introduction

Diffusion models, generating samples from initial noise by learning a reverse
diffusion process, have achieved remarkable success in multi-modality content
generation, such as image generation [5, 26, 30, 35, 4, 46], audio generation
[26, 18, 28], and video generation [14]. These achievements owe much to
several fundamental theoretical research, namely Denoising Diffusion
Probabilistic Modeling (DDPM) [38, 13], Stochastic Differential Equations
(SDE) [40], and Ordinary Differential Equations (ODE) [40, 39]. These
approaches are all based on the assumption that the reverse diffusion process
shares the same functional form as the forward process. Although an indirect
validation of this assumption is given by Song et al. [40], it heavily relies
on the existence of solutions to the Kolmogorov forward and backward equations
[1], which encounters singularities at the endpoints of time intervals where
$t=0$ and $t=1$.

The singularity issue is not only a gap in the theoretical formulation of
current diffusion models, but also affects the quality of the content they
generate. Current applications simply ignore singularity points in their
implementation [42, 31, 27] and restrict the time interval to
$[\varepsilon_{1},1-\varepsilon_{2}]$. As a result, the average brightness of
the generated images typically hovers around 0 [9, 19] (normalizing brightness
to $[-1,1]$). For example, as shown in Fig. LABEL:fig:fig1, existing pre-
trained diffusion models, such as _Stable Diffusion 1.5_ (SD-1.5) [31] and
_Stable Diffusion 2.0-base_ (SD-2.0-base), fail in generating images with pure
white or black backgrounds. To address this challenge, Guttenberg et al. [9]
add extra offset noise during the training process to allow the network could
learn the overall brightness changes of the image. Unfortunately, the offset
noise usually disrupts the pre-defined marginal probability distribution and
further invalidates the original sampling formula. Lin et al. [19] re-scales
the noise schedule to enforce a zero terminal signal-to-noise ratio, and
employ the $v$-prediction technique [36] to circumvent the issue of division
by zero at $t=1$. However, this method only supports models in $v$-prediction
manner and requires substantial training to fine-tune the entire model.
Therefore, it is advantageous to devise a plug-and-play method which
effectively deals with the singularity issue for any practicable diffusion
model without extra training efforts.

In this paper, we begin with a theoretical exploration of the singularity
issue at the endpoints of time intervals in diffusion model, and then devise a
novel plug-and-play module to address the accompanying average brightness
problem in image generation. Through establishing mathematical error bounds
rigorously, we first prove the approximate Gaussian characteristics of the
reverse diffusion process at all sampling time steps, especially where the
singularity issue appears. Following this, we conduct a thorough analysis of
this approximation in the vicinity of singularities, and arrive at two
significant conclusions: 1) by computing the limit of ‘zero divided by zero’
at $t=1$, we confirm the corresponding initial singularity is removable; 2)
the singularity at $t=0$ is an inherent property of diffusion models, thus we
should adhere to this form rather than simply avoiding the singularity.
Following the aforementioned theoretical analysis, we propose the
SingDiffusion method, specifically tailored to address the challenge of
sampling during the initial singularity. Especially, this method can
seamlessly integrate into existing sampling processes in a plug-and-play
manner without requiring additional training efforts.

As demonstrated in Fig. LABEL:fig:fig1 and experiments in the appendix, our
novel plug-and-play initial sampling step can effectively resolve the average
brightness issue. It can be easily applied to a wide range of diffusion
models, including the SD-1.5 and SD-2.0-base with $\epsilon$-prediction,
SD-2.0 with $v$-prediction, and various pre-trained models available on the
CIVITAI website111https://civitai.com/, thanks to its one-time training
strategy. Furthermore, our method can generally enhance the generative
capabilities of these pre-trained diffusion models in notably improving the
FID [11] scores at the same CLIP [29] level on the COCO dataset [20].

##  2 Related Work

###  2.1 Reverse process approximation

Denoising Diffusion Probability Models (DDPM) [38, 13] establish a hand-
designed forward Markov chain in discrete time, and model the distribution of
data in the reverse process. In contrast, Song et al. [40] establish the
diffusion model in continuous time, framed as a Stochastic Differential
Equation (SDE). Moreover, they reveal an Ordinary Differential Equation (ODE)
termed ‘probability flow’, sharing the same single-time marginal distribution
as the original SDE. Notably, ODE also has discrete counterparts known as
Denoising Diffusion Implicit Models (DDIM) [39].

The assumption that the reversal of the diffusion process has an identical
functional form to the forward process is fundamental in the aforementioned
methods. Several studies aim to prove this assumption. Song et al. [40]
indirectly substantiate this in continuous time by introducing a reverse SDE.
Nevertheless, they do not provide error bounds in discrete-time cases, leaving
this assumption unverified at discrete-time steps. Additionally, the treatment
of singularities at $t=0$ and $t=1$ is not addressed in their work. McAllester
et al. [24] provide proof in discrete time, showing the density of the reverse
process as a mixture of Gaussian distributions. Despite this, it doesn’t
qualify as a pure Gaussian distribution. In contrast, to fill this theoretical
gap, our approach directly substantiates this assumption by establishing error
bounds for the reverse process approximation at both non-singular and singular
time steps.

###  2.2 Singularities in diffusion model

Several studies have focused on investigating the singularity occurring at
$t=0$. Song et al. [40] attempt to bypass this singularity by initiating their
analysis at $t=\varepsilon>0$ instead of $t=0$. Therefore, this approach
didn’t effectively address the core singularity problem. Dockhorn et al. [6]
propose a diffusion model on the product space of position and velocity, and
avoid the singularity through hybrid score matching. Nevertheless, this
approach lacks compatibility with DDPM, SDE and ODE due to the incorporation
of the velocity space. Lu et al. [22] employ the $x$-prediction method to
mitigate singularity during training, but did not address the singularity
issue during the sampling process.

Therefore, a comprehensive solution to the singularity at $t=0$ is still
pending. Moreover, the singularity at $t=1$ remains unexplored. To tackle
these issues, we thoroughly investigate the sampling process at both $t=0$ and
$t=1$, and offer theoretical solutions.

###  2.3 Average brightness issue

Diffusion models have shown significant comprehensive quality and
controllability in computer vision, including text-to-image generation [31,
30, 35, 5, 26, 12], image editing [10, 33, 16, 3, 25, 43], image-to-image
translation [34, 41, 44], surpassing previous generative models [8, 45, 7,
23]. However, most existing diffusion models ignore the sampling at the
initial singular time step, resulting in the inability to generate bright and
dark images, i.e., the average brightness issue. Adding offset noise [9] and
employing $v$-prediction [19, 36] are two ways to tackle this problem.
However, these methods require fine-tuning for each existing model, consuming
a substantial amount of time and limiting their applicability. In contrast, we
propose a novel plug-and-play solution targeting the core of the average
brightness issue, i.e., the singularity at the initial time step. Our method
not only empowers the majority of existing pre-trained models to effectively
generate images with the desired brightness level, but also significantly
enhances their generative capabilities.

##  3 Method

To facilitate the clarity of our exploration into singularities from
theoretical and practical angles, this section is organized as follows: 1) We
start by introducing background and symbols in the Preliminaries. 2) Next, we
derive error bounds for the reverse process approximation, confirming its
Gaussian characteristics at both regular and singular time steps. 3) We then
theoretically analyze and handle the sampling at singular time steps, i.e.,
$t=0$ and $t=1$. 4) Lastly, based on our previous analysis, we propose a plug-
and-play method to address initial singularity time step sampling, effectively
resolving the average brightness issue.

###  3.1 Preliminaries

In the realm of generative models, we are consistently provided with a set of
training samples denoted as $\\{y_{i}\in\mathbb{R}^{d}\\}_{i=1}^{N}$, which
are inherently characterized by a distribution given by [15]:

$p(x,t=0)=\frac{1}{N}\sum_{i=1}^{N}\delta(x-y_{i}),$ (1)

where $\delta(x)$ denotes the Dirac delta function.

Consider a continuous-time Gaussian diffusion process within the interval
$0\leq t\leq 1$. Following [17], the distribution of $x_{t}$ conditioned on
$x_{0}$ is written by
$p(x_{t}|x_{0})=\mathcal{N}(\alpha_{t}x_{0},\sigma_{t}^{2}I)$, where
$\alpha_{t}$ and $\sigma_{t}$ are positive scalar functions of $t$ satisfying
$\alpha_{t}^{2}+\sigma_{t}^{2}=1$, and $\alpha_{t}$ decreases monotonically
from 1 to 0 over time $t$; the distribution of $x_{t}$ conditioned on $x_{s}$
is represented as
$p(x_{t}|x_{s})=\mathcal{N}(\alpha_{t|s}x_{s},\sigma_{t|s}^{2}I)$, where
$0\leq s<t\leq 1$, $\alpha_{t|s}=\alpha_{t}/\alpha_{s}$,
$\sigma_{t|s}^{2}=1-\alpha_{t|s}^{2}$. Consequently, the forward process is
derived as follows:

$x_{t}=\alpha_{t|s}x_{s}+\sqrt{1-\alpha_{t|s}^{2}}z_{s},$ (2)

where the set $\\{z_{t}\\}_{t=0}^{1}$ comprises independent standard Gaussian
random variables.

For a discrete-time diffusion process with $T$ steps, the time
$i\in\\{0,1,...T\\}$ corresponds to $t$ in the continuous case as $i=t\times
T$. Defining $\hat{\beta}_{i}=1-\hat{\alpha}_{i|i-1}^{2}$, where
$\hat{\alpha}_{i|i-1}=\alpha_{i/T|(i-1)/T}$, and ‘ $\hat{}$ ’ denotes the
symbol in the discrete-time process, the forward process in Eq. 2 can be
rewritten as:

$\hat{x}_{i}=\sqrt{1-\hat{\beta}_{i}}\hat{x}_{i-1}+\sqrt{\hat{\beta}_{i}}\hat{z}_{i-1},$
(3)

which is equivalent to the forward process outlined in [13].

Taking into account the initial distribution (Eq. 1), the single-time marginal
distribution of $x_{t}$ is given by:

$p(x_{t},t)=\frac{1}{N}\sum_{i}(2\pi\sigma_{t}^{2})^{-\frac{d}{2}}\exp(-\frac{(x_{t}-\alpha_{t}y_{i})^{2}}{2\sigma_{t}^{2}}),$
(4)

where $d$ is the dimension of the training samples. As a result, the reverse
process can be derived using Bayes’ rule:

$\displaystyle
p(x_{s}|x_{t})=p(x_{t}|x_{s})\frac{p(x_{s},s)}{p(x_{t},t)}=(2\pi\sigma_{s|t})^{-\frac{d}{2}}$
(5)
$\displaystyle~{}~{}\sum_{i}\exp(-\frac{1}{2\sigma_{s|t}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{t|s}^{2}y_{i}}{\sigma_{t}^{2}})^{2})w_{i}(x_{t},t),$

where
$\sigma_{s|t}^{2}=\sigma_{t|s}^{2}\frac{\sigma_{s}^{2}}{\sigma_{t}^{2}}$, and
$w_{i}(x_{t},t)=\frac{\exp(-\frac{(x_{t}-\alpha_{t}y_{i})^{2}}{2\sigma_{t}^{2}})}{\sum_{j}\exp(-\frac{(x_{t}-\alpha_{t}y_{j})^{2}}{2\sigma_{t}^{2}})}$.

###  3.2 Error bound estimation

Existing diffusion models, such as [38, 13] are based on an assumption that
the reverse process in Eq. 5 can be approximated by a Gaussian distribution,
when $\hat{\beta}_{i}$ is small, as given by:

$\displaystyle\tilde{p}(x_{s}|x_{t})$
$\displaystyle=(2\pi\sigma_{s|t}^{2})^{-\frac{d}{2}}$ (6)
$\displaystyle\exp(-\frac{1}{2\sigma_{s|t}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{t|s}^{2}\bar{y}(x_{t},t)}{\sigma_{t}^{2}})^{2}),$

where $\bar{y}(x_{t},t)=\sum_{i}w_{i}(x_{t},t)y_{i}$. However, these studies
did not furnish the error bounds to support this assumption. To address this
theoretical gap, we estimate the error bound as follows.

######  Proposition 1 (Error Bound Estimated by $\sigma_{s|t}$).

$\forall s\in(0,1)$, $\exists\tau\in(s,1)$ and $C>0$, such that $\forall
t\in(s,\tau]$,
$\int_{\mathbb{R}^{d}}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}<C\sqrt{\sigma_{s|t}}$.

Proposition 1 demonstrates that when $\sigma_{s|t}$ is small, the forward and
reverse processes share the same form, i.e., Gaussian distribution. Since
$\hat{\beta}_{i}=\frac{\sigma_{i/T}}{\sigma_{(i-1)/T}}\sigma_{(i-1)/T|i/T}$
features a term in the $\sigma_{s|t}$ form, also supports the inference that
this assumption remains valid when $\hat{\beta}_{i}$ is small, as highlighted
in [38, 13]. It is worth noting that this error is bounded by $\sigma_{s|t}$
instead of $\hat{\beta}_{i}$ strictly.

However, the error bound estimated by $\sigma_{s|t}$ in Proposition 1 is not
sufficient to prove the assumption at the singularity time step $t=1$. The
reason is that when $t=1$, $\sigma_{s|t}$ approaches 1 as $s\to 1$, which is
not a small value. Consequently, the error bound at $t=1$ in Proposition 1
remains non-negligible.

To tackle this issue, we instead utilize $\alpha_{s}$ to bound the error at
time step $t=1$, and present a new proposition:

######  Proposition 2 (Error Bound Estimated by $\alpha_{s}$).

$\exists\nu\in(0,1)$ and $C>0$, such that $\forall\nu\leq s<t\leq 1$,
$\int_{\mathbb{R}^{d}}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}<C\sqrt{\alpha_{s}}$.

According to Proposition 2, setting $t=1$, it has $\alpha_{s}\to 0$ as $s\to
1$. As a result, the error bound at $t=1$ assures a small value, affirming the
validation of the Gaussian approximation assumption at $t=1$.

In sum, through Proposition 1 and 2, we have established that the reverse
process of the diffusion model can be approximated by Gaussian distribution
across all time steps.

###  3.3 Tackling the singularities

With the theoretical foundation provided in Section 3.2, we can delve into the
analysis of singular time steps using Eq. 6. It is worth noting that this
section will mainly focus on addressing the singularities present in the
discrete-time diffusion model [13], while the treatment of the continuous case
[40] will be deferred to the appendix.

####  3.3.1 The singularities at $t=1$

Drawing from Eq. 6, a straightforward way is to train a neural network
$\bar{y}_{\theta}(x_{t},t)$ for estimating $\bar{y}(x_{t},t)$, known as
$x$-prediction. This approach ensures that the approximated reverse process
avoids encountering a singularity at $t=1$. However, for stable training [38],
the mainstream choice among current diffusion models is $\epsilon$-prediction.
It utilizes a neural network $\epsilon_{\theta}(x_{t},t)$ for estimating
$\epsilon(x_{t},t)=\frac{x_{t}-\alpha_{t}\bar{y}(x_{t},t)}{\sigma_{t}}$, which
will encounter singularity. More specifically, substituting
$\epsilon(x_{t},t)$ for $\bar{y}(x_{t},t)$ yields the equation:

$x_{s}=\frac{\alpha_{t|s}\sigma_{s}^{2}}{\sigma_{t}^{2}}x_{t}+\frac{\alpha_{s}\sigma_{t|s}^{2}}{\sigma_{t}^{2}}\frac{x_{t}-\sigma_{t}\epsilon(x_{t},t)}{\alpha_{t}}+\sigma_{s|t}z_{t}.$
(7)

At $t=1$, the denominator $\alpha_{t}$ becomes 0, resulting in a division-by-
zero singularity. Theoretically, this singularity is removable because the
limit exists:

$\lim_{t\to
1^{-}}\frac{x_{t}-\sigma_{t}\epsilon(x_{t},t)}{\alpha_{t}}=\lim_{t\to
1^{-}}\bar{y}(x_{t},t)=\frac{1}{N}\sum_{i}y_{i}.$ (8)

Regrettably, computing this limit during inference is unfeasible, since
$\epsilon(x_{1},1)=x_{1}$ loses all information of the correct sampling
direction. Conversely, $\bar{y}(x_{t},t)$ retains the correct sampling
direction for all $t\in[0,1]$. Particularly at $t=1$,
$\bar{y}(x_{1},1)=\frac{1}{N}\sum_{i}y_{i}$ encapsulates the information for
the initial inference step. Therefore, leveraging $x$-prediction at the
initial time step proves more advantageous than $\epsilon$-prediction.

####  3.3.2 The singularities at $t=0$

When $s=0$ and $t$ is small, the distribution in Eq. 6 degenerates into a
singular distribution, i.e., a Gaussian with zero variance, resembling a Dirac
delta:

$\displaystyle\tilde{p}(x_{0}|x_{t})=\delta(x_{0}-y_{j_{0}}),$ (9)

where $j_{0}=\mathop{\arg\min}\limits_{j}|x_{t}-\alpha_{t}y_{j}|$. This
singularity directs the sampling process to converge at the correct point
$y_{j_{0}}=\bar{y}(x_{0},0)$. Therefore, the singularity at $t=0$ is an
inherent characteristic of diffusion models that do not require avoidance as
long we use suitable sampling techniques. For instance, in the final step of
the original DDPM sampling method, it has $x_{0}=\bar{y}(x_{t},t)$. When $t$
is small, $\bar{y}(x_{t},t)\approx\bar{y}(x_{0},0)$, making this process
equivalent to Eq. 9. Thus, there is no need to avoid the singularity.

Besides, we also arrived at this conclusion within the continuous diffusion
model, i.e., SDE. More detailed elaboration on this topic can be found in the
appendix.

Figure 1: Illusion of the gap between the sampling from $x_{1-\varepsilon}$
and $x_{1}$. Due to the lack of consideration of singular time step sampling
in most of the existing methods, they will encounter the average brightness
issue. To tackle this, we propose a plug-and-play SingDiffusion method
(highlighted in red) to bridge this gap.

###  3.4 SingDiffusion

In addition to theoretical issues, the singular sampling issue can also cause
average brightness issue in applications, as depicted in Fig. 1. This is
mainly because most of the existing method sample images starting at time step
$1-\varepsilon$ using a standard Gaussian distribution, which significantly
diverges from the true distribution $p(x_{1-\varepsilon},1-\varepsilon)$. To
validate this, we select two images (a black dog against a black background
and a white dog against a white background). Then we diffuse them for a time
period of $1-\varepsilon$ respectively, and calculate the mean value of their
latent code. It can be seen that the mean of these two latent codes are -0.14
and +0.28, which are significantly different from the samples obtained from
the standard Gaussian distribution. Moreover, as evident from the
visualization in Fig. 1, the latent code ($x_{1-\varepsilon}$) of the black
dog and white dog are notably darker and brighter compared to $x_{1}$ from the
Gaussian distribution respectively. Under such distributional differences,
according to Proposition 3, employing a Gaussian distribution at
$t=1-\epsilon$ is equivalent to generating images towards an average
brightness of 0 at $t=1$ (with grayscale ranging from [-1, 1]). Consequently,
current methods encounter challenges in generating dark or bright images.

######  Proposition 3.

Setting $x_{1-\epsilon}\sim\mathcal{N}(0,I)$ is equivalent to sampling the
value from standard Gaussian as $\bar{y}(x_{1},1)$ at $t=1$.

Based on our analysis of sampling at singular time steps in Section 3.3.1, we
propose a novel plug-and-play method SingDiffusion to fill the gap at $t=1$,
thus solving the problem of average brightness issue. Considering a pre-
trained model $\epsilon_{\theta}(x_{t},t)$ afflicted by the singularity due to
division by zero, we proceed to train a model using $x$-prediction at $t=1$.
The algorithm of our training and sampling process are shown in Algorithm 1
and Algorithm 2. Firstly, for image-prompt data pairs ($x_{0},c$) in the
training process, we use a U-net [32] $\bar{y}_{\theta}$ to fit
$\bar{y}(x_{1},t=1,c)$, where $\bar{y}(x_{t},t,c)$ is the extension of
$\bar{y}(x_{t},t)$ defined in Section 6 under the condition $c$. The loss
function can be written as:

${\mathcal{L}={\mathbb{E}_{x_{0},c\sim
p(x_{0},c),x_{1}\sim\mathcal{N}(\bm{0},\bm{I})}||\bar{y}_{\theta}(x_{1},c)-x_{0}||}^{2}}.$
(10)

As our model is only amortized with respect to text embeddings and not the
time step, we omit the variable $t$ for $\bar{y}_{\theta}$, and the U-net does
not necessitate a complex architecture.

In the sampling process, we use the new model $\bar{y}_{\theta}$ in the
initial time-step with a DDIM scheduler:

$x_{1-\varepsilon}=\alpha_{1-\varepsilon}\bar{y}_{\theta}(x_{1},c)+\sigma_{1-\varepsilon}x_{1}.$
(11)

This equation guarantees $x_{1-\varepsilon}$ adheres to the distribution
$p(x_{1-\varepsilon},1-\varepsilon)$. Following this, we utilize the existing
pre-trained model $\epsilon_{\theta}(x_{t},t,c)$/$v_{\theta}(x_{t},t,c)$ in
$\epsilon$-prediction/$v$-prediction manner to perform the subsequent sampling
steps until it generates $x_{0}$. It’s worth noting that our method is solely
involved in the sampling at $t=1$, independent of the subsequent sampling
process. Consequently, our approach can be once-trained and seamlessly
integrated into the majority of diffusion models.

1:repeat

2: $x_{0},c\sim p(x_{0},c)$, $x_{1}\sim\mathcal{N}(\bm{0},\bm{I})$

3: Take gradient descent step on
$\nabla_{\theta}\left\|\bar{y}_{\theta}(x_{1},c)-x_{0}\right\|^{2}$

4:until converged

Algorithm 1 Training process of SingDiffusion

1:$x_{1}\sim\mathcal{N}(\bm{0},\bm{I})$

2:$\varepsilon=1/T$

3:$x_{1-\varepsilon}=\alpha_{1-\varepsilon}\bar{y}_{\theta}(x_{1},c)+\sigma_{1-\varepsilon}x_{1}$

4:for $t=1-\varepsilon,\dotsc,\varepsilon$ do

5: Calculate $x_{t-\varepsilon}$ using existing sampling algorithms

6:end for

7:return $x_{0}$

Algorithm 2 Sampling process of SingDiffusion

For further improving matching between generated images and input prompts,
existing diffusion models typically incorporate classifier-free techniques
[26]:

$o_{guidance}=o_{neg}+w\times(o_{pos}-o_{neg}),$ (12)

where $w\geq 1$ represents the guidance scale, $o$ signifies the output of the
diffusion model which can be either $\bar{y}_{\theta}$, $\epsilon_{\theta}$ or
$v_{\theta}$, ‘pos’ and ‘neg’ refer to the outputs corresponding to positive
and negative prompts respectively. Nevertheless, we notice that when applying
this technique at the initial singular time step, the influence of the
guidance scale $w$ predominates the results due to the greater directional
difference between the negative and positive outputs. To tackle this
challenge, we implement a straightforward yet highly effective normalization
method:

${\bar{y}_{\theta_{guidance}}=[\bar{y}_{\theta_{neg}}+w\times(\bar{y}_{\theta_{pos}}-\bar{y}_{\theta_{neg}})]/w.}$
(13)

Figure 2: Comparison of stable diffusion models and SingDiffusion on average
brightness issue.

$29$$29.5$$30$$30.5$$31$$10$$11$$12$$13$$14$$15$$16$$17$CLIP score
$\uparrow$FID score $\downarrow$SD-1.5\+ SingDiffusion

$29.5$$30$$30.5$$31$$31.5$$9$$10$$11$$12$$13$$14$CLIP score $\uparrow$FID
score $\downarrow$SD-2.0-base\+ SingDiffusion

Figure 3: Comparison of Pareto curves between SingDiffusion, SD-1.5, and
SD-2.0-base on 30k COCO images, across various guidance scales in [1.5, 2, 3,
4, 5, 6, 7, 8]. Figure 4: Our method can be trained once and seamlessly
integrated into the pre-trained models on CIVITAI in a plug-and-play fashion.

##  4 Experiment

###  4.1 Implement details

To create a versatile plug-and-play model, our model is trained on the
Laion2B-en dataset [37], including 2.32 billion text-image pairs. The images
are center-cropped to 512 $\times$ 512\. Our $\bar{y}_{\theta}$ follows the
U-net structure similar to stable diffusion models but with reduced
parameters, totaling 140 million. We utilize the AdamW optimizer [21] with a
learning rate of 1e-4 to train our $\bar{y}_{\theta}$. The training is
executed across 64 Nvidia V100 chips with a batch size of 3072, and completes
after 180K steps.

During the testing process, all steps, including our initial sampling, are
carried out using the default guidance scale of $w=7.5$. After our initial
sampling at the singular time step, all existing pre-trained diffusion models
generate images in the DDIM pipeline, executing 50 steps to produce images
following a schedule with a total time of $T=1000$.

Table 1: Comparison of average brightness of 100 generated images between stable diffusion models and our SingDiffusion under different prompt conditions. For ’white’/’black’ prompts, higher/lower average brightness is better. Model |  | ”Solid white
---
background”
| ”Solid black
---
background”
| ”Monochrome
---
line-art logo on a
white background”
| ”Monochrome
---
line-art logo on a
black background”
SD-1.5 | 141.43 | 83.09 | 137.95 | 113.66
\+ Ours | 212.59 | 3.04 | 223.68 | 11.52
SD-2.0-base | 150.52 | 99.67 | 136.13 | 104.45
\+ Ours | 227.43 | 0.29 | 228.68 | 10.87
|  |  |  |

###  4.2 Average brightness issue

To validate the effectiveness of SingDiffusion in addressing the average
brightness issue, we select four extreme prompts, including “Solid white/black
background”, and “Monochrome line-art logo on a white/black background”. For
each prompt, we generate 100 images using SD-1.5, SD-2.0-base and
SingDiffusion methods, and then calculate their average brightness. The
results are shown in Table 1. It is remarkably clear that the stable diffusion
methods, whether using prompts with ‘black’ or ‘white’ descriptors, tend to
generate images with average brightness. However, SingDiffusion, implementing
initial singularity sampling, effectively corrects the average brightness of
the generated images. For example, under the prompt “solid black background,”
our method notably lowers the average brightness from 99.67 to 0.29 for images
generated by SD-2.0-base.

Moreover, we also provide visualization results for these prompts in Fig. 2.
It can be seen that images generated by stable diffusion methods predominantly
exhibit a gray tone. In contrast, our SingDiffusion method successfully
overcomes this issue, and is capable of generating both dark and bright
images.

###  4.3 Improvement on image quality

To validate our improvements in general image quality, following [35], we
randomly select 30k prompts from the COCO dataset [20] as the test set. We
employ two metrics for evaluation, including the FID score and CLIP score.
Specifically, Fréchet Inception Distance (FID) [11] calculates the Fréchet
distance between the real data and the generated data. A lower FID implies
more realistic generated data. While the Contrastive Language-Image Pre-
training (CLIP) [29] score measures the similarity between the generated
images and the given prompts. A higher CLIP score means the generated images
better match the input prompts.

Table 2: Comparison of stable diffusion model and SingDiffusion on FID score and CLIP score without classifier guidance. Model | SD-1.5 | SD-2.0-base
---|---|---
5 |  |  |  |
| FID $\downarrow$ | CLIP $\uparrow$ | FID $\downarrow$ | CLIP $\uparrow$
Original | 31.86 | 26.70 | 25.17 | 27.48
\+ SingDiffusion | 21.09 | 27.71 | 18.01 | 28.23
|  |  |  |

First of all, we compare SingDiffusion with SD-1.5 and SD-2.0-base in Table 2
to gauge the model’s inherent fitting capability, without using guidance-free
techniques. It is evident that SingDiffusion significantly outperforms the
existing stable diffusion methods in both FID and CLIP scores. These results
highlight the yet-to-be-utilized fitting potential in current stable diffusion
models, emphasizing the significance of sampling at the initial singular time
step.

Furthermore, inspired by Imagen [35], we plot CLIP v.s. FID Pareto curves by
varying guidance values within the range [1.5, 2, 3, 4, 5, 6, 7, 8] in Fig. 3.
SingDiffusion exhibits substantial improvements over stable diffusion models,
especially noticeable with smaller guidance scales. As the guidance scale
increases, SingDiffusion consistently maintains a lower FID compared to stable
diffusion for achieving a similar CLIP score. This emphasizes that our
approach not only enhances image realism but also ensures better adherence to
the input prompts.

###  4.4 Plug-and-play on other pre-trained models

Since our method is extensively trained on the Laion dataset, it can be easily
integrated into the majority of existing diffusion models. To validate this,
we download several pre-trained models from the CIVITAI website, each
specializing in different image domains like anime, animals, clothing, and so
on. To facilitate the application of SingDiffusion to these models, we
integrate SingDiffusion into the popular Stable Diffusion Web UI [2] system,
and sample images with the Euler ancient method after using our initial
singularity sampling process. The outcomes, displayed in Fig. LABEL:fig:fig1
and Fig. 4, showcase SingDiffusion effectively resolving the average
brightness issue across all models while preserving their original generative
capacities. This demonstrates the adaptability and practicality of our
approach.

$29.4$$29.6$$29.8$$30$$30.2$$30.4$$30.6$$30.8$$31$$31.2$$31.4$$10$$11$$12$$13$$14$$15$$16$CLIP
score $\uparrow$FID score $\downarrow$SingDiffusion w/o normSingDiffusion
Figure 5: Ablation study of guidance normalization on SD-1.5 across various
guidance scales in [1.5, 2, 3, 4, 5, 6, 7, 8]. Figure 6: Visualization of
${\bar{y}_{\theta}}$ for various prompts.

###  4.5 Effect on classifier guidance normalization

We conduct an ablation study on the guidance normalization operation (Eq. 13),
and represent the CLIP v.s. FID Pareto curves in Fig. 5. It can be seen that
the method without normalization exhibits inferior results compared with the
full method. As the guidance scale increases, the gap in FID between these two
methods gradually widens. This phenomenon primarily arises from significant
disparities between ${\bar{y}_{{\theta}_{pos}}}$ and
${\bar{y}_{{\theta}_{neg}}}$. According to Eq. 13, larger guidance scales may
lead to overflow issues. In contrast, normalizing the results with the
guidance scale helps keep the outputs within a typical range, thus restoring
the FID score.

Figure 7: SingDiffusion integrates seamlessly with ControlNet.

###  4.6 Visualization of the ${\bar{y}_{\theta}}$

According to Eq. 10, our main goal with $\bar{y}_{\theta}$ is to model the
average image corresponding to each prompt. To confirm this, we employ the
prompts presented in Fig. LABEL:fig:fig1 and visualize their corresponding
$\bar{y}_{\theta}$. As demonstrated in Fig. 6, it is clear that
$\bar{y}_{\theta}$ does indeed represent a smoothed average image. For
instance, the first four images of the second row resemble average faces,
aligning with the input prompt for generating celebrities. Additionally, we
notice that $\bar{y}_{\theta}$ adapts to the prompt’s brightness suggestion,
appearing brighter or darker accordingly. This highlights our method’s ability
to effectively capture pertinent lighting conditions, and underscores the
significance of the initial singular time step sampling.

###  4.7 Application on ControlNet

Our model can seamlessly integrate with existing diffusion model plugins, such
as ControlNet [44]. Since our $\bar{y}_{\theta}$ is structurally different
from ControlNet, ControlNet is adopted after our initial sampling step. The
results in Fig. 7 demonstrate that our method is fully compatible with
ControlNet and effectively resolves its average brightness issue.

##  5 Conclusion

In this study, we delve into the singularities of time intervals in diffusion
models, exploring both theoretical and practical aspects. Firstly, we
demonstrate that at both regular and singular time steps, the reverse process
can be approximated by a Gaussian distribution. Leveraging these theoretical
insights, we conduct an in-depth analysis of singular time step sampling and
propose a theoretical solution. Finally, we introduce a novel plug-and-play
method SingDiffusion, addressing the initial time-step sampling challenge.
Remarkably, this module substantially mitigates the average brightness issue
prevalent in most current diffusion models, and also enhances their generative
capabilities.

##### Acknowledgments.

This project is supported by the National Natural Science Foundation of China
(62072482, U22A2095)

## References

  * Anderson [1982] Brian D.O. Anderson.  Reverse-time diffusion equation models.  _Stochastic Processes and their Applications_ , pages 313–326, 1982.
  * AUTOMATIC1111 [2022] AUTOMATIC1111.  Stable diffusion web ui.  https://github.com/AUTOMATIC1111/stable-diffusion-webui, 2022.
  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18208–18218, 2022.
  * Campbell et al. [2022] Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet.  A continuous time framework for discrete denoising models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 8780–8794, 2021.
  * Dockhorn et al. [2022] Tim Dockhorn, Arash Vahdat, and Karsten Kreis.  Score-based generative modeling with critically-damped langevin diffusion.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12873–12883, 2021.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , 2014.
  * Guttenberg [2023] Nicholas Guttenberg.  Diffusion with offset noise.  https://www.crosslabs.org/blog/diffusion-with-offset-noise, 2023.
  * Hertz et al. [2023] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or.  Prompt-to-prompt image editing with cross-attention control.  In _International Conference on Learning Representations (ICLR)_ , 2023.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_ , 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 6840–6851, 2020.
  * Ho et al. [2022] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Video diffusion models.  In _ICLR Workshop on Deep Generative Models for Highly Structured Data_ , 2022.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 26565–26577, 2022.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6007–6017, 2023.
  * Kingma et al. [2021] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 21696–21707, 2021.
  * Kong et al. [2021] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Lin et al. [2023] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang.  Common diffusion noise schedules and sample steps are flawed.  _arXiv preprint arXiv:2305.08891_ , 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European Conference on Computer Vision (ECCV)_ , pages 740–755, 2014.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Lu et al. [2023] Yubin Lu, Zhongjian Wang, and Guillaume Bal.  Mathematical analysis of singularities in the diffusion model under the submanifold assumption.  _arXiv preprint arXiv:2301.07882_ , 2023.
  * Lugmayr et al. [2020] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In _European Conference on Computer Vision (ECCV)_ , page 715–732, 2020.
  * McAllester [2023] David McAllester.  On the mathematics of diffusion models.  _arXiv preprint arXiv:2301.11108_ , 2023.
  * Meng et al. [2022] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning (ICML)_ , pages 16784–16804, 2022.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Popov et al. [2021] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov.  Grad-tts: A diffusion probabilistic model for text-to-speech.  In _International Conference on Machine Learning (ICML)_ , pages 8599–8608, 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning (ICML)_ , pages 8748–8763, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning (ICML)_ , pages 8821–8831, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , pages 234–241, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 22500–22510, 2023.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM SIGGRAPH Conference Proceedings_ , 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 36479–36494, 2022b.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 25278–25294, 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning (ICML)_ , pages 2256–2265, 2015.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations (ICLR)_ , 2021a.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations (ICLR)_ , 2021b.
  * Su et al. [2023] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon.  Dual diffusion implicit bridges for image-to-image translation.  In _International Conference on Learning Representations (ICLR)_ , 2023.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Yang et al. [2023] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen.  Paint by example: Exemplar-based image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18381–18391, 2023.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3836–3847, 2023a.
  * Zhang et al. [2022] Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie.  Exploring dual-task correlation for pose guided person image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7713–7722, 2022.
  * Zhang et al. [2023b] Pengze Zhang, Hubery Yin, Chen Li, and Xiaohua Xie.  Formulating discrete probability flow through optimal transport.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023b.

Supplementary Material

##  Appendix A Singularities for SDE

In this section, we’ll address the singularities within the continuous
diffusion process of the SDE. The SDE is represented by the Ornstein-Uhlenbeck
equation:

$dx_{t}=f(t)x_{t}dt+g(t)dB_{t},$ (14)

where $f(t)$ is the drift coefficient, $g(t)$ is the diffusion coefficient,
and $B_{t}$ is the Brownian Motion. The solution to Eq. 14 is:

$\displaystyle x_{t}=$
$\displaystyle\exp(\int_{s}^{t}f(\tau)d\tau)x_{s}\\!+\\!\int_{s}^{t}g(\nu)\exp(-\\!\int_{\nu}^{t}f(\tau)d\tau)dB_{\nu}$
(15) $\displaystyle=$
$\displaystyle\frac{\alpha_{t}}{\alpha_{s}}x_{s}+(\int_{s}^{t}g^{2}(\nu)\frac{\alpha_{t}^{2}}{\alpha_{\nu}^{2}}d\nu)^{\frac{1}{2}}B_{1},$

where $\alpha_{t}=\exp(\int_{0}^{t}f(\tau)d\tau)$. In order to maintain a
constant variance and achieve convergence to a standard Gaussian distribution,
it is necessary to satisfy the conditions of $f(t)=-\frac{1}{2}g^{2}(t)$ and
$\lim_{t\to 1}f(t)=-\infty$. By fulfilling these requirements, Eq. 15 can be
rewritten as follows:

$x_{t}=\alpha_{t|s}x_{s}+\sigma_{t|s}B_{1},$ (16)

where $\alpha_{t|s}=\alpha_{t}/\alpha_{s}$,
$\sigma_{t|s}=\sqrt{1-\alpha_{t|s}^{2}}$. The single-time marginal
distribution $p(x_{t},t)$ of this process remains consistent with Eq. 4, while
the forward conditional distribution $p(x_{t}|x_{s})$ is given by:

$\displaystyle
p(x_{t}|x_{s})=(2\pi\sigma_{t|s}^{2})^{-\frac{1}{2}}\exp(-\frac{(x_{t}-\alpha_{t|s}x_{s})^{2}}{2\sigma_{t|s}^{2}}).$
(17)

According to [40, 1], the reverse-time SDE is:

$dx_{t}=(f(t)x_{t}-g^{2}(t)\nabla_{x_{t}}p(x_{t},t))dt+g(t)d\tilde{B}_{t},$
(18)

where $\tilde{B}_{t}$ is a reverse time Brownian motion. The term
$\nabla_{x_{t}}p(x_{t},t)$, is metiond as the score function:

$\displaystyle
s(x_{t},t):=\nabla_{x_{t}}p(x_{t},t)=\sum_{i}w_{i}(x_{t})\frac{\alpha_{t}y_{i}-x_{t}}{\sigma_{t}^{2}},$
(19)

where
$w_{i}(x_{t},t)=\frac{\exp(-\frac{(x_{t}-\alpha_{t}y_{i})^{2}}{2\sigma_{t}^{2}})}{\sum_{j}\exp(-\frac{(x_{t}-\alpha_{t}y_{j})^{2}}{2\sigma_{t}^{2}})}$.
This score function can be fitted by a neural network $s_{\theta}(x_{t},t)$
using score matching.

###  A.1 The singularity at $t=1$

####  A.1.1 Singularity in SDE Solution

To solve the SDE, the initial step involves rewriting the Eq. 14 as follows:

$\displaystyle d\frac{x_{t}}{\alpha_{t}}$
$\displaystyle=-f(t)\frac{x_{t}}{\alpha_{t}}dt+\frac{1}{\alpha_{t}}(f(t)x_{t}dt+g(t)dB_{t})$
(20) $\displaystyle=\frac{g(t)}{\alpha_{t}}dB_{t}.$

However, it’s crucial to note that a singularity occurs when $\alpha_{1}=0$,
leading to a division-by-zero issue. Consequently, Eq. 16 and Eq. 17 may not
hold at $t=1$. Nonetheless, these equations remain valid for all $s\in[0,1)$,
allowing us to deduce the following:

######  Proposition 4.

$\forall 0\leq s<t\leq 1$, the probability density $p(x_{t}|x_{s})$ converges
to $\mathcal{N}(\bm{0},\bm{I})$ point-wisely, as $t\to 1$.

######  Corollary 1.

The probability density $p(x_{t},t)$ converges to $\mathcal{N}(\bm{0},\bm{I})$
point-wisely, as $t\to 1$.

These claims demonstrate that the Eq. 16 and Eq. 17 can be extended to the
interval $[0,1]$.

####  A.1.2 Singularity in Kolmogorov Equations

According to [1], the reverse process in Eq. 18 relies on the Kolmogorov
forward equation and backward equation:

$\displaystyle\frac{\partial{p(x_{t}|x_{s})}}{\partial{t}}=-f(t)\nabla_{x_{t}}(x_{t}p(x_{t}|x_{s}))+\frac{1}{2}g^{2}(t)\Delta
p(x_{t}|x_{s}),$ (21)
$\displaystyle\frac{\partial{p(x_{t}|x_{s})}}{\partial{s}}=-f(t)x_{t}\nabla_{x_{t}}p(x_{t}|x_{s})-\frac{1}{2}g^{2}(t)\Delta
p(x_{t}|x_{s}).$

However, as $-f(1)=\frac{1}{2}g^{2}(1)=\infty$, these equations may not hold
at $t=1$. As a result, the reverse process (18) may not establish at $t=1$.

####  A.1.3 Singularity in Discretization

With the trained network $s_{\theta}(x_{t})$, [40] introduce a sampling
technique referred to as _reverse diffusion sampling_. It discretizes the Eq.
18 using the Euler–Maruyama method:

$\displaystyle
x_{s}=x_{t}-(f(t)x_{t}-g^{2}(t)s_{\theta}(x_{t}))(t-s)+g(t)\sqrt{t-s}z_{t}.$
(22)

As $-f(1)=\frac{1}{2}g^{2}(1)=\infty$ and $s_{\theta}(x_{1})\approx x_{1}$,
the right-hand side of this equation becomes $\infty-\infty$ at $t=1$. This
observation highlights the limitation of naively discretizing Eq. 18 near
$t=1$.

####  A.1.4 Tackling the Singularity

To mitigate the singularity challenge in the Kolmogorov equations and the
reverse SDE, we scrutinize the reverse conditional distribution at $t=1$,
denoted as $p(x_{s}|x_{1})$. Given that Proposition 4 and Corollary 1 affirm
the equivalence of both $p(x_{1},1)$ and $p(x_{1}|x_{s})$, following a
standard Gaussian distribution, we deduce:

$p(x_{s}|x_{1})=p(x_{1}|x_{s})\frac{p(x_{s},s)}{p(x_{1},1)}=p(x_{s},s),$ (23)

which holds for all $s\in[0,1]$. Therefore, we successfully solved the reverse
equation at $t=1$, circumventing the Kolmogorov equations.

We now possess the necessary tools to overcome the singularity present in Eq.
22. By setting $t=1$ in proposition 2, we obtain the following corollary:

######  Corollary 2.

$\exists\tau\in[0,1)$ and $C>0$, such that $\forall t\in[\tau,1]$,
$\int_{\mathbb{R}^{d}}|p(x_{t},t)-\tilde{p}(x_{t},t)|dx_{t}<C\sqrt{\alpha_{t}}$,
where

$\displaystyle\tilde{p}(x_{t},t)=(2\pi\sigma_{t}^{2})^{-\frac{1}{2}}\exp(-\frac{(x_{t}-\alpha_{t}\bar{y}(x_{t},t))^{2}}{2\sigma_{t}^{2}})$
(24)

As a result, at $t=1$, we’re able to generate a step by employing Eq. 23 and
Eq. 24, followed by sampling the subsequent steps utilizing the reverse
diffusion sampling method outlined in Eq. 22. Remarkably, this process aligns
with the approach we employ to tackle the singularity in DDPM.

It is worth to note that Eq. 23 also offers additional information as follows:

######  Proposition 5.

The generated sample $x_{0}$ is independent with the initial noise point
$x_{1}$

Consider the _sample space_ as the set formed by the diffusion model’s
outputs. Within this framework, we can assert the following corollary:

######  Corollary 3.

Given any fixed initial noise point ${x_{1}}_{fix}$, the entire sample space
can be generated with the reverse diffusion sampling method, or equivalently,
the DDPM sampling method.

###  A.2 The singularity at $t=0$

Similar to DDPM, the SDE also encounters singularities at $t=0$. These
singularities stem from the score function:

$\displaystyle s(x_{t},t)$ $\displaystyle=\nabla_{x_{t}}\log
p(x_{t},t)=-\sum_{i}w_{i}(x_{t})\frac{x_{t}-\alpha_{t}y_{i}}{\sigma_{t}^{2}}$
(25)
$\displaystyle=-\frac{x_{t}-\alpha_{t}\sum_{i}w_{i}(x_{t})y_{i}}{\sigma_{t}^{2}}=-\frac{x_{t}-\alpha_{t}\bar{y}(x_{t},t)}{\sigma_{t}^{2}}.$

Let $j_{0}=\mathop{\arg\min}\limits_{j}|x_{t}-\alpha_{t}y_{j}|$, we have

$\displaystyle\lim_{t\to
0^{+}}-\sum_{i}w_{i}(x_{t})\frac{(x_{t}-\alpha_{t}y_{i})}{\sigma_{t}^{2}}$
(26) $\displaystyle=\lim_{t\to
0^{+}}\sum_{i}\frac{-\frac{(x_{t}-\alpha_{t}y_{i})}{\sigma_{t}^{2}}}{\sum_{j}\exp\frac{1}{2\sigma_{t}^{2}}((x_{t}-\alpha_{t}y_{i})^{2}-(x_{t}-\alpha_{t}y_{j})^{2})}$
$\displaystyle=\begin{cases}0,&\text{if}~{}~{}x_{t=0}=y_{j_{0}},\\\
(x_{j_{0}}-x_{t=0})*\infty,&\text{if}~{}~{}x_{t=0}\neq y_{j_{0}},\end{cases}$

where $(x_{j_{0}}-x_{t=0})*\infty$ signifies that the vector is oriented in
the direction of $x_{j_{0}}-x_{t=0}$ and possesses an infinite norm. This
Equation demonstrates that as $t$ approaches 0, $x_{t}$ is driven towards the
nearest training sample. This aligns with the discrete scenario depicted in
Eq. 9. Despite the presence of singularities in the score function, it guides
the sampling process toward convergence at a specific point. Consequently, the
behavior near $t=0$ becomes predictable, indicating that the singularities do
not need avoidance if we employ suitable sampling techniques. Similar to DDPM,
the final sampling step is $x_{0}=\bar{y}(x_{t},t)$.

##  Appendix B Singularities for Probability Flow ODE

###  B.1 Singularites for the Reverse Process

The probability flow is defined by [40]:

$dx_{t}=(f(t)x_{t}-\frac{1}{2}g^{2}(t)s(x_{t},t))dt.$ (27)

Expanding the expression for $s(x_{t},t)$, we obtain:

$\frac{dx_{t}}{dt}=-\frac{\alpha_{t}\alpha_{t}^{\prime}}{1-\alpha_{t}^{2}}x_{t}+\frac{\alpha_{t}^{\prime}}{1-\alpha_{t}^{2}}\bar{y}(x_{t},t),$
(28)

where $\alpha_{t}^{\prime}$ represents the time derivative of $\alpha_{t}$. In
practical implementations, the design of $\alpha_{t}$, such as the cosine
function, linear function, etc., does not result in an infinite derivative at
$t=1$. Consequently, there are no singularities present at $t=1$ by using
$x$-prediction for the ODE.

At $t=0$, the score function will drive $x_{t}$ towards the nearest training
sample, which aligns with the behavior described in section A.2.

###  B.2 The score function can not be Lipschitz continuous at $t=0$

Several previous papers have mentioned the Lipschitz problem of the ODE near
$t=0$. However, it is important to note that this behavior is not a “problem”
but rather a characteristic of the ODE.

We can disprove the assumption of the ODE’s Lipschitz continuity at $t=0$
through a counter-proof. As $s(x_{t},t)$ is Lipschitz continuous in
$[\epsilon,1],\forall\epsilon>0$, if we assume the ODE to be Lipschitz
continuous near $t=0$, then it will be continuous throughout the interval
$[0,1]$. According to the Picard-Lindelöf theorem, the solution $x_{t}$
remains unique on $[0,1]$ for any initial condition. Consequently, the single-
time marginal distribution $p(x_{t},t)$ consistently comprises a sum of Dirac
deltas, given the initial distribution (Eq. 1) as a sum of Dirac deltas.
However, this leads to a discrepancy with the expected form in Eq. 4,
resulting in a contradiction. Consequently, the right-hand side of Eq. 27
cannot exhibit Lipschitz continuity near $t=0$. This leads us to the following
proposition:

######  Proposition 6.

The right-hand side of Eq. 27 and the score function $s(x_{t},t)$ cannot be
Lipschitz continuous across the space $\mathbb{R}^{d}\times[0,1]$.

The absence of Lipschitz continuity implies the potential existence of
multiple solutions for a given initial condition. These solutions must adhere
to specific distributions to ensure the single-time marginal distribution in
Eq. 4. Consequently, in the reverse process, different $x_{t}$ values will
converge to the points within the training samples $\\{y_{i}\\}_{i=1}^{N}$.

##  Appendix C Singularities for DDIM

Since DDIM is a discretization of the ODE [39], according to Section B.1,
there are no singularities by using $x$-prediction at $t=1$. We can obtain
DDIM sampling by discretizing Eq. 28 as follows:

$\displaystyle x_{t}-x_{s}=$
$\displaystyle-\frac{\alpha_{t}}{\sigma_{t}^{2}}(\alpha_{t}-\alpha_{s})x_{t}+\frac{\alpha_{t}-\alpha_{s}}{\sigma_{t}^{2}}\bar{y}(x_{t},t)$
(29) $\displaystyle+o(|\alpha_{t}-\alpha_{s}|),$

Rearranging the formula, we have:

$\displaystyle x_{s}$
$\displaystyle=\frac{\alpha_{s}-\alpha_{t}}{\sigma_{t}^{2}}\bar{y}(x_{t},t)+\frac{1-\alpha_{s}\alpha_{t}}{\sigma_{t}^{2}}x_{t}+o(|\alpha_{t}-\alpha_{s}|)$
(30)
$\displaystyle=(\alpha_{s}-\alpha_{t}\frac{\sqrt{\sigma_{s}^{2}\sigma_{t}^{2}+(\alpha_{t}-\alpha_{s})^{2}}}{\sigma_{t}^{2}})\bar{y}(x_{t},t)$
$\displaystyle~{}~{}~{}~{}+\frac{\sqrt{\sigma_{s}^{2}\sigma_{t}^{2}+(\alpha_{t}-\alpha_{s})^{2}}}{\sigma_{t}^{2}}x_{t}+o(|\alpha_{t}-\alpha_{s}|)$
$\displaystyle=(\alpha_{s}-\frac{\sigma_{s}}{\sigma_{t}}\alpha_{t})\bar{y}(x_{t},t)+\frac{\sigma_{s}}{\sigma_{t}}x_{t}+o(|\alpha_{t}-\alpha_{s}|).$

In Algorithm 2, we utilize DDIM to sample the initial step of the reverse
process.

##  Appendix D Proofs

######  Lemma 1.

$\forall~{}1\leq i\leq N$ and $t\in[0,1]$, there is a constant $M$ such that

$|y_{i}-\bar{y}(x_{t},t)|\leq M$ (31)

and

$|\bar{y}(x_{t},t)|\leq M.$ (32)

##### Proof:

Let $M_{1}=\max\\{|y_{i}-y_{j}|:1\leq i,j\leq N\\}$ and
$M_{2}=\max\\{|y_{i}|\\}$.

$\displaystyle|y_{i}-\bar{y}(x_{t},t)|\leq\sum_{j}w_{j}(x_{t},t)|y_{i}-y_{j}|\leq
M_{1},$ (33)

since $\sum_{i}w_{i}(x_{t},t)=1$. And then

$|\bar{y}(x_{t},t)|\leq M_{1}+\max\\{|y_{i}|\\}=M_{1}+M_{2}.$ (34)

$M=M_{1}+M_{2}$ is the constant. $\square$

######  Lemma 2.

$\forall 0\leq s<t\leq 1$ and $s$ fixed, $\sigma_{s|t}$ is non-decreasing with
respect to $t$.

##### Proof:

$\displaystyle\sigma_{s|t}^{2}=$
$\displaystyle(1-\frac{\alpha_{t}^{2}}{\alpha_{s}^{2}})\frac{1-\alpha_{s}^{2}}{1-\alpha_{t}^{2}}$
(35) $\displaystyle=$
$\displaystyle\frac{1-\alpha_{s}^{2}}{\alpha_{s}^{2}}(1+\frac{1-\alpha_{s}^{2}}{\alpha_{t}^{2}-1}).$

Since $\alpha_{t}\in[0,1)$, $\sigma_{s|t}$ is non-increasing with respect to
$\alpha_{t}^{2}$. As $\alpha_{t}^{2}$ is non-increasing with respect to $t$,
the claim is established. In addition, the range of $\sigma_{s|t}$ is
$[0,\sigma_{s}]$ $\square$

######  Lemma 3.

Let $M$ be the constant in Lemma 1, $\forall s\in(0,1)$

(1). there is a $\tau_{1}$ such that $\forall t\in(s,\tau_{1}]$ and $1\leq
i\leq N$

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|\leq\sqrt{\sigma_{s|t}}$
(36)

indicates

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}|\leq
2\sqrt{\sigma_{s|t}}$ (37)

(2). there is a $\tau_{2}$ such that $\forall t\in(s,\tau_{2}]$ and $1\leq
i\leq N$

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|>\sqrt{\sigma_{s|t}}$
(38)

indicates

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|>\frac{\sqrt{\sigma_{s|t}}}{2}$
(39)

##### Proof:

(1) Let
$\sigma_{s|\tau_{1}}=min((\frac{\sigma_{s}^{2}}{\alpha_{s}M})^{\frac{2}{3}},0.9\sigma_{s})$,
then $s<t\leq\tau_{1}$ indicates
$\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}M\leq\sqrt{\sigma_{s|t}}$.
Thus

$\displaystyle|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}|$
(40)
$\displaystyle\leq|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|+\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}|\bar{y}(x_{t},t)|$
$\displaystyle\leq\sqrt{\sigma_{s|t}}+\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}M$
$\displaystyle\leq 2\sqrt{\sigma_{s|t}}.$

(2) Let
$\sigma_{s|\tau_{2}}=min((\frac{\sigma_{s}^{2}}{2\alpha_{s}M})^{\frac{2}{3}},0.9\sigma_{s})$,
then $s\leq t\leq\tau_{2}$ indicates
$\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}M\leq\frac{\sqrt{\sigma_{s|t}}}{2}$.
Thus

$\displaystyle|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|$
(41)
$\displaystyle\geq|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|-\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}|y_{i}-\bar{y}(x_{t},t)|$
$\displaystyle>\sqrt{\sigma_{s|t}}-\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}M$
$\displaystyle>\frac{\sqrt{\sigma_{s|t}}}{2}.$

######  Lemma 4.

Let $M$ be the constant in Lemma 1

(1). there is a $\nu_{1}$ such that $\nu_{1}\leq s<t\leq 1$ and $1\leq i\leq
N$

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|\leq\frac{1}{\sqrt{\alpha_{s}}}$
(42)

indicates

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}|\leq\frac{2}{\sqrt{\alpha_{s}}}$
(43)

(2). there is a $\nu_{2}$ such that $\nu_{2}\leq s<t\leq 1$ and $1\leq i\leq
N$

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|>\frac{1}{\sqrt{\alpha_{s}}}$
(44)

indicates

$|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|>\frac{1}{2\sqrt{\alpha_{s}}}$
(45)

##### Proof:

(1) Let $\alpha_{\nu_{1}}=min((\frac{1}{M})^{\frac{2}{3}},0.9)$, then
$\nu_{1}\leq s<t\leq 1$ indicates
$\frac{\alpha_{s}\sigma_{s|t}^{2}M}{\sigma_{s}^{2}}\leq\alpha_{s}M\leq\frac{1}{\sqrt{\alpha_{s}}}$.
Thus

$\displaystyle|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}|$
(46)
$\displaystyle\leq|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|+|\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|$
$\displaystyle\leq\frac{1}{\sqrt{\alpha_{s}}}+\frac{1}{\sqrt{\alpha_{s}}}=\frac{2}{\sqrt{\alpha_{s}}}.$

(2) Let $\alpha_{\nu_{2}}=min((\frac{1}{2M})^{\frac{2}{3}},0.9)$, then
$\nu_{2}\leq s<t\leq 1$ indicates
$\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}M\leq\alpha_{s}M\leq\frac{1}{2\sqrt{\alpha_{s}}}$.
Thus

$\displaystyle|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|$
(47)
$\displaystyle\geq|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|$
$\displaystyle~{}~{}~{}~{}-|\frac{\alpha_{s}\sigma_{s|t}^{2}}{\sigma_{s}^{2}}(\bar{y}(x_{t},t)-y_{i})|$
$\displaystyle>\frac{1}{\sqrt{\alpha_{s}}}-\frac{1}{2\sqrt{\alpha_{s}}}=\frac{1}{2\sqrt{\alpha_{s}}}.$

######  Proposition 1.

$\forall s\in(0,1)$, $\exists\tau\in(s,1)$ and $C>0$, such that $\forall
t\in(s,\tau]$,
$\int_{\mathbb{R}^{d}}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}<C\sqrt{\sigma_{s|t}}$.

##### Proof:

Let $A(x_{t},y,c)$ denote the set
$\\{x_{s}:|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y}{\sigma_{s}^{2}}|>c\sqrt{\sigma_{s|t}}\\}$.

$\displaystyle\tilde{P}(A(x_{t},\bar{y}(x_{t},t),1)|x_{t})$ (48)
$\displaystyle=\tilde{P}(|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|>\sqrt{\sigma_{s|t}}|x_{t})$
$\displaystyle\leq\frac{\sigma_{s|t}^{2}}{\sigma_{s|t}}=\sigma_{s|t}.$

The last inequality is from the Chebyshev inequality. Let $\tau_{2}$ be the
value in Lemma 3(2), then $\forall t\in(s,\tau_{2}]$ $x_{s}\in
A(x_{t},\bar{y}(x_{t},t),1)$ indicates $x_{s}\in
A(x_{t},y_{i},\frac{1}{2}),1\leq i\leq N$, then

$\displaystyle P(A(x_{t},\bar{y}(x_{t},t),1)|x_{t},y_{i})$ (49)
$\displaystyle\leq P(A(x_{t},y_{i},\frac{1}{2})|x_{t},y_{i})$
$\displaystyle=P(|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|>\frac{1}{2}\sqrt{\sigma_{s|t}}|x_{t},y_{i})$
$\displaystyle\leq\frac{\sigma_{s|t}^{2}}{\frac{1}{4}\sigma_{s|t}}=4\sigma_{s|t}.$

As a result

$\displaystyle\int_{A(x_{t},\bar{y}(x_{t},t),1)}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
(50)
$\displaystyle\leq\sum_{i}\int_{A(x_{t},\bar{y}(x_{t},t),1)}w_{i}(x_{t},t)|p(x_{s}|x_{t},y_{i})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
$\displaystyle\leq\sum_{i}\int_{A(x_{t},\bar{y}(x_{t},t),1)}w_{i}(x_{t},t)(p(x_{s}|x_{t},y_{i})+\tilde{p}(x_{s}|x_{t}))dx_{s}$
$\displaystyle\leq\sum_{i}w_{i}(x_{t},t)5\sigma_{s|t}=5\sigma_{s|t}.$

On the other hand, if $x_{s}\in A^{c}(x_{t},\bar{y}(x_{t},t),1)$, where
$A^{c}(x_{t},y,c)$ is the complement of $A(x_{t},y,c)$ denoting the set
$\\{x_{s}:|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y}{\sigma_{s}^{2}}|\leq
c\sqrt{\sigma_{s|t}}\\}$. Let $\tau_{1}$ be the value in Lemma 3(1), $\forall
t\in(s,\tau_{1}]$, then $x_{s}\in A^{c}(x_{t},\bar{y}(x_{t},t),1)$ indicates
$x_{s}\in A^{c}(x_{t},0,2)$. Denoting

$\displaystyle C_{i}$
$\displaystyle=|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|^{2}$
(51)
$\displaystyle~{}~{}~{}~{}-|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|^{2},$

we have

$\displaystyle|\frac{C_{i}}{2\sigma_{s|t}^{2}}|$
$\displaystyle=|\frac{1}{2\sigma_{s|t}^{2}}(|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|^{2}$
(52)
$\displaystyle~{}~{}~{}~{}-|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|^{2})|$
$\displaystyle=|\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))$
$\displaystyle~{}~{}~{}~{}+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}(\bar{y}^{2}(x_{t},t)-y_{i}^{2})|$
$\displaystyle\leq\frac{\alpha_{s}}{\sigma_{s}^{2}}2\sqrt{\sigma_{s|t}}M+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}2M^{2}$

then

$\displaystyle\exp$
$\displaystyle(\frac{C_{i}}{2\sigma_{s|t}^{2}})=\exp(\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))$
(53)
$\displaystyle~{}~{}~{}~{}+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}(\bar{y}^{2}(x_{t},t)-y_{i}^{2}))$
$\displaystyle=1+\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))$
$\displaystyle~{}~{}~{}~{}+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}(\bar{y}^{2}(x_{t},t)-y_{i}^{2})$
$\displaystyle~{}~{}~{}~{}+o(\frac{\alpha_{s}}{\sigma_{s}^{2}}2\sqrt{\sigma_{s|t}}M+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}2M^{2})$
$\displaystyle=1+\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))+o(\sqrt{\sigma_{s|t}}).$

Thus

$\displaystyle|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|$ (54)
$\displaystyle=\tilde{p}(x_{s}|x_{t})|\frac{p(x_{s}|x_{t})}{\tilde{p}(x_{s}|x_{t})}-1|$
$\displaystyle=\tilde{p}(x_{s}|x_{t})|\sum_{i}w_{i}(x_{t},t)\exp(\frac{C_{i}}{2\sigma_{s|t}^{2}})-1|$
$\displaystyle=\tilde{p}(x_{s}|x_{t})|\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(\sum_{i}w_{i}(x_{t},t)y_{i}-\bar{y}(x_{t},t))$
$\displaystyle~{}~{}~{}~{}+o(\sqrt{\sigma_{s|t}})|$
$\displaystyle=\tilde{p}(x_{s}|x_{t})(o(\sqrt{\sigma_{s|t}})),$

which means $\exists\tau_{3}\in(s,1],C_{1}>0$, such that
$|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|<\tilde{p}(x_{s}|x_{t})*C_{1}\sqrt{\sigma_{s|t}}$,
$\forall t\in(s,\tau_{3}]$

Let $\tau=\min(\tau_{1},\tau_{2},\tau_{3})$. As a result

$\displaystyle\int_{\mathbb{R}^{d}}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
(55)
$\displaystyle=\int_{A(x_{t},\bar{y}(x_{t},t),1)}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
$\displaystyle~{}~{}~{}~{}+\int_{A^{c}(x_{t},\bar{y}(x_{t},t),1)}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
$\displaystyle\leq
5\sigma_{s|t}+\int_{A^{c}(x_{t},\bar{y}(x_{t},t),1)}\tilde{p}(x_{s}|x_{t})*C_{1}\sqrt{\sigma_{s|t}}ds$
$\displaystyle\leq
5\sigma_{s|t}+C_{1}\sqrt{\sigma_{s|t}}<(5+C_{1})\sqrt{\sigma_{s|t}}$

Let $C=5+C_{1}$, this is the constant required in this proposition. $\square$

######  Proposition 2.

$\exists\nu\in(0,1)$ and $C>0$, such that $\forall\nu\leq s<t\leq 1$,
$\int_{\mathbb{R}^{d}}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}<C\sqrt{\alpha_{s}}$.

##### Proof:

Let $A(x_{t},y,c)$ denote the set
$\\{x_{s}:|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y}{\sigma_{s}^{2}}|>c\frac{1}{\sqrt{\alpha_{s}}}\\}$.

$\displaystyle\tilde{P}(A(x_{t},\bar{y}(x_{t},t),1)|x_{t})$ (56)
$\displaystyle=\tilde{P}(|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|>\frac{1}{\sqrt{\alpha_{s}}}|x_{t})$
$\displaystyle\leq\sigma_{s|t}^{2}\alpha_{s}\leq\alpha_{s}.$

The last inequality is from the Chebyshev inequality. Let $\nu_{2}$ be the
value in Lemma 4(2), then $\forall\nu_{2}\leq s<t\leq t$, $x_{s}\in
A(x_{t},\bar{y}(x_{t},t),1)$ indicates $x_{s}\in
A(x_{t},y_{i},\frac{1}{2}),1\leq i\leq N$, then

$\displaystyle P(A(x_{t},\bar{y}(x_{t},t),1)|x_{t},y_{i})$ (57)
$\displaystyle\leq P(A(x_{t},y_{i},\frac{1}{2})|x_{t},y_{i})$
$\displaystyle=P(|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|>\frac{1}{2\sqrt{\alpha_{s}}}|x_{t},y_{i})$
$\displaystyle\leq\sigma_{s|t}^{2}4\alpha_{s}\leq 4\alpha_{s}.$

As a result

$\displaystyle\int_{A(x_{t},\bar{y}(x_{t},t),1)}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
(58)
$\displaystyle\leq\sum_{i}\int_{A(x_{t},\bar{y}(x_{t},t),1)}w_{i}(x_{t},t)|p(x_{s}|x_{t},y_{i})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
$\displaystyle\leq\sum_{i}\int_{A(x_{t},\bar{y}(x_{t},t),1)}w_{i}(x_{t},t)(p(x_{s}|x_{t},y_{i})+\tilde{p}(x_{s}|x_{t}))dx_{s}$
$\displaystyle\leq\sum_{i}w_{i}(x_{t},t)5\alpha_{s}=5\alpha_{s}.$

On the other hand, if $x_{s}\in A^{c}(x_{t},\bar{y}(x_{t},t),1)$. Let
$\nu_{1}$ be the value in Lemma 4(1), $\forall\nu_{2}\leq s<t\leq 1$, then
$x_{s}\in A^{c}(x_{t},\bar{y}(x_{t},t),1)$ indicates $x_{s}\in
A^{c}(x_{t},0,2)$. Denoting

$\displaystyle C_{i}$
$\displaystyle=|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|^{2}$
(59)
$\displaystyle~{}~{}~{}~{}-|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|^{2},$

we have

$\displaystyle|\frac{C_{i}}{2\sigma_{s|t}^{2}}|$
$\displaystyle=|\frac{1}{2\sigma_{s|t}^{2}}(|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}\bar{y}(x_{t},t)}{\sigma_{s}^{2}}|^{2}$
(60)
$\displaystyle~{}~{}~{}~{}-|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y_{i}}{\sigma_{s}^{2}}|^{2})|$
$\displaystyle=|\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))$
$\displaystyle~{}~{}~{}~{}+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}(\bar{y}^{2}(x_{t},t)-y_{i}^{2})|$
$\displaystyle\leq\frac{\sqrt{\alpha_{s}}}{\sigma_{s}^{2}}2M+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}2M^{2},$

then

$\displaystyle\exp$
$\displaystyle(\frac{C_{i}}{2\sigma_{s|t}^{2}})=\exp(\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))$
(61)
$\displaystyle~{}~{}~{}~{}+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}(\bar{y}^{2}(x_{t},t)-y_{i}^{2}))$
$\displaystyle=1+\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))$
$\displaystyle~{}~{}~{}~{}+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}(\bar{y}^{2}(x_{t},t)-y_{i}^{2})$
$\displaystyle~{}~{}~{}~{}+o(\frac{\sqrt{\alpha_{s}}}{\sigma_{s}^{2}}2M+\frac{\alpha_{s}^{2}\sigma_{s|t}^{2}}{2\sigma_{s}^{4}}2M^{2})$
$\displaystyle=1+\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(y_{i}-\bar{y}(x_{t},t))+o(\sqrt{\alpha_{s}}).$

Thus

$\displaystyle|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|$ (62)
$\displaystyle=\tilde{p}(x_{s}|x_{t})|\frac{p(x_{s}|x_{t})}{\tilde{p}(x_{s}|x_{t})}-1|$
$\displaystyle=\tilde{p}(x_{s}|x_{t})|\sum_{i}w_{i}(x_{t},t)\exp(\frac{C_{i}}{2\sigma_{s|t}^{2}})-1|$
$\displaystyle=\tilde{p}(x_{s}|x_{t})|\frac{\alpha_{s}}{\sigma_{s}^{2}}(x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}})^{T}(\sum_{i}w_{i}(x_{t},t)y_{i}$
$\displaystyle~{}~{}~{}~{}-\bar{y}(x_{t},t))+o(\sqrt{\alpha_{s}})|$
$\displaystyle=\tilde{p}(x_{s}|x_{t})(o(\sqrt{\alpha_{s}})),$

which means $\exists\nu_{3}\in(s,1],C_{1}>0$, such that
$|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|<\tilde{p}(x_{s}|x_{t})C_{1}\sqrt{\alpha_{s}}$,
$\forall t\in(s,\nu_{3}]$.

Let $\nu=\min(\nu_{1},\nu_{2},\nu_{3})$. As a result

$\displaystyle\int_{\mathbb{R}^{d}}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
(63)
$\displaystyle=\int_{A(x_{t},\bar{y}(x_{t},t),1)}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
$\displaystyle~{}~{}~{}~{}+\int_{A^{c}(x_{t},\bar{y}(x_{t},t),1)}|p(x_{s}|x_{t})-\tilde{p}(x_{s}|x_{t})|dx_{s}$
$\displaystyle\leq
5\alpha_{s}+\int_{A^{c}(x_{t},\bar{y}(x_{t},t),1)}\tilde{p}(x_{s}|x_{t})*C_{1}\sqrt{\alpha_{s}}ds$
$\displaystyle\leq
5\alpha_{s}+C_{1}\sqrt{\alpha_{s}}<(5+C_{1})\sqrt{\alpha_{s}}$

Let $C=5+C_{1}$, this is the constant required in this proposition. $\square$

Remark. Observing that the two parts of the bound in Eq. 55 is $\sigma_{s|t}$
and $\sqrt{\sigma_{s|t}}$, which have different order. We can balance the two
orders by setting the set $A(x_{t},y,c)$ to be
$\\{x_{s}:|x_{s}-\frac{\alpha_{t|s}\sigma_{s}^{2}x_{t}}{\sigma_{t}^{2}}-\frac{\alpha_{s}\sigma_{s|t}^{2}y}{\sigma_{s}^{2}}|>c(\sigma_{s|t})^{\frac{2}{3}}\\}$.
With the same proof process, we can get a bound
$(\sigma_{s|t})^{\frac{2}{3}}$, which is better than $\sqrt{\sigma_{s|t}}$.
Similarly, we can also get a better bound for Proposition 2.

######  Proposition 3.

Setting $x_{1-\epsilon}\sim\mathcal{N}(0,I)$ is equivalent to sampling the
value from standard Gaussian as $\bar{y}(x_{1},1)$ at $t=1$.

##### Proof:

We will demonstrate this proposition in two scenarios, DDPM and DDIM.

(1) Sampling by DDPM:

The sampling step for DDPM is

$x_{s}=\frac{\alpha_{t|s}\sigma_{s}^{2}}{\sigma_{t}^{2}}x_{t}+\frac{\alpha_{s}\sigma_{t|s}^{2}}{\sigma_{t}^{2}}\bar{y}(x_{t},t)+\sigma_{s|t}z_{t}.$
(64)

Applying the conditions that $\alpha_{1}=0$ and $\sigma_{1}=1$ to this
equation, we obtain

$x_{1-\epsilon}=\alpha_{1-\epsilon}\bar{y}(x_{1},1)+\sigma_{1-\epsilon}z_{1},$
(65)

where $z_{1}$ follows a standard Gaussian distribution. Assuming that
$x_{1-\epsilon}\sim\mathcal{N}(0,I)$ and is independent of $z_{1}$, it leads
us to conclude that

$\bar{y}(x_{1},1)=\frac{x_{1-\epsilon}-\sigma_{1-\epsilon}z_{1}}{\alpha_{1-\epsilon}},$
(66)

also follows a Gaussian distribution. Given that
$\alpha_{1-\epsilon}^{2}+\sigma_{1-\epsilon}^{2}=1$, we can deduce the
following quantities

$\displaystyle
E(\bar{y}(x_{1},1))=\frac{0-\sigma_{1-\epsilon}0}{\alpha_{1-\epsilon}}=0,$
(67) $\displaystyle
Var(\bar{y}(x_{1},1))=\frac{I-\sigma_{1-\epsilon}^{2}I}{\alpha_{1-\epsilon}^{2}}=I.$

This implies that $\bar{y}(x_{1},1)$ follows a standard Gaussian distribution.

(2) Sampling by DDIM:

The sampling step for DDPM from $1$ to $1-\epsilon$ is

$x_{1-\epsilon}=\alpha_{1-\epsilon}\bar{y}(x_{1},1)+\sigma_{1-\epsilon}x_{1}.$
(68)

From this, we can deduce that

$\bar{y}(x_{1},1)=\frac{x_{1-\epsilon}-\sigma_{1-\epsilon}x_{1}}{\alpha_{1-\epsilon}},$
(69)

is normally distributed. Assuming that $x_{1-\epsilon}\sim\mathcal{N}(0,I)$
and is independent of $x_{1}$, we can calculate the following quantities

$\displaystyle
E(\bar{y}(x_{1},1))=\frac{0-\sigma_{1-\epsilon}0}{\alpha_{1-\epsilon}}=0,$
(70) $\displaystyle
Var(\bar{y}(x_{1},1))=\frac{I-\sigma_{1-\epsilon}^{2}I}{\alpha_{1-\epsilon}^{2}}=I.$

Therefore, $\bar{y}(x_{1},1)$ is confirmed to be a standard Gaussian
distribution.

$\square$

Table 3: Comparison of average brightness of 100 generated images between SD-2.0 and our SingDiffusion under different prompt conditions. For ’white’ prompts, higher average brightness is preferable, while for ’black’ prompts, lower average brightness is better. Model |  | ”Solid white
---
background”
| ”Solid black
---
background”
| ”Monochrome
---
line-art logo on a
white background”
| ”Monochrome
---
line-art logo on a
black background”
SD-2.0 | 137.64 | 84.94 | 130.68 | 105.07
\+ Ours | 231.26 | 1.54 | 238.67 | 9.22
|  |  |  |
Table 4: Comparison of SD-2.0 model and SingDiffusion on FID score and CLIP score without classifier guidance. Model |  FID $\downarrow$ |  CLIP $\uparrow$
---|---|---
Original | 36.42 | 27.46
\+ SingDiffusion | 20.13 | 28.43
|  |
Figure 8: Comparison of SD-2.0 and our SingDiffusion on average brightness
issue.
$29.5$$30$$30.5$$31$$31.5$$32$$9$$10$$11$$12$$13$$14$$15$$16$$17$$18$$19$$20$$21$CLIP
score $\uparrow$FID score $\downarrow$SD-2.0\+ SingDiffusion Figure 9:
Comparison of Pareto curves between SD-2.0 and our SingDiffusion on 30k COCO
images, across various guidance scales in [1.5, 2, 3, 4, 5, 6, 7, 8].

##  Appendix E Application on SD-2.0

Stable diffusion 2.0 model (SD-2.0) samples images using $v$-prediction, which
differs from SD-2.0-base that utilizes $\epsilon$-prediction. To validate the
applicability of our SingDiffusion method to the $v$-prediction approach, we
conducted identical experiments on SD-2.0 as with SD-2.0-base. Firstly, we
present the outcomes of our method in addressing the average brightness issue
on SD-2.0, as depicted in Figure 8 and Table 3. From the visualizations, it is
evident that our method significantly alleviates SD-2.0’s challenge in
generating both bright and dark images. Furthermore, we compare the generation
capabilities of SD-2.0 on the COCO dataset before and after applying our
method, as detailed in Table 4. It is noticeable that our method substantially
reduces the FID score, resulting in more realistic outcomes. Lastly, we
compared the CLIP vs. FID Pareto curves of the two methods across various
guidance scales. As illustrated in Figure 9, our method achieves lower FID
scores for the same CLIP score.