#  Score identity Distillation: Exponentially Fast Distillation of Pretrained
Diffusion Models for One-Step Generation

Mingyuan Zhou  The University of Texas at Austin  Google  Huangjie Zheng  The
University of Texas at Austin  Zhendong Wang  The University of Texas at
Austin  Mingzhang Yin  The University of Florida  Hai Huang  Google

###### Abstract

We introduce Score identity Distillation (SiD), an innovative data-free method
that distills the generative capabilities of pretrained diffusion models into
a single-step generator. SiD not only facilitates an exponentially fast
reduction in Fréchet inception distance (FID) during distillation but also
approaches or even exceeds the FID performance of the original teacher
diffusion models. By reformulating forward diffusion processes as semi-
implicit distributions, we leverage three score-related identities to create
an innovative loss mechanism. This mechanism achieves rapid FID reduction by
training the generator using its own synthesized images, eliminating the need
for real data or reverse-diffusion-based generation, all accomplished within
significantly shortened generation time. Upon evaluation across four benchmark
datasets, the SiD algorithm demonstrates high iteration efficiency during
distillation and surpasses competing distillation approaches, whether they are
one-step or few-step, data-free, or dependent on training data, in terms of
generation quality. This achievement not only redefines the benchmarks for
efficiency and effectiveness in diffusion distillation but also in the broader
field of diffusion-based generation. Our PyTorch implementation will be
publicly accessible on GitHub.

Figure 1: Rapid advancements in the distillation of a pretrained ImageNet
64x64 diffusion model are shown using the proposed SiD method, with settings
$\alpha=1.0$, a batch size of 1024, and a learning rate of 5e-6. The series of
images, generated from the same set of random noises post-training the SiD
generator with varying counts of synthesized images, illustrates progressions
at 0, 0.1, 0.2, 0.5, 1, 2, 5, 10, 20, and 50 million images. These are
equivalent to roughly 0, 100, 200, 500, 1K, 2K, 5K, 10K, 20K, and 49K training
iterations respectively, organized from the top left to the bottom right. The
associated FIDs for these iterations are 153.52, 34.83, 37.42, 18.08, 10.82,
7.74, 5.94, 4.49, 3.40, and 3.07, in order. The progression of FIDs is
detailed in Fig. 9 in the Appendix.

##  1 Introduction

Diffusion models, also known as score-based generative models, have emerged as
the leading approach for generating high-dimensional data (Sohl-Dickstein et
al., 2015; Song & Ermon, 2019; Ho et al., 2020). These models are appreciated
for their training simplicity and stability, their robustness against mode
collapse during generation, and their ability to produce high-resolution,
diverse, and photorealistic images (Dhariwal & Nichol, 2021; Ho et al., 2022;
Ramesh et al., 2022; Rombach et al., 2022; Saharia et al., 2022; Peebles &
Xie, 2023; Zheng et al., 2023c).

However, the process of generating data with diffusion models involves
iterative refinement-based reverse diffusion, necessitating multiple
iterations through the same generative network. This multi-step generation
process, initially requiring hundreds or even thousands of steps, stands in
contrast to the single-step generation capabilities of previous deep
generative models such as variational auto encoders (VAEs) (Kingma & Welling,
2014; Rezende et al., 2014) and generative adversarial networks (GANs)
(Goodfellow et al., 2014; Karras et al., 2020), which only require forwarding
the noise through the generation network once. Diffusion models necessitate
multi-step generation, making them much more expensive at inference time. A
wide variety of methods have been introduced to reduce the number of sampling
steps during reverse diffusion, but they often still require quite a few
number of function evaluations (NFE), such as 35 NFE for CIFAR-10 32x32 and
511 NFE for ImageNet 64x64 in EDM (Karras et al., 2022), to achieve good
performance.

In this study, we aim to introduce a single-step generator designed to distill
the knowledge on training data embedded in the score-estimation network of a
pretrained diffusion model. To achieve this goal, we propose training the
generator by minimizing a model-based score-matching loss between the scores
of the diffused real data and the diffused generator-synthesized fake data
distributions at various noise levels. However, estimating this model-based
score-matching loss, which is a form of Fisher divergence, at any given noise
level proves to be intractable. To overcome this challenge, we offer a fresh
perspective by viewing the forward diffusion processes of diffusion models
through the lens of semi-implicit distributions. We introduce three
corresponding score-related identities and illustrate their integration to
formulate an innovative loss mechanism. This mechanism involves both score
estimation and Monte Carlo estimation techniques to handle intractable
expectations. Our method, designated as Score identity Distillation (SiD), is
named to underscore its roots in these three identities.

We validate the effectiveness and efficiency of SiD across all four benchmark
datasets considered in Karras et al. (2022): CIFAR-10 32x32, ImageNet 64x64,
FFHQ 64x64, and AFHQv2 64x64. The SiD single-step generator is trained using
the VP-EDM checkpoints as the teacher diffusion models. It achieves state-of-
the-art performance across all four datasets in providing high-quality
generation, measured by Fréchet inception distance (FID) (Heusel et al.,
2017), and also facilitates an exponentially fast reduction in FID as the
distillation progresses. This is visually corroborated by Figs. 1 and 2 and
detailed in the experiments section.

##  2 Related Work

Significant efforts have been directed towards executing the reverse diffusion
process in fewer steps. A prominent line of research involves interpreting the
diffusion model through the lens of stochastic differential equations (SDE) or
ordinary differential equations (ODE), followed by employing advanced
numerical solvers for SDE/ODE (Song et al., 2020; Liu et al., 2022a; Lu et
al., 2022; Zhang & Chen, 2023; Karras et al., 2022). Despite these
advancements, there remains a pronounced trade-off between reducing steps and
preserving visual quality. Another line of work considers the diffusion model
within the framework of flow matching, applying strategies to simplify the
reverse diffusion process into more linear trajectories, thereby facilitating
larger step advancements (Liu et al., 2022b; Lipman et al., 2022). To achieve
generation within fewer steps, researchers also propose to truncate the
diffusion chain and starting the generation from an implicit distribution
instead of white Gaussian noise (Pandey et al., 2022; Zheng et al., 2023a; Lyu
et al., 2022) and combining it with GANs for faster generation (Xiao et al.,
2022; Wang et al., 2023c).

A unique avenue of research focuses on distilling the reverse diffusion chains
(Luhman & Luhman, 2021; Salimans & Ho, 2022; Zheng et al., 2023b; Luo et al.,
2023b). Salimans & Ho (2022) pioneered the concept of progressive
distillation, which Meng et al. (2023) took further into the realm of guided
diffusion models equipped with classifier-free guidance. Subsequent
advancements introduced consistency models (Song et al., 2023) as an
innovative strategy for distilling diffusion models, which promotes output
consistency throughout the ODE trajectory. Song & Dhariwal (2023) further
enhanced the generation quality of these consistency models through extensive
engineering efforts and new theoretical insights. Pushing the boundaries
further, Kim et al. (2023) improved prediction consistency at any intermediate
stage and incorporated GAN-based loss to elevate image quality. Extending
these principles, Luo et al. (2023a) applied consistency distillation
techniques to text-guided latent diffusion models (Ramesh et al., 2022),
facilitating efficient and high-fidelity text-to-image generation.

Recent research has focused on distilling diffusion models into generators
capable of one or two step operations through adversarial training (Sauer et
al., 2023). Following Diffusion-GAN (Wang et al., 2023c), which trains a one-
step generator by minimizing the Jensen–Shannon divergence (JSD) at each
diffusion time step, Xu et al. (2023) introduced UFOGen, which distills
diffusion models using a time-step dependent discriminator, mirroring the
initialization of the generator. UFOGen has shown proficiency in one-step
text-guided image generation. Text-to-3D synthesis, using a pretrained 2D
text-to-image diffusion model, effectively acts as a distillation process, and
leverages the direction indicated by the score function of the 2D diffusion
model to guide the generation of various views of 3D objects (Poole et al.,
2022; Wang et al., 2023b). Building on this concept, Diff-Instruct (Luo et
al., 2023c) applies this principle to distill general pretrained diffusion
models into a single-step generator, while SwiftBrush (Nguyen & Tran, 2023)
further illustrates its effectiveness in distilling pretrained stable
diffusion models. Distribution Matching Distillation (DMD) of Yin et al.
(2023) aligns closely with this principle, and further introduces an
additional regression loss term to improve the quality of distillation.

It is important to note that both Diff-Instruct and DMD are fundamentally
aligned with the approach first introduced by Diffusion-GAN (Wang et al.,
2023c). This method entails training the generator by aligning the diffused
real and fake distributions. The primary distinction lies in whether the JSD
or KL divergence is employed for any given noise level. From this perspective,
the proposed SiD method adheres to the framework established by Diffusion-GAN
and subsequently embraced by Diff-Instruct and DMD. However, SiD distinguishes
itself by implementing a model-based score-matching loss, notably a variant of
Fisher divergence, moving away from the traditional use of JSD or KL
divergence applied to diffused real and fake distributions. Furthermore, it
uncovers an effective strategy to approximate this loss, which is analytically
intractable. In the sections that follow, we delve into both the distinctive
loss mechanism and the method for its approximation, illuminating SiD’s
innovative strategy.

##  3 Forward Diffusion as Semi-Implicit Distribution: Exploring Score
Identities

The marginal of a mixture distribution can be expressed as $p(\bm{x})=\int
p(\bm{x}|\bm{z})p(\bm{z})d\bm{z}.$ In cases where $p(\bm{x}|\bm{z})$ is
analytically defined and $p(\bm{z})$ is straightforward to sample from, yet
the marginal distribution is intractable or difficult to compute, we follow
Yin & Zhou (2018) to refer to it as a semi-implicit distribution. This semi-
implicit framework and its derivatives have been widely used to develop
flexible variational distributions with tractable parameter inference, as
evidenced by a series of studies (Yin & Zhou, 2018; Molchanov et al., 2019;
Hasanzadeh et al., 2019; Titsias & Ruiz, 2019; Sobolev & Vetrov, 2019; Lawson
et al., 2019; Moens et al., 2021; Yu & Zhang, 2023; Yu et al., 2023).

In our study, we explore the vital role of semi-implicit distributions in the
forward diffusion process. We observe that both the observed real data and the
generated fake data adhere to semi-implicit distributions in this process. The
gradients of their log-likelihoods, commonly known as scores, can be
formulated as the expectation of certain random variables. These expectations
are amenable to approximation through deep neural networks or Monte Carlo
estimation. This reformulation of the scores is enabled through the
application of the semi-implicit framework, allowing for the introduction of
three critical identities pertinent to score estimation, as detailed
subsequently.

###  3.1 Forward Diffusions and Semi-Implicit Distributions

The forward marginal of a diffusion model is an exemplary illustration of a
semi-implicit distribution, expressed as:

$\displaystyle\textstyle p_{\text{data}}(\bm{x}_{t})=\int
q(\bm{x}_{t}\,|\,\bm{x}_{0})p_{\text{data}}(\bm{x}_{0})\,d\bm{x}_{0},$ (1)

where the forward conditional $q(\bm{x}_{t}\,|\,\bm{x}_{0})$ is analytically
defined, but the data distribution $p_{\text{data}}(\bm{x}_{0})$ remains
unknown and is typically represented through empirical samples. In this paper,
we focus on Gaussian diffusion models, where the forward conditional follows a
Gaussian distribution:

$\displaystyle
q(\bm{x}_{t}\,|\,\bm{x}_{0})=\mathcal{N}(a_{t}\bm{x}_{0},\sigma_{t}^{2}),$

with $a_{t}\in[0,1]$. To generate a diffused sample from $\bm{x}_{0}\sim
p_{\text{data}}(\bm{x}_{0})$, reparameterization is often employed:

$\displaystyle\bm{x}_{t}:=a_{t}\bm{x}_{0}+\sigma_{t}\bm{\epsilon}_{t},\quad\bm{\epsilon}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).$

As $a_{t}$ can be assimilated into the preconditioning of neural network
inputs without sacrificing generality, we set $a_{t}=1$ for simplicity, in
line with Karras et al. (2022).

While the exact form of $p_{\text{data}}(\bm{x}_{t})$ and hence the score
$S(\bm{x}_{t}):=\nabla_{\bm{x}_{t}}\ln p_{\text{data}}(\bm{x}_{t})$ are not
known, the score of the forward conditional $q(\bm{x}_{t}\,|\,\bm{x}_{0})$ has
an analytic expression as

$\displaystyle\nabla_{\bm{x}_{t}}\ln
q(\bm{x}_{t}\,|\,\bm{x}_{0})=\sigma_{t}^{-2}{(\bm{x}_{0}-\bm{x}_{t})}=-\sigma_{t}^{-1}{\bm{\epsilon}_{t}}.$
(2)

In this work, we explore an implicit generator $p_{\theta}(\bm{x}_{g})$,
parameterized by $\theta$, which generates random samples as
$\bm{x}_{g}=G_{\theta}(\bm{z}),\bm{z}\sim p(\bm{z}),$ where
$G_{\theta}(\cdot)$ represents a neural network, parameterized by $\theta$,
that deterministically transforms noise $\bm{z}\sim p(\bm{z})$ into generated
data $\bm{x}_{g}$. If the distribution of $p_{\theta}(\bm{x}_{g})$ matches
that of $p_{\text{data}}(\bm{x}_{0})$, it then follows that the semi-implicit
distribution

$\displaystyle\textstyle p_{\theta}(\bm{x}_{t})=\int
q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})\,d\bm{x}_{g}$ (3)

would be identical to $p_{\text{data}}(\bm{x}_{t})$ for any $t$. Conversely,
as proved in Wang et al. (2023c), if $p_{\theta}(\bm{x}_{t})$ coincides with
$p_{\text{data}}(\bm{x}_{t})$ for any $t$, this implies a match between the
generator distribution $p_{\theta}(\bm{x}_{g})$ and the data distribution
$p_{\text{data}}(\bm{x}_{0})$.

###  3.2 Score Identities

In this paper, we illustrate that the semi-implicit distribution defined in
(3) is characterized by three crucial identities, each playing a vital role in
score-based distillation. The first identity concerns the diffused real data
distribution, a well-established concept fundamental to denoising score
matching. The second identity is analogous to the first but applies to
diffused generator distributions. The third identity, though not as widely
recognized, is essential for the development of our proposed method.

######  Identity 1 (Tweedie’s Formula for Diffused Real Data).

Consider the semi-implicit distribution $p_{\text{data}}(\bm{x}_{t})$ in (1),
defined by diffusing real data. The expected value of $\bm{x}_{0}$ given
$\bm{x}_{t}$, in line with
$q(\bm{x}_{0}\,|\,\bm{x}_{t})=\frac{q(\bm{x}_{t}\,|\,\bm{x}_{0})p_{\emph{\text{data}}}(\bm{x}_{0})}{p_{\emph{\text{data}}}(\bm{x}_{t})}$
as per Bayes’ rule, is connected to the score of
$p_{\emph{\text{data}}}(\bm{x}_{t})$ as

(4)

This identity, known as Tweedie’s Formula (Robbins, 1992; Efron, 2011),
provides an estimate for the original data $\bm{x}_{0}$ given $\bm{x}_{t}$,
where $\bm{x}_{t}$ is generated as
$\bm{x}_{t}\sim\mathcal{N}(\bm{x}_{0},\sigma_{t}^{2}),\bm{x}_{0}\sim
p_{\text{data}}(\bm{x}_{0})$. The significance of this relationship has been
discussed in Luo (2022) and Chung et al. (2022). An equivalent identity
applies to the diffused fake data distribution.

######  Identity 2 (Tweedie’s Formula for Diffused Fake Data).

For the semi-implicit distribution $p_{\theta}(\bm{x}_{t})$ defined in (3),
resulting from diffusing fake data, the expected value of $\bm{x}_{g}$ given
$\bm{x}_{t}$, following
$q(\bm{x}_{g}\,|\,\bm{x}_{t})=\frac{q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})}{p_{\theta}(\bm{x}_{t})}$
according to Bayes’ rule, is associated with the score of
$p_{\theta}(\bm{x}_{t})$ as

(5)

Transitioning from the initial two identities, we introduce a third identity
that is crucial for the proposed computational methodology of score
distillation, taking advantage of the properties of semi-implicit
distributions.

######  Identity 3 (Score Projection Identity).

Given the intractability of $\nabla_{\bm{x}_{t}}\ln p_{\theta}(\bm{x}_{t})$,
we introduce a projection vector to estimate the expected value of its product
with the score:

$\displaystyle\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}\left[u^{T}(\bm{x}_{t})\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t})\right]$
$\displaystyle\\!\\!=\mathbb{E}_{(\bm{x}_{t},\bm{x}_{g})\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})}\left[u^{T}(\bm{x}_{t})\nabla_{\bm{x}_{t}}\ln
q(\bm{x}_{t}\,|\,\bm{x}_{g})\right].$

This identity was leveraged by Vincent (2011) to draw a parallel between the
explicit score matching (ESM) loss,

(6)

and denoising score matching (DSM) loss, given by

(7)

Integrating the DSM loss with Unet architectures (Ronneberger et al., 2015)
and stochastic-gradient Langevin dynamics (Welling & Teh, 2011) based reverse
sampling, Song & Ermon (2019) have elevated score-based, or diffusion, models
to a prominent position in deep generative modeling. Additionally, Yu & Zhang
(2023) used this identity for semi-implicit variational inference (Yin & Zhou,
2018), while Yu et al. (2023) applied it to refine multi-step reverse
diffusion.

Distinct from these prior applications of this identity, we integrate it with
two previously discussed identities. This fusion, combined with a model-based
score-matching loss, culminates in a unique loss mechanism facilitating
single-step distillation of a pretrained diffusion model.

##  4 SiD: Score identity Distillation

In this section, we introduce the model-based score-matching loss as the
theoretical basis for our distillation loss. We then demonstrate how the three
identities previously discussed can be fused to approximate this loss.

###  4.1 Model-based Explicit Score Matching (MESM)

Assuming the existence of a diffusion model for the data, with parameter
$\phi$ pretrained to estimate the score $\nabla_{\bm{x}_{t}}\ln
p_{\text{data}}(\bm{x}_{t})$, we use the following approximation:

$\displaystyle\textstyle\nabla_{\bm{x}_{t}}\ln
p_{\text{data}}(\bm{x}_{t})\approx
S_{\phi}(\bm{x}_{t}):={\sigma_{t}^{-2}}({f_{\phi}(\bm{x}_{t},t)-\bm{x}_{t}}).$

In other words, we adopt
$f_{\phi}(\bm{x}_{t},t)\approx\mathbb{E}[\bm{x}_{0}|\bm{x}_{t}]$ as our
approximation, according to (4) in Identity 1. Our goal is to distill the
knowledge encapsulated in $\phi$, extracting which for data generation
typically requires many iterations through the same network
$f_{\phi}(\bm{\cdot},\bm{\cdot})$.

We use the pretrained score $S_{\phi}(\bm{x}_{t})$ to construct our
distillation loss. Our aim is to train $G_{\theta}$ to distill the iterative,
multi-step reverse diffusion process into a single network evaluation step.
For a specific reverse diffusion time step $t\sim p(t)$, we define the
theoretical distillation loss as

$\displaystyle{\mathcal{L}}_{\theta}=\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[\|\delta_{\phi,\theta}(\bm{x}_{t})\|_{2}^{2}],$ (8)
$\displaystyle\delta_{\phi,\theta}(\bm{x}_{t}):=S_{\phi}(\bm{x}_{t})-\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t}).$ (9)

We refer to $\delta_{\phi,\theta}(\bm{x}_{t})$ as score difference and
designate its L2 norm ${\mathcal{L}}_{\theta}$ as the model-based explicit
score-matching (MESM) loss, also known in the literature as a Fisher
divergence (Lyu, 2009; Holmes & Walker, 2017; Yang et al., 2019; Yu & Zhang,
2023). This differs from the ESM loss defined in (6) as the expectation is
computed with respect to the diffused fake data distribution
$p_{\theta}(\bm{x}_{t})$ rather than diffused real data distribution
$p_{\text{data}}(\bm{x}_{t})$.

A common assumption is that the performance of the student model used for
distillation will be limited by the outcomes of reverse diffusion using
$S_{\phi}(\bm{x}_{t})$, the teacher model. However, our results demonstrate
that the student model, utilizing single-step generation, can indeed exceed
the performance of the teacher model, EDM of Karras et al. (2022), which
relies on iterative refinement. This indicates that the aforementioned
hypothesis might not necessarily hold true. It implies that reverse diffusion
could accumulate errors throughout its process, even with very fine-grained
reverse steps and the use of advanced numerical solvers designed to counteract
error accumulations.

###  4.2 Loss Approximation based on Identities 1 and 2

To estimate the score $\nabla_{\bm{x}_{t}}\ln p_{\theta}(\bm{x}_{t})$, an
initial thought would be to adopt a deep neural network-based approximation
$f_{\psi}(\bm{x}_{t},t)\approx\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}]$ by (5),
which can be trained with the usual diffusion or denoising score-matching loss
as

(10)

where the timestep distribution $t\sim p(t)$ and weighting function
$\gamma(t)$ can be defined as in Karras et al. (2022). Assuming
$\bm{x}_{t}\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g}),\,\bm{x}_{g}=G_{\theta}(\bm{z}),\,\bm{z}\sim
p(\bm{z})$, the optimal solution $\psi^{*}(\theta)$, which depends on the
generator distribution determined by $\theta$, satisfies

$f_{\psi^{*}(\theta)}(\bm{x}_{t},t)=\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}]=\bm{x}_{t}+\sigma_{t}^{2}\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t})$

and we can express the score difference defined in (9) as

$\displaystyle\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})=\sigma_{t}^{-2}(f_{\phi}(\bm{x}_{t},t)-f_{\psi^{*}(\theta)}(\bm{x}_{t},t)).$
(11)

As $\psi^{*}(\theta)$ depends on $\theta$, the minimization of
$\mathcal{L}_{\theta}$ in (8) could potentially be cast as a bilevel
optimization problem (Ye et al., 1997; Hong et al., 2023; Shen et al., 2023).

It is tempting to estimate the score difference
$\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})$ using an approximated score
difference defined as

$\displaystyle{\delta}_{\phi,\psi}(\bm{x}_{t}):=\sigma_{t}^{-2}(f_{\phi}(\bm{x}_{t},t)-f_{\psi}(\bm{x}_{t},t)),$
(12)

which means we approximate $\psi^{*}(\theta)$ with $\psi$, ignoring its
dependence on $\theta$, and define an approximated MESM loss
${\mathcal{L}}_{\theta}^{(1)}$ as

$\displaystyle{\mathcal{L}}_{\theta}\approx{\mathcal{L}}_{\theta}^{(1)}:=\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}\left[\|{\delta}_{\phi,\psi}(\bm{x}_{t})\|_{2}^{2}\right].$
(13)

However, defining the score approximation error as

$\displaystyle\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t}):=\sigma_{t}^{-2}({f_{\psi}(\bm{x}_{t},t)-f_{\psi^{*}(\theta)}(\bm{x}_{t},t)}),$
(14)

we have
${\delta}_{\phi,\psi}(\bm{x}_{t})={\delta}_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})-\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})$
and

$\displaystyle{\mathcal{L}}_{\theta}^{(1)}={\mathcal{L}}_{\theta}\,+\,$
$\displaystyle\mathbb{E}_{p_{\theta}(\bm{x}_{t})}\big{[}\|\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})\|_{2}^{2}$
$\displaystyle-2\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})\big{]}.$
(15)

Therefore, how well ${\mathcal{L}}_{\theta}^{(1)}$ approximates the true loss
${\mathcal{L}}_{\theta}$ heavily depends on not only the score approximation
error $\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})$ but also the score
difference $\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})$. For a given $\theta$,
although one can control $\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})$ by
minimizing (10), it would be difficult to ensure that influence of the score
difference $\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})$ would not dominate the
true loss ${\mathcal{L}}_{\theta}$, especially during the intial phase of
training when $p_{\theta}(\bm{x}_{t})$ does not match
$p_{\text{data}}(\bm{x}_{t})$ well.

This concern is confirmed through the distillation of EDM models pretrained on
CIFAR-10, employing a loss estimated via reparameterization and Monte Carlo
estimation as

$\displaystyle\hat{{\mathcal{L}}}_{\theta}^{(1)}=\|{\delta}_{\phi,\psi}(\bm{x}_{t})\|_{2}^{2},\leavevmode\nobreak\
$ (16)
$\displaystyle\bm{x}_{t}=\bm{x}_{g}+\sigma_{t}\bm{\epsilon}_{t},\,\,\bm{\epsilon}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
(17)
$\displaystyle\bm{x}_{g}=G_{\theta}(\sigma_{\text{init}}\bm{z}),\leavevmode\nobreak\
\bm{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}).$ (18)

This loss fails to yield meaningful results. Below, we present a toy example
that highlights a failure case when using $\hat{\mathcal{L}}_{\theta}^{(1)}$
as the loss function to optimize $\theta$.

######  Proposition 4 (An example failure case).

Suppose $p_{\text{data}}(x_{0})=\mathcal{N}(0,1)$,
$p_{data}(x_{t})=\mathcal{N}(0,1+\sigma_{t}^{2})$,
$q(x_{t}\,|\,x_{g})=\mathcal{N}(x_{g},\sigma_{t}^{2})$, and
$p_{\theta}(x_{g})=\mathcal{N}(\theta,1)$. Assume $\psi^{*}(\theta)=\theta$
and
$f_{\psi}(x_{t},t)=x_{t}(1+\sigma_{t}^{2})^{-1}+\psi\sigma_{t}^{2}(1+\sigma_{t}^{2})^{-1}$.
Then we have

  1. (i)

$\delta_{\phi,\psi^{*}(\theta)}(x_{t})=-\frac{\theta}{1+\sigma_{t}^{2}}$,
$\delta_{\phi,\psi}(x_{t})=-\frac{\psi}{1+\sigma_{t}^{2}}$;

  2. (ii)

$\mathcal{L}_{\theta}=\frac{\theta^{2}}{(1+\sigma_{t}^{2})^{2}}$,
$\hat{\mathcal{L}}_{\theta}^{(1)}=\frac{\psi^{2}}{(1+\sigma_{t}^{2})^{2}}$.

The proof is presented in Appendix B. The example in this proposition shows
that although minimizing the objective $\mathcal{L}_{\theta}$ leads to the
optimal generator parameter $\theta^{*}=0$, the loss
$\hat{\mathcal{L}}_{\theta}^{(1)}$ would provide no meaningful gradient
towards $\theta^{*}$.

###  4.3 Loss Approximation via Projected Score Matching

We provide an alternative formulation of the MESM loss:

######  Theorem 5 (Projected Score Matching).

The MESM loss in (8) can be equivalently expressed as

(19)

The proof, based on Identity 3, is deferred to Appendix A. We approximate the
loss by substituting $\psi^{*}(\theta)$ in (19) with its approximation $\psi$,
leading to an approximated loss ${\mathcal{L}}_{\theta}^{(2)}$ as

$\displaystyle{\mathcal{L}}_{\theta}^{(2)}$

$=\mathbb{E}_{q(\bm{x}_{t}\,|\,\bm{x}_{g},t)p_{\theta}(\bm{x}_{g})}\left[\sigma_{t}^{-2}{\delta_{\phi,\psi}(\bm{x}_{t})^{T}(f_{\phi}(\bm{x}_{t},t)-\bm{x}_{g})}\right]$

$\displaystyle\\!\\!={\mathcal{L}}_{\theta}-\,\mathbb{E}_{q(\bm{x}_{t}\,|\,\bm{x}_{g},t)p_{\theta}(\bm{x}_{g})}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\left[\sigma_{t}^{-2}{\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}(f_{\phi}(\bm{x}_{t},t)-\bm{x}_{g})}\right].$
(20)

Comparing (20) to (15) indicates that ${\mathcal{L}}_{\theta}^{(2)}$ is
directly influenced by neither the norm
$\|\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})\|_{2}^{2}$ nor the score
difference $\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})$ given by (11).
Initially in training, the discrepancy between the estimated and actual scores
for the generator distribution may amplify the value of
$\triangle_{\psi,\psi^{*}(\theta)}(\bm{x}_{t})$, whereas the difference
between the pre-trained score for the real data distribution and the actual
score for the generator distribution may inflate
$\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})$. By contrast, the term
$f_{\phi}(\bm{x}_{t},t)-\bm{x}_{g}$ within (20) reflects the efficacy of the
pre-trained model in denoising corrupted fake data, which tends to be more
stable.

Let’s verify the failure case for ${\mathcal{L}}_{\theta}^{(1)}$ and see
whether it is still the case for ${\mathcal{L}}_{\theta}^{(2)}$.

######  Proposition 6.

Under the setting of Proposition 4, the gradient of loss
${\mathcal{L}}_{\theta}^{(2)}$ can be estimated as

$\displaystyle\nabla_{\theta}\hat{L}_{\theta}^{(2)}=-({1+\sigma_{t}^{2}})^{-1}{\delta}_{\phi,\psi}(\bm{x}_{t})\nabla_{\theta}G_{\theta}(\sigma_{\text{init}}\bm{z}),$

which involves the product of the approximated score difference
${\delta}_{\phi,\psi}(\bm{x}_{t})=-\frac{\psi}{1+\sigma_{t}^{2}}$ and the
gradient of the generator.

We note the product of the approximated score difference and
$\nabla_{\theta}G_{\theta}(\sigma_{\text{init}}\bm{z})$ is used to construct
the loss for Diff-Instruct (Luo et al., 2023c), which has been shown to be
able to distill a pretrained diffusion model with satisfactory performance.
Thus for the toy example where ${\mathcal{L}}_{\theta}^{(1)}$ fails, using
${\mathcal{L}}_{\theta}^{(2)}$ can provide useful gradient to guide the
generator.

 102k (400)

$\alpha=-0.25$ | $\alpha=0$ | $\alpha=0.5$ | $\alpha=0.75$ | $\alpha=1$ | $\alpha=1.2$ | $\alpha=1.5$
---|---|---|---|---|---|---
|  |  |  |  |  |

 512k (2000)

|  |  |  |  |  |
---|---|---|---|---|---|---

  1024k (4000)

|  |  |  |  |  |
---|---|---|---|---|---|---

  10240k (40000)

|  |  |  |  |  |
---|---|---|---|---|---|---

Figure 2:  Ablation Study of $\alpha$: The SiD generator, configured with
various $\alpha$ values, was trained with its own synthesized images at a
batch size of 256. The results, sorted by specific $\alpha$ values, are
displayed in columns. Sequentially from top to bottom, the rows are labeled
with both the total number of training images and the corresponding number of
iterations, denoted as “number of images (iterations).” This labeling approach
indicates the cumulative count of fake images utilized during training,
corresponding to iterations of 400, 2,000, 4,000, and 40,000, progressing from
the first row to the last. Across the $\alpha$ values of 0.5, 0.75, 1, and
1.2, minor differences are noted in both the Inception Score (IS) and visual
quality, yet the Fréchet Inception Distance (FID) shows notable variations, as
detailed in Fig. 3.

Figure 3: Ablation Study of $\alpha$: Each plot illustrates the relation
between the performance, measured by Inception Score and FID vs. the number of
every thousand images processed during the distillation of the EDM model
pretrained on CIFAR-10 (unconditional), across varying values of $\alpha$. The
study underscores the impact of $\alpha$ on both training efficiency and
generative fidelity, leading us to select $\alpha\in\\{1.0,1.2\\}$ for all
subsequent experiments.

###  4.4 Fused Loss of SiD

Examining ${\mathcal{L}}_{\theta}^{(2)}$ and ${\mathcal{L}}_{\theta}^{(1)}$
unveils their interconnections:

$\displaystyle{\mathcal{L}}_{\theta}^{(2)}=\leavevmode\nobreak\ $
$\displaystyle{\mathcal{L}}_{\theta}^{(1)}+\mathbb{E}_{\bm{x}_{g}\sim
p_{\theta}(\bm{x}_{g})}\mathbb{E}_{\bm{x}_{t}\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g},t)}$
$\displaystyle\left[{\sigma_{t}^{-2}}{{\delta}_{\phi,\psi}(\bm{x}_{t})^{T}(f_{\psi}(\bm{x}_{t},t)-\bm{x}_{g})}\right].$
(21)

Empirically, while $\hat{{\mathcal{L}}}_{\theta}^{(1)}$ fails, our
distillation experiments on CIFAR-10 reveal that
$\hat{{\mathcal{L}}}_{\theta}^{(2)}$ performs well in terms of Inception Score
(IS), but yields poor FID. This outcome is illustrated in the visualizations
for $\alpha=0$ in Figs. 2 and 3.

Visual inspection indicates that the generated images are darker in comparison
to the training images. Given that $\hat{{\mathcal{L}}}_{\theta}^{(1)}$ fails
while $\hat{{\mathcal{L}}}_{\theta}^{(2)}$ shows promis, albeit with poor FID
due to mismatched color, we hypothesize that the difference term

$\hat{{\mathcal{L}}}_{\theta}^{\triangle}=\hat{{\mathcal{L}}}_{\theta}^{(2)}-\hat{{\mathcal{L}}}_{\theta}^{(1)}={\sigma_{t}^{-2}}{\delta}_{\phi,\psi}(\bm{x}_{t})^{T}(f_{\psi}(\bm{x}_{t},t)-\bm{x}_{g})$

directs the gradient towards the desired direction.

Thus we are propelled to consider the loss

$\displaystyle{\mathcal{L}}_{\theta}^{(2)}-\alpha{\mathcal{L}}_{\theta}^{(1)}=(1-\alpha){\mathcal{L}}_{\theta}^{(1)}+{\mathcal{L}}_{\theta}^{\triangle}.$
(22)

We empirically find that setting $\alpha\in[-0.25,1.2]$ produces visually
coherent images, with $\alpha\in[0.75,1.2]$ typically leading to superior
results, as shown in Figs. 2 and 3.

In summary, the weighted loss is expressed as

$\displaystyle\textstyle\tilde{L}_{\theta}(\bm{x}_{t},t,\phi,\psi)=(1-\alpha)\frac{\omega(t)}{{\sigma_{t}^{4}}}\|f_{\phi}(\bm{x}_{t},t)-f_{\psi}(\bm{x}_{t},t)\|_{2}^{2}$
$\displaystyle\textstyle+\frac{\omega(t)}{{\sigma_{t}^{4}}}(f_{\phi}(\bm{x}_{t},t)-f_{\psi}(\bm{x}_{t},t))^{T}(f_{\psi}(\bm{x}_{t},t)-\bm{x}_{g}),$
(23)

where $\bm{x}_{t}$ is generated as in (18) and $\omega(t)$ are weighted
coefficients that need to be specified. To compute the gradient of the above
equation, SiD backpropagates the gradient through both $\phi$ and $\psi$ by
calculating two score gradients ($i.e.$, gradients of scores) as

$\displaystyle\textstyle\nabla_{\theta}f_{\phi}(\bm{x}_{t},t)=\frac{\partial
f_{\phi}(\bm{x}_{t},t)}{\partial\bm{x}_{t}}\nabla_{\theta}G_{\theta}(\sigma_{\text{init}}\bm{z})$
(24)
$\displaystyle\textstyle\nabla_{\theta}f_{\psi}(\bm{x}_{t},t)=\frac{\partial
f_{\psi}(\bm{x}_{t},t)}{\partial\bm{x}_{t}}\nabla_{\theta}G_{\theta}(\sigma_{\text{init}}\bm{z}).$

This feature distinguishes SiD from Diff-Instruct and DMD that do not use
score gradients $\frac{\partial
f_{\phi}(\mathbf{x}_{t},t)}{\partial\bm{x}_{t}}$ and $\frac{\partial
f_{\psi}(\bm{x}_{t},t)}{\partial\bm{x}_{t}}$.

###  4.5 Noise Weighting and Scheduling

The proposed SiD algorithm iteratively updates the score estimation parameters
$\psi$, given $\theta$, following (10), and updates the generator parameters
$\theta$, given $\psi$, as per (23). This alternating update scheme aligns
with related approaches (Wang et al., 2023c; Luo et al., 2023c; Yin et al.,
2023). Consequently, we largely adopt the methodology outlined by Luo et al.
(2023c) and Yin et al. (2023) for setting model parameters, including
weighting coefficients $\omega(t)$ and the distribution of $t\sim p(t)$.
Specifically, denoting $C$ as the total pixel count of an image and
$\|\cdot\|_{1,sg}$ as the L1 norm combined with the stop gradient operation,
we define

$\displaystyle\omega(t)=C\sigma_{t}^{4}/\|\bm{x}_{g}-f_{\phi}(\bm{x}_{t},t)\|_{1,sg}\leavevmode\nobreak\
\leavevmode\nobreak\ .$ (25)

Choosing $\sigma_{\min}=0.002$, $\sigma_{\max}=80$, $\rho=7.0$, and
$t_{\max}\in[0,1000]$, we sample $t\sim\mbox{Unif}[0,t_{\max}/1000]$ and
define the noise levels as

$\displaystyle\sigma_{t}=\big{(}\sigma_{\max}^{\frac{1}{\rho}}+\textstyle(1-t)(\sigma_{\min}^{\frac{1}{\rho}}-\sigma_{\max}^{\frac{1}{\rho}})\big{)}^{\rho}.$
(26)

The distillation process is outlined in Algorithm 1. The one-step generation
procedure is straightforward:

$\bm{x}=G_{\theta}(\sigma_{\text{init}}\bm{z}),\leavevmode\nobreak\
\bm{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$

where $\sigma_{\text{init}}$, by default set to 2.5, remains consistent
throughout distillation and generation.

Algorithm 1 Score identity Distillation (SiD)

Input: Pretrained score network $f_{\phi}$, generator $G_{\theta}$, generator
score network $f_{\psi}$, $\sigma_{\text{init}}=2.5$, $t_{\max}=800$,
$\alpha=1.2$

Initialization $\theta\leftarrow\phi,\psi\leftarrow\phi$

repeat

Sample $\bm{z}\sim\mathcal{N}(0,\mathbf{I})$ and
$\bm{x}_{g}=G_{\theta}(\sigma_{\text{init}}\bm{z})$

Sample $t\sim p(t),\leavevmode\nobreak\
\bm{\epsilon}_{t}\sim\mathcal{N}(0,\mathbf{I})$ and
$\bm{x}_{t}=\bm{x}_{g}+\sigma_{t}\bm{\epsilon}_{t}$

Update $\psi$ with Equation 10:

$\mathcal{L}_{\psi}={\gamma(t)}\|f_{\psi}(\bm{x}_{t},t)-\bm{x}_{g}\|_{2}^{2}$

$\psi=\psi-\eta\nabla_{\psi}\mathcal{L}_{\psi}$

where the timestep distribution $t\sim p(t)$, noise level $\sigma_{t}$, and
weighting function $\gamma(t)$ are defined as in Karras et al. (2022).

Sample $t\sim\mbox{Unif}[0,t_{\max}/1000]$, compute $\sigma_{t}$ with Equation
26, compute $\omega_{t}$ with Equation 25, and update $G_{\theta}$ with
Equation 23:

$\tilde{\mathcal{L}}_{\theta}=(1-\alpha)\frac{\omega(t)}{{\sigma_{t}^{4}}}\|f_{\phi}(\bm{x}_{t},t)-f_{\psi}(\bm{x}_{t},t)\|_{2}^{2}$

$+\frac{\omega(t)}{{\sigma_{t}^{4}}}(f_{\phi}(\bm{x}_{t},t)-f_{\psi}(\bm{x}_{t},t))^{T}(f_{\psi}(\bm{x}_{t},t)-\bm{x}_{g})$

$\theta=\theta-\eta\nabla_{\theta}\tilde{\mathcal{L}}_{\theta}$

until the FID plateaus or the training budget is exhausted

Output: $G_{\theta}$

Table 1: Comparison of various deep generative models trained on CIFAR-10
without label conditioning. The best and second-best one/few-step generators
under the FID or IS metric are highlighted with bold and italic bold,
respectively.

Family Model NFE FID ($\downarrow$) IS ($\uparrow$) Teacher VP-EDM (Karras et
al., 2022) 35 1.97 9.68 Diffusion DDPM (Ho et al., 2020) 1000 3.17
9.46$\pm$0.11 DDIM (Song et al., 2020) 100 4.16 DPM-Solver-3 (Lu et al., 2022)
48 2.65 VDM (Kingma et al., 2021) 1000 4.00 iDDPM (Nichol & Dhariwal, 2021)
4000 2.90 HSIVI-SM (Yu et al., 2023) 15 4.17 TDPM (Zheng et al., 2023a) 5 3.34
TDPM+ (Zheng et al., 2023a) 100 2.83 9.34 VP-EDM+LEGO-PR (Zheng et al., 2023c)
35 1.88 9.84 One Step NVAE (Vahdat & Kautz, 2020) 1 23.5 StyleGAN2+ADA (Karras
et al., 2020) 1 5.33$\pm$0.35 10.02$\pm$0.07 StyleGAN2+ADA+Tune (Karras et
al., 2020) 1 2.92$\pm$0.05 9.83$\pm$0.04 CT-StyleGAN2 (Zheng & Zhou, 2021) 1
2.9$\pm$0.4 10.1$\pm$0.1 StyleGAN2+ DiffAug (Zhao et al., 2020) 1 5.79
ProjectedGAN (Sauer et al., 2021) 1 3.10 DiffusionGAN (Wang et al., 2023c) 1
3.19 Diffusion ProjectedGAN (Wang et al., 2023c) 1 2.54 KD (Luhman & Luhman,
2021) 1 9.36 TDPM (Zheng et al., 2023a) 1 7.34 PD (Salimans & Ho, 2022) 1 8.34
8.69 Score Mismatching (Ye & Liu, 2023) 1 8.10 2-ReFlow (Liu et al., 2022b) 1
4.85 9.01 DFNO (Zheng et al., 2023b) 1 3.78 CD-LPIPS (Song et al., 2023) 1
3.55 9.48 iCT (Song & Dhariwal, 2023) 1 2.83 9.54 iCT-deep (Song & Dhariwal,
2023) 1 2.51 9.76 G-distill (Meng et al., 2023) ($w$=0.3) 1 7.34 8.9 GET-Base
(Geng et al., 2023) 1 6.91 9.16 Diff-Instruct (Luo et al., 2023c) 1 4.53 9.89
StyleGAN2+ADA+Tune+DI (Luo et al., 2023c) 1 2.71 9.86$\pm$0.04 PID (Tee et
al., 2024) 1 3.92 9.13 TRACT (Berthelot et al., 2023) 1 3.78 DMD (Yin et al.,
2023) 1 3.77 CTM (Kim et al., 2023) 1 1.98 SiD (ours), $\alpha=1.0$ 1 2.028
$\pm$0.020 10.017$\pm$0.047 SiD (ours), $\alpha=1.2$ 1 1.923$\pm$0.017
9.980$\pm$ 0.042

Figure 4: Evolution of FIDs for the SiD generator during the distillation of
the EDM teacher model pretrained on CIFAR-10 (unconditional), using
$\alpha=1.0$ or $\alpha=1.2$ and a batch size of 256. The performance of EDM,
along with DMD and Diff-Instruct, is depicted with horizontal lines in purple,
green, and red, respectively.

##  5 Experimental Results

Initially, we demonstrate the capability of the Score Identity Distillation
(SiD) generator to rapidly train and generate photo-realistic images by
leveraging the pretrained score network and its own synthesized fake images.
Subsequently, we conduct an ablation study to investigate the impact of the
parameter $\alpha$ and discuss the settings of several other parameters.
Through extensive experimentation, we assess both the effectiveness and
efficiency of SiD in the context of diffusion-based image generation.

Table 2: Analogous to Table 1 for CIFAR-10 (conditional).

Family Model NFE FID ($\downarrow$) Teacher VP-EDM (Karras et al., 2022) 35
1.79 Direct generation BigGAN (Brock et al., 2019) 1 14.73 StyleGAN2+ADA
(Karras et al., 2020) 1 3.49$\pm$0.17 StyleGAN2+ADA+Tune (Karras et al., 2020)
1 2.42$\pm$0.04 Distillation GET-Base (Geng et al., 2023) 1 6.25 Diff-Instruct
(Luo et al., 2023c) 1 4.19 StyleGAN2+ADA+Tune+DI (Luo et al., 2023c) 1 2.27
DMD (Yin et al., 2023) 1 2.66 DMD (w.o. KL) (Yin et al., 2023) 1 3.82 DMD
(w.o. reg.) (Yin et al., 2023) 1 5.58 CTM (Kim et al., 2023) 1 1.73 SiD
(ours), $\alpha=1.0$ 1 1.932$\pm$0.019 SiD (ours), $\alpha=1.2$ 1
1.710$\pm$0.011

Figure 5: Analogous to Fig. 4 for CIFAR-10 (conditional).

Datasets.  To thoroughly assess the effectiveness of SiD, we utilize four
representative datasets considered in Karras et al. (2022), including CIFAR-10
$32\times 32$ (cond/uncond) (Krizhevsky et al., 2009), ImageNet $64\times 64$
(Deng et al., 2009), FFHQ $64\times 64$ (Karras et al., 2019), and AFHQ-v2
$64\times 64$ (Choi et al., 2020).

Evaluation protocol.  We measure image generation quality using FID and
Inception Score (IS; Salimans et al. (2016)). Following Karras et al. (2019,
2022), we measure FIDs using 50k generated samples, with the training set used
by the EDM teacher model111https://github.com/NVlabs/edm as reference. We also
consider Precision and Recall (Kynkäänniemi et al., 2019) when evaluating SiD
on ImageNet 64x64, where we use a predefined reference
batch222https://openaipublic.blob.core.windows.net/diffusion/jul-2021/ref_batches/imagenet/64/VIRTUAL_imagenet64_labeled.npz
to compute both metrics333https://github.com/openai/guided-
diffusion/tree/main/evaluations (Dhariwal & Nichol, 2021; Nichol & Dhariwal,
2021; Song et al., 2023; Song & Dhariwal, 2023).

Implementation details.  We implement SiD based on the EDM (Karras et al.,
2022) code base and we initialize both the generator $G_{\theta}$ and its
score estimation network $f_{\psi}$ by copying the architecture and parameters
of the pretrained score network $f_{\phi}$ from EDM (Karras et al., 2022). We
provide the other implementation details in Appendix C.

Table 3: Analogous to Table 1 for ImageNet 64x64 with label conditioning. The
Precision and Recall metrics are also included.

Family Model NFE FID ($\downarrow$) Prec. ($\uparrow$) Rec. ($\uparrow$)
Teacher VP-EDM (Karras et al., 2022) 511 1.36 79 2.64 0.71 0.67 Direct
generation RIN (Jabri et al., 2022) 1000 1.23 DDPM (Ho et al., 2020) 250 11.00
0.67 0.58 ADM (Dhariwal & Nichol, 2021) 250 2.07 0.74 0.63 DPM-Solver-3 (Lu et
al., 2022) 50 17.52 HSIVI-SM (Yu et al., 2023) 15 15.49 U-ViT (Bao et al.,
2022) 50 4.26 DiT-L/2 (Peebles & Xie, 2023) 250 2.91 LEGO (Zheng et al.,
2023c) 250 2.16 iCT (Song & Dhariwal, 2023) 1 4.02 0.70 0.63 iCT-deep (Song &
Dhariwal, 2023) 1 3.25 0.72 0.63 Distillation PD (Salimans & Ho, 2022) 2 8.95
0.63 0.65 PD (Salimans & Ho, 2022) 1 15.39 0.59 0.62 G-distill (Meng et al.,
2023) ($w$=1.0) 1 7.54 G-distill (Meng et al., 2023) ($w$=0.3) 8 2.05 BOOT (Gu
et al., 2023) 1 16.3 0.68 0.36 PID (Tee et al., 2024) 1 9.49 DFNO (Zheng et
al., 2023b) 1 7.83 0.61 CD-LPIPS (Song et al., 2023) 2 4.70 0.69 0.64 CD-LPIPS
(Song et al., 2023) 1 6.20 0.68 0.63 Diff-Instruct (Luo et al., 2023c) 1 5.57
TRACT (Berthelot et al., 2023) 2 4.97 TRACT (Berthelot et al., 2023) 1 7.43
DMD (Yin et al., 2023) 1 2.62 CTM (Kim et al., 2023) 1 1.92 0.57 CTM (Kim et
al., 2023) 2 1.73 0.57 DMD (w.o. KL) (Yin et al., 2023) 1 9.21 DMD (w.o. reg.)
(Yin et al., 2023) 1 5.61 SiD (ours), $\alpha=1.0$ 1 2.022$\pm$0.031 0.73 0.63
SiD (ours), $\alpha=1.2$ 1 1.524$\pm$0.009 0.74 0.63

Figure 6: Analogous plot to Fig. 4 for ImageNet 64x64. The batch size is 8192.
See the results of batch size 1024 in Fig. 9.

Table 4: Analogous to Table 1 for FFHQ 64x64.

Family Model NFE FID ($\downarrow$) Teacher VP-EDM (Karras et al., 2022) 79
2.39 Diffusion VP-EDM (Karras et al., 2022) 50 2.60 Patch-Diffusion (Wang et
al., 2023a) 50 3.11 Distillation BOOT (Gu et al., 2023) 1 9.0 SiD (ours),
$\alpha=1.0$ 1 1.710 $\pm$ 0.018 SiD (ours), $\alpha=1.2$ 1 1.550 $\pm$ 0.017

Figure 7: Analogous plot to Fig. 4 for FFHQ 64x64. The batch size is 512.

Table 5: Analogous to Table 1 for AFHQ-v2 64x64.

Family Model NFE FID ($\downarrow$) Teacher VP-EDM (Karras et al., 2022) 79
1.96 Distillation SiD (ours), $\alpha=1.0$ 1 1.628$\pm$0.017 SiD (ours),
$\alpha=1.2$ 1 1.711$\pm$0.020

Figure 8: Ananagous plot to Fig. 4 for AFHQ-v2 64x64. The batch size is 512.

###  5.1 Ablation Study and Parameter Settings

Impact of $\alpha$. We conduct an ablation study to examine the impact of
$\alpha$ on SiD. In Fig. 2, we investigate a range of $\alpha$ values
$[-0.25,0.0,0.5,0.75,1.0,1.2,1.5]$ during SiD training on
CIFAR10-unconditional and visualize the changes in generation as training
progresses under each $\alpha$. For instance, the last row illustrates the
generation results when 10.24 million images (equivalent to 40,000 iterations
with a batch size of 256) are processed by SiD. In Fig. 3, we illustrate the
evolution of the FID and IS from iterations 0 to 8000 (corresponding to 0 to
1.024 million images), where the first plot depicts the IS evolution, while
the second plot shows the trajectory of FID.

The results indicate a stable performance of the model when $\alpha$ varies
from 0 to 1.2. A negative value of $\alpha$ results in large FIDs. This
observation supports our analysis in section 4.2 that directly optimizing
$\mathcal{L}_{\theta}^{(1)}$ given by (13) may not lead to meaningful
improvement, as our loss, shown in (22), is
$\mathcal{L}_{\theta}^{(2)}-\alpha\mathcal{L}_{\theta}^{(1)}$. As $\alpha$
increases within the tested range, we observe a gradual improvement in IS and
FID performance, peaking at $\alpha=1$ or $\alpha=1.2$. Based on these
findings, we select $\alpha=1$ or $\alpha=1.2$ for all our experiments,
although a more refined grid search on $\alpha$ might reveal even better
performance outcomes.

Setting of $\beta_{1}$. We investigate the $\beta_{1}$ parameter of the Adam
optimizer for the generator score network $f_{\psi}$ and the generator
$G_{\theta}$ by setting it as either $\beta_{1}=0$, the value used in
StyleGAN2 (Karras et al., 2020) and Diff-Instruct (Luo et al., 2023c), or
$\beta_{1}=0.9$, a commonly used value. We find that setting $\beta_{1}=0.9$
for $f_{\psi}$ often does not result in convergence, so we retain
$\beta_{1}=0$ for $f_{\psi}$ for all datasets. For learning $G_{\theta}$, we
did not observe significant benefits between setting $\beta_{1}=0$ and
$\beta_{1}=0.9$, except for the FFHQ dataset, where the FID improved by more
than 0.15 when changing from $\beta_{1}=0$ to $\beta_{1}=1$. Therefore, we set
$\beta_{1}=1$ for FFHQ while retaining $\beta_{1}=0$ for all other datasets.

Batch Size for ImageNet 64x64. For ImageNet 64x64, we initially set the batch
size to 1024 and observed an exponential decline in FID until it suddenly
diverged upon reaching or surpassing 2.62, the FID obtained by DMD (Yin et
al., 2023). The exact reason for this divergence is still unclear, but we
suspect it may be related to the FP16 precision used during optimization.
While switching to FP32 could potentially address the issue, we have not
explored this option due to its much higher computational and memory costs.

Instead, we increased the overall batch size from 1024 to 8192 (while keeping
the batch per GPU unchanged at 16, requiring more gradient accumulation
rounds) and reduced the learning rate from 5e-6 to 4e-6. Under $\alpha=1$, we
observed stable performance, while under $\alpha=1.2$, we observed occasional
spikes in FID. Upon examining the generations corresponding to these spikes,
as shown in the fifth image of Fig. 11 and Fig. 12 in the Appendix, we found
interesting patterns where certain uncommon features, such as nests containing
birds, were exaggerated. However, with a batch size as large as 8192, these
occasional spikes did not seem to significantly impact the overall declining
trend, which was roughly log-log linear initially and gradually leveled off.
With that said, when the batch size was reduced to 1024, the sudden divergence
could potentially be caused by such a spike, as observed in Fig. 9.

The drawback of using a larger batch size in this case is that it takes SiD
longer to outperform Diff-instruct and DMD, as clearly shown by comparing the
FID trajectories in Figs. 6 and 9. Although it’s feasible to develop more
advanced strategies, including progressively increasing the batch size,
annealing the learning rate, and implementing gradient clipping, we’ll reserve
these for future study.

###  5.2 Benchmark Performance

Our comprehensive evaluation compares SiD against leading deep generative
models, encompassing both distilled diffusion models and those built from
scratch. Random images generated by SiD in a single step are displayed in
Figs. 13-16 in the Appendix.

The comparative analysis, detailed in Tables 1-5 and illustrated in Figs. 4-8,
underlines the single-step SiD generator’s proficiency in leveraging the
insights from the pretrained EDM (teacher diffusion model) across a variety of
benchmarks, including CIFAR-10 (both conditional and unconditional formats),
ImageNet 64x64, FFHQ 64x64, and AFHQ-v2 64x64. Remarkably, the SiD-trained
generator surpasses the EDM teacher in nearly all tested environments,
showcasing its enhanced performance not just relative to the original multi-
step teacher model but also against a broad spectrum of cutting-edge models,
from traditional multi-step diffusion models to the latest single-step
distilled models and GANs. The sole deviation in this pattern occurs with
ImageNet 64x64, where SiD, at $\alpha=1.2$, attains an FID of 1.524, which is
exceeded by Jabri et al. (2022)’s RIN at 1.23 FID with 1000 steps and the
teacher model VP-EDM’s 1.36 FID with 511 steps.

Our assessment of SiD across various benchmarks has established, with the
exception of ImageNet 64x64, potentially the first instance, to our knowledge,
where a data-free diffusion distillation method outperforms the teacher
diffusion model using just a single generation step. This remarkable outcome
implies that reverse sampling, which utilizes the pretrained score function
for generating images across multiple steps and naturally accumulates
discretization errors during reverse diffusion, might not be as efficient as a
single-step distillation process. The latter, by sidestepping error
accumulation, could theoretically align perfectly with the true data
distribution when the model-based score-matching loss is completely minimized.

Among the single-step generators we’ve evaluated, CTM (Kim et al., 2023) is
SiD’s closest competitor in terms of generation performance. Despite the tight
competition, SiD not only surpasses CTM but is also noteworthy for its
independence from training data. In contrast, CTM’s performance relies on
access to training data and is augmented by the inclusion of an auxiliary GAN
loss. This distinction significantly amplifies SiD’s value, particularly in
contexts where accessing the original training data is either restricted or
impractical, and where data-specific GAN adjustments are undesirable.

In summary, SiD not only stands out in terms of performance metrics but also
simplifies the distillation process remarkably, operating without the need for
real data. It sets itself apart by employing a notably straightforward
distillation approach, unlike the complex multi-stage distillation strategy
seen in Salimans & Ho (2022), the dependency on pairwise regression data in
Yin et al. (2023), the use of additional GAN loss in Kim et al. (2023), or the
need to access training data outlined in Song et al. (2023).

Training Iterations.  In exploring SiD’s performance threshold, we initially
process 500 million SiD-generated synthetic images across most benchmarks. For
CIFAR-10 with label conditioning, this figure increases to 800 million
synthetic images for SiD with $\alpha=1.2$. In the case of ImageNet 64x64, we
extend the training for SiD with $\alpha=1.2$ to involve 1 billion synthetic
images. Through this extensive training, SiD demonstrates superior performance
over the EDM teacher model across all evaluated benchmarks, with the sole
exception of ImageNet 64x64, where EDM utilized 511 NFE. While we note a
gradual slowing down in the rate of FID improvements, the limit of potential
further reductions is not clear, indicating that with more iterations, SiD
might eventually outperform EDM on ImageNet 64x64 as well.

It’s noteworthy that to eclipse the achievements of rivals like Diff-instruct
and DMD, SiD requires significantly fewer synthetic images than the 500
million mark, thanks to its rapid FID reduction rate. This decline often
continues without evident stagnation, surpassing the teacher model’s
performance before the conclusion of the training. We delve into this aspect
further below.

Convergence Speed.  Our SiD generator, designed for distilling pretrained
diffusion models, rapidly achieves the capability to generate photo-realistic
images in a single step. This efficiency is showcased in Fig. 1 for the EDM
model pretrained on ImageNet 64x64 and in Fig. 2 for CIFAR 32x32
(unconditional). The performance of the SiD method is further highlighted in
Figs. 4-8, where the x-axis represents the thousands of images processed
during training. These figures track the FID’s evolution across four datasets
for both $\alpha=1$ and $\alpha=1.2$, demonstrating a roughly linear
relationship between the logarithm of the FID and the logarithm of the number
of processed images. This relationship indicates that FID decreases
exponentially as distillation progresses, a trend that is observed or expected
to eventually slow down and approach a steady state.

For instance, on CIFAR-10 (unconditional), SiD outperforms both Diff-Instruct
and DMD after processing under 20M images, achievable within fewer than 10
hours on 16 A100-40GB GPUs, or 20 hours on 8 V100-16GB GPUs. In the case of
ImageNet 64x64 with a batch size of 1024 and $\alpha=1.0$, SiD exceeds
Progressive Distillation (FID 15.39) of Salimans & Ho (2022) after only around
500k generator-synthesized images (equivalent to roughly 500 iterations with a
batch size of 1024), achieving FIDs lower than 5 after 7.5M images, below 4
after 13M, and under 3 after 31M images. It outperforms Diff-Instruct with
fewer than 7M images processed and DMD with under 40M images. When using a
larger batch size of 8192, SiD’s convergence is slower, yet it attains lower
FIDs: with $\alpha=1$, it outstrips Diff-Instruct after processing less than
20M images (under 20 hours on 16 A100-40GB GPUs), and with $\alpha=1.2$, it
beats DMD after fewer than 90M images (in under 45 hours on 16 A100-40GB
GPUs).

Limitations.  Despite setting a new benchmark in diffusion-based generation,
SiD entails the simultaneous management of three networks during the
distillation process: the pretrained score network $f_{\phi}$, the generator
score network $f_{\psi}$, and the generator $f_{\theta}$, which, in this
study, are maintained at equal sizes. This setup demands more memory compared
to traditional diffusion model training, which only necessitates retaining
$f_{\phi}$. However, the memory footprint of the two additional networks could
be notably reduced by employing LoRA (Hu et al., 2022) for both $f_{\psi}$ and
$f_{\theta}$, a possibility we aim to explore in future research.

Relative to Diff-Instruct, acknowledged for its memory and computational
efficiency in distillation, as detailed in Table 6 in the Appendix for 16
A100-40GB GPUs, the memory allocation per GPU of SiD has seen a rise of around
50% for ImageNet 64x64 and about 70% for CIFAR-10, FFHQ, and AFHQ. The
iteration time has increased by approximately 28% for CIFAR-10 and ImageNet
64x64, and by roughly 36% for the FFHQ and AFHQ datasets. This increase is
because Diff-Instruct does not require computing score gradients, as defined
in (24). By contrast, SiD necessitates computing score gradients, involving
backpropagation through both the pretrained and generator score networks—a
step not needed in Diff-Instruct—leading to about a one-third increase in
computing time per iteration.

##  6 Conclusion

We present Score Identity Distillation (SiD), an innovative method that
transforms pretrained diffusion models into a single-step generator. By
employing semi-implicit distributions, SiD aims to accomplish distillation
through the minimization of a model-based score-matching loss that aligns the
scores of diffused real and generative distributions across different noise
intensities. Experimental outcomes underscore SiD’s capability to
significantly reduce the Fréchet inception distance with remarkable efficiency
and outperform established generative approaches. This superiority extends
across various conditions, including those using single or multiple steps,
those requiring or not requiring access to training data, and those needing
additional loss functions in image generation.

## Impact Statement

The positive aspect of distilled diffusion models lies in their potential to
save energy and reduce costs. By simplifying and compressing large models, the
deployment of distilled models often requires less computational resources,
making them more energy-efficient and cost-effective. This can lead to
advancements in sustainable AI practices, especially in resource-intensive
applications.

However, the negative aspect arises when considering the ease of distilling
models trained on violent or pornographic data. This poses significant ethical
concerns, as deploying such models may inadvertently facilitate the generation
and dissemination of harmful content. The distillation process, intended to
transfer knowledge efficiently, could unintentionally amplify and perpetuate
inappropriate patterns present in the original data. This not only jeopardizes
user safety but also raises ethical and societal questions about the
responsible use of AI technology. Striking a balance between the positive
gains in energy efficiency and the potential negative consequences of
distilling inappropriate content is crucial for the responsible development
and deployment of AI models. Stringent ethical guidelines and oversight are
essential to mitigate these risks and ensure the responsible use of distilled
diffusion models.

## Acknowledgments

The authors would like to express their gratitude to Dr. Zhuoran Yang for his
valuable comments and suggestions.

## References

  * Bao et al. (2022) Bao, F., Li, C., Cao, Y., and Zhu, J.  All are worth words: A ViT backbone for score-based diffusion models.  _arXiv preprint arXiv:2209.12152_ , 2022.
  * Berthelot et al. (2023) Berthelot, D., Autef, A., Lin, J., Yap, D. A., Zhai, S., Hu, S., Zheng, D., Talbott, W., and Gu, E.  Tract: Denoising diffusion models with transitive closure time-distillation.  _arXiv preprint arXiv:2303.04248_ , 2023.
  * Brock et al. (2019) Brock, A., Donahue, J., and Simonyan, K.  Large scale GAN training for high fidelity natural image synthesis.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=B1xsqj09Fm.
  * Choi et al. (2020) Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8188–8197, 2020.
  * Chung et al. (2022) Chung, H., Sim, B., Ryu, D., and Ye, J. C.  Improving diffusion models for inverse problems using manifold constraints.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=nJJjv0JDJju.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  ImageNet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Efron (2011) Efron, B.  Tweedie’s formula and selection bias.  _Journal of the American Statistical Association_ , 106(496):1602–1614, 2011.
  * Geng et al. (2023) Geng, Z., Pokle, A., and Kolter, J. Z.  One-step diffusion distillation via deep equilibrium models.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , pp. 2672–2680, 2014.
  * Gu et al. (2023) Gu, J., Zhai, S., Zhang, Y., Liu, L., and Susskind, J. M.  BOOT: Data-free distillation of denoising diffusion models with bootstrapping.  In _ICML 2023 Workshop on Structured Probabilistic Inference $\\{$$\backslash$&$\\}$ Generative Modeling_, 2023.
  * Hasanzadeh et al. (2019) Hasanzadeh, A., Hajiramezanali, E., Narayanan, K., Duffield, N., Zhou, M., and Qian, X.  Semi-implicit graph variational auto-encoders.  _Advances in neural information processing systems_ , 32, 2019.
  * Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  In _Advances in Neural Information Processing Systems_ , pp. 6626–6637, 2017.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Ho et al. (2022) Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T.  Cascaded diffusion models for high fidelity image generation.  _J. Mach. Learn. Res._ , 23(47):1–33, 2022.
  * Holmes & Walker (2017) Holmes, C. C. and Walker, S. G.  Assigning a value to a power likelihood in a general Bayesian model.  _Biometrika_ , 104(2):497–503, 2017.
  * Hong et al. (2023) Hong, M., Wai, H.-T., Wang, Z., and Yang, Z.  A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic.  _SIAM Journal on Optimization_ , 33(1):147–180, 2023.
  * Hu et al. (2022) Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Jabri et al. (2022) Jabri, A., Fleet, D., and Chen, T.  Scalable adaptive computation for iterative generation.  _arXiv preprint arXiv:2212.11972_ , 2022.
  * Karras et al. (2019) Karras, T., Laine, S., and Aila, T.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4401–4410, 2019.
  * Karras et al. (2020) Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T.  Analyzing and improving the image quality of StyleGAN.  In _Proc. CVPR_ , 2020.
  * Karras et al. (2022) Karras, T., Aittala, M., Aila, T., and Laine, S.  Elucidating the design space of diffusion-based generative models.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=k7FuTOWMOc7.
  * Kim et al. (2023) Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S.  Consistency trajectory models: Learning probability flow ode trajectory of diffusion.  _arXiv preprint arXiv:2310.02279_ , 2023.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational Bayes.  In _International Conference on Learning Representations_ , 2014.
  * Kingma et al. (2021) Kingma, D. P., Salimans, T., Poole, B., and Ho, J.  Variational diffusion models.  _arXiv preprint arXiv:2107.00630_ , 2021.
  * Krizhevsky et al. (2009) Krizhevsky, A. et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kynkäänniemi et al. (2019) Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T.  Improved precision and recall metric for assessing generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lawson et al. (2019) Lawson, J., Tucker, G., Dai, B., and Ranganath, R.  Energy-inspired models: Learning with sampler-induced distributions.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lipman et al. (2022) Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M.  Flow matching for generative modeling.  _arXiv preprint arXiv:2210.02747_ , 2022.
  * Liu et al. (2022a) Liu, L., Ren, Y., Lin, Z., and Zhao, Z.  Pseudo numerical methods for diffusion models on manifolds.  In _International Conference on Learning Representations_ , 2022a.  URL https://openreview.net/forum?id=PlKWVd2yBkY.
  * Liu et al. (2022b) Liu, X., Gong, C., and Liu, Q.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  _arXiv preprint arXiv:2209.03003_ , 2022b.
  * Lu et al. (2022) Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J.  DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps.  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=2uAaGwlP_V.
  * Luhman & Luhman (2021) Luhman, E. and Luhman, T.  Knowledge distillation in iterative generative models for improved sampling speed.  _arXiv preprint arXiv:2101.02388_ , 2021.
  * Luo (2022) Luo, C.  Understanding diffusion models: A unified perspective.  _arXiv preprint arXiv:2208.11970_ , 2022.
  * Luo et al. (2023a) Luo, S., Tan, Y., Huang, L., Li, J., and Zhao, H.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _ArXiv_ , abs/2310.04378, 2023a.
  * Luo et al. (2023b) Luo, S., Tan, Y., Patil, S., Gu, D., von Platen, P., Passos, A., Huang, L., Li, J., and Zhao, H.  Lcm-lora: A universal stable-diffusion acceleration module, 2023b.
  * Luo et al. (2023c) Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, Z.  Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023c.  URL https://openreview.net/forum?id=MLIs5iRq4w.
  * Lyu (2009) Lyu, S.  Interpretation and generalization of score matching.  In _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_ , pp. 359–366, 2009.
  * Lyu et al. (2022) Lyu, Z., Xu, X., Yang, C., Lin, D., and Dai, B.  Accelerating diffusion models via early stop of the diffusion process.  _arXiv preprint arXiv:2205.12524_ , 2022.
  * Meng et al. (2023) Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14297–14306, 2023.
  * Moens et al. (2021) Moens, V., Ren, H., Maraval, A., Tutunov, R., Wang, J., and Ammar, H.  Efficient semi-implicit variational inference.  _arXiv preprint arXiv:2101.06070_ , 2021.
  * Molchanov et al. (2019) Molchanov, D., Kharitonov, V., Sobolev, A., and Vetrov, D.  Doubly semi-implicit variational inference.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pp. 2593–2602. PMLR, 2019.
  * Nguyen & Tran (2023) Nguyen, T. H. and Tran, A.  SwiftBrush: One-step text-to-image diffusion model with variational score distillation.  _arXiv preprint arXiv:2312.05239_ , 2023.
  * Nichol & Dhariwal (2021) Nichol, A. Q. and Dhariwal, P.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Pandey et al. (2022) Pandey, K., Mukherjee, A., Rai, P., and Kumar, A.  DiffuseVAE: Efficient, controllable and high-fidelity generation from low-dimensional latents.  _arXiv preprint arXiv:2201.00308_ , 2022.
  * Peebles & Xie (2023) Peebles, W. and Xie, S.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 4195–4205, 2023.
  * Poole et al. (2022) Poole, B., Jain, A., Barron, J. T., and Mildenhall, B.  Dreamfusion: Text-to-3d using 2d diffusion.  _ArXiv_ , abs/2209.14988, 2022.
  * Ramesh et al. (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M.  Hierarchical text-conditional image generation with CLIP latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rezende et al. (2014) Rezende, D. J., Mohamed, S., and Wierstra, D.  Stochastic backpropagation and approximate inference in deep generative models.  In _Proceedings of the 31st International Conference on Machine Learning_ , pp. 1278–1286, 2014.
  * Robbins (1992) Robbins, H. E.  An empirical Bayes approach to statistics.  In _Breakthroughs in Statistics: Foundations and basic theory_ , pp. 388–394. Springer, 1992.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10684–10695, 2022.
  * Ronneberger et al. (2015) Ronneberger, O., P.Fischer, and Brox, T.  U-Net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , volume 9351 of _LNCS_ , pp. 234–241. Springer, 2015.  URL http://lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a.  (available on arXiv:1505.04597 [cs.CV]).
  * Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans & Ho (2022) Salimans, T. and Ho, J.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=TIdIXIpzhoI.
  * Salimans et al. (2016) Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X.  Improved techniques for training GANs.  In _Advances in Neural Information Processing Systems_ , pp. 2234–2242, 2016.
  * Sauer et al. (2021) Sauer, A., Chitta, K., Müller, J., and Geiger, A.  Projected gans converge faster.  _Advances in Neural Information Processing Systems_ , 34:17480–17492, 2021.
  * Sauer et al. (2023) Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R.  Adversarial diffusion distillation.  _ArXiv_ , abs/2311.17042, 2023.
  * Shen et al. (2023) Shen, H., Xiao, Q., and Chen, T.  On penalty-based bilevel gradient descent method.  _arXiv preprint arXiv:2302.05185_ , 2023.
  * Sobolev & Vetrov (2019) Sobolev, A. and Vetrov, D. P.  Importance weighted hierarchical variational inference.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pp. 2256–2265. PMLR, 2015.
  * Song et al. (2020) Song, J., Meng, C., and Ermon, S.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2020.
  * Song & Dhariwal (2023) Song, Y. and Dhariwal, P.  Improved techniques for training consistency models.  _arXiv preprint arXiv:2310.14189_ , 2023.
  * Song & Ermon (2019) Song, Y. and Ermon, S.  Generative Modeling by Estimating Gradients of the Data Distribution.  In _Advances in Neural Information Processing Systems_ , pp. 11918–11930, 2019.
  * Song et al. (2023) Song, Y., Dhariwal, P., Chen, M., and Sutskever, I.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023.
  * Tee et al. (2024) Tee, J. T. J., Zhang, K., Kim, C., Gowda, D. N., Yoon, H. S., and Yoo, C. D.  Physics informed distillation for diffusion models, 2024.  URL https://openreview.net/forum?id=a24gfxA7jD.
  * Titsias & Ruiz (2019) Titsias, M. K. and Ruiz, F.  Unbiased implicit variational inference.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pp. 167–176. PMLR, 2019.
  * Vahdat & Kautz (2020) Vahdat, A. and Kautz, J.  NVAE: A deep hierarchical variational autoencoder.  In _Advances in neural information processing systems_ , 2020.
  * Vincent (2011) Vincent, P.  A Connection Between Score Matching and Denoising Autoencoders.  _Neural Computation_ , 23(7):1661–1674, 2011\.
  * Wang et al. (2023a) Wang, Z., Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., Chen, W., and Zhou, M.  Patch diffusion: Faster and more data-efficient training of diffusion models.  _arXiv preprint arXiv:2304.12526_ , 2023a.
  * Wang et al. (2023b) Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, J.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, 2023b.
  * Wang et al. (2023c) Wang, Z., Zheng, H., He, P., Chen, W., and Zhou, M.  Diffusion-GAN: Training GANs with diffusion.  In _The Eleventh International Conference on Learning Representations_ , 2023c.  URL https://openreview.net/forum?id=HZf7UbpWHuA.
  * Welling & Teh (2011) Welling, M. and Teh, Y. W.  Bayesian learning via stochastic gradient Langevin dynamics.  In _Proceedings of the 28th international conference on machine learning (ICML-11)_ , pp. 681–688. Citeseer, 2011.
  * Xiao et al. (2022) Xiao, Z., Kreis, K., and Vahdat, A.  Tackling the generative learning trilemma with denoising diffusion GANs.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=JprM0p-q0Co.
  * Xu et al. (2023) Xu, Y., Zhao, Y., Xiao, Z., and Hou, T.  UFOGen: You forward once large scale text-to-image generation via diffusion GANs.  _ArXiv_ , abs/2311.09257, 2023.
  * Yang et al. (2019) Yang, Y., Martin, R., and Bondell, H.  Variational approximations using Fisher divergence.  _arXiv preprint arXiv:1905.05284_ , 2019.
  * Ye et al. (1997) Ye, J., Zhu, D., and Zhu, Q. J.  Exact penalization and necessary optimality conditions for generalized bilevel programming problems.  _SIAM Journal on optimization_ , 7(2):481–507, 1997.
  * Ye & Liu (2023) Ye, S. and Liu, F.  Score mismatching for generative modeling.  _arXiv preprint arXiv:2309.11043_ , 2023.
  * Yin & Zhou (2018) Yin, M. and Zhou, M.  Semi-implicit variational inference.  In _International Conference on Machine Learning_ , pp. 5660–5669, 2018.
  * Yin et al. (2023) Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T.  One-step diffusion with distribution matching distillation, 2023.
  * Yu & Zhang (2023) Yu, L. and Zhang, C.  Semi-implicit variational inference via score matching.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=sd90a2ytrt.
  * Yu et al. (2023) Yu, L., Xie, T., Zhu, Y., Yang, T., Zhang, X., and Zhang, C.  Hierarchical semi-implicit variational inference with application to diffusion model acceleration.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=ghIBaprxsV.
  * Zhang & Chen (2023) Zhang, Q. and Chen, Y.  Fast sampling of diffusion models with exponential integrator.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=Loek7hfb46P.
  * Zhao et al. (2020) Zhao, S., Liu, Z., Lin, J., Zhu, J.-Y., and Han, S.  Differentiable augmentation for data-efficient gan training.  _Advances in Neural Information Processing Systems_ , 33:7559–7570, 2020.
  * Zheng & Zhou (2021) Zheng, H. and Zhou, M.  Exploiting chain rule and Bayes’ theorem to compare probability distributions.  _Advances in Neural Information Processing Systems_ , 34:14993–15006, 2021.
  * Zheng et al. (2023a) Zheng, H., He, P., Chen, W., and Zhou, M.  Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders.  In _The Eleventh International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=HDxgaKk956l.
  * Zheng et al. (2023b) Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K., and Anandkumar, A.  Fast sampling of diffusion models via operator learning.  In _International Conference on Machine Learning_ , pp. 42390–42402. PMLR, 2023b.
  * Zheng et al. (2023c) Zheng, H., Wang, Z., Yuan, J., Ning, G., He, P., You, Q., Yang, H., and Zhou, M.  Learning stackable and skippable LEGO bricks for efficient, reconfigurable, and variable-resolution diffusion modeling, 2023c.

##  Appendix A Proofs

###### Proof of Tweedie’s formula.

For Gaussian diffusion, we have (2), which we explore to derive the identity
shown below. While $p_{\theta}(\bm{x}_{t})$ often does not have an analytic
form, exploiting its semi-implicit construction, its score can be expressed as

$\displaystyle\nabla_{\bm{x}_{t}}\ln p_{\theta}(\bm{x}_{t})$
$\displaystyle=\frac{\int\nabla_{\bm{x}_{t}}q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{{\theta}}(\bm{x}_{g})d\bm{x}_{g}}{p_{\theta}(\bm{x}_{t})}$
$\displaystyle=\frac{\int q(\bm{x}_{t}\,|\,\bm{x}_{g})\nabla_{\bm{x}_{t}}\ln
q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})d\bm{x}_{g}}{p_{\theta}(\bm{x}_{t})}$
$\displaystyle=-\frac{\int
q(\bm{x}_{t}\,|\,\bm{x}_{g})\frac{\bm{x}_{t}-a_{t}\bm{x}_{g}}{\sigma_{t}^{2}}p_{\theta}(\bm{x}_{g})d\bm{x}_{g}}{p_{\theta}(\bm{x}_{t})}$
$\displaystyle=-\frac{\bm{x}_{t}}{\sigma_{t}^{2}}+\frac{a_{t}}{\sigma_{t}^{2}}\frac{\int\bm{x}_{g}q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})d\bm{x}_{g}}{p_{\theta}(\bm{x}_{t})}$
$\displaystyle=-\frac{\bm{x}_{t}}{\sigma_{t}^{2}}+\frac{a_{t}}{\sigma^{2}}\int\bm{x}_{g}q(\bm{x}_{g}\,|\,\bm{x}_{t})d\bm{x}_{g}$
$\displaystyle=-\frac{\bm{x}_{t}}{\sigma_{t}^{2}}+\frac{a_{t}}{\sigma_{t}^{2}}\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}]$
(27)

$\displaystyle\nabla_{\bm{x}_{t}}\ln p_{\text{data}}(\bm{x}_{t})$
$\displaystyle=\frac{\int\nabla_{\bm{x}_{t}}q(\bm{x}_{t}\,|\,\bm{x}_{0})p_{{\phi}}(\bm{x}_{0})d\bm{x}_{0}}{p_{\phi}(\bm{x}_{t})}$
(28)

Therefore, we have

$\displaystyle\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}]=\frac{\bm{x}_{t}+\sigma_{t}^{2}\nabla_{\bm{x}_{t}}\ln
q_{g}(\bm{x}_{t})}{a_{t}}$ (29)

which is known as the Tweedie’s formula. Setting $a_{t}=1$ recovers the
identity presented in the main body of the paper. ∎

###### Proof of Identity 3..

$\displaystyle\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[u^{T}(\bm{x}_{t})\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t})]$ $\displaystyle=\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[u^{T}(\bm{x}_{t})\frac{\nabla_{\bm{x}_{t}}p_{\theta}(\bm{x}_{t})}{p_{\theta}(\bm{x}_{t})}]$
$\displaystyle=\int{u^{T}(\bm{x}_{t})\nabla_{\bm{x}_{t}}p_{\theta}(\bm{x}_{t})}d\bm{x}_{t}$
$\displaystyle=\int
u^{T}(\bm{x}_{t})\int\nabla_{\bm{x}_{t}}q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})d\bm{x}_{g}d\bm{x}_{t}$
$\displaystyle=\int u^{T}(\bm{x}_{t})\int
q(\bm{x}_{t}\,|\,\bm{x}_{g})\nabla_{\bm{x}_{t}}\ln
q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})d\bm{x}_{g}d\bm{x}_{t}$
$\displaystyle=\mathbb{E}_{(\bm{x}_{t},\bm{x}_{g})\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g})p_{\theta}(\bm{x}_{g})}[u^{T}(\bm{x}_{t})\nabla_{\bm{x}_{t}}\ln
q(\bm{x}_{t}\,|\,\bm{x}_{g})].$ (30)

∎

###### Proof of Theorem 5.

Expanding the $L_{2}$ norm, we have

$\displaystyle\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}\|S(\bm{x}_{t})-\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t})\|_{2}^{2}$
$\displaystyle=\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[(\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]-\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}])^{T}(S(\bm{x}_{t})-\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t}))]$
$\displaystyle=\underbrace{\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[(\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]-\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}])^{T}S(\bm{x}_{t})}_{①}-\underbrace{\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[(\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]-\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}])^{T}\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t}))]}_{②}$ (31)

denote

$\displaystyle①$
$\displaystyle=\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}(\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]-\bm{x}_{t})]$
$\displaystyle=\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]]-\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\bm{x}_{t})]$
(32) $\displaystyle②$ $\displaystyle=\mathbb{E}_{\bm{x}_{g}\sim
p_{\theta}(\bm{x}_{g})}\mathbb{E}_{\bm{x}_{t}\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g},t)}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t}\,|\,\bm{x}_{g}))]$
$\displaystyle=\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{g}\sim
p_{\theta}(\bm{x}_{g})}\mathbb{E}_{\bm{x}_{t}\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g},t)}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}(\bm{x}_{g}-\bm{x}_{t})]$
$\displaystyle=\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{g}\sim
p_{\theta}(\bm{x}_{g})}\mathbb{E}_{\bm{x}_{t}\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g},t)}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\bm{x}_{g}]-\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{t}\sim
p_{\theta}(\bm{x}_{t})}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\bm{x}_{t}]$
(33)

Therefore we have

$\displaystyle L=①-②=\frac{1}{\sigma_{t}^{2}}\mathbb{E}_{\bm{x}_{g}\sim
p_{\theta}(\bm{x}_{g})}\mathbb{E}_{\bm{x}_{t}\sim
q(\bm{x}_{t}\,|\,\bm{x}_{g},t)}[\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}(\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]-\bm{x}_{g})]$
(34)

∎

##  Appendix B Analytic study of the toy example

We prove the conclusions in Propositions 4 and 6. Given
$p_{\text{data}}(\bm{x}_{0})=\mathcal{N}(\mathbf{0},\mathbf{I})$,
$p_{\text{data}}(\bm{x}_{t})=\mathcal{N}(0,1+\sigma_{t}^{2})$, and
$q(\bm{x}_{t}\,|\,\bm{x}_{g})=\mathcal{N}(\bm{x}_{g},\sigma_{t}^{2}),\leavevmode\nobreak\
p_{\theta}(\bm{x}_{g})=\mathcal{N}(\theta,1)$, the optimal value of $\theta$
would be $\theta^{*}=0$. The score

$\displaystyle S(\bm{x}_{t})=\nabla_{\bm{x}_{t}}\ln
p_{\text{data}}(\bm{x}_{t})=-\frac{\bm{x}_{t}}{1+\sigma_{t}^{2}}$
$\displaystyle\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t})=-\frac{\bm{x}_{t}-\theta}{1+\sigma_{t}^{2}}.$

Hence, the difference between the scores is
$\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})=-\frac{\theta}{1+\sigma_{t}^{2}}$.
By Tweedie’s formula

$\displaystyle
f_{\phi}(\bm{x}_{t},t)=\mathbb{E}[\bm{x}_{0}\,|\,\bm{x}_{t}]=\frac{\bm{x}_{t}}{1+\sigma_{t}^{2}}$
$\displaystyle\mathbb{E}[\bm{x}_{g}\,|\,\bm{x}_{t}]=\bm{x}_{t}\frac{1}{1+\sigma_{t}^{2}}+\theta\frac{\sigma_{t}^{2}}{1+\sigma_{t}^{2}}$

By assumption we have

$f_{\psi}(\bm{x}_{t},t)=\bm{x}_{t}\frac{1}{1+\sigma_{t}^{2}}+\psi\frac{\sigma_{t}^{2}}{1+\sigma_{t}^{2}},$

which means $\psi^{*}=\theta$, then by Equation 12 we have

$\displaystyle{\delta}_{\phi,\psi}(\bm{x}_{t})=\sigma_{t}^{-2}(f_{\phi}(\bm{x}_{t},t)-f_{\psi}(\bm{x}_{t},t))=-\frac{\psi}{1+\sigma_{t}^{2}}.$
(35)

Accordingly,

$\displaystyle\hat{L}_{\theta}^{(1)}={\delta}_{\phi,\psi}(\bm{x}_{t})^{T}{\delta}_{\phi,\psi}(\bm{x}_{t})=\frac{\psi^{2}}{(1+\sigma_{t}^{2})^{2}}$
(36)

Therefore, while
$\hat{L}_{\theta}=\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})^{T}\delta_{\phi,\psi^{*}(\theta)}(\bm{x}_{t})=\frac{\theta^{2}}{(1+\sigma_{t}^{2})^{2}}$
would provide useful gradient to learn $\theta$, its naive approximation
$\hat{L}_{\theta}^{(1)}$ could fail to provide meaningful gradient.

We can further compute

$\displaystyle\hat{L}_{\theta}^{(2)}$
$\displaystyle=\hat{L}_{\theta}^{(1)}+\frac{{\delta}_{\phi,\psi}(\bm{x}_{t})^{T}(f_{\psi}(\bm{x}_{t},t)-\bm{x}_{g})}{\sigma_{t}^{2}}$
$\displaystyle=\frac{\psi^{2}}{(1+\sigma_{t}^{2})^{2}}-\frac{\psi}{\sigma_{t}^{2}(1+\sigma_{t}^{2})}(\bm{x}_{t}\frac{1}{1+\sigma_{t}^{2}}+\psi\frac{\sigma_{t}^{2}}{1+\sigma_{t}^{2}}-\bm{x}_{g})$
$\displaystyle=\frac{\psi}{(1+\sigma_{t}^{2})^{2}}\left[\bm{x}_{g}-\frac{\epsilon_{t}}{\sigma_{t}}\right].$

Thus

$\displaystyle\nabla_{\theta}\hat{L}_{\theta}^{(2)}$
$\displaystyle=\frac{\psi}{(1+\sigma_{t}^{2})^{2}}\nabla_{\theta}G_{\theta}(z)$
$\displaystyle=-\frac{1}{1+\sigma_{t}^{2}}{\delta}_{\phi,\psi}(\bm{x}_{t})\nabla_{\theta}G_{\theta}(z)$
$\displaystyle\approx\frac{1}{1+\sigma_{t}^{2}}[\nabla_{\bm{x}_{t}}\ln
p_{\theta}(\bm{x}_{t})-S_{\phi}(\bm{x}_{t})]\nabla_{\theta}G_{\theta}(z).$

##  Appendix C Training and Evaluation Details and Additional Results.

The hyperparameters tailored for our study are outlined in Table 6, with all
remaining settings consistent with those in the EDM code (Karras et al.,
2022). The initial development of the SiD algorithm utilized a cluster with 8
Nvidia RTX A5000 GPUs. To support a mini-batch size up to 8192 for ImageNet
64x64, we adopted the gradient accumulation strategy. Extensive evaluations
across four diverse datasets were conducted using cloud computation nodes
equipped with either 16 Nvidia A100-40GB GPUs, 8 Nvidia V100-16GB GPUs, or 8
Nvidia H100-80GB GPUs, with most experiments performed on Nvidia A100-40GB
GPUs.

Comparisons of memory usage and per-iteration computation costs between SiD
and Diff-Instruct, utilizing 16 Nvidia A100-40GB GPUs, are detailed in Table
6.

We note the time and memory costs reported in Table 6 do not include these
used to evaluate the Fréchet Inception Distance (FID) of the single-step
generator during the distillation process. The FID for the SiD generator,
utilizing exponential moving average (ema), was evaluated after processing
each batch of 500k generator-synthesized fake images. We preserve the SiD
generator that achieves the lowest FID, and to ensure accuracy, we re-evaluate
it across 10 independent runs to calculate the corresponding metrics. It’s
worth noting that some prior studies have reported the best metric obtained
across multiple independent random runs, a practice that raises concerns about
reliability and reproducibility. We consciously avoid this approach in our
work to ensure a more robust and credible evaluation.

Table 6: Hyperparameter settings and comparison of distillation time and
memory usage between Diff-Instruct and SiD on 16 NVIDIA A100 GPUs with 40 GB
of memory each.

Method | Hyperparameters | CIFAR-10 32x32 | ImageNet 64x64 | FFHQ 64x64 | AFHQ-v2 64x64
---|---|---|---|---|---
| Batch size | 256 | 8192 | 512 | 512
| Batch size per GPU | 16 | 16 | 32 | 32
| # of GPUs (40G A100) | 16 | 16 | 16 | 16
| Gradient accumulation round | 1 | 32 | 1 | 1
| Learning rate of ($\psi$, $\theta$) | 1e-5 | 5e-6 | 1e-5 | 5e-6
| Loss scaling of ($\psi,\theta$) | (1,100)
| ema | 0.5 | 2 | 0.5 | 0.5
| fp16 | False | True | True | True
| Optimizer Adam (eps) | 1e-8 | 1e-6 | 1e-6 | 1e-6
| Optimizer Adam ($\beta_{1}$) of $\theta$ | 0 | 0 | 0.9 | 0
| Optimizer Adam ($\beta_{1}$) of $\psi$ | 0
| Optimizer Adam ($\beta_{2}$) | 0.999
| $\alpha$ |  $1.0$ and $1.2$
| $\sigma_{\text{init}}$ | 2.5
| $t_{\max}$ | 800
| augment, dropout, cres | The same as in EDM for each corresponding dataset
Diff-Instruct | max memory in GB allocated per GPU | 4.4 | 20.4 | 8.1 | 8.1
max memory in GB reserved per GPU | 4.7 | 23.0 | 10.8 | 10.8
$\sim$seconds per 1k images | 1.4 | 2.8 | 1.1 | 1.1
SiD | max memory allocated per GPU | 7.8 | 31.3 | 17.0 | 17.0
max memory in GB reserved per GPU | 8.1 | 31.9 | 17.2 | 17.2
$\sim$seconds per 1k images | 1.6 | 3.6 | 1.3 | 1.3
$\sim$hours per 10M ($10^{4}$k) images | 4.4 | 10.0 | 3.6 | 3.6
$\sim$days per 100M ($10^{5}$k) images | 1.9 | 4.2 | 1.5 | 1.5
|  $\sim$days per 500M ($5\times 10^{5}$k) images | 9.3 | 20.8 | 7.5 | 7.5

Figure 9: Analogous plot to Fig. 6 for ImageNet 64x64, where the batch size
for SiD is 1024 and learning rate is 5e-6. The FID declines fast until it
suddenly diverges. Increasing the batch size to 8192 and lowering the learning
rate to 4e-6, as shown in Fig. 6, has alleviated the issue of sudden
divergence.

Figure 10: Similar to Fig. 1, this plot showcases the SiD method’s efficacy
with $\alpha=1.0$, a batch size of 8192, and a learning rate of 4e-6. The
images are created using a consistent set of random noises after training the
SiD generator with differing numbers of synthesized images, specifically 0,
0.2, 1, 5, 10, 20, and 50 million images. These correspond to approximately 0,
20, 120, 600, 1.2K, 2.5K, and 6.1K training iterations, respectively,
displayed sequentially from left to right. The corresponding FIDs at these
stages are 153.73, 54.63, 46.07, 11.02, 6.93, 4.68, and 3.34. The progression
of FIDs is illustrated by the dashed blue curve in Fig. 6.

Figure 11: Analogous plot to Fig. 10 for SiD with an adjusted parameter
$\alpha=1.2$. The corresponding FIDs are 154.05, 57.63, 43.55, 16.89, 78.92,
7.45, and 3.22. The progression of FIDs is illustrated by the solid orange
curve in Fig. 6.

Figure 12: All but the last subplot consist of example SiD generated images
corresponding to the spikes of the solid orange curve in Fig. 6, which depicts
the evolution of FIDs of SiD with $\alpha=1.2$. These spikes are observed
after processing around 10, 55, 17, 23, 73, and 88 million images. The last
subplot displays SiD generated images using the generator with the lowest FID.

Figure 13:  Uncoditional CIFAR-10 random images generated with SiD (FID:
1.923).  Figure 14:  Label conditioning CIFAR-10 random images generated with
SiD (FID: 1.710)  Figure 15:  Label conditioning ImageNet 64x64 random images
generated with SiD (FID: 1.524)  Figure 16:  FFHQ random images generated with
SiD (FID: 1.550)  Figure 17:  AFHQ random images generated with SiD (FID:
1.628)