# Accelerating Diffusion Models via Early Stop of the Diffusion Process

Zhaoyang Lyu1,2 Xudong Xu1,2 Ceyuan Yang1,2 Dahua Lin1,2 Bo Dai2
1The Chinese University of Hong Kong
2Shanghai AI Laboratory
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive
performance on various generation tasks. By modeling the reverse process of
gradually diffusing the data distribution into a Gaussian distribution,
generating a sample in DDPMs can be regarded as iteratively denoising a
randomly sampled Gaussian noise. However, in practice DDPMs often need
hundreds or even thousands of denoising steps to obtain a high-quality sample
from the Gaussian noise, leading to extremely low inference efficiency. In
this work, we propose a principled acceleration strategy, referred to as
Early-Stopped DDPM (ES-DDPM), for DDPMs. The key idea is to stop the diffusion
process early where only the few initial diffusing steps are considered and
the reverse denoising process starts from a non-Gaussian distribution. By
further adopting a powerful pre-trained generative model, such as GAN and VAE,
in ES-DDPM, sampling from the target non-Gaussian distribution can be
efficiently achieved by diffusing samples obtained from the pre-trained
generative model. In this way, the number of required denoising steps is
significantly reduced. In the meantime, the sample quality of ES-DDPM also
improves substantially, outperforming both the vanilla DDPM and the adopted
pre-trained generative model. On extensive experiments across CIFAR-10,
CelebA, ImageNet, LSUN-Bedroom and LSUN-Cat, ES-DDPM obtains promising
acceleration effect and performance improvement over representative baseline
methods. Moreover, ES-DDPM also demonstrates several attractive properties,
including being orthogonal to existing acceleration methods, as well as
simultaneously enabling both global semantic and local pixel-level control in
image generation.

##  1 Introduction

Denoising Diffusion Probabilistic Models (DDPMs) Sohl-Dickstein et al. (2015)
are a class of generative models that have received growing attention in
recent years, due to their promising results in both unconditional and
conditional generation tasks, such as image generation (Ho et al., 2020;
Dhariwal & Nichol, 2021; Nichol et al., 2021), image manipulation and
restoration (Nichol et al., 2021; Meng et al., 2021; Saharia et al., 2021b,
a), audio generation (Chen et al., 2020; Kong et al., 2020), as well as 3D
shape generation and completion (Luo & Hu, 2021; Zhou et al., 2021; Lyu et
al., 2022; Zhou et al., 2021). DDPMs regard the generation procedure as the
reverse of a diffusion process, which gradually adds noises to data samples
and transforms the data distribution into a Gaussian distribution. Therefore,
synthesizing a sample from DDPMs is achieved by denoising a randomly sampled
Gaussian noise iteratively.

While DDPMs show great potential in various generation tasks, one major issue
of them is their intrinsic low inference efficiency, as obtaining a high
quality sample from the vanilla DDPM usually requires thousands of iterative
denoising steps, each of which involves a forward evaluation of the underlying
neural network. To accelerate DDPMs, previous attempts (Ho et al., 2020; Kong
& Ping, 2021; Nichol & Dhariwal, 2021) propose to integrate multiple standard
denoising steps into a single jumping denoising step. Although the number of
denoising steps is reduced, such integrated denoising steps break the Gaussian
assumption of DDPMs’ denoising process (Xiao et al., 2021; Sohl-Dickstein et
al., 2015), resulting in dropped sample quality. On the other hand,
acceleration of DDPMs is also indirectly achieved by (Preechakul et al., 2021;
Pandey et al., 2022), which inject image features or VAE-generated images into
the denoising process as additional conditions. Such conditioned denoising
processes are more effective than the original one, requiring fewer denoising
steps to obtain a high-quality sample. However, their acceleration effect is
often limited in practice since there are no modifications to the vanilla
conditional DDPM generation framework in these approaches. It thus remains a
challenging task to accelerate DDPMs in a principled way without compromising
the quality of synthesized samples, especially for unconditional generation or
class-conditioned generation, where no strong conditioning signal is
available.

In this work, we propose a novel acceleration strategy for DDPMs that not only
boosts the efficiency of DDPMs without breaking any of the assumptions, but
also, as an additional benefit, improves the generation quality of DDPMs. The
core idea of our strategy, dubbed as Early-Stopped DDPM (ES-DDPM), is stopping
the diffusion process early as shown in Figure 1. Instead of diffusing the
data distribution into a Gaussian distribution via hundreds to thousands of
iterative steps, ES-DDPM considers only the initial few diffusion steps, so
that the reverse denoising process starts from a non-Gaussian distribution
$q$, leading to a significantly reduced number of denoising steps. One
remaining issue is ES-DDPM requires sampling from the non-Gaussian
distribution $q$ to generate data samples, yet $q$ has no closed form
formulations. To overcome this issue, we observe that $q$ is diffused from the
real data distribution, in the meantime many of existing generative models
provide a good approximation of the real data distribution. Consequently, we
further equip ES-DDPM with a pre-trained generative model (e.g. GAN (Karras et
al., 2019) and VAE (Kingma & Welling, 2013)) so that sampling from $q$ becomes
diffusing samples obtained from the pre-trained generative model. Since both
the sampling and the diffusion operation can be easily achieved in a single
step, the pre-trained generative model brings only minor computational
overhead to ES-DDPM.

The proposed ES-DDPM bears several important advantages. 1) The number of
denoising steps in ES-DDPM is significantly reduced (from 1000 steps to 100
steps in the best case of our experiments). Moreover, such acceleration effect
is achieved without sacrificing the quality of generated samples. In fact, by
combining a pre-trained generative model, ES-DDPM outperforms both the vanilla
DDPM as well as the pre-trained generative model in terms of sample quality.
2) Unlike the acceleration strategies based on jumping denoising steps that
only accelerates the generation process during inference, ES-DDPM provides a
principled way to accelerate DDPMs in both training and inference. 3) While
major properties of the vanilla DDPM is preserved in ES-DDPM, it is orthogonal
to most existing acceleration strategies of DDPMs. As shown in our
experiments, ES-DDPM coupled with other acceleration strategies results in
further improved acceleration effect. 4) ES-DDPM not only maintains the
properties of the vanilla DDPM, it also enjoys the properties of the involved
pre-trained generative model. For instance, when combined with a StyleGAN2 for
unconditional image generation, ES-DDPM enables both global semantic control
and local pixel-level control over generated images, as shown in our
experiments.

Figure 1: Compare Early-Stopped DDPM (ES-DDPM) with the full-length DDPM.

##  2 Background on DDPMs

In Denoising Diffusion Probabilistic Models (DDPMs), the diffusion process is
defined as

$q({\bm{x}}^{1:T}|{\bm{x}}^{0})=\prod_{t=1}^{T}q({\bm{x}}^{t}|{\bm{x}}^{t-1}),\text{
where
}q({\bm{x}}^{t}|{\bm{x}}^{t-1})={\mathcal{N}}({\bm{x}}^{t};\sqrt{1-\beta_{t}}{\bm{x}}^{t-1},\beta_{t}{\bm{I}}),$
(1)

where $\beta_{t}$’s are some predefined small constants, and ${\bm{x}}^{1:T}$
denotes the set of variables
$\\{{\bm{x}}^{1},{\bm{x}}^{2},\cdots,{\bm{x}}^{T}\\}$. Similar notations are
used to denote a set of variables in rest part of the paper. In the diffusion
process, we gradually add noise to the clean image ${\bm{x}}^{0}$. $T$ is set
to be a sufficiently large number so that ${\bm{x}}^{T}$ is close to a
Gaussian noise. $T=1000\sim 4000$ is a typical choice for most works. The
reverse process is defined as

$\displaystyle
p_{\bm{\theta}}({\bm{x}}^{0:(T-1)}|{\bm{x}}^{T})=\prod_{t=1}^{T}p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t}),\text{
where
}p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t})={\mathcal{N}}({\bm{x}}^{t-1};\bm{\mu}_{\bm{\theta}}({\bm{x}}^{t},t),\sigma_{t}^{2}{\bm{I}}).$
(2)

The variance $\sigma_{t}^{2}$’s can be set as time-step dependent constants or
learned by a neural network. The mean is parameterized by a neural network
$\bm{\mu}_{\bm{\theta}}({\bm{x}}^{t},t)$. The neural network is trained to
simulate the reverse process of the diffusion process defined in Equation 1.
To generate an image from the reverse process, we first sample ${\bm{x}}^{T}$
from the Gaussian distribution, and then sample ${\bm{x}}^{t-1}$ from
$p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t})$ for $t=T,T-1,\cdots,1$.
${\bm{x}}^{0}$ is the image that the DDPM generates.

The generation process of a DDPM is extremely slow as it need to sample from
the transition distribution $p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t})$
iteratively, which involves hundreds or even thousands of evaluations of the
output of the neural network $\bm{\mu}_{\bm{\theta}}({\bm{x}}^{t},t)$.

##  3 Methodology

It is necessary to accelerate the sampling process of the DDPM for any
practical usage. Our key insight is that we do not need to diffuse a clean
image thousands of steps to turn it into a Gaussian noise as shown in Equation
1. We can adopt early stop in the diffusion process as shown in Figure 1. We
cut the diffusion process in the middle at $t=T^{\prime}<T$, and use an
encoder to encode the noisy image $x^{T^{\prime}}$ into a low dimension vector
${\bm{z}}$ in one step. Correspondingly, in the sampling process, we first
sample a latent code ${\bm{z}}$ from the Gaussian distribution, then use a
decoder to sample ${\bm{x}}^{T^{\prime}}$ from ${\bm{z}}$. Finally, we use the
DDPM to denoise ${\bm{x}}^{T^{\prime}}$ to a clean image ${\bm{x}}^{0}$ in
fewer steps than the original sampling process. We name the cut DDPM early-
stopped DDPM (ES-DDPM).

Our encoding process, which encodes the clean image ${\bm{x}}^{0}$ to a low
dimension latent code ${\bm{z}}$, is defined as

$\displaystyle
q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})=q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})q_{\psi}({\bm{z}}|{\bm{x}}^{1:T^{\prime}},{\bm{x}}^{0})=q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})q_{\psi}({\bm{z}}|{\bm{x}}^{0}),$
(3)

where $q_{\psi}({\bm{z}}|{\bm{x}}^{0})$ is the encoder. We set it to a
Gaussian distribution whose mean and standard deviation are parameterized by a
neural network. $q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})$ is a partial
diffusion process of Equation 1, and it is defined as

$\displaystyle
q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})=\prod_{t=1}^{T^{\prime}}q({\bm{x}}^{t}|{\bm{x}}^{t-1}),\text{
where
}q({\bm{x}}^{t}|{\bm{x}}^{t-1})={\mathcal{N}}({\bm{x}}^{t};\sqrt{1-\beta_{t}}{\bm{x}}^{t-1},\beta_{t}{\bm{I}}).$
(4)

In this partial diffusion process, we do not need $T^{\prime}$ to be large
enough to guarantee that $x^{T^{\prime}}$ is close to a Gaussian noise.

We define the sampling process as

$\displaystyle
p({\bm{x}}^{0:T^{\prime}},{\bm{z}})=p({\bm{z}})p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}}),$
(5)

where $p({\bm{z}})$ is assumed to follow standard Gaussian distribution,
$p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})$ is the decoder parameterized by a
neural network, and
$p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}})$ is the reverse
process of the partial diffusion process in Equation 4. It is defined as

$\displaystyle
p_{{\theta}}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}})=\prod_{t=1}^{T^{\prime}}p_{{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t}),\text{
where
}p_{{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t})={\mathcal{N}}({\bm{x}}^{t-1};\bm{\mu}_{{\theta}}({\bm{x}}^{t},t),\sigma_{t}^{2}{\bm{I}}).$
(6)

Through variational inference, we can prove that

$\displaystyle\log p({\bm{x}}^{0})\geq-(L_{\text{VAE}}+L_{\text{DDPM}}),$ (7)
$\displaystyle\text{where }L_{\text{VAE}}$
$\displaystyle=D_{\text{KL}}(q_{\psi}({\bm{z}}|{\bm{x}}^{0})||p({\bm{z}}))+\mathbb{E}_{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})),$
(8) $\displaystyle L_{\text{DDPM}}$
$\displaystyle=\sum_{t=2}^{T^{\prime}}\mathbb{E}_{q({\bm{x}}^{t}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})||p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t}))-\mathbb{E}_{q({\bm{x}}^{1}|{\bm{x}}^{0})}\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1}).$ (9)

$D_{\text{KL}}$ denotes Kullback–Leibler divergence. See complete proof in
Appendix Section A.1. To maximize the data log-likelihood $\log
p({\bm{x}}^{0})$, we need to minimize the two loss terms: $L_{\text{DDPM}}$
and $L_{\text{VAE}}$. We will demonstrate that this means we can train an ES-
DDPM and a variational autoencoder (VAE) separately, and then combine them
together to obtain a new generative model. First, The loss term
$L_{\text{DDPM}}$ is the same of a normal DDPM, except that we do not need to
train the full Markov chain range from $1$ to $T$. Instead, we only need to
train the first several steps range from $1$ to $T^{\prime}$. A normal DDPM
trained on the loss of full $T$ terms would also fulfil the requirements here,
but as we will show in later experiments, the training process of an ES-DDPM
is faster than a normal full-length DDPM.

Next, we show that the loss term $L_{\text{VAE}}$ amounts to train a VAE on
the clean images ${\bm{x}}^{0}$ in the dataset. The first term in
$L_{\text{VAE}}$ is the same as in a standard VAE, which requires the encoder
to encode clean images to a latent space that follows the Gaussian
distribution. The second term in $L_{\text{VAE}}$ means that we want
$p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})$ to match
$q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})$ as close as possible. It is proven by
previous works (Sohl-Dickstein et al., 2015; Ho et al., 2020) that

$\displaystyle q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})\sim
N({\bm{x}}^{T^{\prime}};\sqrt{\bar{\alpha}_{t}}{\bm{x}}^{0},(1-\bar{\alpha}_{t}){\bm{I}}),\text{
where }\alpha_{t}=1-\beta_{t},\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i}.$
(10)

Correspondingly, we choose to set $p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})$
to

$\displaystyle p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})\sim
N({\bm{x}}^{T^{\prime}};\sqrt{\bar{\alpha}_{t}}{\bm{f}}_{\phi}({\bm{z}}),(1-\bar{\alpha}_{t}){\bm{I}}).$
(11)

Then

$\displaystyle
D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}}))=C_{1}||{\bm{x}}^{0}-{\bm{f}}_{\phi}({\bm{z}})||^{2}+C_{2},$
(12)

where $C_{1},C_{2}$ are some scalar constants. Now the second term in
$L_{\text{VAE}}$ means that the decoder ${\bm{f}}_{\phi}({\bm{z}})$ need to
reconstruct the original clean image ${\bm{x}}^{0}$, based on the latent code
${\bm{z}}$ sampled from the encoder $q_{\psi}({\bm{z}}|{\bm{x}}^{0})$. This is
exactly the same procedure as training a VAE on clean images ${\bm{x}}^{0}$ in
the dataset.

#### Sampling Procedure.

After training the ES-DDPM and the VAE separately, we can combine them to form
a new generative model. We follow Equation 5 to generate samples and it is
illustrated in Figure 1: First sample ${\bm{z}}$ from the standard Gaussian
distribution, then use the decoder of the VAE to generate an image
${\bm{f}}_{\phi}({\bm{z}})$. Next, sample ${\bm{x}}^{T^{\prime}}$ from the
distribution described in Equation 11. Finally, use the ES-DDPM to sample
${\bm{x}}^{t-1}$ from $p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t})$ for
$t=T^{\prime},T^{\prime}-1,\cdots,1$, and output the generated image
${\bm{x}}^{0}$.

Note that in the above sampling process, we only need to use the decoder
${\bm{f}}_{\phi}({\bm{z}})$, while the encoder
$q_{\psi}({\bm{z}}|{\bm{x}}^{0})$ is not used. This means that we can not only
combine the ES-DDPM with a VAE, but also with any generative model that can
map the latent code ${\bm{z}}$ to a clean image, such as a GAN. Since after a
GAN is trained, its generator is pretty much like the decoder of a VAE, except
that the generator’s encoder is not known, but we simply do not need the
encoder to generate samples from the combined model. Therefore, we can also
think of ${\bm{f}}_{\phi}({\bm{z}})$ as the generator of a trained GAN. In
fact, we will demonstrate that combining an ES-DDPM with a GAN is more
effective than combining it with a VAE in later experiments. For conditional
generation tasks, we can also combine a conditional ES-DDPM with a conditional
VAE or GAN to achieve acceleration. See details in Appendix A.2.

##  4 Related Works

Several works (Ho et al., 2020; Kong & Ping, 2021; Nichol & Dhariwal, 2021)
propose to accelerate the sampling procedure of DDPMs by making jumping steps
in the reverse steps. Watson et al. (2021) propose a dynamic programming
algorithm that finds the optimal denoising time-step schedule for DDPMs.
Watson et al. (2021) optimize DDPM fast samplers by differentiating through
sample quality scores. Salimans & Ho (2022) propose to progressively distill a
trained DDPM for fast sampling. Xiao et al. (2021) propose denoising diffusion
GANs that can balance sample quality, mode coverage, and fast sampling. We
think our method is orthogonal to these acceleration methods. In principle,
these methods can be coupled with our method to accelerate the $T^{\prime}$
denoising steps in our ES-DDPM.

Some works find that combining DDPMs with VAEs can help accelerate the
sampling process of DDPMs. Preechakul et al. (2021) condition the reverse
process of the DDPM on an encoded vector of an image. Pandey et al. (2022)
condition the reverse process of the DDPM on an reconstructed image by a VAE.
Kingma et al. (2021) design variational diffusion models that can generate
images in fewer steps than a DDPM. However, all of them still need a complete
diffusion process to destroy the input image, and in their sampling process,
they still need to run the full denoising process that starts with a Gaussian
noise or a very noisy image. In comparison, we adopt early stop in the
diffusion process, and we start from a partially diffused image generated by a
VAE or GAN in our denoising process. Another major difference is that these
methods need to alter the training process and train a DDPM that is specific
to the VAE model they want to combine with, while our method can combine a
pre-trained DDPM with any other generative models.

We find that a concurrent work (Zheng et al., 2022) adopts a similar
acceleration method to our ES-DDPM, where ES-DDPM represents the non-Gaussian
distribution of ${\bm{x}}^{T^{\prime}}$ by diffusing clean samples obtained
from a pre-trained generative model, while the concurrent work proposes to
directly learn the distribution of ${\bm{x}}^{T^{\prime}}$ with a GAN or a
conditional transport. Consequently, ES-DDPM bears the following advantages:
First, by diffusing the real data distribution approximated by a pre-trained
generative model, ES-DDPM avoids learning the distribution of
${\bm{x}}^{T^{\prime}}$ directly, which brings extra computational complexity,
and learning the distribution of noisy images ${\bm{x}}^{T^{\prime}}$ may be
harder than learning the real data distribution. Second, ES-DDPM can use the
same pre-trained generative model for any value of $T^{\prime}$, while the
concurrent work requires to learn a new model for each possible value of
$T^{\prime}$. Third, without re-training, ES-DDPM can easily utilize the
widely-available pre-trained generative models in the community, and
seamlessly integrate advances in the related fields. In this way, attractive
properties, such as controllable generation, possessed by existing generative
models can be effectively introduced to DDPMs.

##  5 Experiments

###  5.1 Image Generation

In this section, we evaluate the image generation performance of our ES-DDPM
and compare it with other generative models.

#### Datasets.

We conduct experiments on CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et
al., 2015), ImageNet (Russakovsky et al., 2015), LSUN-Bedroom and LSUN-Cat (Yu
et al., 2015). For the CelebA dataset, we conduct experiments at two
resolutions: $64\times 64$ and $128\times 128$. For the ImageNet dataset, we
use a resolution of $64\times 64$, and we use $256\times 256$ for the LSUN
datasets.

#### Evaluation Metrics.

We use the commonly used Fréchet inception distance (Heusel et al., 2017)
(FID) and Inception Score (Salimans et al., 2016) (IS) to evaluate the quality
of generated images. We also use Improved Precision and Recall metrics
(Kynkäänniemi et al., 2019) to explicitly measure sample fidelity and
diversity. sFID (Nash et al., 2021) is also used.

#### CIFAR-10 Experiment.

We combine ES-DDPM with other generative models, i.e., vannila VAE (Kingma &
Welling, 2013), DCGAN (Radford et al., 2015), StyleGAN2 (Karras et al., 2020),
and test their performance in terms of unconditional image generation on
CIFAR-10 (Krizhevsky et al., 2009) dataset. We use the pre-trained 1000-step
DDPM on CIFAR-10 from the work (Ho et al., 2020). We insert the images
generated by the other generative model (VAE or GAN) to different steps in the
reverse process of the DDPM, namely, use different $T^{\prime}$ in Equation 5.
We report FID of the combined generative models in Table 1.

We can see that DDPM can improve the quality of images generated by the other
generative models. And roughly speaking, the larger the number of denoising
steps $T^{\prime}$ is, the better the quality is. This is because
$p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})$ in Equation 11 gradually approaches
to a standard Gaussian noise as $T^{\prime}$ increases, and hence the combined
model gradually matches the original DDPM as $T^{\prime}$ increases. Another
observation is that we need to combine ES-DDPM with a strong generative model
like StyleGAN2 if we want to obtain high quality images in few denoising
steps. One rather surprising observation is that the combined model can even
outperform the original DDPM. In other words, we can combine a DDPM with a
less powerful generative model, and with less denoising steps, to obtain a
even better combined model, which means we can accelerate the DDPM for free!
As we will show in later experiments, the same results can be achieved on
other datasets as well.

#### CelebA Experiment.

We combine ES-DDPM with StyleGAN2 and test its performance on the CelebA (Liu
et al., 2015) dataset at two resolutions: CelebA-64 and CelebA-128. At each
resolution, we train two DDPMs with $T^{\prime}=100$ and $T^{\prime}=200$
using the loss function in Equation 9, and combine them with a self-trained
StyleGAN2, respectively. We compare our combined models with self-trained
$1000$-step DDPMs and other generative models in Table 2. The DDPMs are
trained with a linear $\beta$ schedule following the same procedures in the
works (Ho et al., 2020; Song et al., 2020). We can see that when combined with
StyleGAN2, our ES-DDPM can achieve lower FIDs than the original DDPM and other
score-based methods in even less denoising steps.

Table 1: Unconditional image generation performance on CIFAR-10. We report FID
between randomly generated $50000$ images by the combined generative model and
the training set. $T^{\prime}=0$ refers to the original VAE or GAN model.
$T^{\prime}=1000$ is the original 1000-step DDPM.

Denoising Steps $T^{\prime}$ | 0 | 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 | 900 | 1000
---|---|---|---|---|---|---|---|---|---|---|---
VAE+ES-DDPM | 158.61 | 49.62 | 20.88 | 11.03 | 5.96 | 3.69 | 3.17 | 3.15 | 3.12 | 3.18 | 3.20
DCGAN+ES-DDPM | 33.31 | 15.54 | 9.97 | 7.21 | 5.49 | 3.98 | 3.37 | 3.13 | 3.14 | 3.24
StyleGAN2+ES-DDPM | 7.18 | 5.52 | 5.02 | 4.60 | 4.03 | 3.51 | 3.26 | 3.11 | 3.16 | 3.17
|  |  |  |  |  |  |  |  |  |  |

Table 2: Performance comparison of CelebA unconditional image generation. “*”
means the model is trained by ourselves. For our method and self trained
models, we report FID between randomly generated $50000$ images and the whole
dataset.

CelebA-64 | Method | FID
---|---|---
Hybrid Models | StyleGAN2+$100$-step ES-DDPM (Ours) | 3.01
StyleGAN2+$200$-step ES-DDPM (Ours) | 2.55
$1000$-step DiffuseVAE (Pandey et al., 2022) | 4.76
Score-based Methods | 1000-step DDPM* (Ho et al., 2020) | 3.26
$250$-step PNDM (Liu et al., 2022) | 2.71
NCSN (Song & Ermon, 2019) | 25.30
NCSNv2 (Song & Ermon, 2020) | 10.23
GAN-based Methods | COCO-GAN (Lin et al., 2019) | 4.00
StyleGAN2* (Karras et al., 2020) | 4.55
QA-GAN (Parimala & Channappayya, 2019) | 6.42
VAE-based Methods | NCP-VAE (Aneja et al., 2020) | 5.25
|  |
CelebA-128 | Method | FID
Hybrid Models | StyleGAN2+$100$-step ES-DDPM (Ours) | 1.76
StyleGAN2+$200$-step ES-DDPM (Ours) | 1.79
Score-based Methods | 1000-step DDPM* (Ho et al., 2020) | 5.65
GAN-based Methods | COCO-GAN (Lin et al., 2019) | 5.74
StyleGAN2* (Karras et al., 2020) | 2.13
PresGAN (Dieng et al., 2019) | 29.12
|  |

Compared with CIFAR-10 experiment results in Table 1, we find that ES-DDPM
combined with StyleGAN2 can outperform the full-length DDPM in much less
denoising steps on the CelebA dataset. In Table 2, $T^{\prime}=100$ or
$T^{\prime}=200$ can already outperform the $1000$-step DDPM, while in Table
1, we need $700$ to $800$ denoising steps to achieve a lower FID than the
full-length DDPM. We think it is because CIFAR-10 is a multi-category dataset
while CelebA is a single-category dataset. In other words, CIFAR-10 has more
data diversity than CelebA. GANs are known to be able to generate high quality
samples, but have poor mode coverage (Zhao et al., 2018; Xiao et al., 2021).
On the other hand, DDPMs demonstrate both high sample quality and good mode
coverage (Dhariwal & Nichol, 2021). Therefore, for an ES-DDPM to improve the
generation diversity of a GAN on a dataset of complex distribution, we need
large $T^{\prime}$s in Equation 6. Because for small $T^{\prime}$s, the ES-
DDPM generated image ${\bm{x}}^{0}$ will be very similar to the GAN generated
image ${\bm{f}}_{\phi}({\bm{z}})$, as there is little diversity and
stochasticity in Equation 6 for small $T^{\prime}$s. On the other hand, for
large $T^{\prime}$s, the generated image ${\bm{x}}^{0}$ can be vastly
different from ${\bm{f}}_{\phi}({\bm{z}})$, because ${\bm{x}}^{T^{\prime}}$
contains very little information of ${\bm{f}}_{\phi}({\bm{z}})$, and thus the
GAN’s generation diversity can be improved. We will explicitly verify this
point of view using the Precision and Recall metrics in the following ImageNet
and LSUN Experiments.

#### ImageNet and LSUN Experiment.

For ImageNet-64, we combine ES-DDPM with BigGAN-deep (Brock et al., 2018) and
test its class-conditional image generation performance. For LSUN-Bedroom and
LSUN-Cat, we combine ES-DDPM with StyleGAN (Karras et al., 2019) and
StyleGAN2, respectively. For these experiments, we use pre-trained DDPM models
and GAN generated images provided in the work (Dhariwal & Nichol, 2021). The
DDPMs are of length $1000$ and the variance schedule is learned by a neural
network. We follow the same evaluation methods in the work (Dhariwal & Nichol,
2021): FIDs are computed between randomly generated $50000$ images and the
whole training set. IS’s are computed over the randomly generated $50000$
images. Precision and Recall are computed between the randomly generated
$50000$ images and random $10000$ images from the training set.

Experiment results are shown in Table 3, Table 5 and Table 5. Same as CIFAR-10
and CelebA, we can see that ES-DDPM can improve the generation quality of
GANs, and the combined model can outperform the original DDPM in terms of FID,
sFID and IS. We also observe that BigGAN-deep has low Recall on the complex
multi-category ImageNet dataset, its Recall is gradually improved as
$T^{\prime}$ increases. Therefore, the combined model need large $T^{\prime}$s
around $900$ to achieve a lower FID than the full-length DDPM. On the other
hand, for the single-category datasets, LSUN-Bedroom and LSUN-Cat, StyleGAN
and StyleGAN2 already have a not too bad Recall compared with the full-length
DDPMs. Therefore, the ES-DDPMs can improve the GANs’ Recall in less denoising
steps, and our combined model can outperform the original full-length DDPM in
terms of FID in just $100$ steps. In conclusion, for multi-category datasets
of complex data distribution, our combined models provide a trade-off between
sample diversity and denoising steps. For single-category datasets where GANs
already have a good Recall, our combined models provide better sample quality
than both the GANs and the original full-length DDPMs, with much fewer
denoising steps than the original full-length DDPMs.

Table 3: ImageNet-64 class-conditional image generation performance of ES-
DDPM+BigGAN-deep. For the DDPM and ES-DDPM experiments, we follow the work
(Dhariwal & Nichol, 2021) to use Timestep-Respacing with jumping interval $4$.
The actual denoising steps is $T^{\prime}/4$. We do not use classifier
guidance.

ES-DDPM Length $T^{\prime}$ | 0(BigGAN-deep) | 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 | 900 | 1000(DDPM)
---|---|---|---|---|---|---|---|---|---|---|---
FID | 4.06 | 3.75 | 3.47 | 3.30 | 3.16 | 2.92 | 2.71 | 2.50 | 2.31 | 2.07 | 2.13
sFID | 3.96 | 3.91 | 3.96 | 3.93 | 3.90 | 3.83 | 3.84 | 3.84 | 3.84 | 3.90 | 4.28
IS | 45.00 | 48.63 | 50.17 | 51.45 | 52.32 | 53.85 | 54.41 | 55.11 | 55.42 | 55.29 | 52.52
Precision | 0.795 | 0.798 | 0.800 | 0.794 | 0.792 | 0.787 | 0.784 | 0.778 | 0.771 | 0.756 | 0.739
Recall | 0.483 | 0.504 | 0.514 | 0.529 | 0.533 | 0.547 | 0.565 | 0.577 | 0.593 | 0.608 | 0.631
|  |  |  |  |  |  |  |  |  |  |

Table 4: ES-DDPM+StyleGAN on LSUN-Bedroom.

Denoising | 0 | 100 | 200 | 300 | 1000
---|---|---|---|---|---
Steps $T^{\prime}$ | (StyleGAN) | (DDPM)
FID | 2.35 | 1.85 | 1.70 | 1.68 | 1.86
sFID | 6.61 | 5.93 | 5.70 | 5.75 | 5.64
IS | 2.55 | 2.53 | 2.54 | 2.54 | 2.39
Precision | 0.590 | 0.636 | 0.642 | 0.643 | 0.653
Recall | 0.483 | 0.464 | 0.472 | 0.495 | 0.500
|  |  |  |  |

Table 5: ES-DDPM+StyleGAN2 on LSUN-Cat.

Denoising | 0 | 100 | 200 | 300 | 1000
---|---|---|---|---|---
Steps $T^{\prime}$ | (StyleGAN2) | (DDPM)
FID | 7.26 | 5.47 | 5.15 | 4.89 | 5.68
sFID | 6.33 | 6.47 | 6.57 | 6.70 | 6.72
IS | 4.84 | 5.12 | 5.23 | 5.33 | 5.23
Precision | 0.576 | 0.626 | 0.633 | 0.631 | 0.628
Recall | 0.432 | 0.463 | 0.475 | 0.489 | 0.520
|  |  |  |  |

###  5.2 Coupled With Other DDPM Acceleration Methods

Our method can be easily coupled with other DDPM acceleration methods to
achieve further acceleration. Recall that our method need to use the ES-DDPM
to denoise $x^{T^{\prime}}$ for $T^{\prime}$ steps as shown in Equation 6.
This denoising process is actually the same as the last $T^{\prime}$ steps of
the reverse process in a normal full-length DDPM. Any method that aims at
accelerating the full $T$-step reverse process of a DDPM, can in principle be
applied to accelerate the last $T^{\prime}$ denoising steps. We use DDIM (Song
et al., 2020) and Timestep-Respacing (TR) (Nichol & Dhariwal, 2021) as a
demonstration in our experiments. Both DDIM and Timestep-Respacing enable one
to sampling from a pre-trained DDPM only on a subsequence of the original
denoising steps. We choose a uniform subsequence in our $T^{\prime}$ denoising
steps. We conduct experiments on the CelebA, ImageNet, LSUN-bedroom and LSUN-
Cat datasets. CelebA and ImageNet results are shown in Table 6, Table 7. LSUN-
bedroom and LSUN-Cat results are shown in Table 8 and Table 9 in Appendix A.4.
We can see that our combined model can indeed achieve further acceleration
when coupled with other acceleration methods. And compared with other
acceleration methods, our combined model bears better sample quality when
using the same number of denoising steps in most cases.

Table 6: FID comparison of different DDPM acceleration methods on CelebA. For
our models, denoising steps less than $100$ steps are achieved by accelerating
ES-DDPM($T^{\prime}=100$) using DDIM.

| Denoising Steps | 200 | 100 | 50 | 20 | 10 | 5
---|---|---|---|---|---|---|---
CelebA-64 | ES-DDPM+StyleGAN2+DDIM(Ours) | 2.55 | 3.01 | 3.97 | 4.90 | 6.44 | 9.15
DDIM (Song et al., 2020) | - | 6.53 | 9.17 | 13.73 | 17.33 | -
FastDPM (Kong & Ping, 2021) | - | 7.85 | 8.31 | 10.69 | 15.31 | -
PNDM (Liu et al., 2022) | 2.71 | 2.81 | 3.34 | 5.51 | 7.71 | 11.3
Diffusion Autoencoder (Preechakul et al., 2021) | - | 5.30 | 7.05 | 10.18 | 12.92 | -
TDPM-GAN (Zheng et al., 2022) | - | 3.64 | 3.28 | - | - | -
CelebA-128 | ES-DDPM+StyleGAN2+DDIM(Ours) | 1.79 | 1.76 | 1.86 | 2.27 | 3.26 | 6.15
|  |  |  |  |  |  |

Table 7: FID and IS comparison of different DDPM acceleration methods on the
Imagenet-64 dataset. “*” means the images for evaluation are generated by
ourselves for this method.

| FID | IS
---|---|---
Denoising Steps | 25 | 20 | 15 | 10 | 5 | 25 | 20 | 15 | 10 | 5
ES-DDPM($T^{\prime}=100$)+BigGAN-deep+TR(Ours) | 3.75 | 3.77 | 3.80 | 3.93 | 4.25 | 48.63 | 48.92 | 49.17 | 48.81 | 48.04
DDIM* (Song et al., 2020) | 5.90 | 6.68 | 8.59 | 13.48 | 67.70 | 38.24 | 36.72 | 34.85 | 30.79 | 13.17
TR* (Nichol & Dhariwal, 2021) | 7.99 | 11.08 | 16.49 | 27.74 | 78.59 | 49.62 | 45.83 | 39.32 | 29.32 | 12.65
DDSS (Watson et al., 2021) | 18.40 | 20.69 | 24.69 | 37.32 | 55.14 | 18.12 | 17.92 | 17.23 | 14.76 | 12.90
|  |  |  |  |  |  |  |  |  |

###  5.3 Accelerate Training

We have shown that our combined model can accelerate the sampling process of
DDPMs in the previous sections. We also expect our method to be able to
accelerate the training process of DDPMs, since we only need to train a
denoising chain of length $T^{\prime}$, which is shorter than the original
denoising chain of length $T$. This is because our DDPM loss in Equation 9
contains only $T^{\prime}$ terms, compared with $T$ terms in the original
full-length DDPM loss. We compare the training process of our combined models
with the full-length $1000$-step DDPM on the CelebA dataset. We plot FIDs
during the training process of our combined model and the original DDPM in
Figure 2. Note that we have already taken into account the training time of
the StyleGAN2 by converting the training time of the StyleGAN2 to the
equivalent training steps of the DDPM, and it is annotated by a red dashed
line in Figure 2. For CelebA-64, we can see that our combined model still
converges faster than the full-length DDPM even if we take into account the
training time of the StyleGAN2. Specifically, it takes the full-length DDPM
$9\times 10^{5}$ iterations, about $80$ hours to achieve an FID of $3.26$. In
comparison, our ES-DDPM($T^{\prime}=100$) only uses $4.5\times 10^{4}$
iterations, about $4$ hours to achieve an FID of $3.09$. Even if we take into
account the training time of StyleGAN2, which is about $25$ hours, the total
training time of our combined model ($29$ hours) is still much shorter than
the training time of the full-length DDPM ($80$ hours). As for CelebA-128, we
find the full-length DDPM could not achieve an FID comparable to our combined
model even we train it for a very long time.

(a) CelebA-64

(b) CelebA-128

Figure 2: Training time comparison between our combined models and the
original full-length DDPMs. DDPMs are trained with batchsize $128$ on
CelebA-64 and with batchsize $48$ on CelebA-128. The red dashed line denotes
the training time of StyleGAN2.

###  5.4 Controllable Image Generation

Figure 3: On LSUN-Bedroom-256, we use StyleGAN to control the lighting
condition of a bedroom, and then use the ES-DDPM($T^{\prime}=200$) to control
local details.

In previous sections, we have shown that combining GANs and DDPMs can improve
their image generation quality, and accelerate the sampling and training
process of DDPMs. In this section, we further show that the combined models
can benefit from both GANs and DDPMs in terms of controllable generation.
DDPMs have achieved impressive results on image local editing (Nichol et al.,
2021; Meng et al., 2021). We think this can be largely attributed to DDPMs’
latent space ${\bm{x}}^{t}$, which has the same dimension as the input image
and thus allows us to perform pixel-level manipulation on this latent space.
However, this latent space lack semantic meanings to control high-level
semantics of the generated image. On the other hand, GANs and VAEs can learn a
meaningful representation of an image that can be used to control high-level
semantics, but it’s challenging for them to achieve pixel-level control of the
generated image. We think that DDPMs and GANs (or VAEs) are complementary to
each other in terms of controllable generation. We can achieve both high-level
semantic control and pixel-level control of the generated image in our
combined models.

It is achieved by following Equation 5 to generate images: First use a GAN or
VAE to generate an image ${\bm{f}}_{\phi}({\bm{z}})$. In this step, the GAN or
VAE can control the high-level semantics of the image. Next, we can perform
some coarse local editing on the image. Then, sample ${\bm{x}}^{T^{\prime}}$
from the distribution
$N({\bm{x}}^{T^{\prime}};\sqrt{\bar{\alpha}_{t}}{\bm{f}}_{\phi}({\bm{z}}),(1-\bar{\alpha}_{t}){\bm{I}})$.
Finally, use the ES-DDPM to denoise ${\bm{x}}^{T^{\prime}}$ by sampling
${\bm{x}}^{t-1}$ from $p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t})$ for
$t=T^{\prime},T^{\prime}-1,\cdots,1$, and output the generated image
${\bm{x}}^{0}$. The denoising process will preserve high-level semantics in
${\bm{f}}_{\phi}({\bm{z}})$ as long as $T^{\prime}$ is not too large, and it
will refine the local editing we performed and mitigate artifacts in the
image.

We conduct experiments to verify the effect of our method on controllable
generation. Figure 3 shows that we can use StyleGAN to control the lighting
condition of a bedroom, and then perform local editing on the generated images
(similar to the stroke-based editing in the work (Meng et al., 2021)).
Finally, we use the ES-DDPM to refine the edited images, mitigate artifacts
and make the images photo-realistic. In this way, our combined model can
control both the high-level lighting condition and local details of the
bedroom. Additional examples are provided in Appendix Section A.5.

##  6 Conclusion

In this work, we propose to adopt early stop in the diffusion process of a
DDPM to achieve acceleration. We can combine an early-stopped DDPM wth another
generative model such as VAE or GAN to obtain a new generative model. The
combined model can outperform the original DDPM in terms of sample quality,
sampling speed and training speed. We can further accelerate the combined
model by coupling it with other DDPM acceleration methods. The combined model
also has more control over the generation process. We demonstrate that we can
control both high-level semantics and local details of the generated image by
combining DDPMs with GANs.

Our method has achieved great acceleration effect and sample quality on single
category datasets. However, for multi-category datasets, it still need many
denoising steps to outperform the full-length DDPM in terms of sample quality
due to the poor mode coverage of GANs. In the future, we could combine ES-DDPM
with other generative models that have better mode coverage to achieve high
sample quality with less denoising steps on even complex multi-category
datasets. In addition, as a generative model, more studies are needed on how
to prevent the model from learning biases in the training dataset and from
misuse such as deception.

## References

  * Aneja et al. (2020) Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat.  Ncp-vae: Variational autoencoders with noise contrastive priors.  2020\.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Chen et al. (2020) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ , 2020.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Dieng et al. (2019) Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis K Titsias.  Prescribed generative adversarial networks.  _arXiv preprint arXiv:1910.04302_ , 2019.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4401–4410, 2019.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8110–8119, 2020.
  * Kingma & Welling (2013) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kingma et al. (2021) Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _arXiv preprint arXiv:2107.00630_ , 2021.
  * Kong & Ping (2021) Zhifeng Kong and Wei Ping.  On fast sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2106.00132_ , 2021.
  * Kong et al. (2020) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kynkäänniemi et al. (2019) Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lin et al. (2019) Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chen.  Coco-gan: Generation by parts via conditional coordinating.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 4512–4521, 2019.
  * Liu et al. (2022) Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=PlKWVd2yBkY.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Luo & Hu (2021) Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2837–2845, 2021.
  * Lyu et al. (2022) Zhaoyang Lyu, Zhifeng Kong, Xudong XU, Liang Pan, and Dahua Lin.  A conditional point diffusion-refinement paradigm for 3d point cloud completion.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=wqD6TfbYkrn.
  * Meng et al. (2021) Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.
  * Nash et al. (2021) Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia.  Generating images with sparse representations.  _arXiv preprint arXiv:2103.03841_ , 2021.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nichol & Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Pandey et al. (2022) Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar.  Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents.  _arXiv preprint arXiv:2201.00308_ , 2022.
  * Parimala & Channappayya (2019) Kancharla Parimala and Sumohana Channappayya.  Quality aware generative adversarial networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Preechakul et al. (2021) Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn.  Diffusion autoencoders: Toward a meaningful and decodable representation.  _arXiv preprint arXiv:2111.15640_ , 2021.
  * Radford et al. (2015) Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ , 2015.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Saharia et al. (2021a) Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  _arXiv preprint arXiv:2111.05826_ , 2021a.
  * Saharia et al. (2021b) Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _arXiv preprint arXiv:2104.07636_ , 2021b.
  * Salimans & Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pp. 2256–2265. PMLR, 2015.
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Song & Ermon (2019) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Song & Ermon (2020) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33:12438–12448, 2020.
  * Watson et al. (2021) Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi.  Learning fast samplers for diffusion models by differentiating through sample quality.  In _International Conference on Learning Representations_ , 2021.
  * Xiao et al. (2021) Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the generative learning trilemma with denoising diffusion gans.  _arXiv preprint arXiv:2112.07804_ , 2021.
  * Yu et al. (2015) Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zhao et al. (2018) Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon.  Bias and generalization in deep generative models: An empirical study.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Zheng et al. (2022) Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou.  Truncated diffusion probabilistic models.  _arXiv preprint arXiv:2202.09671_ , 2022.
  * Zhou et al. (2021) Linqi Zhou, Yilun Du, and Jiajun Wu.  3d shape generation and completion through point-voxel diffusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 5826–5835, 2021.

##  Appendix A Appendix

###  A.1 ELBO Proof

Prove

$\displaystyle\log p({\bm{x}}^{0})\geq-(L_{\text{VAE}}+L_{\text{DDPM}})$ (13)

in Equation 7 in the main text.

###### Proof.

Through variational inference, we have

$\displaystyle\log p({\bm{x}}^{0})$ $\displaystyle=\log\int
p({\bm{x}}^{0:T^{\prime}},{\bm{z}})d{\bm{x}}^{1:T^{\prime}}d{\bm{z}}$ (14)
$\displaystyle=\log\int
q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})\frac{p({\bm{x}}^{0:T^{\prime}},{\bm{z}})}{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}d{\bm{x}}^{1:T^{\prime}}d{\bm{z}}$
(15) $\displaystyle\geq\int
q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})\log\frac{p({\bm{x}}^{0:T^{\prime}},{\bm{z}})}{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}d{\bm{x}}^{1:T^{\prime}}d{\bm{z}}$
(16)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}\log\frac{p({\bm{x}}^{0:T^{\prime}},{\bm{z}})}{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}.$
(17)

This is because $\log$ is a concave function and

$\displaystyle\int
q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})d{\bm{x}}^{1:T^{\prime}}d{\bm{z}}=1.$
(18)

We plug in the definition that

$\displaystyle p({\bm{x}}^{0:T^{\prime}},{\bm{z}})$
$\displaystyle=p({\bm{z}})p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}}),$
(19) $\displaystyle q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})$
$\displaystyle=q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})q_{\psi}({\bm{z}}|{\bm{x}}^{0})$
(20)

and obtain

$\displaystyle\log p({\bm{x}}^{0})$
$\displaystyle\geq\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}\log\frac{p({\bm{z}})p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}})}{q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})q_{\psi}({\bm{z}}|{\bm{x}}^{0})}$
(21)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}[\log\frac{p({\bm{z}})}{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}+\log\frac{p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}})}{q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})}].$
(22)

We set the loss function as

$\displaystyle
L=-\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}[\log\frac{p({\bm{z}})}{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}+\log\frac{p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}})}{q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})}].$
(23)

The first term is:

$\displaystyle L_{1}$
$\displaystyle=-\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}\log\frac{p({\bm{z}})}{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}$
(24)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}\log\frac{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}{p({\bm{z}})}$
(25)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})q_{\psi}({\bm{z}}|{\bm{x}}^{0})}\log\frac{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}{p({\bm{z}})}$
(26)
$\displaystyle=D_{\text{KL}}(q_{\psi}({\bm{z}}|{\bm{x}}^{0})||p({\bm{z}})).$
(27)

The second term is:

$\displaystyle L_{2}$
$\displaystyle=-\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}\log\frac{p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:({T^{\prime}}-1)}|{\bm{x}}^{{T^{\prime}}})}{q({\bm{x}}^{1:{T^{\prime}}}|{\bm{x}}_{0})}$
(28)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}\log\frac{q({\bm{x}}^{1:{T^{\prime}}}|{\bm{x}}_{0})}{p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})p_{\theta}({\bm{x}}^{0:({T^{\prime}}-1)}|{\bm{x}}^{{T^{\prime}}})}$
(29)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\log\frac{\prod_{t=1}^{T^{\prime}}q({\bm{x}}^{t}|{\bm{x}}^{t-1})}{\prod_{t=1}^{T^{\prime}}p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (30)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=1}^{T^{\prime}}\log\frac{q({\bm{x}}^{t}|{\bm{x}}^{t-1})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (31)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t}|{\bm{x}}^{t-1})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}+\log\frac{q({\bm{x}}^{1}|{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})}-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (32)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=2}^{T^{\prime}}\log(\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}\frac{q({\bm{x}}^{t}|{\bm{x}}^{0})}{q({\bm{x}}^{t-1}|{\bm{x}}^{0})})+\log\frac{q({\bm{x}}^{1}|{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})}-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (33)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}+\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t}|{\bm{x}}^{0})}{q({\bm{x}}^{t-1}|{\bm{x}}^{0})}+\log\frac{q({\bm{x}}^{1}|{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})}-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (34)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}+\log\frac{q({\bm{x}}^{{T^{\prime}}}|{\bm{x}}^{0})}{q({\bm{x}}^{1}|{\bm{x}}^{0})}+\log\frac{q({\bm{x}}^{1}|{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})}-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (35)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}+\log
q({\bm{x}}^{{T^{\prime}}}|{\bm{x}}^{0})-\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})-\log
p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})]$ (36)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\log\frac{q({\bm{x}}^{t}|{\bm{x}}_{0})}{p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})}+\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}-\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})]$ (37)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}\log\frac{q({\bm{x}}^{t}|{\bm{x}}_{0})}{p_{\phi}({\bm{x}}^{{T^{\prime}}}|{\bm{z}})}+L_{\text{DDPM}},$
(38)

where Line 33 is because

$\displaystyle
q({\bm{x}}^{t}|{\bm{x}}^{t-1})q({\bm{x}}^{t-1}|{\bm{x}}^{0})=q({\bm{x}}^{t-1},{\bm{x}}^{t}|{\bm{x}}^{0})=q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})q({\bm{x}}^{t}|{\bm{x}}^{0})\Longrightarrow$
(39) $\displaystyle
q({\bm{x}}^{t}|{\bm{x}}^{t-1})=\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})q({\bm{x}}^{t}|{\bm{x}}^{0})}{q({\bm{x}}^{t-1}|{\bm{x}}^{0})},$
(40)

and

$\displaystyle L_{\text{DDPM}}$
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}[\sum_{t=2}^{T^{\prime}}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}-\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})]$ (41)
$\displaystyle=\sum_{t=2}^{T^{\prime}}\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}-\mathbb{E}_{q({\bm{x}}^{1:{T^{\prime}}},{\bm{z}}|{\bm{x}}_{0})}\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})$ (42)
$\displaystyle=\sum_{t=2}^{T^{\prime}}\mathbb{E}_{q({\bm{x}}^{t}|{\bm{x}}^{0})q({\bm{x}}^{t-1}|{\bm{x}}^{0},{\bm{x}}^{t})q({\bm{x}}^{1:{T^{\prime}}}\setminus\\{{\bm{x}}^{t-1},{\bm{x}}^{t}\\},{\bm{z}}|{\bm{x}}^{0},{\bm{x}}^{t-1},{\bm{x}}^{t})}\log\frac{q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})}{p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t})}$
(43)
$\displaystyle\;\;\;\;-\mathbb{E}_{q({\bm{x}}^{2:{T^{\prime}}},{\bm{z}}|{\bm{x}}^{0},{\bm{x}}^{1})q({\bm{x}}^{1}|{\bm{x}}^{0})}\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1})$ (44)
$\displaystyle=\sum_{t=2}^{T^{\prime}}\mathbb{E}_{q({\bm{x}}^{t}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})||p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t}))-\mathbb{E}_{q({\bm{x}}^{1}|{\bm{x}}^{0})}\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1}).$ (45)

The first term in $L_{2}$ is

$\displaystyle\mathbb{E}_{q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0})}\log\frac{q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})}{p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})}$
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:(T^{\prime}-1)},{\bm{z}}|{\bm{x}}^{0},{\bm{x}}^{T^{\prime}})q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})}\log\frac{q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})}{p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})}$
(46)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:(T^{\prime}-1)},{\bm{z}}|{\bm{x}}^{0},{\bm{x}}^{T^{\prime}})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}}))$
(47)
$\displaystyle=\mathbb{E}_{q({\bm{x}}^{1:(T^{\prime}-1)}|{\bm{x}}^{0},{\bm{x}}^{T^{\prime}},{\bm{z}})q({\bm{z}}|{\bm{x}}^{0},{\bm{x}}^{T^{\prime}})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}}))$
(48)
$\displaystyle=\mathbb{E}_{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})).$
(49)

Now the total loss is

$\displaystyle L$ $\displaystyle=L_{1}+L_{2}$ (50)
$\displaystyle=D_{\text{KL}}(q_{\psi}({\bm{z}}|{\bm{x}}^{0})||p({\bm{z}}))+\mathbb{E}_{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}}))+L_{\text{DDPM}}$
(51) $\displaystyle=L_{\text{VAE}}+L_{\text{DDPM}},$ (52)

where $L_{\text{VAE}}$ is defined as

$\displaystyle
L_{\text{VAE}}=D_{\text{KL}}(q_{\psi}({\bm{z}}|{\bm{x}}^{0})||p({\bm{z}}))+\mathbb{E}_{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}})).$
(53)

We have finished the proof that

$\displaystyle\log p({\bm{x}}^{0})\geq-L=-(L_{\text{VAE}}+L_{\text{DDPM}}).$
(54)

∎

###  A.2 Conditional Generation

In the case of conditional generation, we use ${\bm{c}}$ to denote the
condition. It contains some information about the clean image ${\bm{x}}^{0}$.
For example, ${\bm{c}}$ could be the class of ${\bm{x}}^{0}$, a low resolution
image of ${\bm{x}}^{0}$, or a grayscale image of ${\bm{x}}^{0}$.

We adapt the encoding process in Equation 3 to

$\displaystyle
q({\bm{x}}^{1:T^{\prime}},{\bm{z}}|{\bm{x}}^{0},{\bm{c}})=q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0},{\bm{c}})q_{\psi}({\bm{z}}|{\bm{x}}^{1:T^{\prime}},{\bm{x}}^{0},{\bm{c}})=q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})q_{\psi}({\bm{z}}|{\bm{c}}),$
(55)

where $q_{\psi}({\bm{z}}|{\bm{c}})$ is the encoder. We set it to be a Gaussian
distribution whose mean and standard deviation are parameterized by a neural
network. $q({\bm{x}}^{1:T^{\prime}}|{\bm{x}}^{0})$ is the same as Equation 4.
We then adapt the sampling process in Equation 5 to

$\displaystyle
p({\bm{x}}^{0:T^{\prime}},{\bm{z}}|{\bm{c}})=p({\bm{z}})p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}},{\bm{c}}),$
(56)

where $p({\bm{z}})$ is assumed to follow standard Gaussian distribution,
$p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}})$ is the decoder
parameterized by a neural network, and
$p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}},{\bm{c}})$ is
the reverse process of the partial diffusion process in Equation 4. It is
defined as

$\displaystyle
p_{{\theta}}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}},{\bm{c}})=\prod_{t=1}^{T^{\prime}}p_{{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{c}}),\text{
where
}p_{{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{c}})={\mathcal{N}}({\bm{x}}^{t-1};\bm{\mu}_{{\theta}}({\bm{x}}^{t},{\bm{c}},t),\sigma_{t}^{2}{\bm{I}}).$
(57)

Through variational inference, we can prove that

$\displaystyle\log
p({\bm{x}}^{0}|{\bm{c}})\geq-(L_{\text{VAE}}+L_{\text{DDPM}}),$ (58)
$\displaystyle\text{where }L_{\text{VAE}}$
$\displaystyle=D_{\text{KL}}(q_{\psi}({\bm{z}}|{\bm{c}})||p({\bm{z}}))+\mathbb{E}_{q_{\psi}({\bm{z}}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}})),$
(59) $\displaystyle L_{\text{DDPM}}$
$\displaystyle=\sum_{t=2}^{T^{\prime}}\mathbb{E}_{q({\bm{x}}^{t}|{\bm{x}}^{0})}D_{\text{KL}}(q({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{x}}^{0})||p_{\theta}({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{c}}))-\mathbb{E}_{q({\bm{x}}^{1}|{\bm{x}}^{0})}\log
p_{\theta}({\bm{x}}^{0}|{\bm{x}}^{1},{\bm{c}}).$ (60)

The proof is the same as Section A.1. Now we have two loss terms,
$L_{\text{DDPM}}$ and $L_{\text{VAE}}$. This means that we can train a
conditional DDPM and a VAE-like model separately, and then combine them
together to obtain a conditional generative model. First, The loss term
$L_{\text{DDPM}}$ is the same of a normal conditional DDPM, except that we do
not need to train the full chain range from $1$ to $T$. Instead, we only need
to train the first several steps range from $1$ to $T^{\prime}$.

Next, we will show that the loss term $L_{\text{VAE}}$ amounts to train a VAE-
like conditional generative model. Similarly to Equation 11, we can set
$p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}})$ to

$\displaystyle p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}})\sim
N({\bm{x}}^{T^{\prime}};\sqrt{\bar{\alpha}_{t}}{\bm{f}}_{\phi}({\bm{z}},{\bm{c}}),(1-\bar{\alpha}_{t}){\bm{I}}).$
(61)

Then

$\displaystyle
D_{\text{KL}}(q({\bm{x}}^{T^{\prime}}|{\bm{x}}^{0})||p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}}))=C_{1}||{\bm{x}}^{0}-{\bm{f}}_{\phi}({\bm{z}},{\bm{c}})||^{2}+C_{2},$
(62)

where $C_{1},C_{2}$ are some scalar constants. Now the second term in
$L_{\text{VAE}}$ means that the decoder ${\bm{f}}_{\phi}({\bm{z}},{\bm{c}})$
need to reconstruct the original image ${\bm{x}}_{0}$, based on the latent
code ${\bm{z}}$ sampled from the encoder $q_{\psi}({\bm{z}}|{\bm{c}})$ and the
condition ${\bm{c}}$. Therefore, $L_{\text{VAE}}$ can be optimized by
following a similar procedure to train a VAE on clean images ${\bm{x}}^{0}$,
except that we need to change the input of the encoder $q_{\psi}$ from
${\bm{x}}^{0}$ to ${\bm{c}}$, and allow the decoder ${\bm{f}}_{\phi}$ to
utilize the condition ${\bm{c}}$ when reconstructing the original image
${\bm{x}}^{0}$.

#### Sampling Procedure.

After training the conditional DDPM and the VAE-like model separately, we can
combine them to form a conditional generative model. We can use the following
equation to generate samples given condition ${\bm{c}}$:

$\displaystyle
p({\bm{x}}^{0:T^{\prime}},{\bm{z}}|{\bm{c}})=q_{\psi}({\bm{z}}|{\bm{c}})p_{\phi}({\bm{x}}^{T^{\prime}}|{\bm{z}},{\bm{c}})p_{\theta}({\bm{x}}^{0:(T^{\prime}-1)}|{\bm{x}}^{T^{\prime}},{\bm{c}}).$
(63)

First sample the latent code ${\bm{z}}$ from $q_{\psi}({\bm{z}}|{\bm{c}})$ ,
then use the decoder of the VAE to generate an image
${\bm{f}}_{\phi}({\bm{z}},{\bm{c}})$. Next, sample ${\bm{x}}^{T^{\prime}}$
from the distribution described in Equation 61. Finally, use the DDPM to
sample ${\bm{x}}^{t-1}$ from
$p_{\bm{\theta}}({\bm{x}}^{t-1}|{\bm{x}}^{t},{\bm{c}})$ for
$t=T^{\prime},T^{\prime}-1,\cdots,1$, and output the generated image
${\bm{x}}^{0}$.

During sampling, we do not need to restrict how
${\bm{f}}_{\phi}({\bm{z}},{\bm{c}})$ is trained, or what kind of model it is.
It could be a conditional GAN or any other conditional generation model.

###  A.3 Computational Cost

On the CelebA dataset, we train the StyleGAN2, the ES-DDPM and the full-length
DDPM on 8 NVIDIA RTX 2080Ti GPUs. For CelebA-64, the StyleGAN2 and the full-
length DDPM are trained for $10^{5}$ (around 25 hours) iterations and $9\times
10^{5}$ (around 80 hours) respectively. For CelebA-128, the StyleGAN2 and the
full-length DDPM are trained for $2.4\times 10^{5}$ (around 183 hours)
iterations and $4\times 10^{6}$ (around 448 hours) respectively. As for ES-
DDPMs, their training time of each iteration is the same as full-length DDPMs
and can be inferred from Figure 2. For image generation experiments, it takes
around one day for the 1000-step DDPM to generate $50000$ random Celeba-128
images on 8 NVIDIA RTX 2080Ti GPUs. Other image generation experiment on
CelebA takes less time.

For the ImageNet-64 dataset, we use pre-trained DDPM models and pre-generated
BigGAN-deep images from the work (Dhariwal & Nichol, 2021). It takes about 8
hours for the 250-step DDPM to generate $50000$ random images on 4 NVIDIA A100
GPUs. Generation time of different length ES-DDPMs can be inferred
accordingly.

For LSUN-Bedroom and LSUN-Cat datasets, we use pre-trained DDPM models and
pre-generated StyleGAN images from the work (Dhariwal & Nichol, 2021). It
takes about one day for the 1000-step DDPM to generate $50000$ random LSUN-
Bedroom or LSUN-Cat images on 24 NVIDIA A100 GPUs. Generation time of
different length ES-DDPMs can be inferred accordingly.

###  A.4 DDPM Acceleration Experiement Results on LSUN Datasets

Table 8: FID comparison of different DDPM acceleration methods on the LSUN-Bedroom dataset. Denoising Steps | 500 | 100 | 50 | 20 | 10 | 5
---|---|---|---|---|---|---
ES-DDPM($T^{\prime}=100$)+StyleGAN+TR(Ours) |  | 1.85 | 1.83 | 2.03 | 2.67 | 3.84
DDIM (Song et al., 2020) |  | 6.62 | 6.75 | 8.89 | 16.95 |
PNDM (Liu et al., 2022) |  | 6.91 | 6.44 | 5.68 | 6.99 | 12.6
FastDPM (Kong & Ping, 2021) |  | 7.98 | 8.37 | 9.86 | 19.07 |
TDPM-GAN (Zheng et al., 2022) | 3.95 |  | 4.10 |  |  |
TDPM-CT (Zheng et al., 2022) | 4.01 |  | 4.95 |  |  |
|  |  |  |  |  |
Table 9: FID comparison of different DDPM acceleration methods on the LSUN-Cat dataset. “*” means the images for evaluation are generated by ourselves for this method. Denoising Steps | 100 | 50 | 20 | 10 | 5
---|---|---|---|---|---
ES-DDPM($T^{\prime}=100$)+StyleGAN2+TR(Ours) | 5.47 | 5.88 | 7.27 | 9.72 | 13.48
DDIM* (Song et al., 2020) | 7.43 | 8.80 | 12.40 | 20.11 | 48.39
TR* (Nichol & Dhariwal, 2021) | 9.88 | 14.60 | 28.86 | 56.62 | 114.00
|  |  |  |  |

###  A.5 Additional Controllable Generation Experiments

Figure 4: On CelebA-128, ES-DDPM can preserve high-level semantics in the
images generated by StyleGAN2 for not too large $T^{\prime}$’s. Hence our
combined model can control high-level semantics such as gender and age in
generated images. Figure 5: On LSUN-Cat-256, we perform stroke-based local
editing on StyleGAN2 generated cats. ES-DDPM($T^{\prime}=300$) can refine the
editing we made and remove artifacts.

Figure 4 shows that we can use StyleGAN2 to control the age and the gender of
the generated face, and the ES-DDPM can preserve these high-level semantic
information. In this way, we can control high-level semantics for our combined
model. Other works (Preechakul et al., 2021; Pandey et al., 2022) have also
shown that it is possible to combine DDPMs and VAEs to control high-level
semantics of the generated images. Compared with them, our method can achieve
better sample quality using the same number of denoising steps as shown in
Table 2 and Table 6. In addition, these methods need to alter the training
process of the DDPM and train a DDPM that is specific to the other model they
want to combine DDPM with. On the contrary, our method can combine a pre-
trained DDPM with any other generative model that has the ability to control
high-level semantics to achieve high-level controllable generation for the
DDPM.

Figure 5 shows that we can perform stroke-based local editing on the cat
images generated by StyleGAN2, and then use the ES-DDPM to refine it, mitigate
artifacts and make the images photo-realistic. This local editing technique is
the same as the one proposed in the work (Meng et al., 2021).