Despite the growing popularity of diffusion models, gaining a deep understanding of the model class remains somewhat elusive for the uninitiated in non-equilibrium statistical physics. With that in mind, we present what we believe is a more straightforward introduction to diffusion models using directed graphical modelling and variational Bayesian principles, which imposes relatively fewer prerequisites on the average reader. Our exposition constitutes a comprehensive technical review spanning from foundational concepts like deep latent variable models to recent advances in continuous-time diffusion-based modelling, highlighting theoretical connections between model classes along the way. We provide additional mathematical insights that were omitted in the seminal works whenever possible to aid in understanding, while avoiding the introduction of new notation. We envision this article serving as a useful educational supplement for both researchers and practitioners in the area, and we welcome feedback and contributions from the community.[<https://github.com/biomedia-mira/demystifying-diffusion>]

§ NOTATION

Description     Section

$\mathbf{x}$     Observed datapoint, e.g. input image     <ref>

$t$     Time variable $t \in \{1,\dots,T\}$, or $t \in [0,1]$ for continuous-time     <ref>, <ref>

$\mathbf{z}_t$     Latent variable at time $t$

$\mathbf{z}_{1:T}$     Finite set of latent variables $\mathbf{z}_1, \mathbf{z}_2,\dots,\mathbf{z}_T$     <ref>

$\mathbf{z}_{0:1}$     Set of latent variables in continuous-time from $t{=}0$ to $t{=}1$     <ref>

$\alpha_t$     Noise schedule coefficient $\alpha_t \in (0, 1)$     <ref>

$\sigma^2_t$     Noise schedule variance $\sigma^2_t \in (0, 1)$     <ref>

$\boldsymbol{\epsilon}_t$     Isotropic random noise, $\boldsymbol{\epsilon}_t \sim \mathcal{N}(0,\mathbf{I})$     <ref>, <ref>

$\mathrm{SNR}(t)$     Signal-to-noise ratio function, defined as $\alpha^2_t / \sigma^2_t$     <ref>

$q(\mathbf{z}_t \mid \mathbf{x})$     Latent variable distribution     <ref>

$q(\mathbf{z}_t \mid \mathbf{z}_s)$     Transition distribution from time $s$ to time $t$, where $s < t$     <ref>

$\alpha_{t|s}$     Transition coefficient from time $s$ to $t$     <ref>

$\sigma^2_{t|s}$     Variance of transition distribution     <ref>

$q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$     Top-down posterior distribution at time $s < t$     <ref>, <ref>

$\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x};s,t)$     Mean of top-down posterior distribution at time $s$; $\boldsymbol{\mu}_Q$ for short     <ref>

$\sigma^2_Q(s,t)$     Variance of top-down posterior distribution; $\sigma^2_Q$ for short     <ref>

$p(\mathbf{z}_s \mid \mathbf{z}_t)$     Generative transition distribution defined as $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x}=\hat{\mathbf{x}}_{\boldsymbol{\theta}}(\mathbf{z}_t, t))$     <ref>

$p(\mathbf{x} \mid \mathbf{z}_0)$     Image likelihood, equiv. $p(\mathbf{x} \mid \mathbf{z}_1)$ in discrete-time     <ref>, <ref>

$\boldsymbol{\phi}$     Variational parameters pertaining to $q_{\boldsymbol{\phi}}$     <ref>

$\boldsymbol{\theta}$     Model parameters pertaining to $p_{\boldsymbol{\theta}}$     <ref>, <ref>

$\hat{\mathbf{x}}_{\boldsymbol{\theta}}(\mathbf{z}_t, t)$     Denoising model for mapping any $\mathbf{z}_t$ to $\mathbf{x}$     <ref>

$\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{z}_t, t)$     Noise prediction model, approximates $\nabla_{\mathbf{z}_t} \log q(\mathbf{z}_t)$     <ref>

$\hat{\mathbf{s}}_{\boldsymbol{\theta}}(\mathbf{z}_t, t)$     Score model, equal to $-\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{z}_t, t) / \sigma_t$     <ref>

$\boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{z}_t;s,t)$     Predicted posterior mean at time $s$; $\boldsymbol{\mu}_{\boldsymbol{\theta}}$ for short     <ref>

$\mathrm{VLB}(\mathbf{x})$     Single-datapoint variational lower bound; equiv. $\mathrm{ELBO}(\mathbf{x})$     <ref>, <ref>

$\mathcal{L}_T(\mathbf{x})$     Discrete-time diffusion loss     <ref>

$\mathcal{L}_{\infty}(\mathbf{x})$     Continuous-time diffusion loss     <ref>, <ref>

$\mathcal{L}_w(\mathbf{x})$     Weighted diffusion loss; also $\mathcal{L}_{\infty}(\mathbf{x}, w)$     <ref>, <ref>

$\boldsymbol{\gamma}_{\boldsymbol{\eta}}(t)$     Neural network with parameters $\boldsymbol{\eta}$ for learning the noise schedule     <ref>

$w(\cdot)$     Noise level weighting function     <ref>

$\lambda$     Logarithm of the signal-to-noise ratio $\mathrm{SNR}(t)$; also $\lambda_t$     <ref>

$\lambda_{\mathrm{min}}$     Lowest log signal-to-noise ratio given by $f_\lambda(t=1)$     <ref>

$\lambda_{\mathrm{max}}$     Highest log signal-to-noise ratio given by $f_\lambda(t=0)$     <ref>

$p(\lambda)$     Density over noise levels     <ref>

$f_\lambda(t)$     Noise schedule function, mapping $t$ to $\lambda$     <ref>

$\mathcal{L}(t;\mathbf{x})$     Joint KL divergence up to time $t$     <ref>

$p_w(t)$     Data augmentation kernel specified by $w(\cdot)$     <ref>

§ INTRODUCTION

The basic setup of generative modelling involves using a dataset of observations of $\mathbf{x}$ to estimate the marginal distribution $p(\mathbf{x})$. Estimating $p(\mathbf{x})$ accurately enables various useful things such as: (i) sample generation; (ii) density estimation; (iii) compression; (iv) data imputation; (v) model selection, etc. Since $p(\mathbf{x})$ is unknown, we must approximate it with a model $p_{\boldsymbol{\theta}}(\mathbf{x}) \approx p(\mathbf{x})$, by optimizing some parameters $\boldsymbol{\theta}$. There are many ways to estimate $p(\mathbf{x})$; we focus on Variational Diffusion Models (VDMs) [22, 23], which are a family of diffusion-based generative models [47]. Despite the growing popularity of diffusion models, gaining a deep understanding of the model class remains somewhat elusive for the uninitiated in non-equilibrium statistical physics. Hence, we present a more straightforward introduction to diffusion models using directed graphical modelling and variational inference principles, which imposes relatively fewer prerequisites on the average reader.

With that in mind, the goal of this paper is to provide a thorough introduction to VDMs, without overlooking mathematical details or introducing new notation relative to the seminal works. We start by reviewing the basic fundamental principles and motivations behind Variational Autoencoders (VAEs) [25, 38], and their hierarchical counterparts [45, 48, 24]. We then introduce diffusion probabilistic models as a natural extension of discrete-time hierarchical VAEs with a particular choice of inference and generative model, before delving into continuous-time variants which represent infinitely deep VAEs.

Variational perspectives on diffusion have also been studied by [55, 17, 57]; we focus on VDMs since they represent the smoothest transition from VAEs to diffusion models. Our work is complementary to [28], as they too provide an introduction to diffusion models. However, our exposition is far more comprehensive, up-to-date, and mathematically consistent with the seminal works on VDMs [22, 23]. Furthermore, we cover recent material on VDMs [23] and provide additional instructive insights which we hope will contribute to the dissemination and understanding of this model class.

§.§ Variational Autoencoder

Variational autoencoder models assume that data $\mathbf{x} \in \mathcal{X}^D$ are generated by some random process involving an unobserved random variable $\mathbf{z} \in \mathcal{Z}^K$. The marginal distribution of $\mathbf{x}$ is: $p(\mathbf{x}) = \int p(\mathbf{x}, \mathbf{z})\mathop{\mathrm{d}\mathbf{z}}$.

The generative process is straightforward: (i) sample a latent variable from a prior distribution $\mathbf{z} \sim p(\mathbf{z})$; (ii) sample an observation from a conditional distribution $\mathbf{x}\sim p(\mathbf{x} \mid \mathbf{z})$. If we choose $\mathbf{z}$ to be a discrete random variable and $p(\mathbf{x} \mid \mathbf{z})$ to be a Gaussian distribution, then $p(\mathbf{x})$ is a Gaussian mixture. If we instead choose $\mathbf{z}$ to be a continuous random variable, then $p(\mathbf{x})$ represents an infinite mixture of Gaussians.

For complicated non-linear likelihood functions where $p(\mathbf{x} \mid \mathbf{z})$ is parameterized by a deep neural network, integrating out the latent variable $\mathbf{z}$ to compute $p(\mathbf{x})$ has no analytic solution, so we must rely on approximations. A straightforward Monte Carlo approximation of $p(\mathbf{x})$ is certainly possible:

\begin{align}
&p(\mathbf{x}) = \mathbb{E}_{\mathbf{z} \sim p(\mathbf{z})} \left[p(\mathbf{x} \mid \mathbf{z})\right] \approx
\frac{1}{N}\sum_{i=1}^N p(\mathbf{x} \mid \mathbf{z}_i), &\mathbf{z}_1,\dots,\mathbf{z}_N \mathop{\sim}\limits^{\mathrm{iid}} p(\mathbf{z}), &
\end{align}

but is subject to the curse of dimensionality, since the number of samples needed to properly cover the latent space grows exponentially with the dimensionality of the latent variable $\mathbf{z}$.

Alternatively, we can turn to variational methods, which pose probabilistic inference as an optimization problem [19]. The first thing to note is that the intractability of $p(\mathbf{x})$ is related to the intractability of the true posterior over the latent variable $p(\mathbf{z} \mid \mathbf{x})$ through a basic identity:

\begin{align}
&&p(\mathbf{x}) = \frac{p(\mathbf{x} \mid \mathbf{z})p(\mathbf{z})}{p(\mathbf{z} \mid \mathbf{x})},
\mathrm{where}
\mid \mathbf{x}) = \frac{p(\mathbf{x} \mid \mathbf{z})p(\mathbf{z})}{\int p(\mathbf{x} \mid \mathbf{z})p(\mathbf{z})\mathop{\mathrm{d}\mathbf{z}}}.&&
\end{align}

Using a complicated neural network-based likelihood renders the integral on the RHS intractable. To estimate $p(\mathbf{x})$ we can approximate the true posterior $p(\mathbf{z} \mid \mathbf{x})$ via a parametric inference model $q(\mathbf{z} \mid \mathbf{x})$ of our choice, such that $q(\mathbf{z} \mid \mathbf{x}) \approx p(\mathbf{z} \mid \mathbf{x})$. Learning a single function with shared variational parameters $\boldsymbol{\phi}$ to map each datapoint $\mathbf{x}$ to a posterior distribution $q(\mathbf{z} \mid \mathbf{x})$ is known as amortized inference.

We may optionally write $q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$ and $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ to explicitly state that these are parametric distributions realized by an encoder-decoder setup with variational parameters $\boldsymbol{\phi}$ and model parameters $\boldsymbol{\theta}$. The typical VAE setup specifies a prior $p(\mathbf{z})$ with no learnable parameters, and it is often chosen to be standard Gaussian: $p(\mathbf{z}) = \mathcal{N}(\mathbf{z}; 0, \mathbf{I})$. It is important to note that unlike the latent variable(s) $\mathbf{z}$ which are local, the parameters $\{\boldsymbol{\phi}, \boldsymbol{\theta}\}$ are global since they are shared for all datapoints. To improve our approximation, we'd like to minimize the Kullback-Leibler (KL) divergence $\argmin_{q(\mathbf{z} \mid \mathbf{x})} D_{\mathrm{KL}} \left(q(\mathbf{z} \mid \mathbf{x}) \parallel p(\mathbf{z} \mid \mathbf{x}) \right)$, but it is not possible do so directly as we do not have access to the true posterior $p(\mathbf{z} \mid \mathbf{x})$ for evaluation.

VAEs maximise the Variational Lower Bound (VLB) of $\log p(\mathbf{x})$:

\begin{align}
D_{\mathrm{KL}} \left(q(\mathbf{z} \mid \mathbf{x}) \parallel p(\mathbf{z} \mid \mathbf{x}) \right) &=  \int q(\mathbf{z} \mid \mathbf{x})  \log \frac{q(\mathbf{z} \mid \mathbf{x})}{p(\mathbf{z} \mid \mathbf{x})} \mathop{\mathrm{d}\mathbf{z}}
\\[5pt] &= \mathbb{E}_{q(\mathbf{z} \mid \mathbf{x})} \left[  \log q(\mathbf{z} \mid \mathbf{x}) - \log \frac{p(\mathbf{x}, \mathbf{z})}{p(\mathbf{x})}\right]
\\[5pt] &= \mathbb{E}_{q(\mathbf{z} \mid \mathbf{x})} \left[  \log q(\mathbf{z} \mid \mathbf{x}) - \log p(\mathbf{x}, \mathbf{z}) \right] + \log p(\mathbf{x})
\\[5pt] &= -\mathrm{VLB}(\mathbf{x}) + \log p(\mathbf{x}),
\end{align}

adding $\mathrm{VLB}(\mathbf{x})$ to both sides reveals:

\begin{align}
D_{\mathrm{KL}} \left(q(\mathbf{z} \mid \mathbf{x}) \parallel p(\mathbf{z} \mid \mathbf{x}) \right) + \mathrm{VLB}(\mathbf{x}) = \log p(\mathbf{x}) \implies \log p(\mathbf{x}) \geq \mathrm{VLB}(\mathbf{x}),
\end{align}

as $D_{\mathrm{KL}} \left(q(\mathbf{z} \mid \mathbf{x}) \parallel p(\mathbf{z} \mid \mathbf{x}) \right) \geq 0$ by Gibbs' inequality. Hence, maximizing the VLB implicitly minimizes the KL divergence of $q(\mathbf{z} \mid \mathbf{x})$ from the true posterior $p(\mathbf{z} \mid \mathbf{x})$ as desired. The VLB is also known as the Evidence Lower BOund (ELBO) since $p(\mathbf{x})$ is called the evidence. The VLB optimized by VAEs is:

\begin{align}
\mathrm{VLB}(\mathbf{x}) &= \mathbb{E}_{q(\mathbf{z} \mid \mathbf{x})} \left[\log \frac{p(\mathbf{x}, \mathbf{z})}{q(\mathbf{z} \mid \mathbf{x})}\right]
\\[5pt] &= \mathbb{E}_{q(\mathbf{z} \mid \mathbf{x})} \left[\log p(\mathbf{x} \mid \mathbf{z})\right]
+ \mathbb{E}_{q(\mathbf{z} \mid \mathbf{x})} \left[\log \frac{p(\mathbf{z})}{q(\mathbf{z} \mid \mathbf{x})} \right]
\\[5pt] &= \mathbb{E}_{q(\mathbf{z} \mid \mathbf{x})} \left[\log p(\mathbf{x} \mid \mathbf{z})\right] - D_{\mathrm{KL}} \left(q(\mathbf{z} \mid \mathbf{x}) \parallel p(\mathbf{z}) \right),
\end{align}

which amounts to an expected likelihood objective regularized by the KL of the posterior from the prior.

If we let $\mathcal{D}$ be a dataset of i.i.d. data, then $\mathrm{VLB}(\mathcal{D}) = \sum_{\mathbf{x} \in \mathcal{D}}\mathrm{VLB}(\mathbf{x})$. We can use stochastic variational inference [13] and the reparameterization trick [25, 38] to jointly optimize the VLB w.r.t. the model parameters $\boldsymbol{\theta}$, and variational parameters $\boldsymbol{\phi}$. For more details on this procedure, the reader may refer to [26] and [3].

[thick,scale=1, every node/.style=scale=1]
[latent] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[obs, below=15pt of z1] (x) $\mathbf{x}$;
[latent, draw=none, right=-2pt of z2, yshift=-16pt] (eq2) $p(\mathbf{z}_1 \mid \mathbf{z}_2)$;
[latent, draw=none, right=-2pt of z1, yshift=-16pt] (eq1) $p(\mathbf{x} \mid \mathbf{z}_1)$;
[latent, draw=none, right=-2pt of z3, xshift=6pt, yshift=-0pt] (eq3) $p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)$;

Hierarchical Generative Model

[thick,scale=1, every node/.style=scale=1]
[latent] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=13pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=6pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[obs, below=15pt of z1] (x) $\mathbf{x}$;
[latent, draw=none, right=-2pt of z2, yshift=-16pt] (eq2) $q(\mathbf{z}_2 \mid \mathbf{z}_1)$;
[latent, draw=none, right=-2pt of z1, yshift=-16pt] (eq1) $q(\mathbf{z}_1 \mid \mathbf{x})$;
[latent, draw=none, right=-2pt of z3, xshift=6pt, yshift=-0pt] (eq3) $q(\mathbf{z}_{t} \mid \mathbf{z}_{t-1})$;

Bottom-up Inference Model

[thick,scale=1, every node/.style=scale=1]
[obs] (x) $\mathbf{x}$;
[latent, below=15pt of x] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[-Latex[scale=1.0]] (x) to [out=225,in=135] (z2);
[-Latex[scale=1.0]] (x) to [out=225,in=135] (z1);
[latent, draw=none, right=-2pt of x, yshift=-16pt] (eq1) $q(\mathbf{z}_T \mid \mathbf{x})$;
[latent, draw=none, right=-2pt of z2, yshift=-16pt] (eq2) $q(\mathbf{z}_1 \mid \mathbf{z}_2, \mathbf{x})$;
[latent, draw=none, right=-2pt of z3, xshift=6pt, yshift=-0pt] (eq3) $q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x})$;

Top-down Inference Model

Hierarchical latent variable graphical models. (a) The generative model $p(\mathbf{x}, \mathbf{z}_{1:T})$ of a hierarchical VAE with $T$ latent variables is a Markov chain. (b) The standard bottom-up inference model $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ of a hierarchical VAE is a Markov chain in the reverse direction. (c) The top-down inference model follows the same topological ordering of the latent variables as the generative model. This top-down structure is used to specify diffusion models. In diffusion models the posterior $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ is tractable due to Gaussian conjugacy, which enables us to specify the generative model transitions as $p(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}) = q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x} = \hat{\mathbf{x}}_{\boldsymbol{\theta}}(\mathbf{z}_t; t))$,  where the data $\mathbf{x}$ is replaced by a denoising model $\hat{\mathbf{x}}_{\boldsymbol{\theta}}(\mathbf{z}_t; t)$.

§.§ Hierarchical VAE

A hierarchical VAE is a deep latent variable model comprised of a hierarchy of latent variables $\mathbf{z}_1,\mathbf{z}_2 \dots, \mathbf{z}_T$. Introducing additional (auxiliary) latent variables significantly improves the flexibility of both inference and generative models [45, 36, 30].

The joint distribution $p(\mathbf{x}, \mathbf{z}_{1:T})$ specifying a generative model of $\mathbf{x}$ is a variational Markov chain $\mathbf{z}_T \to \mathbf{z}_{T-1} \to \cdots \to \mathbf{z}_1 \to \mathbf{x}$:

\begin{align}
p(\mathbf{x}, \mathbf{z}_{1:T}) &= p(\mathbf{z}_T)p(\mathbf{z}_{T-1} \mid \mathbf{z}_T) \cdots p(\mathbf{z}_1 \mid \mathbf{z}_2) p(\mathbf{x} \mid \mathbf{z}_1)
\\[5pt] &
= p(\mathbf{z}_T) \left[\prod_{t=2}^{T}p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)\right]p(\mathbf{x} \mid \mathbf{z}_1). \label{eq: hvae_gen}
\end{align}

The approximate posterior $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ is a Markov chain in the reverse (bottom-up) direction $\mathbf{z}_T \leftarrow \mathbf{z}_{T-1} \leftarrow \cdots \leftarrow \mathbf{z}_1 \leftarrow \mathbf{x}$:

\begin{align}
q(\mathbf{z}_{1:T} \mid \mathbf{x}) &= q(\mathbf{z}_1 \mid \mathbf{x})q(\mathbf{z}_2 \mid \mathbf{z}_1)q(\mathbf{z}_3 \mid \mathbf{z}_2) \cdots q(\mathbf{z}_T \mid \mathbf{z}_{T-1})
\\[5pt] &
= q(\mathbf{z}_1 \mid \mathbf{x}) \prod_{t=2}^{T}q(\mathbf{z}_{t} \mid \mathbf{z}_{t-1}). \label{eq: hvae_inf}
\end{align}

The marginal likelihood $p(\mathbf{x})$ is obtained by marginalizing out the latent variables:

\begin{align}
&& p(\mathbf{x})
= \int p(\mathbf{x} \mid  \mathbf{z}_{1})p(\mathbf{z}_1) \mathop{\mathrm{d}\mathbf{z}_{1}},
% \mathrm{where}&&
= \int p(\mathbf{z}_t \mid  \mathbf{z}_{t+1})p(\mathbf{z}_{t+1}) \mathop{\mathrm{d}\mathbf{z}_{t+1}},&&t = 1,2,\dots,T-1,&&
\end{align}

and the model is fit by maximizing the VLB of $\log p(\mathbf{x})$:

\begin{align}
\log p(\mathbf{x}) \geq \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \left[ \log \frac{p(\mathbf{x}, \mathbf{z}_{1:T})}{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \right] \eqqcolon \mathrm{VLB}(\mathbf{x}).
\end{align}

§.§ Generative Feedback

[thick,scale=1, every node/.style=scale=1]
[latent] (zt) $\hat{\mathbf{z}}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\hat{\mathbf{z}}_2$;
[latent, below=15pt of z2] (z1) $\hat{\mathbf{z}}_1$;

[latent, left=25pt of zt] (zzt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=13pt of zzt] (zz3) 90$\mathbf{\cdots}$;
[latent, left=25pt of z2] (zz2) $\mathbf{z}_2$;
[latent, left=25pt of z1] (zz1) $\mathbf{z}_1$;
[obs, below=15pt of zz1] (xx) $\mathbf{x}$;

[latent, draw=none, left=15pt of zzt] (st) $\mathcal{N}(0,\sigma_T^2)$;
[latent, draw=none, left=15pt of zz2] (s2) $\mathcal{N}(0,\sigma_2^2)$;
[latent, draw=none, left=15pt of zz1] (s1) $\mathcal{N}(0,\sigma_1^2)$;

[latent, draw=none, right=25pt of zt] (dt) $\mathbf{d}_T$;
node[draw=none, scale=0.75, below=13pt of dt] (d3) 90$\mathbf{\cdots}$;
[latent, draw=none, right=25pt of z2] (d2) $\mathbf{d}_2$;
[latent, draw=none, right=25pt of z1] (d1) $\mathbf{d}_1$;
[obs, below=15pt of d1] (x) $\mathbf{x}$;

[blue, -Latex[scale=1.0]]d3dt
[blue, -]d2d3
[blue, -Latex[scale=1.0]]d1d2
[blue, -Latex[scale=1.0]]xd1
[densely dashed,-]dtzt
[densely dashed,-]d2z2
[densely dashed,-]d1z1

A Ladder Network. The latent variables $\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_T$ are noisy representations of $\mathbf{x}$, and $\mathbf{d}_1, \mathbf{d}_2, \dots, \mathbf{d}_T$ are clean representations; both sets are produced by a shared encoder (blue arrows). The variables $\hat{\mathbf{z}}_1, \hat{\mathbf{z}}_2, \dots, \hat{\mathbf{z}}_T$ are outputs of denoising functions where $\hat{\mathbf{z}}_t = g_t(\mathbf{z}_t, \hat{\mathbf{z}}_{t+1})$. Notice how $g_t(\cdot)$ receives both bottom-up and top-down information. The dashed horizontal lines denote local cost functions used to minimize  $\|\hat{\mathbf{z}}_t - \mathbf{d}_t \|^2_2$. The main difference compared to denoising diffusion models is that here the denoising targets $\mathbf{d}_t$ are learned representations of $\mathbf{x}$.

The trouble with hierarchical latent variable models with bottom-up inference is bottom-up inference. [4] and [48] both found hierarchical models with purely bottom-up inference are typically not capable of utilizing more than two layers of latent variables. This often manifests as posterior collapse, whereby the posterior distribution (of the top-most layer, say) collapses to a standard Gaussian prior, failing to learn meaningful representations and effectively deactivating latent variables.

To understand why bottom-up inference is challenging for even modestly deep hierarchies, we start by noting the asymmetry between the associated generative and inference models in Equations <ref> and <ref> respectively. [4, 47] point this out as a source of difficulty in training the inference model efficiently, since there is no way to express each term in the VLB as an expectation under a distribution over a single latent variable. [28] present a similar efficiency-based argument against using bottom-up inference in hierarchical latent variable models.

We claim that efficiency arguments paint an incomplete picture; the main reason one should avoid bottom-up inference is the lack of direct feedback from the generative model.
To show why generative feedback is important, we stress that the purpose of the inference model is to perform Bayesian inference at any given layer in the hierarchy. That is, to compute the posterior distribution $q(\mathbf{z}_t \mid \mathbf{x})$ over each latent variable $\mathbf{z}_t$:

\begin{align}
& q(\mathbf{z}_t \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid \mathbf{z}_t)p(\mathbf{z}_t)}{p(\mathbf{x})}
\propto p(\mathbf{x} \mid \mathbf{z}_t) \int p(\mathbf{z}_t \mid  \mathbf{z}_{t+1})p(\mathbf{z}_{t+1}) \mathop{\mathrm{d}\mathbf{z}_{t+1}}, \label{eq: qzx_bayes}
\end{align}

which clearly shows that the posterior is not only proportional to the current layer's prior $p(\mathbf{z}_t)$ but also to the layer above's $p(\mathbf{z}_{t+1})$, and so on, following the reverse of the generative Markov chain (Equation <ref>).

It therefore stands to reason that interleaving feedback from each transition in the generative model into each respective transition in the inference model can only make the inference network more accurate. To that end, we can take Equation <ref> and rewrite the posterior distribution over each $\mathbf{z}_t$ such that it contains a more explicit dependency on the preceding latent variables as prescribed:

\begin{align}
& q(\mathbf{z}_t \mid \mathbf{x}) = \int q(\mathbf{z}_t \mid \mathbf{z}_{t+1}, \mathbf{x}) \mathop{\mathrm{d}\mathbf{z}_{t+1}}
% \\[5pt] &
\propto \int p(\mathbf{x} \mid \mathbf{z}_t) p(\mathbf{z}_t \mid  \mathbf{z}_{t+1})p(\mathbf{z}_{t+1})\mathop{\mathrm{d}\mathbf{z}_{t+1}}
\\[5pt] \implies \ & q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x}) \propto p(\mathbf{x} \mid \mathbf{z}_{t-1}) p(\mathbf{z}_{t-1} \mid  \mathbf{z}_{t})p(\mathbf{z}_{t}).
\end{align}

The posterior $q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x})$ now follows the same topological ordering of the latent variables as the prior $p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)$, and it coincides with the top-down inference model in HVAEs [48, 24]. Figure <ref> shows how this top-down structure compares to the bottom-up approach. An added benefit of the top-down approach is that the generative model can also receive data-dependent feedback from the inference procedure, which [48] found to be beneficial in practice.

[58, 37] were the first to introduce such lateral feedback connections between the inference and generative paths in hierarchical latent variable models. They called their denoising autoencoder a ladder network (see Figure <ref>), which later inspired the ladder VAE [48]. [58] argue that incorporating lateral feedback connections enables the higher layers to learn abstract invariant representations as they no longer have to retain all the details about the input. Concretely, as depicted in Figure <ref>, each denoised variable $\hat{\mathbf{z}}_t \coloneqq g_t(\mathbf{z}_t, \hat{\mathbf{z}}_{t+1})$ is computed using a denoising function $g_t(\cdot)$ which receives bottom-up feedback from $\mathbf{z}_t$ and top-down feedback from $\hat{\mathbf{z}}_{t+1}$.

§.§ Top-down Inference

The joint approximate posterior $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ can be alternatively factorized into the top-down inference model [48, 24]. As mentioned in Section <ref>, the top-down inference model follows the same topological ordering of the latent variables as the generative model, that is:

\begin{align}
q(\mathbf{z}_{1:T} \mid \mathbf{x}) & = q(\mathbf{z}_T \mid \mathbf{x}) q(\mathbf{z}_{T-1} \mid \mathbf{z}_{T}, \mathbf{x}) \cdots q(\mathbf{z}_2 \mid \mathbf{z}_3, \mathbf{x})q(\mathbf{z}_1 \mid \mathbf{z}_2, \mathbf{x})
\\[5pt] &
= q(\mathbf{z}_T \mid \mathbf{x}) \prod_{t=2}^{T} q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{x}).
\end{align}

Variants of the top-down inference model have featured in much deeper state-of-the-art HVAEs for sample generation [29, 56, 5, 46] and approximate counterfactual inference [7, 33]. As we will explain later, the top-down hierarchical latent variable model serves as the basis for parameterizing denoising diffusion models [47, 11, 22].

For now, we derive the corresponding VLB to obtain a concrete optimization objective:

\begin{align}
\mathrm{VLB}(\mathbf{x}) & = \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \left[ \log \frac{p(\mathbf{x}, \mathbf{z}_{1:T})}{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \right]
\\[5pt] & = \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \left[ \log \frac{p(\mathbf{z}_T) p(\mathbf{x} \mid \mathbf{z}_1) \prod_{t=2}^{T}p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)}{q(\mathbf{z}_T \mid \mathbf{x}) \prod_{t=2}^{T} q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{x})} \right] \customtag{factor the joint}
\\[5pt] & = \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})}\left[\log p(\mathbf{x} \mid \mathbf{z}_{1}) + \log \frac{p(\mathbf{z}_T)}{ q(\mathbf{z}_T \mid \mathbf{x})}\right] + \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})}\left[\sum_{t=2}^T \log  \frac{p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)}{q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{x})} \right]
\\[5pt] & = \mathbb{E}_{q(\mathbf{z}_{1} \mid  \mathbf{x})}\left[\log p(\mathbf{x} \mid \mathbf{z}_{1})\right] + \mathbb{E}_{q(\mathbf{z}_{T} \mid \mathbf{x})}\left[\log \frac{p(\mathbf{z}_T)}{ q(\mathbf{z}_T \mid \mathbf{x})}\right] \nonumber \\[2pt] & \qquad\qquad\qquad\qquad\qquad\qquad\qquad + \sum_{t=2}^T \mathbb{E}_{q(\mathbf{z}_{t-1}, \mathbf{z}_{t} \mid \mathbf{x})} \left[\log \frac{p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)}{q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{x})} \right]
\\[5pt] & = \mathbb{E}_{q(\mathbf{z}_{1} \mid  \mathbf{x})}\left[\log p(\mathbf{x} \mid \mathbf{z}_{1})\right] + \mathbb{E}_{q(\mathbf{z}_{T} \mid \mathbf{x})}\left[\log \frac{p(\mathbf{z}_T)}{ q(\mathbf{z}_T \mid \mathbf{x})}\right] \nonumber \\[2pt] & \qquad\qquad\qquad\qquad\qquad\qquad\qquad + \sum_{t=2}^T \mathbb{E}_{q(\mathbf{z}_{t} \mid \mathbf{x})} \left[\mathbb{E}_{q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x})} \left[ \log \frac{p(\mathbf{z}_{t-1} \mid \mathbf{z}_t)}{q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{x})} \right]\right]
\\[5pt] & = \mathbb{E}_{q(\mathbf{z}_{1} \mid \mathbf{x})}\left[\log p(\mathbf{x} \mid \mathbf{z}_{1})\right] - D_{\mathrm{KL}}(q(\mathbf{z}_{T} \mid \mathbf{x}) \parallel p(\mathbf{z}_{T})) \nonumber \\[2pt] & \qquad\qquad\qquad\qquad\qquad\qquad\qquad - \sum_{t=2}^T \mathbb{E}_{q(\mathbf{z}_{t} \mid \mathbf{x})} \left[D_{\mathrm{KL}}(q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x}) \parallel p(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}))\right]. \label{eq: hvae_elbo}
\end{align}

It is well worth dedicating some time to understanding the details of the above derivation and the resulting expression, as it is the exact objective optimized by diffusion models as well.

One thing to notice is that it comprises the familiar trade-off between minimizing input reconstruction error and keeping the (hierarchical) approximate posterior $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ close to the prior $p(\mathbf{z}_{1:T})$. In contrast to standard VAEs, the prior in HVAEs is learned from data rather than being fixed, as this affords greater flexibility [24, 14, 54].

Demonstration of the hole problem in VAEs. Results are from a single stochastic layer VAE trained on a 2D toy dataset with five clusters. The latent variable $\mathbf{z}$ is also 2-dimensional for illustration purposes.
The leftmost column shows the dataset, overlaid with reconstructed datapoints (red border) and random samples from the generative model (blue border). The remaining columns show the assumed prior $p(\mathbf{z}) = \mathcal{N}(\mathbf{z};0, \mathbf{I})$ (blue contours) overlaid with the aggregate posterior $q(\mathbf{z}) = \sum_{i=1}^N q(\mathbf{z} \mid \mathbf{x}_i) / N$. As shown, there are regions with high density under the prior which are assigned low density under the aggregate posterior. This affects the quality of the random samples since we are likely to sample from regions in $p(\mathbf{z})$ not covered by the data. Further, the bottom row shows a common occurrence in VAEs where latent variable(s) are not activated/used at all by the model, in this case, $\mathbf{z}_2$ was not used.

The Problem with VAEs.

A primary issue with VAEs is the hole problem [39]. The hole problem refers to the mismatch between the so-called aggregate posterior $q(\mathbf{z})$ and the prior $p(\mathbf{z})$ over the latent variables [31, 14]. The aggregate posterior is simply the average posterior distribution over the dataset $\mathcal{D} = \{\mathbf{x}_i\}_{i=1}^N$, that is:

\begin{align}
&&q(\mathbf{z}) = \int q(\mathbf{z} \mid \mathbf{x})p_\mathcal{D}(\mathbf{x}) \mathop{\mathrm{d}\mathbf{x}}
, && p_{\mathcal{D}}(\mathbf{x}) = \frac{1}{N}\sum_{i=1}^N \delta(\mathbf{x} - \mathbf{x}_i),&&
\end{align}

where $p_\mathcal{D}(\mathbf{x})$ is the empirical distribution, constructed by a Dirac delta function $\delta(\cdot)$ centered on each training datapoint $\mathbf{x}_i$. As shown in Figure <ref>, there can be regions with high probability density under the prior which have low density under the aggregate posterior. This affects the quality of generated samples when the decoder receives $\mathbf{z}$'s sampled from regions not covered by the data. As we will show, diffusion models circumvent this by defining the aggregate posterior to be equal to the prior by construction.

§ VARIATIONAL DIFFUSION MODELS

[thick,scale=1, every node/.style=scale=1]
[obs] (x) $\mathbf{x}$;
[latent, below=15pt of x] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[-Latex[scale=1.0]] (x) to [out=225,in=135] (z2);
[-Latex[scale=1.0]] (x) to [out=225,in=135] (z1);
[latent, draw=none, right=-4pt of x, yshift=-16pt] (eq1) $q(\mathbf{z}_T \mid \mathbf{x})$;
[latent, draw=none, right=-4pt of z2, yshift=-16pt] (eq2) $q(\mathbf{z}_1 \mid \mathbf{z}_2, \mathbf{x})$;
[latent, draw=none, right=-4pt of z3, xshift=6pt, yshift=-0pt] (eq3) $q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x})$;

Top-down Hierarchy

[thick,scale=1, every node/.style=scale=1]
[obs] (x) $\mathbf{x}$;
[latent, below=15pt of x] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[-Latex[scale=1.0]] (x) to [out=225,in=135] (z2);
[-Latex[scale=1.0]] (x) to [out=225,in=135] (z1);
[densely dashed, -Latex[scale=1.0]] (zt) to [out=45,in=-45] (x);
[densely dashed, -Latex[scale=1.0]] (z2) to [out=45,in=-45] (x);
[densely dashed, -Latex[scale=1.0]] (z1) to [out=45,in=-45] (x);

Diffusion Model

[thick,scale=1, every node/.style=scale=1]
[latent] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[latent, draw=none, right=-4pt of z2, yshift=-16pt] (eq2) $q(\mathbf{z}_1 \mid \mathbf{z}_2, \mathbf{d}_1)$;
[latent, draw=none, right=-4pt of z3, xshift=6pt, yshift=-0pt] (eq3) $q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{d}_t)$;

[latent, draw=none, left=15pt of zt] (dt) $\mathbf{d}_T$;
node[draw=none, scale=0.75, below=13pt of dt] (d3) 90$\mathbf{\cdots}$;
[latent, draw=none, left=15pt of z2] (d2) $\mathbf{d}_2$;
[latent, draw=none, left=15pt of z1] (d1) $\mathbf{d}_1$;
[obs, below=15pt of d1] (x) $\mathbf{x}$;

HVAE Inference Model

[thick,scale=1, every node/.style=scale=1]
[latent, below] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.75, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=13pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=15pt of z2] (z1) $\mathbf{z}_1$;
[obs, below=15pt of z1] (x) $\mathbf{x}$;
[latent, draw=none, left=15pt of z1] (d1) ${\mathbf{d}}_1$;
[latent, draw=none, left=15pt of z2] (d2) ${\mathbf{d}}_2$;

node[draw=none, scale=0.75, left=4pt of z3] (d3) 55$\mathbf{\cdots}$;

[blue, -Latex[scale=1.0]]z2d1

[latent, draw=none, right=-4pt of z2, yshift=-16pt] (eq2) $q(\mathbf{z}_1 \mid \mathbf{z}_2, \mathbf{d}_1)$;
[latent, draw=none, right=-4pt of z3, xshift=6pt, yshift=-0pt] (eq3) $q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{d}_t)$;

Reverse Process

Probabilistic graphical models of HVAEs and diffusion models. (a) The general top-down hierarchical latent variable model. (b) The top-down model used to specify diffusion models, where $q(\mathbf{z}_T \mid \mathbf{x}) = q(\mathbf{z}_T)$ by construction. Here the posterior $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ is a fixed noising process, so the modelling task is bottom-up prediction of $\mathbf{x}$ from each $\mathbf{z}_t$, i.e. denoising (dashed lines). (c) The top-down model used for posterior inference in HVAEs. It consists of a deterministic bottom-up pass to compute $\mathbf{d}_1,\dots,\mathbf{d}_T$, followed a stochastic top-down pass to compute $\mathbf{z}_T,\dots,\mathbf{z}_1$. (d) The reverse process of a diffusion model, i.e. the generative model. The main differences compared to (c) are that here the deterministic variables $\mathbf{d}_{T-1},\dots,\mathbf{d}_1$ do not depend on $\mathbf{x}$ nor have their own hierarchical dependencies. Further, the blue lines represent a denoising model $\hat{\mathbf{x}}_{\boldsymbol{\theta}} :\mathbf{z}_t \to \mathbf{d}_t$ which is shared across the hierarchy.

A diffusion probabilistic model [47] can be understood as a hierarchical VAE with a particular choice of inference and generative model. Like HVAEs, diffusion models are deep latent variable models that maximize the variational lower bound of the log-likelihood of the data (i.e. the ELBO). Diffusion models were largely inspired by ideas from statistical physics rather than variational Bayesian methods, so they come with a different set of modelling choices and advantages. The general idea behind diffusion models is to define a fixed forward (inference) diffusion process that converts any complex data distribution into a tractable distribution, and then learn a generative model that reverses this diffusion process. Figure <ref> compares diffusion models with (top-down inference) HVAEs.

Diffusion probabilistic models have the following distinctive properties:

* The joint posterior $q(\mathbf{z}_{1:T} \mid \mathbf{x})$ is fixed rather than learned from observed data. This amounts to having a fixed encoder defining a Gaussian diffusion process of the data;

* Each latent variable $\mathbf{z}_t$ has the same dimensionality as the input data $\mathbf{x}$;

* The aggregate posterior $q(\mathbf{z}_T)$ is equal to the prior $p(\mathbf{z}_T)$ by construction;

* The functional form of the inference model is identical to that of the generative model. This corresponds exactly to the top-down inference model structure used in HVAEs;

* A single neural network is shared across all levels of the latent variable hierarchy, and each layer can be trained without having to compute the preceding ones;

* They maximize a particular weighted objective which seems to better align with human perception by suppressing modelling effort on imperceptible details.

[thick,scale=1, every node/.style=scale=1]
[obs] (x) $\mathbf{x}$;
[latent, above=20pt of x] (z1) $\mathbf{z}_1$;
[latent, above=20pt of z1] (z2) $\mathbf{z}_{2}$;
node[draw=none, scale=0.8, above=9pt of z2] (z3) 90$\mathbf{\cdots}$;
[-Latex[scale=1.0]] (x) to [out=45,in=-45] (z2);
[latent, above=9pt of z3] (zt) $\mathbf{z}_T$;
[rectangle, draw=none, fill=white, scale=1, above=2pt of x, xshift=-12pt] (eq1) $\alpha_1$;
[-Latex[scale=1.0]] (x) to [out=45,in=-45] (zt);
[rectangle, draw=none, fill=white, scale=1, right=22pt of z2, yshift=-13pt] (eq2) $\alpha_T$;
[rectangle, draw=none, scale=1, left=15pt of zt] (nt) $\mathcal{N}\big(0, \sigma^2_T\mathbf{I}\big)$;
[rectangle, draw=none, scale=1, left=15pt of z2] (n2) $\mathcal{N}\big(0, \sigma^2_2\mathbf{I}\big)$;
[rectangle, draw=none, scale=1, left=15pt of z1] (n1) $\mathcal{N}\big(0, \sigma^2_1\mathbf{I}\big)$;
[rectangle, draw=none, fill=white, scale=1, right=11pt of z1, yshift=-1pt] (eq2) $\alpha_2$;

Gaussian Diffusion

[thick,scale=1, every node/.style=scale=1]
[obs] (x) $\mathbf{x}$;
[latent, above=20pt of x] (z1) $\mathbf{z}_1$;
[latent, above=20pt of z1] (z2) $\mathbf{z}_{2}$;
node[draw=none, scale=0.8, above=6pt of z2] (z3) 90$\mathbf{\cdots}$;
[latent, above=12pt of z3] (zt) $\mathbf{z}_T$;
[rectangle, draw=none, fill=white, scale=1, above=1.5pt of x, xshift=18pt] (eq1) $\alpha_{1|0}$;
[rectangle, draw=none, fill=white, scale=1, above=1.5pt of z1, xshift=18pt] (eq2) $\alpha_{2|1}$;
[rectangle, draw=none, fill=white, scale=1, right=-0.4pt of z3] (eq3) $\alpha_{t|t-1}$;

[rectangle, draw=none, scale=1, left=15pt of zt] (nt) $\mathcal{N}\big(0, \sigma^2_{T|T-1}\mathbf{I}\big)$;
[rectangle, draw=none, scale=1, left=15pt of z2] (n2) $\mathcal{N}\big(0, \sigma^2_{2|1}\mathbf{I}\big)$;
[rectangle, draw=none, scale=1, left=15pt of z1] (n1) $\mathcal{N}\big(0, \sigma^2_{1|0}\mathbf{I}\big)$;

Markovian Transitions

[thick,scale=1, every node/.style=scale=1]
[obs] (x) $\mathbf{x}$;
[latent, below=20pt of x] (zt) $\mathbf{z}_T$;
node[draw=none, scale=0.8, below=6pt of zt] (z3) 90$\mathbf{\cdots}$;
[latent, below=12pt of z3] (z2) $\mathbf{z}_2$;
[latent, below=20pt of z2] (z1) $\mathbf{z}_1$;
[rectangle, draw=none, fill=white, scale=0.9, right=21pt of z2, yshift=30pt] (eq5) $\alpha_{1}\sigma^2_{2|1}/\sigma^{2}_{2}$;
[-Latex[scale=1.0]] (x) to [out=-45,in=45] (z1);
[rectangle, draw=none, fill=white, scale=0.9, right=12pt of zt] (eq6) $\alpha_{2}\sigma^2_{3|2}/\sigma^{2}_{3}$;
[-Latex[scale=1.0]] (x) to [out=-45,in=45] (z2);
[rectangle, draw=none, scale=1, left=15pt of zt] (nt) $\mathcal{N}\big(0, \mathbf{I}\big)$;
[rectangle, draw=none, scale=1, left=15pt of z2] (n2) $\mathcal{N}\big(0, \sigma^2_{2|3}\mathbf{I}\big)$;
[rectangle, draw=none, scale=1, left=15pt of z1] (n1) $\mathcal{N}\big(0, \sigma^2_{1|2}\mathbf{I}\big)$;
[rectangle, draw=none, fill=white, scale=0.9, left=-6pt of z1, yshift=22pt] (eq3) $\alpha_{2|1}\sigma^2_{1}/\sigma^{2}_{2}$;
[rectangle, draw=none, fill=white, scale=0.9, left=-6pt of z2, yshift=31pt] (eq4) $\alpha_{t|t-1}\sigma^2_{t-1}/\sigma^{2}_{t}$;

Top-down Posterior

Graphical model(s) describing a discrete-time Gaussian diffusion process ($T$ timesteps in total). (a) Parameterization of the forward process in terms of the conditionals $q(\mathbf{z}_t \mid \mathbf{x})$ (ref. Section <ref>). Each latent variable $\mathbf{z}_t$ is a noisy version of $\mathbf{x}$ given by: $\mathbf{z}_t = \alpha_t \mathbf{x} + \sigma_t \boldsymbol{\epsilon}_t$, and $\boldsymbol{\epsilon}_t \sim \mathcal{N}(0, \mathbf{I})$. (b) Markov chain formed by a sequence of transition distributions $q(\mathbf{z}_t \mid \mathbf{z}_{t-1})$ (ref. Section <ref>). Each latent variable is given by: $\mathbf{z}_t = \alpha_{t|t-1} \mathbf{z}_{t-1} + \sigma_{t|t-1} \boldsymbol{\epsilon}_t$, with parameters $\alpha_{t|t-1} \coloneqq \alpha_{t} / \alpha_{t-1}$ and $\sigma^2_{t|t-1} \coloneqq \sigma^2_{t} - \alpha^2_{t|t-1}\sigma^2_{t-1}$. (c) The top-down posterior is tractable due to Gaussian conjugacy: $q(\mathbf{z}_{t-1} \mid \mathbf{z}_t, \mathbf{x}) \propto q(\mathbf{z}_{t} \mid \mathbf{z}_{t-1})q(\mathbf{z}_{t-1} \mid \mathbf{x})$ (ref. Section <ref>), where $q(\mathbf{z}_{t-1} \mid \mathbf{x})$ acts as a Gaussian prior and $q(\mathbf{z}_{t} \mid \mathbf{z}_{t-1})$ as a Gaussian likelihood.
This top-down posterior is used to specify the generative model transitions as $p(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}) = q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x} = \hat{\mathbf{x}}_{\boldsymbol{\theta}}(\mathbf{z}_t; t))$, where the data $\mathbf{x}$ is replaced by a learnable denoising model $\hat{\mathbf{x}}_{\boldsymbol{\theta}}(\mathbf{z}_t; t)$.

Recent model innovations [11] – along with insights from stochastic processes [1] and score-based generative modelling [18, 59, 50, 53] – have yielded a myriad of impressive synthesis results at scale [34, 8, 35, 12, 40, 41, 16].

[22, 23] introduced a family of diffusion-based generative models they call Variational Diffusion Models (VDMs), and showed us that:

* The latent hierarchy can be made infinitely deep[This notion was concurrently explored by [53, 17, 57].] via a continuous-time model where $T \to \infty$;

* The continuous-time VLB is invariant to the noise schedule[Except for the signal-to-noise ratio at its endpoints (see Section <ref>).], meaning we can learn/adapt our noise schedule such that it minimizes the variance of the resulting Monte Carlo estimator of the loss;

* Although weighted diffusion objectives appear markedly different from regular maximum likelihood training, they all implicitly optimize some instance of the ELBO;

* VDMs are capable of state-of-the-art image synthesis, showing that standard maximum likelihood-based training objectives (i.e. the ELBO) are not inherently at odds with perceptual quality.

One important distinction to make between HVAEs and diffusion probabilistic models at this stage is that the role of the latent variables $\mathbf{z}_{1:T}$ are very different from a representation learning perspective. In HVAEs, the posterior latents $\mathbf{z}_{1:T}$ are useful learned representations of $\mathbf{x}$, which increase in semantic informativeness w.r.t. $\mathbf{x}$ as we go from $\mathbf{z}_1$ to $\mathbf{z}_T$. In diffusion probabilistic models, the latent variables $\mathbf{z}_{1:T}$ generally have no semantic meaning, and they decrease in informativeness w.r.t. $\mathbf{x}$ as we go from $\mathbf{z}_1$ to $\mathbf{z}_T$. This is because each $\mathbf{z}_t$ is simply a noisy version of $\mathbf{x}$ following a Gaussian diffusion process.

Gaussian diffusion process ($T{=}100$). Showing two popular noise schedules in terms of $\alpha$, $\sigma^2$ as per Section <ref>: (top) cosine [34]; (bottom) EDM [21].

§.§ Forward Process: Gaussian Diffusion

A Gaussian diffusion process gradually transforms data $\mathbf{x}$ into random noise by adding increasing amounts of Gaussian noise at each timestep $t=0,\dots,1$ resulting in a set of latent variables $\mathbf{z}_0,\dots,\mathbf{z}_1$.[For consistency with the continuous-time case where $T \to \infty$, we denote the latent variables as $\mathbf{z}_{0:1}$ rather than $\mathbf{z}_{1:T}$.]
Each latent variable $\mathbf{z}_t$ is simply a noisy version of $\mathbf{x}$, and its distribution conditional on $\mathbf{x}$ is given by:

\begin{align}
&& q(\mathbf{z}_t \mid \mathbf{x}) = \mathcal{N}\left(\mathbf{z}_t;\alpha_{t} \mathbf{x}, \sigma^2_{t} \mathbf{I}\right), && \mathbf{z}_t = \alpha_{t} \mathbf{x} + \sigma_{t}\boldsymbol{\epsilon}_t, && \boldsymbol{\epsilon}_t \sim \mathcal{N}\left(\boldsymbol{\epsilon}_t; 0, \mathbf{I}\right),&&
\end{align}

where $\alpha_{t} \in (0,1)$ and $\sigma^2_{t} \in (0,1)$ are chosen scalar valued functions of time $t \in [0,1]$. See Figures <ref>, <ref>.

The key idea is to define the forward diffusion process such that the noisiest latent variable $\mathbf{z}_1$ at time $t=1$ is standard Gaussian distributed: $q(\mathbf{z}_1 \mid \mathbf{x}) = \mathcal{N}(\mathbf{z}_1; 0, \mathbf{I})$, thus $q(\mathbf{z}_1 \mid \mathbf{x}) = q(\mathbf{z}_1)$. To that end, the scaling coefficients $\alpha_{0} > \ldots > \alpha_{1}$ decrease w.r.t. time $t$, whereas the noise variances $\sigma_{0}^2 < \ldots < \sigma_{1}^2$ increase w.r.t. $t$. As we will show, this enables us to learn a generative Markov chain which starts from $\mathbf{z}_1 \sim q(\mathbf{z}_1)$ and reverses the forward diffusion process to obtain samples from the data distribution. The implications of this are profound; the aggregate posterior $q(\mathbf{z}_1)$ is equal to the prior $p(\mathbf{z}_1)$ by construction, which circumvents the hole problem in VAEs (see Figure <ref>). [14] showed that the optimal prior is the aggregate posterior, as long as our posterior approximation is good enough.

A variance-preserving process is achieved by solving for the value of $\alpha_t$ such that the variance of the respective latent variable $\mathbb{V}[\mathbf{z}_t]$ is equal to the variance of the input data $\mathbb{V}[\mathbf{x}]$. This can be important from a modelling perspective, as adding increasing amounts of noise to the input alters its statistics.

We can first apply some basic properties of variance to simplify $\mathbb{V}[\mathbf{z}_t]$ as follows:

\begin{align}
\mathbb{V}[\mathbf{z}_t] = \mathbb{V}[\alpha_{t} \mathbf{x} + \sigma_{t}\boldsymbol{\epsilon}_t] = \mathbb{V}[\alpha_{t} \mathbf{x}] + \mathbb{V}[\sigma_{t}\boldsymbol{\epsilon}_t] = \alpha_{t}^2\mathbb{V}[\mathbf{x}] + \sigma_{t}^2\mathbb{V}[\boldsymbol{\epsilon}_t] = \alpha_{t}^2\mathbb{V}[\mathbf{x}] + \sigma_{t}^2,
\end{align}

since $\mathbb{V}[\boldsymbol{\epsilon}_t] = 1$ by definition. Taking the result and solving for $\alpha_t$ yields

\begin{align}
\alpha_{t}^2\mathbb{V}[\mathbf{x}] + \sigma_{t}^2 & = \mathbb{V}[\mathbf{x}]
\\[2pt] \alpha_{t}^2 & = \frac{\mathbb{V}[\mathbf{x}] - \sigma_{t}^2}{\mathbb{V}[\mathbf{x}]}
\\[2pt] \implies \mathbb{V}[\mathbf{z}_t] & = \mathbb{V}[\mathbf{x}] \iff \alpha_{t}^2 = 1 - \frac{\sigma_{t}^2}{\mathbb{V}[\mathbf{x}]},
\end{align}

which further simplifies to $\alpha_t^2 = 1 - \sigma_t^2$ as long as our input data is standardized.

§.§.§ Linear Gaussian Transitions: $q(\mathbf{z}_t \mid \mathbf{z}_{s})$

The conditional distribution of $\mathbf{z}_t$ given a preceding latent variable $\mathbf{z}_s$, for any timestep $s < t$, is:

\begin{align}
&&q(\mathbf{z}_t \mid \mathbf{z}_{s}) = \mathcal{N}\left(\mathbf{z}_t;\alpha_{t|s}\mathbf{z}_s, \sigma_{t|s}^2 \mathbf{I} \right), && \mathbf{z}_t = \alpha_{t|s} \mathbf{z}_s + \sigma_{t|s}\boldsymbol{\epsilon}_t, && \boldsymbol{\epsilon}_t \sim \mathcal{N}\left(\boldsymbol{\epsilon}_t; 0, \mathbf{I}\right),&&
\end{align}

which forms a Markov chain: $\mathbf{z}_1 \leftarrow \mathbf{z}_{(T-1)/T} \leftarrow \mathbf{z}_{(T-2)/T} \leftarrow \cdots \leftarrow \mathbf{z}_0 \leftarrow \mathbf{x}$, see Figure <ref> for an example. In the continuous-time case where $T \to \infty$, each transition is w.r.t. an infinitesimal change in time $\mathrm{d}t$.

The transition distribution $q(\mathbf{z}_t \mid \mathbf{z}_{s})$ is useful for computing closed-form expressions for the parameters of the posterior $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$, which defines our reverse-process, i.e. the generative model (ref. Section <ref>).

Let's focus on deriving $\alpha_{t|s}$ first. By construction, we know that each $\mathbf{z}_t$ is given by:

\begin{align}
\mathbf{z}_t = \alpha_{t} \mathbf{x} + \sigma_{t}\boldsymbol{\epsilon}_t = \alpha_{t} \left(\frac{\mathbf{z}_s - \sigma_s \boldsymbol{\epsilon}_s}{\alpha_s} \right) + \sigma_{t}\boldsymbol{\epsilon}_t,
\end{align}

since $\mathbf{x} = (\mathbf{z}_s - \sigma_s \boldsymbol{\epsilon}_s) / \alpha_s$ for any $s<t$. The conditional mean of $q(\mathbf{z}_t \mid \mathbf{z}_{s})$ is then readily given by:

\begin{align}
\mathbb{E} \left[ \mathbf{z}_t \mid \mathbf{z}_s \right] &= \alpha_{t} \left(\frac{\mathbf{z}_s - \sigma_s \mathbb{E} \left[\boldsymbol{\epsilon}_s\right]}{\alpha_s} \right) + \sigma_{t}\mathbb{E} \left[\boldsymbol{\epsilon}_t\right]
\\[5pt] &
= \frac{\alpha_t}{\alpha_s}\mathbf{z}_s \customtag{since $\mathbb{E}[\boldsymbol{\epsilon}_t] = 0, \ \forall t$}
\\[5pt] &
\eqqcolon \alpha_{t|s}\mathbf{z}_s. \label{eq: cond_alpha}
\end{align}

To compute a closed-form expression for the variance $\sigma^2_{t|s}$ of the transition distribution $q(\mathbf{z}_t \mid \mathbf{z}_{s})$, we can start by rewriting the equation for $\mathbf{z}_t$ in terms of the preceding latent $\mathbf{z}_s$ as follows:

\begin{align}
\mathbf{z}_t &= \alpha_{t|s} \mathbf{z}_s + \sigma_{t|s}\boldsymbol{\epsilon}_{t}
\\[5pt] &= \frac{\alpha_{t}}{\alpha_{s}} \left( \alpha_s \mathbf{x} + \sigma_s \boldsymbol{\epsilon}_s\right) + \sigma_{t|s}\boldsymbol{\epsilon}_{t} \customtag{substitute $\alpha_{t|s}$ and $\mathbf{z}_s$}
\\[5pt] & = \alpha_t \mathbf{x} + \frac{\alpha_t}{\alpha_s} \sigma_s \boldsymbol{\epsilon}_s + \sigma_{t|s}\boldsymbol{\epsilon}_{t}
\\[5pt] \implies \sigma_t \boldsymbol{\epsilon}_{t} & = \frac{\alpha_t}{\alpha_s} \sigma_s \boldsymbol{\epsilon}_s + \sigma_{t|s}\boldsymbol{\epsilon}_{t}. \customtag{since $\mathbf{z}_t = \alpha_t \mathbf{x} + \sigma_t \boldsymbol{\epsilon}_t$}
\end{align}

The above implication allows us to compute the variance $\sigma_{t|s}^2$ straightforwardly. Firstly, recall that variance is invariant to changes in a location parameter, therefore: $\mathbb{V}\left[ cX \right] = c^2\mathbb{V}\left[ X \right]$ for some constant $c$ and random variable $X$. Secondly, the variance of a sum of $n$ independent random variables is simply the sum of their variances: $\mathbb{V}\left[\sum_{i=1}^n X_n\right] = \sum_{i=1}^n\mathbb{V}\left[ X_i\right]$. Using these two properties we can show that:

\begin{align}
\mathbb{V}\left[\sigma_t \boldsymbol{\epsilon}_t\right] & = \mathbb{V}\left[\frac{\alpha_t}{\alpha_s} \sigma_s \boldsymbol{\epsilon}_s + \sigma_{t|s}\boldsymbol{\epsilon}_{t}\right]
\\[5pt] \sigma^2_t\mathbb{V}\left[ \boldsymbol{\epsilon}_t\right] & = \left(\frac{\alpha_t}{\alpha_s}\right)^2 \sigma_s^2\mathbb{V}\left[ \boldsymbol{\epsilon}_s\right] + \sigma_{t|s}^2 \mathbb{V}\left[\boldsymbol{\epsilon}_{t}\right]
\\[5pt] \sigma^2_t & = \left(\frac{\alpha_t}{\alpha_s}\right)^2 \sigma_s^2 + \sigma_{t|s}^2
\\[5pt] \sigma_{t|s}^2 & = \sigma^2_t - \alpha_{t|s}^2 \sigma_s^2. \label{eq: post_var}
\end{align}

§.§.§ Top-down Posterior: $q(\mathbf{z}_s \mid \mathbf{z}_{t}, \mathbf{x})$

Since the forward process is a Markov chain, the joint distribution of any two latent variables $\mathbf{z}_t$ and $\mathbf{z}_s$ where $t > s$ factorizes as: $q(\mathbf{z}_s, \mathbf{z}_t \mid \mathbf{x}) =
q(\mathbf{z}_t \mid \mathbf{z}_s)q(\mathbf{z}_s \mid \mathbf{x})$. Using Bayes' theorem, it is then possible to derive closed-form expressions for the parameters of the posterior distribution $q(\mathbf{z}_s \mid \mathbf{z}_{t}, \mathbf{x})$, which is itself Gaussian due to conjugacy, where $q(\mathbf{z}_s \mid \mathbf{x})$ acts as a Gaussian prior and $q(\mathbf{z}_t \mid \mathbf{z}_s)$ a Gaussian likelihood:

\begin{align}
&&q(\mathbf{z}_s \mid \mathbf{z}_{t}, \mathbf{x}) = \mathcal{N} \left(\mathbf{z}_s;\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x}; s, t), \sigma^2_Q(s,t) \mathbf{I}\right), && \mathbf{z}_s = \boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x}; s, t) + \sigma_{Q}(s,t)\boldsymbol{\epsilon}_t, &&
\end{align}

with $\boldsymbol{\epsilon}_t \sim \mathcal{N}\left(\boldsymbol{\epsilon}_t; 0, \mathbf{I}\right)$. In the following, we will derive closed-form expressions for the posterior parameters $\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x}; s, t)$ and $\sigma^2_Q(s,t)$ in detail. For a graphical model of the posterior see Figure <ref>.

Before proceeding, we note that this posterior distribution will be instrumental in defining our generative model (i.e. the reverse process) as explained later on in Section <ref>. Furthermore, notice that the posterior $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$ coincides with the top-down inference model specification of a hierarchical VAE.

For simplicity, let $D$ denote the dimensionality of $\mathbf{z}_t$, satisfying $\mathrm{dim}(\mathbf{z}_t) = \mathrm{dim}(\mathbf{x}), \forall t$. Furthermore, recall that our covariance matrix of choice is isotropic/spherical: $\sigma^2_Q \mathbf{I}$. The posterior is then given by

\begin{align}
q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x}) &
= \frac{q(\mathbf{z}_t \mid \mathbf{z}_s)q(\mathbf{z}_s \mid \mathbf{x})}{q(\mathbf{z}_t \mid \mathbf{x})}
\\ & \propto q(\mathbf{z}_t \mid \mathbf{z}_s)q(\mathbf{z}_s \mid \mathbf{x})
\\[5pt] &
= \mathcal{N}\left(\mathbf{z}_t; \alpha_{t|s} \mathbf{z}_s, \sigma^2_{t|s}\mathbf{I}\right)\cdot \mathcal{N}\left(\mathbf{z}_s;\alpha_s\mathbf{x}, \sigma^2_s\mathbf{I} \right)
\\[5pt] & = \prod_{i=1}^D \frac{1}{\sigma_{t|s}\sqrt{2\pi}}\exp\left\{-\frac{1}{2\sigma_{t|s}^2}\left(\mathbf{z}_{t,i} - \alpha_{t|s}\mathbf{z}_{s,i}\right)^2 \right\} \cdot \prod_{i=1}^D \frac{1}{\sigma_{s}\sqrt{2\pi}}\exp\left\{-\frac{1}{2\sigma^2_{s}}\left(\mathbf{z}_{s,i} - \alpha_{s}\mathbf{x}_i\right)^2 \right\}
\\[5pt] & \propto \prod_{i=1}^D \exp\left\{-\frac{1}{2\sigma_{t|s}^2}\left(\mathbf{z}_{t,i} - \alpha_{t|s}\mathbf{z}_{s,i}\right)^2 \right\} \cdot \prod_{i=1}^D \exp\left\{-\frac{1}{2\sigma^2_{s}}\left(\mathbf{z}_{s,i} - \alpha_{s}\mathbf{x}_i\right)^2 \right\}
\\[5pt] & = \prod_{i=1}^D \exp\left\{-\frac{1}{2\sigma^2_{t|s}}\left(\mathbf{z}_{t,i}^2 - 2\mathbf{z}_{t,i}\alpha_{t|s}\mathbf{z}_{s,i} + \alpha_{t|s}^2\mathbf{z}_{s,i}^2\right) -\frac{1}{2\sigma^2_{s}}\left(\mathbf{z}_{s,i}^2 - 2\mathbf{z}_{s,i}\alpha_{s}\mathbf{x}_i + \alpha_{s}^2\mathbf{x}_i^2\right) \right\}
\\[5pt] & = \prod_{i=1}^D \exp\left\{-\frac{1}{2}\left[\frac{\mathbf{z}_{t,i}^2 - 2\mathbf{z}_{t,i}\alpha_{t|s}\mathbf{z}_{s,i} + \alpha_{t|s}^2\mathbf{z}_{s,i}^2}{\sigma^2_{t|s}} + \frac{\mathbf{z}_{s,i}^2  -2\mathbf{z}_{s,i}\alpha_{s}\mathbf{x}_i + \alpha_{s}^2\mathbf{x}_i^2}{\sigma^2_{s}} \right]\right\}
\\[5pt] & = \prod_{i=1}^D \exp\left\{-\frac{1}{2}\left[
\mathbf{z}_{s,i}^2\left(\frac{\alpha_{t|s}^2}{\sigma_{t|s}^2} + \frac{1}{\sigma_s^2}\right) -2\mathbf{z}_{s,i}\left( \frac{\alpha_{t|s}\mathbf{z}_{t,i}}{\sigma_{t|s}^2}+\frac{\alpha_s\mathbf{x}_i}{\sigma_s^2}\right)
+\frac{\mathbf{z}_{t,i}^2}{\sigma_{t|s}^2} + \frac{\alpha_s^2\mathbf{x}_i^2}{\sigma_{s}^2}
\right]\right\}.
\label{eq:mm}
\end{align}

The next step is to `match the moments' from Equation (<ref>) with what we expect to see in a Gaussian distribution, i.e. something of the form: $\mathcal{N}\left(x;\mu, \sigma^2 \right)\propto \exp\left\{- \frac{x^2}{2\sigma^2} + \frac{\mu x}{\sigma^2} - \frac{\mu^2}{2\sigma_2}
\right\}$. This exercise yields closed-form expressions for the parameters of the posterior distribution as desired. Without loss of generality, consider the $D=1$ dimensional case for brevity.

Matching the first term in Eq. (<ref>) with $-\frac{x^2}{2\sigma^2}$ we can see that:

\begin{align}
-\frac{\mathbf{z}_s^2}{2}\left(\frac{\alpha_{t|s}^2}{\sigma_{t|s}^2} + \frac{1}{\sigma_s^2}\right) \implies \frac{1}{\sigma^2_Q} = \frac{\alpha_{t|s}^2}{\sigma_{t|s}^2} + \frac{1}{\sigma_s^2},
\end{align}

where $\sigma^2_Q$ is the variance of the posterior $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$. Matching the second term in Eq. (<ref>) with $\frac{\mu x}{\sigma^2}$ we get:

\begin{align}
\mathbf{z}_s\left(\frac{\alpha_{t|s}\mathbf{z}_t}{\sigma_{t|s}^2} + \frac{\alpha_s\mathbf{x}}{\sigma_s^2}\right) \implies \frac{\boldsymbol{\mu}_Q}{\sigma^2_Q} = \frac{\alpha_{t|s}\mathbf{z}_t}{\sigma_{t|s}^2} + \frac{\alpha_s\mathbf{x}}{\sigma_s^2} \implies \boldsymbol{\mu}_Q = {\sigma^2_Q} \left(\frac{\alpha_{t|s}\mathbf{z}_t}{\sigma_{t|s}^2} + \frac{\alpha_s\mathbf{x}}{\sigma_s^2}\right),
\end{align}

where $\boldsymbol{\mu}_Q$ is the mean of the posterior $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$.

The closed-form expressions for $\boldsymbol{\mu}_Q$, $\sigma^2_Q$ simplify quite significantly:

\begin{align}
\frac{1}{\sigma^2_Q} &= \frac{\sigma^2_s}{\sigma^2_s}\cdot
\frac{\alpha_{t|s}^2}{\sigma_{t|s}^2} + \frac{\sigma^2_{t|s}}{\sigma^2_{t|s}} \cdot \frac{1}{\sigma_s^2} \\[5pt] & = \frac{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2}{\sigma_{t|s}^2\sigma_s^2} \implies \sigma_Q^2 = \frac{\sigma_{t|s}^2\sigma_s^2}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2},
\end{align}

and for the posterior mean we then have:

\begin{align}
\boldsymbol{\mu}_Q & =
{\sigma^2_Q} \left(\frac{\alpha_{t|s}\mathbf{z}_t}{\sigma_{t|s}^2} + \frac{\alpha_s\mathbf{x}}{\sigma_s^2}\right)
\\[5pt] &= \frac{\sigma_{t|s}^2\sigma_s^2}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2} \cdot \frac{\sigma_s^2\alpha_{t|s}\mathbf{z}_t + \sigma_{t|s}^2\alpha_s\mathbf{x}}{\sigma_{t|s}^2\sigma_s^2}
\\[5pt] &
= \frac{\sigma_s^2\alpha_{t|s}\mathbf{z}_t + \sigma_{t|s}^2\alpha_s\mathbf{x}}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2}
\\[5pt] &= \frac{\alpha_{t|s}\sigma_s^2}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2}\mathbf{z}_t + \frac{\alpha_s \sigma^2_{t|s}}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2}\mathbf{x}.
\end{align}

Using the fact that $\sigma_{t|s}^2 = \sigma^2_t - \alpha_{t|s}^2 \sigma_s^2$ as in Equation <ref>, we get the final expression:

\begin{align}
\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x};s,t) = \frac{\alpha_{t|s}\sigma_s^2}{\sigma^2_{t}}\mathbf{z}_t + \frac{\alpha_s \sigma^2_{t|s}}{\sigma_{t}^2}\mathbf{x}, \label{eq: post_mu}
\end{align}

revealing that the posterior mean $\boldsymbol{\mu}_Q$, equivalently denoted as $\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x};s,t)$ by [22], is essentially a weighted average of the conditioning set $\{\mathbf{z}_t, \mathbf{x}\}$ of the posterior distribution $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$.

In summary, the top-down posterior distribution is given by:

\begin{align}
q(\mathbf{z}_s \mid \mathbf{z}_{t}, \mathbf{x}) & = \mathcal{N} \left(\mathbf{z}_s;\frac{\alpha_{t|s}\sigma_s^2}{\sigma^2_{t}}\mathbf{z}_t + \frac{\alpha_s \sigma^2_{t|s}}{\sigma_{t}^2}\mathbf{x}, \frac{\sigma_{t|s}^2\sigma_s^2}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2} \mathbf{I}\right)
\\[5pt] & = \mathcal{N} \left(\mathbf{z}_s;\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x}; s, t), \sigma^2_Q(s,t) \mathbf{I}\right).
\end{align}

To conclude, Table <ref> provides a concise breakdown of all the distributions involved in defining a Gaussian diffusion, along with the respective closed-form expressions of their parameters.

Distribution     Mean     Covariance

$q(\mathbf{z}_t \mid \mathbf{x})$ (<ref>)     $\alpha_t \mathbf{x}$      $\sigma_t^2\mathbf{I}$

$q(\mathbf{z}_t \mid \mathbf{z}_s)$ (<ref>)     $\alpha_{t|s} \mathbf{z}_s$     $\sigma_{t|s}^2\mathbf{I}$

$q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$ (<ref>)     $\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x}; s, t)$      $\sigma^2_Q(s,t)\mathbf{I}$

Parameter     Expression

$\alpha_{t|s}$     $\alpha_t / \alpha_s$

$\sigma^2_{t|s}$     $\sigma^2_t - \alpha_{t|s}^2 \sigma_s^2$

$\boldsymbol{\mu}_Q(\mathbf{z}_t, \mathbf{x}; s, t)$     $\displaystyle\frac{\alpha_{t|s}\sigma_s^2}{\sigma^2_{t}}\mathbf{z}_t + \frac{\alpha_s \sigma^2_{t|s}}{\sigma_{t}^2}\mathbf{x}$

$\sigma^2_Q(s,t)$     $\displaystyle \frac{\sigma_{t|s}^2\sigma_s^2}{\alpha_{t|s}^2 \sigma_s^2 + \sigma_{t|s}^2}$

Breakdown of the distributions involved in defining a typical Gaussian diffusion (LHS), along with closed-form expressions for their respective parameters (RHS). Note that $s$ denotes a preceding timestep relative to timestep $t$, i.e. $s < t$. The top-down posterior distribution $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x})$ is tractable due to Gaussian conjugacy: $q(\mathbf{z}_s \mid \mathbf{z}_t, \mathbf{x}) \propto q(\mathbf{z}_t \mid \mathbf{z}_s) q(\mathbf{z}_s \mid \mathbf{x}) $, where $q(\mathbf{z}_s \mid \mathbf{x})$ plays the role of a conjugate (Gaussian) prior and $q(\mathbf{z}_t \mid \mathbf{z}_s)$ the plays the role of a Gaussian likelihood.

§.§.§ Learning the Noise Schedule

Perturbing data with multiple noise scales and choosing an appropriate noise schedule is instrumental to the success of diffusion models. The noise schedule of the forward process is typically pre-specified and has no learnable parameters, however, VDMs learn the noise schedule via the parameterization:

\begin{align}
\sigma_t^2 = \mathrm{sigmoid}\left(\gamma_{\boldsymbol{\eta}}(t)\right),
\end{align}

where $\gamma_{\boldsymbol{\eta}}(t)$ is a monotonic neural network comprised of linear layers with weights $\boldsymbol{\eta}$ restricted to be positive. A monotonic function is a function defined on a subset of the real numbers which is either entirely non-increasing or entirely non-decreasing. As explained later, the noise schedule can be conveniently parameterized in terms of the signal-to-noise ratio. The signal-to-noise ratio (SNR) is defined as $\mathrm{SNR}(t) = \alpha_t^2 / \sigma_t^{2}$, and since $\mathbf{z}_t$ grow noisier over time we have that: $\mathrm{SNR}(t) < \mathrm{SNR}(s)$ for any $t > s$.

For now, we provide some straightforward derivations of the expressions for $\alpha_t^2$ and $\mathrm{SNR}(t)$ as a function of $\gamma_{\boldsymbol{\eta}}(t)$. Recall that in a variance-preserving diffusion process $\alpha_t^2 = 1 - \sigma_t^2$, therefore:

\begin{align}
\alpha_t^2 &= 1- \sigma_t^2 = 1 - \mathrm{sigmoid}\left(\gamma_{\boldsymbol{\eta}}(t)\right) \implies \alpha_t^2 = \mathrm{sigmoid}\left(-\gamma_{\boldsymbol{\eta}}(t)\right),
\end{align}

as for an input $x \in \mathbb{R}$ the following holds

\begin{align}
1 - \mathrm{sigmoid}\left(x\right) &= 1 - \frac{1}{1 + e^{-x}} = \frac{1 + e^{-x}}{1 + e^{-x}} - \frac{1}{1 + e^{-x}} = \frac{e^{-x}}{1 + e^{-x}} \cdot \frac{e^{x}}{e^{x}}
% \\[5pt] &
= \mathrm{sigmoid}\left(-x\right).
\end{align}

To derive $\mathrm{SNR}(t)$ as a function of $\gamma_{\boldsymbol{\eta}}(t)$, we simply substitute in the above equations and simplify:

\begin{align}
\mathrm{SNR}(t) &= \frac{\alpha_t^2}{\sigma_t^2} = \frac{\mathrm{sigmoid}\left(-\gamma_{\boldsymbol{\eta}}(t)\right)}{\mathrm{sigmoid}\left(\gamma_{\boldsymbol{\eta}}(t)\right)} \customtag{by definition}
\\[5pt] &= \frac{(1+e^{\gamma_{\boldsymbol{\eta}}(t)})^{-1}}{(1+e^{-\gamma_{\boldsymbol{\eta}}(t)})^{-1}} = \frac{1+e^{-\gamma_{\boldsymbol{\eta}}(t)}}{1+e^{\gamma_{\boldsymbol{\eta}}(t)}} = \frac{\frac{e^{\gamma_{\boldsymbol{\eta}}(t)}}{e^{\gamma_{\boldsymbol{\eta}}(t)}}+\frac{1}{e^{\gamma_{\boldsymbol{\eta}}(t)}
}}{1+e^{\gamma_{\boldsymbol{\eta}}(t)}} \cdot \frac{e^{\gamma_{\boldsymbol{\eta}}(t)}}{e^{\gamma_{\boldsymbol{\eta}}(t)}} = \frac{e^{\gamma_{\boldsymbol{\eta}}(t)}+1}{e^{\gamma_{\boldsymbol{\eta}}(t)}(1+e^{\gamma_{\boldsymbol{\eta}}(t)})}
\\[5pt] &
= \frac{1}{e^{\gamma_{\boldsymbol{\eta}}(t)}}, \label{eq: snrt}
\end{align}

which is equivalently expressed as $\mathrm{SNR}(t) = \mathrm{exp}(-\gamma_{\boldsymbol{\eta}}(t))$.

§ DISCUSSION

Despite the growing popularity of diffusion models, gaining a deep understanding of the model class remained somewhat elusive for the uninitiated in non-equilibrium statistical physics. With that in mind, we have presented a more straightforward introduction to diffusion models using directed graphical modelling and variational inference principles, which imposes relatively fewer prerequisites on the reader.

Our exposition began with a basic review of latent variable models like VAEs. We then reviewed their deep hierarchical counterparts and established a unifying graphical modelling-based perspective on their connection with diffusion models. We showed that diffusion models share a specific top-down latent variable hierarchy structure with ladder networks [58] and top-down inference HVAEs [48], which among other things explains why they share the same optimization objective.
Although introducing additional (auxiliary) latent variables significantly improves the flexibility of both the inference and generative models, it comes with additional challenges. We highlighted the difficulties with using purely bottom-up inference procedures in deep latent variable hierarchies, including posterior collapse for instance, whereby the posterior distribution (of the top-most layer, say) may collapse to a standard Gaussian prior, failing to learn meaningful representations and deactivating latent variables. Both [4] and [47] point to the asymmetry between the associated generative and inference models in HVAEs as a source of difficulty in training the inference model efficiently, since there is no way to express each term in the variational lower bound as an expectation under a distribution over a single latent variable. [28, 2] present a similar efficiency-based argument against using bottom-up inference in hierarchical latent variable models.

We claim that efficiency arguments paint an incomplete picture; the main reason one should avoid bottom-up inference in hierarchical latent variable models is the lack of direct feedback from the generative model. We argue that since the purpose of the inference model is to perform Bayesian inference at any given layer in the hierarchy, it stands to reason that interleaving feedback from each transition in the generative model into each respective transition in the inference model can only make both the inference and generative models more aligned and accurate. Although this rationale may not apply to diffusion models in quite the same way as it applies to HVAEs, because the inference model of the former is typically fixed, the top-down latent hierarchy structure is nonetheless ubiquitous. Moreover, since the top-down posterior in diffusion models is tractable by definition, and follows the same topological ordering of the latent variables as the generative model, it can be used to specify the generative model transitions by simply replacing the data in our conditioning set with a denoising model. This offers an intuitive view of diffusion models as a specific instantiation of ladder networks and/or HVAEs with top-down inference.

A major problem with VAEs and their hierarchical counterparts, which is not present in diffusion models, is the hole problem. The hole problem refers to the mismatch between the so-called aggregate posterior (i.e. simply the average posterior distribution over the dataset) and the prior over the latent variables. As shown in Figure <ref>, there can be regions with high probability density under the prior which have low probability density under the aggregate posterior. This then affects the quality of generated samples, as the decoder may be tasked with decoding latent variables sampled from regions not covered by the training data. Moreover, the higher the dimensionality of our input data, the less likely it is that our finite dataset covers the entirety of the input space. The manifold hypothesis posits that high-dimensional datasets lie along a much lower-dimensional latent manifold. However, since providing latent variable identifiability guarantees is very challenging for most interesting problems, in practice, we often resort to unfalsifiable assumptions about both the functional form and dimensionality of the latent space.

Diffusion models cleverly circumvent both of the aforementioned issues by: (i) defining the aggregate posterior to be equal to the prior by construction; and (ii) sacrificing the ability to learn reusable representations by fixing the posterior distribution according to a predefined noise schedule. The first point (i) ensures a smooth transition between the prior $p(\mathbf{z}_T)$ and the model $p(\mathbf{x} \mid \mathbf{z}_1)$, so we avoid sampling latent variables from regions of low density under the aggregate posterior $q(\mathbf{z}_T)$. The second point (ii) entails manually specifying how smooth this transition is by defining the latent variables $\mathbf{z}_{1:T}$ to simply be incrementally noisier versions of the input according to a judicious choice of noise schedule. This added noise can be interpreted as a kind of data augmentation technique, which helps smooth out the data density landscape and connect distant modes of the underlying data distribution. As explained in Section <ref>, the noise schedule is specified by parameters $\alpha_t$, $\sigma^2_t$ and in combination with a weighting function $w(\alpha^2_t/\sigma^2_t)$, stipulates the relative importance of each noise level in the diffusion objective. It turns out that the main difference between most diffusion model objectives boils down to the implied weighting function being used as a result [22, 23]. If our primary goal is high-quality sample generation, it suffices to adjust the noise schedule and weighting function such that the model focuses on perceptually important information and ignores imperceptible bits. Indeed, the majority of bits in images are allocated to imperceptible details and can in principle be ignored. Moreover, by encouraging the model to focus on some noise levels more than others, we are implicitly prescribing a preference for modelling low, mid, and/or high-frequency details at different noise levels. [11] found that although their diffusion models were not competitive with state-of-the-art likelihood-based models in terms of lossless codelengths, the samples were of high quality nonetheless. This demonstrates that diffusion models possess excellent inductive biases for image data.

Since the Markovian transitions between latent states $q(\mathbf{z}_t \mid \mathbf{z}_{t-1})$ are chosen to be linear Gaussian with isotropic covariances, the top-down posterior $q(\mathbf{z}_{t-1} \mid \mathbf{z}_{t}, \mathbf{x})$ is tractable through Gaussian conjugacy and the KL divergence terms in the associated VLB simplify significantly down to squared-error terms (Section <ref>). Given that the Gaussian diffusion process can be defined directly in terms of the conditionals $q(\mathbf{z}_t \mid \mathbf{x})$ (ref. Section <ref>), it is possible to: (i) train any level of the latent variable hierarchy independently of the others; (ii) share the same denoising model across the whole hierarchy. This constitutes a critical advantage over ladder networks and top-down HVAEs, as they both induce hierarchical dependencies between the latent states which prevent training individual layers independently. This advantage is particularly salient for infinitely deep latent variable hierarchies. As explained in detail in Section <ref>, diffusion models provide a principled framework for making the latent variable hierarchy infinitely deep. Such models are trained in continuous-time where $T \to \infty$, and can be shown to always improve the diffusion loss compared to hierarchical latent variable models like top-down HVAEs and discrete-time diffusion models [22] (ref. Section <ref>). It is also worth reiterating the ease with which it is possible to recast the denoising task in diffusion models in terms of noise prediction [11] rather than image prediction (see e.g. Section <ref>), that noise prediction seems to perform better in practice, and that the resulting setup has close connections to score-based generative modeling [18, 59, 50, 53].

The success of diffusion models can be partly attributed to an additional reduction in degrees of freedom compared to top-down HVAEs. In VAEs, several simplifying assumptions are made to ensure the inference problem is both tractable and scalable: (i) amortized variational inference; (ii) mean-field variational family assumption; (iii) assumed parametric distributions for both the prior and likelihood; (iv) stochastic optimization of a Monte Carlo estimator of the evidence lower bound. Given the close connection between top-down HVAEs and diffusion models established in Section <ref>, we can see that comparatively speaking diffusion models constitute yet another simplifying assumption by fixing the inference distribution to follow a pre-defined noise schedule. This transforms the learning problem from involving the minimization of the reverse KL divergence to improve our posterior approximation: $\argmin_{q \in \mathcal{Q}}D_{\mathrm{KL}}( q(\mathbf{z}_{1:T} \mid \mathbf{x}) \parallel p(\mathbf{z}_{1:T}))$, where the prior $p$ may be fixed, to minimization of the forward KL divergence: $\argmin_{p \in \mathcal{P}} D_{\mathrm{KL}}( q(\mathbf{z}_{1:T} \mid \mathbf{x}) \parallel p(\mathbf{z}_{1:T}))$, where the posterior $q$ is fixed:

\begin{align}
\argmin_{p \in \mathcal{P}} D_{\mathrm{KL}}( q(\mathbf{z}_{1:T} \mid \mathbf{x}) \parallel p(\mathbf{z}_{1:T}))
& = \argmin_{p \in \mathcal{P}} \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \left[-\log p(\mathbf{z}_{1:T}) \right] - \mathcal{H}(q(\mathbf{z}_{1:T} \mid \mathbf{x}))
\\[0pt] & = \argmax_{p \in \mathcal{P}} \mathbb{E}_{q(\mathbf{z}_{1:T} \mid \mathbf{x})} \left[\log p(\mathbf{z}_{1:T}) \right] + c, \label{eq: erm}
\end{align}

which essentially amounts to a supervised learning problem with noise-augmented i.i.d. data, optimized via maximum likelihood. Indeed, if all we care about is image synthesis quality, then it is intuitively advantageous to sacrifice the ability to learn reusable representations by fixing the posterior $q$ and focusing on leveraging the tried-and-tested machinery of supervised learning to train a good generative model. This rationale also motivated ladder networks [58]. Purely unsupervised learning methods try to represent all the information about $p(\mathbf{x})$, which includes imperceptible details and complicates the learning problem. Conversely, supervised learning is effective at filtering out unnecessary information for the task at hand, which in combination with carefully weighted diffusion objectives, explains how/why diffusion models are capable of high-quality image synthesis that better aligns with human perception.

In Section <ref>, we provided a deeper understanding of the various weighted diffusion objectives in literature. By further analyzing the objective in Equation <ref>, it is possible to show that the diffusion loss is equivalent to the ELBO under simple data augmentation using Gaussian additive noise [23], so long as the weighting function $w(\lambda_t)$ is monotonically increasing w.r.t. time $t$. We showed that if $w$ is a valid CDF, then we can define a valid probability distribution $p_w(t)$ specified by the weighting function, which acts as a data augmentation kernel and dictates the importance of different noise levels as outlined by [23]. However, multiple works report impressive image synthesis results using non-monotonic weighting functions [34, 21], which somewhat peculiarly implies that the ELBO is being minimized at certain noise levels. This seems to reaffirm the widespread belief that maximum likelihood may not be the appropriate objective to use for high-quality sample generation. However, [23] showed that likelihood maximization need not be intrinsically at odds with high-quality image synthesis, as they achieved state-of-the-art FID scores on the high-resolution ImageNet benchmark by optimizing the ELBO (under data augmentation).

We further argue that reconnecting weighted diffusion objectives with maximum likelihood (i.e. the ELBO) is particularly important because we know from Shannon's source coding theorem that the average codelength of the optimal compression scheme is the entropy of the data $\mathbb{\mathcal{H}}(X) = \mathbb{E}[-\log p(\mathbf{x})]$. Thus, as long as we are minimizing codelengths given by the information content $-\log p_{\boldsymbol{\theta}}(\mathbf{x})$ defined by a probabilistic model $p_{\boldsymbol{\theta}}$ (e.g. by maximizing likelihood), then the resulting average codelength $\mathbb{E}[-\log p_{\boldsymbol{\theta}}(\mathbf{x})]$ approaches the entropy of the true data distribution. This is the fundamental goal of generative modelling and compression, a goal with which maximum likelihood is well aligned.

To conclude, the success of diffusion models is arguably as much a product of collective engineering effort and scale as it is a product of algorithmic and theoretical insight. Nonetheless, identifying analogies between model classes undoubtedly aids in understanding, and recognizing the unique properties of specific models helps refine our intuitions about what may or may not work in the future. Fertile ground for future work includes enabling diffusion models to learn semantic latent representations, expanding forward diffusion processes beyond linear Gaussian transitions, and leveraging the identifiability guarantees of probability flow ODEs [53] for causal representation learning and inference.

Acknowledgements. We would like to thank Charles Jones, Rajat Rasal and Avinash Kori for fruitful discussions and valuable feedback. The authors are supported with funding from the ERC under the EU’s Horizon 2020 research and innovation programme (Project MIRA, grant No. 757173).

[1]
Anderson, B. D. (1982).
Reverse-time diffusion equation models.
Stochastic Processes and their Applications, 12(3):313–326.

[2]
Bishop, C. M. and Bishop, H. (2023).
Deep Learning: Foundations and Concepts.

[3]
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. (2017).
Variational inference: A review for statisticians.
Journal of the American statistical Association,

[4]
Burda, Y., Grosse, R., and Salakhutdinov, R. (2015).
Importance weighted autoencoders.
arXiv preprint arXiv:1509.00519.

[5]
Child, R. (2020).
Very deep vaes generalize autoregressive models and can outperform
them on images.
In International Conference on Learning Representations.

[6]
Choi, J., Lee, J., Shin, C., Kim, S., Kim, H., and Yoon, S. (2022).
Perception prioritized training of diffusion models.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 11472–11481.

[7]
De Sousa Ribeiro, F., Xia, T., Monteiro, M., Pawlowski, N., and Glocker, B.
High fidelity image counterfactuals with probabilistic causal models.
In Proceedings of the 40th International Conference on Machine
Learning, volume 202 of Proceedings of Machine Learning Research,
pages 7390–7425. PMLR.

[8]
Dhariwal, P. and Nichol, A. (2021).
Diffusion models beat gans on image synthesis.
Advances in neural information processing systems,

[9]
Hang, T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., Geng, X., and Guo, B.
Efficient diffusion training via min-snr weighting strategy.
arXiv preprint arXiv:2303.09556.

[10]
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S.
Gans trained by a two time-scale update rule converge to a local nash
Advances in neural information processing systems, 30.

[11]
Ho, J., Jain, A., and Abbeel, P. (2020).
Denoising diffusion probabilistic models.
Advances in neural information processing systems,

[12]
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T.
Cascaded diffusion models for high fidelity image generation.
The Journal of Machine Learning Research, 23(1):2249–2281.

[13]
Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, J. (2013).
Stochastic variational inference.
Journal of Machine Learning Research.

[14]
Hoffman, M. D. and Johnson, M. J. (2016).
Elbo surgery: yet another way to carve up the variational evidence
lower bound.
In Workshop in Advances in Approximate Bayesian Inference,
NIPS, volume 1.

[15]
Hoogeboom, E., Heek, J., and Salimans, T. (2023).
simple diffusion: End-to-end diffusion for high resolution images.
In Proceedings of the 40th International Conference on Machine
Learning, volume 202 of Proceedings of Machine Learning Research,
pages 13213–13232. PMLR.

[16]
Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M. (2022).
Equivariant diffusion for molecule generation in 3d.
In International conference on machine learning, pages
8867–8887. PMLR.

[17]
Huang, C.-W., Lim, J. H., and Courville, A. C. (2021).
A variational perspective on diffusion-based generative models and
score matching.
Advances in Neural Information Processing Systems,

[18]
Hyvärinen, A. and Dayan, P. (2005).
Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 6(4).

[19]
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. (1999).
An introduction to variational methods for graphical models.
Machine learning, 37:183–233.

[20]
Jun, H., Child, R., Chen, M., Schulman, J., Ramesh, A., Radford, A., and
Sutskever, I. (2020).
Distribution augmentation for generative modeling.
In International Conference on Machine Learning, pages
5006–5019. PMLR.

[21]
Karras, T., Aittala, M., Aila, T., and Laine, S. (2022).
Elucidating the design space of diffusion-based generative models.
Advances in Neural Information Processing Systems,

[22]
Kingma, D., Salimans, T., Poole, B., and Ho, J. (2021).
Variational diffusion models.
Advances in neural information processing systems,

[23]
Kingma, D. P. and Gao, R. (2023).
Understanding the diffusion objective as a weighted integral of
arXiv preprint arXiv:2303.00848.

[24]
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and
Welling, M. (2016).
Improved variational inference with inverse autoregressive flow.
Advances in neural information processing systems, 29.

[25]
Kingma, D. P. and Welling, M. (2013).
Auto-encoding variational bayes.
arXiv preprint arXiv:1312.6114.

[26]
Kingma, D. P., Welling, M., et al. (2019).
An introduction to variational autoencoders.
Foundations and Trends® in Machine Learning,

[27]
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. (2023).
Flow matching for generative modeling.
In The Eleventh International Conference on Learning

[28]
Luo, C. (2022).
Understanding diffusion models: A unified perspective.
arXiv preprint arXiv:2208.11970.

[29]
Maaløe, L., Fraccaro, M., Liévin, V., and Winther, O. (2019).
Biva: A very deep hierarchy of latent variables for generative
Advances in neural information processing systems, 32.

[30]
Maaløe, L., Sønderby, C. K., Sønderby, S. K., and Winther, O. (2016).
Auxiliary deep generative models.
In International conference on machine learning, pages
1445–1453. PMLR.

[31]
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., and Frey, B. (2015).
Adversarial autoencoders.
arXiv preprint arXiv:1511.05644.

[32]
Meng, C., Song, J., Song, Y., Zhao, S., and Ermon, S. (2020).
Improved autoregressive modeling with distribution smoothing.
In International Conference on Learning Representations.

[33]
Monteiro, M., Ribeiro, F. D. S., Pawlowski, N., Castro, D. C., and Glocker, B.
Measuring axiomatic soundness of counterfactual image models.
In The Eleventh International Conference on Learning

[34]
Nichol, A. Q. and Dhariwal, P. (2021).
Improved denoising diffusion probabilistic models.
In International Conference on Machine Learning, pages
8162–8171. PMLR.

[35]
Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B.,
Sutskever, I., and Chen, M. (2022).
Glide: Towards photorealistic image generation and editing with
text-guided diffusion models.
In International Conference on Machine Learning, pages
16784–16804. PMLR.

[36]
Ranganath, R., Tran, D., and Blei, D. (2016).
Hierarchical variational models.
In International conference on machine learning, pages
324–333. PMLR.

[37]
Rasmus, A., Berglund, M., Honkala, M., Valpola, H., and Raiko, T. (2015).
Semi-supervised learning with ladder networks.
Advances in neural information processing systems, 28.

[38]
Rezende, D. J., Mohamed, S., and Wierstra, D. (2014).
Stochastic backpropagation and approximate inference in deep
generative models.
In International conference on machine learning, pages
1278–1286. PMLR.

[39]
Rezende, D. J. and Viola, F. (2018).
Taming vaes.
arXiv preprint arXiv:1810.00597.

[40]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022).
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 10684–10695.

[41]
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L.,
Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.
Photorealistic text-to-image diffusion models with deep language
Advances in Neural Information Processing Systems,

[42]
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen,
X. (2016).
Improved techniques for training gans.
Advances in neural information processing systems, 29.

[43]
Salimans, T. and Ho, J. (2021).
Should EBMs model the energy or the score?
In Energy Based Models Workshop - ICLR 2021.

[44]
Salimans, T. and Ho, J. (2022).
Progressive distillation for fast sampling of diffusion models.
In International Conference on Learning Representations.

[45]
Salimans, T., Kingma, D., and Welling, M. (2015).
Markov chain monte carlo and variational inference: Bridging the gap.
In International conference on machine learning, pages
1218–1226. PMLR.

[46]
Shu, R. and Ermon, S. (2022).
Bit prioritization in variational autoencoders via progressive
In International Conference on Machine Learning, pages
20141–20155. PMLR.

[47]
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015).
Deep unsupervised learning using nonequilibrium thermodynamics.
In International conference on machine learning, pages
2256–2265. PMLR.

[48]
Sønderby, C. K., Raiko, T., Maaløe, L., Sønderby, S. K., and Winther,
O. (2016).
Ladder variational autoencoders.
Advances in neural information processing systems, 29.

[49]
Song, Y., Durkan, C., Murray, I., and Ermon, S. (2021a).
Maximum likelihood training of score-based diffusion models.
Advances in Neural Information Processing Systems,

[50]
Song, Y. and Ermon, S. (2019).
Generative modeling by estimating gradients of the data distribution.
Advances in neural information processing systems, 32.

[51]
Song, Y. and Ermon, S. (2020).
Improved techniques for training score-based generative models.
Advances in neural information processing systems,

[52]
Song, Y. and Kingma, D. P. (2021).
How to train your energy-based models.
arXiv preprint arXiv:2101.03288.

[53]
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole,
B. (2021b).
Score-based generative modeling through stochastic differential
In International Conference on Learning Representations.

[54]
Tomczak, J. and Welling, M. (2018).
Vae with a vampprior.
In International Conference on Artificial Intelligence and
Statistics, pages 1214–1223. PMLR.

[55]
Tzen, B. and Raginsky, M. (2019).
Neural stochastic differential equations: Deep latent gaussian models
in the diffusion limit.
arXiv preprint arXiv:1905.09883.

[56]
Vahdat, A. and Kautz, J. (2020).
Nvae: A deep hierarchical variational autoencoder.
Advances in Neural Information Processing Systems,

[57]
Vahdat, A., Kreis, K., and Kautz, J. (2021).
Score-based generative modeling in latent space.
Advances in Neural Information Processing Systems,

[58]
Valpola, H. (2015).
From neural pca to deep unsupervised learning.
In Advances in independent component analysis and learning
machines, pages 143–171. Elsevier.

[59]
Vincent, P. (2011).
A connection between score matching and denoising autoencoders.
Neural computation, 23(7):1661–1674.

[60]
Wasserman, L. (2004).
All of statistics: a concise course in statistical inference,
volume 26.