# Reward Guided Latent Consistency Distillation

Jiachen Li1, Weixi Feng1, Wenhu Chen2, William Yang Wang1
1UC Santa Barbara, 2University of Waterloo
1{jiachen_li, weixifeng<EMAIL_ADDRESS><EMAIL_ADDRESS>

Project Page: https://rg-lcd.github.io

###### Abstract

Latent Consistency Distillation (LCD) has emerged as a promising paradigm for
efficient text-to-image synthesis. By distilling a latent consistency model
(LCM) from a pre-trained teacher latent diffusion model (LDM), LCD facilitates
the generation of high-fidelity images within merely 2 to 4 inference steps.
However, the LCM’s efficient inference is obtained at the cost of the sample
quality. In this paper, we propose compensating the quality loss by aligning
LCM’s output with human preference during training. Specifically, we introduce
Reward Guided LCD (RG-LCD), which integrates feedback from a reward model (RM)
into the LCD process by augmenting the original LCD loss with the objective of
maximizing the reward associated with LCM’s single-step generation. As
validated through human evaluation, when trained with the feedback of a good
RM, the 2-step generations from our RG-LCM are favored by humans over the
50-step DDIM [52] samples from the teacher LDM, representing a 25 times
inference acceleration without quality loss.

As directly optimizing towards differentiable RMs can suffer from over-
optimization, we overcome this difficulty by proposing the use of a latent
proxy RM (LRM). This novel component serves as an intermediary, connecting our
LCM with the RM. Empirically, we demonstrate that incorporating the LRM into
our RG-LCD successfully avoids high-frequency noise in the generated images,
contributing to both improved FID on MS-COCO [29] and a higher HPSv2.1 score
on HPSv2 [64]’s test set, surpassing those achieved by the baseline LCM.

Figure 1: Even with merely 2-4 sampling steps, our RG-LCMs that learned from
the CLIP Score and HPSv2.1 can produce high-quality images.

##  1 Introduction

In the realm of modern generative AI (GenAI) models, computational resources
are typically allocated across three key areas: pretraining [5, 1, 28, 41, 43,
44, 3], alignment [73, 57, 39, 18, 7, 42], and inference [69, 13, 60, 49].
Normally, increasing the computational budget across these areas leads to
improvements in sample quality. For instance, the most advanced text-to-image
(T2I) models, such as DALLE-3 [3], ImageGen [44], and Stable Diffusion [43]
are built from diffusion models (DMs) [51, 17, 54]. These models are
pretrained on massive web-scale datasets [47, 6], aligned with human
preference on curated high-quality images [8, 43], and benefit from the
iterative sampling process of DMs.

However, DM’s iterative sampling requires performing 10 - 2000 sequential
function evaluations (FEs) [17, 52], thus impeding rapid inference. While
there have been many works proposed to address this issue [30, 31, 68, 46, 14,
37, 56], consistency model (CM) [56] emerges as a new family of GenAI model
that facilitates fast sampling. A CM is trained to perform single-step
generation while supporting multi-step sampling to trade compute for sample
quality. We can distill a CM from a pretrained DM, a process known as
consistency distillation (CD). For instance, Luo et al. distill a Latent CM
(LCM) [35] from a pretrained Stable Diffusion [43], achieving high-fidelity
image generation in just 2 to 4 FE steps. However, the sample quality of LCM
is inherently constrained by the pretrained LDM’s capabilities [53].
Additionally, the reduced inference computational resources stemming from the
limited number of FE steps compromise LCM’s sample quality.

In this paper, we aim to offset LCM’s sample quality by dedicating additional
computational resources to the LCD process. As demonstrated by the recent
success of large language models [2, 59], aligning a GenAI model to a reward
model (RM) that mirrors human preferences dramatically improves its sample
quality [39, 42]. We are thus motivated to integrate feedback from a
differentiable RM into the LCD process. Facilitated by the fact that the LCD
loss is calculated based on a single-step generation from the LCM, we propose
to simultaneously train the LCM to maximize the reward associated with the
generation, obviating the need for backpropagating gradients through the
extensive denoising steps typically required in optimizing a DM [7, 66]. We
dub our method _Reward Guided Latent Consistency Distillation_ (RG-LCD). As
evidenced by human evaluation, our RG-LCM significantly outperforms the LCM
derived from standard LCD methods. Moreover, it even exceeds the capabilities
of the pretrained LDM, which generates images using 50 DDIM steps, with just 2
FE steps, achieving a 25-fold increase in inference speed without compromising
image quality.

While our RG-LCD is conceptually simple and already achieves impressive
results, it can suffer from reward overestimation [22, 70] due to direct
optimization with the gradient from the RM. As shown in the top row of Fig. 3,
performing RG-LCD with ImageReward [66] causes high-frequency noise in the
generated images. To tackle this challenge, we propose learning a latent proxy
RM to serve as the intermediary that connects our LCM with the RM. Instead of
directly optimizing towards the RM, we optimize the LCM towards the LRM while
finetuning the LRM to match the preference of the expert RM in each RG-LCD
iteration. This novel strategy allows us to optimize the expert RM indirectly,
even allowing for learning from non-differentiable RMs. We empirically verify
that incorporating the LRM into our RG-LCD successfully eliminates the high-
frequency noise in the generated image, contributing to improved FID on MS-
COCO [29] and a higher HPSv2.1 score on HPSv2’s test set [64], outperforming
the baseline LCM.

In summary, our contributions are threefold:

  * •

Introduction of RG-LCD framework, which incorporates feedback from an RM that
mirrors human preference into the LCD process.

  * •

Introduction of the LRM, which enables indirect optimization towards the RM,
mitigating the issue of reward over-optimization.

  * •

A 25 times inference acceleration over teacher LDM (Stable Diffusion v2.1)
without compromising sample quality.

##  2 Related Work

Accelerating DM inference. Centering around DM’s SDE formulation [55], various
methods have been proposed to accelerate the sampling process of a DM. For
example, faster numerical ODE solvers [52, 30, 31, 71, 10, 20] and
distillation techniques [34, 45, 36, 72]. Recent advances explore enhancing
the single-step generation quality by incorporating an adversarial loss [46]
or by distillation [37]. Consistency Model [56] is also trained for single-
step generation. We leverage this property and directly maximize the reward of
this single-step generation given by a differentiable RM.

Consistency Model has emerged as a new family of GenAI model [56] that
facilitates fast inference. While it is trained to perform single-step
generation by mapping arbitrary points in the PF-ODE trajectory to the origin,
CM also supports multi-step sampling, allowing for trading compute for better
sample quality. A CM can be trained as a standalone GenAI model (consistency
training). Recently, Song et al. proposed improved techniques [53] to support
better consistency training. A CM can also be distilled from a pretrained DM.
For instance, Luo et al. [35] learn an LCM by distilling from a pretrained
Stable Diffusion [43]. We defer more technical details to Sec. 3.

Vision-and-language reward models. Motivated by the significant success of
reinforcement from human feedback (RLHF), there have been many works delving
into training an RM to mirror human preferences on a pair of text and image,
including HPSv2 [64], ImageReward [66], PickScore [24]. These RMs are normally
derived by finetuning a vision and language model, e.g., CLIP [41] and BLIP
[27], on human preference data. Since these RMs are differentiable, our RG-LCD
augments the standard LCD with the objective of maximizing the differentiable
reward associated with its single-step during training.

Aligning DMs to Human preference has been extensively studied recently,
including RL based methods [12, 40, 70] and reward finetuning methods [7, 66].
Recently, Diffusion-DPO [62] is proposed by extending DPO [42] to train DMs on
preference data. Moreover, other works focus on modifying the training data
distribution [65, 26, 11, 58, 8] to finetune DMs on visually appealing and
textually cohered data. Additionally, alternative techniques [4, 48] re-
caption pre-collected web images to enhance text accuracy. On the other hand,
DOODL [63] is proposed to optimize the RM during inference time. However, its
improvement is made at the cost of inference speed. While we also propose to
directly finetune our model with the gradient given by an RM, finetuning an
LCM during LCD is much simpler than finetuning a DM, as we only tackle the
single-step generation, circumventing the need to pass gradients through DM’s
complicated iterative sampling process.

##  3 Background

###  3.1 Diffusion Model

Diffusion models (DMs) [51, 54, 17, 38] progressively inject Gaussian noise
into data in the forward process and sequentially denoise the data to create
samples in the reverse denoising process. The forward process perturbs the
original data distribution $p_{data}(\mathbf{x})\equiv p_{0}(\mathbf{x}_{0})$
to the marginal distributional $p_{t}(\mathbf{x}_{t})$. From a continuous-time
perspective, we can represent the forward process with a stochastic
differential equation (SDE) [55, 21]

$\mathrm{d}\mathbf{x}_{t}=\boldsymbol{\mu}(t)\mathbf{x}_{t}\mathrm{d}t+\sigma(t)\mathrm{d}\mathbf{w}_{t},\quad\mathbf{x}_{0}\sim
p_{\text{data }}\left(\mathbf{x}_{0}\right),$ (1)

where $\boldsymbol{\mu}(\cdot)$ and $\mu(\cdot)$ are the drift and diffusion
coefficients respectively, and $\mathbf{w}_{t}$ denotes the standard Wiener
process. The reverse time SDE above corresponds to an ordinary differential
equation (ODE)[55], named Probability Flow (PF-ODE), which is given by

$\mathrm{d}\mathbf{x}_{t}=\left[\boldsymbol{\mu}\left(t\right)\mathbf{x}_{t}-\frac{1}{2}\sigma(t)^{2}\nabla\log
p_{t}\left(\mathbf{x}_{t}\right)\right]\mathrm{d}t,\quad\mathbf{x}_{T}\sim
p_{T}(\mathbf{x}_{T}).$ (2)

PF-ODE’s solution trajectories sampled at $t$ are distributed the same as
$p_{t}(\mathbf{x}_{t})$. Empirically, we learn a denoising model
$\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ to fit $-\nabla\log
p_{t}(\mathbf{x}_{t})$ (score function) via score matching [19, 54, 17].
During sampling, we start from the sample
$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\tilde{\sigma}^{2}\mathbf{I})$ and
follow the empirical PF-ODE below

$\mathrm{d}\mathbf{x}_{t}=\left[\boldsymbol{\mu}\left(t\right)\mathbf{x}_{t}+\frac{1}{2}\sigma(t)^{2}\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},t)\right]\mathrm{d}t,\quad\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\tilde{\sigma}^{2}\mathbf{I}).$
(3)

In this paper, we focus on conditional LDM that operates on the image latent
space $\mathcal{Z}$ and includes a text prompt $\mathbf{c}$ passed to the
denoising model $\boldsymbol{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c},t)$,
where $\mathbf{z}_{t}=\mathcal{E}(\mathbf{x}_{t})\in\mathcal{Z}$ is encoded by
a VAE [23] encoder $\mathcal{E}$. Moreover, we utilize Classifier-Free
Guidance (CFG) [16] to improve the quality of conditional sampling by
replacing the noise prediction with a linear combination of conditional and
unconditional noise prediction for denoising, i.e.,
$\tilde{\boldsymbol{\epsilon}}_{\theta}\left(\mathbf{z}_{t},\omega,\mathbf{c},t\right)=(1+\omega)\boldsymbol{\epsilon}_{\theta}\left(\mathbf{z}_{t},\boldsymbol{c},t\right)-\omega\boldsymbol{\epsilon}_{\theta}(\mathbf{z},\varnothing,t)$,
where $\omega$ is the CFG scale.

###  3.2 Consistency Model

Consistency model (CM) [56] is proposed to facilitate efficient generation. At
its core, CM learns a consistency function
$\boldsymbol{f}:\left(\mathbf{x}_{t},t\right)\mapsto\mathbf{x}_{\epsilon}$
that can map any point $\mathbf{x}_{t}$ on the same PF-ODE trajectory to the
trajectory’s origin, where $\epsilon$ is a fixed small positive number.
Learning the consistency function involves enforcing the _self-consistency_
property

$\boldsymbol{f}(\mathbf{x}_{t},t)=\boldsymbol{f}(\mathbf{x}_{t}^{\prime},t^{\prime}),\forall
t,t^{\prime}\in[\epsilon,T],$ (4)

where $\mathbf{x}_{t}$ and $\mathbf{x}_{t}^{\prime}$ belong to the same PF-
ODE. The consistency function $\boldsymbol{f}$ is modeled with a CM
$\boldsymbol{f}_{\theta}$. To ensure
$\boldsymbol{f}_{\theta}(\mathbf{x},\epsilon)=\mathbf{x}$,
$\boldsymbol{f}_{\theta}$ is parameteried as

$\boldsymbol{f}_{\theta}(\mathbf{x},t)=c_{\text{skip}}(t)\mathbf{x}+c_{\text{out}}(t)F_{\theta}(\mathbf{x},t),$
(5)

where $c_{\text{skip}}(t)$ and $c_{\text{out}}(t)$ are differentiable
functions with $c_{\text{skip}}(\epsilon)=1$ and $c_{\text{out}}(\epsilon)=0$,
and $F_{\theta}$ is a neural network. We can learn a CM
$\boldsymbol{f}_{\theta}$ by distilling from a pretrained DM, known as
_consistency distillation_ (CD) [56]. The CD loss is given by

$L_{\text{CD}}\left(\theta,\theta^{-};\Phi\right)=\mathbb{E}_{\mathbf{x},t}\left[d\left(\boldsymbol{f}_{\theta}\left(\mathbf{x}_{t_{n+1}},t_{n+1}\right),\boldsymbol{f}_{\theta^{-}}\left(\hat{\mathbf{x}}_{t_{n}}^{\phi},t_{n}\right)\right)\right].$
(6)

where $d(\cdot,\cdot)$ measures the distance between two samples. $\theta^{-}$
is the parameter of the target CM $\boldsymbol{f}_{\theta^{-}}$, updated by
the exponential moving average (EMA) of $\theta$, i.e.,
$\theta^{-}\leftarrow\texttt{stop\\_grad}\left(\mu\theta+(1-\mu)\theta^{-}\right)$.
$\hat{\mathbf{x}}_{t_{n}}^{\phi}$ is an estimation of $\mathbf{x}_{t_{n}}$
from $\mathbf{x}_{t_{n+1}}$ using the one-step ODE solver $\Phi$:

$\hat{\mathbf{x}}_{t_{n}}^{\phi}\leftarrow\mathbf{x}_{t_{n+1}}+\left(t_{n}-t_{n+1}\right)\Phi\left(\mathbf{x}_{t_{n+1}},t_{n+1};\phi\right).$
(7)

Note that the parameter $\phi$ corresponds to the parameter of the pretrained
DM, which is used to construct the ODE solver $\Phi$. We includes the
algorithm for sampling from a CM in Appendix B.

###  3.3 Latent Consistency Model

Luo et al. [35] extends CM to work on latent space $\mathcal{Z}$ and focuses
on conditional generation. Specifically, a Latent CM (LCM)
$\boldsymbol{f}_{\theta}:\left(\mathbf{z}_{\boldsymbol{t}},\omega,\boldsymbol{c},t\right)\mapsto\mathbf{z}_{\mathbf{0}}$
is trained to minimized the LCD loss

$L_{\text{LCD}}\left(\theta,\theta^{-};\Psi\right)=\mathbb{E}_{\mathbf{z},\mathbf{c},\omega,n}\left[d\left(\boldsymbol{f}_{\theta}\left(\mathbf{z}_{t_{n+k}},\omega,\mathbf{c},t_{n+k}\right),\boldsymbol{f}_{\theta^{-}}\left(\hat{\mathbf{z}}_{t_{n}}^{\Psi,\omega},\omega,\mathbf{c},t_{n}\right)\right)\right],$
(8)

where $\hat{\mathbf{z}}_{t_{n}}^{\Psi,\omega}$ is an estimate of $z_{t_{n}}$
obtained by the numerical augmented PF-ODE solver $\Psi$ and $k$ is skipping
interval.

$\hat{\mathbf{z}}_{t_{n}}^{\Psi,\omega}\leftarrow\mathbf{z}_{t_{n+k}}+(1+\omega)\Psi(\mathbf{z}_{t_{n+k}},t_{n+k},t_{n},c;\psi)-\omega\Psi(\mathbf{z}_{t_{n}+k},t_{n+k},t_{n},\varnothing;\psi).$
(9)

In this paper, we use DDIM [52] as the ODE solver $\Psi$ to distill from a
pretrained Stable Diffusion [43] and refer interested readers to the original
LCM paper for formula of the DDIM solver. We use huber loss as our distance
function $d(\cdot,\cdot)$.

Figure 2: Overview of our RG-LCD. We integrate feedback from a differentiable
RM into the standard LCD procedures by training the LCM to maximize the reward
associated with its single-step generation.

##  4 Reward Guided Latent Consistency Distillation

In this section, we start by presenting the core components of our RG-LCD
framework, which augments the standard LCD loss (8) with an objective towards
maximizing a differentiable RM, as shown in Fig. 2 (Sec. 4.1). We then
motivate the development of a latent proxy RM (LRM) to support indirect RM
optimization by illustrating the risk of suffering from reward over-
optimization when directly optimizing towards the RM with a gradient-based
method. Following this, we then detail the procedure to pretrain and finetune
the LRM to match the preference of the RGB-based RM during RG-LCD (Sec. 4.2).

###  4.1 RG-LCD with Differentiable RMs

Recall that each LCD iteration samples a timestep $t_{n+k}$, and construct the
noisy latent $\mathbf{z}_{t_{n+k}}$ by perturb the image latent
$\mathbf{z}=\mathcal{E}(\mathbf{x})$ with a Gaussian noise, given a sampled
CFG scale $\omega$ and text prompt $\mathbf{c}$. As the LCM
$\boldsymbol{f}_{\theta}$ maps the $\mathbf{z}_{t_{n+k}}$ to the PF-ODE origin
$\hat{\mathbf{z}}_{0}=\boldsymbol{f}_{\theta}\left(\mathbf{z}_{t_{n+k}},\omega,\mathbf{c},t_{n+k}\right)$,
we construct the following objective to maximize the reward associated with
$\mathcal{D}(\hat{\mathbf{z}_{0}})$

$J(\theta)=\mathbb{E}_{\mathbf{z},\mathbf{c},\omega,n}\left[\mathcal{R}\left(\mathcal{D}\left(\boldsymbol{f}_{\theta}\left(\mathbf{z}_{t_{n+k}},\omega,\mathbf{c},t_{n+k}\right)\right),\mathbf{c}\right)\right],$
(10)

where $\mathcal{R}$ is a differentiable RM that calculates the rewards
associated with a pair of text and image. We define the training loss of our
RG-LCD by a linear combination of the LCD loss in (8) and $J(\theta)$ with a
weighting parameter $\beta$

$L_{\text{RG-
LCD}}\left(\theta,\theta^{-};\Psi\right)=L_{\text{LCD}}\left(\theta,\theta^{-};\Psi\right)-\beta
J(\theta)$ (11)

Appendix B includes pseudo-codes for our RG-LCD training.

###  4.2 RG-LCD with a Latent Proxy RM

Figure 3: (Top) Optimizing the RG-LCM with the gradient from ImageReward [66]
results in high-frequency noise in the generated images. (Bottom) Indirectly
optimizing the ImageReward through the latent proxy RM eliminates the high-
frequency noise, avoiding reward over-optimization.

When training the LCM $\boldsymbol{f}_{\theta}$ towards $J(\theta)$ with a
gradient-based method, we may suffer from the issue of reward over-
optimization. As shown in the top row of Fig. 3, performing RG-LCD with
ImageReward [66] causes high-frequency noise in the generated images. To
mitigate this issue, we propose learning a latent proxy RM
$\mathcal{R}^{\text{L}}_{\sigma}$ to serve as an intermediary to connect
$\boldsymbol{f}_{\theta}$ and the RGB-based RM $\mathcal{R}^{\text{E}}$.
Specifically, we train $\boldsymbol{f}_{\theta}$ to optimize the reward given
by $\mathcal{R}^{\text{L}}_{\sigma}$ while simultaneously finetuning the
$\mathcal{R}^{\text{L}}_{\sigma}$ to matches the preference given by the
expert RM $\mathcal{R}^{\text{E}}$ that process RGB images.

Ideally, the LRM $\mathcal{R}^{\text{L}}_{\sigma}$ should be capable of
accessing the text-image pair even at the beginning of RG-LCD. We thus
initialize $\mathcal{R}^{\text{L}}_{\sigma}$ with a pretrained CLIP [41] text
encoder, complemented by pretraining its latent encoder from scratch. This
latent encoder is pretrained following the same methodology used for CLIP
visual encoders, ensuring it aligns effectively with the text encoder’s
representation.

After pretraining, we finetune $\mathcal{R}^{\text{L}}_{\sigma}$ to match the
preference of $\mathcal{R}^{\text{E}}$. Note that we do not need to assume a
differentiable $\mathcal{R}^{\text{E}}$ anymore, allowing us to learn from the
feedback from a wider range of RGB-based RM, e.g., LLMScore [33], VIEScore
[25] and DA-score [50]. Given $\mathbf{z}_{0}=\mathbf{z}$,
$\mathbf{z}_{1}=\boldsymbol{f}_{\theta}\left(\mathbf{z}_{t_{n+k}},\omega,\mathbf{c},t_{n+k}\right)$,
and
$\mathbf{z}_{2}=\boldsymbol{f}_{\theta^{-}}\left(\mathbf{z}_{t_{n}},\omega,\mathbf{c},t_{n}\right)$
in each RG-LCD iteration, we train the the LRM
$\mathcal{R}_{\sigma}^{\text{L}}$ to match the $\mathcal{R}^{\text{E}}$’s
preference by

$\begin{aligned}
&L_{\text{RM}}(\sigma)=\mathbb{E}_{\mathbf{z},\mathbf{c},\omega,n}\left[\sum^{1}_{i=0}\sum^{2}_{j=i+1}D_{\text{KL}}\left(P^{\sigma}_{i,j}||\texttt{stop\\_grad}(Q_{i,j)}\right)\right],\,\,\text{where}\\\
&P^{\sigma}_{i,j}(k)=\frac{s^{\text{L}}_{\sigma}(\mathbf{z}_{k},\mathbf{c})}{s^{\text{L}}_{\sigma}(\mathbf{z}_{i},\mathbf{c})+s^{\text{L}}_{\sigma}(\mathbf{z}_{j},\mathbf{c})},Q_{i,j}(k)=\frac{s^{\text{E}}(\mathbf{z}_{k},\mathbf{c})}{s^{\text{E}}(\mathbf{z}_{i},\mathbf{c})+s^{\text{E}}(\mathbf{z}_{j},\textbf{c})},k\in\\{i,j\\}\\\
&s^{\text{L}}_{\sigma}(\mathbf{z},\mathbf{c})=\exp\left(\mathcal{R}^{\text{L}}_{\sigma}\left(\mathbf{z},\mathbf{c}\right)/{\tau_{L}}\right),s^{\text{E}}(\mathbf{z},\mathbf{c})=\exp\left({\mathcal{R}^{\text{E}}\left(\mathcal{D}\left(\mathbf{z}\right),\mathbf{c}\right)}/{\tau_{E}}\right)\end{aligned}.$
(12)

$\tau_{L}$ and $\tau_{E}$ are temperature parameters. $L_{\text{RM}}(\sigma)$
also supports matching a $\mathcal{R}^{\text{E}}$ that only output preference
over two images. In this case, we can set ${\tau_{E}}$ to a small positive
number and only give a non-zero positive reward to the sample favored by the
expert. Moreover, since $\mathbf{z}_{0}=\mathbf{z}$ corresponds to the latent
of a real image, we can adjust $Q_{0,j}$ to prefer $k=0$ with a increased
likelihood.

While calculating $s^{\text{E}}(\mathbf{z},\mathbf{c})$ still requires
decoding the latent, the application of the stop_grad operation eliminates the
need for gradient transmission through $\mathcal{D}$, leading to a substantial
reduction in memory usage. Moreover, optimizing
$\mathcal{R}^{\text{L}}_{\sigma}$ with $L_{\text{RM}}(\sigma)$ is independent
from optimizing $\boldsymbol{f}_{\theta}$ with $L_{\text{RG-LCD}}$. Therefore,
we can use a smaller batch size to optimize $\mathcal{R}^{\text{L}}_{\sigma}$
without affecting the batch size used to optimize $\boldsymbol{f}_{\theta}$.

In essence, our LRM acts as an proxy connecting the LCM
$\boldsymbol{f}_{\theta}$ and the expert RM $\mathcal{R}^{\text{E}}$. As we
will show Sec. 5.2, using this indirect feedback from the expert mitigate the
issue of reward over-optimization, avoiding high-frequency noise in the
generated images.

##  5 Experiment

We perform thorough experiments to demonstrate the effectiveness of our RG-
LCD. Sec. 5.1 conducts human evaluation to compare the performance of our
methods with baselines. Sec. 5.2 further increases the experiment scales to
experiment with a wider array of RMs with automatic metrics. By connecting
both evaluation results, we identify problems with the current RMs. Finally,
Sec. 5.3 conducts ablation studies on critical design choices.

Settings Our training are conducted on the CC12M datasets [6], as the LAION-
Aesthetics datasets [47] used by the original LCM [35] are no longer
accessible111https://laion.ai/notes/laion-maintanence.We distill our LCM from
the Stable Diffusion-v2.1 [43] by training for 10K iterations on 8 NVIDIA A100
GPUs without gradient accumulation and set the batch size to reach the maximum
capacity of our GPUs. We follow the hyperparameter settings listed in the
diffusers [61] library by setting learning rate $1e-6$, EMA rate $\mu=0.95$
and the guidance scale range $[\omega_{\min},\omega_{\max}]=[5,15]$. As
mentioned in Sec. 3.3, we use DDIM [52] as our ODE solver $\Psi$ with a
skipping step $k=20$. We include more training details in Appendix A.

###  5.1 Evaluating RG-LCD with Human

Figure 4: Human evaluation results on the PartiPrompt (1632 prompts) across
three evaluation questions. Top row evaluates the RG-LCM (CLIP). Bottom row
evaluates the RG-LCM (HPS). Figure 5: Human evaluation results on the HPSv2
test set (3200 prompts) across three evaluation questions. Top row evaluates
the RG-LCM (CLIP). Bottom row evaluates the RG-LCM (HPS).

We train RG-LCM (HPS) and RG-LCM (CLIP) utilizing feedback from HPSv2.1 [64]
and CLIPScore [41], respectively. CLIPScore evaluates the relevance between
text and images, whereas HPSv2.1, derived by fine-tuning CLIPScore with human
preference data, is expected to mirror human preferences more accurately. We
choose the teacher LDM (Stable Diffusion v2.1) and a standard LCM distilled
from the same teacher LDM as the baseline methods. To demonstrate the efficacy
of our methods, we compare the performance of our RG-LCMs over 2-step and
4-step generations against the 50-step generations from the teacher LDM and
evaluate the 4-step generation quality of our RG-LCMs against the standard
LCM.

We follow a similar evaluation protocol as in [62] to generate images by
conditioning on prompts from Partiprompt [67] (1632 prompts) and of HPSv2’s
test set [64] (3200 prompts). We hire labelers from Amazon Mechanical Turk for
a head-to-head comparison of images based on three criteria: Q1 General
Preference (Which image do you prefer given the prompt?), Q2 Visual Appeal
(Which image is more visually appealing, irrespective of the prompt?), and Q3
Prompt Alignment (Which image better matches the text description?).

The full human evaluation results in Fig. 4 and 5 show that the 2-step
generations from RG-LCM (CLIP) are generally preferred (Q1) over the 50-step
generations of the teacher LDM in both prompt sets, representing a 25-fold
acceleration in inference speed. Even with CLIPScore feedback, the 4-step
generations from our RG-LCM are generally preferred (Q1) over the baseline
methods. This indicates a noteworthy achievement, given that CLIPScore does
not train on human preference data. Surprisingly, on the HPSv2 prompt set, the
4-step generations from the RG-LCM (CLIP) are more preferred (59.4% against
50-step DDIM samples from SD and 81.7% against 4-step LCM samples) compared to
the 4-step generations of the RG-LCM (CLIP) (57.1% against 50-step DDIM
samples from SD, and 69.0% against 4-step LCM samples).

To investigate this phenomenon, we observe that both RG-LCMs score similarly
in General Preference (Q1) and Prompt Alignment (Q3). However, the RG-LCM
(CLIP) is rated slightly lower in Visual Appeal (Q2) than in the other
criteria, whereas the RG-LCM (HPS) is rated significantly higher for Q2
compared to Q1 and Q3. This distinction highlights that CLIPScore’s primary
contribution is enhancing text-image alignment, whereas an RM like HPSv2.1
particularly focuses on improving visual quality. Thus, when over-optimizing
towards HPSv2.1, the RG-LCM (HPS) can be biased in generating visually
appealing samples by sacrificing prompt alignment.

###  5.2 Evaluating RG-LCD with Automatic Metrics

Models | NFEs | Human Preference Score v2.1 $\uparrow$ | FID-30K $\downarrow$
---|---|---|---
|  | Anime | Photo | Concept-Art | Paintings | MS-COCO
LCM | 4 | 22.40 | 19.17 | 18.86 | 20.55 | 19.05
Stable Diffusion v2.1 | 50 | 25.66 | 24.37 | 24.58 | 25.72 | $\mathbf{12.66}$
RG-LCM (CLIP) | 2 | 26.32 | 25.01 | 25.27 | 26.71 | 18.06
RG-LCM (CLIP) | 4 | 27.80 | 26.92 | 27.04 | 28.11 | 19.22
RG-LCM (Pick) | 2 | 26.44 | 28.26 | 28.24 | 29.04 | 22.84
RG-LCM (Pick) | 4 | 27.33 | 29.42 | 29.29 | 30.26 | 22.02
RG-LCM (Pick) + LRM | 2 | 23.82 | 21.31 | 21.90 | 22.99 | 15.91
RG-LCM (Pick) + LRM | 4 | 25.17 | 23.06 | 22.90 | 24.87 | 16.27
RG-LCM (ImgRwd) | 2 | 29.65 | 31.03 | 31.15 | 32.00 | $\underline{32.12}$
RG-LCM (ImgRwd) | 4 | 30.26 | 31.83 | 31.88 | 32.73 | $\underline{42.69}$
RG-LCM (ImgRwd) + LRM | 2 | 25.64 | 25.61 | 25.82 | 25.75 | 17.57
RG-LCM (ImgRwd) + LRM | 4 | 26.84 | 26.72 | 26.72 | 27.30 | 17.20
RG-LCM (HPS) | 2 | 30.85 | 33.66 | 33.35 | 33.66 | 24.04
RG-LCM (HPS) | 4 | $\mathbf{31.83}$ | $\mathbf{34.84}$ | $\mathbf{34.43}$ | $\mathbf{34.75}$ | 25.11
RG-LCM (HPS) + LRM | 2 | 24.22 | 22.43 | 22.94 | 23.16 | 15.60
RG-LCM (HPS) + LRM | 4 | 25.60 | 24.24 | 24.08 | 25.14 | 15.25
Table 1: Evaluation of our RG-LCMs on the HPSv2 test prompts and MS-COCO
datasets. NFEs denotes the number of function evaluation during inference. We
train RG-LCMs with CLIPScore, PickScore, ImageReward (ImgRwd) and HPSv2.1. We
employ the HPSv2.1 to evaluate the generations on HPSv2 Benchmark’s test set.
We calculate the FID of the generations on the MS-COCO. Except trained with
CLIPScore, our RG-LCMs achieve better HPSv2.1 scores on HPSv2 test prompts at
the expense of higher FIDs on MS-COCO. Integrating a LRM into our RG-LCD
process allows for simultaneously improvement on HPSv2.1 scores on HPSV2 test
prompts and FID on MS-COCO against the baseline LCM.

In this section, we further train RG-LCD (ImgRwd) and RG-LCD (Pick) by
leveraging feedback from ImageReward [66] and PickScore [24]. Both of these
RMs are trained on human preference data. We will use automatic metrics to
perform a large-scale evaluation of the performance of different models. As we
have human evaluation results for RG-LCD (HPS) and RG-LCD (CLIP), we can also
evaluate the quality of the automatic metrics. For each RG-LCD, we collect
their 2-step and 4-step generations by conditioning on prompts from HPSv2’s
test set and measuring the HPSv2.1 score associated with the samples. To
comprehensively understand the sample quality from different models, we
further generate images conditioned on the prompts of MS-COCO [29] and measure
their Fréchet Inception Distance (FID) to the ground truth images.

Table 1 presents the full evaluation results with the automatic metrics.
Except for RG-LCM (CLIP), all the other RG-LCMs achieve higher HPSv2.1 scores
than the baseline LCM but at the expense of higher FID values on the MS-COCO
dataset. Specifically, the RG-LCM (ImgRwd) model exhibits a notably high FID
value, yet it still secures an impressive HPSv2.1 score when evaluated on
HPSv2 test prompts. The elevated FID value aligns with expectations, as Figure
3 illustrates that optimization directed towards ImageReward tends to
introduce a significant amount of high-frequency noise into the generated
images. Surprisingly, these high-frequency noises do not adversely affect the
HPSv2.1 scores. Furthermore, the HPSv2.1 scores do not capture the human
preference for the 4-step samples from RG-LCM (CLIP) by giving the highest
score to RG-LCM (HPS)’s 4-step samples, contrary to what is depicted in human
evaluation shown in Fig. 5.

These observations suggest that the HPSv2.1 score, as a metric, has
limitations and requires further refinement. We conjecture that the _Resize_
operation, which happens during the preprocessing phase, causes the HPSv2.1
model to overlook the high-frequency noise during reward calculation. As
illustrated in Fig. 3, the high-frequency noise becomes less perceptible when
images are reduced in size. Although resizing operations enhance efficiency in
tasks such as image classification [32, 9, 15] and facilitate high-level text-
image understanding [41], they prevent the model from capturing critical
visual nuances that are vital for accurately reflecting human preferences.
Consequently, we advocate for future RMs to exclude the _Resize_ operation.
One potential approach could involve training an LRM, as in our paper, to
learn human preferences in the latent space, circumventing the need for
resizing images.

Connecting Table 1 with the human evaluation results in Fig. 5 suggests that
images that achieve a high HPSv2.1 score and a low FID on MS-COCO are more
aligned with human preferences. Moreover, this desirable outcome can be
accomplished by integrating an LRM into our RG-LCD. Although these
correlations do not imply causality, they underscore the potential benefits of
utilizing an LRM in the RG-LCD process. As depicted in the bottom row in Fig.
3, the images generated by RG-LCM (ImgRwd) that integrates an LRM do not
suffer from high-frequency noise, contributing to their improved FID on MS-
COCO. In Appendix C, we include additional samples for each RG-LCM in Table 1.

###  5.3 Ablation Study

Figure 6: Ablating the reward scale $\beta$ for different reward functions.
All samples are generated with 4 steps. We observe that over-optimizing RMs
that trained with preference data prioritizes visual appeal over text
alignment. In contrast, over-optimizing CLIPScore compromises visual
attractiveness in favor of text alignment. Figure 7: Ablation study on the
number of training iterations. We generate all samples with 4 steps. We
observe that RG-LCM, which learns from an RM that prioritizes visual appeal,
can generate high-quality images with fewer training iterations.

Ablation on the reward scale $\beta$. We use the hyperparameter $\beta$ to
determine the optimization strength towards the RM. We are especially
interested in the impact of an extremely large $\beta$, which can lead to
reward over-optimization [22]. We already know that over-optimizing the
ImageReward can lead to the introduction of high-frequency noise in the
generated images. To expand our understanding, we conduct experiments a wider
array of RMs including HPSv2.1, PickScore and CLIPScore and evaluate whether
over-optimizing these RMs will also leads to similar high-frequency noise.

The results in Figure 6 reveal that an extremely large $\beta$ value does not
introduce the high-frequency noise when using HPSv2.1, PickScore, and
CLIPScore, even though all these metrics resize input images to 224x224 pixels
as in ImageReward. Notably, over-optimization of HPSv2.1 leads to generating
images with repetitive objects as described in the text prompts and increases
color saturation. Conversely, over-optimization of PickScore tends to result
in images with more muted colors. On the other hand, excessive optimization of
CLIPScore results in images where the text prompts are visibly incorporated
into the imagery. These findings align with the discussions in Sec. 5.1,
suggesting that optimizing towards a preference-trained RM generally
prioritizes visual appeal over text alignment. In contrast, over-optimizing
CLIPScore compromises visual attractiveness in favor of text alignment. We
include additional image samples in Appendix C.

Ablation on the training iterations. In total, we train each RG-LCM for 10K
iterations. We take checkpoints from 1K, 2K, 4K, and 10K iterations and sample
images with the same prompts and seeds. We can observe performing RG-LCD with
RMs that facilitate the visual appeal of the generated images also results in
fast training, as the checkpoint at the 2K iterations can already produce
high-quality images. In contrast, the images generated by RG-LCM (CLIP) still
generate blurry images after training for 2K iterations.

##  6 Conclusion

In this paper, we introduce RG-LCD, a novel strategy that integrates feedback
from an RM into the LCD process. The RG-LCM learned via our method enjoys
better sample quality while facilitating fast inference, benefiting from
additional computational resources allocated to align with human preferences.
By evaluating using prompts from the HPSv2 [64] test set and PartiPrompt [67],
we empirically show that humans favor the 2-step generations of our RG-LCD
(HPS) over the 50-step DDIM generations of the teacher LSM. This represents a
25-fold increase in terms of inference speed without a loss in quality.
Moreover, even when using CLIPScore—a model not fine-tuned on human
preferences—our method’s 4-step generations still surpass the 50-step DDIM
generations from the teacher LDM.

We also identify that directly optimizing towards an imperfect RM, e.g.,
ImageReward, can cause high-frequency noise in generated images. To reconcile
the issue, we propose integrating an LRM into the RG-LCD framework. Notably,
our methods not only prevents reward over-optimization but also avoids passing
gradients through the VAE decoder and facilitates learning from non-
differentiable RMs.

## References

  * Achiam et al. [2023a] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023a.
  * Achiam et al. [2023b] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023b.
  * Betker et al. [2023a] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2(3):8, 2023a.
  * Betker et al. [2023b] J. Betker, G. Goh, L. Jing, TimBrooks, J. Wang, L. Li, LongOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu, YunxinJiao, and A. Ramesh.  Improving image generation with better captions.  2023b.  URL https://api.semanticscholar.org/CorpusID:264403242.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Changpinyo et al. [2021] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568, 2021.
  * Clark et al. [2023] K. Clark, P. Vicol, K. Swersky, and D. J. Fleet.  Directly fine-tuning diffusion models on differentiable rewards.  _arXiv preprint arXiv:2309.17400_ , 2023.
  * Dai et al. [2023] X. Dai, J. Hou, C.-Y. Ma, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al.  Emu: Enhancing image generation models using photogenic needles in a haystack.  _arXiv preprint arXiv:2309.15807_ , 2023.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dockhorn et al. [2022] T. Dockhorn, A. Vahdat, and K. Kreis.  Genie: Higher-order denoising diffusion solvers.  _Advances in Neural Information Processing Systems_ , 35:30150–30166, 2022.
  * Dong et al. [2023] H. Dong, W. Xiong, D. Goyal, R. Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_ , 2023.
  * Fan et al. [2024] Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee.  Reinforcement learning for fine-tuning text-to-image diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Feng et al. [2023] X. Feng, Z. Wan, M. Wen, Y. Wen, W. Zhang, and J. Wang.  Alphazero-like tree-search can guide large language model decoding and training.  _arXiv preprint arXiv:2309.17179_ , 2023.
  * Geng et al. [2024] Z. Geng, A. Pokle, and J. Z. Kolter.  One-step diffusion distillation via deep equilibrium models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Ho and Salimans [2022] J. Ho and T. Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hu et al. [2024] H. Hu, K. C. Chan, Y.-C. Su, W. Chen, Y. Li, K. Sohn, Y. Zhao, X. Ben, B. Gong, W. Cohen, et al.  Instruct-imagen: Image generation with multi-modal instruction.  _arXiv preprint arXiv:2401.01952_ , 2024.
  * Hyvärinen and Dayan [2005] A. Hyvärinen and P. Dayan.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Jolicoeur-Martineau et al. [2021] A. Jolicoeur-Martineau, K. Li, R. Piché-Taillefer, T. Kachman, and I. Mitliagkas.  Gotta go fast when generating data with score-based models.  _arXiv preprint arXiv:2105.14080_ , 2021.
  * Karras et al. [2022] T. Karras, M. Aittala, T. Aila, and S. Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kim et al. [2023] K. Kim, J. Jeong, M. An, M. Ghavamzadeh, K. D. Dvijotham, J. Shin, and K. Lee.  Confidence-aware reward optimization for fine-tuning text-to-image models.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Kingma et al. [2021] D. Kingma, T. Salimans, B. Poole, and J. Ho.  Variational diffusion models.  _Advances in neural information processing systems_ , 34:21696–21707, 2021.
  * Kirstain et al. [2024] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Ku et al. [2023] M. Ku, D. Jiang, C. Wei, X. Yue, and W. Chen.  Viescore: Towards explainable metrics for conditional image synthesis evaluation.  _arXiv preprint arXiv:2312.14867_ , 2023.
  * Lee et al. [2023] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu.  Aligning text-to-image models using human feedback.  _arXiv preprint arXiv:2302.12192_ , 2023.
  * Li et al. [2022a] J. Li, D. Li, C. Xiong, and S. Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , 2022a.
  * Li et al. [2022b] J. Li, T. Tang, W. X. Zhao, J.-Y. Nie, and J.-R. Wen.  Pretrained language models for text generation: A survey.  _arXiv preprint arXiv:2201.05273_ , 2022b.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lu et al. [2022a] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022a.
  * Lu et al. [2022b] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Lu and Weng [2007] D. Lu and Q. Weng.  A survey of image classification methods and techniques for improving classification performance.  _International journal of Remote sensing_ , 28(5):823–870, 2007.
  * Lu et al. [2024] Y. Lu, X. Yang, X. Li, X. E. Wang, and W. Y. Wang.  Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Luhman and Luhman [2021] E. Luhman and T. Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  _arXiv preprint arXiv:2101.02388_ , 2021.
  * Luo et al. [2023] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _arXiv preprint arXiv:2310.04378_ , 2023.
  * Meng et al. [2023] C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14297–14306, 2023.
  * Nguyen and Tran [2023] T. H. Nguyen and A. Tran.  Swiftbrush: One-step text-to-image diffusion model with variational score distillation.  _arXiv preprint arXiv:2312.05239_ , 2023.
  * Nichol and Dhariwal [2021] A. Q. Nichol and P. Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Prabhudesai et al. [2023] M. Prabhudesai, A. Goyal, D. Pathak, and K. Fragkiadaki.  Aligning text-to-image diffusion models with reward backpropagation.  _arXiv preprint arXiv:2310.03739_ , 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rafailov et al. [2024] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans and Ho [2022] T. Salimans and J. Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Sauer et al. [2023] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach.  Adversarial diffusion distillation.  _arXiv preprint arXiv:2311.17042_ , 2023.
  * Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Segalis et al. [2023] E. Segalis, D. Valevski, D. Lumen, Y. Matias, and Y. Leviathan.  A picture is worth a thousand words: Principled recaptioning improves image generation.  _arXiv preprint arXiv:2310.16656_ , 2023.
  * Shih et al. [2024] A. Shih, S. Belkhale, S. Ermon, D. Sadigh, and N. Anari.  Parallel sampling of diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Singh and Zheng [2024] J. Singh and L. Zheng.  Divide, evaluate, and refine: Evaluating and improving text-to-image alignment with iterative vqa feedback.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Sohl-Dickstein et al. [2015] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. [2020a] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song and Dhariwal [2023] Y. Song and P. Dhariwal.  Improved techniques for training consistency models.  _arXiv preprint arXiv:2310.14189_ , 2023.
  * Song and Ermon [2019] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. [2020b] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Song et al. [2023] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever.  Consistency models.  _International conference on machine learning_ , 2023.
  * Stiennon et al. [2020] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Sun et al. [2023] J. Sun, D. Fu, Y. Hu, S. Wang, R. Rassin, D.-C. Juan, D. Alon, C. Herrmann, S. van Steenkiste, R. Krishna, et al.  Dreamsync: Aligning text-to-image generation with image understanding feedback.  _arXiv preprint arXiv:2311.17946_ , 2023.
  * Team et al. [2023] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Vijayakumar et al. [2016] A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. Crandall, and D. Batra.  Diverse beam search: Decoding diverse solutions from neural sequence models.  _arXiv preprint arXiv:1610.02424_ , 2016.
  * von Platen et al. [2022] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wallace et al. [2023a] B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik.  Diffusion model alignment using direct preference optimization.  _arXiv preprint arXiv:2311.12908_ , 2023a.
  * Wallace et al. [2023b] B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik.  Diffusion model alignment using direct preference optimization.  _arXiv preprint arXiv:2311.12908_ , 2023b.
  * Wu et al. [2023a] X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li.  Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.  _arXiv preprint arXiv:2306.09341_ , 2023a.
  * Wu et al. [2023b] X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li.  Human preference score: Better aligning text-to-image models with human preference.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2096–2105, 2023b.
  * Xu et al. [2024] J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yu et al. [2022] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022.
  * Zhang et al. [2023] K. Zhang, X. Yang, W. Y. Wang, and L. Li.  Redi: Efficient learning-free diffusion inference via trajectory retrieval.  _arXiv preprint arXiv:2302.02285_ , 2023.
  * Zhang and Chen [2022] Q. Zhang and Y. Chen.  Fast sampling of diffusion models with exponential integrator.  _arXiv preprint arXiv:2204.13902_ , 2022.
  * Zhang et al. [2024] Y. Zhang, E. Tzeng, Y. Du, and D. Kislyuk.  Large-scale reinforcement learning for diffusion models.  _arXiv preprint arXiv:2401.12244_ , 2024.
  * Zheng et al. [2022] H. Zheng, P. He, W. Chen, and M. Zhou.  Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders.  _arXiv preprint arXiv:2202.09671_ , 2022.
  * Zheng et al. [2023] H. Zheng, W. Nie, A. Vahdat, K. Azizzadenesheli, and A. Anandkumar.  Fast sampling of diffusion models via operator learning.  In _International Conference on Machine Learning_ , pages 42390–42402. PMLR, 2023.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

Appendix

Figure 8: Samples from our RG-LCM (HPSv2.1) with the teacher Stable Diffusion
v2.1-base. The resolution is 512 x 512.

In the main paper, we distill our RG-LCD from the Stable Diffusion-v2.1 (768 x
768) [43]. Fig. 8 further shows the samples from our RG-LCM (HPSv) distilled
from Stable Diffusion-v2.1-base (512 x 512) [43]. The rest of the appendix is
structured as below

  * •

Appendix A details the experimental setup and hyperparameter configurations.

  * •

Appendix B elaborates on the training processes and sampling techniques from a
(latent) CM.

  * •

Appendix C shows extra samples generated by various models.

##  Appendix A Additional Experimental Details and Hyperparameters (HPs)

For qualitative evaluation, we ensure consistency across all methods by using
the same random seed for head-to-head image comparisons.

As mentioned in Sec. 5, our training are conducted on the CC12M datasets [6],
as the LAION-Aesthetics datasets [47] used in the original LCM paper [35] are
not accessible222https://laion.ai/notes/laion-maintanence. We train all LCMs
(including RG-LCMs and the standard LCM) by distilling from the teacher LDM
Stable Diffusion-v2.1 [43] for 10K gradient steps on 8 NVIDIA A100 GPUs. When
learning the standard LCM, we use a batch size 32 on each GPU (256 effective
batch size). For RG-LCMs, we use a batch size 5 on each GPU (40 effective
batch size). Interestingly, we observe that different batch sizes do not
impact the final performance too much.

We use the same set of hyperparameters (HP) for training RG-LCM and the
standard LCM by following the settings listed in the diffusers [61] library,
except that RG-LCM has a unique HP $\beta$. Specifically, we set the learning
rate $1e-6$, EMA rate $\mu=0.95$ and the guidance scale range
$[\omega_{\min},\omega_{\max}]=[5,15]$. We include more training details in
Appendix A. Following the practice in [35], we initialize
$\boldsymbol{f}_{\theta}$ with the same parameters as the teacher LDM. We
further encode the CFG scale $\omega$ by applying the Fourier embedding to
$\omega$ and integrate it into the LCM backbone by adding the projected
$\omega$-embedding into the original embedding as in [36].

As mentioned in Sec. 3.3, we use DDIM [52] as our ODE solver $\Psi$ with a
skipping step $k=20$, the formula of the DDIM ODE solver $\Psi_{\text{DDIM}}$
from $t_{n+k}$ to $t_{n}$ is given below [35]

$\begin{aligned}
\Psi_{\text{DDIM}}&\left(\boldsymbol{z}_{t_{n+k}},t_{n+k},t_{n},\boldsymbol{c}\right)=\\\
&\underbrace{\frac{\alpha_{t_{n}}}{\alpha_{t_{n+k}}}\boldsymbol{z}_{t_{n+k}}-\beta_{t_{n}}\left(\frac{\beta_{t_{n+k}}\cdot\alpha_{t_{n}}}{\alpha_{t_{n+k}}\cdot\beta_{t_{n}}}-1\right)\hat{\boldsymbol{\epsilon}}_{\psi}\left(\boldsymbol{z}_{t_{n+k}},\boldsymbol{c},t_{n+k}\right)}_{\text{DDIM
Estimated }\boldsymbol{z}_{t_{n}}}-\boldsymbol{z}_{t_{n+k}}\end{aligned},$
(13)

where $\hat{\boldsymbol{\epsilon}}_{\psi}$ denotes the noise prediction model
from the teacher LDM. $\alpha_{t_{n}}$ and $\beta_{t_{n}}$ specify the noise
schedule. For the forward process SDE defined in (1), we have

$\boldsymbol{\mu}(t)=\frac{\mathrm{d}\log\alpha(t)}{\mathrm{d}t},\quad\sigma^{2}(t)=\frac{\mathrm{d}\beta^{2}(t)}{\mathrm{d}t}-2\frac{\mathrm{d}\log\alpha(t)}{\mathrm{d}t}\beta^{2}(t).$
(14)

As a result, we have
$p_{t}(\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t}|\alpha(t),\beta^{2}(t)\mathbf{I})$.
We refer interested readers to the original LCM paper [35] for further
details.

Reward scale ${\beta}$ for different RG-LCMs with different RMs. In Sec. 5.1
and 5.2, we train our RG-LCMs with different RMs, including CLIPScore [41],
PickScore [24], ImageReward [66] and HPSv2.1 [64]. Table 2 shows the $\beta$
we used for different RMs when obtaining the results in Fig. 4, 5 and Table 1.

| CLIPScore | PickScore | ImageReward | HPSv2.1
---|---|---|---|---
$\beta$ | 5.0 | 5.0 | 1.0 | 1.0
Table 2: $\beta$ for different RG-LCMs when training with different RMs.

Details for integrating an LRM into RG-LCM As discussed in Sec. 4.2, the LRM
admits a similar architecture as the CLIP [41] model, with the distinction of
replacing the visual encoder with a latent encoder. We retain the original
pretrained text encoder and focus on pretraining the latent encoder from
scratch. This process mirrors the CLIP’s pretraining approach, minimizing the
same contrastive loss on the CC12M datasets [6]. The image latent is extracted
with the same VAE encoder used in the teacher LDM Stable Diffusion-v2.1. Upon
completing the pretraining phase, the LRM demonstrates promising initial
results, achieving a zero-shot Top-1 Accuracy of $38.8\%$ and Top-5 Accuracy
of $66.47\%$ on the ImageNet validation set [9]. These metrics underscore the
model’s fundamental capability in understanding text-image alignments.

During the RG-LCD process, we finetune the LRM to match the preference of an
expert RM. We train the last 2 layer of the latent encoder and the last 5
layers of the text encoder. We set the learning rate to $0.0000033$ following
[64]. Note that we do not perform heavy HP searches to determine their optimal
values.

##  Appendix B Training and Sampling from (Latent) CM

###  B.1 Multistep sampling from a learned CM and LCM

We provide the pseudo-codes for multistep consistency sampling [56] and
multistep latent consistency sampling [35] procedures in Algorithm 1 and
Algorithm 2, respectively. The multistep sampling procedures alternate between
the consistency mapping and noise-injection steps, trading additional
computation resources for better sample quality. In the $n$-th iteration, we
first perturb the predicted sample $\mathbf{x}$ (or $\mathbf{z}$) with
Gaussian noise to obtain $\hat{\mathbf{x}}_{\tau_{n}}$ (or
$\hat{\mathbf{z}}_{\tau_{n}}$). We then map the noisy sample
$\hat{\mathbf{x}}_{\tau_{n}}$ (or $\hat{\mathbf{z}}_{\tau_{n}}$) to obtain a
new $\mathbf{x}$ (or $\mathbf{z}$).

Algorithm 1 Multistep Consistency Sampling

CM $\boldsymbol{f}_{\theta}$, steps $N$, timestep sequence
$\tau_{1}>\tau_{2}>\dots>\tau_{N-1}$, noise schedule $\alpha(t),\beta(t)$.

Sample initial noise
$\hat{\mathbf{x}}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

$\mathbf{x}\leftarrow\boldsymbol{f}_{\theta}(\hat{\mathbf{x}}_{T},T)$

for $n=1,\ldots,N-1$ do

Sample
$\hat{\mathbf{x}}_{\tau_{n}}\sim\mathcal{N}(\alpha(\tau_{n})\mathbf{x},\beta^{2}(\tau_{n})\mathbf{I})$

$\mathbf{x}\leftarrow\boldsymbol{f}(\hat{\mathbf{x}}_{\tau_{n}},\tau_{n})$

end for

Return $\mathbf{x}$

Algorithm 2 Multistep Latent Consistency Sampling

LCM $\boldsymbol{f}_{\theta}$, steps $N$, timestep sequence
$\tau_{1}>\tau_{2}>\dots>\tau_{N-1}$, noise schedule $\alpha(t),\beta(t)$,
text prompt $\mathbf{c}$, CFG scale $\omega$, VAE decoder $\mathcal{D}$.

Sample initial noise
$\hat{\mathbf{z}}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$

$\mathbf{z}\leftarrow\boldsymbol{f}_{\theta}(\hat{\mathbf{z}}_{T},\omega,\mathbf{c},T)$

for $n=1,\ldots,N-1$ do

Sample
$\hat{\mathbf{z}}_{\tau_{n}}\sim\mathcal{N}(\alpha(\tau_{n})\mathbf{z},\beta^{2}(\tau_{n})\mathbf{I})$

$\mathbf{z}\leftarrow\boldsymbol{f}_{\theta}(\hat{\mathbf{z}}_{\tau_{n}},\omega,\mathbf{c},T)$

end for

Return $\mathcal{D}(\mathbf{z})$

###  B.2 Training procedures of RG-LCD

Algorithm 3 Reward Guided Latent Consistency Distillation

dataset $\mathcal{D}$, initial model parameter ${\theta}$, learning rate
$\eta$, ODE solver $\Psi$, distance metric $d$, EMA rate $\mu$, noise schedule
$\alpha(t),\beta(t)$, guidance scale
$\left[\omega_{\min},\omega_{\max}\right]$, skipping interval $k$, VAE encoder
$\mathcal{E}$, decoder $\mathcal{D}$, reward model $\mathcal{R}$, reward scale
$\beta$

Encoding training data into latent space:
$\mathcal{D}_{z}=\\{(\boldsymbol{z},\boldsymbol{c})\mid\boldsymbol{z}=E(\boldsymbol{x}),(\boldsymbol{x},\boldsymbol{c})\in\mathcal{D}\\}$

$\theta^{-}\leftarrow\theta$

repeat

Sample
$(\boldsymbol{z},\boldsymbol{c})\sim\mathcal{D}_{z},n\sim\mathcal{U}[1,N-k]$
and $\omega\sim\left[\omega_{\min},\omega_{\max}\right]$

Sample
$\boldsymbol{z}_{t_{n+k}}\sim\mathcal{N}\left(\alpha\left(t_{n+k}\right)\boldsymbol{z};\sigma^{2}\left(t_{n+k}\right)\mathbf{I}\right)$

$\hat{\boldsymbol{z}}_{t_{n}}^{\Psi,\omega}\leftarrow\boldsymbol{z}_{t_{n+k}}+(1+\omega)\Psi\left(\boldsymbol{z}_{t_{n+k}},t_{n+k},t_{n},\boldsymbol{c}\right)-\omega\Psi\left(\boldsymbol{z}_{t_{n+k}},t_{n+k},t_{n},\varnothing\right)$

$J(\theta)\leftarrow\mathcal{R}\left(\mathcal{D}\left(\boldsymbol{f}_{\theta}\left(\mathbf{z}_{t_{n+k}},\omega,\mathbf{c},t_{n+k}\right)\right),\mathbf{c}\right)$

$\mathcal{L}\left({\theta},{\theta}^{-};\Psi\right)\leftarrow
d\left(\boldsymbol{f}_{\theta}\left(\boldsymbol{z}_{t_{n+k}},\omega,\boldsymbol{c},t_{n+k}\right),\boldsymbol{f}_{\theta^{-}}\left(\hat{\boldsymbol{z}}_{t_{n}}^{\Psi,\omega},\omega,\boldsymbol{c},t_{n}\right)\right)$
\- $\beta\cdot J(\theta)$

${\theta}\leftarrow{\theta}-\eta\nabla_{{\theta}}\mathcal{L}\left({\theta},{\theta}^{-}\right)$

${\theta}^{-}\leftarrow\texttt{stop\\_grad}\left(\mu{\theta}^{-}+(1-\mu){\theta}\right)$

until convergence

Algorithm 3 presents the pseudo-codes for our RG-LCD. We use the red color to
highlight the difference between our RG-LCD and the standard LCD [35].

##  Appendix C Additional Qualititive Results

Figure 9: Samples from our RG-LCMs with different RMs compared with the
baseline LCM and teacher Stable Diffusion v2.1. Figure 10: More samples from
our RG-LCMs with different RMs compared with the baseline LCM and teacher
Stable Diffusion v2.1. Figure 11: Effect of the Latent proxy RM (LRM).
Integrating the LRM into our RG-LCD procedures makes the generated images
natural, corresponding to the lower FID in Table 1. Moreover, it helps
eliminate the high-frequency noise in the generated images. Figure 12:
Additional images to study the impact of the reward scale $\beta$. We generate
all samples with 4 steps.

We provide additional samples from our RG-LCMs with different RMs compared
with the baseline LCM and teacher Stable Diffusion v2.1 in Fig. 9 and 10.

The prompts for images in Fig. 9 in the left-to-right order are given below

  * •

Van Gogh painting of a teacup on the desk

  * •

Impressionist painting of a cat, textured, hypermodern

  * •

photo of a kid playing , snow filling the air

  * •

A fluffy owl sits atop a stack of antique books in a detailed and moody
illustration.

  * •

a deer reading a book

  * •

a photo of a monkey wearing glasses in a suit

The prompts for images in Fig. 10 in the left-to-right order are given below

  * •

ornate archway inset with matching fireplace in room

  * •

Poster of a mechanical cat, techical Schematics viewed from front

  * •

portrait of a person with Cthulhu features, painted by Bouguereau.

  * •

a serene nighttime cityscape with lake reflections, fruit trees

  * •

Teddy bears working on new AI research on the moon in the 1980s.

  * •

A cinematic shot of robot with colorful feathers

Fig. 11 presents image samples when integrating a latent proxy RM (LRM) into
our RG-LCD procedures. The prompts in the left-to-right order are given below

  * •

a man in a brown blazer standing in front of smoke, backlit, in the style of
gritty hollywood glamour, light brown and emerald, movie still, emphasis on
facial expression, robert bevan, violent, dappled

  * •

a cute pokemon resembling a blue duck wearing a puffy coat

  * •

Highly detailed photograph of a meal with many dishes.

Fig. 12 includes additional samples for the ablation on the reward scale
$\beta$. The prompts in the top-to-bottom order are given below

  * •

Ultra realistic photo of a single light bulb, dramatic lighting

  * •

Pirate ship trapped in a cosmic maelstrom nebula

  * •

A golden retriever wearing VR goggles.

  * •

Highly detailed portrait of a woman with long hairs, stephen bliss, unreal
engine, fantasy art by greg rutkowski.

  * •

A stunning beautiful oil painting of a lion, cinematic lighting, golden hour
light.

  * •

A raccoon wearing a tophat and suit, holding a briefcase, standing in front of
a city skyline.