figurec

#  ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise
Optimization

Luca Eyring1,2,3, Shyamgopal Karthik2,3,4,11footnotemark: 1 Karsten Roth2,3,4
Alexey Dosovitskiy5 Zeynep Akata1,2,3

1Technical University of Munich 2Munich Center of Machine Learning
3Helmholtz Munich 4University of Tübingen 5Inceptive
<EMAIL_ADDRESS><EMAIL_ADDRESS>
equal contribution

###### Abstract

Text-to-Image (T2I) models have made significant advancements in recent years,
but they still struggle to accurately capture intricate details specified in
complex compositional prompts. While fine-tuning T2I models with reward
objectives has shown promise, it suffers from "reward hacking" and may not
generalize well to unseen prompt distributions. In this work, we propose
Reward-based Noise Optimization (ReNO), a novel approach that enhances T2I
models at inference by optimizing the initial noise based on the signal from
one or multiple human preference reward models. Remarkably, solving this
optimization problem with gradient ascent for 50 iterations yields impressive
results on four different one-step models across two competitive benchmarks,
T2I-CompBench and GenEval. Within a computational budget of 20-50 seconds,
ReNO-enhanced one-step models consistently surpass the performance of all
current open-source Text-to-Image models. Extensive user studies demonstrate
that our model is preferred nearly twice as often compared to the popular SDXL
model and is on par with the proprietary Stable Diffusion 3 with 8B
parameters. Moreover, given the same computational resources, a ReNO-optimized
one-step model outperforms widely-used open-source models such as SDXL and
PixArt-$\alpha$, highlighting the efficiency and effectiveness of ReNO in
enhancing T2I model performance at inference time. Code is available at
https://github.com/ExplainableML/ReNO.

Figure 1: Qualitative results of four different one-step Text-to-Image models
with and without ReNO over different prompts. The same initial random noise is
used for the one-step generation and the initialization of ReNO. ReNO
significantly improves upon the initially generated image with respect to both
prompt faithfulness as well as aesthetic quality for all four models. Best
viewed zoomed in.

##  1 Introduction

Advancements in Text-to-Image (T2I) models have been achieved in recent years,
largely due to the availability of massive image-text datasets [24, 73, 74]
and the development of denoising diffusion models [75, 32, 17, 66]. Despite
these improvements, T2I models often struggle to accurately capture the
intricate details specified in complex compositional prompts [33, 3]. Common
challenges include incorrect text rendering, difficulties with attribute
binding, generation of unlikely object combinations, and color leakage. While
recent models have begun to address these issues by employing enhanced
language encoders, larger diffusion models, and better data curation [12, 11,
20, 6], these approaches typically involve training larger models from
scratch, making them inapplicable to existing models.

As a more efficient alternative, fine-tuning T2I models has gained significant
attention. This approach can be tailored either toward specific preferences
[94, 68, 29] or general human preferences. Inspired by the success of
Reinforcement Learning from Human Feedback (RLHF) [27, 14] in Large Language
Models (LLMs), several works [64, 21, 16, 95, 10] propose aligning T2I models
by fine-tuning them on human-preferred prompt-image sets using RLHF-inspired
techniques. Additionally, human preference reward models, such as PickScore
[41], HPSv2 [87], and ImageReward [89], have gained popularity. These models
are trained to output a score reflecting human preference for an image given a
specific prompt, typically by measuring human preferences for various images
generated from the same prompt. The scores predicted by these models have been
utilized as evaluation metrics for the quality of generated images.
Furthermore, Clark et al. [15], Prabhudesai et al. [62], Li et al. [46]
directly fine-tune T2I models on these differentiable reward models to
maximize the predicted reward of generated images. This approach is efficient
due to the directly differentiable objective.

Fine-tuning T2I models with reward objectives has a major drawback of “reward
hacking”, which occurs when a reward model gives a high score to an
undesirable image. Reward hacking points to deficiencies in existing reward
models, highlighting gaps between the desired behavior and the behavior
implicitly captured by the reward model, which is especially prone to appear
when explicitly fine-tuning for a reward [15, 46, 18]. Additionally, these
models are often fine-tuned on a small scale (e.g., <50 prompts in some cases
[8, 21]) and thus may not generalize well to unseen prompt distributions with
complex compositional structures. In this work, our aim is to enhance T2I
models at inference for each unique generation, similar to the paradigm of
test-time training for classification models [79, 25]. Fine-tuning diffusion
models for every single prompt would both be computationally expensive
(Dreambooth [68] takes 5 minutes on 1 A100), and susceptible to “reward-
hacking”.

We sidestep the challenge of fine-tuning the model’s parameters and instead
explore optimizing the initial random noise during inference without adapting
any of the model’s parameters. To obtain more optimal noise and a higher-
quality generated image, we introduce Reward-based Noise Optimization (ReNO),
where the initial noise is updated based on the signal from a reward model
evaluated on the generated image. The main challenges in this approach are
twofold. First, backpropagating the gradient through the denoising steps can
lead to exploding/vanishing gradients, rendering the optimization process
unstable. Our insight is that by employing a distilled one-step T2I model [71,
65, 91, 11], we circumvent the issue of exploding/vanishing gradients since
backpropagation is performed through a single step. Second, naively optimizing
the initial latent for an arbitrary objective can lead to collapse due to
reward hacking. To mitigate this, we propose the use of a combination of
reward objectives to not overfit to any single reward. Moreover, given a well-
calibrated one-step T2I model with frozen parameters, the generated images
should not exhibit reward hacking if the initial noise remains in the
proximity of the initial noise distribution. Therefore, we propose an
optimization scheme with limited steps, regularization of the noise to stay
in-distribution, and gradient clipping.

In essence, ReNO involves optimizing the initial latent noise given an one-
step T2I model (e.g., SD/SDXL-Turbo) and a reward model (e.g., ImageReward)
for a limited number of iterations (10-50 steps). On the popular evaluation
benchmarks T2I-Compbench and GenEval, our noise optimization strategy (ReNO)
significantly improves performance, increasing scores by over 20% in some
cases. This enhancement allows SD2.1-Turbo models to approach the performance
of closed-source proprietary models such as DALL-E 3 [6] and SD3 [20]. We
demonstrate that ReNO substantially improves the performance of four different
one-step T2I models (e.g. Figure 1), both in terms of quantitative evaluation
and extensive user studies, while only requiring 20-50 seconds to generate an
image. Moreover, given the same computational budget, ReNO surpasses the
performance of competing multi-step models, offering an attractive trade-off
between performance and inference speed. ReNO not only motivates the
development of more robust reward models but also provides a compelling
benchmark for their evaluation. Finally, our results highlight the importance
of the noise distribution in T2I models and encourage further research into
understanding and adapting it.

##  2 Reward-based Noise Optimization (ReNO)

Despite the remarkable progress in Text-to-Image (T2I) generation, current
state-of-the-art models still struggle to consistently produce visually
satisfactory images that fully adhere to the input prompt. Recent studies have
highlighted the significant impact of the initial noise vector
${\bm{\varepsilon}}$ on the quality of the generated image [76, 17]. In fact,
selecting and re-ranking images generated from a set of initial noises based
on reward models has been shown to substantially improve performance [41, 37].
This observation naturally leads to the question of whether it is possible to
identify an optimal noise vector that maximizes a given goodness measure for
the generated image. In this section, we first provide an overview of one-step
diffusion models, which serve as the foundation for our work. We then
introduce our simple yet principled approach that enables practical noise
optimization to enhance the performance of one-step T2I models based on human-
preference reward models, addressing the challenge of generating high-quality
images that align with the input prompt.

###  2.1 Background: One-Step Diffusion Models

T2I models aim to generate images ${\mathbf{x}}_{0}$ conditioned on a given
textual prompt p. A generative model $G_{\theta}$ parameterized by $\theta$
takes as input a noise vector
${\bm{\varepsilon}}\sim\mathcal{N}(0,\mathbf{I})$ and a prompt p, and outputs
an image $G_{\theta}({\bm{\varepsilon}},\textnormal{p})={\mathbf{x}}_{0}$. The
objective is to learn the parameters $\theta$, such that the generated image
${\mathbf{x}}_{0}$ aligns with the semantics of the prompt p. This is
typically achieved by training the model on a large dataset of paired text and
images. Recent models are based on a time-dependent formulation between a
standard Gaussian distribution
${\bm{\varepsilon}}\sim\mathcal{N}(0,\mathbf{I})$, and data
$\mathbf{x}_{0}\sim p_{0}(\mathbf{x})$. These models define a probability path
between the initial noise distribution and the target data distribution, such
that

$\displaystyle\mathbf{x}_{t}=\alpha_{t}{\mathbf{x}}_{0}+\sigma_{t}{\bm{\varepsilon}},$
(1)

where $\alpha_{t}$ is a decreasing and $\sigma_{t}$ is an increasing function
of $t$. Score-based diffusion [77, 40, 36] and flow matching [48, 50, 1]
models share the observation that the process ${\mathbf{x}}_{t}$ can be
sampled dynamically using a stochastic or ordinary differential equation (SDE
or ODE). Consider the forward SDE that transforms data into noise as $t$
increases
$\mathop{}\\!\textnormal{d}{\mathbf{x}}_{t}={\mathbf{u}}({\mathbf{x}}_{t},t)\mathop{}\\!\textnormal{d}t+g(t)\mathop{}\\!\textnormal{d}{\mathbf{w}}_{t}$,
where ${\mathbf{u}}_{t}(x_{t},t)$ denotes the drift, ${\mathbf{w}}_{t}$ is a
Wiener process and $g(t)$ represents the diffusion schedule. Then, the
marginal probability distribution $p_{t}({\mathbf{x}})$ of ${\mathbf{x}}_{t}$
in (1) coincides with the distribution of the probability flow ODE [77, 40],
as well as the reverse-time SDE [2]

$\mathop{}\\!\textnormal{d}{\mathbf{x}}_{t}=[{\mathbf{u}}({\mathbf{x}}_{t},t)-g(t)^{2}{\mathbf{s}}({\mathbf{x}}_{t},t)]\mathop{}\\!\textnormal{d}t+g(t)\mathop{}\\!\textnormal{d}\bar{\mathbf{w}}_{t},$
(2)

where ${\mathbf{s}}({\mathbf{x}},t)=\nabla\log p_{t}({\mathbf{x}})$ is the
score function. By solving either the ODE or SDE backward in time from
${\mathbf{x}}_{T}={\bm{\varepsilon}}\sim\mathcal{N}(0,\mathbf{I})$, we can
generate samples from $p_{0}({\mathbf{x}})$. This relies on a good estimate of
the parameterized score ${\mathbf{s}}_{\theta}({\mathbf{x}}_{t},t)$. The
choice of functions $\alpha_{t}$ and $\sigma_{t}$ are defined implicitly based
on the forward SDE [78, 36, 76, 40]. Furthermore, the process
${\mathbf{x}}_{t}$ is considered on an interval $[0,T]$ with $T$ sufficiently
large such that ${\mathbf{x}}_{T}$ approximates the initial noise distribution
$\mathcal{N}(0,\mathbf{I})$. Then, it has been shown that the score can be
approximated efficiently based on, e.g. the denoising loss [32]

$\mathcal{L}_{\mathrm{s}}(\theta)=\mathbb{E}_{{\mathbf{x}}_{0}\sim
p({\mathbf{x}}_{0}),{\bm{\varepsilon}}\sim\mathcal{N}(0,\mathbf{I}),t\sim\mathcal{U}(0,T)}[\|\sigma_{t}{\mathbf{s}}_{\theta}({\mathbf{x}}_{t},t)+{\bm{\varepsilon}}\|^{2}].$
(3)

During inference, these models simulate an ODE/SDE through discretization for
a number of steps. This can be computationally expensive as the trained model
must be evaluated sequentially.

##### Distillation.

As a means to reduce inference time, distillation techniques have recently
gained traction with the intent to learn a student model that approximates the
solution of the simulated differential equation with a trained teacher model
given fewer inference steps, e.g. score distillation [61] penalizes the
estimated score to the real data distribution. Furthermore, several methods
have been proposed to distill models into one-step generators, which learn to
approximate the full ODE or SDE in one step. Our work builds upon the
following one-step T2I models which we refer to as
$\smash{\tilde{G}_{\theta}}$. Adversarial Diffusion Distillation (ADD) [71]
combines score distillation with an adversarial loss and is employed to train
SD-Turbo based on SD 2.1 [66] as a teacher and SDXL-Turbo [72] based on SDXL
[59]. Diffusion Matching Distillation (DMD) [91] additionally leverages a
distributional loss based on an approximated KL divergence and is applied for
PixArt-$\alpha$ DMD [12, 11]. Lastly, Trajectory Segmented Consistency
Distillation (TSCD) [65] introduces a progressive segment-wise consistency
distillation [78, 39] loss to train HyperSDXL [65] with reward fine-tuning.
All these models are trained in latent space such that during inference, an
image is generated by first generating a sample in latent space and then
decoding it
$G_{\theta}({\bm{\varepsilon}},\textnormal{p})=\mathcal{D}(\smash{\tilde{G}}_{\theta}({\bm{\varepsilon}},\textnormal{p}))$
with a pre-trained decoder $\mathcal{D}$.

Figure 2: Overview of our proposed ReNO framework. Given reward models based
on human preferences, we optimize the initial latent noise to maximize the
reward scores (consisting HPSv2 [87], PickScore [41], ImageReward [89], and
CLIP [63]) for the images generated by the one-step T2I model. Over 50
iterations, the quality of the images and the prompt faithfulness are
improved.

###  2.2 Initial Noise Optimization

Given a Text-to-Image generative model
$G_{\theta}({\bm{\varepsilon}},\textnormal{p})$ that generates images based on
a noise ${\bm{\varepsilon}}$ and a prompt p, we defined the following
optimization problem following previous work [81, 38, 70, 5] with the
objective of optimizing the noise ${\bm{\varepsilon}}$ based on a criterion
function $\mathcal{C}:\mathbb{R}^{H\times W\times c}\rightarrow\mathbb{R}$
evaluated on the generated image

${\bm{\varepsilon}}^{\star}=\operatorname*{arg\,max}_{\bm{\varepsilon}}\mathcal{C}(G_{\theta}({\bm{\varepsilon}},\textnormal{p})).$
(4)

Then, given a differentiable $\mathcal{C}$, (4) can be solved through
iterative optimization via standard gradient ascent techniques. However,
backpropagating through
$\mathcal{C}(G_{\theta}({\bm{\varepsilon}},\textnormal{p}))$ is non-trivial as
current Text-to-Image models are based on the simulation of ODEs or SDEs
(Section 2.1). Several methods have been proposed to enable backpropagation
through time-dependent generative models [13, 53, 15, 81], based on e.g., the
adjoint method [60]. Our method, in contrast, leverages the crucial
observation that selecting a one-step model as $G_{\theta}$ enables efficient
backpropagation through (4). Although this realization may initially appear
trivial, it proves to be a fundamental step in facilitating practical noise
optimization in Text-to-Image models. Current methods require between 10 [81]
and 40 [5] minutes to optimize noise and thus, to generate a single image. Our
approach achieves image generation, including noise optimization, in 20-50
seconds, making it suitable for practical applications.

##### Noise regularization.

One important consideration, is that it is desirable for ${\bm{\varepsilon}}$
to stay within the proximity of the initial noise distribution
$\mathcal{N}(0,\mathbf{I})$ as otherwise $G_{\theta}$ might provide unwanted
generations. This can be realized by including a regularization term inside of
$\mathcal{C}$. Samuel et al. [69] propose instead of directly optimizing the
likelihood of $p_{T}({\bm{\varepsilon}})$, to instead consider the likelihood
of the norm of the noise $r=||{\bm{\varepsilon}}||$, which is distributed
according to a
$\smash{{\mathchoice{\raisebox{0.0pt}{$\displaystyle\chi$}}{\raisebox{0.0pt}{$\textstyle\chi$}}{\raisebox{0.0pt}{$\scriptstyle\chi$}}{\raisebox{0.0pt}{$\scriptscriptstyle\chi$}}}^{d}}$
distribution $p(r)$. Thus, following Samuel et al. [70], Ben-Hamu et al. [5]
we maximize the log-likelihood of the norm of a noise sample
$K({\bm{\varepsilon}})=(d-1)\text{log}(||{\bm{\varepsilon}}||)-||{\bm{\varepsilon}}||^{2}/2$.
In our framework, this corresponds to employing a regularized criterion
function given by
${\mathcal{C}}({\mathbf{x}}_{0},{\bm{\varepsilon}})=\smash{\tilde{\mathcal{C}}}({\mathbf{x}}_{0})+K({\bm{\varepsilon}})$,
which can be plugged into (7).

In Figure 3, we provide an illustrative example where we chose the criterion
to maximize

Figure 3: Initial noise optimization for one-step $G_{\theta}$ HyperSDXL with
two color channel criterions (5).

a selected color channel $c$ of the generated image while minimizing the other
two $\bar{c}_{1},\bar{c}_{2}$

$\tilde{\mathcal{C}}({\mathbf{x}}_{0})=\sum\nolimits_{i,j}{\mathbf{x}}_{0}^{i,j,c}-{\mathbf{x}}_{0}^{i,j,\bar{c}_{1}}-{\mathbf{x}}_{0}^{i,j,\bar{c}_{2}},$
(5)

where ${\mathbf{x}}_{0}^{i,j,c}$ denotes the channel $c$ of the pixel at
$(i,j)$. Note that due to the calibration of the trained model and the noise
staying in-distribution, the noise does not collapse to the optimal
${\bm{\varepsilon}}^{\star}$, which would result in the generation of a fully
blue or red image. Also, the optimization first adapts the color of the car
and then starts changing the background. Here, 10 optimization steps provide
satisfactory results illustrating the efficacy of the proposed one-step noise
optimization framework.

###  2.3 Human Preference Reward Models and Our Reward Criterion

Inspired by the success of Reinforcement Learning From Human Feedback [27, 14]
in aligning LLMs with human preferences, similar methods have been explored
for T2I generation. The underlying idea is to train a model
$\mathcal{R}_{\psi}$ that takes in an input along with the generated output
(in this case a prompt and the corresponding image) and provides a score for
the “goodness” of the generated output. Notable open-source human preference
reward models for T2I include ImageReward [89] based on BLIP [44] and human
preferences collected for the DiffusionDB dataset, PickScore [41], and HPSv2
[87] both based on a CLIP [63] ViT-H/14 backbone. These reward models provide
a quantitative measure of the image’s quality and relevance to the prompt
through a prediction by a differentiable neural network. Thus, they have not
only been employed for the evaluation of T2I models but also to fine-tune them
[89, 15, 16] as a means of achieving higher reward scores. Lastly, CLIPScore
[31] has also been leveraged to measure the prompt alignment of a generated
image.

To generally enhance the performance of Text-to-Image models without any fine-
tuning, we propose to leverage a Reward-based criterion function $\mathcal{C}$
for Noise Optimization (ReNO). Specifically, we propose to use a weighted
combination of a number $n$ of pre-trained reward models
${\mathcal{R}^{0}_{\psi},\dots\,\mathcal{R}^{n}_{\psi}}$ as the criterion
function

$\tilde{\mathcal{C}}({\mathbf{x}}_{0},\textnormal{p})=\sum\nolimits^{n}_{i}\lambda_{i}\mathcal{R}^{i}_{\psi}({\mathbf{x}}_{0},\textnormal{p}),$
(6)

where $\lambda_{\text{i}}$ denotes the weighting for reward model
$\mathcal{R}_{\psi}^{i}$. Employing a combination of reward models can help
prevent “reward-hacking” and allow capturing various aspects of image quality
and prompt adherence, as different reward models are trained on different
prompt and preference sets. This not only effectively combines the strengths
of multiple reward models, but also helps mitigate their weaknesses. ReNO then
boils down to iteratively solving (4) with gradient ascent

${\bm{\varepsilon}}^{t+1}={\bm{\varepsilon}}^{t}+\eta\nabla_{{\bm{\varepsilon}}^{t}}[K({\bm{\varepsilon}}^{t})+\sum\nolimits^{n}_{i}\lambda_{i}\mathcal{R}^{i}_{\psi}(G_{\theta}({\bm{\varepsilon}}^{t},\textnormal{p}),\textnormal{p})],$
(7)

where $\eta$ is the learning rate. Similar to the color example in Figure 3,
it is actually not desirable to find the optimal ${\bm{\varepsilon}}^{\star}$
as we want to prevent adversarial samples that exploit the reward models. We
find that already a few optimization steps (<50) of ReNO lead to significant
improvements in both prompt following and visual aesthetics, striking a good
balance between reward optimization and the prevention of reward hacking. Due
to the efficacy of the proposed framework, generating one image, including
noise optimization, takes between 20-50 seconds, depending on the model and
image size, enabling its practical use. We provide a sketch of ReNO in Figure
2 and full details in Algorithm 1.

##  3 Related Work

##### Initial Noise Optimization.

The initial noise optimization framework was first introduced in DOODL [81]
for improved guidance in Text-to-Image models. Subsequently, it was leveraged
by Karunratanakul et al. [38] for 3D universal motion priors, for rare-
concept, generation in text-to-image models [70, 69, 54], and by D-Flow [5]
for solving inverse problems in various settings. While these methods mainly
focus on controlling the generated sample for specific applications, our
proposed method is designed to generally improve Text-to-Image models without
the need for additional techniques to mitigate exploding or vanishing
gradients on the optimization process. Most related to our work is DOODL [81],
which also proposes to improve the textual alignment of text-to-image models
using a CLIP-score-based criterion function, which we similarly employ in our
method. These existing methods, however, take 10 (DOODL) to 40 (D-Flow)
minutes to generate a single image due to their application on time-dependent
generative models with a large number of denoising steps. To mitigate this,
Samuel et al. [70] propose a bootstrap-based method to increase the efficiency
of generating a batch of images. However, this method is limited to settings
where the goal is to generate samples including a concept jointly represented
by a set of input images.

##### Reward Optimization for Text-to-Image Models.

Reward models [89, 41, 88, 87, 42] were first introduced to mimic human
preferences given an input prompt and generated images. There have been
several attempts at incorporating these signals to enhance text-to-image
generation. One notable direction is the idea of using reinforcement learning
based algorithms to fine-tune text-to-image models to better align with these
rewards either with an explicit reward model [8, 21, 10, 16, 28, 95] or by
bypassing it entirely with Direct Preference Optimization [64, 90, 45, 82].
However, this can be expensive, requiring thousands of queries to generalize,
and therefore a lot of work has explored directly fine-tuning diffusion models
[15, 46, 62] using differentiable rewards [89, 42, 88, 41, 35]. Additionally,
there has also been works exploring the concept of using reward models to
perform classifier-guidance [4, 30] as well as using rewards to distill
diffusion models into fewer steps [43, 65]. Differently from these works, we
focus on adapting a diffusion model during inference by purely optimizing the
initial latent noise using a differentiable objective.

##  4 Experiments

##### Experimental Setup.

We extensively evaluate the effectiveness of our proposed method, ReNO, using
four open-source one-step image generation models: SD-Turbo, SDXL-Turbo,
PixArt-$\alpha$ DMD, and HyperSDXL. HyperSDXL generates images of size
$1024\times 1024$ while the others generate $512\times 512$. To assess the
performance across diverse scenarios, we consider three challenging tasks.
First, we evaluate on T2I-CompBench [33], which comprises 6000 compositional
prompts spanning six categories, using a VQA, object detection, and image-text
matching scores. Second, we employ GenEval [26], consisting of 552 object-
focused prompts, measuring the quality of the generated images using a pre-
trained object detector. Finally, we utilize Parti-Prompts [92], a collection
of more than 1600 complex prompts, and assess the generated images using both
reward-based metrics and extensive human evaluation. Throughout all
experiments, we optimize Equation (7) for 50 steps using gradient ascent with
Nesterov momentum and gradient norm clipping for stability. Lastly, we select
the image with the highest reward score from the optimization trajectory for
evaluation.

Table 1: SD-Turbo evaluated on the attribute binding categories of
T2I-CompBench and the LAION aesthetic score predictor [74] for different
reward models.

Reward Attribute Binding Aesthetic $\uparrow$ Color $\uparrow$ Shape
$\uparrow$ Texture $\uparrow$ SD-Turbo 0.5513 0.4448 0.5690 5.647 \+ CLIPScore
0.6625 0.5501 0.6621 5.475 \+ HPSv2 0.6443 0.5451 0.6859 5.752 \+ ImageReward
0.7720 0.6104 0.7334 5.611 \+ PickScore 0.6341 0.5069 0.6242 5.711 \+ All
0.7830 0.6244 0.7466 5.704

###  4.1 Effect of Reward Models

We analyze the effect of various reward models in Table 1. We see that
optimizing ImageReward or CLIPScore alone improves the text-image faithfulness
(i.e., attribute binding from T2I-Compbench). However, this comes at the cost
of decreased aesthetic score. PickScore and HPSv2 improve the image quality,
however the gains in faithfulness are modest. Combining all the rewards leads
to having strong improvements in faithfulness, while also increasing the image
quality. Thus, we employ ReNO with all four reward models. We report further
details in Appendix C, including the performance of all combinations of reward
models.

Table 2: Quantitative Results on T2I-CompBench. ReNO combined with (1)
PixArt-$\alpha$ DMD [12, 11, 91], (2) SD-Turbo [72], (3) SDXL-Turbo [72],
HyperSD [65] demonstrates superior compositional generation ability in both
attribute binding, object relationships, and complex compositions. The best
value is bolded, and the second-best value is underlined. Multi-step results
taken from [12, 20].

Model | Attribute Binding | Object Relationship | Complex$\uparrow$
---|---|---|---
Color $\uparrow$ | Shape$\uparrow$ | Texture$\uparrow$ | Spatial$\uparrow$ | Non-Spatial$\uparrow$
SD v1.4 | 0.38 | 0.36 | 0.42 | 0.12 | 0.31 | 0.31
SD v2.1 | 0.51 | 0.42 | 0.49 | 0.13 | 0.31 | 0.34
SDXL | 0.64 | 0.54 | 0.56 | 0.20 | 0.31 | 0.41
PixArt-$\alpha$ | 0.69 | 0.56 | 0.70 | 0.21 | 0.32 | 0.41
DALL-E 2 | 0.57 | 0.55 | 0.64 | 0.13 | 0.30 | 0.37
DALL-E 3 | 0.81 | 0.68 | 0.81 | - | - | -
(1) PixArt-$\alpha$ DMD | 0.38 | 0.34 | 0.47 | 0.19 | 0.30 | 0.36
(1) + ReNO (Ours) | 0.64 | 0.57 | 0.72 | 0.25 | 0.31 | 0.46
(2) SD-Turbo | 0.55 | 0.44 | 0.57 | 0.17 | 0.31 | 0.41
(2) + ReNO (Ours) | 0.78 | 0.62 | 0.75 | 0.22 | 0.32 | 0.48
(3) SDXL-Turbo | 0.61 | 0.44 | 0.60 | 0.24 | 0.31 | 0.43
(3) + ReNO (Ours) | 0.78 | 0.60 | 0.74 | 0.26 | 0.31 | 0.47
(4) HyperSDXL | 0.65 | 0.50 | 0.65 | 0.25 | 0.31 | 0.46
(4) + ReNO (Ours) | 0.79 | 0.63 | 0.77 | 0.26 | 0.31 | 0.48

Table 3: Quantitative Results on GenEval. ReNO combined with (1)
PixArt-$\alpha$ DMD [12, 11, 91], (2) SD-Turbo [72], (3) SDXL-Turbo [72],
HyperSDXL [65] improves results across all categories. The best value is
bolded, and the second-best value is underlined. Multi-step results taken from
[20].

Model | Mean $\uparrow$ | Single$\uparrow$ | Two$\uparrow$ | Counting$\uparrow$ | Colors$\uparrow$ | Position$\uparrow$ | Color Attribution$\uparrow$
---|---|---|---|---|---|---|---
SD v2.1 | 0.50 | 0.98 | 0.51 | 0.44 | 0.85 | 0.07 | 0.17
SDXL | 0.55 | 0.98 | 0.74 | 0.39 | 0.85 | 0.15 | 0.23
IF-XL | 0.61 | 0.97 | 0.74 | 0.66 | 0.81 | 0.13 | 0.35
PixArt-$\alpha$ | 0.48 | 0.98 | 0.50 | 0.44 | 0.80 | 0.08 | 0.07
DALL-E 2 | 0.52 | 0.94 | 0.66 | 0.49 | 0.77 | 0.10 | 0.19
DALL-E 3 | 0.67 | 0.96 | 0.87 | 0.47 | 0.83 | 0.43 | 0.45
SD3 (8B) | 0.68 | 0.98 | 0.84 | 0.66 | 0.74 | 0.40 | 0.43
(1) PixArt-$\alpha$ DMD | 0.45 | 0.95 | 0.38 | 0.46 | 0.76 | 0.05 | 0.09
(1) + ReNO (Ours) | 0.59 | 0.98 | 0.72 | 0.58 | 0.85 | 0.15 | 0.27
(2) SD-Turbo | 0.49 | 0.99 | 0.51 | 0.38 | 0.85 | 0.07 | 0.14
(2) + ReNO (Ours) | 0.62 | 1.00 | 0.82 | 0.60 | 0.88 | 0.12 | 0.33
(3) SDXL-Turbo | 0.54 | 1.00 | 0.66 | 0.45 | 0.84 | 0.09 | 0.20
(3) + ReNO (Ours) | 0.65 | 1.00 | 0.84 | 0.68 | 0.90 | 0.13 | 0.35
(4) HyperSDXL | 0.56 | 1.00 | 0.76 | 0.43 | 0.87 | 0.10 | 0.21
(4) + ReNO (Ours) | 0.65 | 1.00 | 0.90 | 0.56 | 0.91 | 0.17 | 0.33

###  4.2 Quantitative Results

Table 2 presents the quantitative results of ReNO on T2I-Compbench. Most
notably, we observe that for both Pixart-$\alpha$ DMD and SD-Turbo, there are
improvements of over 20% in the Color, Shape, and Texture Categories. For
instance, on Color SD-Turbo improves from 55% to 78%, which is only slightly
below DALL-E 3. Similar improvements can also be seen for SDXL-Turbo and
HyperSDXL models where performance increases by over 10 percentage points in
these categories. Even outside this, there are significant boosts in the
Spatial, Non-Spatial, and Complex categories, highlighting both the efficacy
of the noise optimization framework, as well as the utility of human
preference models for improving T2I generation at inference. Similar trends
can also be noticed for GenEval in Table 3, where applying our noise
optimization framework helps improve the performance of various one-step
diffusion models. For instance, SD-Turbo improves its mean score from 0.49 to
0.62. Notably, our strongest model, HyperSDXL + ReNO, comes very close to the
proprietary DALL-E 3 and SD3, i.e., beating DALL-E 3 on 4/6 categories in
GenEval. In both of these benchmarks, our noise optimization framework
improves results for all the models in all the categories. It is also
important to note that both T2I-Compbench and GenEval use a variety of methods
unrelated to human preference rewards, such as VQA models and object
detectors, to detect different objects in the generated images. We report
further quantitative results including comparisons to other test-time-based
methods in Appendix B. Additionally, these quantitative results are supported
by the qualitative results reported in Figure 1 and Appendix A.

Figure 4: User Study Results for ReNO

###  4.3 User Study Results

To further validate ReNO we perform a user study on the commonly used Parti-
Prompts [92] with Amazon Mechanical Turk (AMT). Parti-Prompts generally
includes longer complex prompts that test artistic generation capabilities as
opposed to T2I-Compbench and GenEval, which purely focus on faithfulness. We
conducted user studies with ReNO applied to SD-Turbo for $512\times 512$ and
HyperSDXL for $1024\times 1024$ generation. We compare SD-Turbo + ReNO against
SD-Turbo, SDXL-Turbo, SD2.1 (50 Steps), and SDXL-Base (50 Steps). The results
in Figure 4 confirm our findings in the quantitative evaluation. SD-Turbo +
ReNO has an above 60% win rate against all benchmarked models reaching up to
77% against the SD-Turbo base. To contextualize these results, SD3 [20]
conducts a similar user study on Parti-Prompts and reports a 70% win rate
against SDXL (50 steps). Our strongest base model, HyperSDXL, already beats
SDXL (50 steps) [65] without ReNO. Thus, we compare it with and without ReNO
as well as against the proprietary SD3 (8B) [20]. Again, HyperSDXL + ReNO
achieves an above 60% win rate, and notably, it also narrowly beats SD3 with
54%. This confirms our finding in ReNO, which substantially improves overall
generative quality, pushing results at least close to the ones of even current
state-of-the-art proprietary models. Lastly, we note that user studies on AMT
can potentially be noisy and, therefore, view the results holistically along
with quantitative evaluation. We provide a detailed breakdown of the
preference for image quality and faithfulness, as well as full details of the
user study in Appendix D.

Figure 5: Attribute binding results on T2I-CompBench with varying number of
iterations.

###  4.4 Computational Cost of ReNO

The primary concern of our proposed method is the increased inference cost
since existing methods (e.g. DOODL, D-Flow) are impractical for regular T2I
generation usage. However, we circumvent this issue through our restriction to
one-step models and 50 optimization steps, which makes ReNO run in 20-50
seconds. To analyze the performance of ReNO with respect to the number of
optimization steps we evaluate its performance over a set of reference points.
We report results on the attribute binding part of T2I-CompBench for SD-Turbo
+ ReNO in Figure 5 and visually corroborate these results with Figure 6. Note
that even when restricted to the same compute budget as SDXL (50 steps,
~7sec), SD-Turbo + ReNO significantly outperforms it while in this comparison
PixArt-$\alpha$ (20 steps, ~7sec) lies shortly below the Pareto-frontier of
ReNO.

Figure 6: The initial images are generated with four different one-step models
$G_{\theta}$ given the prompt p "A yellow reindeer and a blue elephant" and
randomly initialized noise ${\bm{\varepsilon}}^{0}$. Each column shows the
result of optimizing the noise latent ${\bm{\varepsilon}}^{t}$ for $t$ steps
with respect to our reward-based criterion.

##### Limitations.

An interesting observation in our experiments is that despite using different
image generation models of varying architectures and sizes, they converge to
similar performance on both T2I-Compbench and GenEval. In addition to the
limitations of the generative models, we hypothesize that this could be due to
the limitations of the reward models themselves, given their limited
compositional reasoning abilities [93]. Stronger reward models and preference
data would be crucial in enhancing results further.

Table 4: Computational cost comparison of ReNO optimizing four reward models
on an A100 GPU.

Method sec/iter (total) VRAM #params Img size SD-Turbo 0.4s (20s) 15GB 860M
$512\times 512$ SDXL-Turbo 0.6s (30s) 21GB 2.6B $512\times 512$
PixArt-$\alpha$ DMD 0.5s (25s) 25GB 600M $512\times 512$ HyperSDXL 1.0s (50s)
39GB 2.6B $1024\times 1024$

Secondly, not only the runtime but also the amount of needed GPU VRAM is
significantly higher when using ReNO. We reduce it by leveraging fp16
quantization and the pytorch [57] memory reduction technique introduced in
Bhatia and Dangel [7], which for ReNO lowers the VRAM by another ~15%. Then,
all of the models can be optimized on a single A100 GPU in 20-50 seconds, and
e.g., SD-Turbo requires only 15GB VRAM for the entire optimization process.
Note, however, that the amount of VRAM also scales with the size of the
generated image. Thus, HyperSDXL needs 39GB of VRAM. We provide a summary of
the computational cost of ReNO in Table 4, which lays out ReNO’s main
limitation. Finally, current T2I models struggle with generating humans,
rendering text, and also modeling complex compositional relations. While our
work attempts to alleviate these issues and provides a flexible framework for
further improvements, future work is required to resolve these issues.

##  5 Conclusion

This work introduces ReNO, a test-time optimization strategy for enhancing
text-to-image generation without any fine-tuning. Not only do we achieve the
strongest results among all open-source models on T2I-Compbench and GenEval,
but images from ReNO on a single-step SD-Turbo have over a 60% win rate
against a 50-step SDXL model and is competitive with the 8B parameter SD3
model on user studies. We also demonstrate that ReNO outperforms SDXL even
when restricted to the same computational budget, highlighting the benefits of
ReNO for practical use cases. The performance gains from ReNO underscore the
importance of developing even better and more robust reward models and,
moreover, establish a valuable benchmark for assessing their effectiveness.
Furthermore, the substantial impact of optimizing the initial noise
distribution motivates further research into understanding, manipulating, and
controlling this crucial aspect of generative models.

## Acknowledgements

This work was supported by BMBF FKZ: 01IS18039A, by the ERC (853489 - DEXIM),
by EXC number 2064/1 – project number 390727645. Shyamgopal Karthik and
Karsten Roth thank the International Max Planck Research School for
Intelligent Systems (IMPRS-IS) for support. Luca Eyring and Karsten Roth would
also like to thank the European Laboratory for Learning and Intelligent
Systems (ELLIS) PhD program for support.

## References

  * Albergo and Vanden-Eijnden [2023] Michael S Albergo and Eric Vanden-Eijnden.  Building normalizing flows with stochastic interpolants.  In _ICLR_ , 2023.
  * Anderson [1982] Brian D.O. Anderson.  Reverse-time diffusion equation models.  _Stochastic Processes and their Applications_ , 1982.
  * Bakr et al. [2023] Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny.  Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models.  _arXiv preprint arXiv:2304.05390_ , 2023.
  * Bansal et al. [2024] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  In _ICLR_ , 2024.
  * Ben-Hamu et al. [2024] Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman.  D-flow: Differentiating through flows for controlled generation.  In _ICML_ , 2024.
  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  _OpenAI Technical Report_ , 2023.
  * Bhatia and Dangel [2024] Samarth Bhatia and Felix Dangel.  Lowering pytorch’s memory consumption for selective differentiation.  2024\.
  * Black et al. [2024] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.  Training diffusion models with reinforcement learning.  In _ICLR_ , 2024.
  * Chefer et al. [2023] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  In _SIGGRAPH_ , 2023.
  * Chen et al. [2023] Chaofeng Chen, Annan Wang, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin.  Enhancing diffusion models with text-encoder reinforcement learning.  _arXiv preprint arXiv:2311.15657_ , 2023.
  * Chen et al. [2024a] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.  Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation.  _arXiv preprint arXiv:2403.04692_ , 2024a.
  * Chen et al. [2024b] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  In _ICLR_ , 2024b.
  * Chen et al. [2018] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud.  Neural ordinary differential equations.  In _NeurIPS_ , 2018.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _NIPS_ , 2017.
  * Clark et al. [2024] Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet.  Directly fine-tuning diffusion models on differentiable rewards.  In _ICLR_ , 2024.
  * Deng et al. [2024] Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, and Tingbo Hou.  Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models.  In _CVPR_ , 2024.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _NeurIPS_ , 2021.
  * Dong et al. [2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang.  Raft: Reward ranked finetuning for generative foundation model alignment.  _TMLR_ , 2023.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.  Scaling rectified flow transformers for high-resolution image synthesis.  _arXiv preprint arXiv:2403.03206_ , 2024.
  * Fan et al. [2023] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Reinforcement learning for fine-tuning text-to-image diffusion models.  _NeurIPS_ , 2023.
  * Feng et al. [2023a] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  In _ICLR_ , 2023a.
  * Feng et al. [2023b] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Layoutgpt: Compositional visual planning and generation with large language models.  In _NeurIPS_ , 2023b.
  * Gadre et al. [2023] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  _NeurIPS_ , 2023.
  * Gandelsman et al. [2022] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros.  Test-time training with masked autoencoders.  In _NeurIPS_ , 2022.
  * Ghosh et al. [2023] Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt.  Geneval: An object-focused framework for evaluating text-to-image alignment.  In _NeurIPS_ , 2023.
  * Griffith et al. [2013] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz.  Policy shaping: Integrating human feedback with reinforcement learning.  _NIPS_ , 2013.
  * Guo et al. [2024a] Jianshu Guo, Wenhao Chai, Jie Deng, Hsiang-Wei Huang, Tian Ye, Yichen Xu, Jiawei Zhang, Jenq-Neng Hwang, and Gaoang Wang.  Versat2i: Improving text-to-image models with versatile reward.  _arXiv preprint arXiv:2403.18493_ , 2024a.
  * Guo et al. [2024b] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  In _ICLR_ , 2024b.
  * He et al. [2024] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, et al.  Manifold preserving guided diffusion.  In _ICLR_ , 2024.
  * Hessel et al. [2022] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Huang et al. [2023] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.  T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.  In _NeurIPS_ , 2023.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Isajanyan et al. [2024] Arman Isajanyan, Artur Shatveryan, David Kocharyan, Zhangyang Wang, and Humphrey Shi.  Social reward: Evaluating and enhancing generative ai through million-user feedback from an online creative community.  In _ICLR_ , 2024.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  In _NeurIPS_ , 2022.
  * Karthik et al. [2023] Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata.  If at first you don’t succeed, try, try again: Faithful diffusion-based text-to-image generation by selection.  _arXiv preprint arXiv:2305.13308_ , 2023.
  * Karunratanakul et al. [2024] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakorn, and Siyu Tang.  Optimizing diffusion noise can serve as universal motion priors.  In _CVPR_ , 2024.
  * Kim et al. [2024] Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon.  Consistency trajectory models: Learning probability flow ode trajectory of diffusion.  2024\.
  * Kingma et al. [2021] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  In _NeurIPS_ , 2021.
  * Kirstain et al. [2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  In _NeurIPS_ , 2023.
  * Lee et al. [2023] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu.  Aligning text-to-image models using human feedback.  _arXiv preprint arXiv:2302.12192_ , 2023.
  * Li et al. [2024a] Jiachen Li, Weixi Feng, Wenhu Chen, and William Yang Wang.  Reward guided latent consistency distillation, 2024a.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , 2022.
  * Li et al. [2024b] Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka.  Aligning diffusion models by optimizing human utility.  _arXiv preprint arXiv:2404.04465_ , 2024b.
  * Li et al. [2024c] Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev, Dhritiman Sagar, Yanzhi Wang, Sergey Tulyakov, and Jian Ren.  Textcraftor: Your text encoder can be image quality controller.  In _CVPR_ , 2024c.
  * Lian et al. [2024] Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.  Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.  _TMLR_ , 2024.
  * Lipman et al. [2023] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.  Flow matching for generative modeling.  In _ICLR_ , 2023.
  * Liu et al. [2022] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum.  Compositional visual generation with composable diffusion models.  In _ECCV_ , 2022.
  * Liu et al. [2023] Xingchao Liu, Chengyue Gong, and Qiang Liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  In _ICLR_ , 2023.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Ma et al. [2024] Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie.  Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers, 2024.
  * Marion et al. [2024] Pierre Marion, Anna Korba, Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Arnaud Doucet, Felipe Llinares-López, Courtney Paquette, and Quentin Berthet.  Implicit diffusion: Efficient optimization through stochastic sampling.  2024\.
  * Meiri et al. [2023] Barak Meiri, Dvir Samuel, Nir Darshan, Gal Chechik, Shai Avidan, and Rami Ben-Ari.  Fixed-point inversion for text-to-image diffusion models.  _arXiv preprint arXiv:2312.12540_ , 2023.
  * Otani et al. [2023] Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh.  Toward verifiable and reproducible human evaluation for text-to-image generation.  In _CVPR_ , 2023.
  * Park et al. [2021] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach.  Benchmark for compositional text-to-image synthesis.  In _NeurIPS Datasets and Benchmarks Track_ , 2021.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _NeurIPS_ , 2019.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers, 2023.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
  * Pontryagin [1987] L.S. Pontryagin.  _Mathematical Theory of Optimal Processes_.  Classics of Soviet Mathematics. 1987.  ISBN 9782881240775.  URL https://books.google.de/books?id=kwzq0F4cBVAC.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion, 2022.
  * Prabhudesai et al. [2023] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki.  Aligning text-to-image diffusion models with reward backpropagation.  _arXiv preprint arXiv:2310.03739_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _NeurIPS_ , 2023.
  * Ren et al. [2024] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao.  Hyper-sd: Trajectory segmented consistency model for efficient image synthesis, 2024.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _NeurIPS_ , 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _CVPR_ , 2023.
  * Samuel et al. [2023] Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik.  Norm-guided latent space exploration for text-to-image generation.  In _NeurIPS_ , 2023.
  * Samuel et al. [2024] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik.  Generating images of rare concepts using pre-trained diffusion models.  In _AAAI_ , 2024.
  * Sauer et al. [2023a] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation, 2023a.
  * Sauer et al. [2023b] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation.  _arXiv preprint arXiv:2311.17042_ , 2023b.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _NeurIPS_ , 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _ICLR_ , 2021a.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _ICLR_ , 2021b.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  In _ICML_ , 2023.
  * Sun et al. [2020] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In _ICML_ , 2020.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wallace et al. [2023] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik.  End-to-end diffusion latent optimization improves classifier guidance.  In _ICCV_ , 2023.
  * Wallace et al. [2024] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik.  Diffusion model alignment using direct preference optimization.  In _CVPR_ , 2024.
  * Wang et al. [2024a] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin.  Compositional text-to-image synthesis with attention map control of diffusion models.  In _AAAI_ , 2024a.
  * Wang et al. [2024b] Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li.  Divide and conquer: Language models can plan and self-correct for compositional text-to-image generation.  _arXiv preprint arXiv:2401.15688_ , 2024b.
  * Wang et al. [2022] Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau.  Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models.  _arXiv preprint arXiv:2210.14896_ , 2022.
  * Wang et al. [2023] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu.  Tokencompose: Grounding diffusion with token-level supervision, 2023.
  * Wu et al. [2023a] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li.  Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.  _arXiv preprint arXiv:2306.09341_ , 2023a.
  * Wu et al. [2023b] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Better aligning text-to-image models with human preference.  In _ICCV_ , 2023b.
  * Xu et al. [2023] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  In _NeurIPS_ , 2023.
  * Yang et al. [2024] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li.  Using human feedback to fine-tune diffusion models without any reward model.  In _CVPR_ , 2024.
  * Yin et al. [2023] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation.  _arXiv preprint arXiv:2311.18828_ , 2023.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _TMLR_ , 2022.
  * Yuksekgonul et al. [2022] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In _ICLR_ , 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023.
  * Zhang et al. [2024] Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk.  Large-scale reinforcement learning for diffusion models.  _arXiv preprint arXiv:2401.12244_ , 2024.

## Appendix

##  Appendix A Further Qualitative Results

Here, we report further qualitative results. We separate them into $512\times
512$ and $1024\times 1024$ generated images. First, in Figure 7, we show
examples of ReNO applied to SD-Turbo, SDXL-Turbo, and PixArt-$\alpha$ DMD.
Then, in Figures 8 and 9, we report HyperSDXL + ReNO against competing
methods. Broadly, we see that ReNO not only fixes the artifacts occurring in
one-step models but also improves compositional understanding (color
attribution, spatial reasoning) as well as the quality of generated faces and
pushes current one-step models to be broadly on par with proprietary models in
these settings.

Figure 7: Comparison of images generated with and without ReNO at $512\times
512$ resolution across various one-step models and SD v2.1. The noise used to
generate the initial image is the same one that is used to initialize ReNO.
Figure 8: Images generated with and without ReNO using HyperSDXL at
$1024\times 1024$ resolution compared to competing T2I models SDXL, SD3, and
DALL-E 3. ReNO helps to fix artifacts and generates images of comparable
quality to even closed-source models. The noise used to generate the initial
image is the same one that is used to initialize ReNO. Figure 9: Comparison of
generated images from different models (HyperSD, HyperSD + ReNO, SDXL (50
steps), SD3, DALLE-3) for various prompts. Each row corresponds to a specific
prompt, and each column represents a different model.

##  Appendix B Further Quantitative Results

###  B.1 Comparison to Direct Preference Optimization

Direct Preference Optimization has recently been applied in the context of
Diffusion models [8, 90, 45]. Here, we compare against an SDXL model that has
been preference-tuned on a dataset of over 800k preferences in Table 5. We see
that while DPO improves both attribute binding and the aesthetic score of the
generated images, it underperforms the SDXL-Turbo with ReNO. This highlights
the potential of test-time/online optimization compared to traditional fine-
tuning, since it can generalize much better to unseen prompt distributions.

Table 5: Comparison of ReNO and Direct Preference Optimization (DPO) with a SDXL-based model. SDXL Base result taken from [12]. Method | Attribute Binding |
---|---|---
| Color $\uparrow$ | Shape $\uparrow$ | Texture $\uparrow$ | Aesthetic
SDXL Base [59] | 0.6369 | 0.5408 | 0.5637 | 5.604
DPO-SDXL [82] | 0.6793 | 0.5316 | 0.6513 | 5.687
SDXL-Turbo + ReNO | 0.7800 | 0.5955 | 0.7396 | 6.024
Table 6: Quantitative Results on T2I-CompBench. Full comparison against
different Compositional Text-to-Image methods. The best value is bolded, and
the second-best value is underlined. Results for compositional methods taken
from [56, 84].

Model | Attribute Binding | Object Relationship | Complex$\uparrow$
---|---|---|---
Color $\uparrow$ | Shape$\uparrow$ | Texture$\uparrow$ | Spatial$\uparrow$ | Non-Spatial$\uparrow$
SD2.1 | 0.5065 | 0.4221 | 0.4922 | 0.1342 | 0.3096 | 0.3386
\+ Composable Diffusion [49] | 0.4063 | 0.3299 | 0.3645 | 0.0800 | 0.2980 | 0.2898
\+ Attn-Mask-Control [83] | 0.4119 | 0.4649 | 0.4505 | 0.1249 | 0.3046 | 0.3779
\+ StructureDiffusion [22] | 0.4990 | 0.4218 | 0.4900 | 0.1386 | 0.3111 | 0.3355
\+ TokenCompose [86] | 0.5055 | 0.4852 | 0.5881 | 0.1815 | 0.3173 | 0.2937
\+ Attn-Exct [9] | 0.6400 | 0.4517 | 0.5963 | 0.1455 | 0.3109 | 0.3401
\+ GORS [33] | 0.6603 | 0.4785 | 0.6287 | 0.1815 | 0.3193 | 0.3328
SD-Turbo + ImageSelect [37] | 0.7222 | 0.5552 | 0.6919 | 0.2216 | 0.3154 | 0.4618
(1) PixArt-$\alpha$ DMD | 0.3824 | 0.3414 | 0.4691 | 0.1906 | 0.3060 | 0.3643
(1) + ReNO (Ours) | 0.6454 | 0.5658 | 0.7186 | 0.2508 | 0.3138 | 0.4554
(2) SD-Turbo | 0.5513 | 0.4448 | 0.5690 | 0.1739 | 0.3101 | 0.4052
(2)+ ReNO (Ours) | 0.7830 | 0.6244 | 0.7466 | 0.2235 | 0.3161 | 0.4829
(3) SDXL-Turbo | 0.6149 | 0.4366 | 0.6001 | 0.2401 | 0.3118 | 0.4250
(3)) + ReNO (Ours) | 0.7800 | 0.5955 | 0.7396 | 0.2551 | 0.3147 | 0.4690
(4) HyperSDXL | 0.6535 | 0.4956 | 0.6496 | 0.2509 | 0.3108 | 0.4582
(4) + ReNO (Ours) | 0.7904 | 0.6324 | 0.7671 | 0.2616 | 0.3145 | 0.4766

###  B.2 Compositional Text-to-Image Methods.

We show the results for several methods that have been tailored to
compositional Text-to-Image generation in Table 6. These methods either
explicitly finetune the model for improved compositional generation, or modify
the inference process, or repeat the sampling over multiple iterations. We see
that ReNO consistently outperforms specific methods tailored for this task.

We also note that some methods use LLMs and other tools (image-editing,
customization etc.) to plan out or correct generations [47, 84, 23]. However,
these methods significantly impact the generation process through, e.g.,
iterative generation and planning. In contrast, ReNO only changes the initial
noise and doesn’t alter the generative model at all. Thus, our method could
also be incorporated into these tools to further improve performance.

##  Appendix C Analysis of Reward Models

We show all results for all combinations of the reward models in Table 7.
Broadly, adding all reward models ensures that a meaningful improvement is
achieved both on attribute binding and on the aesthetic score. In addition to
this, we perform a leave-one-out analysis on Parti-Prompts in Table 8, where
one reward is excluded from ReNO and subsequently analyzed.

Even when a particular reward is not optimized for, we see that there is a
consistent improvement in the metrics, and in most cases, at least 80% of the
images improve even on the left-out reward. This phenomenon across a variety
of models (e.g. CLIP, BLIP) trained on differing datasets certainly indicates
that there are significant improvements made by ReNO across most of the
images. While the reward increase and the percental improvement, can differ
based on the one-step model, CLIPScore and ImageReward seem to be less
correlated to the other rewards, which could be explained based on the similar
backbone employed by HPSv2 and PickScore. Interestingly, PixArt-$\alpha$ DMD
achieves the highest reward scores after optimization, which does not follow
the quantitative results for T2I-CompBench and GenEval as reported in Section
4.2.

Table 7: Full results for all different reward model combinations considered in ReNO over the attribute binding categories of T2I-CompBench and the LAION aesthetic score predictor [74]. We highlight the best and second-best results per number of reward models. Reward Models | Attribute Binding | Aesthetic
---|---|---
Color $\uparrow$ | Shape $\uparrow$ | Texture $\uparrow$
Base (SD-Turbo) | 0.5513 | 0.4448 | 0.5690 | 5.647
\+ CLIPScore | 0.6625 | 0.5501 | 0.6621 | 5.475
\+ HPSv2 | 0.6443 | 0.5451 | 0.6859 | 5.752
\+ ImageReward | 0.7720 | 0.6104 | 0.7334 | 5.611
\+ PickScore | 0.6341 | 0.5069 | 0.6242 | 5.711
\+ CLIPScore + HPSv2 | 0.6691 | 0.5664 | 0.6979 | 5.714
\+ CLIPScore + ImageReward | 0.7749 | 0.6218 | 0.7415 | 5.579
\+ HPSv2 + ImageReward | 0.7710 | 0.6228 | 0.7518 | 5.692
\+ PickScore + CLIP | 0.6606 | 0.5500 | 0.6735 | 5.615
\+ PickScore + HPSv2 | 0.6593 | 0.5571 | 0.6766 | 5.776
\+ PickScore + ImageReward | 0.7798 | 0.6298 | 0.7354 | 5.662
\+ CLIPScore + HPSv2 + ImageReward | 0.7735 | 0.6238 | 0.7524 | 5.677
\+ PickScore + CLIPScore + HPSv2 | 0.6886 | 0.5599 | 0.7012 | 5.733
\+ PickScore + CLIPScore + ImageReward | 0.7797 | 0.6218 | 0.7513 | 5.620
\+ PickScore + HPSv2 + ImageReward | 0.7778 | 0.6298 | 0.7457 | 5.713
\+ All | 0.7830 | 0.6244 | 0.7466 | 5.704

##### Reward Weighting.

The four reward models that we employ output scores in different ranges.
Specifically, HPSv2 mostly ranges between $0.2$-$0.4$, while PickScore is in
the range of $20$-$30$ for most of the images. ImageReward is in the range of
$-2$ to $+2$ for the majority of images, and CLIPScore is between 0 and 1. For
all our experiments, we use weights of $1.0$ for ImageReward, $5.0$ for HPSv2,
$0.05$ for PickScore, and $1.0$ for CLIPScore. When each score range is scaled
to $[0,1]$, then these weights correspond to $4.0$ for ImageReward, $1.0$ for
HPSv2, $0.5$ for PickScore, and $1.0$ for CLIPScore. These weights ensure that
the losses from each reward model are roughly similar, with a higher emphasis
on ImageReward.

Table 8: Leave-one-out reward evaluation on Parti-Prompts. The listed reward in the first column is left out in ReNO and subsequently analyzed with respect to its change as well as the percentage of generations where ReNO improves this reward. | Initial Reward $\uparrow$ | ReNO Reward$\uparrow$ | Change $\uparrow$ | Improve % $\uparrow$
---|---|---|---|---
PixArt-$\alpha$ DMD |  |  |  |
CLIPScore [0, 1] | 0.332 | 0.386 | +0.054 | 95.1
PickScore [20, 30] | 22.235 | 23.788 | +1.553 | 97.6
HPSv2 [0.2, 0.4] | 0.281 | 0.324 | +0.043 | 97.7
ImageReward [-2, 2] | 0.896 | 1.367 | +0.471 | 88.7
SD-Turbo |  |  |  |
CLIPScore [0, 1] | 0.353 | 0.386 | +0.033 | 80.4
PickScore [20, 30] | 22.028 | 23.232 | +1.204 | 91.1
HPSv2 [0.2, 0.4] | 0.266 | 0.310 | +0.044 | 94.7
ImageReward [-2, 2] | 0.552 | 1.243 | +0.691 | 90.3
SDXL-Turbo |  |  |  |
CLIPScore [0, 1] | 0.360 | 0.379 | +0.019 | 74.2
PickScore [20, 30] | 22.505 | 23.185 | +0.680 | 82.3
HPSv2 [0.2, 0.4] | 0.280 | 0.310 | +0.030 | 93.4
ImageReward [-2, 2] | 0.920 | 1.270 | +0.350 | 81.0

##  Appendix D User Study

We perform our user study on Amazon Mechanical Turk, and pay participants
based on prior guidelines [55], which also ensures the compensation is above
the minimum wage. We use pairwise preferences due to its simplicity, allowing
users to mark ties between images that are equally good/bad. Each pairwise
comparison is treated as an individual entity and handed to an individual user
to minimize user biases. In particular, each pairwise comparison between the
two models has involved at least 339 unique users (and 673 maximal), with the
average being 495. To reduce the number of user comparisons, we perform the
user study on a subset of Parti-Prompts totaling slightly above 1000 prompts
(excluding challenges [’Basic’, ’Imagination’, ’Perspective’, ’Linguistic
Structures’] and categories [’Abstract’, ’Indoor Scenes’, ’Produce &
Plants’]).

We ask users to answer the following three questions:

  * •

On personal preference: "Which image would you personally prefer getting given
the input text (based on your personal tradeoff between faithfulness and
aesthetics)?"

  * •

On aestheticness: "Which image do you find more aesthetically pleasing?

  * •

On faithfulness: "Which image is more faithful to the input text?"

We also provide additional information on terminology:

  * •

Faithfulness: The generated image should reflect all key concepts, their
relations and their attributes given in the text prompt.

  * •

Aestheticness: Refers to the style, coloring and interpretation in the
depiction of concepts (i.e. "looks better").

  * •

Personal preference: Some generations can be more faithful, but less
aesthetic, or the other way around. Choose which you prefer :).

(a) User Preferences for Image Quality

(b) User Preferences for Prompt Faithfulness

Figure 10: User study results on aestheticness and faithfulness based on
Parti-Prompts.

For the competing methods, we use default parameters: one-step generation
without classifier-free guidance (CFG), and $\text{CFG}=7.5$ for SD2.1 and
$\text{CFG}=5.0$ for SDXL. For the proprietary SD3, we generate images through
the API provided at https://platform.stability.ai/. Note that to compare SDXL
with SD-Turbo + ReNO, we generate images in $1024\times 1024$ for SDXL as this
is its native resolution and then afterward downsize them to $512\times 512$.

In Figure 10, we report the results for the specific questions on faithfulness
and aestheticness. Interestingly, for all models, the preference for
aestheticness is even larger than that of faithfulness. While the quantitative
results on T2I-CompBench and GenEval reported in Section 4.2 mainly benchmark
the prompt following improvements of ReNO, this result confirms ReNO’s
benefits in improving the general quality of generated images.

##  Appendix E Algorithm

We outline the overall algorithm for ReNO in Algorithm 1. Note that, we choose
gradient ascent with Nesterov momentum as we found this for our computational
budget to yield the best results. Although line-search-based methods such as
L-BFGS are viable options [5], we find that even without them gradient ascent
provides efficient and effective optimization of the criterion function.
However, L-BFGS or gradient ascent without momentum might also be viable
optimization methods for ReNO.

Algorithm 1 ReNO

Input: p (prompt), $G_{\theta}$ (One-Step T2I Model),
$\mathcal{R}_{\psi}^{0,1...n}$ (Reward Functions), $\lambda_{0,1...n}$ (Reward
Weights), $m$ (# Optimization Steps), $\eta$ (Learning Rate),
$\lambda_{\text{reg}}$ (Regularization Strength)

Initialize $v_{-1}=0.0$, ${\bm{\varepsilon}}^{0}=\mathcal{N}(0,\mathbf{I})$,
$R^{\star}=-\inf$.

for $t=0$ to $m$ do

Generate image ${\mathbf{x}}_{0}^{t}=G_{\theta}({\bm{\varepsilon}}^{t},p)$

Compute reward-based criterion
$R^{t}=\sum\nolimits^{n}_{i}\lambda_{i}\mathcal{R}^{i}_{\psi}({\mathbf{x}}_{0}^{t},\textnormal{p})$

$\text{grad}_{t}=\nabla_{{\bm{\varepsilon}}^{t}}[\lambda_{\text{reg}}K({\bm{\varepsilon}}^{t})+R^{t}]$

$\text{grad}_{t}=\text{GradNormClip}(\text{grad}_{t},0.1)$

$v_{t}=0.9\cdot v_{t-1}+\eta\cdot\text{grad}_{t}$

${\bm{\varepsilon}}^{t+1}={\bm{\varepsilon}}^{t}+v_{t}$

if $R^{t}>R^{\star}$ then

${\mathbf{x}}_{0}^{\star}={\mathbf{x}}_{0}^{t}$, $R^{\star}=R^{t}$

end if

end for

return ${\mathbf{x}}_{0}^{\star}$

##  Appendix F Implementation Details

Our code is built with Pytorch [57] and is mainly based on the diffusers
library [80]. It is available at https://github.com/ExplainableML/ReNO.

###  F.1 ReNO hyperparameters

As detailed in Algorithm 1 the main hyperparameters in ReNO are the learning
rate $\mu$, the regularization strength $\lambda_{\text{reg}}$ and the choice
for reward models, which we explore in Appendix C. We use
$\lambda_{\text{reg}}=0.01$ for all our experiments. For the learning rate, we
use $\mu=5$ for all our $512\times 512$ models and $\mu=10$ for HyperSDXL that
generates $1024\times 1024$ as we found this to give a good balance between
exploration, improvements, and fast convergence. Note that in combination with
gradient norm clipping, this also prevents major changes in the noise that
would completely change the generated image. This effect can be observed in
Figures 1 and 6, and Appendix A, as the image after ReNO optimization still
shares significant details with the initially generated image.

###  F.2 Models

SD-Turbo, SDXL-Turbo [72], and HyperSDXL [65] are built with a UNet [67]
architecture similar to the one proposed in Rombach et al. [66]. On the other
hand, PixArt-$\alpha$ DMD [12, 11] leverages a Diffusion Transformer [19, 58,
52] based architecture. We use the checkpoints of SD-Turbo, SDXL-Turbo,
HyperSDXL, and PixArt-$\alpha$ DMD supplied through huggingface. For
HyperSDXL, we use the one-step UNet checkpoint (as opposed to the LoRA
version).

###  F.3 Rewards

In this work, we employ the four following reward models for ReNO.

####  F.3.1 Human Preference Score v2 (HPSv2)

HPSv2 [87] is an improved version of the HPS [88] model, which uses an
OpenCLIP ViT-H/14 model and is trained on prompts collected from DiffusionDB
[85] and other sources. Note that here we employ the further improved HPSv2.1
checkpoint.

####  F.3.2 PickScore

PickScore also uses the same ViT-H/14 model, however is trained on the Pick-a-
Pic dataset which consists of 500k+ preferences that are collected through
crowd-sourced prompts and comparisons.

####  F.3.3 ImageReward

ImageReward [89] trains a MLP over the features extracted from a BLIP model
[44]. This is trained on a dataset of images collected from the DiffusionDB
[85] prompts.

####  F.3.4 CLIPScore

Lastly, we use CLIPScore [63, 31], which was not designed specifically as a
human preference reward model. However, it measures the text-image alignment
with a score between 0 and 1. Thus, it offers a way of evaluating the prompt
faithfulness of the generated image that can be optimized. We use the model
provided by OpenCLIP [34] with a ViT-H/14 backbone.

###  F.4 Metrics

Apart from the user study (details in Appendix D) and the reward models
themselves in Table 8, we benchmark ReNO with three different evaluation
schemes as detailed in the following.

####  F.4.1 T2I-CompBench

T2I-CompBench is a comprehensive benchmark proposed by Park et al. [56] for
evaluating the compositional capabilities of text-to-image generation models.
The benchmark consists of three categories and six sub-categories of
compositional text prompts: (1) Attribute binding, which includes color,
shape, and texture sub-categories, where the model should bind the attributes
with the correct objects to generate the complex scene; (2) Object
relationships, which includes spatial and non-spatial relationship sub-
categories, where the prompts contain at least two objects with specified
relationships; and (3) Complex compositions, where the prompts contain more
than two objects or more than two sub-categories. The attribute binding
subtasks are evaluated using BLIP-VQA (i.e., generating questions based on the
prompt and applying VQA on the generated image), spatial relationships are
evaluated using an object detector, non-spatial relationships are evaluated
through CLIPScore (CLIP ViT-B/32), and complex compositions are evaluated
using all three models.

####  F.4.2 GenEval

GenEval is an object-focused framework introduced by Ghosh et al. [26] for
evaluating the alignment between text prompts and generated images from Text-
to-Image (T2I) models. Unlike holistic metrics such as FID or CLIPScore,
GenEval leverages existing object detection methods to perform a fine-grained,
instance-level analysis of compositional capabilities. The framework assesses
various aspects of image generation, including object co-occurrence, position,
count, and color. By linking the object detection pipeline with other
discriminative vision models, GenEval can further verify properties like
object color. All the metrics on the GenEval benchmarks are evaluated using a
MaskFormer object detection model with a Swin Transformer [51] backbone.
Lastly, GenEval is evaluated over four seeds and reports the mean for each
metric, which we follow.

####  F.4.3 LAION Aesthetic Score Predictor

Furthermore, we employ the improved LAION Aesthetic Predictor as an evaluation
metric. It consists of an MLP trained on top of a CLIP [63] backbone.
Importantly, this predictor does not take the prompt as a joint input with the
image. Thus, the aestheticness of an image is always evaluated independently
of what prompt was used to generate it. This predictor can also be used as an
objective to improve the aesthetic quality of generated images, which we
briefly investigated. We found that while numerically, the generated images
achieve a higher score, their actual visual quality does not seem to always be
higher. We hypothesize that this is because the predictor is independent of
the given prompt and thus might be more prone to reward-hacking.

##  Appendix G Broader Impact

Text-to-Image models have a wide variety of uses in different settings. While
they can be used for harmful purposes, practcal deployments of these models
(including ours) must be made with a safety checker/filter to prevent the
generation of NSFW content. In our work, we rely on existing pretrained
models, and therefore would inhereit its biases. However, we believe that our
reward optimization framework is flexible to also include safety and fairness
and potential objectives which would be an option to mitigate the harms of
existing image generation models.