# Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion
Models

Chen Henry Wu, Fernando De la Torre
Carnegie Mellon University, Pittsburgh, PA
<EMAIL_ADDRESS>
###### Abstract

Text-to-image diffusion models have achieved remarkable performance in image
synthesis, while the text interface does not always provide fine-grained
control over certain image factors. For instance, changing a single token in
the text can have unintended effects on the image. This paper shows a simple
modification of classifier-free guidance can help disentangle image factors in
text-to-image models. The key idea of our method, Contrastive Guidance, is to
characterize an intended factor with two prompts that differ in minimal
tokens: the positive prompt describes the image to be synthesized, and the
baseline prompt serves as a “baseline” that disentangles other factors.
Contrastive Guidance is a general method we illustrate whose benefits in three
scenarios: (1) to guide domain-specific diffusion models trained on an object
class, (2) to gain continuous, rig-like controls for text-to-image generation,
and (3) to improve the performance of zero-shot image editors. Code available
at: https://github.com/humansensinglab/contrastive-guidance

Figure 1: This paper explores disentangled control for text-to-image diffusion
models. We show that using a pair of contrastive prompts can (1) guide domain
experts (e.g., trained on an object class) with text, (2) enable rig-like
control for text-to-image diffusion models (similar to steering a StyleGAN),
and (3) improve text-to-image diffusion-based image editing methods.

##  1 Introduction

Many recent methods for text-to-image synthesis are built upon diffusion
models trained on large-scale datasets (e.g., GLIDE [38], DALL$\cdot$E 2 [44],
Imagen [47], and Stable Diffusion [45]). Besides generating images with
unprecedented visual quality, these models also show remarkable generalization
ability to novel texts with complex compositions of concepts.

Prompt engineering and guidance [6, 16] are two (overlapping) directions for
steering the outputs of these text-to-image models. Prompt engineering focuses
on how to design better text as input to the model, while guidance focuses on
how the image distribution can be modified by text. However, prompt
engineering and previous guidance methods [6, 16, 38, 29] still face
challenges in fine-grained control over image factors. They can affect many
image factors, not only what is intended. Thus, in this paper, we raise the
following question. How can we steer the outputs of text-to-image diffusion
models over certain image factors, without affecting other factors?

In this paper, we show that a simple modification of classifier-free guidance
can help disentangle image factors in text-to-image diffusion models. Our key
idea is to instantiate an intended image factor with two text prompts that
differ only in several tokens: the positive prompt describes the image to be
synthesized, while the baseline prompt serves as a “baseline” that
disentangles other factors. The difference between the two score functions
conditioned on the two prompts is used as guidance. For instance, suppose we
already have a diffusion model for high-quality cat images. To synthesize a
cat on the beach, one can set the positive prompt as a portrait photo of a cat
on the beach, and the baseline prompt as a portrait photo of a cat. The
baseline prompt helps the model focus on on the beach, preventing it from
contributing to what a portrait photo of cat looks like.

Besides the above intuition, we also formalize Contrastive Guidance in a
probabilistic framework. Specifically, by applying the Bayes rule, we show
that the positive and baseline prompts together build a binary generative
classifier over the noisy images. This binary generative classifier can be
used to guide the denoising process following classifier guidance [6]. Note
that, in classifier guidance, the score function of the classifier is computed
by backpropagating through the classifier. Different from them, we show that
our generative classifier has a gradient-free solution that corresponds to the
score functions of the two prompts, and Contrastive Guidance is proportional
to this solution up to a constant.

Since our derivation is based on the unified SDE/ODE formulation [51],
Contrastive Guidance can be plugged into different diffusion models and
samplers/schedulers in various scenarios:

  * •

Guiding Domain Expert (Figure 1 top): Given a domain expert trained on high-
quality cat faces, how can we combine the domain knowledge of the domain
expert with the generalization ability of the text-to-image model? We show
that compared with classifier-free guidance [16], our Contrastive Guidance can
better disentangle the domain knowledge and text condition.

  * •

Continuous, Rig-like Control (Figure 1 middle): How could we gain continuous,
rig-like control over the color while maintaining others? It helps synthesize
the same object with interpolated color.

  * •

Improving Image Editing (Figure 1 bottom): Previous works have used text-to-
image diffusion models for image editing [14, 55, 35, 25]. Disentanglement is
the key to this application since the user only wants the intended edit (e.g.,
adding people) without altering other factors (e.g., the car and the
background).

Our experiments verify these applications. Furthermore, our analysis
demonstrates that our Contrastive Guidance is more disentangled than
classifier-free guidance [16].

##  2 Related Work

#### Text-to-image diffusion model:

Recent years have witnessed unprecedented progress in text-to-image synthesis,
driven by large generative models such as diffusion models [17, 50, 51, 6]
(e.g., GLIDE [38], DALL$\cdot$E 2 [44], Imagen [47], and Stable Diffusion
[45]) and VQ Transformers [9] (e.g., DALL$\cdot$E [43], CogView [7, 8], and
Parti [57]). This paper studies how to extract disentangled image factors from
text-to-image diffusion models.

#### Guidance for diffusion models.

Guidance methods modify the output distribution of pre-trained diffusion
models, based on additional inputs such as class labels [6], text [38], and
corrupted images [24, 32]. The first guidance method is classifier guidance
[6], for which a class classifier is finetuned on noisy images. Similarly,
CLIP guidance [38, 30] finetunes a CLIP model [42] to support text input. To
avoid finetuning classifiers or CLIP, classifier-free guidance (CFG) [16]
jointly trains a conditional and an unconditional diffusion model and combines
their score estimates, and CFG has become the default for text-to-image tasks
[38, 45]. To compose multiple texts, composable diffusion [29] combines score
estimates with different text inputs. Besides user-specified conditions,
several works showed that even guidance based on model outputs [3] or
representations [18] can improve the quality of images. In this paper, we
explore how to disentangle image factors with texts to gain fine-grained
control.

#### Image editing with diffusion models.

Recent works have shown that diffusion models are capable of unpaired image-
to-image translation [35, 52, 4, 55]. A more recent trend of works have
explored zero-shot image editing with text-to-image diffusion models [55, 14,
25]. One of the applications of our Contrastive Guidance is to improve the
intended edit of some of these zero-shot image editors.

#### Guidance for other generative models.

Guidance has also been widely studied for GANs [12] and autoregressive
language models (LMs). For GANs, the conditions for guidance include class
labels [37, 39] text prompts [41, 56], corrupted images [36], and de-biasing
[19, 56]. Although LMs have very different formulations from generative models
for images, similar ideas to ours have been explored. For instance, GeDi [26]
uses a small LM with two contrastive prompts to guide a large LM, and
contrastive decoding [27] leverages a small language model to correct mistakes
by a large LM.

##  3 Method

###  3.1 Background on Diffusion Models

Diffusion models sample data via iterative denoising. Suppose noises are added
to the image $\bm{x}$ via
$\mathrm{d}\bm{x}=\bm{f}(\bm{x},t)\mathrm{d}t+g(t)\mathrm{d}\bm{w}$, where
$t\in[0,T]$ is the time step, and $\bm{f}$ and $g$ are pre-defined functions
that specify how exactly the noises are added. It has been shown that the
denoising problem can be formulated by the following SDE [51]:

$\mathrm{d}\bm{x}=\Big{[}\bm{f}(\bm{x},t)-g(t)^{2}\nabla_{\bm{x}}\log
p_{t}(\bm{x})\Big{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},$ (1)

where $\nabla_{\bm{x}}\log p_{t}(\bm{x})$ is the score function, the core of
diffusion models, which provides information about the denoising direction.
Besides the SDE-based formulation, previous works have also proposed non-
Markovian [49, 54] and ODE-based formulations [51, 49, 48, 28, 31, 23, 58].
Since the true score function is inaccessible, diffusion models learn to
approximate it $\nabla_{\bm{x}}\log
p_{t}(\bm{x})\approx{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}(\bm{x},t)}$.
For instance, we can plug it into the SDE and get the denoising process:

$\mathrm{d}\bm{x}=\Big{[}\bm{f}(\bm{x},t)-g(t)^{2}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}(\bm{x},t)}\Big{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.$
(2)

This connection between diffusion models and differential equations provides a
unified formulation for guided image synthesis. Given a condition $c$, one
only need to compute the conditional score function $\nabla_{\bm{x}}\log
p_{t}(\bm{x}|c)$ and replace $\nabla_{\bm{x}}\log p_{t}(\bm{x})$ with
$\nabla_{\bm{x}}\log p_{t}(\bm{x}|c)$ in the denoising process. For instance,
we can plug it into the above SDE and get the conditional denoising process
[51]:

$\mathrm{d}\bm{x}=\Big{[}\bm{f}(\bm{x},t)-g(t)^{2}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|c)\Big{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.$ (3)

The true conditional score function $\nabla_{\bm{x}}\log p_{t}(\bm{x}|c)$ is
also inaccessible, and each guidance method aims to approximate it. For
instance, given a text-to-image diffusion model $\bm{s}_{\bm{\theta}}$ and a
text $\bm{y}$ as the condition, classifier-free guidance (CFG) [16] models the
conditional denoising process as

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}\tau_{t}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y})\
-\\\
&\quad\quad\quad\quad\quad\quad(\tau_{t}-1)\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},\end{split}$
(4)

where $\varnothing$ is an empty string and $\tau_{t}$ is a hyperparameter
indicating the guidance strength. It implies that CFG approximates
$\nabla_{\bm{x}}\log
p_{t}(\bm{x}|c)\approx\tau_{t}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y})-(\tau_{t}-1)\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)$.

###  3.2 Contrastive Guidance

We introduce Contrastive Guidance, which uses text-to-image diffusion models
to steer certain image factors while not affecting others. Our key idea is to
instantiate a condition $c$ with two prompts $(\bm{y}^{+},\bm{y}^{-})$ that
differ in a few tokens: the positive prompt $\bm{y}^{+}$ describes the
intended image, while the baseline prompt $\bm{y}^{-}$ serves as a “baseline”
that disentangles other factors.111Although we use $\bm{y}^{-}$ to denote the
baseline prompt, it does not need to be a “negative prompt”. That is, the
baseline prompt need not describe what not to generate. See the following text
for an example.  Suppose we have a domain expert that can generate high-
quality images of cats. To generate a cat with eyeglasses, one can set the
positive prompt $\bm{y}^{+}$ as a portrait photo of a cat with eyeglasses, and
the baseline prompt $\bm{y}^{-}$ as a portrait photo of a cat without
eyeglasses, or simply, a portrait photo of a cat. The baseline prompt helps
the model focus on with eyeglasses, preventing it from contributing to what a
portrait photo of a cat looks like. In this example, cats are better modeled
by the domain expert.

Figure 2:  Overview of Contrastive Guidance. Given a model to be guided,
Contrastive Guidance use two prompts with minimal differences as guidance
signal, which helps disentangle the intended image factors. ∗The guided model
can be a domain expert, the text-to-image model itself, or an image editing
method (Section 3.3).

In the following, we formalize the above intuition. Our goal is to derive
$\nabla_{\bm{x}}\log p_{t}(\bm{x}|c)$. First, we rewrite $p_{t}(\bm{x}|c)$ as

$\displaystyle p_{t}(\bm{x}|c)\propto p_{t}(\bm{x})p_{t}(c|\bm{x}).$ (5)

We will discuss the model to be guided $p_{t}(\bm{x})$ later, which has
several variations. The key idea of our formulation is to rewrite the term
$p_{t}(c|\bm{x})$ as a generative classifier using the Bayes rule. It is a
binary classifier that infers whether the noisy $\bm{x}$ is from the positive
prompt $\bm{y}^{+}$ or the baseline prompt $\bm{y}^{-}$. Formally, we define
the generative classifier as

$\displaystyle
p_{t}(c|\bm{x}):=\frac{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}},$
(6)

where the temperature $\gamma_{t}$ controls the sharpness of the distribution.
In the following, we will combine Eq. (5) and Eq. (6), using simplified
notations: $p^{+}:=p(\bm{y}^{+})$, $p^{-}:=p(\bm{y}^{-})$,
$p_{t}^{+}(\bm{x}):=p_{t}(\bm{x}|\bm{y}^{+})$, and
$p_{t}^{-}(\bm{x}):=p_{t}(\bm{x}|\bm{y}^{-})$. Using the log-derivative trick,
we have (see details in Appendix A.1):

$\displaystyle\begin{split}&\quad\ \nabla_{\bm{x}}\log p_{t}(\bm{x}|c)\\\
&=\nabla_{\bm{x}}\log\Big{(}p_{t}(\bm{x})\frac{p^{+}p_{t}^{+}(\bm{x})^{\gamma_{t}}}{p^{+}p_{t}^{+}(\bm{x})^{\gamma_{t}}+p^{-}p_{t}^{-}(\bm{x})^{\gamma_{t}}}\Big{)}\end{split}$
(7) $\displaystyle\begin{split}&=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\nabla_{\bm{x}}\log p_{t}^{+}(\bm{x})^{\gamma_{t}}\\\
&\quad\quad\quad\quad\quad\quad-\frac{\nabla_{\bm{x}}\Big{(}p^{+}p_{t}^{+}(\bm{x})^{\gamma_{t}}+p^{-}p_{t}^{-}(\bm{x})^{\gamma_{t}}\Big{)}}{p^{+}p_{t}^{+}(\bm{x})^{\gamma_{t}}+p^{-}p_{t}^{-}(\bm{x})^{\gamma_{t}}}\end{split}$
(8) $\displaystyle\begin{split}&=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\underbrace{\frac{\gamma_{t}\cdot
p^{-}p_{t}^{-}(\bm{x})^{\gamma_{t}}}{p^{+}p_{t}^{+}(\bm{x})^{\gamma_{t}}+p^{-}p_{t}^{-}(\bm{x})^{\gamma_{t}}}}_{\lambda_{t}}\cdot\\\
&\quad\quad\quad\quad\quad\quad\Big{(}\nabla_{\bm{x}}\log
p_{t}^{+}(\bm{x})-\nabla_{\bm{x}}\log p_{t}^{-}(\bm{x})\Big{)}\end{split}$ (9)
$\displaystyle\begin{split}&=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\lambda_{t}\Big{(}\nabla_{\bm{x}}\log p_{t}^{+}(\bm{x})\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-\nabla_{\bm{x}}\log
p_{t}^{-}(\bm{x})\Big{)}\end{split}$ (10)
$\displaystyle\approx{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}(\bm{x},t)}+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\big{)}},$
(11)

where $\bm{s}_{\bm{\theta}}$ is the text-to-image diffusion model. We will
introduce several choices of the score estimate
${\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}(\bm{x},t)}$
in Section 3.3, e.g., domain experts, text-to-image models themselves, and
image editors. Please see Figure 2 for an illustration of how the term is
computed with pretrained text-to-image diffusion models and the two prompt
conditions.

#### (Not) estimating $\lambda_{t}$

A remaining question is how to estimate the coefficient $\lambda_{t}$. Since
$\gamma_{t}$, $p(\bm{y}^{+})$ and $p(\bm{y}^{-})$ do not depend on $\bm{x}$,
the only terms to estimate are $p_{t}(\bm{x}|\bm{y}^{+})$ and
$p_{t}(\bm{x}|\bm{y}^{-})$. We can leverage the fact that each SDE has an ODE
with the same marginal density at each time $t$ [33, 51]. Estimating the
density of ODEs is tractable by calling an ODE solver [2, 13] from $t$ to $T$
(see Appendix A.2). However, this requires calling the ODE solver at all
$t$’s, resulting in the total complexity of $\mathcal{O}(N^{2})$ where $N$ is
the number of discretization steps. To avoid this, this paper assumes that the
temperature $\gamma_{t}$ is adaptive such that $\lambda_{t}$ does not depend
on $\bm{x}$, giving us $\mathcal{O}(N)$ complexity. Further, our experiments
show that a fixed $\lambda_{t}$ for all time $t$ provides reasonable results.

#### Comparison to CFG and negative prompts

It is worth noting that although the derivation of the ultimate solution is
complex, the final expression in Eq. (11) is simple. In terms of the final
expression, our approach can be seen as an extension upon classifier-free
guidance (CFG) [16], where we employ a non-trivial baseline prompt in place of
an empty string. Similarly, negative prompts have become a common practice in
most diffusion model applications. Typical negative prompts are phrases
describing unwanted outputs such as “low quality” and “NSFW”. We would like to
highlight the idea of “specifying the image factor with a contrastive baseline
prompt”, which is why our method achieves better disentanglement than CFG
(Section 4). In the following sections, we will also demonstrate how this
formulation leads to new applications, such as merging a text-to-image
diffusion model and a domain-specific diffusion model, rig-like controls, and
image editing.

###  3.3 Applications

#### Guiding Domain Experts

Text-to-image models sometimes underperform domain-specific models in highly
specialized domains (e.g., faces). To combine the best of both worlds, we can
use Contrastive Guidance to guide domain experts. Suppose we have a domain
expert
${\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}_{\bm{\phi}}(\bm{x},t)}$
for high-quality cat faces. To generate a cat with eyeglasses, one can set
$\bm{y}^{+}$ as a portrait photo of a cat with eyeglasses, and $\bm{y}^{-}$ as
a portrait photo of a cat without eyeglasses, or simply, a portrait photo of a
cat. In this case, the denoising process becomes:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}_{\bm{\phi}}(\bm{x},t)}\
+\\\
&\quad{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\big{)}}\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.\end{split}$
(12)

#### Continuous, Rig-like Control

The text-only input of text-to-image diffusion models does not allow
continuous, rig-like control because it cannot be accurately described by the
text. To gain this control, we can use Contrastive Guidance and change
$\lambda_{t}$ continuously. For instance, when the prompt $\bm{y}$ is a photo
of a waterfall, the positive prompt $\bm{y}^{+}$ can be a beautiful photo of a
waterfall, and the negative prompt $\bm{y}^{-}$ can bea not beautiful photo of
a waterfall. In these cases, the denoising process becomes:222Previous works
[45] have shown that text-to-image models need classifier-free guidance (CFG)
[16] to generate reasonable output. In this paper,
${\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}}$
refers to the score function after CFG. See details in Appendices C and B.2.

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y})}\
+\\\
&\quad{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\big{)}}\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.\end{split}$
(13)

One can tune the value of $\lambda_{t}$ to gain continuous, rig-like control
for text-to-image diffusion models. Specifically, when $\lambda_{t}$ is
positive, the generation is guided towards the positive prompt; when
$\lambda_{t}$ is negative, the generation is guided towards the opposite
direction of the positive prompt.

Figure 3:  Contrastive Guidance improves disentanglement (e.g., attributes,
background, foreground, and objects). Within each row, we fixed all random
variables during the sampling process. The gray-scale images visualize the
norm of the pixel-wise distance between the two images before and after the
guidance.

#### Guiding Image Editors

Zero-shot image editing aims at modifying a real image $\bm{x}$ given a source
text $\bm{y}$ and a target text $\hat{\bm{y}}$ [55]. For instance, suppose one
wants to change the season of a real image, taken from an aerial view, from
autumn to winter. The source text $\bm{y}$ can be an aerial view of autumn
scene, and the target text $\hat{\bm{y}}$ can be an aerial view of winter
scene. Since disentangling image factors is a key to meaningful editing,
Contrastive Guidance can be helpful to this task. In this paper, we focus on
two editors: SDEdit [35] and CycleDiffusion [55]. SDEdit is based on the
finding that, when noises are added and then removed up to a suitable noise
level, the structure of the image can preserve. For zero-shot image editing,
SDEdit first adds noise to the source image $\bm{x}$ to get a noisy image
$\bm{x}_{t_{\text{SD}}}$. Then, conditioned on the target text $\hat{\bm{y}}$,
it denoises $\bm{x}_{t_{\text{e}}}$ into a target image $\hat{\bm{x}}$ via

$\mathrm{d}\hat{\bm{x}}=\Big{[}\bm{f}(\hat{\bm{x}},t)-g(t)^{2}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\hat{\bm{x}},t,\hat{\bm{y}})}\Big{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.$
(14)

CycleDiffusion leverages the observation that fixing all random variables in
the diffusion process helps produce similar images. Similar to SDEdit,
CycleDiffusion also adds noise to get $\bm{x}_{t_{\text{e}}}$ and denoises it
with Eq. (14), but the difference is that the random variable
$\mathrm{d}\bar{\bm{w}}$ in the denoising process in Eq. (14) is not randomly
sampled. Instead, CycleDiffusion introduces an encoding process that infers
the $\mathrm{d}\bar{\bm{w}}$ that are likely to help reconstruct the source
image given the source text. In the denoising process, the encoded
$\mathrm{d}\bar{\bm{w}}$ are applied to Eq. (14). This helps improve content
preservation during editing. Contrastive Guidance can strengthen the intended
editing for both methods by modifying the denoising process in Eq. (14).
Specifically, we set the positive prompt $\bm{y}^{+}$ as the target text
$\hat{\bm{y}}$ and the baseline prompt $\bm{y}^{-}$ as the source text
$\bm{y}$. In this case, the denoising process now becomes:

$\begin{split}\mathrm{d}\hat{\bm{x}}&=\Big{[}\bm{f}(\hat{\bm{x}},t)-g(t)^{2}\Big{(}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\hat{\bm{x}},t,\hat{\bm{y}})}\
+\\\ &\quad\ \
{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\big{(}\bm{s}_{\bm{\theta}}(\hat{\bm{x}},t,\hat{\bm{y}})-\bm{s}_{\bm{\theta}}(\hat{\bm{x}},t,\bm{y})\big{)}}\Big{)}\Big{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.\end{split}$
(15)

AFHQ Cat AFHQ Dog FFHQ Classifier-Free Guidance 0.341 0.287 0.118 Contrastive
Guidance (ours) 0.271 0.265 0.104

Table 1:  Contrastive Guidance improves disentanglement. The metric is the
average pixel-wise $L_{2}$ distance between the two images before and after
the guidance.

AFHQ Cat AFHQ Dog FFHQ Method FID$\downarrow$ KID $\times 10^{3}$$\downarrow$
$\mathcal{S}_{\text{CLIP}}$$\uparrow$ FID$\downarrow$ KID $\times
10^{3}$$\downarrow$ $\mathcal{S}_{\text{CLIP}}$$\uparrow$ FID$\downarrow$
$\mathcal{S}_{\text{CLIP}}$$\uparrow$ Text-to-Image Only 48.5 35.2 0.327 83.4
51.4 0.328 69.8 0.305 Finetuned Text-to-Image Only 41.3 35.2 0.316 67.1 38.9
0.319 47.3 0.280 Domain Expert + Classifier-Free Guidance 43.9 39.5 0.311 78.0
51.6 0.315 50.0 0.279 Domain Expert + Negative Guidance 124.6 110.7 0.230 94.5
60.7 0.239 168.1 0.175 Domain Expert + Contrastive Guidance 32.0 18.8 0.333
52.8 27.0 0.329 42.2 0.280

Table 2:  Contrastive Guidance improve domain-specific text-to-image modeling.

##  4 Disentanglement Analysis

In this section, we show that Contrastive Guidance provides more disentangled
guidance than classifier-free guidance [16] (CFG; for this baseline, the
positive prompt is used as input). To study disentanglement, we study the
pixel-wise $L_{2}$ distance between pairs of images before the guidance (i.e.,
the domain expert) and after the guidance (i.e., Contrastive Guidance or CFG).
All random variables are fixed during sampling. Figure 3 shows samples and
visualizations of the $L_{2}$ distance. We can see regions of the tongue, the
background, the snowflakes, and the tennis ball clearly when using Contrastive
Guidance, which are not observed when using CFG. Table 1 also quantitatively
verifies this observation. These results show that the guidance by Contrastive
Guidance is more disentangled than CFG.

##  5 Applications

This section provides experiments of Contrastive Guidance in several
applications: (1) guiding domain experts for domain-specific text-to-image
synthesis (Section 5.1), (2) guiding text-to-image models to correct their own
mistakes and to gain continuous, rig-like controls (Section 5.2), and (3)
guiding image editors to strengthen the intended edit (Section 5.3).

###  5.1 Guiding Domain Experts

#### Datasets:

We used three datasets: AFHQ Cat [5], AFHQ Dog [5], and FFHQ [20]. For each
dataset, we created a set of 64 text pairs $(\bm{y}^{+},\bm{y}^{-})$ (see
Appendix B.1). The training sets of these datasets are used as (1) training
data for domain experts and (2) references for FID and KID, which we will
detail later. The validation sets are not used here.

#### Pre-trained models:

We used the SD-v1-5 Stable Diffusion model [45] as the text-to-image diffusion
model. Since Stable Diffusion is a latent diffusion model, we need the domain
expert to share the encoder and decoder with it. To this effect, one can use
the encoder of Stable Diffusion to encode images in each of the three datasets
and then train a UNet on the encoded representations. To save computation, we
finetune a copy of the text-to-image diffusion model into a domain expert:333
In practice, sometimes data owners may not want to share domain data for
finetuning. Instead, they can still provide a domain expert to be guided. In
these cases, the Finetuned Text-to-Image Only baseline does not apply, while
our Contrastive Guidance still applies.  we used a fixed text input: A
portrait photo of [domain] *, where [domain] is the domain identifier, e.g.,
cat. The UNet and the embedding of * are finetuned, and the latter has a
larger learning rate. The described finetuning method can be viewed as a
combination of textual inversion [10] and DreamBooth [46].

#### Baselines:

(1) Text-to-Image Only samples images from the text-to-image diffusion model,
conditioned on $\bm{y}^{+}$, (2) Finetuned Text-to-Image Only3 samples images
from the finetuned text-to-image diffusion model, conditioned on $\bm{y}^{+}$,
(3) Classifier-Free Guidance [16] leverages the positive prompt $\bm{y}^{+}$
to guide the domain expert, and (4) Expert + Negative Guidance [29] uses the
baseline prompt $\bm{y}^{-}$ to negatively guide the domain expert.

#### Metrics:

For each text pair, we sample 16 samples from each method (1024 samples for
each method in total). To evaluate the realism and domain specificity, we used
Clean FID [40] to compute the Frechet Inception Distance (FID [15]) and Kernel
Inception Distance (KID [1]) of the 1024 samples using the training set as
reference. On AFHQ, we evaluated with resolution $512\times 512$, while on
FFHQ, we evaluated with resolution $256\times 256$, both with pre-computed
statistics by [40]. Since statistics are not available for KID on FFHQ, we
omitted this term. To evaluate the closeness to the text, we reported the CLIP
score, i.e., the cosine similarity between the normalized CLIP embeddings of
${\bm{x}}$ and $\bm{y}^{+}$:

$\mathcal{S}_{\text{CLIP}}({\bm{x}},\bm{y}^{+})=\cos\big{\langle}\text{CLIP}_{\text{img}}({\bm{x}}),\text{CLIP}_{\text{text}}(\bm{y}^{+})\big{\rangle}.$
(16)

Table 2 demonstrates that Contrastive Guidance achieves the best realism and
domain specificity (i.e., FID and KID); also, images from our method have
higher or comparable closeness to the positive text (i.e.,
$\mathcal{S}_{\text{CLIP}}$) compared to Text-to-Image Only, which ignores
domain specificity.

Figure 4:  Contrastive Guidance for continuous, rig-like controls, which
cannot be described accurately by language. Results show that Contrastive
Guidance can guide text-to-image diffusion models to generate images with
continuously changed variations. The positive and baseline prompts are
provided in Table 8. Images are resized to $256\times 256$, and please zoom in
to see the details.  Figure 5:  Contrastive Guidance mitigate contradiction.
We focused on failure cases of the text-to-image diffusion model such as
understanding negation (without eyeglasses, no umbrellas, without beard),
relation (in water), and one unexpected contradiction (colored photo). The
prompts used in this experiment are provided in Table 7.

###  5.2 Guiding Text-to-Image Models

In this section, we use Contrastive Guidance to help a text-to-image diffusion
model (SD-v1-5, following Section 5.1) to guide itself for negation,
relations, and continuous, rig-like controls. The prompts we used are provided
in Appendix C. Since vision-language models like CLIP are not reliable for
verifying text-image consistency in challenging cases [53], we provide image
samples in this section.

Figure 5 shows that Contrastive Guidance improves the consistency between the
synthesized images and some text prompts with negation, relation, and unlikely
scenarios. For instance, the text-to-image diffusion model tends to synthesize
contradictory images when the text contains without eyeglasses, a heavy
downpour with no umbrellas [34], colored photo, without beard, and a bottle in
water [53]. We leveraged Contrastive Guidance to mitigate such contradiction
by specifying baseline prompts such as with eyeglasses and water in a bottle.

Figure 4 shows that Contrastive Guidance enables continuous, rig-like controls
($\lambda_{t}\in\\{-8,-4,0,4,8\\}$) such as the cuteness of the fox, the
polarity of the weather (e.g., sunny vs. rainy), and the aesthetics of the
scene. The positive and baseline prompts used in this experiment are provided
in Table 8.

Figure 6:  Contrastive Guidance can improve zero-shot image editing methods
such as SDEdit [35] and CycleDiffusion [55]. We did not use a fixed
combination of hyperparameters because every input can have its best
combination of hyperparameters and even random seed. Following [55], for each
input, we ran 15 random trials for each combination of hyperparameters and
report the one with the highest $\mathcal{S}_{\text{D-CLIP}}$. All images are
resized to $256\times 256$, and please zoom in to see the details.

###  5.3 Guiding Image Editors

This section demonstrates that Contrastive Guidance can improve zero-shot
image editors such as SDEdit and CycleDiffusion. The input is a triplet
$(\bm{x},\bm{y},\hat{\bm{y}})$, where $\bm{x}$ is the source image, $\bm{y}$
is the source text, and $\hat{\bm{y}}$ is the target text; the output is a
target image $\hat{\bm{x}}$ to be generated. As detailed in Section 3.3, we
apply Contrastive Guidance to the decoding of SDEdit and CycleDiffusion, by
setting the positive prompt $\bm{y}^{+}$ as the target text $\hat{\bm{y}}$ and
the baseline prompt $\bm{y}^{-}$ as the source text $\bm{y}$. Please find more
details in Appendix D.

#### Datasets and pre-trained diffusion models:

We used the set of 150 triplets $({\bm{x}},\bm{y},\hat{\bm{y}})$ in [55],
covering editing of image styles, attributes, and objects. We used the SD-v1-5
Stable Diffusion as the text-to-image diffusion model.

#### Metrics:

We reported (1) PSNR and SSIM, which evaluate the faithfulness of the edited
images to the source images (i.e., minimality of editing); (2) CLIP score,
which evaluates the authenticity of the edited images to the target text:

$\mathcal{S}_{\text{CLIP}}({\hat{\bm{x}}},\hat{\bm{y}})=\cos\big{\langle}\text{CLIP}_{\text{img}}({\hat{\bm{x}}}),\text{CLIP}_{\text{text}}(\hat{\bm{y}})\big{\rangle};$
(17)

(3) directional CLIP score used by [55]:

$\begin{split}&\mathcal{S}_{\text{D-CLIP}}({\hat{\bm{x}}},{\bm{x}},\bm{y},\hat{\bm{y}})\
=\\\ &\quad\quad\quad\ \
\cos\Big{\langle}\text{CLIP}_{\text{img}}({\hat{\bm{x}}})-\text{CLIP}_{\text{img}}({\bm{x}}),\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{CLIP}_{\text{text}}(\hat{\bm{y}})-\text{CLIP}_{\text{text}}(\bm{y})\Big{\rangle}.\end{split}$
(18)

Seen in Table 3 are results for zero-shot image-to-image translation. For each
method (SDEdit and CycleDiffusion), using Contrastive Guidance results in
higher performances in terms of all metrics. Also, the use of Contrastive
Guidance allows us to use a less number of hyperparameter enumerations and
random trials for all methods as is done in [55] (see Appendix D), making the
three methods nearly $4\times$ more efficient while achieving higher
performances.

$\mathcal{S}_{\text{CLIP}}$$\uparrow$ $\mathcal{S}_{\text{D-CLIP}}$$\uparrow$
PSNR$\uparrow$ SSIM$\uparrow$ SDEdit 0.344 0.258 15.93 0.512 SDEdit + CG 0.336
0.275 17.37 0.553 CycleDiffusion 0.334 0.272 21.92 0.731 CycleDiffusion + CG
0.330 0.278 22.65 0.742

Table 3:  Contrastive Guidance (denoted as + CG) can improve SDEdit and
CycleDiffusion for zero-shot image editing. We did not use a fixed combination
of hyperparameters, and neither did we plot the trade-off curve. The reason is
that every input can have its best combination of hyperparameters and even
random seed. Instead, following [55], for each input, we ran 15 random trials
for each combination of hyperparameters and report the one with the highest
$\mathcal{S}_{\text{D-CLIP}}$. Please find more details in Appendix D.

##  6 Conclusions and Discussion

This paper advocates the idea of using text-to-image diffusion models as
disentangled guidance for image synthesis. The key idea is to disentangle
image factors using two prompts that have minimal differences. We formalize
this intuition by defining a generative classifier with the text-to-image
model and the two prompts, whose score function is proportional to a simple
subtraction term. We have shown the efficacy of Contrastive Guidance in
several applications: (1) guiding domain experts for domain-specific but
generalizable text-to-image synthesis, (2) guiding text-to-image models
themselves to gain continuous, rig-like controls, and (3) guiding image
editors to improve the intended edit. Our experiments show that Contrastive
Guidance is both qualitatively and quantitatively better than the baselines.

This paper also leaves several questions for further exploration. For
efficiency, we assume that the temperature $\gamma_{t}$ is adaptive such that
$\lambda_{t}$ does not depend on $\bm{x}$; although its effectiveness is
validated by our experiments, it should be considered with reservation because
the validity may not hold on other domains. Moreover, one may study how the
choice of the prompt pairs $(\bm{y}^{+},\bm{y}^{-})$ affects the performance.
Please see our discussion on errors and biases in Appendix B.4.

## References

  * Bińkowski et al. [2018] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton.  Demystifying MMD GANs.  _ICLR_ , 2018.
  * Chen et al. [2018] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Kristjanson Duvenaud.  Neural ordinary differential equations.  _NeurIPS_ , 2018.
  * Chen et al. [2022] Ting Chen, Ruixiang Zhang, and Geo rey E. Hinton.  Analog bits: Generating discrete data using diffusion models with self-conditioning.  _ArXiv_ , 2022.
  * Choi et al. [2021] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon.  ILVR: Conditioning method for denoising diffusion probabilistic models.  _ICCV_ , 2021.
  * Choi et al. [2020] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  StarGAN v2: Diverse image synthesis for multiple domains.  _CVPR_ , 2020.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Quinn Nichol.  Diffusion models beat GANs on image synthesis.  _NeurIPS_ , 2021.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang.  CogView: Mastering text-to-image generation via Transformers.  _NeurIPS_ , 2021.
  * Ding et al. [2022] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  CogView2: Faster and better text-to-image generation via hierarchical Transformers.  _NeurIPS_ , 2022.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Björn Ommer.  Taming Transformers for high-resolution image synthesis.  _CVPR_ , 2021.
  * Gal et al. [2022a] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _ArXiv_ , 2022a.
  * Gal et al. [2022b] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  StyleGAN-NADA: CLIP-guided domain adaptation of image generators.  _SIGGRAPH_ , 2022b.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  _NIPS_ , 2014.
  * Grathwohl et al. [2019] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Kristjanson Duvenaud.  FFJORD: Free-form continuous dynamics for scalable reversible generative models.  _ICLR_ , 2019.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay M. Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _ArXiv_ , 2022.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  _NIPS_ , 2017.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _NeurIPS Workshop_ , 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _NeurIPS_ , 2020.
  * Hong et al. [2022] Susung Hong, Gyuseong Lee, Woo-Yool Jang, and Seung Wook Kim.  Improving sample quality of diffusion models using self-attention guidance.  _ArXiv_ , 2022.
  * Karakas et al. [2022] Cemre Karakas, Alara Dirik, Eylül Yalçınkaya, and Pinar Yanardag.  FairStyle: Debiasing StyleGAN2 with style channel manipulations.  _ArXiv_ , 2022.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  _CVPR_ , 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  _CVPR_ , 2020.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _NeurIPS_ , 2021.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _NeurIPS_ , 2022.
  * Kawar et al. [2022a] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  _NeurIPS_ , 2022a.
  * Kawar et al. [2022b] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui-Tang Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  _ArXiv_ , 2022b.
  * Krause et al. [2021] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani.  GeDi: Generative discriminator guided sequence generation.  _Findings of EMNLP_ , 2021.
  * Li et al. [2022] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.  Contrastive decoding: Open-ended text generation as optimization.  _ArXiv_ , 2022.
  * Liu et al. [2022a] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  _ICLR_ , 2022a.
  * Liu et al. [2022b] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum.  Compositional visual generation with composable diffusion models.  _ECCV_ , 2022b.
  * Liu et al. [2021] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell.  More control for free! Image synthesis with semantic diffusion guidance.  _ArXiv_ , 2021.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  DPM-Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps.  _NeurIPS_ , 2022.
  * Lugmayr et al. [2022] Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  RePaint: Inpainting using denoising diffusion probabilistic models.  _CVPR_ , 2022.
  * Maoutsa et al. [2020] Dimitra Maoutsa, Sebastian Reich, and Manfred Opper.  Interacting particle solutions of Fokker–Planck equations through gradient–log–density estimation.  _Entropy_ , 2020.
  * Marcus et al. [2022] Gary Marcus, Ernest Davis, and Scott Aaronson.  A very preliminary analysis of DALL-E 2.  _ArXiv_ , 2022.
  * Meng et al. [2022] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  SDEdit: Guided image synthesis and editing with stochastic differential equations.  _ICLR_ , 2022.
  * Menon et al. [2020] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.  PULSE: Self-supervised photo upsampling via latent space exploration of generative models.  _CVPR_ , 2020.
  * Nguyen et al. [2017] Anh M Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  _CVPR_ , 2017.
  * Nichol et al. [2022] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  _ICML_ , 2022.
  * Nie et al. [2021] Weili Nie, Arash Vahdat, and Anima Anandkumar.  Controllable and compositional generation with latent-space energy-based models.  _NeurIPS_ , 2021.
  * Parmar et al. [2022] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On aliased resizing and surprising subtleties in GAN evaluation.  _CVPR_ , 2022.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and D. Lischinski.  StyleCLIP: Text-driven manipulation of StyleGAN imagery.  _ICCV_ , 2021.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _ICML_ , 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  _ICML_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  _ArXiv_ , 2022.
  * Rombach et al. [2022] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _CVPR_ , 2022.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _ArXiv_ , 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  _NeurIPS_ , 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _ICLR_ , 2022.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _ICLR_ , 2021a.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _NeurIPS_ , 2019.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _ICLR_ , 2021b.
  * Su et al. [2022] Xu Su, Jiaming Song, Chenlin Meng, and Stefano Ermon.  Dual diffusion implicit bridges for image-to-image translation.  _ArXiv_ , 2022.
  * Thrush et al. [2022] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  _ArXiv_ , 2022.
  * Watson et al. [2022] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi.  Learning fast samplers for diffusion models by differentiating through sample quality.  _ICLR_ , 2022.
  * Wu and la Torre [2022] Chen Henry Wu and Fernando De la Torre.  Unifying diffusion models’ latent space, with applications to CycleDiffusion and guidance.  _ArXiv_ , 2022.
  * Wu et al. [2022] Chen Henry Wu, Saman Motamed, Shaunak Srivastava, and Fernando De la Torre.  Generative visual prompt: Unifying distributional control of pre-trained generative models.  _NeurIPS_ , 2022.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation.  _ArXiv_ , 2022.
  * Zhang and Chen [2022] Qinsheng Zhang and Yongxin Chen.  Fast sampling of diffusion models with exponential integrator.  _ArXiv_ , 2022.

##  Appendix A Mathematical Details

###  A.1 Derivation of Score Function of Generative Classifier

Here, we derive $\nabla_{\bm{x}}\log p_{t}(\bm{x}|c)$ of the generative
classifier $p(\bm{x}|c)$ defined by Contrastive Guidance. Recall that, given a
positive prompt $\bm{y}^{+}$ and a baseline prompt $\bm{y}^{-}$, in Section
3.2 we have rewritten this generative classifier as

$\displaystyle p_{t}(\bm{x}|c)\propto
p_{t}(\bm{x})p_{t}(c|\bm{x})=p_{t}(\bm{x})\frac{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}},$
(19)

where the temperature $\gamma_{t}$ determines how sharp the generative
classifier is. Given enough space, here we do not simplify notations as done
in Section 3.2. Using the log-derivative trick is used once in Eq. (23), we
derive the score function as

$\displaystyle\begin{split}&\quad\ \nabla_{\bm{x}}\log p_{t}(\bm{x}|c)\\\
&=\nabla_{\bm{x}}\log\Big{(}p_{t}(\bm{x})\frac{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}\Big{)}\end{split}$
(20) $\displaystyle=\nabla_{\bm{x}}\log p_{t}(\bm{x})+0+\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}-\nabla_{\bm{x}}\log\Big{(}p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}\Big{)}$
(21) $\displaystyle=\nabla_{\bm{x}}\log p_{t}(\bm{x})+\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}-\frac{\nabla_{\bm{x}}\Big{(}p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}\Big{)}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}$
(22) $\displaystyle\begin{split}&=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\nabla_{\bm{x}}\log p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad-\frac{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}\end{split}$
(23) $\displaystyle=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\frac{p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}\Big{(}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}-\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}\Big{)}$ (24)
$\displaystyle=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\underbrace{\frac{\gamma_{t}\cdot
p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}{p(\bm{y}^{+})p_{t}(\bm{x}|\bm{y}^{+})^{\gamma_{t}}+p(\bm{y}^{-})p_{t}(\bm{x}|\bm{y}^{-})^{\gamma_{t}}}}_{\lambda_{t}}\Big{(}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})-\nabla_{\bm{x}}\log p_{t}(\bm{x}|\bm{y}^{-})\Big{)}$
(25) $\displaystyle=\nabla_{\bm{x}}\log
p_{t}(\bm{x})+\lambda_{t}\Big{(}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})-\nabla_{\bm{x}}\log p_{t}(\bm{x}|\bm{y}^{-})\Big{)}$
(26)
$\displaystyle\approx{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}(\bm{x},t)}+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\Big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\Big{)}}.$
(27)

###  A.2 Derivation of $\lambda_{t}$

In Section 3.2 and Appendix A.1, we have derived the score function
$\nabla_{\bm{x}}\log p_{t}(\bm{x}|c)$ of the generative classifier
$p(\bm{x}|c)$ defined by Contrastive Guidance. However, a remaining question
is how to estimate the coefficient $\lambda_{t}$ in Eq. (LABEL:eq:contain-
lambda) and Eq. (25). In the expression of $\lambda_{t}$, $\gamma_{t}$ is a
hyperparameter, and $p(\bm{y}^{+})$ and $p(\bm{y}^{-})$ are priors of text
prompts; therefore, to estimate $\lambda_{t}$, we only need to estimate
$p_{t}(\bm{x}|\bm{y}^{+})$ and $p_{t}(\bm{x}|\bm{y}^{-})$. In the following,
we show how to estimate $p_{t}(\bm{x}|\bm{y}^{+})$, and the estimate of
$p_{t}(\bm{x}|\bm{y}^{-})$ can be derived similarly by replacing $\bm{y}^{+}$
with $\bm{y}^{-}$.

Recall that $p_{t}(\bm{x}|\bm{y}^{+})$ is the marginal distribution of
$\bm{x}$ of the following SDE, with initial state distribution
$p_{T}(\bm{x})$:

$\mathrm{d}\bm{x}=\Big{[}\bm{f}(\bm{x},t)-g(t)^{2}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})\Big{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.$ (28)

It has been shown that each SDE has a corresponding probability flow ODE with
the same marginal density at each time $t$ [33, 51]. Specifically, probability
flow ODE of the SDE in Eq. (28) is

$\mathrm{d}\bm{x}=\Big{[}\bm{f}(\bm{x},t)-\frac{1}{2}g(t)^{2}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})\Big{]}\mathrm{d}t.$ (29)

Since the SDE and the probability flow ODE have the same marginal
distribution, we can estimate $p_{t}(\bm{x}|\bm{y}^{+})$ by estimating the
marginal $p_{t}^{\text{ODE}}(\bm{x}|\bm{y}^{+})$ of the probability flow ODE
in Eq. (29). Since $p_{T}^{\text{ODE}}(\bm{x}|\bm{y}^{+})$ is the isometric
Gaussian distribution $\mathcal{N}(\bm{x};\bm{0},\bm{I})$, the marginal
$p_{t}^{\text{ODE}}(\bm{x}|\bm{y}^{+})$ is tractable by integrating the
following ODE [2, 13] from $t$ to $T$.

$\mathrm{d}\begin{bmatrix}\bm{x}\\\ \log
p_{t}^{\text{ODE}}(\bm{x}|\bm{y}^{+})\end{bmatrix}=\begin{bmatrix}\displaystyle\bm{f}(\bm{x},t)-\frac{1}{2}g(t)^{2}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})\\\
\displaystyle-\mathrm{tr}\Big{(}\frac{\mathrm{d}}{\mathrm{d}\bm{x}}\big{(}\bm{f}(\bm{x},t)-\frac{1}{2}g(t)^{2}\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})\big{)}\Big{)}\end{bmatrix}\mathrm{d}t.$ (30)

The generalist text-to-image diffusion model approximates $\nabla_{\bm{x}}\log
p_{t}(\bm{x}|\bm{y}^{+})$ with $\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})$,
and Eq. (30) becomes

$\mathrm{d}\begin{bmatrix}\bm{x}\\\ \log
p_{t}^{\text{ODE}}(\bm{x}|\bm{y}^{+})\end{bmatrix}=\begin{bmatrix}\displaystyle\bm{f}(\bm{x},t)-\frac{1}{2}g(t)^{2}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})\\\
\displaystyle-\mathrm{tr}\Big{(}\frac{\mathrm{d}}{\mathrm{d}\bm{x}}\big{(}\bm{f}(\bm{x},t)-\frac{1}{2}g(t)^{2}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})\big{)}\Big{)}\end{bmatrix}\mathrm{d}t.$
(31)

Although the above estimation is tractable, we need to run it for all $t$’s,
resulting in $\mathcal{O}(N^{2})$ complexity where $N$ is the number of
discretization steps. Therefore, although $\lambda_{t}$ can be estimated as we
show above, this paper tries to avoid such estimation. To this effect, we
assume that the temperature $\gamma_{t}$ is adaptive such that $\lambda_{t}$
does not depend on $\bm{x}$, giving us $\mathcal{O}(N)$ complexity.

##  Appendix B Guiding Domain Experts (Section 5.1): Experimental Details

###  B.1 Task Input

Contrastive Guidance allows generalists to guide domain experts for domain-
specific text-to-image synthesis. The input to this task is a pair of positive
and baseline prompts $(\bm{y}^{+},\bm{y}^{-})$ (note that some baselines can
only take one of them as input). Tables 4-6 provide several examples for
$(\bm{y}^{+},\bm{y}^{-})$. The full lists of samples are provided in our code
release.

###  B.2 Baseline Details

Generalist Only:  Following the notations in the main text, Generalist Only
means sampling from the reverse-time SDE:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y}^{+})}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},\end{split}$
(32)

where
${\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y}^{+})}$
means the score function after classifier-free guidance [16]. It means that
Generalist Only samples from:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}\tau_{t}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-(\tau_{t}-1)\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},\end{split}$
(33)

where $\varnothing$ is the null token that stands for unconditional sampling,
and $\tau_{t}$ is a hyperparameter indicating the strength of guidance. We
encourage readers to read [16] for more details of classifier-free guidance.

Positive Guidance:  Positive Guidance is essentially classifier-free guidance
for the domain expert, using the positive prompt. Specifically, this baseline
samples from the following reverse-time SDE:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}_{\bm{\phi}}(\bm{x},t)}+\tau_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)\big{)}\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},\end{split}$
(34)

where $\varnothing$ is the null token that stands for unconditional sampling,
and $\tau_{t}$ is a hyperparameter indicating the strength of guidance.

Negative Guidance:  Negative Guidance is essentially the negation operation
[29] applied to the domain expert. Specifically, this baseline samples from
the following reverse-time SDE:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\bm{s}_{\bm{\phi}}(\bm{x},t)}+\tau_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\big{)}\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},\end{split}$
(35)

where $\varnothing$ is the null token that stands for unconditional sampling,
and $\tau_{t}$ is a hyperparameter indicating the strength of guidance.

StyleCLIP:  We used the latent optimization variant of StyleCLIP [41] as our
baseline. Given the positive prompt $\bm{y}^{+}$, StyleCLIP begins by randomly
sampling a style vector from a StyleGAN [20, 21, 22], followed by iteratively
optimizing this style vector to minimize the CLIP embedding distance between
the generated image and the positive prompt $\bm{y}^{+}$. We used the
implementation in the Google Colab Notebook provided by the authors of the
original paper.

StyleGAN-NADA:  Similar to StyleCLIP, StyleGAN-NADA [11] also leverages the
CLIP model to help generalize to text conditions, while StyleGAN-NADA
finetunes the StyleGAN generator for each positive prompt $\bm{y}^{+}$ and
keeps the style vector fixed. We used the implementation in the Google Colab
Notebook provided by the authors of the original paper.

Positive prompt $\bm{y}^{+}$ | Baseline prompt $\bm{y}^{-}$
---|---
a portrait photo of a cat with eyeglasses. | a portrait photo of a cat.
a portrait photo of a cat, indoor. | a portrait photo of a cat, outdoor.
a portrait photo of a yellow cat. | a portrait photo of a cat.
a portrait photo of a hairy cat. | a portrait photo of a hairless cat.
a portrait photo of a cat with open mouth. | a portrait photo of a cat with closed mouth.
a portrait photo of a cat at night. | a portrait photo of a cat in the daytime.
a portrait photo of a cat with fireworks behind. | a portrait photo of a cat.
a portrait photo of a cat with a tennis ball in front of it. | a portrait photo of a cat.
a portrait photo of a cat when rain comes down. | a portrait photo of a cat.
a portrait photo of a British Shorthair cat. | a portrait photo of a cat.
a portrait photo of a cat with snow. | a portrait photo of a cat without snow.
a portrait photo of a cat when snow comes down. | a portrait photo of a cat.
a portrait photo of a cat underwater. | a portrait photo of a cat.
Table 4:  Examples of text prompts for AFHQ Cat (Section 5.1). See image samples from our method in Figures 7-9.  Positive prompt $\bm{y}^{+}$ | Baseline prompt $\bm{y}^{-}$
---|---
a portrait photo of a dog with eyeglasses. | a portrait photo of a dog.
a portrait photo of a dog, indoor. | a portrait photo of a dog, outdoor.
a portrait photo of a yellow dog. | a portrait photo of a dog.
a portrait photo of a hairy dog. | a portrait photo of a hairless dog.
a portrait photo of a dog with open mouth. | a portrait photo of a dog with closed mouth.
a portrait photo of a dog at night. | a portrait photo of a dog in the daytime.
a portrait photo of a dog with fireworks behind. | a portrait photo of a dog.
a portrait photo of a dog with a tennis ball in front of it. | a portrait photo of a dog.
a portrait photo of a dog when rain comes down. | a portrait photo of a dog.
a portrait photo of a German Shepherd dog. | a portrait photo of a dog.
a portrait photo of a dog with snow. | a portrait photo of a dog without snow.
a portrait photo of a snow when snow comes down. | a portrait photo of a dog.
a portrait photo of a dog underwater. | a portrait photo of a dog.
Table 5:  Examples of text prompts for AFHQ Dog (Section 5.1).  Positive prompt $\bm{y}^{+}$ | Baseline prompt $\bm{y}^{-}$
---|---
a portrait photo of a person with eyeglasses. | a portrait photo of a person.
a portrait photo of a person, indoor. | a portrait photo of a person, outdoor.
a portrait photo of a black woman. | a portrait photo of a person.
a portrait photo of a hairy person. | a portrait photo of a hairless person.
a portrait photo of a person with open mouth. | a portrait photo of a person with closed mouth.
a portrait photo of a person at night. | a portrait photo of a person in the daytime.
a portrait photo of a person with fireworks behind. | a portrait photo of a person.
a portrait photo of a person with a tennis ball in front of it. | a portrait photo of a person.
a portrait photo of a person when rain comes down. | a portrait photo of a person.
a portrait photo of a person with a headphone. | a portrait photo of a person without a headphone.
a portrait photo of a person with snow. | a portrait photo of a person without snow.
a portrait photo of a person when snow comes down. | a portrait photo of a person.
a portrait photo of a person underwater. | a portrait photo of a person.
Table 6:  Examples of text prompts for FFHQ (Section 5.1).  Figure 7:  Samples
for text-to-image synthesis in the AFHQ Cat domain, using our Contrastive
Guidance. A single triangle beside an image stands for it fails sometimes,
while double triangles beside an image stand for it usually fails. The
positive text $\bm{y}^{+}$ is displayed below each image, in which $\ldots$
stands for “a portrait photo of”.  Figure 8:  Samples for text-to-image
synthesis in the AFHQ Cat domain, using our Contrastive Guidance. Notations
follow Figure 7.  Figure 9:  Samples for text-to-image synthesis in the AFHQ
Cat domain, using our Contrastive Guidance. Notations follow Figure 7.

###  B.3 Additional Samples for Domain-Specific Text-to-Image Modeling

Figure 7, Figure 8, and Figure 9 present more samples on the AFHQ Cat domain.
Limited by the file size (50 MB), we provide more samples on the AFHQ Dog
domain and the FFHQ domain in our code release.

###  B.4 Error Analysis

Our text prompts cover objects, poses, environment, attributes, lighting, and
sub-population. Among them, we find some failure cases produced by our
Contrastive Guidance. In terms of objects, our method sometimes fails to model
“with AR glasses” and “while another cat is behind it”. In terms of pose, as
expected, it does not model ambiguous descriptions consistently, such as
“looking left” and “looking right”. In terms of environment, unexpectedly, it
fails to model “with storm”. In terms of attributes, it sometimes fails to
model low-density attributes such as “closed eyes”, “red eyes”, “purple eyes”,
and “asymmetric face”. In terms of lighting, it fails to model “underexposure”
and “overexposure”. We also observe some unexpected biases. For instance, “wet
cats” tend to be black; “cats in the dark” tend to be gray; “underexposure
cats” tend to be black instead of being underexposure; “overexposure cats”
tend to be white instead of being overexposure.

Prompt $\bm{y}$ Positive prompt $\bm{y}^{+}$ Baseline prompt $\bm{y}^{-}$ an
oil painting of a couple in formal evening wear going home get caught in a
heavy downpour with no umbrellas. an oil painting of a couple in formal
evening wear going home get caught in a heavy downpour with no umbrellas. an
oil painting of a couple in formal evening wear going home get caught in a
heavy downpour with an umbrella. a photo of a professor without eyeglasses. a
photo of a professor without eyeglasses. a photo of a professor with
eyeglasses. a colored photo of a man without beard. a colored photo of a man
without beard. a black-and-white photo of a man with beard. a bottle is in
water. a bottle is in water. water is in a bottle.

Table 7:  Text prompts for generalists guiding themselves to mitigate
contradiction (Section 5.2). See image samples in Figure 5.

Prompt $\bm{y}$ Positive prompt $\bm{y}^{+}$ Baseline prompt $\bm{y}^{-}$ a
portrait photo of a fox. a portrait photo of a cute fox. a portrait photo of a
fox. an oil painting of a couple in formal evening wear going home. an oil
painting of a couple in formal evening wear going home in a heavy downpour. an
oil painting of a couple in formal evening wear going home in a sunny day. a
photo of a valley. a beautiful photo of a valley. a not beautiful photo of a
valley. a photo of a waterfall. a beautiful photo of a waterfall. a not
beautiful photo of a waterfall.

Table 8:  Text prompts for generalists guiding themselves for continuous, rig-
like control (Section 5.2). See image samples in Figure 4.

##  Appendix C Guiding Generalists (Section 5.2): Experimental Details

###  C.1 Task Input

Contrastive Guidance allows generalists to guide themselves (1) to improve
challenging cases of text-to-image synthesis and (2) to enable continuous,
rig-like controls. The input is a tuple $(\bm{y},\bm{y}^{+},\bm{y}^{-})$. For
(1), the tuples for the images in Figure 5 are shown in Table 7. For (2), the
tuples for the images in Figure 4 are shown in Table 8.

###  C.2 Method Details

Generalist Only:  Following the notations in the main text, Generalist Only
means sampling from the reverse-time SDE:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y})}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.\end{split}$
(36)

where
${\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y})}$
means the score function after classifier-free guidance [16]. It means that
Generalist Only samples:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}\tau_{t}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y})-(\tau_{t}-1)\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}},\end{split}$
(37)

where $\varnothing$ is the null token that stands for unconditional sampling,
and $\tau_{t}$ is a hyperparameter indicating the strength of guidance. We
encourage readers to read [16] for more details of classifier-free guidance.

Contrastive Guidance:  When using Contrastive Guidance to guide the
generalist, we sample from the reverse-time SDE:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}{\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y})}+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\big{)}}\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.\end{split}$
(38)

where
${\color[rgb]{0.27,0.42,0.81}\definecolor[named]{pgfstrokecolor}{rgb}{0.27,0.42,0.81}\tilde{\bm{s}_{\bm{\theta}}}(\bm{x},t,\bm{y})}$
means the score function after classifier-free guidance [16]. It means that
Contrastive Guidance samples from the following reverse-time SDE:

$\begin{split}&\mathrm{d}\bm{x}=\bigg{[}\bm{f}(\bm{x},t)-g(t)^{2}\Big{(}\tau_{t}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y})-(\tau_{t}-1)\bm{s}_{\bm{\theta}}(\bm{x},t,\varnothing)+{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\lambda_{t}\big{(}\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{+})-\bm{s}_{\bm{\theta}}(\bm{x},t,\bm{y}^{-})\big{)}}\Big{)}\bigg{]}\mathrm{d}t+g(t)\mathrm{d}\bar{\bm{w}}.\end{split}$
(39)

##  Appendix D Guiding Image Editors (Section 5.3): Experimental Details

For all methods, we follow the setting described in [55] to guarantee that the
best hyperparameter combination is chosen for each method, instead of using a
predefined hyperparameter combination. Specifically, for each test sample, we
allow each method to enumerate some combinations of hyperparameters, each
combination with multiple random trials (detailed below). To select the best
hyperparameter combination for each sample, we used the directional CLIP score
$\mathcal{S}_{\text{D-CLIP}}$ as the criterion (higher is better). For each
method, we used the DDIM sampler ($\eta=0.1$) with 100 steps.

SDEdit:  We enumerated the classifier-free guidance of the decoding step as
$\\{1,1.5,2,3,4,5\\}$; we enumerated the encoding step as
$\\{15,20,25,30,40,50\\}$; we ran $15$ trials for each combination.

CycleDiffusion:  We set the classifier-free guidance of the encoding step as
$1$; we enumerated the classifier-free guidance of the decoding step as
$\\{1,1.5,2,3,4,5\\}$; we enumerated the encoding step as
$\\{15,20,25,30,40,50\\}$; we ran $15$ trials for each hyperparameter
combination.

Contrastive Guidance for SDEdit:  Since Contrastive Guidance helps strengthen
the indented editing, we used fewer combinations to enumerate. Specifically,
we enumerated the classifier-free guidance of the decoding step as
$\\{1,1.5,2,3\\}$; we enumerated the encoding step as $\\{30,40,50\\}$; we ran
$15$ trials for each combination. We set $\lambda_{t}=10$.

Contrastive Guidance for CycleDiffusion:  Since Contrastive Guidance helps
strengthen the indented editing, we used fewer combinations to enumerate. We
set the classifier-free guidance of the encoding step as $1$; we enumerated
the classifier-free guidance of the decoding step as $\\{1,1.5,2,3\\}$; we
enumerated the encoding step as $\\{30,40,50\\}$; we ran $15$ trials for each
hyperparameter combination. We set $\lambda_{t}=6$.