(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Department of Statistics and Data Science
University of California, Los Angeles
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image
Diffusion Models

Yasi Zhang  Peiyu Yu  Ying Nian Wu

###### Abstract

Text-to-image diffusion models have shown great success in generating high-
quality text-guided images. Yet, these models may still fail to semantically
align generated images with the provided text prompts, leading to problems
like incorrect attribute binding and/or catastrophic object neglect. Given the
pervasive object-oriented structure underlying text prompts, we introduce a
novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method
to address the aforementioned problems. We show that an object-centric
attribute binding loss naturally emerges by approximately maximizing the log-
likelihood of a $z$-parameterized energy-based model with the help of the
negative sampling technique. We further propose an object-centric intensity
regularizer to prevent excessive shifts of objects attention towards their
attributes. Extensive qualitative and quantitative experiments, including
human evaluation, on several challenging benchmarks demonstrate the superior
performance of our method over previous strong counterparts. With better
aligned attention maps, our approach shows great promise in further enhancing
the text-controlled image editing ability of diffusion models.

###### Keywords:

Attention Map AlignmentEnergy-Based Models Text-to-Image Diffusion Models

##  1 Introduction

Recently, large-scale text-to-image diffusion models [25, 18, 21, 26, 1, 10]
have showcased remarkable capabilities in producing diverse, imaginative,
high-resolution visual content based on free-form text prompts. Despite their
revolutionary progress, however, these models may not consistently capture and
convey the full semantic meaning of the provided text prompts [4, 24]. Some
well-known issues include omission, hallucination, or duplication of details
[29], semantic leakage of attributes between entities [24], and
miscomprehension of intricate textual descriptions [26].

Figure 1: Key observations of the generation process of diffusion models. The
given prompt is “a purple crown and a blue suitcase”. In panel (c), we
hypothesize that if the intensity level of any object in the prompt does not
remain high during the first half of the denoising process, e.g. the crown in
SD and SG, the model would fail to generate the object in the final image. The
panel (d) suggests that if the attention map distributions of any attribute-
object pair are not aligned, the model would struggle to correctly bind
attributes to their respective objects, e.g. ‘purple’ and ‘crown’ in SD and
AnE. The generated images are displayed in the panel (e). All methods share
the same random seed.

Many previous works have focused on addressing the semantic misalignment
issues, particularly concerning multiple-object generation and attribute
binding. Composable Diffusion (CD) [15] composes multiple output noises guided
by different objects in a text prompt during the generation process. However,
this approach often results in a blend of objects, failing to distinctly
separate them. Prompt-to-Prompt (PtP) [7] observes a strong correlation
between cross-attention maps and the layout of an image. Building on this,
Structured Diffusion (StrD) [6] experiments with averaging attention maps
generated by different noun phrases for the same queried image latent
representation. Yet, a simple average is inadequate for consistently
generating images with multiple objects possessing complex attributes. Attend-
and-Excite (AnE) [3] proposes a novel approach of maximizing the attention map
scores of object tokens by updating the latent at each sampling step. However,
we note that artifacts and incorrect attribute binding are likely when AnE
maximizes the attention weights of object tokens without any concerns on
attributes. In response, SynGen (SG) [23] proposes an attribute-object pair-
centric objective, aiming to minimize the distribution distance within the
pair while maximizing it from other tokens, based on the assumption that
normlized attention maps follow a multinomial distribution. Our findings (see
Figs. 1 and 4-6) indicate that this approach still struggles with object
neglect due to its pair-centric nature. We argue that multiple-object
generation is more critical than attribute binding, as attributes cannot
manifest without the presence of objects. Furthermore, in scenarios with
multiple objects and no explicit attributes in a prompt, SG is degraded to
standard Stable Diffusion Models (SD) [25]. Diverging from these methods,
Energy-Based Cross Attention (EBCA) [19] introduces an Energy-Based Model
(EBM) framework [28, 31, 30, 32] for queries and keys within cross-attention
mechanisms, proposing updates to text embeddings instead of latent noise
representations.

A closer look at both the fluctuations of attention intensities and the
attention distributions of attribute-object pairs in these methods shed light
on the root cause of the misalignment issues. As illustrated in Fig. 1,
alignment in attribute-object attention maps (e.g., ‘purple crown’ in SG)
encourages attribute binding. However, attention map alignment alone does not
guarantee complete semantic alignment, as the intensity levels of object
attention maps are crucial in determining the presence of an object in the
final image. For example, in the image generated by SG, the crown is notably
absent. Conversely, despite successful generation of both objects with strong
intensities, AnE binds the attribute ‘purple’ incorrectly to the suitcase
resulting from misaligned attention distributions of attribute-object pairs.
Motivated by these key observations, we introduce a novel object-conditioned
Energy-Based Attention Map Alignment (EBAMA) method to hopefully address both
the incorrect attribute binding and the catastrophic object neglect problems
in a unified framework. Notably, we show that approximately maximizing the
log-likelihood for a $z$-parameterized EBM effectively leads to optimizing an
object-centric binding loss, which emphasizes both the object attention map
intensity levels and the attribute-object attention map alignment. We further
develope an object-centric intensity regularizer to prevent excessive shifts
of objects towards their attributes, providing an extra degree of freedom
balancing the trade-off between correct attribute binding and the necessary
presence of objects.

We summarize our contributions as follows: i) we introduce a novel object-
conditioned EBAMA method to address both the incorrect attribute binding and
the catastrophic object neglect problems in text-controlled image generation;
ii) extensive qualitative and quantitative experiments, including human
evaluation, on several challenging benchmarks demonstrate the superior
performance of our method over strong previous approaches. iii) We showcase
that our approach has great promise in further enhancing the text-controlled
image editing ability of diffusion models.

##  2 Related Work

#### EBM Framework for Attention Mechanisms

Recent advancements in the theoretical exploration of attention mechanisms
have increasingly embraced the EBM framework [12, 16, 22]. Modern Hopfield
Networks [22] showcases that one of the proposed energy minima is equivalent
to the attention mechanism. Building on this groundwork, Energy Transformer
[11] designs an engineered energy function to extract the relationships
between tokens. Furthering this approach, EBCA [19] first formulates EBMs of
query values condtioned on key values in each cross-attention layer.
Similarly, our method seeks to exploit the theoretical potential of EBMs,
focusing on the unique formulation of object-conditioned EBMs for attention
maps.

#### Text-to-Image Diffusion Models

Most large-scale text-to-image diffusion models [25, 18, 21, 26, 1, 10]
utilize classifier-free guidance [8] for improved conditional synthesis
results. However, due to its strong linguistic and visual priors injected from
the training dataset, these models suffer from diverse semantic misalignment
issues related to the objects in the provided text prompts and their
attribute(s) [4, 26, 24, 29]. Our approach better aligns images with its
provided texts by mitigating the issues of object neglect and incorrect
attribute binding without fine-tuning the diffusion models or additional
training datasets.

#### Attention-Based Enhancement

PtP [7] identifies a correlation between cross-attention maps and image
layout. Expanding on this, StrD [6] experiments with averaging attention maps
from different noun phrases to mitigate object neglect and attribute leakage.
AnE [3] introduces a method to enhance object presence by maximizing attention
map weights for object tokens. SG [23] proposes minimizing distribution
distances of the attention maps within attribute-object pairs and maximizing
the distances between the pairs and the other tokens. Different from the
previous approaches, EBCA [19] adopts an EBM framework, focusing on updating
text embeddings within cross-attention mechanisms. Our work also introduces an
energy-inspired attention map alignment objective, while our objective has a
specific emphasis on the object tokens.

##  3 Background

###  3.1 Stable Diffusion Models

For fair comparison with previous approaches, we also conduct experiments on
open-sourced state-of-the-art Stable Diffusion Models (SD) [25]. SD first
encodes an image $x$ into the latent space using a pretrained encoder [5],
i.e., $z=\mathcal{E}(x)$. Given a text prompt $y$, SD optimizes the
conditional denoising autoencoder $\epsilon_{\theta}$ by minimizing the
objective

$\displaystyle\mathcal{L}_{\theta}=\mathbb{E}_{t,\epsilon\sim\mathcal{N}(0,1),z\sim\mathcal{E}(x)}||\epsilon-\epsilon_{\theta}(z_{t},t,\phi(y))||^{2},$
(1)

where $\phi$ is a frozen CLIP text encoder [20], $z_{t}$ is a noised version
of the latent $z$, and the time step $t$ is uniformly sampled from
$\\{1,\ldots,T\\}$. During sampling, $z_{T}$ is randomly sampled from standard
Gaussian and denoised iteratively by the denoising autoencoder
$\epsilon_{\theta}$ from time $T$ to $0$. Finally, a decoder $\mathcal{D}$
reconstructs the image as $\tilde{x}=\mathcal{D}(z_{0})$.

###  3.2 Cross-Attention Mechanism

In the cross-attention mechanism, $K$ is the linear projections of $W_{y}$,
the CLIP-encoded text embeddings of text prompt $y$. $Q$ is the linear
projection of the intermediate image representation parameterized by latent
variables $z$. Given a set of queries $Q$ and keys $K$, the (unnormalized)
attention features and (softmax-normalized) scores between these two matrices
are

$\displaystyle
A=\frac{QK^{T}}{\sqrt{m}},~{}\tilde{A}=\text{softmax}\left(\frac{QK^{T}}{\sqrt{m}}\right),$
(2)

where $m$ is the feature dimension. We consider both attention features and
scores for our modeling here, which we denote as $A_{s}$ and $\tilde{A}_{s}$
for token $s$, respectively.

Figure 2: An overview of our workflow for optimizing diffusion models. It
includes aggregation of attention maps, computation of object-centric
attention loss, and updates to $z_{t}$.

##  4 Method

The key idea of the proposed method derives from the object-oriented structure
that underlies most prompts for text-to-image generation. To be specific,
syntactically the majority of prompts can be parsed as the modifiers and the
entity-nouns, i.e., the nouns that correspond to objects in the generated
image, such as “A red metal crown”, and “A girl in red”, etc. Based on the
observation, we propose to exploit the object-oriented structure by employing
an ensemble of object-centric cross-attention losses for inference-time
optimization. Our aim is to address the semantic misalignment issues including
both the attribute binding (e.g., semantic leakage and attribute neglect [23])
and catastrophic object neglect [3], with the principally derived and
deliberately designed optimization objective. We then discuss the key
components of our method as follows.

###  4.1 Extraction of the Object-Oriented Structure

Following the pre-processing step in [23], we parse the prompt using Spacy’s
[9] transformer-based dependency parser to extract the object-oriented
structure. We identify a set $S$ of object tokens ${s}$ from the prompt, whose
tag is either NOUN (noun) such as ‘backpack’ or PROPN (proper noun) such as
‘Tesla company’ using the parser; we exclude nouns that serve as direct
modifiers of other nouns. The remaining modifiers are grouped by their
corresponding object tokens, denoted as the modifier sets for each object
token $s$, i.e., $\mathcal{M}(s)$. Note that $\mathcal{M}(s)=\emptyset$ if
there are no modifiers corresponding to the object token $s$. We refer to
supplementary material for more details about the parsing process.

###  4.2 Object-Conditioned Energy-Based Model

We assume that the distribution of the modifier tokens
$l\in\bigcup_{s}\mathcal{M}(s)$ given the object token $s$ is

$p_{z}(l|s)=\frac{1}{Z(s)}\exp\left(f(A_{l},A_{s})\right),$ (3)

where $Z(s)=\sum_{l}\exp\left(f(A_{l},A_{s})\right)$ is the normalizing
constant and $f$ is the negative energy function. How to choose the energy
function for attention maps remains an interesting and open problem. Prior
works [22, 19] utilize a log-sum-exp term to model the exponential interaction
and alignment between state patterns (query) and stored patterns (key).
However, the alignment measurement for attentions maps across different tokens
is unclear. To bridge the gap, we propose the application of a non-crafted yet
effective energy function — cosine similarity defined as
$f(A_{l},A_{s})=\left<A_{l},A_{s}\right>/(||A_{l}||\cdot||A_{s}||)$. The
efficacy of this choice of energy function is validated in the experimental
section. Eqn. (3) therefore defines a multinomial token distribution as a
$z$-parameterized conditional energy-based model, where $z$ is the latent
variables of SD. The inference-time optimization over the latent variables $z$
is then equivalently maximizing the log-likelihood of this EBM, which
increases the probabilities of the syntatically related modifier tokens of the
given object $s$. To be specific, it can be shown that (see the supplementary
material)

$\nabla_{z}\log
p_{z}(l|s)=\nabla_{z}f(A_{l},A_{s})-\mathbb{E}_{p_{z}(l|s)}\left[\nabla_{z}f(A_{l},A_{s})\right].$
(4)

Since the vocabulary size of modifier tokens can be large in practice (in the
order of $10^{4}$), we consider resorting to negative sampling [17] for the
approximation of the expectation term, where we uniformly sample tokens
unrelated to the object token and calculate the Monte Carlo average. This
particular implementation choice of Eqn. (4) then leads to the object-centric
attribute binding loss below.

###  4.3 Object-Conditioned Energy-Based Attention Map Alignment

For each object token $s\in S$, we design the following two components that
consist of the object-centric attention loss:

#### Object-centric attribute binding

First, instead of operating on the noun-modifier normalized attention score
pairs as in [23], we focus on optimizing the log-likelihood of the object-
conditioned EBM using negative sampling. This gives us the attribute binding
loss:

$\displaystyle
L_{b}^{(s)}=-\frac{1}{|\mathcal{M}(s)|}\sum_{l\in\mathcal{M}(s)}f(A_{s},A_{l})+\frac{1}{N-|\mathcal{M}(s)|-1}\sum_{l\notin\mathcal{M}(s),l\neq
s}f(A_{s},A_{l}),$ (5)

whose negative gradient w.r.t. $z$ could be seen as the Monte Carlo
approximation of Eqn. (4). The goal of $L_{b}^{(s)}$ is to: i) maximize the
cosine similarity between the given object $s$ and its syntactically-related
modifier tokens, while ii) enforcing the repulsion of grammatically unrelated
ones in the feature space. Note that the loss above only applies to the cases
where $\mathcal{M}(s)$ is a non-empty set. For the case where
$\mathcal{M}(s)=\emptyset$, only the repulsive term of Eqn. (5) is used.

#### Object-centric intensity regularizer

Although the proposed attribute binding loss mitigates the catastrophic object
neglect problem (see $\lambda=0$ entries in Tab. 1), we observe that the
object-related attention feature can still be overly shifted when there are
multiple modifier tokens in the $\mathcal{M}(s)$ or multiple object tokens in
a prompt; this could again potentially leads to the object neglect phenomenon.
To address this issue, we follow [3] and propose an object-centric intensity
regularizer to maintain the attention intensity level of object $s$:

$\displaystyle L_{n}^{(s)}=-||\mathcal{K}(\tilde{A_{s}})||_{\infty},$ (6)

where $\mathcal{K}$ is a 3x3 Gaussian kernel, and $||\cdot||_{\infty}$ denotes
the maximum value of a vector. We use the attention scores in Eqn. (2) as its
input. We refer to $||\mathcal{K}(\tilde{A_{s}})||_{\infty}$ as the intensity
level of the object token $s$.

The final object-centric attention loss $L$ is the linear combination of the
binding loss and the regularizer, i.e.

$\displaystyle L=\sum_{s\in S}L^{(s)}=\sum_{s\in S}L_{b}^{(s)}+\lambda
L_{n}^{(s)},$ (7)

where intensity weight $\lambda$ is a hyper-parameter to specify. $\lambda>0$
enforces the presence of object $s$, but excessively intensified object
attention can hinder the attribute binding performance and lower visual image
quality. We provide empirical analysis on how to tune the weight in practice
(see Section 5.4 and supplementary material for details).

###  4.4 Workflow

Our workflow is illustrated in Fig. 2. To begin, at each time step $t$, we
aggregate the attention map features denoted as $A$ at a resolution of 16x16.
This aggregation is performed after one step of propagating $z_{t}$ through
the denoising model. Subsequently, we calculate the object-centric attention
loss, as described in Eqn. (7). Finally, we backpropagate the computed loss
and update $z_{t}$ for each time step, following the formula
$z_{t}^{\prime}\leftarrow z_{t}-\alpha\nabla_{z_{t}}L$, where $\alpha$
represents the step size. In our experimental setup, we set $\alpha=20$. Note
that we only perform updates on $z_{t}$ during the first half of the sampling
steps, which corresponds to 25 steps since we use a DDIM sampler with a total
of 50 steps. More details about the workflow can be found in the supplementary
material.

##  5 Experiments

Table 1: Comparison of Full Sim., Min. Sim., and T-C Sim. across different methods on the AnE dataset. Note that the performance of SG on Animal-Animal is degraded to SD, as the prompts do not contain any attribute-object pairs. The best and second-best performances are marked in bold numbers and underlines, respectively; tables henceforth follows this format. | Animal-Animal | Animal-Object | Object-Object
---|---|---|---
Method | Full Sim. | Min. Sim. | T-C Sim. | Full Sim. | Min. Sim. | T-C Sim. | Full Sim. | Min. Sim. | T-C Sim.
SD[25] | 0.311 | 0.213 | 0.767 | 0.340 | 0.246 | 0.793 | 0.335 | 0.235 | 0.765
CD[15] | 0.284 | 0.232 | 0.692 | 0.336 | 0.252 | 0.769 | 0.349 | 0.265 | 0.759
StrD[6] | 0.306 | 0.210 | 0.761 | 0.336 | 0.242 | 0.781 | 0.332 | 0.234 | 0.762
EBCA[19] | 0.291 | 0.215 | 0.722 | 0.317 | 0.229 | 0.732 | 0.321 | 0.231 | 0.726
AnE[3] | 0.332 | 0.248 | 0.806 | 0.353 | 0.265 | 0.830 | 0.360 | 0.270 | 0.811
SG[23] | 0.311 | 0.213 | 0.767 | 0.355 | 0.264 | 0.830 | 0.355 | 0.262 | 0.811
Ours($\lambda=0)$ | 0.340 | 0.255 | 0.814 | 0.362 | 0.271 | 0.851 | 0.360 | 0.270 | 0.823
Ours | 0.340 | 0.256 | 0.817 | 0.362 | 0.270 | 0.851 | 0.366 | 0.274 | 0.836

We compare our generation results with SD, CD, StrD, EBCA, AnE, and SG on two
artificial datasets, AnE dataset [3] and DVMP [23], and one natural-language
dataset, ABC-6K [6]. We refer to supplementary material for implementation
details and computational efficiency comparison.

#### Datasets

The AnE dataset [3] comprises three benchmarks: Animal-Animal, Animal-Object,
and Object-Object. Each benchmark varies in complexity and incorporates a
combination of potentially colored animals and objects. The prompt patterns
for these benchmarks include two unattributed animals, one unattributed animal
and one attributed object, and two attributed objects, respectively. The DVMP
dataset [23] features a diverse set of objects (e.g., daily objects, animals,
fruits, etc.) and diverse modifiers including colors, textures and so on. It
features more than three attribute descriptor per prompt. The ABC-6K dataset
[6], derived from natural MSCOCO [14] captions, includes prompts with at least
two color words modifying different objects. The first two datasets are
artificial, while ABC-6K is composed of natural language captions.

Figure 3: Full Sim. results on DVMP and ABC-6K datasets. We randomly sample
200 prompts from each dataset and generate 4 images for each prompt.

###  5.1 Quantitative Comparison

Following the setting of [3], we compare the Text-Image Full Similarity (Full
Sim.), Text-Image Min Similarity (Min. Sim.), and Text-Caption Similarity (T-C
Sim.) on the AnE dataset. Additionally, we present the Full Sim. results on
the DVMP and ABC-6K datasets.

Full Sim. is the CLIP [20] cosine similarity score between the text prompt and
the generated image. Furthermore, we assess CLIP similarity for the most
neglected object independently from the full text by computing the CLIP
similarity scores between each sub-prompt and the generated image. The smaller
score is denoted as Min. Sim.. T-C Sim. is the average CLIP similarity between
the prompt and all captions generated by a pre-trained BLIP image-captioning
model [13] with the generated image as input.

We generate 64 images for each prompt using the same seed across all methods
and compute the average score between each prompt and its corresponding
images. Our method consistently demonstrates superior performance across all
datasets, as shown in Tab. 1. We stress the following advantages of our
method: (1) Our method distinguishes itself from SG by its adaptability to the
Animal-Animal dataset, even when the prompts lack specific attributes; (2) Our
method with $\lambda=0$ surpasses AnE and SG in all cases, underscoring the
effectiveness of our object-centric attribute binding loss; (3) As the dataset
becomes more complicated, our method with hyper-picked $\lambda$ gains a more
significant advantage over that with $\lambda=0$.

In Fig. 3, despite SG’s deliberate design for multi-attribute prompts, our
method consistently surpasses SG. Furthermore, in the ABC-6K dataset, AnE and
SG exhibit performance levels similar to that of SD, while our method
consistently achieves superior results. These advantages are further confirmed
by our human evaluation in 5.3.

###  5.2 Qualitative Comparison

SD |  |  |
---|---|---|---
AnE |  |  |
SG |  |  |
Ours |  |  |
|  an orange backpack and a purple car |  a red suitcase and a blue apple |  a purple crown and a blue suitcase
Figure 4: Qualitative comparison on the AnE dataset. Each column shares the same random seed. SD |  |  |
---|---|---|---
AnE |  |  |
SG |  |  |
Ours |  |  |
|  a purple modern camera and a spotted baby dog and a sliced tomato |  a red metal crown and a white bear and a wooden chair |  an orange suitcase and a sliced strawberry and a baby mouse
Figure 5: Qualitative comparison on the DVMP dataset. Each column shares the same random seed. SD |  |  |
---|---|---|---
AnE |  |  |
SG |  |  |
Ours |  |  |
|  A man with glasses, earrings, and a red shirt with blue tie. |  A red kitty cat sitting on a floor near a dish and a white towel. |  Two tan boats on dock next to large white buildings.
Figure 6: Qualitative comparison on the ABC-6K dataset. Each column shares the
same random seed.

In Figs. 4-6, we identify recurrent failure modes in SG and AnE, attributable
to the ineffectiveness of their objective design. AnE frequently struggles
with incorrect attribute association, whereas SG often fails to generate
multiple objects simultaneously. In contrast, our method attains high-quality
semantic alignment with deliberately designed optimization objective. It also
exhibits more stable performance across different random seed selections.

#### Object Omission

SG, due to its pair-centric approach, frequently omits objects, as evidenced
by missing items like cars, apples, and crowns in Fig. 4, tomatoes, crowns,
strawberries in Fig. 5, and earrings, ties, etc. in Fig. 6.

#### Attribute Omission

Due to a lack of concern for attribute tokens, AnE fails to overcome the
strong visual priors over objects. e.g., the green apple in Fig. 4, the non-
spotted dog in Fig. 5, and the brown cat in Fig. 6.

#### Attribute Leakage

In the case of SG, examples include purple on the wall, blue spilled on the
plants, and purple on the suitcase in Fig. 4, illustrating how attributes
emerge as leakage when the respective object is absent. Additional examples
include the tomato’s color spilling onto the dog and red metal leaking onto
the chair in Fig. 5, as well as blue color leaking and forming artifacts in
Fig. 6. AnE, with its sole focus on intensity, also suffers significantly from
attribute leakage, evident in the purple backpack and blue suitcase in Fig. 4,
the red metal chair in Fig. 5, and the blue glasses and earrings, and red
towel in Fig. 6.

We argue that addressing object neglect or attribute binding in isolation is
insufficient, as these issues are intrinsically interconnected. Our method
adeptly balances these two concerns with a chosen intensity weight $\lambda$,
demonstrating its success in addressing the challenges above.

###  5.3 Human Evaluation

Recent work [33, 2] has found that large Vision-and-Language Models (VLMs)
[20, 27, 13, 34] demonstrate a significant lack of compositional
understanding, failing to reflect human preferences accurately. Given this, we
conducted human evaluations across all three datasets to rigorously assess our
model’s performance.

Raters were enlisted online, with the requirement that each participant
possessed an educational level of a bachelor’s degree or higher. In the
process of evaluation, they were presented with 2-way multiple choice problems
consisting of a text prompt and two images generated by our method and one of
four baselines, including SD, AnE, SG, and our method with $\lambda=0$. For
each dataset, 100 prompts were randomly sampled for evaluation. The
effectiveness of prompt-image alignment was assessed by asking raters, "Which
image better matches the given description?". More details are provided in
supplementary material.

Figure 7: Preference ratio percentage on text-image alignment by human
evaluation. Ours(avg) represents the average preference ratio of our method
compared with the other four methods.

The human evaluation results are shown in Fig. 7. Notably, the object count in
DVMP dataset prompts was evenly distributed among $\\{1,2,3\\}$. In cases with
only one object, $\lambda$ likely has no significant effect, leading to
similar performance in image-text alignment regardless of the choices of
$\lambda$. This observation partly explains why our method performs similarly
to that with $\lambda=0$ . Consequently, we recommend selecting different
intensity weights based on the number of objects in a prompt to optimize our
method’s performance. Furthermore, we observe: 1) our method consistently
surpasses SD, AnE, and SG aligned with quantitative results in Tab. 1 and Fig.
3; 2) our method shows a more pronounced advantage over other methods on the
natural-language ABC-6K dataset. We argue that our object-centric objective,
in harmony with the object-oriented patterns prevalent in naturally occurring
prompts, exhibits superior efficacy in handling complex real-world, natural-
language-based prompts.

Table 2: Ablation results on repulsive term.  Both Ours and Ours($\lambda=0$) benefit from the repulsive term as defined in Eqn. (5). |  | Animal-Animal | Animal-Object | Object-Object
---|---|---|---|---
Method | Repul. | Full Sim. | Min. Sim. | T-C Sim. | Full Sim. | Min. Sim. | T-C Sim. | Full Sim. | Min. Sim. | T-C Sim.
Ours($\lambda=0)$ | ✗ | 0.311 | 0.213 | 0.767 | 0.343 | 0.246 | 0.794 | 0.334 | 0.237 | 0.765
| ✓ | 0.340 | 0.255 | 0.814 | 0.362 | 0.271 | 0.851 | 0.360 | 0.270 | 0.823
Ours | ✗ | 0.338 | 0.250 | 0.810 | 0.360 | 0.267 | 0.841 | 0.359 | 0.269 | 0.819
| ✓ | 0.340 | 0.256 | 0.817 | 0.362 | 0.270 | 0.851 | 0.366 | 0.274 | 0.836

###  5.4 Ablation Study

$\lambda=0.0$ |  |  |
---|---|---|---
$\lambda=0.5$ |  |  |
$\lambda=1.0$ |  |  |
| (a) | (b) | (c)
Figure 8: Ablation demonstration for intensity weight $\lambda$. (a) a sliced
apple and a purple camera and a teal lion; (b) a brown bear with red hat and
scarf and a small stuffed bear; (c) a gray crown and a purple apple. We have
selected one prompt from each dataset to showcase the stability of our method.
Each column shares the same random seed.

#### Repulsive Term

Tab. 2 presents the results of ours(${\lambda=0}$) w/o and w/ the repulsive
term in rows 1 and 2, and similarly, ours w/o and w/ this term in rows 3 and
4, under the same settings as Tab. 1. Row 2/4 demonstrates a significant
performance increase than Row 1/3 due to the repulsive term, validating the
effectiveness of negative sampling approximation.

#### Intensity Weight

We demonstrate the impact of different choices for the intensity weight
$\lambda$, which plays a role in enhancing the intensity level. In Fig. 8, we
present some representative examples where the model needs to generate
multiple objects with certain modifiers. When $\lambda=0.5$, the generation is
balanced. However, when $\lambda=0.0$, all images more or less suffer from
object neglect, e.g. the camera in (a), the stuffed bear in (b), and the crown
in (c). Conversely, when $\lambda=1.0$, artifacts are likely to appear, e.g.
the camera in (a), the brown bear in (b), and the apple in (c); attribute
binding becomes less effective, e.g. the purple camera in (a), the stuffed
bear in (b) and the purple apple in (c).

###  5.5 Augmented Attribute Editing

PtP [7] allows local editing with word swap, adding new phrase, or re-
weighting by replacing the attention weights of unchanged tokens, or re-
weighting the attention maps of target tokens. This heavily relies on the
semantic coupling between tokens and their attention maps. In Fig. 9, we
categorize the failure cases in PtP shown in the left panel into four
situations: (a) ineffective editing with aligned text-to-image generation; (b)
ineffective editing with incorrect attribute binding, e.g. the semantic
leakage of ‘pink’; (c) ineffective editing with object neglect, e.g. the
‘apple’; and (d) insignificant editing with aligned text-to-image generation,
e.g. the property ‘metal’ for the drum. In contrast, our method effectively
enhances the semantic distribution of attention maps, allowing PtP with our
approach to apply effective and significant local attribute editing to the
original images, as demonstrated in the right panel. More implementation
details are provided in supplementary material.

Original $\longrightarrow$ Edited | Original $\longrightarrow$ Edited
---|---
|  |  |
(a) a yellow($\rightarrow$ green) guitar
|  |  |
(b) a (+ blue) frog on a pink bench
|  |  |
(c) a green backpack and a black($\rightarrow$ purple) apple
|  |  |
(d) a dog is playing a leather($\rightarrow$ metal) drum on the beach
Figure 9: Augmented attribute editing with our method. The left two columns
demonstrates the attribute editing results from PtP, while the right two
columns demonstrates the results from PtP w. ours. Each row shares the same
random seed.

##  6 Limitations

We list the limitations of our model as follows: (1) The effectiveness of our
method is limited by the expressive power of the standard Stable Diffusion
model. (2) Our energy-based model is conditioned on the object token $s\in S$.
Typically, $|S|\neq 0$. However, when $|S|=0$, it indicates that there are no
objects in the prompt, and our method is degraded to regular diffusion model
generation.

##  7 Conclusion

We introduce an object-conditioned EBAMA framework to address the alignment
issues in text-to-image diffusion models. We propose an object-centric
attribute binding loss that maximizes the log-likelihood of the object-
conditioned EBM in the attention feature space. An intensity regularizer is
further designed to provide an extra degree of freedom balancing the trade-off
between correct attribute binding and the necessary presence of objects.
Extensive quantitative and qualitative comparisions demonstrate the
superiority of our method in aligned text-to-image generation. This
advancement promises great improvements in text-controlled attention-based
image editing with semantically aligned attention maps.

##  8 Societal Impact and Ethical Concerns

This paper introduces a novel method for aligning attention maps to improve
the compositional generation of images from text prompts, using diffusion
models. Our approach aims to advance AI-assisted visual content creation in
the creative, media, and communication sectors. Although we foresee no
significant ethical issues unique to our method, it inherits general concerns
associated with generative models, such as privacy, copyright infringement,
misuse for deceptive content, and content bias.

## References

  * [1] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al.: ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
  * [2] Chang, Y., Zhang, Y., Fang, Z., Wu, Y., Bisk, Y., Gao, F.: Skews in the phenomenon space hinder generalization in text-to-image generation. arXiv preprint arXiv:2403.16394 (2024)
  * [3] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG) 42(4), 1–10 (2023)
  * [4] Conwell, C., Ullman, T.: Testing relational understanding in text-guided image generation. arXiv preprint arXiv:2208.00005 (2022)
  * [5] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873–12883 (2021)
  * [6] Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A.R., Narayana, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-free structured diffusion guidance for compositional text-to-image synthesis. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=PUIqjT4rzq7
  * [7] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-or, D.: Prompt-to-prompt image editing with cross-attention control. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=_CDixzkzeyb
  * [8] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
  * [9] Honnibal, M., Montani, I.: spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear 7(1), 411–420 (2017)
  * [10] Hoogeboom, E., Heek, J., Salimans, T.: simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301.11093 (2023)
  * [11] Hoover, B., Liang, Y., Pham, B., Panda, R., Strobelt, H., Chau, D.H., Zaki, M.J., Krotov, D.: Energy transformer. arXiv preprint arXiv:2302.07253 (2023)
  * [12] Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences 79(8), 2554–2558 (1982)
  * [13] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [14] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [15] Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models. In: European Conference on Computer Vision. pp. 423–439. Springer (2022)
  * [16] McEliece, R., Posner, E., Rodemich, E., Venkatesh, S.: The capacity of the hopfield associative memory. IEEE transactions on Information Theory 33(4), 461–482 (1987)
  * [17] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013)
  * [18] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)
  * [19] Park, G.Y., Kim, J., Kim, B., Lee, S.W., Ye, J.C.: Energy-based cross attention for bayesian context update in text-to-image diffusion models. arXiv preprint arXiv:2306.09869 (2023)
  * [20] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [21] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
  * [22] Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G.K., et al.: Hopfield networks is all you need. arXiv preprint arXiv:2008.02217 (2020)
  * [23] Rassin, R., Hirsch, E., Glickman, D., Ravfogel, S., Goldberg, Y., Chechik, G.: Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment (2023)
  * [24] Rassin, R., Ravfogel, S., Goldberg, Y.: Dalle-2 is seeing double: Flaws in word-to-concept mapping in text2image models. arXiv preprint arXiv:2210.10606 (2022)
  * [25] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10684–10695 (June 2022)
  * [26] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding (2022)
  * [27] Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., Kiela, D.: Flava: A foundational language and vision alignment model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15638–15650 (2022)
  * [28] Xie, J., Lu, Y., Zhu, S.C., Wu, Y.: A theory of generative convnet. In: International Conference on Machine Learning. pp. 2635–2644. PMLR (2016)
  * [29] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B.K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., Wu, Y.: Scaling autoregressive models for content-rich text-to-image generation (2022)
  * [30] Yu, P., Xie, S., Ma, X., Jia, B., Pang, B., Gao, R., Zhu, Y., Zhu, S.C., Wu, Y.N.: Latent diffusion energy-based model for interpretable text modeling. arXiv preprint arXiv:2206.05895 (2022)
  * [31] Yu, P., Xie, S., Ma, X., Zhu, Y., Wu, Y.N., Zhu, S.C.: Unsupervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems 34, 14264–14279 (2021)
  * [32] Yu, P., Zhu, Y., Xie, S., Ma, X.S., Gao, R., Zhu, S.C., Wu, Y.N.: Learning energy-based prior model with diffusion-amortized mcmc. Advances in Neural Information Processing Systems 36 (2024)
  * [33] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? In: The Eleventh International Conference on Learning Representations (2022)
  * [34] Zeng, Y., Zhang, X., Li, H.: Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276 (2021)

## Supplementary Material

##  Appendix A Object-Conditioned Energy-Based Model

In this section, we present the proof for Equ. (4). Specifically, we elaborate
on the derivation of the gradient of the log-likelihood for the EBM as defined
in Equ. (3):

$\displaystyle\nabla_{z}\log p_{z}(l|s)$ $\displaystyle=$
$\displaystyle\nabla_{z}\log(\exp(f(A_{l},A_{s})))-\nabla_{z}\log(\sum_{l}\exp(f(A_{l},A_{s})))$
$\displaystyle=$
$\displaystyle\nabla_{z}f(A_{l},A_{s})-\sum_{l}\frac{\exp(f(A_{l},A_{s}))}{\sum_{l}\exp(f(A_{l},A_{s}))}\nabla_{z}f(A_{l},A_{s})$
$\displaystyle=$
$\displaystyle\nabla_{z}f(A_{l},A_{s})-\sum_{l}p_{z}(l|s)\nabla_{z}f(A_{l},A_{s})$
$\displaystyle=$
$\displaystyle\nabla_{z}f(A_{l},A_{s})-\mathbb{E}_{p_{z}(l|s)}\left[\nabla_{z}f(A_{l},A_{s})\right].$

##  Appendix B Algorithm

Our workflow can be outlined in Algo. 1. Initially, for the first half of the
denoising steps, the latent variable $z_{t}$ is updated using the gradient of
the loss function, i.e. Equ. (7). The latter half of the denoising steps
follows the standard generation process of diffusion models.

Algorithm 1 Energy-Based Attention Map Alignment

Input: A text prompt $y$, a set of object tokens $S$, a set of modifier tokens
$\\{\mathcal{M}(s)\\}_{s\in S}$, a pretrained Stable Diffusion model $SD$,
total sampling steps $T$, an image decoder $\mathcal{D}$
Output: An image $x$ aligned with the prompt $y$

1:Initialize $z_{T}\sim\mathcal{N}(0,1)$

2:for $t$ in $T:[T/2]+1$ do

3: $\\_,A,\tilde{A}\leftarrow SD(z_{t},t,y)$

4: Compute attention loss $L$ according to Equ. (7)

5: $z_{t}^{\prime}\leftarrow z_{t}-\nabla_{z_{t}}L$

6: $z_{t-1},\\_,\\_\leftarrow SD(z_{t}^{\prime},t,y)$

7:end for

8:for $t$ in $[T/2]:1$ do

9: $z_{t-1},\\_,\\_\leftarrow SD(z_{t},t,y)$

10:end for

11:$x\leftarrow\mathcal{D}(z_{0})$

12:return $x$

##  Appendix C Implementation Details

Experiments were conducted on a Linux-based system equipped with 4 Nvidia
R9000 GPUs, each of them has 48GB of memory. To ensure a fair comparison with
previous methods, we utilized the official Stable Diffusion v1.4 text-to-image
model with the CLIP ViT-L/14 text encoder.

###  C.1 Hyperparameters

In our approach, we utilize a default fixed guidance scale of 7.5. The update
step size is selected as $\alpha=20$. We employ a DDIM sampler with a total of
50 steps. The update of the latent variable $z_{t}$ is confined to the first
half of the denoising process, which, in this context, corresponds to the
initial 25 steps. Further discussion regarding the step size and the updated
timesteps is in Appendix F.

###  C.2 Parser

Following [23], we utilize the spaCy parser [9], specifically employing the
transformer-based en_core_web_trf model. Initially, we identify tokens within
the prompt that are tagged as either NOUN or PNOUN , thereby constituting our
object set. Subsequently, we extract all modifiers within this set based on a
predefined set of syntactic dependencies, which include amod, nmod, compound,
npadvmod, and conj. Finally, any NOUN or PNOUN that functions as a modifier
for other entity-nouns within the object set is excluded.

###  C.3 Attention Map Extraction

The aggregated attention features $A_{t}$ comprises $N$ spatial attention
maps, each corresponding to a token of the input prompt $y$. The CLIP text
encoder appends a specialized $\langle\texttt{SOT}\rangle$ token at the
beginning of $y$ to signify the start of the text. It has been observed that
in Stable Diffusion, the $\langle\texttt{SOT}\rangle$ token consistently
receives the highest attention among all the tokens. Following [3], we exclude
the attention allocated to $\langle\texttt{SOT}\rangle$ and then apply a
softmax operation to the remaining tokens to obtain attention scores
$\tilde{A}_{t}$.

###  C.4 Augmented Attribute Editing Setup

We utilize and slightly adapt the official repository from [7] for conducting
attention editing. A cross-replace step of 0.8, i.e. replacing the first 80%
steps of cross-attention maps, is employed for all editing tasks.
Additionally, in line with the repository’s provisions, self-attention maps
also play a role in preserving the image’s shape. For this purpose, we set the
self-replace step at 0.4, i.e. replacing the first 40% steps of self-attention
maps, for all the experiments.

##  Appendix D Computational Efficiency

We randomly sampled a total of 100 prompts from the ABC-6K dataset and
compared the time required for each method. Since there are no updates in SD,
it provides us with the lower bound of the time needed for generating the
images. As shown in Tab. 3, we can observe that AnE takes the longest time,
approximately 47.0 minutes, to complete the generation process, while our
method and SG require less than half of that time. The reason for this
discrepancy is that AnE employs a technique called iterative latent
refinement, which may require multiple updates at certain steps when the loss
objective does not meet specific thresholds. We emphasize that our method is
computationally efficient while still achieving top performance.

Table 3: Computational time comparison. We compare the time required for generating images for 100 prompts in ABC-6K dataset for each method. | SD | AnE | SG | Ours
---|---|---|---|---
Time(min) | 13.2 | 47.0 | 21.9 | 20.9

##  Appendix E External modifiers incorporation

We modify our workflow shown in Fig. 2 to include external modifiers without
affecting the original prompt’s guidance: (a) We first append the modifier(s)
to the prompt to obtain attention maps for both original and external tokens.
(b) We calculate the final loss using these attention maps. (c) After updating
$z_{t}$, we drop the external modifier(s) and proceed with the original
prompt. Tab. 4 presents the results of incorporating arbitrary syntactically
unrelated modifiers, with their number denoted as $n$. The modifiers chosen
are ‘red’($n=1$), and ‘red blue green’($n=3$). The performance decreases with
$n$ increasing. This decline may be due to external tokens interfering with
the repulsion of other non-modifier tokens, as the attribute binding loss has
to balance the repulsion of external tokens and that of non-modifier tokens.

$n$ | A-A | A-O | O-O
---|---|---|---
0 | 0.340/0.256/0.817 | 0.362/0.270/0.851 | 0.366/0.274/0.836
1 | 0.337/0.254/0.812 | 0.360/0.269/0.844 | 0.364/0.272/0.826
3 | 0.331/0.248/0.822 | 0.357/0.266/0.846 | 0.362/0.270/0.823
Table 4: Results on external modifiers incorporation under the same setting as
Tab. 1 . $n$ denotes the number of external modifiers appended.

##  Appendix F Additional Ablation Experiments

#### Intensity Weight $\lambda$

We explore various settings of the intensity weight parameter $\lambda$ as
illustrated in Fig. 10, where the metrics are computed across 10 images for
each prompt. The values of Text-Image Full Similarity (Full. Sim.) and Text-
Caption Similarity (T-C Sim.) are presented as functions of varying $\lambda$.
At $\lambda=0$, the intensity level is disregarded by the method. Conversely,
increasing $\lambda$ shifts the focus more towards the intensity level, at the
expense of distribution alignment in attention maps.

For Animal-Animal and Object-Object , both metrics peak at $\lambda=0.5$. For
the Animal-Object dataset, the Text-Image similarity attains its highest score
at $\lambda=0$ or $\lambda=0.25$. Given that Text-Caption Similarity is
maximal at $\lambda=0.25$, this value is selected for the Animal-Object
dataset.

Our analysis indicates that $\lambda$ effectively balances the trade-off
between intensity level and attribute binding. Extremes of $\lambda$ (e.g.,
$1.0$ or $0.0$) yield suboptimal generation results. Based on these findings,
and considering that the ABC-6K dataset is more akin to the Object-Object
dataset – notably, they both contain at least two attributes and two objects –
we select $\lambda=0.5$ for the ABC-6K dataset in all our experiments. The
situation with the DVMP dataset is more intricate due to its potential for
containing one to three objects and an unlimited number of attributes. As we
suggest in the main text, selecting different intensity weights based on the
number of objects in a prompt is recommended to optimize our method’s
performance.

Figure 10: Ablation study for $\lambda$. We generated 10 images for each prompt with the same seed across all methods. The results indicate that for datasets with Animal-Animal and Object-Object pairings, a setting of $\lambda=0.5$ is optimal; whereas for the Animal-Object dataset, $\lambda=0.25$ yields the best performance. $T^{\prime}=0$ |  |  |
---|---|---|---
$T^{\prime}=25$ |  |  |
$T^{\prime}=50$ |  |  |
| (a) | (b) | (c)
Figure 11: Ablation demonstration for updated timesteps $T^{\prime}$. (a) an
orange and white cat sitting in the grass near some yellow flowers; (b) red
roses in a square green vase; (c) A room with pink walls and white display
shelves and chair. Each column shares the same random seed.

#### Number of updated timesteps $T^{\prime}$

We explore different settings for the updated timesteps, denoted as
$T^{\prime}$, which refer to the timestep numbers of updating the latent
variable $z_{t}$. This exploration is depicted in Fig. 11. When
$T^{\prime}=0$, our method defaults to the standard stable diffusion
generation, with no updates applied to the model. In this configuration, due
to the lack of interventions during the generation process, the generated
images often exhibit semantic misalignments. Examples include the yellow
flowers in (a), the red roses in (b), and the pink walls in (c). Conversely,
setting $T^{\prime}=25$ implements our proposed method, which produces images
better aligned with the input text. However, increasing $T^{\prime}$ to 50,
where $z_{t}$ is updated throughout the generation process, can introduce
artifacts. Notable instances of these artifacts are visible in the
representations of the cat in (a), the vase in (b), and the chair in (c).

$\alpha=1$ |  |  |
---|---|---|---
$\alpha=20$ |  |  |
$\alpha=40$ |  |  |
| (a) | (b) | (c)
Figure 12: Ablation demonstration for step size $\alpha$. (a) a blue zebra and
a spotted crown; (b) a living room with white walls and blue trim; (c) a green
and white sign on a black pole and some buildings. Each column shares the same
random seed.

#### Step Size $\alpha$

We investigate different settings for the step size $\alpha$, as depicted in
Fig. 12. When $\alpha$ is set to 1, the step size is too small, leading to
insufficient attribute binding and the inability to generate multiple objects
effectively. This is evident from the examples of blue walls in (b), a green
pole in (c), and the missing crown in (a). In this setting, the performance of
(a) is similar to that of SD as shown in the middle panel of Fig I.
Conversely, with $\alpha$ set to 40, the step size becomes excessively large,
causing an overemphasis on certain attributes, e.g. blue in (a), blue in (b),
black in (c) (note that the building behind is also black).

##  Appendix G Additional Details on Human Evaluation

Raters were enlisted via an online platform under conditions of anonymity,
with the requirement that each participant possessed an educational level of a
bachelor’s degree or higher. Additionally, they were assured of the protection
of their privacy and the confidentiality of their identities throughout the
process.

Fig. 13 provides a screenshot of the rating interface. The order of images was
randomized.

Figure 13: A screenshot of the rating interface. The order of images was
randomized.

##  Appendix H Additional Qualitative Results

We present additional qualitative results comparing our method with others in
Figs. I-I. It is observed that AnE struggles with incorrect attribute binding,
whereas SG exhibits issues related to object neglect.

##  Appendix I Additional Explanatory Results

We present more attention intensity levels and distribution maps in Figs.
16-18.

SD |  |  |
---|---|---|---
AnE |  |  |
SG |  |  |
Ours |  |  |
|  a gray crown and |  |
|  a purple balloon and |  |
|  a green chair and |  |
| a purple car |  |

Figure 14: Additional qualitative comparison. Each column shares the same
random seed. AnE: incorrect attribute binding, e.g., red apple, purple clock,
green car. SG: object neglect, e.g. crown, clock, car.

SD |  |  |
---|---|---|---
AnE |  |  |
SG |  |  |
Ours |  |  |
|  A brown bear with red hat and scarf and a small stuffed bear. |  a blue zebra and |
|  A boy in a red shirt with a helmet and yellow bat. | a spotted crown |

Figure 15: Additional qualitative comparison. Each column shares the same
random seed. AnE: object neglect, e.g. red hat, zebra; incorrect attribute
binding, e.g. gray shirt, red bat. SG: object neglect, e.g. small stuffed
bear, crown, helmet.

| Intensity Level | “blue” “apple” | Generated Image
---|---|---|---
SD |  |  |
AnE |  |  |
SG |  |  |
Ours |  |  |
Figure 16: Additional explanatory results. The text prompt is “a red suitcase
and a blue apple”. Each column shares the same random seed.

| Intensity Level | “purple” “car” | Generated Image
---|---|---|---
SD |  |  |
AnE |  |  |
SG |  |  |
Ours |  |  |
Figure 17: Additional explanatory results. The text prompt is “an orange
backpack and a purple car”. Each column shares the same random seed.

| Intensity Level | “sliced” “tomato” | Generated Image
---|---|---|---
SD |  |  |
AnE |  |  |
SG |  |  |
Ours |  |  |
Figure 18: Additional explanatory results. The text prompt is “a purple modern
camera and a spotted baby dog and a sliced tomato”. Each column shares the
same random seed.