# AttenCraft: Attention-guided Disentanglement of Multiple Concepts for Text-
to-Image Customization

Junjie Shentu1 Matthew Watson1 Noura Al Moubayed1
1Department of Computer Science  Durham University  UK
{junjie.shentu, matthew.s.watson<EMAIL_ADDRESS>
###### Abstract

With the unprecedented performance being achieved by text-to-image (T2I)
diffusion models, T2I customization further empowers users to tailor the
diffusion model to new concepts absent in the pre-training dataset, termed
subject-driven generation. Moreover, extracting several new concepts from a
single image enables the model to learn multiple concepts, and simultaneously
decreases the difficulties of training data preparation, urging the
disentanglement of multiple concepts to be a new challenge. However, existing
models for disentanglement commonly require pre-determined masks or retain
background elements. To this end, we propose an attention-guided method,
AttenCraft, for multiple concept disentanglement. In particular, our method
leverages self-attention and cross-attention maps to create accurate masks for
each concept within a single initialization step, omitting any required mask
preparation by humans or other models. The created masks are then applied to
guide the cross-attention activation of each target concept during training
and achieve concept disentanglement. Additionally, we introduce Uniform
sampling and Reweighted sampling schemes to alleviate the non-synchronicity of
feature acquisition from different concepts, and improve generation quality.
Our method outperforms baseline models in terms of image-alignment, and
behaves comparably on text-alignment. Finally, we showcase the applicability
of AttenCraft to more complicated settings, such as an input image containing
three concepts. The project is available at https://github.com/junjie-
shentu/AttenCraft.

##  1 Introduction

Diffusion models have demonstrated unprecedented capabilities in generating
high-quality and diverse images [1, 2]. By leveraging cross-attention layers
within the UNet architecture [3], diffusion models can be conditioned on
information in different modalities. Text-to-image (T2I) diffusion models, a
specific subclass of conditional diffusion models, exhibit remarkable
proficiency in producing images semantically aligned with natural language
prompts [4, 5, 6, 7].

Despite the strong image-text connections established by T2I models,
introducing new concepts not present in pre-training datasets remains
challenging [8]. In response, studies aimed at "customizing" T2I models for
generalization to newly introduced concepts propose fine-tuning pre-trained
models with a few or even a single image of the target object, creating
subject-driven T2I models [8, 9, 10, 11, 12, 13, 14, 15, 16]. During subject-
driven T2I learning, the visual representation is "revered" to an identifier
token $\rm[V]$ through the cross-attention mechanism and then generalized to
various contexts [8]. However, prevalent subject-driven T2I models are
designed to learn from images containing a single new concept, imposing high
requirements for data preparation and demanding a prolonged fine-tuning
process when introducing multiple concepts [10].

A few studies have explored learning multiple concepts from a single image or
a local part of the image [17, 18, 19, 20, 21]. There are two prominent
strategies for disentangling multiple concepts that can be identified. The
first leverages masks [17, 18, 19, 20] as conditional information to guide the
cross-attention activation during training, while the second directly
calibrates the cross-attention to focus on different concepts in the given
image. However, both strategies show drawbacks. The first strategy requires
masks provided by specialized segmentation models or humans [17, 20], and the
second strategy retains the background [21].

In this study, we propose a novel method, termed AttenCraft, to disentangle
multiple concepts from a single image for subject-driven T2I generation.
Specifically, we utilize both self-attention and cross-attention maps to
create accurate masks for each target concept within a single step without
relying on specialized segmentation models or human intervention. Several
techniques are incorporated, including Cross-attention suppression, Self-
attention enhancement, and Delta masking, to refine the masks. The masks are
updated during training to cover more details, and we also apply two
regularization techniques to maintain mask quality. The masks are then
leveraged to guide the cross-attention activation to disentangle multiple
concepts during training. By approaching the cross-attention map between the
input image and identifier token $\rm[V]$ to the corresponding mask, a
distinct connection between $\rm[V]$ and the visual representation of the
target concept can be established.

Another challenge emerges as the non-synchronized learning between each
individual concept and the concept group when applying the existing models, as
shown in Fig. 1. We hypothesize the non-synchronicity is induced by an
improper ratio of concept sampling, with a more detailed interpretation given
in Section 3.3. Therefore, we investigate the effects of the sampling scheme
and propose a Uniform sampling scheme that can better balance the feature
acquisition of various target concepts. Following the results obtained from
Uniform sampling, we introduce the Reweighted sampling scheme to further
improve the synchronicity and generation quality through the adaptive sampling
ratio for each target concept. A qualitative comparison of feature learning
synchronicity on the "baby & toy" dataset is presented in Fig. 1, indicating a
significant improvement in learning synchronicity achieved by our method. In
summary, our contributions are:

  * •

We propose AttenCraft to disentangle multiple concepts in a single image for
customized T2I generation, which creates accurate masks per target concept
within a single step based on attention maps, without requiring human labeling
or specialized segmentation models.

  * •

We introduce two sampling strategies, Uniform sampling and Reweighted
sampling, which enhance the learning synchronicity of various concepts and
improve generation quality.

  * •

We evaluate the performance of our method and baseline models, proving that
our method achieves the best image-alignment and behaves comparably to
baseline models on text-alignment. We also demonstrate our method applies to
input images containing more concepts.

Figure 1: Synchronicity in learning multiple concepts. Given an image
containing two concepts, DisenDiff overfits the concept group and generates
collapsed images. The baby image is proper but the toy image is not authentic.
Break-a-scene shows a collapsed baby in the group image and the baby image,
while the toy image lacks details of the target toy. In contrast, our method
with uniform sampling presents proper details of each concept and no collapse
appears, and our method with Reweighted sampling shows more details.

##  2 Related Work

#### Diffusion models and customized T2I generation.

Diffusion models achieve state-of-the-art performance in image generation,
surpassing other generative models in terms of authenticity and edibility [1,
2]. Utilizing pre-trained text encoders [22, 23], diffusion models implement
the T2I diffusion model in pixel space under classifier-free guidance [4, 24,
25]. Stable Diffusion (SD) [5] trains the denoising UNet [3] in latent space
by applying Variational Autoencoder (VAE) [26] and the text encoder of
Contrastive Language-Image Pre-training (CLIP) [27]. Furthermore, subject-
driven T2I models [8, 9] train the model to learn visual representations of a
new concept from several images and reverse to an identifier token $\rm[V]$
for generalization. In addition, parameter-efficient tuning (PEFT) [28] is
conducted to reduce training time by utilizing a smaller set of trainable
parameters, comprising the cross-attention layers [10, 29, 21, 30], Low-rank
Adaptation (LoRA [31]) parameters [15, 32, 33, 34], and additional encoder,
adapter or weight offset [35, 13, 36]. Moreover, some studies pre-train a
universal encoder capable of directly encoding input images [16, 11, 37, 38,
39]. However, the majority of current subject-driven T2I models focus on input
images containing a single concept, neglecting the exploration of extracting
multiple concepts from a single image.

#### Attention application in diffusion models.

The Attention mechanism plays a crucial role in T2I generation by manipulating
feature dependencies between images and text. Guided by cross-attention during
inference, pre-trained diffusion models can exhibit superior semantic
alignment with provided text prompts [40, 41, 42, 43], achieve image editing
[44, 45], and provide positional control [46, 47, 48, 43, 36]. Moreover,
cross-attention guidance is applied during model training to eliminate
background interference or concentrate on specific regions in input images
using provided masks [16, 38, 35, 33, 17, 20, 49]. Meanwhile, self-attention
can promote subject consistency across different contexts [50] or facilitate
subject swaps while preserving style consistency [51]. Furthermore, the self-
attention and cross-attention maps are applied to achieve unsupervised
segmentation [52] and augmentation of the segmentation datasets [53, 54, 55].

#### Disentangling multiple concepts from a single image.

Break-a-scene disentangles multiple concepts from a single image by applying
masks provided by users or specialized segmentation models [56] to guide
cross-attention activation. Meanwhile, Saffee et al. [20] adopt human-
determined masks to learn local features of a given concept and applies them
to locally edit other images. It also proposes an automatic mask
identification method that leverages the cross-attention map of a special
token. Similarly, Jin et al. [18] apply a fixed threshold on the cross-
attention maps to obtain the mask. Furthermore, Rahman et al. [19] utilize
dense conditional random field (CRF) [57], and Hao et al. [35] applies Otsu
thresholding [58], to obtain masks from cross-attention maps. However, these
automatic masks are typically coarse [18, 19], time-consuming [20], and often
fail to separate different concepts [35]. DisenDiff [21] calibrates cross-
attention to encourage the model to distribute its attention among different
concepts and achieve disentanglement without requiring masks, but fails to
exclude the background. Unlike existing methods, our proposed approach does
not rely on pre-determined masks and can efficiently disentangle multiple
concepts, along with the background, from a single input image using self-
generated accurate masks guided by the attention mechanism.

##  3 Proposed Method

Figure 2: Method overview. (a) Our method creates accurate masks for each
target concept within a single mask initialization step based on attention
maps. The initialized masks are then applied to guide the cross-attention
activation for disentanglement. (b) The mask creation process includes several
techniques: Cross-attention suppression, Self-attention enhancement, and Delta
masking, which transform coarse cross-attention maps to accurate masks. (c)
The masks are updated during training to cover more details in the target
region.

###  3.1 Preliminaries

For an input image $x\in\mathbb{R}^{H\times W\times 3}$, Stable Diffusion (SD)
projects $x$ into a latent representation $z\in\mathbb{R}^{h\times w\times c}$
via a VAE encoder $\mathcal{E}$ [26], where $c$ is the latent feature
dimension. The downsampling follows a scale factor $f=H/h=W/w$. In T2I
generation, the text prompts $y$ are projected into text embeddings
$\tau_{\theta}(y)$, where $\tau_{\theta}$ is a pre-trained CLIP text encoder.
The UNet is trained to predict the randomly added noise $\varepsilon$ given
the noisy latent $z_{t}$, the timestep $t$, and conditioning:

$L_{LDM}=\mathbb{E}_{z,y,t,\varepsilon\sim\mathcal{N}(0,1)}\left[\left\|\varepsilon-\varepsilon_{\theta}\left(z_{t},t,\tau_{\theta}(y)\right)\right\|^{2}_{2}\right]$
(1)

where $\varepsilon$ and $\varepsilon_{\theta}$ are standard Gaussian noise and
predicted noise residue, respectively. The UNet in SD comprises 16 Transformer
layers incorporating self-attention and cross-attention layers to capture the
dependencies within the input data [22, 5]. Specifically, the self-attention
layers capture the global attention across the image while the cross-attention
layers learn to attend between the image and text prompts. The attention maps
$A^{(i)}_{S}$ and $A^{(i)}_{C}$ in layer $i$ can be calculated as follows:

$A_{S}^{(i)}=softmax\left(\frac{Q^{(i)}_{I}K_{I}^{{(i)}^{T}}}{\sqrt{d_{i}}}\right),A_{C}^{(i)}=softmax\left(\frac{Q^{(i)}_{I}K_{T}^{{(i)}^{T}}}{\sqrt{d_{i}}}\right)$
(2)

where $d_{i}$ denotes the output dimension of the query ($Q$) and key ($K$)
features in layer $i$. $Q_{I}$, $K_{I}$ and $K_{T}$ represent the query
feature of $z_{t}$, the key feature of $z_{t}$, and the key feature of
$\tau_{\theta}(y)$, respectively.

###  3.2 Attention-guided mask creation

#### Creating masks based on attention.

The cross-attention map $A_{C}$ delineates the location and outline of the
target concept. However, the raw cross-attention maps exhibit coarse
granularity and noise, presenting two challenges in mask creation: (1) While
$A_{C}$ displays robust attention activation within the target region, weaker
activation may occur elsewhere; (2) attention activation is unevenly
distributed, resulting in an incomplete shape, as shown in Fig. 2(b). To
mitigate the first challenge, we apply Cross-attention suppression [21]
following the left part of Eq. 3:

$\hat{A}_{C}=\left(A_{C}\right)^{\upsilon},A_{C}^{S}=\left(A_{S}\right)^{\tau}\otimes\hat{A}_{C}$
(3)

Since the attention maps are produced by a Softmax operation, the activation
value ranges from 0 to 1. Consequently, element-wise multiplication of $A_{C}$
by $\upsilon$ can diminish weak activation in non-target regions, but it also
exacerbates uneven activation. To this end, we employ Self-attention
enhancement [54], which multiplies self-attention maps $A_{S}$ to the power of
$\tau$, and then multiplies $A_{S}^{\tau}$ with $\hat{A}_{C}$ to enhance the
smoothness and precision of $\hat{A}_{C}$, as depicted in the right part of
Eq. 3. $A_{S}$ encompasses pairwise correlations among patches in $z_{t}$,
facilitating the propagation of attention activation to regions associated
with the same concept, with reduced activation in non-target regions.
Similarly, performing element-wise multiplication on $A_{S}$ to the power of
$\tau$ reduces the correlation between patches of different concepts.

Given $A_{C}^{S}$, we observe that the attention activation of different
tokens emphasizes various regions across the entire map. Therefore, masks can
be inferred based on the activation difference of different tokens.
Specifically, we calculate the difference between patch ${\left(p,q\right)}$
on $A_{C_{i}}^{S}$ and $A_{C_{j}}^{S}$ ($i\neq j$), and set the value of mask
$M_{i}$ to 1 if the maximum difference exceeds a predetermined threshold
$\gamma$; otherwise, it is set to 0. We term this process Delta masking, and
the calculation process is expressed as:

$M_{i}{\left(p,q\right)}=\left\\{\begin{matrix}1&\textit{if}\max(A_{C_{i}}^{S}{\left(p,q\right)}-A_{C_{j}}^{S}{\left(p,q\right)})>\gamma,i\neq
j\\\ 0&\textit{Otherwise}\end{matrix}\right.$ (4)

The process of attention-guided mask creation is depicted in Fig. 2(b).

#### Mask initialization and updating.

In our proposed method, masks can be initiated within a single step before
training. As illustrated in Fig. 2(a), we select an initial denoising timestep
$t_{0}$ ranging from 0 to 300 in the DDPM noise schedule [1] for noise
prediction. During prediction, the attention layers process correlations
across the input data, forming attention maps. The noisy latent $z_{t_{0}}$ in
the interval of $\left[0,300\right]$ contains finer details, conducive to
obtaining precise attention maps, while a higher $t_{0}$ (e.g., 900) results
in a noisy latent $z_{t_{0}}$ with coarser information, limiting the quality
of masks [45]. An illustration of the attention maps and masks generated at
different $t_{0}$ is present in A.1. Note that the calculated denoising loss
is not back-propagated during the initialization.

The initial mask is not flawless, as illustrated in Fig. 2, where the cat mask
excludes the ears. To address this, we propose regularly updating masks
following the pipeline depicted in Fig. 2(b) after every $n$ training steps.
During training, the cross-attention maps are guided to align with the masks,
resulting in more concentrated attention activation within the target regions,
thereby enhancing mask accuracy. Conversely, refined masks guide cross-
attention activation to encompass more details in the target regions. This
reciprocal process forms a cycle of synergistic optimization, and the mask
update is depicted in Fig. 2(c).

#### Regularization.

In Self-attention enhancement, the multiplication between
$\left(A_{S}\right)^{\tau}$ and $\hat{A}_{C}$ may induce "attention leakage".
Concretely, if attention activation extends beyond the target region to the
background, multiplication with $A_{S}$ intensifies the activation in the
background, leading the masks to cover the background. Consequently, attention
activation may divert to the background region in subsequent training steps.
To address this issue, we introduce a background mask as regularization.
Specifically, we incorporate the cross-attention map of the <sot> token, which
contains background information [48], and update the background mask during
training. Additionally, since the denoising timesteps $t$ are randomly sampled
for noise predictions during training, they introduce instability to mask
creation as attention maps are coarse at a high $t$, degrading the mask
quality. Thus, we apply another regularization by evaluating the Structural
Similarity Index (SSIM) score [59] between the current and updated masks. The
current masks will be retained until the next update if the calculated SSIM
score is lower than a critical value $\eta$.

###  3.3 Uniform sampling and Reweighted sampling

Figure 1 illustrates the unsynchronized learning of multiple concepts. The
baseline models fail to capture the details of each individual concept and the
concept group, demonstrating overfitting and underfitting across different
targets. We hypothesize that an improper sampling ratio induces this non-
synchronicity. Specifically, DisenDiff utilizes a consistent text prompt
encompassing all target concepts throughout the training process, resulting in
the overfitting of the concept group. While the boy image is adequately
learned, the toy image is underfitted, suggesting varied difficulties in
learning different concepts. Break-a-scene employs a union sampling scheme,
which randomly selects a subset of $k\leq N$ concepts with the corresponding
text prompts and masks in each step, where $N$ denotes the total number of
target concepts. Paired with a masked reconstruction loss, this scheme
enhances the feature learning of subsets (e.g., single concept) and achieves
relatively better learning synchronicity than DisenDiff. Under such
conditions, only the baby image experiences collapse, while the quality of the
toy image improves. In the two-concept scenario of union sampling, the
sampling ratio for "Concept A-Concept B-Concept group" is 25%-25%-50%.

Given this observation, we posit that the sampling scheme plays a pivotal role
in balancing feature acquisition from various concepts. Therefore, we propose
reassigning the weight for sampling the single concept and the concept group.
Initially, a uniform sampling ratio is distributed for each individual concept
and concept group, termed uniform sampling. Afterward, an optimal sampling
ratio can be determined by leveraging the results of Uniform sampling as prior
knowledge, leading to more stable and synchronized learning across various
concepts, termed Reweighted sampling.

###  3.4 Training objective

Combining the attention-guided masks and new sampling schemes, the overall
training objectives are learning high-quality visual representations and
building explicit connections between them and identifier tokensm, which are
achieved by jointly applying masked reconstruction loss and cross-attention
loss. The masked reconstruction loss is formulated as:

$L_{rec}=\mathbb{E}_{z,y_{s},t,\varepsilon\sim\mathcal{N}(0,1)}\left[\left\|\varepsilon\odot
M_{s}-\varepsilon_{\theta}\left(z_{t},t,\tau_{\theta}(y_{s})\right)\odot
M_{s}\right\|^{2}_{2}\right]$ (5)

where $y_{s}$ and $M_{s}$ are text prompts and masks in the sampled subsets.
$L_{rec}$ solely focuses on reconstructing the non-masked region, thereby
mitigating the non-synchronicity issue when combined with a proper sampling
scheme. In addition, the cross-attention loss is formulated as Jensen-Shannon
divergence [60] between the cross-attention maps of the identifier tokens and
corresponding masks:

$L_{attn}=\mathbb{E}_{z,k,t}\left[\frac{1}{2}D\left(A_{C}(V_{s_{i}},z_{t})\parallel
N\right)+\frac{1}{2}D\left(M_{s_{i}}\parallel
N\right)\right],N=\frac{1}{2}\left[A_{C}(V_{s_{i}},z_{t})+M_{s_{i}}\right]$
(6)

where $A_{C}(V_{s_{i}},z_{t})$ represents the cross-attention map of the $i$th
identifier token in a sampled subset $s$ given the noisy latent $z_{t}$, and
$M_{s_{i}}$ denotes the $i$th mask in the subset. The function $D(\cdot)$
denotes the Kullback–Leibler divergence [61], given by:

$D(P\parallel Q)=\sum P(x)\log\left(\frac{P(x)}{Q(x)}\right)$ (7)

In addition, a prior preservation loss [9] is included to prevent language
drift:

$L_{prior}=\mathbb{E}_{z_{pr},y_{pr},t,\varepsilon\sim\mathcal{N}(0,1)}\left[\left\|\varepsilon-\varepsilon_{\theta}\left(z_{{pr}_{t}},t,\tau_{\theta}(y_{pr})\right)\right\|^{2}_{2}\right]$
(8)

where $z_{pr}$ is the noisy latent generated by the prior preservation prompt
$y_{pr}$. In general, the loss function during training is formulated as
follows with two scaling coefficients:

$L=L_{rec}+\alpha L_{attn}+\beta L_{prior}$ (9)

##  4 Experiments

###  4.1 Experimental settings

#### Datasets.

We perform experiments on fifteen datasets spanning various categories,
including human, animal, furniture, and object. Ten datasets are collected
from DisenDiff, characterized by relatively simple backgrounds. We also
synthesize five datasets using Gen4Gen [62], which combines multiple
personalized concepts into complex backgrounds sourced from copyright-free
platforms 111https://unsplash.com. The concepts for synthesis are curated from
the DreamBooth dataset[9] and CustomConcept101 [10].

#### Baselines.

We compare our method with Break-a-scene [17] and DisenDiff [21], which are
two representative methods for concept disentanglement. Break-a-scene uses
pre-determined masks to guide cross-attention during training, whereas
DisenDiff adopts a mask-free framework that directly calibrates the cross-
attention. Additionally, we implement Custom Diffusion [10] on our datasets to
demonstrate the effect of disentanglement.

#### Evaluation metrics.

Following baseline models [10, 17, 21], we evaluate the image- and text-
alignment. To assess image-alignment, we calculate the cosine similarity
between the CLIP embeddings of the generated images and the input images. For
text-alignment, we compute the cosine similarity between the CLIP embeddings
of the generated images and the corresponding text prompts. In particular,
identifier tokens are replaced by the class names of the target concepts for
Break-a-scene and our proposed method, while they are deleted for DisenDiff
and Custom Diffusion when evaluating text-alignment. For each dataset, we
prepare 10 text prompts for each individual concept and the concept group, and
generate 10 images under each text prompt using 50 steps of PSNR sampler with
a guidance scale of 7.5, resulting in an evaluation set consisting of 300
images.

#### Implementation details.

We utilize SD v2.1 trained on the LAION-5B dataset [63] as the foundational
model. Consistent with Textual Inversion [8], we initialize each identifier
token $\rm[V]$ using the text embedding of the corresponding class name. From
the attention layers, we extract cross-attention and self-attention maps of
dimensions $16\times 16$ and $32\times 32$, respectively, which have been
shown to contain the most semantic and visual information [44, 54]. We set the
powers $\upsilon$ and $\tau$ to 2 and 4, respectively [21, 54]. The threshold
$\gamma$ for Mask clipping is empirically set to 0.1. An illustration of
initialized masks is presented in A.4. All experiments are conducted on an
NVIDIA A100 GPU with a single input image, a batch size of 1, and a learning
rate of $5\times 10^{-5}$ for 200 steps. To save computational costs, only the
$W_{k}$ and $W_{v}$ matrices in the cross-attention layers are trained [10].
Additionally, we update the masks every $n=20$ steps and set the critical
value to $\eta=0.8$ for SSIM regularization. We present other implementation
details of baseline models and our method in A.3

###  4.2 Qualitative comparisons

We present a qualitative comparison between our method and baseline models in
Fig. 3. Upon examination, Custom Diffusion exhibits difficulty in learning
each concept separately, often matching only one of the concepts to the
identifier token. For instance, it struggles to reproduce the vase and dog in
the first row. Both DisenDiff and Break-a-scene are capable of disentangling
multiple concepts, but DisenDiff struggles to exclude background elements,
presenting occasional inclusions from the background (e.g., boot, vase, and
horse & dog). They also present deviations in the visual features of the
target concepts. For example, DisenDiff makes the horse and dog in the fifth
row the same size, and Break-a-scene does not retain all the identities of the
target concept, such as the vase and the dog in the fifth row. Moreover, image
collapse can be observed in all the baseline models, such as the "horse &
dog". In contrast, our method demonstrates proficiency in disentangling
multiple concepts, excluding background information, and preserving the
identity of target concepts. Furthermore, compared to Uniform sampling, our
method with Reweighted sampling retains more details of the target concepts,
as seen in the vase and horse examples. More examples can be found in A.4

Figure 3: Qualitative results. The qualitative comparisons between our method
and baseline models in generating an individual concept (first three rows) and
concept group (last three rows).

We also showcase the cross-attention maps of the identifier tokens alongside
the generated images in Fig. 4. Despite all models generating both target
concepts, Custom Diffusion exhibits attention activation on both concepts in
the $[\rm V_{1}]$ map in a large region encompassing the background in the
$[\rm V_{2}]$ map, suggesting its inability to disentangle multiple concepts.
Conversely, the other three methods successfully separate the cross-attention.
However, DisenDiff displays minor activation on the dog and background in the
$[\rm V_{1}]$ map, consistent with the appearance of background elements shown
in Fig. 3. Both Break-a-scene and our method demonstrate effective control of
cross-attention, with attention activations well confined to their respective
regions.

Figure 4: Visualization of cross-attention maps. Generated images and the
cross-attention maps of identifier token under text prompt: "A watercolor
painting of $[\rm V_{1}]$ and $[\rm V_{2}]$".

###  4.3 Quantitative comparisons

(a)

(b)

(c)

Figure 5: Results of quantitative comparisons. (a) Comparisons between our
method with Uniform sampling and the baseline models on all datasets; (b)
Comparisons between our method with Uniform sampling and Reweighted sampling;
(c) Ablation study.

Quantitative comparisons between our method with Uniform sampling and baseline
models are shown in Fig. 5(a). Despite Custom Diffusion achieving the highest
text-alignment score and the second-highest score in concept group image-
alignment, it exhibits the lowest image-alignment in single concept generation
due to its inability to disentangle multiple concepts, as evidenced by the
qualitative results. Break-a-scene displays lower performance in single
concept generation compared to DisenDiff but outperforms it in concept group
generation. Significantly, our method surpasses DisenDiff and Break-a-scene in
terms of image-alignment in both single concept generation and concept group
generation. Regarding text-alignment, our method also outperforms these two
baselines in concept group generation and demonstrates comparable performance
in single concept generation, with minor differences observed. Detailed
results can be referred to A.4.

Additionally, we investigate the effects of the Reweighted sampling based on
the results of Uniform Sampling. By observing the images generated by Uniform
Sampling, we select 9 datasets that present non-synchronicity in the feature
learning of different concepts, and establish an adaptive sampling ratio for
each dataset to highlight a specific concept or the group. The averaged image-
alignment and text-alignment scores of single concept generation and concept
group generation are depicted in Fig. 5(b). The application of an adaptive
sampling ratio leads to improvements in both metrics compared to using a
uniform sampling ratio, highlighting the effectiveness of Reweighted sampling.

Figure 6: Applicability to more complex settings. An input image containing
three concepts.

###  4.4 Ablation

We delve into the attention-guided mask creation and ablate various components
of our method to demonstrate their contributions, given that the quality of
the mask directly influences the cross-attention activation, thereby impacting
feature learning. We explore the effect of each component through a series of
experiments with the following modifications: (1) Using the initial masks
without mask updating; (2) Eliminating the cross-attention suppression; (3)
Eliminating the self-attention enhancement; (4) Omitting the background token
regularization; (5) Omitting the SSIM regularization; (6) Substituting Delta
masking with Otsu thresholding. We present the results in Fig. 5(c). Our
complete design achieves the highest image-alignment score and comparable
text-alignment. Utilizing the initial masks throughout training slightly
degrades the model’s performance, as these masks may not encompass the entire
target range, leading to missing details. Eliminating the suppression also
induces a slight degradation in image-alignment but a relatively substantial
decrease in text-alignment. Additionally, removing the two regularization
techniques, the background token and the SSIM, results in a significant
decrease in image-alignment. Substituting Delta masking with Otsu thresholding
leads to degraded image-alignment and text-alignment, as Otsu thresholding may
fail to create accurate masks for each concept, thereby misguiding the cross-
attention activation to the wrong region. Finally, removing the enhancement
causes the most significant decrease in image-alignment. We provide
qualitative results of the ablation in A.4 for further insight.

Figure 7: Failure case. The model only identifies vase in the input when given
text prompt "vase" but omits the plants.

###  4.5 More input concepts

We assess the disentangling capability of our proposed method on images
containing three concepts. While solely adding an additional identifier token
for the third concept, we maintain all other settings consistent, and apply
Uniform sampling. Figure 6 presents the images generated by our method. The
outcomes demonstrate that our method can disentangle multiple concepts in a
more complex setting and generate images of each single concept as well as the
concept group.

##  5 Conclusions and Limitations

In this study, we introduce AttenCraft for disentangling multiple concepts
from a single image in T2I customization, which autonomously creates precise
masks for each target concept within a single initialization step, leveraging
attention maps. These masks guide cross-attention activation during training,
facilitating concept disentanglement. Additionally, we propose Uniform
sampling and Reweighted sampling schemes to enhance learning synchronicity of
features from different concepts, thereby improving generation quality.
AttenCraft achieves state-of-the-art image-alignment scores, while its text-
alignment score is comparable to concurrent methods, striking the optimal
balance between reconstruction and editability among all studied methods.
Furthermore, AttenCraft extends its applicability to more complex inputs which
contain three concepts.

#### Limitations

AttenCraft utilizes attention maps to create the masks, which are influenced
by the input data and the semantic understanding capabilities of the pre-
trained SD model, which may occasionally lead to feature omission. As
illustrated by a failure case in Fig. 7, the identifier token [$\rm V_{2}$]
for the whole vase with plants is initialized by "vase", causing the model to
create a mask only for the vase but omitting the plants, thereby limiting
cross-attention to the vase region and resulting in different plants being
depicted in the generated image under text prompt "[$\rm V_{2}$] in the
bedroom".

## References

  * [1] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [2] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [3] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [4] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [5] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [7] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696–10706, 2022.
  * [8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [9] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [10] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023.
  * [11] Dongxu Li, Junnan Li, and Steven CH Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  arXiv preprint arXiv:2305.14720, 2023.
  * [12] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  arXiv preprint arXiv:2304.02642, 2023.
  * [13] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  ACM Transactions on Graphics (TOG), 42(4):1–13, 2023.
  * [14] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H Bermano.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models.  arXiv preprint arXiv:2307.06925, 2023.
  * [15] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  arXiv preprint arXiv:2307.06949, 2023.
  * [16] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu.  Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning.  arXiv preprint arXiv:2307.11410, 2023.
  * [17] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  In SIGGRAPH Asia 2023 Conference Papers, pages 1–12, 2023.
  * [18] Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, and Philip Teare.  An image is worth multiple words: Learning object level concepts using multi-concept prompt learning.  arXiv preprint arXiv:2310.12274, 2023.
  * [19] Tanzila Rahman, Shweta Mahajan, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, and Leonid Sigal.  Visual concept-driven image generation with text-to-image diffusion model.  arXiv preprint arXiv:2402.11487, 2024.
  * [20] Mehdi Safaee, Aryan Mikaeili, Or Patashnik, Daniel Cohen-Or, and Ali Mahdavi-Amiri.  Clic: Concept learning in context.  arXiv preprint arXiv:2311.17083, 2023.
  * [21] Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang.  Attention calibration for disentangled text-to-image personalization.  arXiv preprint arXiv:2403.18551, 2024.
  * [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [23] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [25] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [26] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [28] Pu Cao, Feng Zhou, Qing Song, and Lu Yang.  Controllable generation with text-to-image diffusion models: A survey.  arXiv preprint arXiv:2403.04279, 2024.
  * [29] Yuxuan Zhang, Jiaming Liu, Yiren Song, Rui Wang, Hao Tang, Jinpeng Yu, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al.  Ssr-encoder: Encoding selective subject representation for subject-driven generation.  arXiv preprint arXiv:2312.16272, 2023.
  * [30] Yufei Cai, Yuxiang Wei, Zhilong Ji, Jinfeng Bai, Hu Han, and Wangmeng Zuo.  Decoupled textual embeddings for customized image generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 909–917, 2024.
  * [31] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [32] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al.  Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [33] Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation.  In The Twelfth International Conference on Learning Representations, 2023.
  * [34] Yang Yang, Wen Wang, Liang Peng, Chaotian Song, Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, et al.  Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models.  arXiv preprint arXiv:2403.11627, 2024.
  * [35] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yee K Wong.  Vico: Detail-preserving visual condition for personalized text-to-image generation.  arXiv preprint arXiv:2306.00971, 2023.
  * [36] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones 2: Customizable image synthesis with multiple subjects.  arXiv preprint arXiv:2305.19327, 2023.
  * [37] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  arXiv preprint arXiv:2304.03411, 2023.
  * [38] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943–15953, 2023.
  * [39] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [40] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  arXiv preprint arXiv:2212.05032, 2022.
  * [41] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  ACM Transactions on Graphics (TOG), 42(4):1–10, 2023.
  * [42] Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin.  Compositional text-to-image synthesis with attention map control of diffusion models.  arXiv preprint arXiv:2305.13921, 2023.
  * [43] Quynh Phung, Songwei Ge, and Jia-Bin Huang.  Grounded text-to-image synthesis with attention refocusing.  arXiv preprint arXiv:2306.05427, 2023.
  * [44] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [45] Trong-Tung Nguyen, Duc-Anh Nguyen, Anh Tran, and Cuong Pham.  Flexedit: Flexible and controllable diffusion-based object-centric image editing.  arXiv preprint arXiv:2403.18605, 2024.
  * [46] Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung.  Directed diffusion: Direct control of object placement through attention guidance.  arXiv preprint arXiv:2302.13153, 2023.
  * [47] Yutong He, Ruslan Salakhutdinov, and J Zico Kolter.  Localized text-to-image generation for free via cross attention control.  arXiv preprint arXiv:2306.14636, 2023.
  * [48] Minghao Chen, Iro Laina, and Andrea Vedaldi.  Training-free layout control with cross-attention guidance.  arXiv preprint arXiv:2304.03373, 2023.
  * [49] Junjie Shentu, Matthew Watson, and Noura Al Moubayed.  Textual localization: Decomposing multi-concept images for subject-driven text-to-image generation.  arXiv preprint arXiv:2402.09966, 2024.
  * [50] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon.  Training-free consistent text-to-image generation.  arXiv preprint arXiv:2402.03286, 2024.
  * [51] Jaeseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, and Youngjung Uh.  Visual style prompting with swapping self-attention.  arXiv preprint arXiv:2402.12974, 2024.
  * [52] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco.  Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion.  arXiv preprint arXiv:2308.12469, 2023.
  * [53] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen.  Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1206–1217, 2023.
  * [54] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen.  Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [55] Pablo Marcos-Manchón, Roberto Alcover-Couso, Juan C SanMiguel, and Jose M Martínez.  Open-vocabulary attention maps with token optimization for semantic segmentation in diffusion models.  arXiv preprint arXiv:2403.14291, 2024.
  * [56] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  * [57] Philipp Krähenbühl and Vladlen Koltun.  Efficient inference in fully connected crfs with gaussian edge potentials.  Advances in neural information processing systems, 24, 2011.
  * [58] Nobuyuki Otsu et al.  A threshold selection method from gray-level histograms.  Automatica, 11(285-296):23–27, 1975.
  * [59] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [60] Bent Fuglede and Flemming Topsoe.  Jensen-shannon divergence and hilbert space embedding.  In International symposium onInformation theory, 2004. ISIT 2004. Proceedings., page 31. IEEE, 2004.
  * [61] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  The annals of mathematical statistics, 22(1):79–86, 1951.
  * [62] Chun-Hsiao Yeh, Ta-Ying Cheng, He-Yen Hsieh, Chuan-En Lin, Yi Ma, Andrew Markham, Niki Trigoni, HT Kung, and Yubei Chen.  Gen4gen: Generative data pipeline for generative multi-concept composition.  arXiv preprint arXiv:2402.15504, 2024.
  * [63] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [64] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * [65] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.

##  Appendix A Supplemental Material

###  A.1 Mask creation under different denoising timesteps

Hertz et al. [44] elucidate that structural information is established in the
early steps, and the details are refined in the later steps of inference. The
DDPM noise scheduler [1] introduces Gaussian noise to the input during the
diffusion process based on a Markov chain, with a variance schedule that
increases along with the diffusion step, which is also called the denoising
step in the reversed denoising process. Consequently, the noisy latent $z_{t}$
primarily contains coarse structural information when the denoising step $t$
is relatively high, whereas more detailed information can be discerned in
$z_{t}$ when $t$ is low.

Figure 8: Attention maps and resultant masks under different denoising
timestep $t$

Therefore, we recommend an initial denoising timestep $t_{0}$ ranging from 0
to 300 for the mask initialization step. We illustrate the cross-attention
maps $A_{C}$, the suppressed cross-attention maps $\hat{A}_{C}$, the enhanced
attention maps $A_{C}^{S}$, and the resulting masks $M$ after Delta masking
under different denoising timesteps $t$ in Fig. 8. Taking the cat in the "cat
& dog" dataset as an example, it can be observed that when $t$ is lower than
400, the outlines of the cat in the cross-attention maps $A_{C}$ are clear,
resulting in high-quality masks. However, when $t$ exceeds 600, the cross-
attention maps $A_{C}$ become blurry, leading to masks that fail to cover the
entire target region. Particularly, when $t$ is 999, $A_{C}$ only indicates
the position of the cat, with most details lost, resulting in a meaningless
mask.

Furthermore, when $t$ is set to 0, the mask $M_{0}$ exhibits lower quality
compared to $M_{200}$ from $t=200$, despite the clear and informative
attention maps. This discrepancy is caused by the Delta masking technique, as
the attention activation displayed on $A_{C}^{S}$ of other identifier tokens
(e.g., dog) is also intense in the cat region, leading to an incomplete cat
mask. Therefore, we recommend sampling a random initial timestep $t_{0}$ for
mask initialization instead of using a fixed step.

###  A.2 Datasets

In this study, we curate fifteen datasets for experiment and evaluation. We
include ten datasets introduced by DisenDiff [21], which generally feature
simple backgrounds. Additionally, we utilized the Gen4Gen dataset creation
pipeline [62] to amalgamate personalized concepts into complex backgrounds
(e.g., fields, mountains, forests) sourced from copyright-free platforms,
resulting in five synthetic datasets. The personalized concepts used for
dataset synthesis were collected from the DreamBooth dataset [9] and
CustomConcept101 [10]. All datasets are presented in Fig. 9, and the class
names for each dataset are: (1) cat & dog; (2) baby & toy; (3) chair & lamp;
(4) chair & vase; (5) dog & pig; (6) horse & dog; (7) woman & dog; (8) cow &
bird; (9) man & woman; (10) woman & boy; (11) boot & backpack; (12) cat &
penguin; (13) dog & bear; (14) backpack & toy; (15) toy & vase.

Figure 9: Illustration of datasets

###  A.3 Implementation details

#### Custom Diffusion. [10]

We utilize the official implementation of Custom Diffusion from the
HuggingFace platform [64] with 200 training steps, a batch size of 1, and a
learning rate of $5\times 10^{-5}$. An AdamW optimizer with $\beta_{1}=0.9$
and $\beta_{2}=0.999$ is applied. During training, the text prompt is "$[\rm
V_{1}]$ $[\rm Class_{1}]$ and $[\rm V_{2}]$ $[\rm Class_{2}]$" to fit the
original design of Custom Diffusion. The same prompt design is also employed
during inference. The identifier tokens are initialized by rare token
embeddings. PEFT is applied in Custom Diffusion so that only the $W_{k}$ and
$W_{v}$ matrices in cross-attention layers of UNet are optimized.

#### DisenDiff. [21]

We implement DisenDiff based on the official implementation with 250 training
steps, a batch size of 1, and a learning rate of $5\times 10^{-5}$. The
optimizer is the AdamW optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ is
applied.with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ is applied. Similar to
Custom Diffusion, the design of text prompt "$[\rm V_{1}]$ $[\rm Class_{1}]$
and $[\rm V_{2}]$ $[\rm Class_{2}]$" is applied in DisenDiff and the
identifier tokens are initialized by rare token emnbeddings. Also, DisenDiff
follows the selection of trainable parameters of Custom Diffusion.

#### Break-a-scene. [17]

We combine the official implementation of Break-a-scene with the
implementation presented in Texutual Localization [49]. Since the original
implementation of Break-a-scene optimizes the whole UNet following DreamBooth
[9], while Textual Localization presents a similar method with PEFT by only
optimizing the $W_{k}$ and $W_{v}$ matrices in cross-attention layers of UNet,
following Custom Diffusion. To ensure a fair comparison, we thereby adapt the
implementation of Break-a-scene with PEFT. We optimize the text embeddings of
identifier tokens with a high learning rate of $5\times 10^{-4}$ for 400 steps
in the first training stage, and train the text encoders and $W_{k}$ and
$W_{v}$ matrices in cross-attention layers with a low learning rate of
$5\times 10^{-5}$ for 200 steps, with a batch size of 1 applied for both
stages. An AdamW optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ is
applied for both stages. The masks are created by jointly using the Grounding
DINO [65] and SAM [56]. Moreover, the design of the text prompt is "$[\rm
V_{1}]$ and $[\rm V_{2}]$" where the identifier tokens are initialized by
corresponding class name embeddings.

#### AttenCraft (Ours).

We detail the implementation of our method in Section 4.1. An AdamW optimizer
with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ is applied. While the
hyperparameter settings introduced are generally applicable to most datasets,
an exception arises with the "man & woman" dataset. As illustrated in Fig. 10,
setting the denoising timestep $t_{0}$ for mask initialization to 0, 200, or
400 results in intense attention activation in the woman’s region within the
cross-attention maps $A_{C}^{[1]}$ of the "man" token, leading to inaccurate
masks for the "woman" concept. This issue may stem from various factors,
including visual similarities between the man and woman in the dataset or
biases present in the pre-training dataset. We demonstrate that our model
relies on the semantic understanding capabilities of the pre-trained SD to
differentiate between different concepts (Section 5). To mitigate this issue,
we explore different initial denoising timesteps $t_{0}$ for mask
initialization and identify an optimal timestep of $t_{0}=600$, under which
the model creates the masks that cover the faces of the man and woman,
respectively.

Figure 10: Dataset requires adaptive hyperparameter setting. We initialize the
masks for the "man &woman" dataset by sampling the initial denoising step
$t_{0}=600$.

###  A.4 Additional experiment results

#### Mask initialization.

In our method, we use a single step to initialize the masks for each concept
in the dataset. To present the effectiveness of our method on mask
initialization, we present the initial masks for each dataset in Fig. 11.

Figure 11: Initial masks for each dataset created by AttenCraft after hte mask
initialization step

There are two exceptions being the "chair & vase" dataset and "man & woman"
dataset, which are discussed in detail in the previous sections. Moreover, our
method tends to present stronger attention on the humans themselves than other
parts of humans, such as the long hair and cloth, as illustrated by the
woman’s mask in the "woman & dog" dataset and the woman’s mask in the "woman &
boy" dataset. Aside of these, our method successfully creates accurate masks
for other datasets.

#### Comparisons between our method and baseline models.

We present additional comparative results between our method with Uniform
sampling and the baseline models. Table 1 summarizes the performance of all
methods across all datasets. Our method, AttenCraft with Uniform sampling,
surpasses the baseline models in image-alignment for both individual concept
generation and concept group generation. Specifically, our method exhibits an
increase of 1.23% and 1.90% over the second-place model in individual concept
generation and concept group generation, respectively. Although it does not
achieve the highest text-alignment, our method performs comparably with the
other two disentangling models, with a difference of less than 0.27% in
individual concept generation. Furthermore, in concept group generation, our
model achieves a higher text-alignment score compared to the other two
disentangling models, with a difference exceeding 0.34%. For detailed
performance on each dataset, refer to Table 2.

Additionally, we provide further qualitative comparisons in Fig. 12 and Fig.
13. As shown in Fig. 12, Custom Diffusion frequently struggles to correctly
match identifier tokens with corresponding visual representations in the
image, resulting in instances where a random concept shares the same class
with the target concept (e.g., the cat in the third row and the bear in the
fourth row). DisenDiff and Break-a-scene also exhibit inaccuracies in
identifying the target concept in some cases, such as the bear in the fourth
row and the cow in the fifth row. Moreover, when generating concept groups, as
depicted in Fig. 13, Custom Diffusion fails to adequately separate concepts
and control them individually, leading to the inability to generate images
according to the text prompt "$[\rm V_{1}]$ painting $[\rm V_{2}]$ on a
canvas" in the first row. It also exhibits feature blending, as both the cow
and bird appear white in the third row. Furthermore, DisenDiff and Break-a-
scene display deviations in presenting the woman & dog in the first row and
the chair & lamp in the fourth row. They also produce collapsed images of the
man & woman in the second row and the dog & bear in the sixth row. In
contrast, our method consistently demonstrates good editability and visual
identity for all concepts in both Fig. 12 and Fig. 13, underscoring the
effectiveness of our approach.

#### Comparisons between our method with different sampling schemes.

We also present additional results comparing Uniform sampling with Reweighted
sampling. By observing the results of Uniform sampling, we pick out the
datasets that present non-synchronicity during feature learning of different
concepts and apply an adaptive sampling ratio to each dataset, resulting in a
subset comprising 9 datasets. Table 3 summarizes the performance of our method
under different sampling schemes. With adaptive sampling ratios, Reweighted
sampling shows an overall improvement compared to Uniform sampling, with
image-alignment and text-alignment increasing by 2.13% and 5.40% in individual
concept generation, and 4.41% and 3.35% in concept group generation. Detailed
comparisons on each dataset can be referred to Table 4. Note that the
"Reweighted ratio" means the sampling ratio of Concept A, Concept B, and the
concept group in the dataset.

Additional qualitative comparisons in individual concept generation and
concept group generation are illustrated in Fig. 14. Our method, under both
sampling schemes, preserves the visual features of the target concepts and
reflects the editability of the generated images. Comparing Uniform sampling
with Reweighted sampling, the latter presents more details of the target
concepts, such as the collar on the dog in the first row and the gap between
the bear’s eyes in the second row.

Table 1: Summarized quantitative comparisons between our method (Uniform
sampling) and baselines

Method | Individual concept | Concept group
---|---|---
Image-alignment | Text-alignment | Image-alignment | Text-alignment
Custom Diffusion |  0.5331 $\pm$ 0.0744 | 0.1881 $\pm$ 0.0222 | 0.5791 $\pm$ 0.0947 | 0.2130 $\pm$ 0.0245
DisenDiff | 0.5511 $\pm$ 0.0784 | 0.1857 $\pm$ 0.0222 |  0.5521 $\pm$ 0.0976 |  0.2014 $\pm$ 0.0281
Break-a-scene |  0.5455 $\pm$ 0.0837 |  0.1850 $\pm$ 0.0197 |  0.5639 $\pm$ 0.0866 |  0.2017 $\pm$ 0.0227
AttenCraft (Ours) | 0.5579 $\pm$ 0.0764 |  0.1855 $\pm$ 0.0187 | 0.5901 $\pm$ 0.0924 | 0.2085 $\pm$ 0.0245

Table 2: Detailed quantitative comparisons between our method (Uniform
sampling) and baselines

Metric | Method | baby & toy | cat & dog | chair & lamp | chair & vase | cow & bird
---|---|---|---|---|---|---
Image-alignment (Individual concept) | Custom Diffusion | 0.5392 | 0.6320 | 0.5431 | 0.5012 | 0.5573
DisenDiff | 0.5256 | 0.6317 | 0.5556 | 0.5512 | 0.5932
Break-a-scene | 0.5326 | 0.6363 | 0.5476 | 0.4863 | 0.5696
AttenCraft (Ours) | 0.5471 | 0.6484 | 0.5439 | 0.4849 | 0.6007
Text-alignment (Individual concept) | Custom Diffusion | 0.1863 | 0.1951 | 0.1887 | 0.1910 | 0.2167
DisenDiff | 0.1751 | 0.1889 | 0.1944 | 0.1907 | 0.2144
Break-a-scene | 0.1870 | 0.1924 | 0.1916 | 0.1859 | 0.2148
AttenCraft (Ours) | 0.1855 | 0.1891 | 0.1946 | 0.1868 | 0.2143
Image-alignment (Concept group) | Custom Diffusion | 0.6185 | 0.6714 | 0.7311 | 0.6341 | 0.6601
DisenDiff | 0.5985 | 0.6399 | 0.7197 | 0.6450 | 0.6286
Break-a-scene | 0.5869 | 0.6354 | 0.7042 | 0.5637 | 0.6741
AttenCraft (Ours) | 0.5964 | 0.6732 | 0.7108 | 0.5691 | 0.7208
Text-alignment (Concept group) | Custom Diffusion | 0.2088 | 0.2210 | 0.2193 | 0.2078 | 0.2605
DisenDiff | 0.2174 | 0.2016 | 0.2182 | 0.2124 | 0.2523
Break-a-scene | 0.2221 | 0.1990 | 0.2138 | 0.1915 | 0.2448
AttenCraft (Ours) | 0.2229 | 0.2116 | 0.2286 | 0.2097 | 0.2564

Metric | Method | dog & pig | horse & dog | man & woman | woman & boy | woman & dog
---|---|---|---|---|---|---
Image-alignment (Individual concept) | Custom Diffusion | 0.6500 | 0.5922 | 0.4626 | 0.4460 | 0.4711
DisenDiff | 0.6539 | 0.5827 | 0.4856 | 0.4998 | 0.5348
Break-a-scene | 0.6781 | 0.6004 | 0.4479 | 0.4600 | 0.4832
AttenCraft (Ours) | 0.6824 | 0.5777 | 0.4555 | 0.5348 | 0.5138
Text-alignment (Individual concept) | Custom Diffusion | 0.1963 | 0.2086 | 0.1500 | 0.1503 | 0.1720
DisenDiff | 0.2034 | 0.2067 | 0.1530 | 0.1631 | 0.1763
Break-a-scene | 0.2011 | 0.2062 | 0.1467 | 0.1672 | 0.1704
AttenCraft (Ours) | 0.1978 | 0.2018 | 0.1486 | 0.1600 | 0.1742
Image-alignment (Concept group) | Custom Diffusion | 0.5671 | 0.6914 | 0.6101 | 0.4833 | 0.5032
DisenDiff | 0.6044 | 0.5736 | 0.4872 | 0.5033 | 0.4841
Break-a-scene | 0.5529 | 0.6803 | 0.4904 | 0.4331 | 0.4768
AttenCraft (Ours) | 0.6045 | 0.7099 | 0.5490 | 0.5803 | 0.5937
Text-alignment (Concept group) | Custom Diffusion | 0.2108 | 0.2678 | 0.1893 | 0.1992 | 0.2015
DisenDiff | 0.2193 | 0.2431 | 0.1819 | 0.1895 | 0.1847
Break-a-scene | 0.1848 | 0.2304 | 0.1829 | 0.1911 | 0.1860
AttenCraft (Ours) | 0.2074 | 0.2475 | 0.1887 | 0.1857 | 0.2032

Metric | Method | boot & backpack | cat & penguin | dog & bear | toy & backpack | toy & vase
---|---|---|---|---|---|---
Image-alignment (Individual concept) | Custom Diffusion | 0.5369 | 0.5466 | 0.4643 | 0.5269 | 0.5270
DisenDiff | 0.5031 | 0.5889 | 0.5419 | 0.5587 | 0.4600
Break-a-scene | 0.5529 | 0.5866 | 0.5588 | 0.4926 | 0.5492
AttenCraft (Ours) | 0.5378 | 0.6151 | 0.5473 | 0.4990 | 0.5797
Text-alignment (Individual concept) | Custom Diffusion | 0.1877 | 0.2072 | 0.1930 | 0.1922 | 0.1872
DisenDiff | 0.1843 | 0.2019 | 0.1810 | 0.1895 | 0.1631
Break-a-scene | 0.1743 | 0.1947 | 0.1809 | 0.1791 | 0.1833
AttenCraft (Ours) | 0.1855 | 0.1960 | 0.1789 | 0.1891 | 0.1799
Image-alignment (Concept group) | Custom Diffusion | 0.4287 | 0.5384 | 0.4519 | 0.6253 | 0.4723
DisenDiff | 0.3795 | 0.5032 | 0.4025 | 0.6328 | 0.4793
Break-a-scene | 0.4392 | 0.5809 | 0.5033 | 0.6149 | 0.5224
AttenCraft (Ours) | 0.3725 | 0.5668 | 0.4892 | 0.6034 | 0.5124
Text-alignment (Concept group) | Custom Diffusion | 0.1809 | 0.2074 | 0.1911 | 0.2317 | 0.1982
DisenDiff | 0.1620 | 0.1883 | 0.1662 | 0.2218 | 0.1616
Break-a-scene | 0.1619 | 0.2184 | 0.1918 | 0.2243 | 0.1828
AttenCraft (Ours) | 0.1660 | 0.2162 | 0.1956 | 0.2113 | 0.1771

Figure 12: Additional qualitative comparisons between our method with Uniform
sampling and the baseline models in individual concept generation Figure 13:
Additional qualitative comparisons between our method with Uniform sampling
and the baseline models in concept group generation Table 3: Summarized
quantitative comparisons between our method with Uniform sampling) and
Reweighted sampling

Sampling scheme | Individual concept | Concept group
---|---|---
Image-alignment | Text-alignment | Image-alignment | Text-alignment
Uniform |  0.5727 $\pm$ 0.0439 |  0.1853 $\pm$ 0.0082 |  0.5713 $\pm$ 0.0755 |  0.2028 $\pm$ 0.0193
Reweighted | 0.5849 $\pm$ 0.0449 | 0.1863 $\pm$ 0.0078 | 0.5965 $\pm$ 0.0773 | 0.2096 $\pm$ 0.0177

Table 4: Detailed quantitative comparisons between our method with Uniform
sampling) and Reweighted sampling

Metric | Sampling scheme | baby & toy | dog & pig | horse & dog | woman & boy | dog & bear
---|---|---|---|---|---|---
Image-alignment (Individual concept) | Uniform | 0.5471 | 0.6824 | 0.5777 | 0.5348 | 0.5473
Reweighted | 0.5711 | 0.6805 | 0.5931 | 0.5361 | 0.5529
Text-alignment (Individual concept) | Uniform | 0.1855 | 0.1978 | 0.2018 | 0.1600 | 0.1789
Reweighted | 0.1855 | 0.1965 | 0.2024 | 0.1605 | 0.1828
Image-alignment (Concept group) | Uniform | 0.5964 | 0.6045 | 0.7099 | 0.5803 | 0.4892
Reweighted | 0.6159 | 0.6256 | 0.7431 | 0.6191 | 0.5096
Text-alignment (Concept group) | Uniform | 0.2229 | 0.2074 | 0.2475 | 0.1857 | 0.1956
Reweighted | 0.2269 | 0.2096 | 0.2528 | 0.1894 | 0.1987
| Reweighted ratio | 0.3,0.5,0.2 | 0.4,0.4,0.2 | 0.3,0.5,0.2 | 0.3,0.4,0.3 | 0.3,0.4,0.3

Metric | Sampling scheme | toy & backpack | toy & vase | cat & dog | boot & backpack
---|---|---|---|---|---
Image-alignment (Individual concept) | Uniform | 0.4990 | 0.5797 | 0.6484 | 0.5378
Reweighted | 0.5169 | 0.5963 | 0.6719 | 0.5456
Text-alignment (Individual concept) | Uniform | 0.1890 | 0.1799 | 0.1891 | 0.1855
Reweighted | 0.1888 | 0.1883 | 0.1904 | 0.1815
Image-alignment (Concept group) | Uniform | 0.6034 | 0.5124 | 0.6732 | 0.3725
Reweighted | 0.6435 | 0.5461 | 0.6800 | 0.3859
Text-alignment (Concept group) | Uniform | 0.2113 | 0.1771 | 0.2116 | 0.1660
Reweighted | 0.2267 | 0.2034 | 0.2117 | 0.1677
| Reweighted ratio | 0.4,0.2,0.4 | 0.2,0.4,0.4 | 0.4,0.4,0.2 | 0.2,0.4,0.4

Figure 14: Additional qualitative comparisons between our method with
different sampling schemes in individual concept generation (left) and concept
group generation

#### Quantitative results of ablation study.

We discussed the effects of various components during attention-guided mask
creation on the model performance in terms of image-alignment and text-
alignment in Section 4.4. In this section, qualitative comparisons of the
created masks and cross-attention maps from various variants of our model are
illustrated in Fig. 15 to provide a visual interpretation of the ablation
study.

As shown in Fig. 15, eliminating Cross-attention suppression could allow weak
attention activation to happen elsewhere, leading to separate activations on
the mask, which guide the attention activation outside of the target region in
reverse. Eliminating Self-attention enhancement would not render the attention
activation to be evenly and smoothly distributed within the target region,
thereby causing low-quality masks and cross-attention maps. Moreover, omitting
the background token regularization would increase the risk of "attention
leakage", so the mask and cross-attention map of an identifier token could
cover the background region, while omitting SSIM regularization would possibly
affect the local mask coverage. For instance, the mask and cross-attention map
of the lamp in the fourth row loses coverage on the lamp’s rod if the SSIM
regularization is omitted. Furthermore, utilizing the initial masks throughout
the training process is dependent on the quality of the initial masks. As can
be seen in the fifth row, the initial mask of the dog loses a small part of
the head, causing the cross-attention map to present comparably weak
activation in the head region. On the other hand, updating the masks during
training can improve the masks, making attention activation more robust in the
head region. Finally, substituting Delta masking with Otsu thresholding may
lead to the failure of separating the masks, therefore a correct connection
between identifier tokens and visual representations cannot be built.

Figure 15: Qualitative illustration of the ablation study

###  A.5 Societal impact

Our method may present a positive impact on content creation, making it easier
for people to customize their personal concepts with new contexts. Therefore,
more inspiration can be invoked. On the other hand, it could cause copyright
issues if users apply our method to copyrighted content. Moreover, there is a
risk of our method being used to create authentic but fake images, thereby
causing public damage.