# Layout-and-Retouch: A Dual-stage Framework for Improving Diversity in
Personalized Image Generation

Kangyeol Kim1,3*, Wooseok Seo2*, Sehyun Nam2, Bodam Kim2,
Suhyeon Jeong2, Wonwoo Cho1,3, Jaegul Choo1,3†, Youngjae Yu2†
1 KAIST AI 2 Yonsei University 3 Letsur Inc.
{kangyeolk, wcho<EMAIL_ADDRESS>
{justin_seo, daniel5253, qhdamm23,
pikachuisabird<EMAIL_ADDRESS>

###### Abstract

Personalized text-to-image (P-T2I) generation aims to create new, text-guided
images featuring the personalized subject with a few reference images.
However, balancing the trade-off relationship between prompt fidelity and
identity preservation remains a crtical challenge. To address the issue, we
propose a novel P-T2I method called Layout-and-Retouch, consisting of two
stages: _1) layout generation_ and _2) retouch_. In the first stage, our step-
blended inference utilizes the inherent sample diversity of vanilla T2I models
to produce diversified layout images, while also enhancing prompt fidelity. In
the second stage, multi-source attention swapping integrates the context image
from the first stage with the reference image, leveraging the structure from
the context image and extracting visual features from the reference image.
This achieves high prompt fidelity while preserving identity characteristics.
Through our extensive experiments, we demonstrate that our method generates a
wide variety of images with diverse layouts while maintaining the unique
identity features of the personalized objects, even with challenging text
prompts. This versatility highlights the potential of our framework to handle
complex conditions, significantly enhancing the diversity and applicability of
personalized image synthesis.

††footnotetext: * These authors contributed equally to this
work.††footnotetext: † Co-corresponding authors

##  1 Introduction

Following the notable success of text-to-image (T2I) generation models,
_e.g.,_ Stable Diffusion [40], which are trained on large-scale datasets of
text-image pairs, there has been increasing interest in personalized text-to-
image (P-T2I) generation problems [11]. Given a few reference images
containing a specific subject, P-T2I models aim to create new, prompt-guided
images that include the personalized subject. To effectively achieve this
goal, previous studies [11, 42, 30] proposed to learn new personalized
concepts by adjusting pre-trained T2I generation models [40, 36]. These
methods have demonstrated promising and visually satisfactory results, leading
to the development of versatile applications with practical potential in real-
world situations.

When evaluating P-T2I models, there exist two primary criteria. 1) _Prompt
fidelity_ examine the extent to which the generated image aligns with the
textual description. 2) On the other hand, _identity preservation_ assesses
whether the appearance of a subject within the image faithfully maintains the
characteristics of the personalized subject. In essence, a trade-off
relationship may exist between prompt fidelity and identity preservation [27],
thereby P-T2I models often miss to illustrate characteristics of personalized
concept when it comes to strictly following prompt guidance as well as
retaining the details of visual attributes of the concept [34, 16].

Figure 1:  Visualizations of center-point distributions of subjects. Using the
ViCo [16] evaluation prompt, we generate 10 images per prompt with both Stable
Diffusion (SD) [40] and Dreambooth [42] trained with (a) the subject of
reference images. For each subject, we locate the bounding box of an object.
We then compute the 2D center-point distribution by fitting Gaussian
distributions to each center point, with the mean being the center point
itself and the variance fixed, and then normalize all distributions. (b)
Vanilla SD places objects across a wider range compared to (c) fine-tuned SD,
indicating that fine-tuned SD has a weaker ability to generate a diverse range
of image layouts. $\sigma^{2}_{avg}$ denotes the averaged variances of 2D
center points, meaning that the center point set is more widely dispersed with
vanilla SD.

Since preserving the identity of each subject is crucial in P-T2I problems, a
line of research [22, 45, 47, 5, 34, 53] has focused on maintaining visual
appearance during the inference phase by explicitly using reference images as
an additional condition. Although these approaches enhance the identity
preservation capabilities of P-T2I models, they do not improve prompt
fidelity. This is because the ability to generate contextually appropriate
images is inherently limited by the intrinsic capacities of the pre-trained
P-T2I model. To investigate the inherent limitations of pre-trained P-T2I
generation models in comparison with vanilla T2I models such as Stable
Diffusion [40], we conduct exploratory experiments, the results of which are
shown in Fig. 1. In fact, requiring the P-T2I model to depict the appearance
of personalized concepts and synthesize non-personalized contexts is highly
demanding.

To overcome the limitations of the previous works and generate personalized
images from diverse and challenging prompts while retaining identity
characteristics, we propose a novel off-the-shelf P-T2I generation strategy
called Layout-and-Retouch. Our core idea involves separating the generation of
non-personalized parts (e.g., background and wearables) from the identity of
target subjects, thus structuring our generation process into two stages: _1)
layout image generation_ and _2) retouch_.

In the layout image generation stage, we primarily focus on improving prompt
fidelity, irrespective of identity preservation. Specifically, our first aim
is to create a layout image that provides structural guidance for the next
step. For this purpose, we carefully design a step-blended denoising approach
to improve the prompt fidelity of pre-trained P-T2I models by leveraging the
diverse expressive capabilities of vanilla T2I generation models. As a result,
we obtain layout image features that depict an object visually resembling the
target subject, occupying a specific position within the image. The remaining
areas are arranged to align with the provided textual prompt.

In the retouch stage, the remaining goal is to precisely calibrate the target
subject while preserving the background context of the layout image obtained
in the previous stage. To achieve this, we propose multi-source attention
swap, a straightforward yet effective method that integrates context from the
layout image and captures detailed visual appearance from a reference image
simultaneously. The principle of the attention swap technique is to transmit
information, such as object appearance and style, from reference visual
characteristics by replacing the queries, keys, and values in the target
denoising steps with those derived from a reference image [5, 12, 13, 9].

Our two-stage method enables the creation of personalized images with
diversified layouts, surpassing existing methods in variety. Extensive
experiments demonstrate its capability to produce diverse images and
effectively manage challenging prompts.

##  2 Related Work

###  2.1 Text-Guided Image Generation and Editing

##### T2I generation models.

T2I generation models [51, 35, 38, 40, 43, 36] have been extensively studied
for their ability to create realistic images from textual descriptions.
Recently, diffusion-based models [38, 40, 43] have demonstrated significant
success in producing photo-realistic images from user-provided text, offering
exceptional controllability. However, models such as Imagen [43], Stable
Diffusion [40], and DALL-E series [39, 38, 3], still face challenges in
generating personalized images. Specifically, these models struggle when it
comes to creating images based on specific or user-defined concepts, where the
identities are difficult to accurately convey through text descriptions.

##### Text-guided image editing.

Building on the advancements in T2I generation models, there have been active
studies in editing specific images based on text inputs [28, 32, 18, 2, 4, 24,
23, 1]. However, editing images while retaining most of the original content
is challenging, as even minor modifications to the text guidance can lead to
significant changes. Therefore, early works such as SDEdit [32] and Blended-
Diffusion [1] have been developed with limited image editing capabilities,
_e.g._ , requiring users to provide a spatial mask to specify the area for
editing. To address these limitations, Imagic [23] and InstructPix2Pix [4]
have been designed using multi-stage training processes and multiple distinct
models, respectively. Despite their effectiveness, these editing models may
not guarantee identity preservation when performing complex text-driven image
transformations.

###  2.2 Text-to-Image Personalization

Addressing the limitations of T2I generation and editing models,
personalization models aim to create new images using a few images of an
object, while preserving the object’s identity consistently.

##### Optimization-based methods.

Previous studies [11, 42, 26, 15, 49, 16, 8, 48, 14] have explored generating
consistent image variations of a specified concept by embedding the concept
within the textual domain of diffusion-based models, often represented by a
particular token. This approach allows for the controlled generation of images
that align closely with a target prompt. Textual Inversion [11] and DreamBooth
[42] are advanced methods for creating personalized images through diffusion-
based models. Textual Inversion optimizes a textual embedding to integrate a
specialized token with the target prompt, while DreamBooth extends this by
adjusting all parameters of the denoising U-Net, targeting a specific token
and the subject’s class category. These methods enhance the precision and
contextual relevance of image generation. Research efforts have continually
advanced by focusing on tuning key components, such as the cross-attention
layer [26, 15, 48], or by incorporating additional adapters [16] to enhance
training efficiency and conditioning performance. While these studies have
shown promising results, they have faced limitations in preserving the
appearance of subjects.

##### Off-the-shelf methods.

To eliminate the necessity of additional fine-tuning steps, researchers have
explored plug-in T2I personalization techniques [5, 33, 10, 29, 53, 46, 19].
These approaches not only enhance computational efficiency but also improve
outcomes by explicitly utilizing reference images as an additional condition
to capture visual appearance during the inference phase. Technically, these
methods manipulate the keys and values of the self-attention module during the
denoising process of the U-Net, effectively altering structures and textures
[5, 34]. MasaCtrl [5], for instance, employs a dual-path pipeline that
synthesizes both a reference and a target image concurrently, replacing the
target’s keys and values in the self-attention module with those from the
reference. More recently, DreamMatcher [34] introduced a technique that
adjusts these replaced target values using flowmap-based semantic matching to
ensure structural correspondences between the target and reference latent
features. However, the P-T2I model alone may present significant limitations
in generating diverse layout images, often making it difficult to handle
complex prompt conditions. To overcome this issue, we propose a two-stage
framework, Layout-and-Retouch, where the vanilla T2I model is responsible for
constructing an initial layout, leading to improvements in both prompt
fidelity and layout diversity.

##  3 Proposed Methods

###  3.1 Preliminaries

##### Latent diffusion model.

Recent advancements in text-to-image diffusion models, such as Stable
Diffusion [40], have achieved robust and efficient image generation by
performing the denoising process within the latent space using a pre-trained
autoencoder. Specifically, a pre-trained encoder compresses an image into a
latent representation $\mathbf{z}$, followed by diffusion and iterative
denoising steps using a conditional diffusion model $\epsilon_{\theta}$.
During denoising, a text condition $\mathbf{y}$ is incorporated through a
cross-attention module, guiding the latent representations to align with the
text condition. The training objective is formulated as

$\mathcal{L}=\mathbb{E}_{\mathbf{y},\mathbf{z},\epsilon,i}\left[\|\epsilon-\epsilon_{\theta}(\mathbf{z}^{i},i,\texttt{CLIP}(\mathbf{y})\|\right],$
(1)

where $\epsilon\sim\mathcal{N}(0,1)$, CLIP denotes the text encoder of CLIP
[37], and $\mathbf{z}^{i}$ denotes a noisy version of the latent
representations $\mathbf{z}$. Also, discrete time steps $i$ are sampled from
the set $\\{1,...,T\\}$ uniformly at random. For effective and robust image
generation, we base our method on Stable Diffusion [40].

The latent diffusion model employs a time-conditional U-net backbone, which
features multiple self-attention and cross-attention layers. Formally, the
attention mechanism can be written as

$\texttt{Attention}(Q,K,V)=\texttt{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}V\right),$

where $Q$ represents the query encoded using latent feature maps, while $K$
and $V$ denote the key and value, respectively. These are encoded using either
the latent feature maps for self-attention layers or textual embeddings for
cross-attention layers.

##### Pre-training personalized model.

In general, training a personalized model requires a set of $m$ reference
images $\\{I^{m}_{r}\\}_{m=1}^{M}$ that describe a target concept [11, 42, 8].
Previous approaches fine-tune either a partial [11] or the entire network [42]
to encapsulate the target concept into the network. After training, special
text tokens such as "<*>" are used to represent the personalized information,
which can be flexibly combined with additional text conditions. Although pre-
trained networks with the special text tokens have shown a remarkable capacity
to convey the target object, the use of only a few training images makes them
highly susceptible to overfitting [52, 54, 7]. One of our observations is that
a pre-trained personalized model has weak capacities for generating images
with diverse configurations as shown in Fig. 1. Throughout this paper, we
denote the text condition incorporating the special text token as
$\mathbf{y_{p}}$, while the text condition with only the special text token
removed is denoted as $\mathbf{y^{-}_{p}}$. Additionally, we define a neutral
text condition, which includes only the concept without any specific text
guidance, as $\mathbf{y}_{r}$ (e.g., a photo of <*>).

Figure 2:  Overall pipeline of Layout-and-Retouch: (a) In the layout
generation step, we perform (I) step-blended denoising using vanilla and
personalized T2I models. Different subject-related words (e.g., red vase and
<*>) are fed to each model. (b) The retouch step focuses on calibrating the
target subject while maintaining the layout image structure. This is achieved
using (II) multi-source attention swap, where intermediate variables from the
attention layers of other denoising paths are used to create the target image,
and (III) adaptive mask blending, which combines $\mathbf{M^{\text{SAM}}}$ and
a cross-attention map to generate an accurate mask for blending feature maps
in the self-attention layer.

###  3.2 Layout-and-Retouch Framework

Given a reference image $I_{r}$ and the corresponding text condition
$\mathbf{y}_{p}$, our goal is to generate a target image $I_{t}$ that adheres
to the text condition while retaining the appearance of $I_{r}$. One solution
might be to leverage a pre-trained P-T2I model with $\mathbf{y_{p}}$; however,
we find that the pre-trained P-T2I model tends to synthesize images within a
restricted layout space (See Fig. 1 and Fig 4). We hypothesize that the model
pre-trained on repetitive layouts limits its capabilities to generate diverse
configurations. To address this issue, we propose the Layout-and-Retouch
framework, as depicted in Fig. 2.

####  3.2.1 Stage 1 - Layout Generation

##### Step-blended denoising.

Our core idea is to utilize the expressiveness of vanilla Stable Diffusion
(SD) [40] by having it create the layout in the initial steps. Delegating the
task of generating the initial layout to vanilla SD broadens the range and
expressiveness of the layouts. We use the text condition $\mathbf{y^{-}_{p}}$
since vanilla SD lacks prior knowledge of the concept. As shown in Fig. 1,
layouts generated by vanilla SD are more diverse than those from pre-trained
P-T2I models. By leveraging vanilla SD’s ability to generate diverse initial
layouts, we facilitate the creation of a broader spectrum of layout images.
The initial layout generation is achieved within $\lambda_{1}$ steps,
determined based on empirical studies. Detailed results and analysis of these
$\lambda_{1}$ iterations are in the Appendix.

Generating the initial layouts within $\lambda_{1}$ steps, the subsequent
steps adjust the initial object appearance to visually align with the target
subject while preserving the initial structure. We use $\mathbf{y_{p}}$ and
the personalized model for the remaining steps. Let $\epsilon_{\theta}$ and
$\epsilon^{*}_{\theta}$ represent the vanilla and personalized denoising
networks, respectively, and let $\mathbf{z}^{i}$ denote the latent
representations at each denoising step $i$. Formally, computing the latent
representations in step-blended denoising can be written as:

$\mathbf{z}_{i-1}=\begin{cases}\texttt{Sample}(\mathbf{z}^{i},\epsilon_{\theta}(\mathbf{z}^{i},\mathbf{y^{-}_{p}},i))&\text{if
}i<\lambda_{1},\\\
\texttt{Sample}(\mathbf{z}^{i},\epsilon^{*}_{\theta}(\mathbf{z}^{i},\mathbf{y_{p}},i))&\text{otherwise},\end{cases}\quad\text{for
}i=T,T-1,\ldots,1,$

where Sample operation determines the next latent representations using a
predicted noise. Following the denoising steps, a decoding process is
performed to generate a layout image $I_{o}$.

####  3.2.2 Stage 2 - Retouch

##### Multi-source attention swap.

Although a layout image $I_{o}$ captures some visual characteristics of the
target subject, it loses finer details. To enhance the details, we use
attention swapping techniques [5, 12] with multiple source images: $I_{o}$ and
$I_{r}$. Specifically, intermediate variables from cross and self-attention
modules derived from $I_{o}$ and $I_{r}$ are passed to the denoising steps of
$I_{t}$. We denote the queries, keys, and values of the cross-attention module
as $Q^{c}_{t},K^{c}_{t},V^{c}_{t}$ and those of the self-attention module as
$Q^{s}_{t},K^{s}_{t},V^{s}_{t}$ in the target denoising process. Additionally,
we use $\mathbf{y}_{r}$ as a text condition to focus on modifying the visual
features of the target object throughout the denoising process.

Algorithm 1 describes an overall process of the Retouch stage. During the
target path denoising, attention swapping replaces
$Q^{c}_{t},K^{c}_{t},V^{c}_{t}$ or $Q^{s}_{t},K^{s}_{t},V^{s}_{t}$ with
variables from the layout and context path. In the early steps, we use
$Q^{c}_{o},K^{c}_{o},V^{c}_{o}$ and $Q^{s}_{o},K^{s}_{o},V^{s}_{o}$ from the
layout path, integrating them into the target path to construct the overall
structure of the noisy image. This sharing helps $I_{t}$ loosely adhere to
$I_{o}$’s structure, making replication easier. In later denoising steps,
$K^{s}_{r}$ and $V^{s}_{r}$ from the reference path replace those in the
target path to infuse detailed features from the reference object.
Additionally, a composite foreground mask $\mathbf{M}$ combines latent
representations of the target and layout path within a self-attention layer,
where the details on $\mathbf{M}$ are provided below.

Algorithm 1 Retouch stage with multi-source attention swapping algorithm

1:Inputs: layout image $I_{o}$, reference image $I_{r}$, neutral text
condition $\mathbf{y}_{r}$, target text condition $\mathbf{y}_{p}$ SAM mask
$\mathbf{M}^{\text{SAM}}$, pre-trained personalized diffusion model
$\epsilon^{*}_{\theta}$

2:$\mathbf{z}^{T}_{r}\leftarrow\texttt{DDIMInversion}(\texttt{Encoder}(I_{r}),``\,")$
$\triangleright$ DDIM inversion for reference image

3:$\mathbf{z}^{T}_{o}\leftarrow\texttt{DDIMInversion}(\texttt{Encoder}(I_{o}),``\,")$
$\triangleright$ DDIM inversion for layout image

4:$\mathbf{z}^{T}_{t}\leftarrow\mathbf{z}^{T}_{o}$ $\triangleright$ Initialize
target latent with layout latent

5:for $i=T,T-1,\ldots,1$ do

6:
$\\{{\color[rgb]{0,0,1}Q^{s}_{r},K^{s}_{r},V^{s}_{r}}\\},\epsilon_{r}\leftarrow\epsilon^{*}_{\theta}(\mathbf{z}^{i}_{r},\mathbf{y}_{r},i)$
$\triangleright$ Reference path

7:
$\\{{\color[rgb]{1,0,0}Q^{c}_{o},K^{c}_{o},V^{c}_{o}}\\},\\{{\color[rgb]{1,.5,0}Q^{s}_{o},K^{s}_{o},V^{s}_{o}}\\},\phi_{o}\leftarrow\epsilon^{*}_{\theta}(\mathbf{z}^{i}_{o},\mathbf{y}_{p},i)$
$\triangleright$ Layout path

8:
$\\{{\color[rgb]{0,.5,.5}Q^{c}_{t},K^{c}_{t},V^{c}_{t}}\\},\\{{\color[rgb]{1,0,1}Q^{s}_{t},K^{s}_{t},V^{s}_{t}}\\}\leftarrow\epsilon^{*}_{\theta}(\mathbf{z}^{t}_{i},\mathbf{y}_{r},i)$
$\triangleright$ Target path

9: if $i>\lambda_{2}$ then

10:
$\epsilon^{*}\leftarrow\epsilon^{*}_{\theta}(\mathbf{z}^{t}_{i},\mathbf{y}_{r},i,\\{{\color[rgb]{1,0,0}Q^{c}_{o},K^{c}_{o},V^{c}_{o}}\\},\\{{\color[rgb]{1,.5,0}Q^{s}_{o},K^{s}_{o},V^{s}_{o}}\\})$
$\triangleright$ Denoise w/ variables

11: else

12:
$\epsilon^{*}\leftarrow\epsilon^{*}_{\theta}(\mathbf{z}^{t}_{i},\mathbf{y}_{r},i,\\{{\color[rgb]{0,.5,.5}Q^{c}_{t},K^{c}_{t},V^{c}_{t}}\\},\\{{\color[rgb]{1,0,1}Q^{s}_{t},}\,{\color[rgb]{0,0,1}K^{s}_{r},V^{s}_{r}}\\},\phi_{o},\mathbf{M}^{\text{OURS}})$
$\triangleright$ Denoise w/ variables and mask

13: end if

14:
$\mathbf{z}^{i-1}_{r}\leftarrow\texttt{Sample}(\mathbf{z}^{i}_{r},\epsilon_{r})$
$\triangleright$ Sample next latent for reference image

15:
$\mathbf{z}^{i-1}_{o}\leftarrow\texttt{Sample}(\mathbf{z}^{i}_{o},\epsilon_{o})$
$\triangleright$ Sample next latent for layout image

16:
$\mathbf{z}^{i-1}_{t}\leftarrow\texttt{Sample}(\mathbf{z}^{i}_{t},\epsilon^{*})$$\triangleright$
Sample next latent for target image

17:end for

18:$I_{t}=\text{Decoder}(\mathbf{z}^{0}_{t})$

19:return $I_{t}$

##### Adaptive mask blending.

To enhance the visual details of the target subject, we explicitly create a
foreground mask to directly blend latent representations of the layout image.
One can make foreground masks using cross-attention maps that correlate to the
object prompt tokens extracted from the decoder as in [5], where the maps are
averaged and then thresholded to produce a binary mask
$\mathbf{M}^{c}\in\mathbb{R}^{h\times w}$, where $h$ and $w$ represent the
spatial dimensions of the latent representations. However, we observe that
$\mathbf{M}^{c}$ often fail to cover the entire foreground and includes noisy
regions. To mitigate this issue, we additionally harness a binary foreground
mask $\mathbf{M}^{\text{SAM}}\in\mathbb{R}^{H\times W}$ where $H,W$ denote
height and width respectively computed by Segment-Anything [25] given a layout
image. We also notice that only using $\mathbf{M}^{\text{SAM}}$ results in
layout-target misalignment, since detailed appearance or locations may shift
during the target generation process (details are explained in the Appendix
C.2). Therefore, we propose an adaptive mask blending technique to mitigate
these issues.

Let $\mathbf{M}[x,y]$ be the value at position $(x,y)$ and
$\texttt{Resize}(\mathbf{M}^{c})\in\mathbb{R}^{H\times W}$ be the resized mask
from the averaged cross-attention map. We first discard noisy regions of
$\mathbf{M}^{c}$ based on $\mathbf{M}^{\text{SAM}}$, _i.e.,_

$\displaystyle\mathbf{M}^{k}[x,y]$
$\displaystyle=\texttt{OR}(\texttt{Resize}(\mathbf{M}^{c})[x,y],\mathbf{M}^{\text{SAM}}[x,y]),\;$

where OR represents pixel-wise operations between two binary values. Then, we
apply a distance transformation [41] to smooth the abrupt transitions between
regions with values of 1 and regions with values of 0. Lastly, we add the area
with 1s of $\mathbf{M}^{c}$ to explicitly insert confident regions based on
the activated values of $\mathbf{M}^{c}$. Formally, this can be formulated as
follows:

$\displaystyle\mathbf{M}[x,y]=\texttt{Normalize}(\mathcal{D}_{l2}(\mathbf{M}^{k}[x,y]))+\mathcal{C}(\mathbf{M}^{c}[x,y]),$

where $\mathcal{D}_{l2}$ denotes the distance transformation using
$l2$-distance and Normalize is a scaling operation to map the values to the
range [0.5, 1.0]. Also a small connected set removal operation $\mathcal{C}$
performs to discard noisy pixel mask areas by removing connected sets below a
specified volume threshold. This allows us to reduce the potential negative
effects caused by noisy masks.

During denoising steps, $\mathbf{M}\in\mathbb{R}^{H\times W}$ is resized to
match the spatial dimension of the latent representations. To blend the latent
representations between the target and layout path, weighted summation is
performed in the self-attention layer:

$\displaystyle\phi^{*}=\mathbf{M}\odot\phi_{t}+(1-\mathbf{M})\odot\phi_{o},$

where $\odot$ represents an element-wise multiplication, and $\phi_{t}$,
$\phi_{o}$ are the output latent representations of the self-attention layer,
for the target and layout path, respectively. The combined latent
representation $\phi^{*}$ is passed to the next layer after the self-attention
layer.

##  4 Experiments

Figure 3:  Qualitative comparisons with challenging prompts. Our method
produces images with poses and scales significantly different from the
reference images compared to other methods. Additionally, it excels at
generating images that accurately follow challenging prompts. It is noteworthy
that our method does not produce images with identical structures, benefiting
from step-blended denoising.

In this section, we describe the experimental setup (Section 4.1), including
benchmark datasets, baselines, and evaluation metrics to evaluate Layout-and-
Retouch and baselines in three aspects:

  1. 1.

Layout Diversity: To assess the model’s ability to generate diverse layouts.

  2. 2.

Identity Preservation: To ensure that objects consistently maintain their
detailed characteristics across images.

  3. 3.

Prompt Fidelity: To verify that the outputs faithfully adhere to the text
guidance.

Comparisons with baselines include both quantitative and qualitative analyses
to investigate the add-on effects of Layout-and-Retouch and to compare it with
various baselines according to the criteria mentioned above (Section 4.2). The
qualitative analysis includes experimental results validating the superiority
of Layout-and-Retouch in generating diverse images, along with an ablation
study demonstrating the contribution of each component of the proposed method.
(Section 4.3)

###  4.1 Experimental Setup

##### Datasets.

We adopt the image-prompt dataset proposed in ViCo [16] for evaluation as a
normal dataset. This dataset includes 16 unique concepts from previous methods
[11, 42, 26] and contains 31 distinct prompts. The categories include 5 live
animals, 6 toys, 3 household goods, 1 accessory, and 1 building, with 4-7
images per object. The evaluation prompt set, with 31 prompts, is referenced
from Dreambooth [42] and covers both living and non-living entities, including
a few type-specific prompts. Additionally, we use the challenging prompt
dataset proposed in DreamMatcher [34], which includes 24 prompts requiring
large displacement or complex scene synthesis. For evaluation, we generate 8
images per object and prompt, following the evaluation protocols of previous
studies [16, 34].

##### Baselines.

Our proposed approach is compatible with any P-T2I model without requiring
additional training process. The primary baselines are the original P-T2I
models such as Textual inversion [11], Dreambooth [42], Custom diffusion [26]
to evaluate potential performance improvements. Furthermore, we compare our
method with other tuning-free plug-in baselines such as FreeU [46],
MagicFusion [53] and DreamMatcher [34]. As a backbone algorithm for evaluating
plug-in baselines, we use Textual Inversion [11] and DreamBooth [42] as a
personalized model for all baselines, including our method.

##### Evaluation metrics.

Following conventional work [11, 42, 26, 16], we focus on measuring two
primary aspects: (1) identity preservation and (2) prompt fidelity. To
evaluate identity preservation, we adopt $\mathbf{I_{CLIP}}$ and
$\mathbf{I_{DINO}}$, which respectively employ CLIP [37] and DINO [6] as
backbone networks to measure subject similarity between
$\\{I^{m}_{r}\\}_{m=1}^{M}$ and the generated images. For evaluating prompt
fidelity, we utilize the CLIP [37] image and text encoder to compute text-
image similarity score, denoted as $\mathbf{T_{CLIP}}$. Lastly, we utilize
Inception Score (IS) [44] to measure the diversity of the generated image set.

Methods | Normal Prompts | Challenging Prompts
---|---|---
$\mathbf{I_{DINO}}\uparrow$ | $\mathbf{I_{CLIP}}\uparrow$ | $\mathbf{T_{CLIP}}\uparrow$ | $\mathbf{I_{DINO}}\uparrow$ | $\mathbf{I_{CLIP}}\uparrow$ | $\mathbf{T_{CLIP}}\uparrow$
DreamMatcher [34] | $0.576$ | $0.781$ | $0.278$ | $0.650$ | $0.824$ | $0.310$
MagicFusion [53] | $0.521$ | $0.759$ | $0.278$ | $0.599$ | $0.801$ | $0.315$
FreeU [46] | $0.526$ | $0.770$ | $0.281$ | $0.587$ | $0.795$ | $0.312$
Ours | 0.600 | 0.789 | 0.297 | 0.665 | $0.824$ | $0.315$
Table 1: Quantitative comparison against various plug-in based models [46, 53,
34] on base prompts curated by ViCo [16]. We used Textual Inversion [11] for
normal prompts baseline, and we used Dreambooth [42] for challenging prompts
baseline.

###  4.2 Comparisons with Baselines

##### Comparisons with plug-in baselines

We compare our approach with tuning-free plug-in baselines: FreeU [46],
MagicFusion [53] and DreamMatcher [34] on both normal and challenging prompts.
Fig. 3 shows visual comparisons of Layout-and-Retouch and baseline outputs. As
seen, the proposed method successfully follows challenging prompt guidance
while dynamically adapting the personalized object.

As shown in Table 1 (Left), the proposed method shows superior and comparable
performances on both normal prompts and challenging prompts. For normal
prompts, our method presents consistent performance increases in
$\mathbf{I_{DINO}},\mathbf{I_{CLIP}}$ and $\mathbf{T_{CLIP}}$. These
improvements emphasize the effectiveness of our method in achieving both goals
of personalized image generation: identity preservation and prompt fidelity.
Evaluation on challenging prompts is shown in Table 1 (Right). Although,
baseline models show comparable results on $\mathbf{T_{CLIP}}$, the
$\mathbf{I_{DINO}}$ and $\mathbf{I_{CLIP}}$ scores indicate that the prompt
condition for generating personalized objects is not being successfully
fulfilled. Meanwhile, our approach effectively balances the dual objectives of
identity preservation and prompt fidelity in the generated images.

We conduct a user study to evaluate three aspects: (1) Identity preservation,
(2) Prompt fidelity, and (3) Diversity. As can be seen in the ranking score of
Table 3, our approach is largely preferred against the other plug-in
baselines. In the diversity aspect, we outperform DreamMatcher [34] by a large
margin, demonstrating strength in generating diverse images. The user study
setup is written in Appendix B.4.

Methods | $\mathbf{I_{DINO}\uparrow}$ | $\mathbf{I_{CLIP}\uparrow}$ | $\mathbf{T_{CLIP}\uparrow}$
---|---|---|---
Textual Inversion | 0.527 | 0.769 | 0.277
\+ Ours |  0.600 (+13.9%) |  0.789 (+2.6%) |  0.297 (+7.2%)
Dreambooth | 0.637 | 0.814 | 0.313
\+ Ours |  0.655 (+2.8%) |  0.815 (+0.1%) |  0.317 (+1.3%)
Custom Diffusion | 0.658 | 0.814 | 0.315
\+ Ours |  0.659 (+0.2%) |  0.814 (+0.0%) |  0.317 (+0.6%)
Table 2: Add-on effects of our method. Our method consistently improve
baselines, validating the effectiveness of the proposed approach. Performance
gains are marked in blue.

Table 3: User study results against plug-in baselines.

##### Improving P-T2I baselines.

We validate the effectiveness of our proposed framework by integrating it with
Textual inversion [11], Dreambooth [42], Custom diffusion [26]. Table 3
presents the performance evaluations for identity preservation and prompt
fidelity. As shown, our approach enhances prompt fidelity across all baselines
while maintaining comparable results in identity preservation. Notably, the
significant and consistent improvements in prompt fidelity demonstrate the
effectiveness of our method, highlighting the importance of initial layouts in
generating prompt-aligned images. We present more qualitative comparisons with
P-T2I baselines in the Appendix C.3.

###  4.3 Qualitative Analysis

##### Analysis on image diversity.

One of the strengths of the proposed method lies in synthesizing images with
diverse layouts. To demonstrate this, we measure the diversity of the
generated images with DreamMatcher [34] and our approach, employing the
diversity metric IS score [44]. Instead of generating a small set across all
prompts, we generate 400 images for 10 randomly sampled prompts out of the 31
available for each of the 8 objects with the aim of faithfully measuring the
score. As seen in Fig. 4 (a), our method surpasses DreamMatcher [34] in terms
of IS score across all objects, indicating the superiority of ours to create
images with diverse configurations. In addition, we evaluate the diversity of
the generated images by projecting three image groups of prompt using pre-
trained ResNet [17], then apply UMAP [31] to visualize the embedding space. As
shown in Fig. 4 (b) visualizes of clusters, our method obviously produces the
images in broader coverage compared to DreamMatcher [34]. This proves the
strength of Layout-and-Retouch in generating diverse and prompt-aligned
images. The sampled prompts list is available in the Appendix 6.

Figure 4:  Illustrations of diversity analysis. (a) We measure the IS score across individual objects, and verifying our method has a strong tendency to generate more diverse images. (b) Our methodology is distributed more broadly within the same prompt. This means that our approach is capable of generating images with diverse configurations.  | Configurations | $\mathbf{I_{DINO}}\uparrow$ | $\mathbf{I_{CLIP}}\uparrow$ | $\mathbf{T_{CLIP}}\uparrow$
---|---|---|---|---
(i@) | Dreambooth [42] | 0.637 | 0.814 | 0.313
(ii@) | (i@) + Step-blended denoising | 0.620 | 0.802 | 0.319
(iii@) | (ii@) + Reference attention swap | 0.707 | 0.812 | 0.280
(iv@) | (ii@) + Multi-source attention swap | 0.615 | 0.798 | 0.320
(v@) | (iv@) + Adaptive mask blending (Ours) | 0.655 | 0.815 | 0.317
Table 4:  Performances of various configurations on ViCo [16] dataset.

##### Ablation study.

We study the contribution of each component by sequentially adding step-
blended denoising, multi-source attention swap, and adaptive mask blending. An
ablation study is conducted based on Dreambooth baseline [42]. Table 4 shows
the performances of different configurations in aspects of identity
preservation ($\mathbf{I_{DINO}}$, $\mathbf{I_{CLIP}}$) and prompt fidelity
($\mathbf{T_{CLIP}}$). (ii@) Building upon Dreambooth [42], we introduce step-
blended denoising, and demonstrate a significant enhancement in prompt
fidelity. (iii@) A natural extension is to incorporate a reference image into
the denoising path by utilizing the corresponding $K$ and $V$ from the self-
attention layer derived from the reference image denoising process. However,
we observe a significant degradation in prompt fidelity, primarily due to the
increased complexity of generating a background while accurately reflecting
the visual characteristics of the reference image. (iv@) To address this, we
adopt a two-stage framework with multi-source attention swapping This method
achieves a significant performance boost in prompt fidelity compared to
(iii@), but it leads to a reduction in identity preservation. (v@) In
response, we employ adaptive mask blending, which enhances identity
preservation and produces superior results in both identity preservation and
prompt fidelity.

##  5 Conclusion

In this paper, we present Layout-and-Retouch, a two-stage framework for
creating personalized images with prompt-aligned, diverse configurations. Our
motivation stems from preliminary observations on the limited layout
generation capacity, which results in a weak ability to handle challenging
prompts. To address this issue, we introduce the step-blended denoising to
diversify image layouts by leveraging the expressive power of the vanilla T2I
model. Furthermore, we propose the multi-source attention swap and adaptive
mask blending to faithfully transfer the visual features of the reference
image while preserving the layout image structure. The combination of these
modules enables our framework to effectively enrich image layouts, thereby
improving its ability to handle challenging prompts. Extensive experiments
demonstrate the superiority of Layout-and-Retouch in generating diverse
prompt-aligned images, particularly in handling challenging prompts compared
to baselines.

## References

  * [1] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European conference on computer vision, pages 707–723. Springer, 2022.
  * [3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.
  * [4] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  * [5] Ming Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22503–22513, 2023.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [7] Daewon Chae, Nokyung Park, Jinkyu Kim, and Kimin Lee.  Instructbooth: Instruction-following personalized text-to-image generation.  arXiv preprint arXiv:2312.03011, 2023.
  * [8] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation.  arXiv preprint arXiv:2305.03374, 2023.
  * [9] Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo.  Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer.  arXiv preprint arXiv:2312.09008, 2023.
  * [10] Xiaoyue Duan, Shuhao Cui, Guoliang Kang, Baochang Zhang, Zhengcong Fei, Mingyuan Fan, and Junshi Huang.  Tuning-free inversion-enhanced control for consistent image editing.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1644–1652, 2024.
  * [11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [12] Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al.  Photoswap: Personalized subject swapping in images.  Advances in Neural Information Processing Systems, 36, 2024.
  * [13] Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, and Xin Eric Wang.  Swapanything: Enabling arbitrary object swapping in personalized visual editing.  arXiv preprint arXiv:2404.05717, 2024.
  * [14] Inhwa Han, Serin Yang, Taesung Kwon, and Jong Chul Ye.  Highly personalized text embedding for image manipulation by stable diffusion.  arXiv preprint arXiv:2303.08767, 2023.
  * [15] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323–7334, 2023.
  * [16] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yee K Wong.  Vico: Detail-preserving visual condition for personalized text-to-image generation.  arXiv preprint arXiv:2306.00971, 2023.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [19] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or.  Style aligned image generation via shared attention.  arXiv preprint arXiv:2312.02133, 2023.
  * [20] Mengqi Huang, Zhendong Mao, Mingcong Liu, Qian He, and Yongdong Zhang.  Realcustom: Narrowing real text word for real-time open-domain text-to-image customization.  arXiv preprint arXiv:2403.00483, 2024.
  * [21] Sangwon Jang, Jaehyeong Jo, Kimin Lee, and Sung Ju Hwang.  Identity decoupling for multi-subject personalization of text-to-image models.  arXiv preprint arXiv:2404.04243, 2024.
  * [22] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  arXiv preprint arXiv:2304.02642, 2023.
  * [23] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023.
  * [24] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  arXiv:2304.02643, 2023.
  * [26] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023.
  * [27] Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin.  Direct consistency optimization for compositional text-to-image personalization, 2024.
  * [28] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr.  Manigan: Text-guided image manipulation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7880–7889, 2020.
  * [29] Henglei Lv, Jiayu Xiao, Liang Li, and Qingming Huang.  Pick-and-draw: Training-free semantic guidance for text-to-image personalization.  arXiv preprint arXiv:2401.16762, 2024.
  * [30] Jiancang Ma, Junhao Liang, Chen Chen, and H. Lu.  Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning.  ArXiv, abs/2307.11410, 2023.
  * [31] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [32] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [33] Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang.  Dragondiffusion: Enabling drag-style manipulation on diffusion models.  arXiv preprint arXiv:2307.02421, 2023.
  * [34] Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang.  Dreammatcher: Appearance matching self-attention for semantically-consistent text-to-image personalization.  arXiv preprint arXiv:2402.09812, 2024.
  * [35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  SDXL: Improving latent diffusion models for high-resolution image synthesis.  In The Twelfth International Conference on Learning Representations, 2024.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [38] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [39] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International conference on machine learning, pages 8821–8831. Pmlr, 2021.
  * [40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [41] Azriel Rosenfeld and John L Pfaltz.  Distance functions on digital pictures.  Pattern recognition, 1(1):33–61, 1968.
  * [42] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in neural information processing systems, 35:36479–36494, 2022.
  * [44] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  Advances in neural information processing systems, 29, 2016.
  * [45] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning, 2023.
  * [46] Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu.  Freeu: Free lunch in diffusion u-net.  arXiv preprint arXiv:2309.11497, 2023.
  * [47] Yu-Chuan Su, Kelvin CK Chan, Yandong Li, Yang Zhao, Han Zhang, Boqing Gong, Huisheng Wang, and Xuhui Jia.  Identity encoder for personalized diffusion.  arXiv preprint arXiv:2304.07429, 2023.
  * [48] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  * [49] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  arXiv preprint arXiv:2303.09522, 2023.
  * [50] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [51] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [52] Weili Zeng, Yichao Yan, Qi Zhu, Zhuo Chen, Pengzhi Chu, Weiming Zhao, and Xiaokang Yang.  Infusion: Preventing customized text-to-image diffusion from overfitting.  arXiv preprint arXiv:2404.14007, 2024.
  * [53] Jing Zhao, Heliang Zheng, Chaoyue Wang, Long Lan, and Wenjing Yang.  Magicfusion: Boosting text-to-image generation performance by fusing diffusion models, 2023.
  * [54] Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu.  Enhancing detail preservation for customized text-to-image generation: A regularization-free approach.  arXiv preprint arXiv:2305.13579, 2023.

In Appendix, we first discuss on broader impacts and limitations of the
proposed method (Section A). Next, experimental details including
implementation details are provided (Section B). Lastly, we present additional
qualitative and quantitative analysis of our method(Section C). This includes
in-depth visualizations of each component as well as further explanation about
component. Furthermore, we present a user study to compare the effectiveness
and user preferences compared to baselines.

##  Appendix A Discussion

###  A.1 Broader Impacts.

The proposed method can significantly enhance personalized image generation
models, which may have various social impacts. Positively, it can facilitate
the creation of more tailored and relevant content for individuals, improving
user experiences across applications like social media, advertising, and
digital art. Additionally, It can also assist in personalized education and
training, providing custom visual aids suited to individual learning styles.
However, there are potential risks, including the misuse of the technology for
improving the quality of deepfakes or unauthorized digital
impersonations,leading to privacy violations and misinformation. Ethical
concerns also arise regarding the data used for personalization, especially if
it involves sensitive or personal information. Therefore, ensuring
transparency, obtaining consent, and implementing robust security measures are
crucial to mitigate these risks.

###  A.2 Limitations.

While the proposed method shows promising performances in personalized image
generation, it often fails to generate an user-intended image due to the
reliance on vanilla Stable Diffusion (SD) model [40]. Specifically, a highly
complicated prompt condition that is beyond the capacity of vanilla SD can
cause a failure in the layout image, subsequently leading to undesirable
target image. As seen in Fig. 5 (a), if the layout image does not adhere to
prompt condition, the final output exhibits similar issues. Furthermore, the
shape similarity between objects in layout and reference image would affect
the identity preservation, as large shape discrepancies are difficult to
rectify during the retouching stage. Fig. 5 (b) illustrates an example of
generating with different layout images. As can be seen, retaining similar
shapes is beneficial for creating a personalized object characteristics in
later stage. We believe that using a more robust foundation model such as SDXL
[36], to enhance prompt understanding and adherence on prompt conditions, is
an effective approach to mitigating this problem. Our future plan includes
utilizing SDXL as vanilla model in step-blended denoising and developing a
technique to minimize shape difference.

Figure 5: Layout Failure cases. (a) shows our generation results when layout
fails to generate appropriate context, (b) shows when layout fails to
faithfully generate shape of personalized concept. Since second stage of our
pipeline lies on layout image, our pipeline can fail in prompts beyond the
capacity of pre-trained SD backbone.

##  Appendix B Experimental Details

###  B.1 Implementation Details

We use SD 1.4 [40] as our baseline vanilla foundation text-to-image model, and
pre-training personalized models also are conducted based on SD 1.4.
Specifically, we utilize all weights of personalized baseline models: Textual
inversion [11], Dreambooth [42], Custom Diffusion [26] released in previous
method [34]. In experiment, we use a DDIM sampler with a total inference time
step to $T=50$. Empirically, we set $\lambda_{1}=5$, $\lambda_{1}=3$ for
normal and challenging prompt respectively and $\lambda_{2}=10$ for all
datasets. Additionally, the mask blend operation starts from $31^{th}$ to the
last step. For our GPU setup, we use a NVIDIA GeForce RTX 3090 GPU for all
experiments, which consumes about 18GiB memory during inference. It takes
about 30 seconds for generating a single image including pre-processing. As
for other parameters such as specific layers used for reference, we follow the
prior works [5, 34].

###  B.2 Prompts for Diversity Experiment

Since one of our core claims is the diversity of synthesized images, we
demonstrate our capabilities using a few different ways. First, we visualize
the center point distributions of the target subjects (see Fig. 1), using all
31 prompts curated by ViCo [16] and generating 10 images per prompt. However,
when measuring diversity using classic metrics such as Inception Score [44] or
visualizing embedding spaces [31], it is insufficient to generate only 10
samples with each prompt. Therefore, we randomly sample 10 prompts from three
categories in total, from normal prompts and challenging prompts used in
DreamMatcher [34]. These categories include large displacement, occlusion, and
novel-view synthesis. We then generate 400 samples per prompt, resulting in a
total of 4,000 images to evaluate Inception Scores for the objects depicted in
Fig. 4. The list of prompts is provided in Fig. 6 below.

Figure 6: The list of 10 randomly sampled prompts that were used to generate
images to compare the diversity of the output. We sampled the prompts
according to the three different categories.

Figure 7: Qualitative examples of $\mathbf{T_{CLIP}}$ and ImageReward [50].
Following Instructbooth [7], we measure both $\mathbf{T_{CLIP}}$ and
ImageReward on 8 displayed images above. Each row illustrates instances where
ImageReward provide more accurate evaluations compared to $\mathbf{T_{CLIP}}$.
Based on these observations, we additionally evaluate our method using
ImageReward.

###  B.3 Analysis on evaluation metrics

Methods | $\mathbf{T_{CLIP}}$ | $\mathbf{ImageReward}$
---|---|---
Textual Inversion | 0.277 | -0.975
\+ Ours |  0.297 (+7.2%) |  -0.437 (+55.2%)
Dreambooth | 0.313 | 0.255
\+ Ours |  0.317 (+1.3%) |  0.420 (+64.7%)
Custom Diffusion | 0.315 | 0.257
\+ Ours |  0.317 (+0.6%) |  0.288 (+12.1%)
Table 5:  The quantitative experimental results of evaluating the effect of
our methodology in prompt fidelity, compared to the baseline optimization-
based methods. We follow the dataset gathered by ViCo [16]. Performance gains
are indicated in blue.

In main manuscript, we utilize distance-based metrics such as
$\mathbf{I_{DINO}}$, $\mathbf{I_{CLIP}}$, and $\mathbf{T_{CLIP}}$ to evaluate
identity preservation and prompt fidelity. Specifically, ViT-S/16 DINO [6] and
ViT-B/32 CLIP [37] are used to extract the embeddings from the generated and
input conditions such as reference images and prompts. Although these metrics
show reasonable evaluation results as previous work [5, 42, 34, 46, 53]
demonstrated, we often face unexpected results on prompt fidelity evaluation
using $\mathbf{T_{CLIP}}$. Fig. 7 illustrates inaccurate evaluation of
$\mathbf{T_{CLIP}}$ on the generated images. Specifically: (1) In Row 1: even
though most images fail to comply with the prompt condition,
$\mathbf{T_{CLIP}}$ remains relatively high. (2) In Row 2: despite most images
successfully adhering to the prompt condition, $\mathbf{T_{CLIP}}$ is
comparatively low.

To address this issue, recent work [7, 21, 20] suggest to employ a neural
network-based evaluation model such as ImageReward [50] that is trained on a
large human feedback dataset. We qualitatively validate its robustness as
evaluation metric by repeatedly measuring the scores of the generated images.
Representative image sets and evaulation results are shown in Fig. 7, where
ImageReward demonstrates accuracy, whereas $\mathbf{T_{CLIP}}$ as previously
discussed tends to relatively inaccurate.

Upon this, we evaluate Layout-and-Retouch using ImageReward to determine how
beneficial it is for improving prompt fidelity when plugged into the baseline.
Similar to main manuscript, we use ViCo [16] dataset and generate 8 images per
object and prompt. Table 5 shows the evaluation on prompt fidelity based on
both $\mathbf{T_{CLIP}}$ and ImageReward. Layout-and-Retouch greatly improves
ImageReward across baselines, demonstrating the strength of our method to
generate prompt-aligned images.

###  B.4 User Study Setup

We conduct user study with aim of evaluating three aspects: (1) Identity
preservation, (2) Prompt alignment, (3) Diversity of outputs. As baselines, we
compare (1), (2) with three different prior work: MagicFusion [53], FreeU
[46], DreamMatcher [34]. Next, we evaluate (3) with DreamMatcher.

The users were given 16 questions to compare identity preservation, 16
questions to compare prompt alignment, and 10 questions to compare diversity
of generation. Two different sets of question sets were randomly distributed
in an attempt to conduct a faithful user study. 48 users answered to 42
questions, resulting in a total of 2,016 responses. Specifically, we design
our questions as follows:

  1. Q1:

Please rank the methods (A,B,C,D) for generating an object most similar to the
one contained in the following reference image.

  2. Q2:

Please choose the method for generating an image most similar to the given
prompt.

  3. Q3:

Please choose the method that generate more diverse images with the given
prompt.

Each question is asked to evaluate (1) Identity preservation, (2) Prompt
alignment, (3) Diversity of outputs, respectively. All samples used for the
user study were randomly chosen from a pool comprising tens of thousands of
images generated from various prompts and object selections using ViCo [16]
dataset. For clear understanding, we present an user interface of the user
study in Fig. 11 and 12

##  Appendix C Additional Analysis on Layout-and-Retouch

In this section, we analyze our framework Layout-and-Retouch by providing
additional experiments with aim of helping to understand the proposed method.

###  C.1 Analysis on Step-blended Denoising

Figure 8: Analysis on the effect of $\lambda_{1}$ of step-blended denoising.
Note that the experiments were conducted based on 31 normal ViCo [16] prompts
using Dreambooth [42] baseline.

Figure 9: Visualization of the cross-attention map during step-blended
denoising. To obtain cross-attention maps, we aggregate the decoder layers of
32 x 32 resolution.

##### Hyperparameter sensitivity.

Since synthesizing initial layout is critical for Layout-and-Retouch, we
empirically analyze the effect of the number of vanilla SD iterations in
denoising steps. As shown in Fig. 8, we observe that an early stop of vanilla
SD denoising leads to higher identity preservation ($\mathbf{I_{DINO}}$,
$\mathbf{I_{CLIP}}$), while simultaneously lowering prompt fidelity
($\mathbf{T_{CLIP}}$, $\mathbf{ImageReward}$). On the other hand, as initial
layout generation steps increase, identity preservation drops rapidly while
the prompt fidelity improves. This demonstrates a trade-off relationship
between identity preservation and prompt fidelity as previous work [27]
pointed out. Empirically, we determine our optimal $\lambda_{1}$ as five,
where identity preservation does not dramatically drop and prompt fidelity is
beyond its criterion confirmed in qualitative observation.

##### Cross-attention map visualization.

We visualize a cross-attention (CA) map during step-blended denoising process
in Fig. 9. The two images shown in the left side are final images generated
using the same seeds, using a solely vanilla denoising and step-blended
denoising with $\lambda_{1}=5$. We can observe that the elements of the image
(e.g., dog pose and background) are remarkably similar, suggesting that five
denoising iterations suffice to convey a diverse layout from the vanilla model
to subsequent steps. Fig. 9 shows detailed visualizations of cross-attention
map, which illustrates the alterations in the dog’s form throughout the
denoising process. This visualization highlights that modifying only the
initial steps with the vanilla model is sufficient for generating diverse and
promptly aligned layouts. Subsequent denoising steps employing the pre-trained
personalized model further enrich the layout image with ample information
about user-specific object

###  C.2 Qualitative Analysis on Mask Variants

Figure 10: Visualization of each foreground mask and its outputs. We
illustrate the masks obtained from different sources and combined mask via
adaptive mask blending alongside the generated image for each mask.

In Fig. 10, we compare our adaptive mask blending method and the mask
extracted with SAM [25] and the cross-attention map after thresholding. In the
main paper, we argued that utilizing only one of the two masks would result in
either noisy region interruption or region misalignment. As seen in Column 2
of Fig. 10, $\mathbf{M}^{c}$ often fail to retrieve the entire foreground
object and contains noisy region, resulting in degraded performance in
retaining visual details and undesirable artifacts (See red box). On the other
hands, Column 2 of Fig. 10 shows that when using only
$\mathbf{M}^{\text{SAM}}$ to define the target region, it often fails to
identify important object part, impairing the necessary components of the
layout image. To address these issues, we apply adaptive mask blending,
successfully overcoming them by finding a balanced mask region and strength.

Figure 11: An example of a user study comparing Layout-and-Retouch with prior
works: We compare Identity Preservation and Prompt Alignment with MagicFusion
[53], FreeU [46] and DreamMatcher [34]. For Identity Preservation, we provide
a reference image and 4 examples generated by the 4 methods with the same
seed. For Prompt Alignment, we additionally provide the prompt that was used
to generate the 4 images. The samples were selected randomly from a wide pool
for fair comparison.

Figure 12: An example of a user study comparing Layout-and-Retouch with prior
works: We compare Image Diversity with DreamMatcher [34]. We provide the users
with a prompt, and the 8 images generated with the same random seeds. The
samples were selected randomly from a wide pool for fair comparison.

###  C.3 Additional qualitative results

We additionally report qualitative results of our work as follows:

  1. 1.

Additional qualitative comparisons with plug-in based baselines across various
objects (Fig. 13, 14, and 15.

  2. 2.

Additional qualitative comparisons by adding the proposed method on baselines
(Fig. 16, 17, and 18).

  3. 3.

Additional comparison with DreamMatcher on image diversity (Fig. 19)

Figure 13: Qualitative comparison with plug-in based baselines [34, 53, 46]:
We use Dreambooth [42] for all baselines. Note that our method does not
produce images with identical structures, as the initial steps of generating
layout images rely on vanilla Stable Diffusion [40].

Figure 14: Qualitative comparison with plug-in based baselines [34, 53, 46]:
We use Dreambooth [42] for all baselines. Note that our method does not
produce images with identical structures, as the initial steps of generating
layout images rely on vanilla Stable Diffusion [40].

Figure 15: Qualitative comparison with plug-in based baselines [34, 53, 46]:
We use Dreambooth [42] for all baselines. Note that our method does not
produce images with identical structures, as the initial steps of generating
layout images rely on vanilla Stable Diffusion [40].

Figure 16: Qualitative comparison with baselines: We compare Layout-and-
Retouch with the three baseline methods [11, 42, 26] to show the effectiveness
of our work. Note that our method does not produce images with identical
structures, as the initial steps of generating layout images rely on vanilla
Stable Diffusion [40].

Figure 17: Qualitative comparison with baselines: We compare Layout-and-
Retouch with the three baseline methods [11, 42, 26] to show the effectiveness
of our work. Note that our method does not produce images with identical
structures, as the initial steps of generating layout images rely on vanilla
Stable Diffusion [40].

Figure 18: Qualitative comparison with baselines: We compare Layout-and-
Retouch with the three baseline methods [11, 42, 26] to show the effectiveness
of our work. Note that our method does not produce images with identical
structures, as the initial steps of generating layout images rely on vanilla
Stable Diffusion [40].

Figure 19: Diversity comparison against DreamMatcher [34]: We present the
generated images of Layout-and-Retouch and DreamMatcher, to highlight the
capacity of the proposed approach in producing diverse images.