11footnotetext: First two authors contributed equally.

# Highly Personalized Text Embedding for Image Manipulation by Stable
Diffusion

Inhwa Han111 , Serin Yang111 , Taesung Kwon, Jong Chul Ye
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South
Korea
{inhwahan, yangsr, star.kwon<EMAIL_ADDRESS>
###### Abstract

Diffusion models have shown superior performance in image generation and
manipulation, but the inherent stochasticity presents challenges in preserving
and manipulating image content and identity. While previous approaches like
DreamBooth [31] and Textual Inversion [7] have proposed model or latent
representation personalization to maintain the content, their reliance on
multiple reference images and complex training limits their practicality. In
this paper, we present a simple yet highly effective approach to
personalization using highly personalized (HiPer) text embedding by
decomposing the CLIP embedding space for personalization and content
manipulation. Our method does not require model fine-tuning or identifiers,
yet still enables manipulation of background, texture, and motion with just a
single image and target text. Through experiments on diverse target texts, we
demonstrate that our approach produces highly personalized and complex
semantic image edits across a wide range of tasks. We believe that the novel
understanding of the text embedding space presented in this work has the
potential to inspire further research across various tasks. Project page is
available at http://hiper0.github.io/

Figure 1: Image manipulation results with highly personalized (HiPer) text
embeddings. In the upper row, the identities of the rabbit and the dog are
well preserved while adequately manipulating the images to align with target
texts. In the bottom row, not only motion and background, but also texture of
the source image is transformed towards corresponding target text.

Figure 2: The proposed method. (Training) First, the source text prompt, which
have the meaning of source image, is converted to text embedding. Some parts
of text embedding, which have no information, are removed. The informative
target embedding part and the personalized embedding is concatenated, and they
are the input of pre-trained U-net. In training, the personalized embedding is
only optimized. Although this figure depicts it as learning in image space,
the embedding is actually optimized in latent space. (Inference) The target
embedding is also cropped and concatenated with personalized embedding.
Personalized embedding vector is calibrated by multiplying it with
$\alpha=0.8$. The pre-trained text-to-image model, which conditioned that
embedding, generates an image which has the meaning of target text and the
subject of source image.

##  1 Introduction

Image manipulation has long been a subject of interest in image processing.
Early works on image manipulation utilized GANs [35, 9, 36, 5, 22]. However,
these approaches required annotated datasets, manual identification of editing
directions, or off-the-shelf classifiers. The advent of CLIP [27] has made
text-guided image manipulation feasible, and combining it with the generative
power of StyleGAN [13] has led to significant successes with only text
prompts, as demonstrated in recent works such as [26, 8].

Diffusion models have also gained significant attention in the domain of text-
guided image synthesis. The inherent stochastic nature of these models has
proven to be particularly useful in generating high-quality images from text
prompts [1, 15, 10, 17]. One notable example of a text-to-image diffusion
model is Stable Diffusion [30], which has shown remarkable performance in
generating images that align well with the conditioning text prompts. However,
despite its impressive results, the stochastic nature of Stable Diffusion can
lead to issues with maintaining critical content information, such as the
shape or identity of the subject in the generated image. This limitation is
not unique to Stable Diffusion but is a fundamental issue with all diffusion
models. Therefore, researchers have focused on developing effective techniques
to ensure that the generated images maintain crucial content information
during the conversion of the style and other visual attributes.

For instance, DreamBooth [31] and Textural Inversion [7] proposed using an
identifier to personalize the generative model or text representation,
respectively. Although both studies succeeded in preserving the identity of
the subject, their application is limited due to the requirement of three to
five reference images for optimal performance, which may not be feasible in
many scenarios. Additionally, model fine-tuning or optimizing intermediate
representation requires a considerable amount of time, which limits their
practical use. Moreover, these prior works have inferior capacity in motion
editing, which further restricts their application in scenarios where motion
editing is required.

Figure 3: Cross Attention maps in the final timestep of text-to-image
diffusion models. The source text is “a standing dog’ and the target text is
“a sitting dog”. Cross Attention maps (a) conditioned with ${{\bm{e}}}_{src}$
(b) conditioned with $[{{\bm{e}}}_{src}^{\prime},{{\bm{e}}}_{hper}]$, (c)
conditioned with $[{{\bm{e}}}_{tgt}^{\prime},{{\bm{e}}}_{hper}]$. (d) Cross
attention maps by Imagic [14] with Stable Diffusion.

Imagic [14] is one of the diffusion-based methods that proposed to solve
semantic image editing. It has shown the potential to edit motion while
preserving the underlying structure and background. However, the method is
computationally demanding and involves multiple optimization steps and model
fine-tuning, resulting in significant time consumption. Moreover, the final
output is obtained through interpolation of the optimized embedding and CLIP
embedding of the target text. This implies that the output is not an exact
embedding aligned with the target text, but rather an approximation in the
embedding space. This makes it hard to maintain the structure information and
requires a significant amount of manual work to sample high-quality final
outputs.

Therefore, it is imperative to develop more efficient and effective methods
that can preserve the identity of the subject while accommodating motion
editing and having lower computational costs to make these approaches more
practical and applicable in real-world scenarios.

In this work, we propose a novel approach to address the limitations of
existing diffusion model-based approaches for image manipulation by text
prompts. Our method is simple yet highly effective in personalizing Stable
Diffusion. One of the key contributions of our work is the discovery of the
unique semantic decomposition of the CLIP embedding space in Stable Diffusion
[30]. We found that while the initial parts of the CLIP embedding space are
useful for image manipulation by text tokens, the tail parts play a crucial
role in preserving the identity of the subject in the source image. By
optimizing the tails, we can retain the identity of the subject while
modifying the remaining parts of the image to align with the target prompts.
Compared to existing methods such as DreamBooth [31] and Textural Inversion
[7], our approach requires only a single image and target text, without
additional training on the diffusion model, resulting in a significant
increase in computational efficiency.

To demonstrate the effectiveness of our method, we conducted experiments on
diverse target texts with varying backgrounds, textures, and motions. Our
results show that our method outperforms existing methods in personalization
and semantic manipulation of images by text prompts. Overall, our approach
offers a simple yet highly effective solution to the limitations of existing
methods and has the potential to advance the field of text-guided image
synthesis. Our contribution can be summarized as following:

  * •

We propose a simple yet highly effective optimization method that does not
require model fine-tuning or identifiers, enabling a short training time of
approximately 3 minutes.

  * •

We propose a novel method to personalize the text embedding using only a
single image. Our approach involves decomposing the CLIP embedding spaces
based on their positions.

  * •

Our approach enables manipulation not only of the background or texture but
also of motion, which was difficult to achieve using previous methods.

##  2 Related Work

###  2.1 Text-guided image synthesis

For text-guided image synthesis using GANs [19, 42, 11, 18, 39], text was
usually used as condition. With image-caption data pairs, GANs were trained to
generate samples by utilizing attention mechanisms or contrastive approaches
[39, 40]. With the introduction of CLIP model [27] which has been trained on a
vast dataset of 400 million image-text pairs, the need for annotated images
has been reduced. By combining the great capacity of CLIP and the generative
power of StyleGAN [13], works on text-guided image manipulation have been
developed with CLIP-driven stylizing losses [26, 8]. Furthermore, CLIPstyler
[16] introduces a CNN encoder-decoder architecture that leverages patch-wise
CLIP loss to capture both content and style properties. As a result, the model
can generate and manipulate images beyond the domains of pre-trained
generators. With autoregressive framework, [29] and [4] are text-to-image
generators that utilize transformer-based joint pretraining for both vision
and language tokens, and have shown superior performance compared to previous
GAN-based methods.

###  2.2 Diffusion models

Diffusion models have become popular in the field of generative models for
their ability to transform a noise vector into an output image through a
diffusion process [12, 37, 24]. Recent research [3, 38] has demonstrated that
the diffusion model exhibits superior image generation quality in comparison
to GANs [2, 41]. The diffusion models are recently used for image manipulation
task [20, 33, 21] by adding noise to an image, and then editing the image
during the reverse diffusion process. Using the diffusion models and CLIP
model, the text-based image editing methods [15, 1, 23] have been studied, and
shown great image manipulation performance. DiffusionCLIP [15] and Blended
Diffusion [1] uses CLIP model that regulates the characteristics of the image
according to the provided textual prompts during diffusion process. In [10]
and [25], the image manipulation methods using diffusion model involves the
controlling of cross-attention map.

Beyond image manipulation, text-based image synthesis models using Diffusion
models [32, 28, 30, 23] has emerged, and generate more accurate image
correspond to the given text prompts. Among them, Stable Diffusion [30] is a
text-to-image generation model that has attracted attention for its
flexibility. It has been recently extended to text-based image manipulation
and enables both local and global editing, as well as personalization,
allowing for a wide range of image manipulation possibilities. Despite the
advantages of diffusion models, they have limitations when it comes to
preserving identity information in the generated images.

This has led to significant works on preserving the subject’s identity in
images by optimizing embeddings or personalizing diffusion models. Imagic [14]
proposed a three-step method involving diffusion model fine-tuning to optimize
the embedding that generates the exact source image. However, it requires
sampling lots of images and may not always produce the desired output, as it
interpolates the optimized embedding with a CLIP embedding from the target
text in the final step. DreamBooth [31] utilizes a method that involves using
identifiers to preserve the identity of the subject in the source image. While
DreamBooth [31] can generate images with different conditions while preserving
the identity of the subject, it requires multiple images for fine-tuning the
diffusion model with complex loss functions, which can be time-consuming and
computationally expensive. Textual Inversion [7] finds new pseudo-words by
conducting personalization in the text embedding space. These pseudo-words
lead to identity preservation. However, Textual Inversion requires multiple
images for optimal performance and takes about an hour in order to optimize
the embedding vector associated with the pseudo-word.

##  3 Highly Personalized Text Embedding (HiPer)

###  3.1 Stable Diffusion Model

Diffusion models [12] attempt to model the data distribution $p_{\rm
data}({\bm{x}})$ by constructing a hierarchical latent variable model.
Specifically, DDPM [12] directly samples $\bm{x}_{t}$ from a clean image
$\bm{x}_{0}$ by adding Gaussian noise with $\beta_{t}\in(0,1)$ at time
$t\in[1,...,T]$,

$\bm{x}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$
(1)

where $\bm{\epsilon}\sim\mathcal{N}(0,\bm{I})$, $\alpha_{t}=1-\beta_{t}$, and
$\bar{\alpha}_{t}=\prod_{i=0}^{t}\alpha_{i}$. The reverse sampling process to
generate a clean image is then given by:

$\bm{x}_{t-1}=\frac{1}{\sqrt{1-\beta_{t}}}\left(\bm{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\bm{\epsilon}_{\theta}(\bm{x}_{t},t)\right)+\sigma_{t}\bm{\epsilon}.$
(2)

where where
$\sigma_{t}:=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$ and the
neural network $\bm{\epsilon}_{\theta}(x_{t},t)$ is trained by

$\displaystyle\min_{\theta}{{\mathbb{E}}}_{{\bm{x}}_{t},{\bm{x}}_{0},\bm{\epsilon}\sim\mathcal{N}(0,{\bm{I}})}\left[\|\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)-\bm{\epsilon}\|_{2}^{2}\right],$

Stable Diffusion Models (SDM) [30] is a publicly available diffusion model
with important modifications. Instead of using diffusion sampling directly in
the image space, SDM is based on the latent diffusion method, where the
forward and reverse diffusion sampling in (1) and (2) are performed in the
latent domain. To achieve this, SDM uses a VQ-GAN [6] encoder $E$ to convert
the image ${\bm{p}}$ to a lower-dimensional latent variable ${\bm{x}}$ (i.e.,
${\bm{x}}=E({\bm{p}})$). The diffusion sampling is then performed in the
latent space, and the final image is obtained using a VQ-GAN decoder $D$.
Another important modification in SDM is the addition of text-based
conditioning. Specifically, SDM uses the tokenizer of the CLIP text encoder
${\tau_{\phi}}$ to extract words from a given text prompt ${\bm{y}}$ and
convert them into numbers called tokens, denoted by
${\bm{e}}=\tau_{\phi}({\bm{y}})$. The tokens are then transformed into text
embeddings, which are used to condition the neural network during training:

$\displaystyle\min_{\theta}{{\mathbb{E}}}_{{\bm{x}}_{t},{\bm{x}}_{0},\bm{\epsilon}\sim\mathcal{N}(0,{\bm{I}})}\left[\|\bm{\epsilon}_{\theta}({\bm{x}}_{t},t,\tau_{\phi}({\bm{y}}))-\bm{\epsilon}\|_{2}^{2}\right],$

Therefore, SDM enables text-driven image generation using the reverse
diffusion sampling in (2) in the latent space. Instead of using
$\epsilon_{\theta}(\bm{x}_{t},t)$, the model uses a text-conditioned neural
network called $\bm{\epsilon}_{\theta}({\bm{x}}_{t},t,\tau_{\phi}({\bm{y}}))$.

The proposed method is implemented using pretrained Stable Diffusion model,
which is publicly available.

Figure 4: The qualitative comparison results. Compared with three stable
diffusion-based text-guided image manipulation methods, our method shows its
superiority. It could preserve the identities of the subject in source images,
while appropriately transforming the semantic information to align with the
CLIP embedding of the target text. Original Imagic results with Imagen show
comparable results by using proprietary text embedding scheme.

Figure 5: By concatenating highly personalized (HiPer) text embeddings with
different target embeddings, we can achieve precise image manipulation
results. This allows us to manipulate the image with high precision while
preserving the subject’s identity in the source image.

###  3.2 Text Embedding Decomposition

#### Key observation

The objective of the proposed method is to optimize text embeddings to better
represent a given subject. In SDM, the input text is first converted into an
embedding by a tokenizer and text encoder, which is then conditioned to
autoencoder. Let ${{\bm{e}}}_{src}$ be the text embedding for source image.
The converted text embedding contains the information of the text. For
example, consider an image that corresponds to the text prompt “a standing
dog”. The source embedding has the information of “a standing dog”. However,
if the length of the input text is shorter than the maximum length of the text
embedding, then the end part of the embedding does not contain any information
about the input text (See Fig. 3(a)). Taking this into account, the proposed
method only optimizes a piece of the text embedding at the end, which has no
information about the source text. If successfully optimized, the piece of
embedding would contain a lot of information about the source image.

#### HiPer embedding

Consider a source text embedding ${\bm{e}}_{src}\in{\mathbb{R}}^{C\times M}$
for a text prompt ${\bm{y}}$, i.e. ${\bm{e}}_{src}=\tau_{\phi}({\bm{y}})$ (See
Figure 2). In the proposed method, we select the last $N$ tokens at the end of
the text as what we call the highly personalized (HiPer) embedding
${{\bm{e}}}_{hper}\in\mathbb{R}^{C\times N}$ for image representation, and the
remaining $M-N$ tokens is denoted by
${{\bm{e}}}_{sem}^{\prime}\in\mathbb{R}^{C\times(M-N)}$. More specifically,
this leads to the following decomposition of the text embedding:

$\displaystyle{\bm{e}}_{src}=\begin{bmatrix}{\bm{e}}^{\prime}_{src},{\bm{e}}_{hper}\end{bmatrix}$
(3)

Since the uninformative part is removed, ${{\bm{e}}}_{src}^{\prime}$ still
contains the information of source text similar to ${{\bm{e}}}_{src}$. Then,
for a given image ${\bm{p}}_{0}$ and its latent
${\bm{x}}_{0}=E({\bm{p}}_{0})$, the HiPer embedding ${\bm{e}}_{hper}$ is only
optimized whereas ${\bm{e}}_{src}^{\prime}$ generated from the source text is
maintained:

$\displaystyle{\bm{e}}_{hper}$
$\displaystyle=\arg\min\limits_{{\bm{e}}_{h}\in\mathbb{R}^{C\times
N}}\mathbb{E}_{\bm{x}_{t},\bm{\epsilon}\mathtt{\sim}\mathcal{N}(0,{\bm{I}})}\left[\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\bm{x}_{t},t,[{{\bm{e}}}_{src}^{\prime},{{\bm{e}}}_{h}])\|^{2}\right]$
(4)

Note that this is different from Textual Inversion [7], where the whole text
embedding ${\bm{e}}_{src}$ is optimized:

$\displaystyle{\bm{e}}_{src}=\arg\min\limits_{{\bm{e}}\in\mathbb{R}^{C\times
M}}\mathbb{E}_{\bm{x}_{t},\bm{\epsilon}\mathtt{\sim}\mathcal{N}(0,{\bm{I}})}\left[\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\bm{x}_{t},t,{\bm{e}})\|^{2}\right]$
(5)

In the inference step, ${{\bm{e}}}_{src}^{\prime}$ is replaced with the
cropped text embedding ${{\bm{e}}}_{tgt}^{\prime}$ of the target prompt, while
the tail $N$ tokens are replaced with the HiPer embedding ${\bm{e}}_{hper}$.
Then, the reverse diffusion generates latent samples by

$\bm{x}_{t-1}=\frac{1}{\sqrt{1-\beta_{t}}}\left(\bm{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\overline{\alpha}_{t}}}\bm{\epsilon}_{\theta}(\bm{x}_{t},t,{\bm{e}}_{cmp})\right)+\sigma_{t}\bm{\epsilon}.$
(6)

where the composite embedding ${\bm{e}}_{cmp}$ is given by

${\bm{e}}_{cmp}=[{{\bm{e}}}_{tgt}^{\prime},{{\bm{e}}}_{hper}].$

Subsequently, the final image is obtained by applying decoder to the final
latent sample, i.e. $\bm{x}_{0}$.

When the composite embedding ${\bm{e}}_{cmp}$ is conditioned to pre-trained
Stable Diffusion model, we found that the output image has the meaning of
target text and the personalization from the subject of the source image. For
instance, if the source image has the meaning of “a standing dog” and the
target text is “a sitting dog”, the target embedding with personalized
embedding generates the source image’s dog, which is sitting. The simplicity
of our method contrasts with its impressive performance results, which will be
demonstrated in the following section.

Figure 6: Text-driven image manipulation results featuring a female doctor.

Figure 7: The effect of the number of personalized tokens, $N$. Increasing $N$
produces outputs similar to the source image. On the other hand, smaller
values of $N$ generate adequately edited outputs but do not sufficiently
preserve the identity.

##  4 Experimental Results

Our method is based on Stable Diffusion [30]. We set the number of
optimization steps as $1000$ and the number of personalized tokens $N=5$. The
dataset used as source images are from Ted of Imagic [14] and LAION [34]. We
experimented with the models using PyTorch library on NVIDIA GeForce GTX 3090.

#### Text-driven image manipulation

As shown in Figure 1, our method could achieve image manipulation in all the
three areas - motion, background, and texture. The rabbit and the dog in the
source images could eat, read books, jump, and swim in diverse backgrounds. In
addition to motion and background, texture of images could be controlled. The
real source images were well translated into diverse painting styles.
Furthermore, the subject identity is well preserved in the generated images.
The shape of the rabbit’s ears and the color of its fur are maintained in the
generated images. Also, the dog’s identity, such as the color, species, and
fur texture, was contained throughout the output images. Even the appearance
of the dog leash is also preserved.

Moreover, our method exhibits impressive capability in simultaneously
manipulating three areas, which are typically challenging to achieve. In
Figure 1, the motion “playing with a cat,” the background “on the grass,” and
the texture “Claude Monet” style are appropriately modulated while preserving
the identity of the dog in the source image.

#### Comparative studies

We compared our model with three diffusion model based image manipulation
baselines, i.e. Imagic [14], DreamBooth [31], and Textual Inversion [7].
Although DreamBooth and Textual Inversion require three to five images for
optimal training, we trained them with a single image because of their
unavailability. We compared our method with these three methods which were
implemented based on Stable Diffusion [30]. Additionally, the Imagic authors
provided us with the official Imagic results that were obtained using Imagen,
as a model that is not publicly available. We also included them for
comparison. The qualitative results are illustrated in Figure 4.

It is noticed from the results that the implementation of Imagic using the
Stable Diffusion model exhibits inferior content preservation capabilities
when compared to the official Imagic result and the other results. Although
Imagic was proposed to edit semantic information while preserving the other
overall shapes, it showed inferior performance with Stable Diffusion model. In
the first row, the bird’s head is different from the source image and the dog
in the second row has lots of hair around its head in the generated image
which is different from the source image. It is due to interpolation in the
inference step, which approximates between the target and the optimized
embedding. The Imagen model-based implementation of Imagic yields
significantly improved image manipulation results, preserving the original
source image while making targeted edits based on specified textual inputs.
However, due to the Imagic model’s strong tendency to preserve the object’s
position and background of the source image, it results in poor performance
when the source image undergoes significant changes due to target textual
inputs. As shown in the last row of the figure, when the white bear in source
image undergoes substantial transformations due to textual inputs, the model
fails to preserve its identity.

In case of DreamBooth, the diffusion model is fine-tuned on a single image
which results in the output image same as the source one. The semantic editing
along the target text is not adequately applied. On the other hand, Textual
Inversion shows more plausible results compared to DreamBooth. However, we
observed in the fourth and the last rows of Figure 4 that its performance
degraded when dealing with simultaneous transformation of motion, background,
and texture. The dog in the generated image plays with a ball but not on the
road. The ball also does not look like a soccer ball. Also, cookies cannot be
seen from the output images in the last row.

Our proposed method overcomes the limitations of existing diffusion-based
methods in editing multiple aspects and is able to generate images that
maintain the same identity as the source while effectively transforming the
attributes to align with the target text. Notably, our method enables a high
level of manipulation, as evidenced by the bear in the last row, which is
shown eating cookies in the river, a transformation that was not possible
using any existing method.

#### More examples

In Figure 5, we present the impressive text-driven image manipulation results
achieved through our method. By leveraging highly personalized (HiPer) text
embeddings and concatenating them with different target embeddings, we are
able to achieve precise image manipulation without compromising the subject’s
identity in the source image. Our method empowers users to control various
visual attributes of the image, such as colors, movement, background, and
detailed scene description, while preserving the original image’s integrity.
This not only fosters creative expression but also has numerous potential
applications in fields such as fashion, advertising, and entertainment.
Moreover, our method is computationally efficient and can be applied to a
diverse range of images, making it a practical and versatile tool for image
manipulation.

Figure 6 showcases the text-driven image manipulation results featuring a
doctor. The original image depicts the doctor wearing a white lab coat and
standing against a plain background. The manipulation results highlight our
method’s ability to modify specific visual attributes of the image based on
the input text while maintaining the integrity of the original image. In this
case, the doctor’s posture and hand gesture were modified based on the input
text, resulting in an image where the doctor appears to be giving the thumbs
up. Additionally, the image was converted to a cartoon style, demonstrating
our method’s versatility in handling different image styles. Overall, this
example illustrates the potential of our method in various applications.

Additional and more comprehensive experimental results can be found in the
Supplementary Materials.

##  5 Discussion

#### Number of personalized tokens

We performed experiments with varying numbers of tokens $N$ to study their
effect on personalization. Figure 7 illustrates that increasing the number of
tokens leads to overfitting of the personalized embedding, producing outputs
similar to the source image and failing to capture the desired changes. In the
last column, the dogs in the generated image are not shown as sleeping, the
background does not feature snow, and the texture has not been transformed
into a pop art style. On the other hand, smaller values of $N$ generate
adequately edited outputs but do not sufficiently preserve the identity. It is
shown that the dogs generated with $N=1$ are different from the source image.
Based on these findings, we chose $N=5$ as our baseline.

#### Comparison of cross-attention maps

Our method is based on a key observation that the cross-attention map in text-
to-image diffusion models does not contain any information at the end of the
token. Figure 3 show the cross-attention map in the final timestep of text-to-
image diffusion model conditioned with other embedding. The total number of
cross attention map is 77, which corresponds to the total number of token. The
source image is the input image of Fig. 2, which correspond the source text of
“a standing dog’, and source embedding of ${{\bm{e}}}_{src}$, The target text
is “a sitting dog” and the corresponding embedding is ${{\bm{e}}}_{tgt}$. With
the proposed method, the personalized embedding ${{\bm{e}}}_{hper}$ is
optimized. Each represents cross-attention map (a) conditioned with
${{\bm{e}}}_{src}$, (b) conditioned with
$[{{\bm{e}}}_{src}^{\prime},{{\bm{e}}}_{hper}]$, (c) conditioned with
$[{{\bm{e}}}_{tgt}^{\prime},{{\bm{e}}}_{hper}]$. Figure 3 (a) of
${{\bm{e}}}_{src}$ shows that the attention map corresponding to each text is
only activated, and the end of the token is not activated. (b) and (c) shows
that the map related to personalized embedding is activated. In the case of
target embedding with personalized embedding (c), some attention map related
the personalized embedding follows the target embedding part’s attention map,
but maintain the distribution of personalized embedding, compared to (b). In
Figure 3(d), the attention map of Imagic with Stable Diffusion is shown,
revealing no discernible organized attention pattern.

The results confirm that our composite embedding approach, utilizing a highly
personalized embedding and a semantic embedding for content, successfully
separates the personalization and manipulation aspects. This approach
outperforms existing methods which do not separate these two aspects, as
demonstrated by our experimental results.

##  6 Conclusion

We present a highly personalized text-to-image generation method using Stable
Diffusion, which is simple yet powerful. With just a single image, our method
can produce highly personalized text tokens, resulting in superior performance
in maintaining the identity of the subject. Furthermore, our approach does not
necessitate model fine-tuning or complex loss functions. These properties
enable us to manipulate images quickly and easily using a simple optimization
process that takes just three minutes. Additionally, we have demonstrated the
remarkable capabilities of our method by demonstrating image editing results
in three areas: motion, background, and texture.

## References

  * [1] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [2] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [3] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [4] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  Advances in Neural Information Processing Systems, 34:19822–19835, 2021.
  * [5] Hao Dong, Simiao Yu, Chao Wu, and Yike Guo.  Semantic image synthesis via adversarial learning.  In Proceedings of the IEEE international conference on computer vision, pages 5706–5714, 2017.
  * [6] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
  * [7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [8] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  * [9] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  Advances in Neural Information Processing Systems, 33:9841–9850, 2020.
  * [10] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [11] Tobias Hinz, Stefan Heinrich, and Stefan Wermter.  Semantic object accuracy for generative text-to-image synthesis.  IEEE transactions on pattern analysis and machine intelligence, 44(3):1552–1565, 2020.
  * [12] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [13] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [14] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [15] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [16] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18062–18071, 2022.
  * [17] Gihyun Kwon and Jong Chul Ye.  Diffusion-based image translation using disentangled style and content representation.  arXiv preprint arXiv:2209.15264, 2022.
  * [18] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr.  Controllable text-to-image generation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [19] Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao.  Object-driven text-to-image synthesis via adversarial training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12174–12182, 2019.
  * [20] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell.  More control for free! image synthesis with semantic diffusion guidance.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 289–299, 2023.
  * [21] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [22] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.  Text-adaptive generative adversarial networks: manipulating images with natural language.  Advances in neural information processing systems, 31, 2018.
  * [23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [24] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [25] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  arXiv preprint arXiv:2302.03027, 2023.
  * [26] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085–2094, October 2021.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [28] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [29] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [31] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [32] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [33] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon.  Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models.  arXiv preprint arXiv:2104.05358, 2021.
  * [34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv preprint arXiv:2210.08402, 2022.
  * [35] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9243–9252, 2020.
  * [36] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  IEEE transactions on pattern analysis and machine intelligence, 44(4):2004–2018, 2020.
  * [37] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [38] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [39] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [40] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji.  Improving text-to-image synthesis using contrastive learning.  arXiv preprint arXiv:2107.02423, 2021.
  * [41] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In International conference on machine learning, pages 7354–7363. PMLR, 2019.
  * [42] Zizhao Zhang, Yuanpu Xie, and Lin Yang.  Photographic text-to-image synthesis with a hierarchically-nested adversarial network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6199–6208, 2018.

Supplementary Material

##  Appendix A Details of experimental setting

#### Implementation details

For optimizing the personalized embedding with Stable Diffusion model, we set
the size of source image to $512\times 512$. We use the highly personalized
(HiPer) embedding which is optimized during 1000 steps, with the learning rate
of $5\times{10}^{-3}$.

#### The number of HiPer embedding tokens

The default number of HiPer embedding tokens was 5. In cases where the
generated image is intended to more closely follow the original image, the
number of tokens can be increased.

#### Calibration of HiPer embedding

Before concatenating the HiPer embedding ${\bm{e}}_{hper}$ with target
embedding ${\bm{e}}_{tgt}^{\prime}$, the HiPer embedding vector is calibrated
by multiplying it with 0.8. This value was obtained through experimentation
and is a constant value throughout the experiment.

##  Appendix B Evaluation

###  B.1 Quantitative analysis

For quantitative comparison, we conducted a user study and calculated CLIP
scores. The details of the user study will be described later. To measure the
preservation of identity, we calculated the distance between the output images
and the source image. In addition, we evaluated the semantic alignment of the
output images with the target prompts by computing the distance between them
in the CLIP embedding space. As illustrated in Table 1, which displays the
average scores from the experiments in Figs. 4 and 8, our method achieved best
score in the user study on semantic alignment. While DreamBooth ranked first
in the user study for identity preservation, its performance in manipulation
was inferior. Its strong focus on identity preservation resulted in inadequate
editing towards the target prompt. As shown in Figure 4, the bird in the first
row was not transformed enough to align with target text prompt. Meanwhile,
Imagic with Stable diffusion showed the good performance in manipulation (cf.
CLIP score). However, its output images could not preserve the identities.
Figure 8 demonstrates that the shape of the purse in the second row and the
dog in the last row differ from the source image.

Methods | User study | CLIP score | Train time
---|---|---|---
w/ Stable Diffusion | Semantic $\uparrow$ | Identity $\uparrow$ | Text $\uparrow$ | Image $\uparrow$
Imagic w/ Stable Diffusion |  3.731 |  3.251 |  0.2148 |  0.8223 |  14.08 min
DreamBooth |  3.099 |  4.503 |  0.1955 |  0.8520 |  14.67 min
Textual Inversion |  3.070 |  3.567 |  0.1633 |  0.8303 |  42.8 min
HiPer |  4.520 |  4.099 |  0.2047 |  0.8443 |  3 min

Table 1: Quantitative results on comparative studies. The bold text denotes
the best score, and the underline indicates the second-best score. All of the
models were implemented using the Stable Diffusion model.

Figure 8: The images generated from the same text prompt using our proposed
method and other methods are shown. As illustrated, our method consistently
produces images that are visually more aligned with the given text prompts,
and are more similar to the source images in terms of identity. The images
generated by the other methods lack detail and often produce unrealistic
results.

Figure 9: Ablation study on the source embeddings.

#### User Study

We conducted a user study to evaluate the performance of our personalized
image generation methods, Imagic [14], Dreambooth [31], and Textual Inversion
[7], on various tasks such as motion, background, and texture changes. To
compare our methods, we used a single source image and conducted the
experiments on publicly available Stable Diffusion models. We obtained
feedback from 20 participants and asked them to rank the resulting images from
other models based on their similarity to the source image (Identity) and
their alignment with the given text (Semantic alignment). To gather
comprehensive feedback, we utilized a personalized opinion evaluation system
through Google Forms, which included a scoring scale of 1-5, with 1
representing the lowest score and 5 representing the highest score.
Respondents had the option of selecting one of five different scores for each
inquiry: 1-very low, 2-low, 3-middle, 4-high, 5-very high.

Figure 10: Ablation study results on the role of ${\bm{e}}_{hper}$. While the
images generated from ${\bm{e}}_{tgt}$ (i.e. Stable Diffusion) cannot preserve
the guitar’s identity, guitars in the images generated from ${\bm{e}}_{hper}$
show same identity as the source image.

Figure 11: The facial editing results are presented. As shown, our method is
capable of generating images with high-quality facial editing, such as
changing the motion, adding glasses, or modifying the facial expression while
retaining the identity.

Figure 12: Personalization results with random seeds. The majority of output
images included the dog with its leash.

Figure 13: The qualitative comparison results with one source image are
presented. As can be seen, our method consistently produces images that are
visually more aligned with the given text prompts and are more similar to the
source image in terms of identity.

##  Appendix C Additional experiments

###  C.1 Ablation study on source embedding ${\bm{e}}_{src}$

We conducted an ablation study using different source embeddings to
investigate the role of ${\bm{e}}^{\prime}_{src}$ during the optimizing phase.
As shown in Figure 9, we found that the identity of the dog in the source
image was well-preserved across all source embeddings. However, the output
images generated from the prompt “a standing dog” demonstrated superior
performance compared to the other embeddings. In some cases, the dogs in the
first row (i.e. “a dog”) did not resemble the dog in the source image, and the
dogs in the third row (i.e. “a jumping dog”) did not appear to be jumping.
Additionally, human-like artifacts were present in the third and last rows.
Based on these observations, we decided to use more descriptive source text
such as “a standing dog” for training ${\bm{e}}_{hper}$ as the baseline.

###  C.2 Ablation study on personalized embedding

In Figure 10, we examined the function of ${\bm{e}}_{hper}$. The images
generated solely using ${\bm{e}}_{tgt}$, which corresponds to the Stable
Diffusion Model, were not able to preserve the identity of the guitar. On the
other hand, the guitars in the images generated from ${\bm{e}}_{hper}$
retained the same identity as the source image.

###  C.3 Additional Results

Our method using HiPer embedding could manipulate human facial images as
illustrated in Figure 11. Using only a single facial image, we could generate
diverse manipulated images. The hair style and facial features of the person
in the source image were accurately reflected in the resulting image. Also, it
appears that the images are well generated in accordance with the given text
prompt.

Furthermore, as demonstrated in Figure 12, we achieved high-quality edited
images across multiple random seeds while maintaining the identity of the dog
and aligning with the target prompts. It is worth noting that the majority of
output images included the dog with its leash.

For further comparison, we generated comparative results for additional result
images (Figure 8 and 13). From the manipulated images, which are obtained with
the Stable Diffusion, it is confirmed that our model produces better results
than other models in maintaining the appearance of objects in the source image
and reflecting the given text in the resulting image.

##  Appendix D Limitations

Although HiPer can successfully manipulate images using a wide range of
prompts, such as motion, background, and artistic styles, it may encounter
difficulties when attempting to manipulate images using certain types of
prompts, such as counting (Figure 14(a)). Although the number of new subjects
(“two cats”) from the source prompt is incorporated into the output image, the
number of the original subject (“two baskets” or “many baskets”) in the source
image could not be altered. Also, while the object of generated image with
HiPer embedding follows the shape of source image’s desired object, there were
instances where the color did not match accurately (Figure 14(b)). If the
characteristics of color are not clear or if the color is not specified in the
target text prompt, the resulting image may differ in color from the source
image. Another drawback of our approach is that it proved to be ineffective in
the production of complex artificial products (Figure 14(c)). On the other
hand, our approach demonstrates superior performance when applied to natural
images such as those featuring dogs, birds, or various scenes.

Our model preserves the identity of the source image, but it does not
currently achieve a level where the generated image appears natural. Since our
focus is on preserving the identity of the source image, there are unnatural
aspects in the outside of the personalization. This is because our image
manipulation with the backbone of Stable Diffusion Model, which is publicly
available, has the limitation of generating realistic image.

##  Appendix E Social impact

Our method enables manipulation of images using arbitrary prompts, therefore,
it is important to exercise caution when selecting images such as those
depicting human faces or bodies. Moreover, text-to-image manipulation method
may have the negative potential to create convincing fake images that could
mislead people. Therefore, we recommend that users use the tool appropriately
and carefully.

Figure 14: Failure cases of (a) counting, (b) color, and (c) complex
artificial products.

##  Appendix F Acknowledgements

We would like to express our gratitude to Shiran Zada and the other authors
from Imagic for their help with the official Imagic results.

##  Appendix G Source text prompt during training

|  | source text
---|---|---
Figure 1 | left | a rabbit
| right | a standing dog
Figure 4 | 1st row | a bird
| 2nd row | a standing dog
| 3rd row | a teddy
| 4th row | a standing dog
| 5th row | a bear
Figure 5 | 1st row | a car
| 2nd row | a standing dog
| 3rd row | a purse
| 4th row | a cat
Figure 6 |  | a doctor
Figure 7 |  | a standing dog
Figure 8 | 1st row | a cup
Figure 9 |  | a standing dog
Figure 10 |  | a guitar
Figure 11 | top | a woman
| bottom | a man
Figure 12 |  | a standing dog
Figure 13 |  | a rabbit
Figure 14 | 1st row | a basket
| 2nd row | a flower
| 3rd row | a kitten-toy

Table 2: Source text prompts for generating the figures.