#  PALP: Prompt Aligned Personalization of Text-to-Image Models

Moab Arar111 1,2, Andrey Voynov2, Amir Hertz2, Omri Avrahami111 2,4,
Shlomi Fruchter1, Yael Pritch 2, Daniel Cohen-Or111 1,2, Ariel Shamir111 2,3
1 Tel-Aviv University, 2 Google Research 3 Reichman University, 4 The Hebrew
University of Jerusalem

###### Abstract

Content creators often aim to create personalized images using personal
subjects that go beyond the capabilities of conventional text-to-image models.
Additionally, they may want the resulting image to encompass a specific
location, style, ambiance, and more. Existing personalization methods may
compromise personalization ability or the alignment to complex textual
prompts. This trade-off can impede the fulfillment of user prompts and subject
fidelity. We propose a new approach focusing on personalization methods for a
_single_ prompt to address this issue. We term our approach prompt-aligned
personalization. While this may seem restrictive, our method excels in
improving text alignment, enabling the creation of images with complex and
intricate prompts, which may pose a challenge for current techniques. In
particular, our method keeps the personalized model aligned with a target
prompt using an additional score distillation sampling term. We demonstrate
the versatility of our method in multi- and single-shot settings and further
show that it can compose multiple subjects or use inspiration from reference
images, such as artworks. We compare our approach quantitatively and
qualitatively with existing baselines and state-of-the-art techniques.

Figure 1: Prompt aligned personalization allow rich and complex scene
generation, including all elements of a condition prompt (right).

11footnotetext: Work done while working at Google.22footnotetext: Project page
available at https://prompt-aligned.github.io/.

##  1 Introduction

Text-to-image models have shown exceptional abilities to generate a diversity
of images in various settings (place, time, style, and appearances), such as
”a sketch of Paris on a rainy day” or ”a Manga drawing of a teddy bear at
night” [36, 41]. Recently, personalization methods even allow one to include
specific subjects (objects, animals, or people) into the generated images [15,
39]. In practice, however, such models are difficult to control and may
require significant prompt engineering and re-sampling to create the specific
image one has in mind. It is even more acute with personalized models, where
it is challenging to include the personal item or character in the image and
simultaneously fulfill the textual prompt describing the content and style.
This work proposes a method for better personalization and prompt alignment,
especially suited for complex prompts.

A key ingredient in personalization methods is fine-tuning pre-trained text-
to-image models on a small set of personal images while relying on heavy
regularization to maintain the model’s capacity. Doing so will preserve the
model’s prior knowledge and allow the user to synthesize images with various
prompts; however, it impairs capturing identifying features of the target
subject. On the other hand, persisting on identification accuracy can hinder
the prompt-alignment capabilities. Generally speaking, the trade-off between
identity preservation and prompt alignment is a core challenge in
personalization methods (see Fig. 2).

Content creators and AI artists frequently have a clear idea of the prompt
they wish to utilize. It may involve stylization and other factors that
current personalization methods struggle to maintain. Therefore, we take a
different approach by focusing on excelling with a single prompt rather than
offering a general-purpose method intended to perform well with a wide range
of prompts. This approach enables both (i) learning the unique features of the
subject from a few or even a single input image and (ii) generating richer
scenes that are better aligned with the user’s desired prompt (see Fig. 1).

Our work is based on the premise that existing models possess knowledge of all
elements within the target prompt except the new personal subject.
Consequently, we leverage the pre-trained model’s prior knowledge to prevent
personalized models from losing their understanding of the target prompt. In
particular, since we know the target prompt during training, we show how to
incorporate score-distillation guidance [34] to constrain the personalized
model’s prediction to stay aligned with the pre-trained one. Therefore, we
introduce a framework comprising two components: personalization, which
teaches the model about our new subject, and prompt alignment which prevents
it from forgetting elements included in the target prompt.

Our approach liberates content creators from constraints associated with
specific prompts, unleashing the full potential of text-to-image models. We
evaluate our method qualitatively and quantitatively. We show superior results
compared with the baselines in multi- and single-shot settings, all without
pre-training on large-scale data [2, 16], which can be difficult for certain
domains. Finally, we show that our method can accommodate multi-subject
personalization with minor modification and offer new applications such as
drawing inspiration from a single artistic painting, and not just text (see
Figure 3).

Figure 2: Previous personalization methods struggle with complex prompts
(e.g., “A sketch inspired by Vitruvian man”) presenting a trade-off between
prompt-alignment and subject-fidelity. Our method, optimizes for both, without
compromising either.

##  2 Related work

#### Text-to-image synthesis

has marked an unprecedented progress in recent years [36, 38, 30, 41, 13],
mostly due to large-scale training on data like LAION-400m [42]. Our approach
uses pre-trained diffusion models [21] to extend their understanding to new
subjects. We use the publicly available Stable-Diffusion model [38] for most
of our experiments since baseline models are mostly open-source on SD. We
further verify our method on a larger latent diffusion model variant [38].

#### Text-based editing

methods rely on contrastive multi-modal models like CLIP [35] as an interface
to guide local and global edits [32, 5, 3, 14, 6]. Recently, Prompt-to-Prompt
(P2P) [19] was proposed as a way to edit and manipulate _generated_ images by
editing the attention maps in the cross-attention layers of a pre-trained
text-to-image model. Later, Mokady et al. [29] extended P2P for real images by
encoding them into the null-conditioning space of classifier-free guidance
[20]. InstructPix2Pix [7] uses an instruction-guided image-to-image
translation network trained on synthetic data. Others preserve image-structure
by using reference attention maps [31] or features extracted using DDIM [44]
inversion [47]. Imagic [27] starts from a target prompt, finds text embedding
to reconstruct an input image, and later interpolates between the two to
achieve the final edit. UniTune [48], on the other hand, performs the
interpolation in pixel space during the denoising backward process. In our
work, we focus on the ability to generate images depicting a given subject,
which may not necessarily maintain the global structure of an input image.

#### Early personalization methods

like Textual Inversion [15] and DreamBooth [39] tune pre-trained text-2-image
models to represent new subjects, either by finding a new soft word-embedding
[15] or calibrating model-weights [39] with existing words to represent the
newly added subject. Later methods improved memory requirements of previous
methods using Low-Rank updates [22, 28, 46, 40] or compact-parameter space
[17]. In another axis, NeTI [1] and $P+$ [51] extend TI [15] using more tokens
to capture the subject-identifying features better. Personalization can also
be used for other tasks. ReVersion [23] showed how to learn relational
features from reference images, and Vinker et al. [50] used personalization to
decompose and visualize concepts at different abstraction levels. Chefer et.
al [9] propose an interpretability method for text-to-image models by
decomposing concepts into interpretable tokens. Another line-of-works pre-
train encoders on large-data for near-instant, single-shot adaptation [16, 2,
55, 49, 10, 56, 53]. Single-image personalization has also been addressed in
Avrahami et al. [4], where the authors use segmentation masks to personalize a
model on different subjects. In our work, we focus on prompt alignment, and
any of the previous personalization methods may be replaced by our baseline
personalization method.

#### Score Distillation Sampling (SDS)

emerged as a technique for leveraging 2D-diffusion models priors [38, 41] for
3D-generation from textual input. Soon, this technique found way to different
applications like SVG generation [25, 24], image-editing [18], and more [45].
Other variant of SDS [34] aim to improve the image-generation quality of SDS,
which suffers from over-saturation and blurriness [26, 52]. In our approach,
we propose a framework that leverages score sampling to maintain alignment
with the target prompt. Alternative score-sampling techniques can be
considered to further boost the text-alignment of the personalized method.

#### Text-to-image alignment

methods address text-related issues that arise in base diffusion generative
models. These issues include neglecting specific text parts, attribute mixing,
and more. Previous methods address these issues through attention-map re-
weighting[12, 54, 33], latent-optimization [8, 37], or re-training with
additional data [43]. However, none of these methods address prompt alignment
of personalization methods, instead, they aim to enhance the base models in
generating text-aligned images.

Figure 3: PALP for multi-subject personalization achieves coherent and prompt-
aligned results. Our method works when the subject has only one image (e.g.,
the ”Wanderer above the Sea of Fog” artwork by Caspar David Friedrich).

##  3 Preliminaries

Figure 4: Method overview. We propose a framework consisting of a
personalization path (left) and a prompt-alignment branch (right) applied
simultaneously in the same training step. We achieve personalization by
finetuning the pre-trained model using a simple reconstruction loss to denoise
the new subject $S$. To keep the model aligned with the target prompt, we
additionally use score sampling to pivot the prediction towards the direction
of the target prompt $y$, e.g., ”A sketch of a cat.” In this example, when
personalization and text alignment are optimized simultaneously, the network
learns to denoise the subject towards a ”sketch” like representation. Finally,
our method does not induce a significant memory overhead due to the efficient
estimation of the score function, following [34].

#### Generative Diffusion Models.

Diffusion models perform a backward diffusion process to generate an image. In
this process, the diffusion model $G$ progressively denoises a source
$x_{T}\sim\mathcal{N}(0,1)$ to produce a real sample $x_{0}$ from an
underlying data distribution $p(X)$. At each timestep $t\in\\{0,1,...,T\\}$,
the model predicts a noise $\hat{\epsilon}=G\left(x_{t},t,y;\theta\right)$
conditioned on a prompt $y$ and timestep $t$. The generative model is trained
by maximizing the evidence lower bound (ELBO) using a denoising score matching
objective [21]:

$\mathcal{L}(\textbf{x},y)=\mathbb{E}_{t\sim[0,T],\epsilon\sim\mathcal{N}(0,1)}\left[\lVert
G\left(x_{t},t,y;\theta\right)-\epsilon\rVert_{2}^{2}\right].$ (1)

Here,
$x_{t}=\sqrt{\bar{\alpha}_{t}}\textbf{x}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$,
and x is a sample from the real-data distribution $p(X)$. Throughout this
section, we will write $G_{\theta}(x_{t},y)=G\left(x_{t},t,y;\theta\right)$.

#### Personalization.

Personalization methods fine-tune the diffusion model $G$ for a new target
subject $S$, by optimizing Eq. 1 on a small set of images representing $S$.
Textual-Inversion [15] optimizes new word embeddings to represent the new
subject. This is done by pairing a generic prompt with the placeholder $[V]$,
e.g., “A photo of [V]”, where $[V]$ is mapped to the newly-added word
embedding. DreamBooth [39] calibrates existing word-embeddings to represent
the personal subject $S$. This can be done by adjusting the model weights, or
using recent more efficient methods that employ a Low-Rank Adaptation (LoRA)
[22] on a smaller subset of weights [28, 40, 46].

##  4 Prompt Alignment Method

###  4.1 Overview

Our primary objective is to teach $G$ to generate images related to a new
subject $S$. However, unlike previous methods, we strongly emphasize achieving
optimal results for a _single textual prompt_ , denoted by $y$. For instance,
consider a scenario where the subject of interest is one’s cat, and the prompt
is $y=\text{``A sketch of [my cat] in Paris.''}$ In this case, we aim to
generate an image that faithfully represents the cat while incorporating all
prompt elements, including the sketchiness and the Parisian context.

The key idea is to optimize two objectives: personalization and prompt
alignment. For the first part, i.e., personalization, we fine-tune $G$ to
reconstruct $S$ from noisy samples. One option for improving prompt alignment
would be using our target prompt $y$ as a condition when tuning $G$ on $S$.
However, this results in sub-optimal outcome since we have no images depicting
$y$ (e.g., we have no sketch images of our cat, nor photos of it in Paris).
Instead, we push $G$’s noise prediction towards the target prompt $y$. In
particular, we steer $G$’s estimation of $S$ towards the distribution density
$p(x|y)$ using score-sampling methods (See Fig. 4). By employing both
objectives simultaneously, we achieve two things: (1) the ability to generate
the target subject $S$ via the backward-diffusion process (personalization)
while (2) ensuring the noise-predictions are aligned with text $y$. We next
explain our method in detail.

###  4.2 Personalization

We follow previous methods [15, 39, 28, 46], and fine-tune $G$ on a small set
of images representing $S$, which can be as small as a single photo. We update
$G$’s weights using LoRA method [22], where we only update a subset of the
network’s weights $\theta_{\text{LoRA}}\subseteq\theta$, namely, the self- and
cross- attention layers. Furthermore, we optimize a new word embedding $[V]$
to represent $S$.

###  4.3 Prompt-Aligned Score Sampling

Long fine-tuning on a small image set could cause the model to overfit. In
this case, the diffusion model predictions always steer the backward denoising
process towards one of the training images, regardless of any conditional
prompt. Indeed, we observed that overfitted models could estimate the inputs
from noisy samples, even in a single denoising step.

In Fig. 5, we visualize the model prediction by analyzing its estimate of
$x_{0}$, the real-sample image. The real-sample estimation $\hat{x}_{0}$ is
derived from the model’s noise prediction via:

$\hat{x}_{0}=\frac{x_{t}-\sqrt{1-\bar{\alpha}_{t}}G_{\theta}(x_{t},y)}{\sqrt{\bar{\alpha}_{t}}}.$
(2)

.

(a) Input

(b) base model

(c) Over-fitted

(d) Prompt aligned

Figure 5: Visualization of $\hat{x}_{0}$. We visualize the model estimation of
$\hat{x}_{0}$ given a pure-noise and the prompt ”A sketch of [V].” The base
model (b) is not personalized to the target subject and predicts mainly the
”Sketch” appearance. Personalization methods (c) tend to overfit the input
image where many image elements, including the background and the subject
colors are restored, suggesting the model does not consider the prompt
condition. Prompt aligned personalization (c) maintains the sketchiness and
does not overfit (see cat-like shape).

As can be seen from Fig. 5, the pre-trained model (before personalization)
steers the image towards a sketch-like image, where the estimate $\hat{x}_{0}$
has a white background, but the subject details are missing. Apply
personalization without PALP overfits, where elements from the training
images, like the background, are more dominant, and sketchiness fades away,
suggesting miss-alignment with the prompt “A sketch” and over-fitting to the
input image. Using PALP, the model prediction steers the backward-denoising
process towards a sketch-like image, while staying personalized, where a cat
like shape is restored.

Our key idea is to encourage the model’s denoising prediction towards the
target prompt. Alternatively, we push the model’s estimation of the real-
sample, denoted by $\hat{x}_{0}$, to be aligned with the prompt $y$.

In our example, where the target prompt is ”A sketch of [V],” we push the
model prediction toward a sketch and prevent overfitting a specific image
background. Together with the personalization loss, this will encourage the
model to focus on capturing [V]’s identifying features rather than
reconstructing the background and other distracting features.

To achieve the latter, we use Score Distillation Sampling (SDS) techniques
[34]. In particular, given a clean target prompt $y^{c}$, that doesn’t contain
the placeholder ”[V]”. Then we use score sampling to optimize the loss:

$\mathcal{L}(\hat{x}_{0},y^{c}),$ (3)

from Eq. 1. Note, we use the pre-trained network weights to evaluate Eq. 3,
and omit all learned placeholder (e.g., $[V]$). Therefore, this ensures that
by minimizing the loss, we will stay aligned with the textual prompt $y$ since
the pre-trained model possesses all knowledge about the prompt elements.

Poole et al. [34] found an elegant and efficient approximation of the score
function using:

$\nabla\mathcal{L}_{SDS}(\textbf{x})=\tilde{w}(t)\left(G^{\alpha}\left(x_{t},t,y^{c};\theta\right)-\epsilon\right)\frac{\partial\textbf{x}}{\partial\phi},$
(4)

where $\phi$ are the weights controlling the appearance of x, and
$\tilde{w}(t)$ is a weighting function. Here $G^{\alpha}$ denotes the
classifier-free guidance prediction, which is an extrapolation of the
conditional and unconditioned ($y=\emptyset$) noise prediction. The scalar
$\alpha\in\mathbb{R}^{+}$ controls the extrapolation via:

$G^{\alpha}\left(x_{t},t,y^{c};\theta\right)=(1-\alpha)\cdot
G_{\theta}\left(x_{t},\emptyset\right)+\alpha\cdot
G_{\theta}\left(x_{t},y^{c}\right).$ (5)

In our case, $\textbf{x}=\hat{x}_{0}$, and it is derived from $G$’s noise
prediction as per Eq. 2 . Therefore, the appearance of $\hat{x}_{0}$ is
directly controlled by the LoRA weights and $[V]$.

###  4.4 Avoiding Over-saturation and Mode Collapse

Guidance by SDS produces less diverse and over-saturated results. Alternative
implementations like [52, 26] improved diversity, but the overall
personalization is still affected (see Appendix A). We argue that the reason
behind this is that the loss in Eq. 4 pushes the prediction $\hat{x}_{0}$
towards the center of distribution of $p(x|y^{c})$, and since the clean target
prompt $y^{c}$ contains only the subject class, i.e., “A sketch of a cat”,
then as a result, the loss will encourage $\hat{x}_{0}$ towards the center of
distribution of a general cat, not our one.

Instead, we found the Delta Denoising Score (DDS) [18] variant to work better.
In particular, we use the residual direction between the personalized model’s
prediction $G^{\beta}_{\theta_{LoRA}}(\hat{x}_{t},y_{P})$, and the pre-trained
one $G^{\alpha}_{\theta}(\hat{x}_{t},y^{c})$, where $\alpha$ and $\beta$ are
their guidance scales, respectively. Here $y_{P}$ is the personalization
prompt, i.e., ”A photo of [V],” and $y_{c}$ is a clean prompt, e.g., ”A sketch
of a cat.” The score can then be estimated by:

$\nabla\mathcal{L}_{PALP}=\tilde{w}(t)(G^{\alpha}_{\theta}(\hat{x}_{t},y^{c})-G^{\beta}_{\theta_{LoRA}}(\hat{x}_{t},y_{P}))\frac{\partial\hat{x}_{0}}{\partial\theta_{\text{LoRA}}},$
(6)

which perturbs the denoising prediction towards the target prompt (see right
part of Fig. 4). Our experiments found imbalanced guidance scales, i.e.,
$\alpha>\beta$, to perform better. We have also considered two variant
implementations: (1) in the first, we use the same noise for the two branches,
i.e., the text-alignment and personalization branches, and (2) we use two
i.i.d noise samples for the branches. Using the same noise achieves better
text alignment than the latter variant.

#### On the Computational Complexity of PALP:

the gradient in Eq. 6, is proportional to the personalization gradient with
the scaling:

$\frac{\partial\hat{x}_{0}}{\partial\theta_{\text{LoRA}}}\propto-\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\nabla
G_{\theta_{LoRA}}(x_{t},y_{P}).$ (7)

and since the gradient $\nabla G_{\theta_{LoRA}}(x_{t},y_{P})$ is also
calculated as part of the derivative of Eq. 1, then, we do not need to back-
propagate through the text-to-image model through the prompt-alignment branch.
This may be useful to use guidance from bigger models to boost smaller model’s
personalization performance. Finally, we also note that the scaling term is
very high for large $t$-values. Re-scaling the gradient back by
$\sqrt{\bar{\alpha}_{t}}/\sqrt{1-\bar{\alpha}_{t}}$ ensures uniform gradient
update across all timesteps and improves numerical stability.

##  5 Results

Input | Ours |  | TI+DB [15, 39] | CD [28] |  $P+$ [51] | NeTI [1]
---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Table 1: Qualitative comparison in multi-shot setting. Our method achieves
state-of-the-art results on complex prompts, better-preserving identity, and
prompt alignment. For TI [15]+DB [39], we use the same seed to generate our
results, emphasizing the gain achieved by incorporating the prompt alignment
path. For other baselines, we chose the best two out of eight samples.

#### Experimental setup:

We use StableDiffusion (SD)-v1.4 [38] for ablation and comparison purposes, as
many official implementations of state-of-the-art methods are available in
SD-v1.4. We further validate our method with larger text-to-image models (see
Appendix B). Complete experimental configuration, including learning rate,
number of steps, and batch size, appear in Appendix C.

#### Evaluation metric:

for evaluation, we follow previous works [39, 15] and use CLIP-score [35] to
measure alignment with the target (clean) prompt $y^{c}_{t}$ (i.e., does not
have the placeholder [V]). For subject preservation, we also use CLIP feature
similarity between the input and generated images with the target prompt. We
use VIT-B32 [11] trained by OpenAI on their proprietary data for both metrics.
This ensures that the underlying CLIP used by SD-v1.4 differs from the one
used for evaluation, which could compromise the validity of the reported
metric.

#### Dataset:

for multi-shot setting, we use data collected by previous methods [15, 28],
with different subjects like animals, toys, personal items, and buildings. For
those subjects, checkpoints of previous methods exist, allowing a fair
comparison.

###  5.1 Ablation studies

Figure 6: Ablation study. We report image alignment (left) and text alignment
(right) as a function of the number of fine-tuning steps. The base model
results are the pre-trained model’s performance when using the class word to
represent the target subject.

For ablation, we start with TI [15] and DB [39] as our baseline
personalization method and gradually add different components contributing to
our final method.

_Early stopping:_ We begin by considering early stopping as a way to control
the text-alignment. The lower the number of iterations, the less likely we are
to hurt the model’s prior knowledge. However, this comes at the cost of
subject fidelity, evident from Fig. 6. The longer we tune the model on the
target subject, the more we risk overfitting the training set.

_Adding SDS guidance:_ improves text alignment, yet it severely harms the
subject fidelity, and image diversity is substantially reduced (see Appendix
A). Alternative distillation sampling guidance [26] improves on top of SDS;
however, since the distillation sampling guides the personalization
optimization towards the center of distribution of the subject class, it still
produces less favorable results.

_Replacing SDS with PALP guidance:_ improves text alignment by a considerable
margin and maintains high fidelity to the subject $S$. We consider two
variants: one where we use the same noise for personalization loss or sample a
new one from the normal distribution. Interestingly, using the same noise
helps with prompt alignment. Furthermore, scaling the score sampling equation
Eq. 6 by $\sqrt{\bar{\alpha}_{t}}/\sqrt{1-\bar{\alpha}_{t}}$ further enhances
the performance.

###  5.2 Comparison with Existing Methods

| Text-Alignment $\uparrow$ | Image-Alignment $\uparrow$
---|---|---
Method | Style | Class | Ambiance 1 | Ambiance 2 | Target Prompt |
P+ | 0.244 | 0.257 | 0.217 | 0.218 | 0.308 | 0.673
NeTI | 0.235 | 0.264 | 0.22 | 0.214 | 0.310 | 0.695
TI+DB | 0.237 | 0.279 | 0.22 | 0.216 | 0.319 | 0.716
Ours | 0.245 | 0.272 | 0.23 | 0.224 | 0.340 | 0.681

Table 2: Comparisons to prior work. Our method presents better prompt-
alignment, without hindering personalization.

Method | Text-Alignment $\uparrow$ | Personalization $\uparrow$
---|---|---
P+ | 68.5 % | 61.2 %
NeTI | 63.2 % | 70.3 %
TI+DB | 73.3 % | 60.4 %
Ours | 91.2 % | 72.1 %

Table 3: User Study results. For text alignment, we report the percent of elements from the prompt that users found in the generated image. For personalization, users rated the similarity of the subject $S$ and the main subject in the generated image. Input |  | Ours |  | Best of previous works
---|---|---|---|---
|  |  |  |  |  |  |  |
|  |  ”3D Render as a chef” |  ”Vector art wearing a hat” |  ”A painting by Da Vinci” |  |  ”3D Render as a chef” |  ”Vector art wearing a hat” |  ”A painting by Da Vinci”
|  |  |  |  |  |  |  |
|  |  ”Anime drawing” |  ”Pop art” |  ”A Caricature” |  |  ”Anime drawing” |  ”Pop art” |  ”A Caricature”
|  |  |  |  |  |  |  |
|  |  ”3D Render as a chef” |  ”Vector art wearing a hat” |  ”A painting by Da Vinci” |  |  ”3D Render as a chef” |  ”Vector art wearing a hat” |  ”A painting by Da Vinci”
|  |  |  |  |  |  |  |
|  |  ”Anime drawing” |  ”Pop art” |  ”A Caricature” |  |  ”Anime drawing” |  ”Pop art” |  ”A Caricature”
Table 4: Qualitative comparison against ProFusion [56], IP-Adapter [55] , E4T
[16], and Face0 [49]. On the left, we show the results of our method on two
individuals using a _single_ image with multiple prompts. To meet space
requirements, we report a single result among all previous methods, based on
our preference, for a given subject and prompt. Full comparison appears in
Tab. 9 Tab. 8.

We compare our method against multi-shot methods, including CustomDiffusion
[28], $P+$ [51], and NeTI [1]. We further compare against TI [15] and DB [39]
using our implementation, which should also highlight the gain we achieve by
incorporating our framework with existing personalization methods. Our
evaluation set contains ten different complex prompts that include at least
four different elements, including style-change (e.g., ”sketch of”, ”anime
drawing of”), time or a place (e.g., ”in Paris”, ”at night”), color palettes
(e.g., ”warm,” ”vintage”). We asked users to count the number of elements that
appear in the image (text-alignment) and rate the similarity of the results to
the input subject (personalization, see Tab. 2).

Our method achieves the best text alignment while maintaining high image
alignment. TI+DB achieves the best image alignment. However, the reason for
this is because TI+DB is prone to over-fitting. Indeed, investigating each
element in the prompt, we find that the TI+DB achieves the best alignment with
the class prompt (e.g., ”A photo of a cat”) while being significantly worse in
the Style prompt (e.g., ”A sketch”). Our method has a slightly worse image
alignment since we expect appearance change for stylized prompts. We validate
this hypothesis with a user study and find that our method achieves the best
user preference in prompt alignment and personalization (see Tab. 3). Full
details on the user study appear in Appendix D.

###  5.3 Applications

#### Single-shot setting:

In a single-shot setting, we aim to personalize text-2-image models using a
single image. This setting is helpful for cases where only a single image
exists for our target subject (e.g., an old photo of a loved one). For this
setting, we qualitatively compare our method with encoder-based methods,
including IP-Adapter [55], ProFusion [56], Face0 [49], and E4T [16]. We use
portraits of two individuals and expect previous methods to generalize to our
selected images since all methods are pre-trained on human faces. Note that
E4T [16] and ProFusion [56] also perform test time optimization.

As seen from Tab. 4, our method is both prompt- and identity-aligned. Previous
methods, on the other hand, struggle more with identity preservation. We note
that optimization-based approaches [16, 56] are more identity-preserving, but
this comes at the cost of text alignment. Finally, our method achieves a
higher success rate, where the quality of the result is independent of the
chosen seed.

#### Multi-concept Personalization:

Our method accommodates multi-subject personalization via simple
modifications. Assume we want to compose two subjects, $S_{1}$ and $S_{2}$, in
a specific scene depicted by a given prompt $y$. To do so, we first allocate
two different placeholders, [V1] and [V2], to represent the target subjects
$S_{1}$ and $S_{2}$, respectively. During training, we randomly sample an
image from a set of images containing $S_{1}$ and $S_{2}$. We assign different
personalization prompts $y_{P}$ for each subject, e.g., ”A photo of [V1]” or
”A painting inspired by [V2]”, depending on the context. Then, we perform PALP
while using the target prompt in mind, e.g., ”A painting of [V1] inspired by
[V2]”. This allows composing different subjects into coherent scenes or using
a single artwork as a reference for generating art-inspired images. Results
appear in Fig. 3; further details and results appear in Appendix E.

##  6 Conclusions

We have introduced a novel personalization method that allows better prompt
alignment. Our approach involves fine-tuning a pre-trained model to learn a
given subject while employing score sampling to maintain alignment with the
target prompt. We achieve favorable results in both prompt- and subject-
alignment and push the boundary of personalization methods to handle complex
prompts, comprising multiple subjects, even when one subject has only a single
reference image.

While the resulting personalized model still generalizes for other prompts, we
must personalize the pre-trained model for different prompts to achieve
optimal results. For practical real-time use cases, there may be better
options. However, future directions employing prompt-aligned adapters could
result in instant time personalization for a specific prompt (e.g., for
sketches). Finally, our work will motivate future methods to excel on a subset
of prompts, allowing more specialized methods to achieve better and more
accurate results.

## References

  * Alaluf et al. [2023] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or.  A neural space-time representation for text-to-image personalization.  _CoRR_ , abs/2305.15391, 2023.
  * Arar et al. [2023] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models.  _CoRR_ , abs/2307.06925, 2023.
  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 18187–18197. IEEE, 2022.
  * Avrahami et al. [2023a] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  _CoRR_ , abs/2305.16311, 2023a.
  * Avrahami et al. [2023b] Omri Avrahami, Ohad Fried, and Dani Lischinski.  Blended latent diffusion.  _ACM Trans. Graph._ , 42(4):149:1–149:11, 2023b.
  * Bar-Tal et al. [2022] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV_ , pages 707–723. Springer, 2022.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 18392–18402. IEEE, 2023.
  * Chefer et al. [2023a] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10, 2023a.
  * Chefer et al. [2023b] Hila Chefer, Oran Lang, Mor Geva, Volodymyr Polosukhin, Assaf Shocher, Michal Irani, Inbar Mosseri, and Lior Wolf.  The hidden language of diffusion models.  _arXiv preprint arXiv:2306.00966_ , 2023b.
  * Chen et al. [2023] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W. Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  _CoRR_ , abs/2304.00186, 2023.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Feng et al. [2022] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  _arXiv preprint arXiv:2212.05032_ , 2022.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV_ , pages 89–106. Springer, 2022.
  * Gal et al. [2022] Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _ACM Trans. Graph._ , 41(4):141:1–141:13, 2022.
  * Gal et al. [2023a] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023a.
  * Gal et al. [2023b] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Trans. Graph._ , 42(4):150:1–150:13, 2023b.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  _arXiv preprint arXiv:2303.11305_ , 2023.
  * Hertz et al. [2023a] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or.  Delta denoising score.  _CoRR_ , abs/2304.07090, 2023a.
  * Hertz et al. [2023b] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross-attention control.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023b.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _CoRR_ , abs/2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Hu et al. [2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * Huang et al. [2023] Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin C. K. Chan, and Ziwei Liu.  Reversion: Diffusion-based relation inversion from images.  _CoRR_ , abs/2303.13495, 2023.
  * Iluz et al. [2023] Shir Iluz, Yael Vinker, Amir Hertz, Daniel Berio, Daniel Cohen-Or, and Ariel Shamir.  Word-as-image for semantic typography.  _ACM Trans. Graph._ , 42(4):151:1–151:11, 2023.
  * Jain et al. [2023] Ajay Jain, Amber Xie, and Pieter Abbeel.  Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 1911–1920. IEEE, 2023.
  * Katzir et al. [2023] Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski.  Noise-free score distillation.  _arXiv preprint arXiv:2310.17590_ , 2023.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 6007–6017. IEEE, 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 1931–1941. IEEE, 2023.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 6038–6047. IEEE, 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Parmar et al. [2023] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation, 2023.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 2065–2074. IEEE, 2021.
  * Phung et al. [2023] Quynh Phung, Songwei Ge, and Jia-Bin Huang.  Grounded text-to-image synthesis with attention refocusing.  _arXiv preprint arXiv:2306.05427_ , 2023.
  * Poole et al. [2023] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _CoRR_ , abs/2103.00020, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  _CoRR_ , abs/2102.12092, 2021.
  * Rassin et al. [2023] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik.  Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment, 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 10674–10685. IEEE, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 22500–22510. IEEE, 2023.
  * Ryu [2023] Simo Ryu.  Low-rank adaptation for fast text-to-image diffusion fine-tuning.  [https://github.com/cloneofsimo/lora](https://github.com/cloneofsimo/lora
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Segalis et al. [2023] Eyal Segalis, Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan.  A picture is worth a thousand words: Principled recaptioning improves image generation.  _arXiv preprint arXiv:2310.16656_ , 2023.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
  * Song et al. [2022] Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris N. Metaxas, and Ahmed Elgammal.  Diffusion guided domain adaptation of image generators.  _CoRR_ , abs/2212.04473, 2022.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023, Los Angeles, CA, USA, August 6-10, 2023_ , pages 12:1–12:11. ACM, 2023.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 1921–1930. IEEE, 2023.
  * Valevski et al. [2022] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  _arXiv preprint arXiv:2210.09477_ , 2022.
  * Valevski et al. [2023] Dani Valevski, Danny Wasserman, Yossi Matias, and Yaniv Leviathan.  Face0: Instantaneously conditioning a text-to-image model on a face.  _CoRR_ , abs/2306.06638, 2023.
  * Vinker et al. [2023] Yael Vinker, Andrey Voynov, Daniel Cohen-Or, and Ariel Shamir.  Concept decomposition for visual exploration and inspiration.  _CoRR_ , abs/2305.18203, 2023.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  P+: extended textual conditioning in text-to-image generation.  _CoRR_ , abs/2303.09522, 2023.
  * Wang et al. [2023] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _CoRR_ , abs/2305.16213, 2023.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Wu et al. [2023] Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang.  Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis.  _CoRR_ , abs/2304.03869, 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _CoRR_ , abs/2308.06721, 2023.
  * Zhou et al. [2023] Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu.  Enhancing detail preservation for customized text-to-image generation: A regularization-free approach.  _CoRR_ , abs/2305.13579, 2023.

##  Appendix A Over-saturation and mode-collapse in SDS

Figure 7: Additional results for Multi-subject personalization. The reference
images appear in Fig. 10.

Figure 8: User study sample questions. On the left, we show a sample question
in which the participants must count the prompt elements perceived from the
generated sample. On the right, users were asked to rate, from 1 to 5, the
similarity between the two main subjects, with one being the least similar.
Participants answers were aggregated and normalized.

(a) SDS [34]

(b) NFSD [26]

(c) Ours

Figure 9: Using SDS [34] guidance for prompt alignment produces over-saturated
results. Alternative improvements like NFSD [26] producess less-diverse
results. Our method produces diverse and prompt-aligned results. The input
prompt is ”A digital art of [V] on a field with lightning in the background.”

We use prompt-aligned score sampling Eq. 6 to guide the model towards the
target prompt $y$. We found this to be more effective than using SDS [34] or
improved versions of it like NFSD [26]. In particular, SDS and NFSD produce
over-saturated or less diverse results. We provide qualitative examples in
Fig. 9(c), and quantitative comparison appears in Sec. 5.1.

##  Appendix B Additional Results

We provide additional qualitative comparison for multi-shot setting, the
results appear in Tab. 7. Moreover, a full qualitative comparison and un-
curated results of the single-shot experiment appear in Tab. 8, Tab. 9, and
Fig. 12. For results obtained using a larger model, with improved text-encoder
and bigger capacity, see Tab. 5 and Tab. 6.

##  Appendix C Implementation Details

All our experiments, except those conducted using a larger model, were
conducted using TPU-v3, with a global batch size of 32. We use a learning rate
of 5e-5. For LoRA [22], we use a rank $r=32$ and only modify the self-and
cross-attention layers’ projection matrices. For most experiments, we use a
classifier free-guidance scale of $\alpha=15.0$ and $\alpha=7.5$; for
composition experiments, we use $\alpha=7.5$ and $\beta=1.0$. For quantitative
comparison, we fine-tuned the model for 500 steps. However, this may be sub-
optimal depending on the subject and prompt complexity.

##  Appendix D User-Study Details

In the user study, we asked 30 individuals to rate prompt alignment and
personalization performances of four methods, including $P+$ [51], NeTI [1],
TI [15]+DB [39], and our method. Our test set includes six different subjects
and ten prompts. We generated eight samples for each prompt and subject using
the four methods. Then, we randomly picked a single photo and asked the
participants to count the number of different prompt elements that appear in
the generated image. We asked the users to rate the similarity between the
main subject in the generated image and the sample. Then, we randomly divided
the questions into three forms and shuffled them between the participants.
Sample questions appear in Fig. 8.

##  Appendix E Multi-subject Personalization

Figure 10: Art inspired composition. Our method blends the target subject and
the reference paintings coherently. Further, we can produce diverse results by
slightly modifying the prompt. For example, we can generate an image of <my
toy> having a picnic, playing with a kite, or simply standing next to a lake
(see top-row third column).

(a) NeTI [1]

(b) w/o PALP Guidnace

(c) Ours

Figure 11: Art-inspired composition ablation. We consider two alternatives:
(a) using a pre-trained personalized model and prompting the artwork name, or
(b) jointly training on both subjects without PALP guidance. In both cases,
the results are sub-optimal, where our method achieves more coherent results.
Reference images appear in Fig. 10.

#### Multi-subject composition:

For multi-subject personalization, we use two placeholder $V_{1}$ and $V_{2}$
to represent the subject $S_{1}$ and $S_{2}$. We use $y_{P}$ = ”A photo of
[$V_{i}$]” as our personalization prompt. We use a descriptive prompt
describing the desired scene for the target clean prompt. For example, ”A 19th
century portrait of a [$C_{1}$] and [$C_{2}$]”, where $C_{1}$ and $C_{2}$
represent the class name of $S_{1}$ and $S_{2}$, respectively. We provide
additional results in Fig. 7.

#### Art-inspired composition:

For Art-inspired composition, we use ”A photo of [$V_{1}$]” to describe the
main subject, while ”An oil painting of [$V_{2}$]” is used for the artistic
image. The clean prompt used is ”An oil painting of [class name],” where
”[class name]” is the subject class (e.g., a cat or a toy). Furthermore,
adding a description to the clean prompt improves alignment, for example, ”An
oil painting of a cat sitting on a rock” for the” Wanderer above the Sea of
Fog” artwork by Caspar David Friedrich. Finally, at test time, we use the
target prompt ”An oil painting of [$V_{1}$] inspired by [$V_{2}$] painting” or
a similar variant, with possibly an additional description of the desired
scene. Results appear in Fig. 10. We further show that joint training is
insufficient for this task, nor is pre-training on $S$ prompting the artwork
or artist name. Both cases produce miss-aligned results (see Fig. 11).

As can be seen from Fig. 5, the pre-trained model (before personalization)
steers the image towards a sketch-like image, where the estimate $\hat{x}_{0}$
has a white background, but the subject details are missing. Apply
personalization without PALP overfits, where elements from the training
images, like the background, are more dominant, and sketchiness fades away,
suggesting miss-alignment with the prompt “A sketch” and over-fitting to the
input image. Using PALP, the model prediction steers the backward-denoising
process towards a sketch-like image, while staying personalized, where a cat
like shape is restored.

Figure 12: Un-curated samples for single-shot setting. The input image appear in Tab. 9. w/o PALP | w/ PALP
---|---
|
|
Table 5: Additional Results on Larger Text-to-Image Models. Reference images appear at the bottom. w/o PALP | w/ PALP
---|---
|
|
Table 6: Additional Results on Larger Text-to-Image Models. Reference images appear at the bottom. Input | Ours |  | TI+DB [15, 39] | CD [28] |  $P+$ [51] | NeTI [1]
---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Table 7: Additional qualitative comparison in multi-shot setting. Input | Prompt | Ours |  | E4T [16] | ProFusion [56] | IP-Adapter [55] | Face0 [49]
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Table 8: Single-shot setting - full qualitative comparison. Input | Prompt | Ours |  | E4T [16] | ProFusion [56] | IP-Adapter [55] | Face0 [49]
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
Table 9: Single-shot setting - full qualitative comparison.


The detection and normalization of diseases in biomedical texts are key biomedical natural language processing tasks. Disease names need not only be identified, but also normalized or linked to clinical taxonomies describing diseases such as . In this paper we describe deep learning methods that tackle both tasks. We train and test our methods on the known NCBI disease benchmark corpus. We propose to represent disease names by leveraging 's graphical structure together with the lexical information available in the taxonomy using graph embeddings. We also show that combining neural named entity recognition models with our graph-based entity linking methods via multitask learning leads to improved disease recognition in the NCBI corpus.

§ INTRODUCTION

In biomedical search applications, e.g. clinical search applications, it is of key importance to query not simply by keywords but rather by concept, viz., resolve names into their synonyms and return all matching documents. This is particularly true regarding diseases [4]. When we issue a query about “ovarian cancer”, we expect the system to return hits mentioning standard synonyms such as “ovarian neoplasms”. To achieve this it is necessary not only to perform named entity recognition (NER) to detect all entities mentioned in a document, but also to disambiguate them against databases of canonical names and synonyms, a task known as entity normalization or linking (EL).

Detecting and normalizing disease mentions are challenging tasks due to linguistic variability (e.g. abbreviations, morphological and orthographic variation, word order) [15, 23]. Interest in these tasks led to the creation of disease knowledge bases and annotated corpora. The Medical Subject Headings () taxonomy [17] is a repository of clinically-relevant terms covering (among others) a large range of standardized disease names, where standard names and known synonyms are organised hierarchically. The NCBI disease corpus [5] is a collection of abstracts with disease mentions manually resolved to or concepts.

Annotated resources allow the training of supervised learning algorithms. The first such system was [15], which used a learning-to-rank approach. More recently, systems based on deep learning have been proposed [16, 2, 25]. The state-of-the-art for NER employs bidirectional recurrent neural networks to parameterize a conditional random field [14]. Normalization is typically based on learning a similarity metric between disease mentions and their standard names in , where mentions and names are represented by a composition of word embeddings [16, 2]. This typically requires first identifying potential candidate diseases, for example by exploiting handcrafted rules [16] or bidirectional recurrent neural networks [6]. While [25] leverages in addition a coherence model that enforces a form of joint disambiguation. Crucially, previous work has consistently ignored the hierarchical structure of turning the taxonomy into a flat list of concepts.

In this paper we study an alternative approach to disease normalization. In particular, we acknowledge the hierarchical structure of the and employ graph encoders—both supervised and unsupervised—to learn representations of disease names (i.e. nodes). To make node representations easier to correlate to textual input, we propose to also exploit lexical information available in scope notes. Finally, given the similarities between NER and EL, we explore multitask learning <cit.> via a shared encoder. MTL has been traditionally applied in biomedical NLP to solve NER over several, though related corpora, covering different, but related entities [24], but has seldom been used to learn related tasks over the same corpus and entities. Our findings suggest that using lexicalized node embeddings improves EL performance, while MTL allows to transfer knowledge from (graph-embedding) EL to NER, with performance topping comparable approaches for the NCBI corpus [8, 24].

§ DATASETS AND METHODS

Identification of APC2, a homologue of the $\red{[}$adenomatous polyposis coli tumour$\red{]}_{\red{\text{D011125}}}$ suppressor.

MeSH Heading    Adenomatous Polyposis Coli

Scope Note       A polyposis syndrome due to an autosomal dominant mutation of the APC

genes (GENES, APC) on CHROMOSOME 5. (…)

Tree Number(s)    C04.557.470.035.215.100 (…)

Entry Term(s)       Polyposis Syndrome, Familial (…)

(Top) An NCBI corpus excerpt: title of abstract 10021369. In italics, the mention, in red the identifier. (Bottom) Snapshot of entry for disease concept D011125. The “Scope Note” is a short (10-20 tokens) definition of the concept.

Knowledge Base

is a bio-medical controlled vocabulary produced by the National Library of Medicine [17] covering a large number of health concepts, including $10,923$ disease concepts, and is used for indexing, cataloging and searching bio-medical and health-related documents and information. There are records of several types divided further into several categories. In this paper we consider only category C Descriptors, in other words on diseases disease nodes are arrayed hierarchically (in the form of an acyclic directed graph or tree) from most generic to most specific in up to thirteen hierarchical levels, and have many attributes, such as,  MeSH Heading, Tree Number(s), Unique ID, Scope Note and Entry Term(s). See Figure <ref> (bottom)[Full example: <https://meshb.nlm.nih.gov/record/ui?ui=D011125>].

We use the NCBI disease corpus [5] for training and validating our methods. It consists of $792$[We drop one training abstract because it is repeated: abstract 8528200.] abstracts separated into training, validation and test subsets (see Table <ref>). The corpus is annotated with disease mentions mapped to either or [9] (whose identifiers we convert to identifiers if possible using CTD [3]), Figure <ref> (top) illustrates an annotated instance in the corpus.

NCBI disease corpus statistics.

Split       Abstracts     Total entities     Unique entities     Unique concept IDs     Tokens
Training      $592$     $5,134$     $1,691$     $657$     $136,088$

Validation     $100$     $787$     $363$     $173$     $23,969$

Test     $100$     $960$     $424$     $201$     $24,497$

Visualization of the MTL architecture.

For NER we use variants of [14] and [18] biLSTM-CRF models. First, we encode words in an abstract using either a fixed and pre-trained contextualized encoder, i.e. bioELMo [10], or a concatenation of a fixed and pre-trained biomedical embedding, i.e. SkipGram  [21], and a trainable character-level encoder (e.g. LSTM [14] or CNN [18]). We then re-encode every token with a trainable biLSTM whose outputs are linearly transformed to the potentials of a linear-chain CRF [13]. All trainable parameters are optimized to maximize the likelihood of observations (i.e. IOB-annotated spans). At prediction time, Viterbi decoding finds the optimum IOB sequence in linear time.

EL with embeddings

We want to capitalize on the structure of the disease branch of . Thus, for every disease node, we learn an embedding with [20, 7]. The algorithm starts from randomly initialized node embeddings which are then optimized to be predictive of neighbourhood in the graph (for this we ignore the directionality inherit to relations in ). We use the framework [7] to generate two types of representations. For type-1, we use as is, that is, with randomly initialized node embeddings. For type-2, we leverage lexical information by initializing a node's embedding with the bioELMo encoding of that node's scope note (we use average pooling). The idea behind type-2 embeddings is to combine lexical features in the scope note and  structural relations in . Our EL model is a logistic regression classifier parameterized by a bilinear product between the embedding $\emb(y)$ of a node $y$ and the contextualized encoding $\enc(x,m)$ of a mention $m$ in an abstract $x$, i.e. $p(y|x,m, \theta) \propto \exp(\enc(x,m)^\top \mathbf W \emb(y))$, where $\theta = \{\mathbf W\}$ are trainable parameters. The node embedding is either a type-1 or type-2 embedding, and the mention embedding is the average of the bioELMo contextualized representation of the tokens in the disease mention. Unlike previous work, we normalize $p(y|x,m, \theta)$ against the set of all known entities in , that is, without a pre-selection of candidates. The model is optimized to maximize the likelihood of  observations—triples $(m, x, y)$ of mentions in an abstract resolved to nodes.

EL with GCN encoders

Graph convolution networks (GCNs) [12] generalize CNNs to general graphs. A GCN is an encoder where node representations are updated in parallel by transforming messages from adjacent nodes in a graph structure. A message is nothing but a parameterized (possibly non-linear) transformation of the representation of the neighbour. Stacking $L$ GCN layers allows information to flow from nodes as far as $L$ hops away in graph structure. In particular we follow [19], where the encoding $\mathbf h_v^{(\ell)}$ of a node $v$ in the taxonomy is computed as $\mathbf{h}_v^{(\ell)} = \sigma \left( \sum_{u \in \mathcal{N}(v)}\mathbf{W}^{(\ell)}\mathbf{h}_u^{(\ell-1)} + \mathbf b^{(\ell)} \right)$, where $\ell$ indexes the layer, $\mathcal{N}(v)$ are neighbors of $v$ in (we again discard directionality), $\mathbf b^{(\ell)}$ and $\mathbf W^{(\ell)}$ are the parameters of the $\ell$th GCN layer, $\sigma$ is a nonlinearity (we use ReLU), and $\mathbf{h}_v^{(0)}=\emb(v)$ is the initial embedding of the disease node $v$. For the $0$th representation of nodes,  we use the bioELMo encoding of a node's scope note. As a trainable encoder, a GCN layer is an integral part of the classifier and its parameters are updated by backpropagation from the downstream objective. Once again, our objective is to maximize the likelihood of observations under a probabilistic classifier, i.e. $p(y|x,m, \theta) \propto \exp(\enc(x, m)^\top \gcn(y; \theta))$, where $\gcn(y; \theta) = \mathbf h_y^{(L)}$ and $\theta = \{(\mathbf W^{(\ell)}, \mathbf b^{(\ell)})_{\ell=1}^L\}$ are the trainable parameters. Note that unlike embeddings, GCN encoders are trained directly on EL supervision, which is arguably of limited availability.

Multitask Learning

Traditionally, the problem of disease normalization is tackled by first identifying the disease names (NER) and then normalizing them (EL). We attempt to learn from both types of supervision by having a NER and an EL model share parts of their architectures. This is known as multitask learning [1]. In particular, we share the encoder of the NER architecture, see Figure <ref>, and derive mention features for the EL model from there.

§ EXPERIMENTS AND RESULTS

We train our models using ADAM [11] and initial learning rate of $10^{-3}$. We split our abstracts into sentences using NLTK[<https://www.nltk.org/>] and process sentences independently. We use $200$-dimensional word2vec  embeddings and $1,024$-dimensional bioELMo encodings.  For NER models, we learn $60$-dimensional character embeddings. We train node embeddings using for $100$ epochs, whereas GCN-based EL models are trained for $500$ epochs. All models employ $1,024$-dimensional node embeddings. We stack $L=2$ GCN layers with $2,048$ hidden and $1,024$ output units. We use $0.5$ dropout regularization [22][The code of our experiments is available at:

Metrics and Results

We report precision (Pre), recall (Rec) and (micro averaged) F1 scores (F1) for NER. For EL it is customary to test whether the target canonical entity (a concept identifier) occurs among the top $k$ predictions [25]. Hence we report precision at confidence rank $k$ (Pre@k) and mean reciprocal rank (MRR) defined as $\text{MRR} = 1/|E| \cdot \sum \{1/\text{rank}_i \mid 1\leq i \leq |E|\}$. We use early stopping for model selection based on performance on the validation set, and report the average result of $5$ independent runs on the test and validation sets.

Table <ref> shows the results for NER. By replacing the word and character embeddings altogether with bioELMo embeddings, the results improve by a large margin. Increasing the number of parameters in the models did not affect model performance. Furthermore, we improve on [8]'s re-implementation of Lample et al.'s biLSTM-CRF model on the NCBI corpus (+$0.028$ F1 score points).

We tested our EL baseline—type-1, encoding only the structural information—against disease embeddings generated by averaging the bioELMo embedding of Scope Notes (see Table <ref>), and found it to perform better except on Pre@30. Node lexicalization, type-2, yields results on a par or better than the bioELMo baseline. Using GCN for training, we do not achieve any improvement. This suggests that both structural and lexical information in are important for normalization, and that taxonomy encoding is best achieved when computed independently from the EL task. This strategy seems on the other hand to underperform other normalization methods reported in the literature [25, 15]. Our systems however do not incorporate many of their optimizations such as: normalization to disease synonym rather than identifier, abbreviation resolution, re-ranking and filtering, or coherence models.

Table <ref> shows, finally, that using MTL results in an improved NER score ($+0.009$ F1 score points w.r.t. our best NER model in Table <ref>). This indicates a certain degree of transfer from the EL to the NER task. It also yields a improvement w.r.t. similar NER and MTL for the NCBI corpus [8, 24] ($+0.028$ points for NER and $+0.015$ for MTL).

Error analysis

Regarding NER, most errors arise when dealing with multi-token entities: e.g., for sporadic T-PLL only the head T-PLL is detected. On the other hand, the bioELMo model is able to identify abbreviated disease names much better than the two baselines: e.g., T-PLL occurring alone is always correctly detected. This might be because bioELMo starts at character-level and gradually captures token- and phrase-level information.

Our best EL model makes errors by confusing diseases with their ancestors and neighbors: e.g., it confuses D016399- Lymphoma, T-Cell with D015458- Leukemia, T-Cell, with which it shares an ancestor: D008232- Lymphoproliferative Disorders. Often, it resolves correctly the first instance of a concept but returns instead a neighbour afterwards: e.g., in “Occasional missense mutations in ATM were also found in tumour DNA from patients with B-cell non-Hodgkins lymphomas (B-NHL) and a B-NHL cell line" B-cell non-Hodgkins lymphomas is correctly resolved to D016393 but B-NHL (its abbreviation) is mapped to D008228 (its child) and the second occurrence to D018239 (another form of cancer). This might be due to the following facts: Both NCBI and are not particularly large resources, hence the mention and disease encodings learnt by the model are not sufficiently discriminative. In addition, GCN and training tends to ignore the direction of taxonomical edges. Finally, DNorm and NormCo normalize diseases to concepts indirectly via their synonyms, allowing them to exploit better lexical features[They also include a wide range of optimizations such as re-ranking, coherence models or abbreviation resolution.].

§ CONCLUSIONS

In this paper, we address the problem of normalizing disease names to concepts (canonical identifiers) by adapting state-of-the-art neural graph embeddings (GCN and ) that exploit both 's hierarchical structure and the description of diseases. Our graph-based disease node encoding is the first of its kind, to best of our knowledge. We also apply multi-tasking to transfer information between disease detection (NER) and resolution (EL) components of the task, leveraging their common signals to improve on the single models. We observe that bioELMo embeddings lead to substantial improvement in NER performance. We demonstrate that node lexicalization does improve over either pure structural or lexical embeddings, and that MTL gives rise to state-of-the-art performance for NER in the corpus used (NCBI corpus).

On the other hand, we do not currently outperform other disease normalization approaches [25, 15], and often confuse neighbour concepts with their true targets. In the future we intend to experiment with the optimizations reported (especially: resolution to synonym rather than concept identifier and re-ranking) by [25, 15]. is also a comparatively small graph with short disease descriptions. As further work we plan to enrich it by linking it to larger scale resources (e.g., Wikipedia).

[1]
Rich Caruana.
Multitask learning.
Machine Learning, 28(1):41–75, 1997.

[2]
Hyejin Cho, Wonjun Choi, and Hyunju Lee.
A method for named entity normalization in biomedical articles:
Application to diseases and plants.
BMC Bioinformatics, 18(1):451, 2017.

[3]
Allan Peter Davis, Cynthia J. Grondin, Robin J. Johnson, Daniela Sciaky, Roy
McMorran, Jolene Wiegers, Thomas C. Wiegers, and Carolyn J. Mattingly.
The Comparative Toxicogenomics Database: update 2019.
Nucleic Acids Research, 47(D1):D948–D954, 2018.

[4]
Rezarta Islamaj Dogan and Zhiyong Lu.
An inference method for disease name normalization.
In AAAI Fall Symposium: Information Retrieval and Knowledge
Discovery in Biomedical Text, 2012.

[5]
Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu.
NCBI disease corpus.
Journal of Biomedical Informatics, 47(C):1–10, 2014.

[6]
Nathan Greenberg, Trapit Bansal, Patrick Verga, and Andrew McCallum.
Marginal likelihood training of bilstm-crf for biomedical named
entity recognition from disjoint label sets.
In Proceedings of EMNLP 2018, 2018.

[7]
Aditya Grover and Jure Leskovec.
Node2Vec: Scalable feature learning for networks.
In Proceedings of KDD 2016, 2016.

[8]
Maryam Habibi, Leon Weber, Mariana Neves, David Luis Wiegandt, and Ulf Leser.
Deep learning with word embeddings improves biomedical named entity
Bioinformatics, 33(14):i37–i48, 2017.

[9]
Ada Hamosh, Alan F. Scott, Joanna S. Amberger, Carol A. Bocchini, and Victor A.
Online Mendelian Inheritance in Man (OMIM), a knowledge base of
human genes and genetic disorders.
Nucleic Acids Research, 33(suppl_1):D514–D517, 2005.

[10]
Qiao Jin, Bhuwan Dhingra, William W Cohen, and Xinghua Lu.
Probing biomedical embeddings from language models.
CoRR, abs/1904.02181, 2019.

[11]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
Proceedings of ICLR 2014.

[12]
Thomas N. Kipf and Max Welling.
Semi-supervised classification with graph convolutional networks.
CoRR, abs/1609.02907, 2016.

[13]
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira.
Conditional random fields: Probabilistic models for segmenting and
labeling sequence data.
In Proceedings of ICML 2001, 2001.

[14]
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and
Chris Dyer.
Neural architectures for named entity recognition.
CoRR, abs/1603.01360, 2016.

[15]
Robert Leaman, Rezarta Islamaj Doğan, and Zhiyong Lu.
DNorm: disease name normalization with pairwise learning to rank.
Bioinformatics, 29(22):2909–2917, 2013.

[16]
Haodi Li, Qingcai Chen, Buzhou Tang, Xiaolong Wang, Hua Xu, Baohua Wang, and
Dong Huang.
CNN-based ranking for biomedical entity normalization.
BMC Bioinformatics, 18, 10 2017.

[17]
C.E. Lipscomb.
Medical subject headings (MeSH).
Bull Medical Library Association, 88(3):265–266, 2000.

[18]
Xuezhe Ma and Eduard H. Hovy.
End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF.
CoRR, abs/1603.01354, 2016.

[19]
Diego Marcheggiani and Ivan Titov.
Encoding sentences with graph convolutional networks for semantic
role labeling.
CoRR, abs/1703.04826, 2017.

[20]
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.
DeepWalk: Online learning of social representations.
In Proceedings of KDD 2014, 2014.

[21]
S. Pyysalo, F. Ginter, H. Moen, T. Salakoski, and S. Ananiadou.
Distributional semantics resources for biomedical text processing.
In Proceedings of LBM 2013, 2013.

[22]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Dropout: A simple way to prevent neural networks from overfitting.
Journal of Machine Learning Research, 15(1):1929–1958, 2014.

[23]
Camilo Thorne and Roman Klinger.
On the semantic similarity of disease mentions in  and
In Proceedings of NLDB 2018, 2018.

[24]
Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang,
Curtis Langlotz, and Jiawei Han.
Cross-type biomedical named entity recognition with deep multi-task
Bioinformatics, 35(10):1745–1752, 2018.

[25]
Dustin Wright, Yannis Katsis, Raghav Mehta, and Chun-Nan Hsu.
NormCo: Deep disease normalization for biomedical knowledge base
In Proceedings of AKBC 2019, 2019.