# Lego: Learning to Disentangle and Invert Concepts Beyond Object Appearance
in Text-to-Image Diffusion Models

Saman Motamed
INSAIT, Sofia University
<EMAIL_ADDRESS>Danda Pani Paudel
INSAIT, Sofia University
ETH Zurich
<EMAIL_ADDRESS>Luc Van Gool
INSAIT, Sofia University
ETH Zurich - KU Leuven
<EMAIL_ADDRESS>
###### Abstract

Diffusion models have revolutionized generative content creation and text-to-
image (T2I) diffusion models in particular have increased the creative freedom
of users by allowing scene synthesis using natural language. T2I models excel
at synthesizing concepts such as nouns, appearances, and styles. To enable
customized content creation based on a few example images of a concept,
methods such as Textual Inversion and DreamBooth invert the desired concept
and enable synthesizing it in new scenes. However, inverting more general
concepts that go beyond object appearance and style (adjectives and verbs)
through natural language, remains a challenge. Two key characteristics of
these concepts contribute to the limitations of current inversion methods. 1)
Adjectives and verbs are entangled with nouns (subject) and can hinder
appearance-based inversion methods, where the subject appearance leaks into
the concept embedding and 2) describing such concepts often extends beyond
single word embeddings (“being frozen in ice”, “walking on a tightrope”, etc.)
that current methods do not handle.

In this study, we introduce Lego, a textual inversion method designed to
invert subject entangled concepts from a few example images. Lego disentangles
concepts from their associated subjects using a simple yet effective Subject
Separation step and employs a Context Loss that guides the inversion of
single/multi-embedding concepts. In a thorough user study, Lego-generated
concepts were preferred over 70% of the time when compared to the baseline.
Additionally, visual question answering using a large language model suggested
Lego-generated concepts are better aligned with the text description of the
concept.

Figure 1:  We showcase Lego’s ability to invert concepts of “frozen in ice”,
“burnt and melted”, and “closed eyes” using as few as just four example images
(two with and two without the concept). Our results cover text-to-image
models, including LDM, Stable Diffusion 2.1, Attend and Excite, and closed-
source DALL.E 2 and 3. Notably, Lego faithfully represents concepts, even with
a less capable backbone (LDM), while more powerful models such as DALL.E,
though artistically impressive, do not consistently capture the intended
concept.

##  1 Introduction

If you saw a Lego figurine frozen in a block of ice or a Rubik’s cube melt and
deform, how confident would you be in your ability to describe the fine
details of the scene by using natural language descriptions alone? And even
then, can text-to-image models generate images that accurately follow such
text descriptions? (see Figure 1). As Mark Twain said; “Actions speak louder
than words” and describing the fine details of any scene is often more
difficult than showing someone / something an example of a similar scene.

Recently, large text-to-image Diffusion models [39, 31, 42, 49, 2] have shown
promising results in synthesizing high quality images. These models empower
users by enabling scene synthesis through natural language descriptions. The
ability to craft personalized content with these models, e.g. a scene
featuring one’s pet dog as Superman, has spurred a research direction aimed at
enhancing the user’s control for customized content creation [13, 41, 24, 21,
52, 18, 44, 28, 23, 19, 29, 14, 47]. Using a few example images of a concept,
text-based inversion methods identify an embedding within the textual
embedding space of a text-to-image model’s text-encoder that can synthesize
that specific concept. This identified embedding can then be injected into
various text descriptions, allowing for the synthesis of the concept in
diverse scenes. Inversion methods capable of inverting appearance-based
concepts include Textual Inversion (TI) [13], DreamBooth [41], Custom-
Diffusion [24] and ELITE [47]. While TI and ELITE kept the diffusion model
frozen, DreamBooth and Custom-Diffusion tuned parts of the model on the
example images. Taking a different approach from those works, ReVersion [21]
aims to invert relations between subjects (e.g, towards, inside, etc.) from a
few example images rather than concepts related to subject appearance.

With vision-language models having shown strong bias towards nouns / objects
[30, 34, 17, 51] and existing inversion and personalization techniques being
predominantly centered around learning appearances, relations and styles, we
redirect our attention to object agnostic concepts, specifically, adjectives
and verbs. Thus, this paper takes a comprehensive approach to examine the
capabilities of text-to-image models in handling adjectives and verbs. We show
that current inversion methods often fail to invert such concepts and our
experiments suggest this challenge is due to two key characteristics inherent
to these concepts. First, such concepts are entangled with a subject (noun).
For instance, the concept of melting gives different shapes and
characteristics to different subjects it is applied to and current inversion
methods are not able to handle subject entangled concepts. Second, describing
such concepts frequently extends beyond single word embeddings. For instance
being frozen in ice is expressed using multiple word embeddings whereas
appearance based concepts can have a single word embedding (e.g., some toy, a
pet, etc.). We introduce Lego, a textual inversion method that augments the TI
framework [13] with two additional components; a Subject Separation step that
disentangles a concept from its associated subject by recovering an explicit
embedding for the subject and a contrastive Context Loss that helps guide
multiple embeddings in the textual embedding space, increasing editability and
accuracy of the learned embeddings. We show Lego’s capabilities to invert
various concepts with comparisons to state of the art T2I and inversion
methods, and demonstrate that Lego is a reliable and stand-alone inversion
method for general concepts applicable to any text-conditioned diffusion
model.

Our major contributions are summarized below:

  * •

We study a new problem, General Concept Inversion of adjectives and verbs. We
show text-guided image synthesis models and current text based inversion and
personalization methods are unable to synthesize such concepts.

  * •

We propose two modifications to TI; Subject Separation and Context Loss that
allow our method to disentangle concepts from subjects and guide the concept’s
embeddings in the textual embedding space. These modifications together lead
to faithful inversion of concepts.

Table 1: A characteristics overview of some recent T2I personalization methods (in order from left to right; Lego, Textual Inversion, ReVersion, Custom-Diffusion, ELITE, ControlNet and Attend and Excite), with a reference to sample images of each method when used for inverting/synthesizing adjective and verb concepts. | Lego |  TI [13] |  RV [21] |  C-Diff [24] |  ELITE [47] |  CNet [52] |  A+E [5]
---|---|---|---|---|---|---|---
Frozen Model | ✓ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓
Text Embedding Optimization | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗
Subject/Concept Disentangling | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
Multi-Embedding Steering | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
Sample Results |  Figs. 1, 5, 8 |  Fig. 6 \- Sec. 11 |  Fig. 4 |  Fig. 12 |  Figs. 1, 5

##  2 Related Work

Diffusion Models [20, 16, 46, 45, 43] have become the go-to generative model
over their counterparts [3, 22, 15, 9, 4, 12] with their superior synthesis
quality and more stable training. Recently, text-to-image (T2I) diffusion
models [32, 36, 37, 40, 42] have shown promise in enabling an intuitive
interface for users to control image generation, using natural language
descriptions. However, gaining granular control and customized image
generation has proven difficult with natural language descriptions alone [11,
27, 48, 26]. Addressing this difficulty has started a line of research for
inverting desired concepts in these large models and better tuning them for
customized content creation. In Table 1 we describe the characteristics of the
most relevant personalization models and give further detail on some of the
recent works that make an attempt at solving this problem below.

Textual Inversion. Given a T2I model, Textual Inversion [13] is tasked with
finding a pseudo-word’s embedding that can represent a subject, given as few
as 3-4 exemplar images of that subject. Without fine-tuning any part of the
network, TI searches the textual embedding space of the diffusion model’s
vision-language encoder (BERT [7], CLIP [35], etc.) to find an embedding that
can synthesize the given object in the reference images. In the ablation study
(Figure 6 and Supplementary Section 11), we show how TI is not enough for
inverting general concepts.

DreamBooth + Custom Diffusion. With similar objective as TI, DreamBooth [41]
and Custom-Diff [24] not only perform optimization in the textual embedding
space, but also train parts of the T2I diffusion network with the exemplar
images in order to achieve a better representation of the given concept.
DreamBooth and Custom-Diff achieve better concept representation compared to
TI. Custom-Diff is similar to DreamBooth with a few differences; 1) allowing
multiple concepts being learned simultaneously, 2) light weight tuning by only
updating the cross-attention parameters of the diffusion network and 3) a
regularization step to stop language drift during tuning. Section 5.1 shows
how these models fail to invert subject entangled concepts.

Figure 2: Top figure is an overview of Lego’s objective and the Subject
Separation step. Learning an explicit embedding to represents the subject
(Rubik’s cube) allows the concept (“melted”) embedding to dissociate from the
subject’s appearance features, as visualized by $<$concept$>$ embedding
(highlighted in blue). The figure below depicts the framework that uses
concept only images (same setting as TI, DreamBooth, etc.). In this setting,
the subject’s features leak into the $<$concept$>$ embedding, (highlighted in
orange and blue), as shown by the concept’s visualization which evinces both
melting effects and Rubik’s cube features.

Attend-and-Excite. Feeding long text descriptions to T2I models often leads to
catastrophic forgetting; where some words in the sentence fail to appear in
the generated image. Attend and Excite [5] enforces the cross-attention units
of the diffusion network to be activated for the user-selected tokens,
encouraging the model to generate all subjects described in the text prompt.
In Figures 1 and 5, we show that selecting the concept tokens and using the
Attend and Excite method is not enough to generate the concepts.

ReVersion. Relation Inversion [21] is the only work besides ours that focuses
on inverting non-appearance concepts, namely relations between subjects.
ReVersion uses a contrastive loss based on InfoNCE [33] that steers the
relation embedding towards the embeddings of prepositions (preposition prior),
with the observation that in natural language, prepositions express the
relation between subjects. ReVersion also uses natural language descriptions
of the subjects in the exemplar images to separate the subjects from the
relation embedding. Relations control the positioning of subjects with respect
to one another and hence, the subjects stand alone. This facilitates the use
of natural language descriptions alone for separating the relation embedding
from subject appearance. In contrast, our focus lies in studying concepts
entangled with the subject. Our ablation study (Section 5.4) shows that
ReVersion’s framework fails to invert subject-entangled concepts.

##  3 The Concept Inversion Problem

The goal of Lego is to learn embeddings
${\mathcal{CPT}=\\{<\\!cpt_{1}\\!>,...,<\\!cpt_{n}\\!>\\}}$ that represent a
general concept $\mathbf{C}$, from a few exemplar images. Let
${\mathcal{I}=\\{I_{1},I_{2},...,I_{m}\\}}$ be a small set of such exemplar
images involving a common subject $\mathbf{S}_{e}$. Lego requires a clear
separation between images of a subset with concepts, say $\mathcal{I}_{C}$,
and the same without, say $\mathcal{I}_{\overline{C}}$, such that
$\mathcal{I}=\mathcal{I}_{C}\cup\mathcal{I}_{\overline{C}}$. For the example
shown in Figure 2 (top), the caption of all the images in
$\mathcal{I}_{\overline{C}}$ is “photo of a Rubik’s cube”, whereas the same in
$\mathcal{I}_{C}$ is “photo of a Rubik’s cube that is melted”, where the
exemplar subject $\mathbf{S}_{e}$ is “Rubik’s cube” and the general concept of
interest $\mathbf{C}$ is “melted”. In this setting, we wish to learn the
embeddings $\mathcal{CPT}$ corresponding to the concept $\mathbf{C}$ such that
the concept can be transferred to any novel target subject $\mathbf{S}_{t}$–
which is “a teddy bear” in the very same example.

Our experiments show the inherent difficulty in synthesizing accurate
representations of numerous verb and adjective concepts using T2I models with
natural language guidance. The ability to learn embeddings capable of
representing such concepts using as few as four exemplar images allows the
user to have greater control over T2I models. In the context of T2I models,
adjective and verb concept inversion, where the concept is entangled with
subjects, has not been previously explored. We demonstrate that such
entanglement poses challenges for inversion methods that have previously aimed
at subject/concept separation, as observed in other concept types like
relations [21]. By exploring the inversion of verbs and adjectives, our work
has the potential to inspire future studies and enhance representations of
more complex concepts that go beyond object appearance.

Figure 3: An overview of Lego’s framework. From left to right, during
embedding optimization, Lego dedicates an embedding $<\\!subj\\!>$ for
inverting the subject $\mathbf{S}_{e}$ in the exemplar images
$\mathcal{I}_{C}$ and $\mathcal{I}_{\overline{C}}$. This stops appearance
leakage to the concept embeddings. Each concept embedding (
$<\\!cpt_{i/j}\\!>$) is separately steered towards user defined words
($\mathcal{P}_{i/j}$) that correspond to the embedding’s semantic word and
away from antonyms of those words ($\mathcal{N}_{i/j}$). After the inversion,
the learned embeddings can together be applied to different target subjects
$\mathbf{S}_{t}$ (“Statue” and “Teddy bear”) to manifest the concept in new
scenes.

##  4 Learning Concepts Beyond Appearance

###  4.1 Preliminaries

T2I Diffusion Models. Diffusion models [20, 43, 8] are a class of generative
models that learn to generate novel scenes by learning to gradually denoise
samples from the Gaussian prior $\text{x}_{T}$ (trained by adding noise
$\epsilon\sim\mathcal{N}(0,1)$ to $\text{x}_{0}$) back to the image
$\text{x}_{0}$. In this work, we study T2I models, namely the Latent Diffusion
Model (LDM) [39] which is the predecessor of Stable Diffusion. Instead of
directly adding noise and learning to denoise images, LDM operates on a
pretrained autoencoder’s projection of images in some latent space. LDM
enables T2I generation by conditioning the denoising network
$\epsilon_{\theta}(.)$ on the encoding of text descriptions $c$ using a text
encoder $\tau_{\theta}(.)$ such as CLIP [35] or BERT [7]. To sample images
using a trained latent diffusion model $\epsilon_{\theta}(.)$, we iteratively
denoise a noise latent $\text{x}_{t}$ for $t$ steps, using the predicted noise
$\epsilon_{\theta}(\text{x}_{t},t,\tau_{\theta}(c))$ to get $\text{x}_{0}$.
$\text{x}_{0}$ is then mapped to the image space using some pre-trained
decoder. The LDM loss is:

$\mathcal{L}_{LDM}(\theta):=\mathbb{E}_{t,\text{x}_{0},\epsilon}[\|\epsilon-\epsilon_{\theta}(\text{x}_{t},t,\tau_{\theta}(c))\|^{2}].$
(1)

Inversion in the Textual Embedding Space. Current textual inversion methods
focus on either appearance inversion of a subject [13, 24, 41, 47] or
inverting a relation between subjects [21]. Given a few exemplar images of a
subject or relation, the aim is to find a text embedding $<$$emb^{*}$$>$ in
the output space of $\epsilon_{\theta}(.)$, such that injecting
$<$$emb^{*}$$>$ in any encoded text $\tau_{\theta}(c)$ allows the
reconstruction of that concept in a new context defined by description $c$.
The embedding inversion loss is defined as:

$\mathcal{L}_{\text{inversion}}=\mathbb{E}_{t,\text{x}_{0},\epsilon}[\|\epsilon-\epsilon_{\theta}(\text{x}_{t},t,\tau_{\theta}(c))\|^{2}],$
(2)

such that: $<\\!emb^{*}\\!>=\underset{<emb>}{\mathrm{\text{arg
min}}}(\mathcal{L}_{\text{inversion}})$, where $<$$emb$$>$ is the concept
embedding being optimized, and is fed into the pretrained T2I model as part of
the text description c.

###  4.2 Method Overview

In this section, we present Lego, a text embedding inversion method for
extracting general concepts (adjectives and verbs) from exemplar images. Lego
augments Textual Inversion with two modifications; Subject Separation and
Context Loss. What separates Lego from works such as Textual Inversion, Custom
Diffusion and ReVersion is that Lego aims to invert concepts that move beyond
objects’ relations, styles, and appearances. These concepts are considered to
be entangled with the subject of the exemplar images. For instance, The
concept of “melting” is not standalone and changes the features of the subject
it is applied to, whereas in subject inversion (TI [13], Custom-Diff [24],
etc.), the subject’s appearance does not change and stands alone. Similarly,
in relation inversion (ReVersion [21]), the relation defines how different
subjects interact and does not change the subjects’ appearance features. For
example, Batman and Superman sitting back to back does not change Batman nor
Superman’s intrinsic appearance features.

###  4.3 Subject Separation

Appearance inversion (enabled by Textual Inversion) focuses on low-level
features in order to learn a single style or subject appearance from a few
sample images. For such subject-centric inversions, a pixel level
reconstruction loss (Equation 2) is sufficient to find an embedding that
represents the subject. Inverting relations between subjects (ReVersion) is a
higher-level concept that requires more than a pixel level loss which
ReVersion addresses using a preposition prior (Section 2). Relations between
subjects however, are not entangled with the subjects’ appearance, hence
ReVersion is able to detach the relation embedding from the subject embedding
by steering it away from the embedding of natural language words that describe
the subjects (e.g., while inverting the relation of Batman and Superman
sitting back to back, the relation embedding is steered away from embeddings
of “Batman” and “Superman”) . Our concepts however are entangled with the
subject such that the same approach as ReVersion leads to appearance leakage
(see Figures 2 \- bottom and 6) and does not allow the concept to be separated
from the subject.

In order to learn embeddings $\mathcal{CPT}$ representing the concept
$\mathbf{C}$ disentangled from the exemplar subject $\mathbf{S}_{e}$, we
dedicate an additional embedding $<\\!subj\\!>$ to separately represent the
subject $\mathbf{S}_{e}$. This embedding gets optimized twice using (i)
$\mathcal{I}_{\overline{C}}$ and (ii) $\mathcal{I}_{{C}}$, separately.
Learning $<$subj$>$ from subject-only images $\mathcal{I}_{\overline{C}}$
naturally enables the concept embeddings $\mathcal{CPT}$ to not have to learn
the subject’s appearance, while reconstructing $\mathcal{I}_{{C}}$ using both
$<$subj$>$ and $\mathcal{CPT}$ embeddings (see Figure 2). In other words,
learning the subject embedding without concept and using its embedding to
generate the subject with concept during optimization allows us to better
perform the desired disentanglement. Our extensive experiments clearly
illustrate that the proposed learning setup prevents the subject-specific
features to leak into the concept embedding. In Figures 2 and 6 and in the
Supplementary Section 11, we show that the absence of the proposed Subject
Separation leads to undesired outcomes. For instance, Figure 2 (bottom) shows
how not performing Subject Separation in learning the concept of melting from
Rubik’s cube images leads to the “$<$melted$>$ toy bear” to have Rubik’s
features, whereas learning an explicit embedding for the cube Figure 2 (top)
keeps the Rubik’s features disentangled from the concept embedding, leading to
desired inversion of the concept.

###  4.4 Contrastive Context Guidance

General concepts of our interest (e.g., frozen in a block of ice, walking on a
tightrope) often require a description of multiple words. Such descriptive
concepts are expressed by multiple words. Hence in this work, we allow for
learning multiple word embeddings per concept. More specifically, Lego enables
learning multiple embeddings
(${\mathcal{CPT}=\\{<\\!cpt_{1}\\!>,...,<\\!cpt_{n}\\!>\\}}$) for a single
concept – such that combining all $n$ embeddings represents the concept,–
unlike, TI [13] and Custom-Diff [24] which learn concepts described by a
single word embedding. Inspired by success of contrastive losses for
representation learning [33, 25, 50, 21], we employ an InfoNCE-based [33] loss
to learn the embeddings $\mathcal{CPT}$ in a contrastive setting.

Learning concept embeddings in the contrastive setting however requires
positive and negative embedding sets. Let $\mathcal{P}_{i}=\\{P_{ik}\\}$ and
$\mathcal{N}_{i}=\\{N_{ik}\\}$ respectively be the positive and negative
embedding sets corresponding the concept embedding $<\\!cpt_{i}\\!>$. Note
that each embedding $<\\!cpt_{i}\\!>$ corresponds to a semantic word. We form
the sets $\mathcal{P}_{i}$ and $\mathcal{N}_{i}$ by embedding the synonyms and
antonyms, respectively, of the semantic word corresponding to
$<\\!cpt_{i}\\!>$. Please refer to Section 8 for more detail on choosing
$\mathcal{P}_{i}$ and $\mathcal{N}_{i}$. In this manner, for the general
concept $\mathbf{C}$, we obtain a set of triplets
$\\{(<\\!cpt_{i}\\!>,\mathcal{P}_{i},\mathcal{N}_{i})\\}_{i=1}^{n}$. This
triplets’ set is then used to compute our modified InfoNCE loss, referred here
as $\mathcal{L}_{\text{context}}$, which is given by,

$\\!\mathcal{L}_{\text{context}}\\!=\\!-\\!\sum\limits_{i=1}^{n}\log\\!\frac{\sum\limits_{\\!k=1\\!}^{|\\!\mathcal{P}_{i}\\!|}e^{<cpt_{i}>^{\intercal}\cdot
P_{ik}}}{\sum\limits_{\\!k=1\\!}^{|\\!\mathcal{P}_{i}\\!|}\\!e^{<cpt_{i}>^{\intercal}\cdot
P_{ik}}\\!+\\!\sum\limits_{\\!k=1\\!}^{|\\!\mathcal{N}_{i}\\!|}\\!e^{<cpt_{i}>^{\intercal}\cdot
N_{ik}}}.$ (3)

During the learning process, the context loss guides each concept embedding
$<\\!cpt_{i}\\!>$ individually, towards the embeddings of the respective
positive words and away from the negative ones. Figure 3 shows an example
where we want to capture the concept of “frozen in ice” with two embeddings
$<\\!cpt_{i}\\!>$ and $<\\!cpt_{j}\\!>$, one representing “frozen” and one
representing “ice”. When combined together in a sentence, they express the
concept $\mathbf{C}$ in the exemplar images $\mathcal{I}_{C}$. Note that the
context loss is applied to only the concept embeddings $\mathcal{CPT}$, but
not to subject embedding $<\\!subj\\!>$. In Section 5.4, we provide examples
of concept inversion with and without the context loss. Existing T2I inversion
methods [13, 47, 41, 21, 24] optimize a single word’s embedding(s) and among
them, relation inversion is the only existing work that also performs the
learning in a contrastive setting. To invert a relation, the embedding is
steered towards a set of preposition words and away from other Parts of Speech
words (i.e. nouns, verbs, etc.), by making the observation that relations
between objects are defined in preposition-dense space of the textual
embedding space. In contrast, our objective of learning descriptive multi-word
concepts, enabled by our context loss, requires the embeddings to be steered
towards their corresponding semantic word, each in a different part of the
text-embedding space. Furthermore, the negative sets of words allow Lego’s
embeddings to be steered away from words that can disturb the concept
inversion. By disturbing the inversion, we refer to words that are in close
distance to our concept embedding, yet associated with a concept we do not
want to represent. For instance, numbers are closely embedded in the text
embedding space. In order to accurately invert cardinality, say number $3$, we
can construct a negative set of words comprised of $\\{1,2,4,5\\}$.

###  4.5 Lego

Lego uses the inversion loss in Equation 2 to learn an embedding
$<$$subj^{*}$$>$ that represents the subject of the exemplar images, by
optimizing an embedding $<$subj$>$. While optimizing the concept embeddings,
the weighted sum of the inversion loss and our context loss is used to obtain
the concept embeddings $<\\!\mathcal{CPT^{*}}\\!>$. Below we show both the
subject and concept inversion losses.

$\begin{split}<\\!subj^{*}\\!>&=\underset{<subj>}{\mathrm{\text{arg
min}}}(\mathcal{L}_{\text{inversion}}),\\\
<\\!\mathcal{CPT^{*}}\\!>&=\underset{<cpt>}{\mathrm{\text{arg
min}}}(\mathcal{L}_{\text{inversion}}+\lambda.\mathcal{L}_{\text{context}}).\end{split}$
(4)

These two embeddings’ recoveries, corresponding to subject and concepts, can
be thought as two parallel processes acting on the same image generator and
the set of exemplar images. Such clear separation in learning process enables
us to achieve the desired disentanglement.

##  5 Experiments

###  5.1 DreamBooth and Custom Diffusion

Both DreamBooth [41] and Custom-Diff [24] pursue the same objective by tuning
parts of the diffusion model, while optimizing a word embedding in order to
achieve better accuracy in synthesizing personalized concepts compared to
methods like Textual Inversion that keep the model frozen. Since Custom-Diff
allows learning multiple embeddings at the same time, we carried out our
experiments using Custom-Diff. With the same exemplar images used in Lego, we
trained Custom-Diff to learn the embeddings of “Rubik’s cube” and “melting”
from Rubik’s $\mathcal{I_{C}}$ and $\mathcal{I}_{\overline{C}}$ images and the
concept of “cat” and “closed eyes” from the cat images. Figure 4 shows that
inverting concepts while training the network does not allow for dissociation
of concepts from subjects and leads to appearance leakage. We show how Custom
Diffusion’s learned embedding for “melting”, when applied to a toy car and a
toy cat, inserts the Rubik’s cube features in those images and similarly, the
learned concept for “closed eyes” from the cat images, inserts the cat’s
features into images of Batman and a doll.

Figure 4:  Custom-Diff is unable to separate the concept embedding from the
subject appearance. Learning “closed eyes” from cat images and “melted” from
Rubik images will carry the subject features when applied to new subjects.

###  5.2 Lego

We tested Lego’s capabilities in inverting 10 various concepts; from
controlling cardinality of subjects (3, 4, 5) to concepts that deform the
subject (melting, crumpling), to the subject performing an action (closed
eyes, walking on a rope, arms raised) and change of state and appearance of
the subject (frozen in ice and having a smiley emoji face).

Figure 5:  Qualitative comparison of Lego with an LDM backbone and learned
concepts from exemplar images, compared to text-guided models such as LDM, SD
2.1, DALL.E 2&3 and Attend & Excite (highlighted words are the given token).
Please zoom in for details. Figure 6:  This figure shows examples of our
ablation study for learning the concept of “burnt and melted” from single
subject (top row) and multi subject (bottom row) example images and
transferring it to a “Toy bear”. We synthesized 100 images for each ablation
category. You can find all 100 images for each category in the Supplementary
Section 11. Please zoom in to see the details in the images. Figure 7:  This
figure (left) shows Lego’s composition capability of combining different
concepts learned from different images and (right) learning more complex
multi-word embedding concepts from a single example.

####  5.2.1 Results

In Figures 1 and 5, we show Lego’s results for a subset of the concepts and
provide comparisons with the latest SOTA methods for T2I generation. We note
that we use a less capable backbone compared to our competitors, LDM (the
original Latent Diffusion Model), for extracting concepts using Lego which can
authentically represent the concept even where more recent models such as
Stable Diffusion 2.1 and DALL.E fail. However, just like TI, Lego is
standalone and can be applied to any T2I model. We will be releasing Stable
Diffusion support under Hugging Face’s Diffusers library [10]. For more
examples of Lego with various subjects, please refer to Figure 8 in the
Supplementary material.

We tested Lego’s concept inversion against natural language for 10 concepts,
generating 200 images per concept using Lego and LDM with language control.
Three concepts specified cardinality (3, 4, or 5 subjects). Participants
counted subjects in each image. Lego consistently outperformed natural
language guidance in producing correct subject counts (see Table 2). For the
remaining seven concepts, we performed two different studies. A Visual
Question Answering (VQA) large language model (Flan-T5 XL) [37] was used to
answer questions about 200 images generated using Lego, versus 200 images
generated using natural language descriptions of each concept. Our experiments
showed that VQA models performed better when asked “Yes” or “No” questions
about specific concepts rather than asking more general questions such as “
What is this image?”. For instance, we generated images of “toy bears frozen
in ice”. We then prompted the VQA model with the question: “ Is the toy bear
frozen in ice?”. Table 2 shows the number of images where the VQA model
confirms the concept is in the image. Consistent with the numerical concepts,
Lego performs better than natural language for all concepts. Please refer to
Supplementary Section 8 where we give a detailed overview of the prompts used
and the generated images.

While VQA suggests Lego outperforms LDM, a more comprehensive evaluation for
generative models, especially concerning general concepts, involves human
preference metrics. To this end, we used Amazon Mechanical Turk [1] and for
each concept, paired the 200 images generated by Lego with the other 200
generated by LDM. We asked users to select the image they think best
represents the concept for each pair of images. Each question was answered by
10 users (total of 14000 answers for all 7 concepts). The majority vote
determined the outcome, showing that Lego was preferred over LDM in over 70%
of cases (Table 2).

Concept |  LLM $\uparrow$ |  Human $\uparrow$
---|---|---
Lego | LDM | Lego | LDM
3 | - | - | 107 | 85
4 | - | - | 63 | 18
5 | - | - | 36 | 18
Frozen in ice | 92 | 55 | 68.5% | 31.5%
Burnt and melted | 136 | 61 | 77% | 23%
Closed eyes | 111 | 75 | 74.5% | 25.5%
Smiley Emoji Face | 199 | 151 | 67.5% | 32.5%
Crumpled and squeezed | 147 | 64 | 60.5% | 39.5%
Walking on a rope | 133 | 4 | 84 % | 16%
Arms raised | 124 | 49 | 70% | 30%
Table 2: LLM metric reports the number (out of 200) of images, where Flan-T5
XL model confirms the image contains the concept. The Human metric reports the
number of images with correct subject cardinality for the numerical concepts,
and the percentage of time users preferred one method’s images compared to the
other.

###  5.3 Concept Composition and Complexity

Lego’s learned concept embeddings can be combined, as demonstrated in Figure 7
(left). It seamlessly composes concepts like “closed eyes” from cat images and
“walking on a rope” from figure images. This enables creating scenes like a
lion walking on a rope with closed eyes. While we have showcased Lego’s
ability with mostly two-word-embedding concepts, the synthetic example in
Figure 7 (right) illustrates its capability to invert more complex scenarios,
learning and applying “walking on a rope with closed eyes” from a single
example to the subject “lion”.

###  5.4 Ablation Study

We studied the effect of Subject Separation and Context Loss on inverting
three concepts; “burnt and melted”, “closed eyes” and “walking on rope”. TI
requires a few images of a subject or subjects in the same style (e.g., a few
Monet paintings), while ReVersion needs multi-subject images for relation
inversion. For the “burnt and melted” concept, we used both single and multi-
subject reference images to cover various settings. Our ablation study
compares Lego, incorporating both Subject Separation and Context Loss, with
three other combinations that remove one or both of these steps. Note that
single / multi-subject experiments with neither Subject Separation nor Context
Loss resemble performing a regular Textual Inversion [13]. Performing single /
multi-subject experiments by adding the Context Loss without Subject
Separation resembles the ReVersion framework [21]. Figure 6 illustrates the
ablation results for “burnt and melted,” highlighting the impact of Subject
Separation and the necessity of Context Loss for effective concept
representation. Complete ablation results are available in Supplementary
Section 11.

##  6 Conclusion

In this work, we took a first look at capabilities of T2I models in
synthesizing adjective and verb concepts. We showed that entanglement of such
concepts with a subject and the need for using multiple word embeddings in
describing more complex concepts hinders current inversion methods. We
proposed Lego that effectively inverts such concepts from as few as 4 example
images by disentangling concepts from subjects with a Subject Separation step
and further enables defining concepts with multiple embeddings by using a
Context Loss that guides each embedding towards a meaningful place in the
textual embedding space. The images generated by our method were clearly
preferred by humans over the baseline. We also showed that our learned
concepts can be combined together to generate images with a composition of
concepts. We seek to learn dynamic concepts using example videos in the
future.

##### Limitations and Ethical Statement:

Lego struggles to invert concepts beyond the capabilities of the backbone,
like facial expressions with earlier versions of Stable Diffusion. Our
research is dedicated to ethical and responsible data use, emphasizing the
need for socially responsible applications in personalized visual media.

## References

  * Amazon [2023] Amazon.  Amazon mechanical turk.  https://www.mturk.com/, 2023.
  * Balaji et al. [2022] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu.  ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers, 2022.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Chang et al. [2023] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al.  Muse: Text-to-image generation via masked generative transformers.  _arXiv preprint arXiv:2301.00704_ , 2023.
  * Chefer et al. [2023] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10, 2023.
  * Daniel et al. [2018] Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh.  Quality control in crowdsourcing.  _ACM Computing Surveys (CSUR)_ , 51:1 – 40, 2018.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Ding et al. [2021] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  _Advances in Neural Information Processing Systems_ , 34:19822–19835, 2021.
  * Face [2023] Hugging Face.  Diffusers library.  https://huggingface.co/docs/diffusers/index, 2023.
  * Feng et al. [2022] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  _arXiv preprint arXiv:2212.05032_ , 2022.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _European Conference on Computer Vision_ , pages 89–106. Springer, 2022.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–13, 2023.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Gu et al. [2022] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10696–10706, 2022.
  * Hendricks and Nematzadeh [2021] Lisa Anne Hendricks and Aida Nematzadeh.  Probing image-language transformers for verb understanding.  _arXiv preprint arXiv:2106.09141_ , 2021.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2023] Ziqi Huang, Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu.  Reversion: Diffusion-based relation inversion from images.  _arXiv preprint arXiv:2303.13495_ , 2023.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ , 34:852–863, 2021.
  * Kim et al. [2022] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2426–2435, 2022.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Le-Khac et al. [2020] Phuc H. Le-Khac, Graham Healy, and Alan F. Smeaton.  Contrastive representation learning: A framework and review.  _IEEE Access_ , 8:193907–193934, 2020.
  * Lee et al. [2023] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu.  Aligning text-to-image models using human feedback.  _arXiv preprint arXiv:2302.12192_ , 2023.
  * Liu et al. [2022] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In _European Conference on Computer Vision_ , pages 423–439. Springer, 2022.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6038–6047, 2023.
  * Momeni et al. [2023] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid.  Verbs in action: Improving verb understanding in video-language models.  _arXiv preprint arXiv:2304.06708_ , 2023.
  * Nichol et al. [2021a] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021a.
  * Nichol et al. [2021b] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021b.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Park et al. [2022] Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach.  Exposing the limits of video-text models through contrast sets.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3574–3586, 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rao et al. [2013] Huaming Rao, Shih-Wen Huang, and Wai-Tat Fu.  What will others choose? how a majority vote reward scheme can improve human computation in a spatial location identification task.  In _AAAI Conference on Human Computation & Crowdsourcing_, 2013.
  * Rombach et al. [2022a] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022a.
  * Rombach et al. [2022b] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022b.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Sohn et al. [2023] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al.  Styledrop: Text-to-image generation in any style.  _arXiv preprint arXiv:2306.00983_ , 2023.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Wu et al. [2023] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang.  Uncovering the disentanglement capability in text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1900–1910, 2023.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022.
  * Yuan et al. [2021] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta.  Multimodal contrastive training for visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6995–7004, 2021.
  * Yuksekgonul et al. [2022] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.

## Lego: Learning to Disentangle and Invert Concepts Beyond Object Appearance
in Text-to-Image Diffusion Models - supplemental document

  * •

To see Lego inverted concepts applied to more subjects, please see Figure 8.

  * •

If you are interested in details of the experimental design such as text
templates and how $\mathcal{P}$ and $\mathcal{N}$ were chosen for each
concept, please refer to Section 8.

  * •

For details on the prompts used in generating concepts using the baseline and
details on the VQA study, please see Section 9.

  * •

200 images generated using Lego and 200 images generated using the baseline
are shown in Figures 13 to 26.

  * •

We show how ELITE [47] and ControlNet [52] perform in inverting general
concepts in Section 10.

  * •

The full results for our ablation study can be found in Section 11.

##  7 Additional Results

In Figure 8, we show the results of Lego learned concepts, applied to four
additional subjects.

Figure 8:  We show Lego learned concepts applied to more subjects. please zoom
in to see details of images.

##  8 Experiment Design

In this section, we go over the design of each concept inversion experiment,
including the template of prompts used during textual inversion and selecting
the appropriate positive $\mathcal{P}_{i}$ and negative $\mathcal{N}_{i}$ set
of words for each concept embedding.

###  8.1 Numerical Concepts

In our work, we inverted concept numbers 3, 4 and 5 by using example images of
1 Lego figurine compared to 3, 4, and 5 Lego figurines, respectively. Figure 9
shows sample text templates used for inverting such concepts. We dedicated an
embedding $<$subj$>$ for learning the subject appearance and an embedding
$<$cpt$>$ to learn the concept. For all numerical concepts, one concept
embedding was used.

Figure 9: Sample templates for inverting numerical concepts. Single subject
images will be make up $\mathcal{I}_{\overline{C}}$ and the templates will
only have the $<$subj$>$ (left) while images with as many subjects as we are
trying to invert the number for will make up $\mathcal{I}_{C}$ with the
template having both $<$subj$>$ and $<$cpt$>$ embeddings (right).

To form the positive and negative sets of words that were used in our context
loss, we constructed the positive set $\mathcal{P}$ to be the digit and word
description of the desired number and further constructed the negative set
$\mathcal{N}$ to be the digit and word descriptions of a few neighbouring
numbers. For instance, to learn the concept of $4$ from the Lego figurines in
Figure 9, we set $\mathcal{P}=\\{\text{four},\text{4}\\}$ and
$\mathcal{N}=\\{\text{1, 2, 3, 5, 6, one, two, three, five, six}\\}$.

###  8.2 Remaining 7 Concepts

In Figure 10, we show template samples for the remaining 7 concepts. We skip
the subject-only templates as they follow the same structure as the one we
showed in Figure 9. To the right of each concept’s template, we provide each
embedding’s corresponding $\mathcal{P}$ and $\mathcal{N}$ that we used in our
experiments.

In order to select $\mathcal{P}$ and $\mathcal{N}$, we used a few synonyms for
the effect we are trying to invert. In the case of numerical concepts, our
negative set was the neighbouring numbers due to the fact that they are in
close embedding distance to the concept number. This is however not the case
in many examples. For instance, for the concept “ closed eyes”, the positive
set for one of the embeddings has the word “closed”. The antonym word “open”
however is not close to the embedding of “closed”. Our experiments showed that
in such cases, the negative set’s words is not as important as the positive
set, yet one should still select words based on the antonym scheme as
selecting arbitrary words can cause the embedding to diverge from a suitable
place in the text embedding space.

We suggest following our sanity check step when using Lego to invert arbitrary
concepts; during optimization, for each concept embedding, we monitor the 10
most and least similar words to the embedding. This allows you to see if your
embedding is indeed getting close to a set of words that describe the concept.
They could also suggest words for building the negative set if the embedding
is also getting close to undesired words that you had not thought of before.

Figure 10: Template samples for the non-numerical concepts explored in this
work. All 7 concepts use two word embedding and we show the positive and
negative words for each embedding.

##  9 LLM and Human Preference Study

In this section, we provide the 200 images generated with Lego and LDM for
each concept in our study. Both the VQA LLM (Flan) and Human preference
(Mechanical Turk) study preferred Lego generated concepts over the baseline;
LDM.

For our experiment with the VQA model, we experimented with asking different
types of questions based on the ground truth exemplar images. “Yes” or “No”
questions about the concept being present in the image or not led to the
highest correct answers based on the example images. Hence, we asked “Yes” or
“No” questions about each image and according to the LLM, Lego generated
concepts had more images aligned with the concept across all concepts (see
Table 2).

For each concept in the Human preference study, 10 sets of questions were
created, each containing 20 pairs of images, one generated using Lego’s
inversion of the concept synthesized through an LDM backbone and one generated
using natural language description (see next paragraph for details) of the
concept, synthesized through LDM. For each pair of images, users were prompted
to pick one image between the two that best represents the concept (users were
given both the exemplar images and text description of the concept for each
question - see Figure 11 for a snapshot of the user study interface for the
concept of walking on rope). We selected the image with majority user vote
[38, 6] for each pair as the preferred concept representation. Lego was
preferred over the baseline in all concepts as shown in Table 2.

Figure 11: The users were shown the image with instructions (left) on the
concept they should be looking for in each pair of images. And below the
instruction, 20 questions, each including 2 images (right). were given to
users where they had to pick one of the two. The order of Lego and baseline
generated concepts was randomized.

In Figures 13 to 26 show the 200 images generated for each concept using Lego
and natural language. With prompt tuning being an important aspect of getting
the desired outcome from generative models, for each concept we described the
concept in 4 different natural language prompts, with LDM generating 50 images
per prompt.

We applied the concept of frozen in ice to “toy bear” (see Figure 13) and we
used the following prompts for LDM; 1) “Photo of a toy bear frozen in ice”, 2)
“ a toy bear stuck in a block of ice”, 3) “a toy bear that is frozen in a
block of ice” and 4) “photo of a toy bear that is stuck in ice” (see Figure
14).

We applied the concept of burnt and melted to “toy bear” (see Figure 15) and
we used the following prompts for LDM; 1) “Photo of a toy bear that is burnt
and melted”, 2) “photo of a melted and burnt toy bear”, 3) “a toy bear that is
burnt and charred” and 4) “photo of a burnt and destroyed toy bear” (see
Figure 16).

We applied the concept of closed eyes to “lion” (see Figure 17) and we used
the following prompts for LDM; 1) “a lion with closed eyes”, 2) “photo of a
lion with its eyes closed”, 3) “photo of a lion with its eyes shut” and 4)
“photo of a lion that has closed eyes” (see Figure 18).

We applied the concept of smiley emoji face to “fluffy monster” (see Figure
19) and we used the following prompts for LDM; 1) “Photo of a fluffy monster
with a smiley emoji face”, 2) “photo of a fluffy monster with a yellow smiley
emoji head”, 3) “a fluffy monster with a yellow smiley emoji face” and 4)
“photo of a fluffy monster having a yellow smiley emoji mask” (see Figure 20).

We applied the concept of raised arms to “toy bear” (see Figure 21) and we
used the following prompts for LDM; 1) “Photo of a toy bear with its arms
raised”, 2) “ photo of a toy bear with arms up”, 3) “photo of a toy bear with
its hands up in the air” and 4) “ photo of a toy bear having raised its arms”
(see Figure 22).

We applied the concept of crumpled and crushed to “toy car” (see Figure 23)
and we used the following prompts for LDM; 1) “Photo of a toy car that is
crushed and crumpled”, 2) “ photo of a crushed and destroyed toy car”, 3)
“photo of a crumpled toy car” and 4) “ photo of a crushed and squeezed toy
car” (see Figure 24).

We applied the concept of walk on rope to “zebra” (see Figure 25) and we used
the following prompts for LDM; 1) “Photo of a zebra walking on a rope”, 2) “
photo of a zebra balancing on a tightrope”, 3) “a zebra walking on a
tightrope” and 4) “ photo of a zebra walking on a wire” (see Figure 26).

##  10 ControlNet and ELITE

recall that ELITE [47] trained mapping networks on a large vision dataset in
order to skip the embedding optimization step of Textual inversion by directly
learning to map an image’s subject to its corresponding embedding. Similar to
TI, ELITE is an appearance based inversion method and also requires an input
mask of the subject to be inverted. In Figure 12 we show ELITE’s attempt at
inverting concepts of walking on a rope, smiley emoji face, frozen in ice and
crumpled and crushed.

ControlNet [52] is a powerful method for adding conditional control to T2I
Diffusion models; such as pose and layout. In Figure 12 we condition Stable
Diffusion on the edge map of example images of our concepts and prompt the
model to follow these outlines. This is too restrictive for general concepts
and as seen in the four concept examples, the results are not satisfactory.

Figure 12:  Results of ControlNet and ELITE for inverting concepts of walking
on a rope, smiley emoji face, frozen in ice and crushed and crumpled Figure
13: Lego generated images with LDM backbone for the concept of frozen in ice,
learned from the Lego figurine frozen in ice example images, applied to a toy
bear. Figure 14: LDM generated images using 4 different prompts, describing a
toy bear that is frozen in ice. Figure 15: Lego generated images with LDM
backbone for the concept of burnt and melted, learned from the burnt and
melted Rubik’s cube example images, applied to a toy bear. Figure 16: LDM
generated images using 4 different prompts, describing a toy bear that has
been burnt and melted. Figure 17: Lego generated images with LDM backbone for
the concept of closed eyes, learned from the cat with closed eyes example
images, applied to a lion. Figure 18: LDM generated images using 4 different
prompts, describing a lion with its eyes closed. Figure 19: Lego generated
images with LDM backbone for the concept of smiley emoji face, learned from
the bird with smiley emoji face example images, applied to a fluffy monster.
Figure 20: LDM generated images using 4 different prompts, describing a fluffy
monster with a yellow smiley emoji face. Figure 21: Lego generated images with
LDM backbone for the concept of arms raised, learned from the Lego figurine
with its arms raised example images, applied to a toy bear. Figure 22: LDM
generated images using 4 different prompts, describing a toy bear with its
arms raised. Figure 23: Lego generated images with LDM backbone for the
concept of crushed and crumpled, learned from the crushed and crumpled soda
can example images, applied to a toy car. Figure 24: LDM generated images
using 4 different prompts, describing a toy car that is squished and crumpled.
Figure 25: Lego generated images with LDM backbone for the concept of walking
on rope, learned from the figure walking on a rope example images, applied to
a zebra. Figure 26: LDM generated images using 4 different prompts, describing
a zebra that is walking on a rope.

##  11 Complete Ablation Results

In Section 5.4 we showed a few ablation results for the concept of burnt and
melted. In Figures 27 and 28, we show the full ablation results for the single
and multi subject burnt and melted concept. In the single subject experiment,
we used $\mathcal{P}$ words that were more focused on burnt and destroyed
whereas in the multi-subject experiment we used words focused on melted and
liquefy. The effect is clear in Lego’s inversion results based on the choice
of positive word selection. We encourage the user to experiment with the words
that they would like to prominently see in the result concept. Figures 29 and
30 show the ablation results for learning the concepts of closed eyes and
walking on rope and applying them to “batman” and “a zebra” respectively.

The ablation for the three concepts clearly show the effect of Subject
Separation and Context Loss on the concept’s representation. In the single
subject melting examples, we can see Rubik’s cube’s features in the toy bear
and in the multi-subject scenario, both the features of the Rubik’s cubes and
Toy Story’s Woody can be seen in the toy bears (notice Woody’s eyes in some of
the bears). Furthermore, in the closed eyes example, the cat’s features can be
found in what is mean to represent Batman’s face. In the walking on rope
example, the figure from the example images is present instead of “a woman”.
The effect of context loss can be seen by comparing Lego generated concepts
with concepts generated by performing subject separation and not context loss.
Not performing context loss leaves most images unfaithful in representing the
concept.

Figure 27:  Ablation results for burnt and melted concept from single subject
example images applied to the subject “toy bear”. Please zoom in to see
details. Figure 28:  Ablation results for burnt and melted concept from multi
subject example images applied to the subject “toy bear”. Please zoom in to
see details. Figure 29:  Ablation results for closed eyes concept applied to
the subject “batman”. Please zoom in to see details. Figure 30:  Ablation
results for walking on rope concept applied to the subject “a woman”. Please
zoom in to see details.