# ComFusion: Personalized Subject Generation in Multiple Specific Scenes From
Single Image

Yan Hong1, Yuxuan Duan2, Bo Zhang2, Haoxing Chen1, Jun Lan1,
Huijia Zhu1, Weiqiang Wang1, Jianfu Zhang3
1Ant Group, 2MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong
University
3Qing Yuan Research Institute, Shanghai Jiao Tong University
<EMAIL_ADDRESS><EMAIL_ADDRESS>2{sjtudyx2016,bo-
<EMAIL_ADDRESS>
1{yelan.lj, huijia.zhj<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

Recent advancements in personalizing text-to-image (T2I) diffusion models have
shown the capability to generate images based on personalized visual concepts
using a limited number of user-provided examples. However, these models often
struggle with maintaining high visual fidelity, particularly in manipulating
scenes as defined by textual inputs. Addressing this, we introduce ComFusion,
a novel approach that leverages pretrained models generating composition of a
few user-provided subject images and predefined-text scenes, effectively
fusing visual-subject instances with textual-specific scenes, resulting in the
generation of high-fidelity instances within diverse scenes. ComFusion
integrates a class-scene prior preservation regularization, which leverages
composites the subject class and scene-specific knowledge from pretrained
models to enhance generation fidelity. Additionally, ComFusion uses coarse
generated images, ensuring they align effectively with both the instance image
and scene texts. Consequently, ComFusion maintains a delicate balance between
capturing the essence of the subject and maintaining scene fidelity. Extensive
evaluations of ComFusion against various baselines in T2I personalization have
demonstrated its qualitative and quantitative superiority.

Figure 1:  Contrasting with existing methods [31, 2] , which often face
challenges in simultaneously preserving instance fidelity and scene fidelity,
ComFusion skillfully composites the instance image with textual prompts and
fuses the visual details of the subject instance with the textual variations
of the scenes, yielding the creation of plausible, personalized images that
exhibit a rich diversity.

##  1 Introduction

Text-to-image (T2I) personalization aims to customize a diffusion-based T2I
model with user-provided visual concepts [31, 9]. This innovative approach
enables the creation of new images that seamlessly integrate these concepts
into diverse scenes. More formally, given a few images of a subject (no more
than five), our objective is embed this subject into the model’s output
domain. This allows for the synthesis of the subject with a unique identifier
in various scenes. The task of rendering such imaginative scenes is
particularly challenging. It entails the synthesis of specific subjects (e.g.,
objects, animals) in new contexts, ensuring their natural and seamless
incorporation into the scene. Such a task demands a delicate balance between
the subject’s distinctive features and the new scene context.

Recently, this field of T2I personalization has attracted significant
attention from the academic community with many works [2, 19, 9, 40, 41, 38,
1, 33] leveraging the capabilities of advanced diffusion-based T2I models [29,
28, 32]. These approaches broadly fall into two categories: The first category
[10, 6, 33, 21, 17] integrates additional modules with a pretrained base
model. This stream enables the creation of personalized subjects without the
need for finetuning during testing. However, it often struggles to maintain
the subject’s identity consistently across different synthesized images. In
contrast, the second category [31, 9, 19, 2] infocuses on finetuning the
pretrained model with a select set of images, employing various regularization
techniques and training strategies. This finetuning process effectively
harnesses the model’s existing class knowledge, combined with the unique
identifier of the subject, thereby allowing for the generation of diverse
variations of the subject in various contexts.

However, finetuning diffusion models presents a challenge known as language
drift [25, 20], where the finetuned model may lose its pre-finetuning acquired
knowledge, including the understanding of a diverse range of classes and
scenes. Furthermore, few-shot learning paradigms are susceptible to
catastrophic neglecting, particularly when generating new images with specific
scenes, leading to inadequate generation or integration of some prompts or
subjects. These issues contribute to a notable decrease in both scene fidelity
and instance fidelity. In Fig. 1, we show an example of the personalized
generation using a specific dog instance image and the text prompt “A dog in
the rain”. The images, generated by existing leading methods [31, 2] and our
proposed approach. Fig. 1 (a) illustrates a lack of instance fidelity, where
the generated images fail to preserve the subject dog’s appearance, resulting
in low-personality output. Fig. 1 (b) highlights examples with insufficient
scene fidelity, failing to accurately represent the rainy scene, thus limiting
the diversity of the generated images.

To address the issues of language drift and catastrophic neglecting and to
improve the instance fidelity and scene fidelity of generated images, we
propose ComFusion (Composite and Fusion). This innovative approach is tailored
to enable personalized subject generation across diverse scenes. ComFusion
utilizes a finetuning strategy to effectively composite and fuse visual
subject features with textual features, thus synthesizing new images with
high-fidelity instances composed with a variety of distinct scenes. To achieve
this, we have established two streams: a composite stream and a fusion stream.
Within the composite stream, we introduce class-scene prior loss to preserve
the pretrained model’s knowledge of class and scene. This strategy leads to
the generation of a diverse array of images that composite the essence of
class and scene priors with subject instances and their contexts, alleviating
language drift in scene representation, enhancing coherent syntheses of both
subject instances and scene contexts. In the fusion stream, we propose a
visual-textual matching loss to effectively fuse the visual information of the
subject instance with the textual information of the scene, ensuring their
integrated representation in the coarsely generated images. Through the fusion
stream, ComFusion mitigates the catastrophic neglecting issue, achieving a
harmonious balance between instance fidelity and scene fidelity. In Fig. 1, we
present some impressive samples obtained by ComFusion. The images illustrate a
remarkable preservation of the dog’s appearance, while the scene ‘in the rain’
is brought to life with vivid details such as rain spots and umbrellas. To
further substantiate ComFusion’s effectiveness, we have conducted extensive
experiments across various subject instances and scenes. These studies confirm
that ComFusion excels both qualitatively and quantitatively, outperforming
existing methods.

##  2 Related Works

Diffuion-Based Text-to-Image Generation: The field of Text-to-Image (T2I)
generation has recently witnessed remarkable advancements [43, 18, 36, 37, 32,
11], predominantly led by pre-trained diffusion models such as Stable
Diffusion [29], DALLE [28], Imagen [32] and etc. These models are renowned for
their exceptional control in producing photorealistic images that closely
align with textual descriptions. Despite their superior capabilities in
generating high-quality images, these models encounter challenges in more
personalized image generation tasks , which are often difficult to be
precisely described with text descriptions. This challenge has sparked
interest in the rapidly evolving field of personalized T2I generation [31, 9,
19, 38, 24].

Personalized Text-to-Image Generation: Given a small set of images of the
subject concept, personalized T2I generation [31, 9, 19, 10, 40, 24, 41, 38,
33, 1, 5, 12, 34, 28, 32] aims to generate new images according to the text
descriptions while maintaining the identity of the subject concept. Early
studies in training generative models in few-shot setting focus on alleviating
mode collapse [35, 23, 39] for generative adversarial networks [7, 13, 8, 42,
14, 15, 22]. Recently, finetuning diffusion-based text-to-image model with a
few images has also been explored in [31, 2]. In stream of diffusion-based
generators, personalized T2I generation methods can be classified into two
categories: The first stream involves the integration of additional modules
(e.g., [26, 16, 43]) with a pretrained base model. The second stream adopts a
strategy of finetuning the pretrained model using a few selected images.

Without Finetuning: These methods [10, 6, 33, 21, 17] generally rely on
additional modules trained on additional new datasets, such as the visual
encoder in [33, 41] and the experts in [6, 21] to directly map the image of
the new subject to the textual space. Specifically, [10] introduces an encoder
that encodes distinctive instance information, enabling rapid integration of
novel concepts from a given domain by training on a diverse range of concepts
within that domain. In [33], a learnable image encoder translates input images
into textual tokens, supplemented by adapter layers in the pre-trained model,
thus facilitating rich visual feature representation and instant text-guided
image personalization without requiring test-time finetuning. DisenBooth [5]
uses weak denoising and contrastive embedding auxiliary tuning objectives for
personalization. ELITE [41] introduces a method for learning both local and
global maps on large-scale datasets, allowing for instant adaptation to unseen
instances using a single image marked with the subject concept for
personalized generation.

Finetuning: Various methods employ diverse training strategies to optimize
different modules within pretrained models [31, 9, 19]. DreamBooth [31] and TI
[9] are two popular subject-driven text-to-image generation methods based on
finetuning. Both approaches map subject images to a special prompt token
during finetuning. They differ in their finetuning focus: TI concentrates on
prompt embedding, while DreamBooth targets the U-Net model and text-encoder.
Recent finetuning-based methods [19, 40] focus on how to design training
strategy to update core parameters of T2I model for subject concepts on user-
provided 4-6 images. A domain-agnostic method is proposed in [1] that
introduce a novel contrastive-based regularization technique. This technique
aims to preserve high fidelity to the subject concept’s characteristics while
keeping the predicted embeddings close to editable regions of the latent
space. Break-A-Scene [2] utilizes the subject concept’s mask and employs a
two-stage process for personalized T2I generation using a single image.
However, this approach is limited in terms of the subject’s diversity. In this
paper, we concentrate on advancing personalized T2I generation. Our goal is to
establish an ideal balance between fidelity to the subject concept and
adaptability to multiple specific scenes.

Figure 2:  The illustration of ComFusion finetuning framework. We show an
example of one-shot personalized generation setting and please note ComFusion
can be applied to few-shot settings. ComFusion consists of a composite stream
(highlighted with green and orange arrows, details in Sec. 3.1 and Sec. 3.2)
and a fusion stream (highlighted with blue arrows, details in Sec. 3.3).

##  3 ComFusion

In this section, we introduce ComFusion, our novel method designed to
facilitate personalized subject generation across various specific scenes.
Formally, with a limited number of subject instance images, typically no more
than five, our objective is to synthesize new images of the subject with high
detail fidelity, guided by text prompts. These prompts drive variations such
as changes in the subject’s location, background, pose, viewpoint, and other
contextually relevant transformations. It’s important to note that there are
no constraints on the capture settings of the subject instance images; they
can represent a wide range of scenarios. IIn the main body of our paper, we
focus on the particularly challenging one-shot setting, where only a single
instance image is used. The generated images aim to be faithful (i.e.,
accurately reflect the content) both the subject instance and the text
prompts, which manifests in two key aspects - instance fidelity: ensuring
visual congruence with the instance image and scene fidelity: aligning the
scenes in the newly created images with the provided prompts. As shown in Fig.
2, we design a two-stream training strategy for ComFusion, consisting of a
composite stream supervised by instance finetune loss and class-scene prior
loss (denoted as $\\{\mathcal{L}_{C}^{I},\mathcal{L}_{C}^{S}\\}$ in green and
orange stream of Fig. 2, also demonstrated in Sec. 3.2) and a fusion stream
supervised by visual-textual matching loss (denoted as
$\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$ in blue stream of Fig. 2, also
demonstrated in Sec. 3.3).

###  3.1 Finetuning Text-to-Image Diffusion Models

ComFusion will finetune specific pretrained diffusion models, e.g., Stable
Diffusion [29], which consists of an auto-encoder (encoder $E$ and a decoder
$D$), text-encoder $\Gamma$, and a denoising model $\epsilon_{\theta}$
architectured with $UNET$ [30]. The auto-encoder subjects at map an image
$\bm{x}$ into low-dimensional latent $\bm{z}$ with encoder $E$ and recover the
original image $\tilde{x}$ with decoder $D$ after the denoising process. The
denoising model $\epsilon_{\theta}$ is trained on the latent to produce
subject latent based on the textual condition source from $\Gamma(\bm{T})$,
where $\bm{T}$ is the user-provided prompt providing the information (e.g.,
subject classes, instance attributes, scenes) of the generated images and
$\Gamma$ denotes the pretrained CLIP text encoder [27]. Given a single subject
instance image $\bm{x}^{I}$ from a subject class, the instance image is
captioned with instance text $\bm{T}^{I}$ = “a [identifier] [class noun]”
(_e.g.,_ “a sks dog”), where “[class noun]” is a coarse class (_e.g.,_ “dog”)
provided by user and “[identifier]” is an unique identifier for subject
concept (_e.g.,_ “sks”). Given a single instance image $\bm{x}^{I}$, the
pretrained models will be finetuned with instance finetune loss:

$\displaystyle\mathcal{L}_{C}^{I}=\mathbb{E}_{\bm{z}\sim\\{\bm{z}^{I}\\},\bm{\epsilon},t}\left[\|\bm{\epsilon}-\epsilon_{\theta}(\bm{z}_{t},t,\Gamma(\bm{T}^{I}))\|_{2}^{2}\right],$
(1)

where $t\sim\mathcal{N}(0,1)$ is the time step,
$\bm{\epsilon}\sim\mathcal{N}(0,I)$ denotes the unscaled noise sampled from
Gaussian distribution, and $\bm{z}_{t}$ are the noisy latent at time $t$,
$\\{\bm{z}^{I}\\}$ represents latent of instance images $\\{\bm{x}^{I}\\}$
processed by encoder $E$. This finetuning process enables the “implantation”
of a new (unique identifier, subject) pair into the diffusion model’s
“dictionary”. The fundamental aim is to leverage the model’s existing class
knowledge and intertwine it with the embedding of the subject’s unique
identifier. This approach enables the effective utilization of the model’s
visual priors for generating novel variations of the subject in diverse
contexts.

The finetuning of diffusion models brings about the challenge of language
drift [25, 20], a phenomenon where models gradually stray from the syntactic
and semantic complexities of language, focusing too intently on task-specific
details. In our research, this issue emerges prominently: the finetuned model
tends to lose the knowledge it acquired before finetuning, such as
comprehending various classes and scenes that are inherent to pretrained
models. This often leads to a significant reduction in scene fidelity. To
tackle this, existing methods employ a specific prior loss to regularize the
model. Typically, this loss function involves using designated prior texts
$\\{\bm{T}^{P}_{i}|_{i=1}^{N}\\}$, input into the pretrained model to generate
prior images $\\{\bm{x}^{P}_{i}|_{i=1}^{N}\\}$ based on prior texts, where $N$
is the number of prior text-image pairs. Such a loss function ensures the
model’s adherence to its pre-trained knowledge base, thus preserving essential
foundational knowledge before embarking on few-shot finetuning. An example or
prior loss is class-specific prior loss proposed by DreamBooth [31]. It
leverages coarse class to form class text $\bm{T}^{C}=$ “a [class noun]”
(_e.g.,_ “a dog”), which is fed into the pretrained model to produce class
prior images $\\{\bm{x}^{C}_{i}|_{i=1}^{N}\\}$ of the coarse class. Those
class prior images resemble the subject class images but has no requirement of
subject instance preservation. Given class prior images
$\\{\bm{x}^{C}_{i}|_{i=1}^{N}\\}$, the pretrained models will be finetuned
with:

$\displaystyle\mathcal{L}_{dream}=\mathbb{E}_{\bm{z}\sim\\{\bm{z}^{C}\\},\bm{\epsilon},t}\left[\|\bm{\epsilon}-\epsilon_{\theta}(\bm{z}_{t},t,\Gamma(\bm{T}^{C}))\|_{2}^{2}\right],$
(2)

where $\\{\bm{z}^{C}\\}$ represents latent of class prior images
$\\{\bm{x}^{C}\\}$ and the other terms are defined similar to Eq. 1. The
class-specific prior-preservation loss supervises the model with
reconstruction of class prior images, and will be trained together with Eq. 1
with a weight hyperparameter. This loss function is specifically tailored to
preserve the unique attributes of instance images related to their class. It
utilizes semantic priors associated with the class, integrated into the
model’s structure, to facilitate the generation of varied instances within the
subject’s class. Intuitively, models trained with $\mathcal{L}_{C}^{I}$ and
$\mathcal{L}_{dream}$, leveraging the knowledge from large-scale T2I
pretrained models capable of generating images with any scenes, should ideally
preserve both instance and scene fidelity. However, this approach primarily
addresses language drift related to the subject class but may overlook drifts
in text prompts describing the scenes of the generated images, leading to
catastrophic neglecting [4]. In large-scale pretrained T2I models like Stable
Diffusion [29], models trained on a vast array of image-text pairs demonstrate
proficiency in generating novel images based on combinations of random texts.
Nonetheless, the neglecting phenomenon remains an issue in certain scenarios,
where some prompts or subjects are not adequately generated or integrated by
these large-scale models [4]. In contrast, few-shot learning paradigms,
relying on a limited set of image-text pairs, often yield less optimal
responses to complex subject instances and scene texts than their large-scale
counterparts, potentially exacerbating the neglecting issue. This can lead to
low diversity in outputs and loss to scene fidelity or adequately respond to
instance images for personalized generation, as illustrated in Fig. 1.

###  3.2 Composite: Class-Scene Prior Loss

Given our objective to composite new subject instance images within various
specific scenes, we updated the prior-preservation loss in Eq. 2 to class-
scene prior loss. This update is specifically designed to maintain the
pretrained model’s knowledge of both class and scene, thereby significantly
enhancing scene fidelity. By integrating class-scene prior loss with instance
finetune loss, ComFusion effectively preserves and leverages the extensive
understanding of class and scene inherent to the pretrained model. To
elaborate, we initially generate a descriptive class-scene text set
$\\{\bm{T}^{CS}\\}$. This set composites the subject class information “[class
noun]” (_e.g.,_ “dog”) and scene information “[scene]” (_e.g.,_ “in the
rain”), resulting in class-scene texts “a [class noun] [scene]” (_e.g.,_ “a
dog in the rain”). The class-scene prior images $\\{(\bm{x}^{CS}\\}$ is
generated by pretrained model with the corresponding $\\{\bm{T}^{CS}\\}$.
Subsequently, these richly detailed class-scene image-text pairs
$(\bm{x}^{CS}_{k},\bm{T}^{CS}_{k})$ combined with instance image-text pairs
$(\bm{x}^{I},\bm{T}^{I})$ are fed into ComFusion to finetune the diffusion
model. In a manner akin to Eq. 2, the trainable parameters of ComFusion are
optimized by class-scene prior loss:

$\displaystyle\mathcal{L}_{C}^{S}=\mathbb{E}_{(\bm{z},\bm{T})\sim\\{(\bm{z}^{CS},\bm{T}^{CS})\\},\bm{\epsilon},t}\left[\|\bm{\epsilon}-\epsilon_{\theta}(\bm{z},t,\Gamma(\bm{T}))\|_{2}^{2}\right],$
(3)

where $\\{(\bm{z}^{CS},\bm{T}^{CS})\\}$ is the set of latent-text pairs
corresponding to $\\{(\bm{x}^{CS},\bm{T}^{CS})\\}$. Similar to
$\mathcal{L}_{dream}$ in Eq. 2, this class-scene prior loss will be trained
together with instance finetune loss $\mathcal{L}_{C}^{I}$ in Eq. 1 with a
$\lambda_{C}$ controls for the relative weight of this term.
$\mathcal{L}_{C}^{I}$ and $\mathcal{L}_{C}^{S}$ formulate the objective of the
composite stream in ComFusion. Different from $\mathcal{L}_{dream}$, the
class-scene prior text $\bm{T}^{CS}$ in $\mathcal{L}_{C}$ articulates a
comprehensive delineation of subject class information and meticulous scene
descriptors as specified by $\bm{T}^{CS}$. This loss formulation adeptly
tackles the language drift issue related to class and scene knowledge in the
finetuned model. It facilitates the generation of a varied collection of
images that capture the essence of both class and scene priors from the
pretrained model, while integrating these elements with subject instances and
their specific contexts. Such integration enables ComFusion to produce images
that accurately depict subject instances within their respective scenes.

Figure 3: The coarse generated results $\tilde{\bm{x}}^{IS}_{k}$ by
supervision of visual-textual fusion loss
$\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$ under denoising steps $\tau$
sampled from $\\{1,3,5\\}$. The instance in coarse generated images is similar
to instance image $\bm{x}^{I}$ , while it maintains consistent with specific
scene in prior images $\bm{x}^{CS}_{k}$.

###  3.3 Fusion: Visual-Textual Matching Loss

In ComFusion, the composite stream effectively finetunes the instance image
while retaining the class-scene prior knowledge embedded in the pretrained
model. This approach effectively maintains both instance fidelity and scene
fidelity. Furthermore, ComFusion employs a visual-textual matching loss to
fuse the visual information of the subject instance with the textual context
of the scene, further enhancing both instance fidelity and scene fidelity.
This ensures a more coherent and accurate depiction of the subject instance
and the scenes. The key idea behind the visual-textual matching loss is to
generate coarse images that encapsulate both the subject instance and scene
texts. This loss then guarantees that the coarsely generated images
effectively merge the specific visual details of the instances with the
textual nuances of the scenes, aligning them with the instance image and scene
texts. Specifically, for a class-scene prior image $\bm{x}^{CS}$ annotated
with detailed class-scene text $\bm{T}^{CS}$, according to the standard
forward process of DDPMs [12], we add a random and appropriate level of noise
$\bm{\epsilon}_{t}$ with timestep $t$ to obtain noisy latent
$\bm{z}^{CS}_{t}$. This is designed to infuse new information while preserving
the structural features of class-scene prior images. We then create instance-
scene text $\bm{T}^{IS}$ by replacing the “[class noun]” with “[identifier]
[class noun]” in class-scene text (_e.g.,_ “a sks dog in the rain”). This
modified text is processed through by the text-encoder $\Gamma$ to obtain
conditional textual information, which is then used to iteratively denoise the
noisy latent $\bm{z}^{CS}_{t}$ to denoised latent $\tilde{\bm{z}}^{IS}$.
Adopting an accelerated generation process in DDIMs [34], we set $\tau$ as the
number of steps required to denoise $\bm{z}^{CS}_{t}$ into
$\tilde{\bm{z}}^{IS}$, with a function expressed as:

$\tilde{\bm{z}}^{IS}=f_{\theta}(\bm{z}^{CS}_{t},\bm{T}^{IS},\tau).$ (4)

Generally, $f_{\theta}$ significantly reduces the computational effort
required for denoising from $t$ to $\tau$, producing a coarse denoised latent.
The specifics of this denoising function $f_{\theta}$ in Eq. 4 will be
detailed in the supplementary materials. The denoised latent
$\tilde{\bm{z}}^{IS}$ is then decoded using decoder $D$, resulting in a coarse
denoised image $\tilde{\bm{x}}^{IS}=D(\tilde{\bm{z}}^{IS})$. This image,
guided by the instance-scene text, fuses both the subject and the scene’s
features. Fig. 3 illustrates examples of coarse denoised images under
different settings of $\tau$. We enable the instance image
$\tilde{\bm{x}}^{IS}$ to have similar textural structure (_resp.,_ visual
appearance) of class-scene prior image $\bm{x}^{CS}$ (_resp.,_ instance image
$\bm{x}^{I}$) by a pair of visual-textual fusion loss:

$\displaystyle\mathcal{L}_{F}^{I}=\mathbb{E}_{\bm{x}\sim\\{\tilde{\bm{x}}^{IS}_{k}\\}}\left[-\textbf{DINO}(\bm{x},\bm{x}^{I})\right],$
(5)
$\displaystyle\mathcal{L}_{F}^{S}=\mathbb{E}_{(\bm{x}^{\prime},\bm{T})\sim\\{(\tilde{\bm{x}}^{IS}_{k},\bm{T}^{CS}_{k})\\}}\left[-\textbf{CLIP}(\bm{x}^{\prime},\bm{T})\right],$

where the first (_resp.,_ second) term is represented by $\mathcal{L}^{I}_{F}$
(_resp.,_ $\mathcal{L}^{S}_{F}$) in Fig. 2. DINO [3] is a self-supervised
pretrained transformer, renowned for its proficiency in extracting visual
information from images. Utilizing self-supervised learning techniques, DINO
effectively discerns and encodes complex visual patterns, making it
exceptionally suitable for image analysis tasks. In contrast, CLIP [27] is at
the forefront of image-text cross-modality pretraining. It is designed to
align visual information with corresponding textual data, thus enabling the
model to comprehend and relate the contents of images with their relevant
textual descriptions. CLIP’s ability to bridge visual and textual domains
renders it an invaluable asset for tasks that demand a thorough understanding
of both visual and textual elements, facilitating precise and effective image-
text alignments. Hence, in the fusion stream of our approach, we employ DINO
for visual matching and CLIP for textual matching, leveraging the strengths of
each model to enhance the overall efficacy of our method.
$\textbf{DINO}(\tilde{\bm{x}}^{IS}_{k},\bm{x}^{I})$ is used to calculate the
cosine similarity between DINO embedding of $\tilde{\bm{x}}^{IS}_{k}$ and
$\bm{x}^{I}$ with pretrained ViT-S/16 DINO [3], the larger similarity means
that generated $\tilde{\bm{x}}^{G}_{k}$ is more similar to instance image
$\bm{x}^{I}$. $\textbf{CLIP}(\tilde{\bm{x}}^{IS}_{k},\bm{T}^{CS}_{k})$ aims at
calculating the cosine similarity of visual embedding of generated image
$\tilde{\bm{x}}^{IS}_{k}$ and textual embedding of class-scene prior text
$\bm{T}^{CS}_{k}$. To the larger similarity, the trained model tend to produce
new images including specific scene information. By applying both visual loss
$\mathcal{L}^{I}_{F}$ and textual loss $\mathcal{L}^{S}_{F}$ on the same
generated image $\tilde{\bm{x}}^{IS}_{k}$, ComFusion mitigates the
catastrophic neglecting problem and achieves a better balance between instance
fidelity and scene fidelity. As a result, it yields a more harmonious and
precise depiction that effectively captures the core characteristics of
instance fidelity and scene fidelity.

###  3.4 Overall Objectives and Inference Process

ComFusion’s objective function integrates the instance finetune loss in Eq. 1
is combined with the class-scene prior loss in Eq. 3 and visual-textual fusion
loss in Eq. 5:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{C}^{I}+\lambda_{C}^{S}\mathcal{L}_{C}^{S}+\lambda_{F}^{I}\mathcal{L}_{F}^{I}+\lambda_{F}^{S}\mathcal{L}_{F}^{S},$
(6)

where $\lambda_{C}^{S}$, $\lambda_{F}^{I}$, and $\lambda_{F}^{S}$ represent
the respective weights of $\mathcal{L}^{S}_{C}$, $\mathcal{L}^{I}_{F}$, and
$\mathcal{L}^{S}_{F}$. $\mathcal{L}_{total}$ is employed to finetune the
trainable parameters of text-encoder $\Gamma$ and UNET $\epsilon_{\theta}$
based on pretrained Stable Diffusion [29]. During this process, the parameters
of the auto-encoders remain fixed. In the inference phase, ComFusion follows
the standard T2I inference protocol: generating a random latent, followed by
denoising this latent using the prompt “a [identifier] [class noun] [scene]”
with the UNet. Finally, the denoised latent is decoded to produce new images.

##  4 Experiments

###  4.1 Experimental Settings and Details

Implementation Details. All methods were applied using a pre-trained Stable
Diffusion (SD) checkpoint 1.5 [29]. We trained ComFusion and DreamBooth for
1200 steps, using a batch size of 1 and learning rate $1\times 10^{-5}$. The
number of prior images $N$ is set as $200$ for fair comparison. During
training, the hyper parameters $\lambda_{C}^{S}$ (_resp.,_ $\lambda_{F}^{S}$,
$\lambda_{F}^{I}$,$\tau$) is set as $1$ (_resp.,_ $0.01$, $0.01$,$3$). All
experiments are conducted with $1$ $A100$ GPU. Detailed implementation
information for all baselines is provided in Supplementary.

Datasets. To evaluate the effectiveness of our proposed methods among
different datasets, we use a combined dataset of the TI [9] dataset of 5
concepts, and the dataset from DreamBooth [31] with 20 concepts. For both
datasets, each concept selects one original image as instance image. We
perform experiments on $25$ subject datasets spanning a variety of categories
and varying training samples. We evaluate all the methods with $15$ distinct
scenes. Also, we use $\bm{T}^{CS}=$ “a [class noun] Scene” with the same scene
prompts to sample prior images with $15$ scenes for ComFusion. Detailed
information about the subject datasets and scene prompts is available in the
supplementary materials. Experiments involving more than one instance image,
other specific scenes, and scenarios without specific scenes for all methods
are also documented in Supplementary.

Baselines. We compare our ComFusion described in Section Sec. 3 with
DreamBooth [31], Textual-Inversion(TI) [9], Custom-Diffusion (CD) [19],
Extended Textual-Inversion (XTI) [40], ELITE [41], and Break-A-Scene [2].
Details of these baseline methods are reported in Supplementary.

Evaluation Metrics. Following DreamBooth [31], for each method, we generated
$10$ images for each of $25$ instances and each of $15$ scenes, totaling
$3750$ images for evaluation of robustness and generalization abilities of
each method. Following DreamBooth [31] and CD [19], we evaluate those methods
on two dimensions including instance fidelity and scene fidelity. CLIP-I [27]
and DINO score [3] were used to evaluate instance fidelity by measuring the
similarity between generated images and instance images, and the alignment
between textual scene with generated images are measured by CLIP-T [27].
Detailed descriptions of these measurement metrics are provided in
Supplementary.

Table 1: Quantitative metric comparison of instance fidelity (DINO, CLIP-I)
and scene fidelity (CLIP-T).

Methods | DINO ($\uparrow$) | CLIP-I ($\uparrow$) | CLIP-T ($\uparrow$)
---|---|---|---
Real Images | 0.795 | 0.859 | N/A
DreamBooth [31] | 0.619 | 0.752 | 0.229
TI [9] | 0.465 | 0.634 | 0.185
CD [19] | 0.615 | 0.724 | 0.205
XTI [40] | 0.435 | 0.601 | 0.198
ELITE [41] | 0.405 | 0.615 | 0.249
Break-A-Scene [2] | 0.632 | 0.771 | 0.294
Ours | 0.658 | 0.814 | 0.321

Table 2: Instance fidelity and scene fidelity user preference.

Methods | Instance fidelity ($\uparrow$) | Scene fidelity ($\uparrow$)
---|---|---
DreamBooth [31] | 3.1% | 3.8%
TI [9] | 0.3% | 0.0%
CD [19] | 6.2% | 1.0%
XTI [40] | 0.3% | 1.8%
ELITE [41] | 0.0% | 11.1%
Break-A-Scene [2] | 34.5% | 20.2%
Ours | 55.6% | 62.1%

Figure 4: Images generated by DreamBooth [31], TI [9],CD [19],XTI [40],ELITE
[41], Break-A-Scene [2], and our proposed ComFusion in multiple specific
scenes from a single instance image.

###  4.2 Comparisons with Personalized Generation Baselines

We perform quantitative and qualitative evaluation on the instance fidelity
and scene fidelity of generated images. Instance fidelity assesses how well
the generated image maintains the identity of the original instance image,
while scene fidelity evaluates the semantic similarity between the generated
image and the input textual scene.

####  4.2.1 Quantitative Evaluations

Quality Assessments of Generated Images. We perform the quantitative
evaluation on the instance fidelity using DINO score and CLIP-I score, and
scene fidelity with CLIP-T score. In Tab. 1, “Real Images” represents measure
the similarity between given single image and remaining real images belonging
to the same subject as given image, providing upper bound of fidelity of
subject. Comparisons in Tab. 1 indicate that our ComFusion achieves the
highest scores for DINO, CLIP-I, and CLIP-T, indicating that it can generate
high-fidelity images with higher instance fidelity and scene fidelity than
baseline methods.

Human Perceptual Study. Further, following DreamBooth [31], we conduct a user
study to evaluate the instance fidelity and scene fidelity of generated
images. In detail, based on generated $3750$ images per method including $6$
baseline methods and our method, we present results generated from different
methods in random order and we ask 12 users to choose. (1) Instance fidelity:
determining which result better preserves the identity of the instance image,
and (2) Scene fidelity: evaluating which result achieves better alignment
between the given textual scene and the generated image. We collect 90,000
votes from 12 users ($12\times 3750\times 2$) for instance fidelity and scene
fidelity, and show the percentage of votes for each method in Tab. 2. The
comparison results demonstrate that the generated results obtained by our
method are preferred more often than those of other methods.

####  4.2.2 Qualitative Evaluations

To evaluate the superiority of our ComFusion in balancing the accuracy of
subjects and the consistency of multiple specific scenes, we visualize
comparison results in Fig. 4. We can see that images generated by TI [9], CD
[19], and XTI [40] are similar to input instance image in terms of structure,
those methods fail to make response to the specific scene in given testing
prompts. ELITE [41] may generate distorted images in unexpected scenes. Images
generated by Break-A-Scene [2] maintain instance fidelity while may fail to
composite subject instance in specific scenes. In contrast, our ComFusion can
generate images of higher instance fidelity and scene fidelity. This is
attributed to the class-scene prior loss can introduce specific scene
information during the process of learning subject instance and visual-textual
matching loss can enhance the fusion between visual instance image and textual
scene context.

###  4.3 Ablation Studies

Effect of Class-Scene Prior Loss Compared with prior preservation loss (Eq. 2)
proposed in DreamBooth [31], our class-scene prior loss $\mathcal{L}_{C}^{S}$
(Eq. 3) utilizes detailed texts for prior images to incorporate multiple
specific scenes. During training, this loss explicitly enforces the model to
retain prior scene knowledge while incorporating new information from instance
images within these scenes. From the visual comparison between ComFusion(“w/o
visual-textual loss $\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$”) and
DreamBooth [31] in Fig. 1, Fig. 4, and quantitative results in Tab. 3, we can
see that class-scene prior loss $\mathcal{L}_{C}^{S}$ significantly improves
the CLIP-T score while achieves comparable CLIP-I(_resp.,_ DINO) score, which
indicates that it can effectively improve scene fidelity without undermining
the instance fidelity.

Table 3: Ablation studies of each loss and alternative designs. Time
represents the totoal training time on 1 A100 GPU.

Methods | DINO ($\uparrow$) | CLIP-I ($\uparrow$) | CLIP-T ($\uparrow$) | Time(/s)
---|---|---|---|---
Real Images | 0.795 | 0.859 | N/A | N/A
DreamBooth | 0.619 | 0.752 | 0.229 | 491.8
Ours (w/o $\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$) | 0.627 | 0.771 | 0.301 | 597.0
Ours (w/o $\mathcal{L}_{F}^{I}$) | 0.586 | 0.697 | 0.342 | 597.3
Ours (w/o $\mathcal{L}_{F}^{S}$) | 0.716 | 0.828 | 0.189 | 597.1
Ours ($\tau=1$) | 0.641 | 0.806 | 0.334 | 537.9
Ours ($\tau=5$) | 0.698 | 0.825 | 0.309 | 623.1
Ours ($\tau=3$) | 0.658 | 0.814 | 0.321 | 597.7

Figure 5: Visual ablative results of ComFusion.

Effect of Visual-Textual Matching Loss We further conduct ablation to evaluate
the effect of the proposed visual-textual loss
$\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$ in Eq. 5. To evaluate the
effect of each item in visual-textual loss, we alternatively remove
$\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$ (_resp.,_
$\mathcal{L}_{F}^{I}$, $\mathcal{L}_{F}^{S}$) from total loss function in Eq.
6, and report visual results in Fig. 5 and quantitative results in Tab. 3. The
comparison results indicate that
$\\{\mathcal{L}_{F}^{I},\mathcal{L}_{F}^{S}\\}$ are well-design to balance the
instance fidelity and scene fidelity, removing them degrades the instance
fidelity in generated images in 2nd row in Fig. 5. To further study effect of
each item in visual-textual loss, removing $\mathcal{L}_{F}^{S}$ leads to
degrading the score of DINO and CLIP-I reflecting by poor instance fidelity in
4th row in Fig. 5, while lower scene fidelity in 3rd row in Fig. 5 is caused
by removing $\mathcal{L}_{F}^{C}$.

Effect of Denoising Timesteps $\tau$ To assess the impact of timesteps $\tau$
in the fusion stream, we experimented with varying $\tau$ values from
$\\{1,3,5\\}$ to train ComFusion. Our observations indicate that a larger
$\tau$ value tends to better preserve instance fidelity but at the expense of
reduced scene fidelity. We selected $\tau=3$ as the default setting for
ComFusion, considering time cost and a balance between instance fidelity and
scene fidelity

##  5 Conclusions

In this paper, we present ComFusion, a novel approach designed to facilitate
personalized subject generation within multiple specific scenes from a single
image. ComFusion introduces a class-scene prior loss to composite knowledge of
subject class and specific scenes from pretrained models. Moreover, a visual-
textual matching loss to further improve the fusion of visual object feature
and textual scene feature. Extensive quantitative and qualitative experiments
demonstrate the effectiveness of ComFusion.

## References

  * Arar et al. [2023] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H Bermano.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06925_ , 2023.
  * Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  _arXiv preprint arXiv:2305.16311_ , 2023.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chefer et al. [2023] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-Excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–10, 2023.
  * Chen et al. [2023a] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation.  _arXiv preprint arXiv:2305.03374_ , 2023a.
  * Chen et al. [2023b] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  _arXiv preprint arXiv:2304.00186_ , 2023b.
  * Clouâtre and Demers [2019] Louis Clouâtre and Marc Demers.  Figr: Few-shot image generation with reptile.  _arXiv preprint arXiv:1901.02199_ , 2019.
  * Ding et al. [2022] Guanqi Ding, Xinzhe Han, Shuhui Wang, Shuzhe Wu, Xin Jin, Dandan Tu, and Qingming Huang.  Attribute group editing for reliable few-shot image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11194–11203, 2022.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–13, 2023.
  * Gu et al. [2022] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10696–10706, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hong et al. [2020a] Yan Hong, Li Niu, Jianfu Zhang, and Liqing Zhang.  Matchinggan: Matching-based few-shot image generation.  In _ICME_ , 2020a.
  * Hong et al. [2020b] Yan Hong, Li Niu, Jianfu Zhang, Weijie Zhao, Chen Fu, and Liqing Zhang.  F2gan: Fusing-and-filling gan for few-shot image generation.  In _ACM MM_ , 2020b.
  * Hong et al. [2022] Yan Hong, Li Niu, Jianfu Zhang, and Liqing Zhang.  DeltaGAN: Towards diverse few-shot image generation with sample-specific delta.  _ECCV_ , 2022.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  _arXiv preprint arXiv:2304.02642_ , 2023.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10124–10134, 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1931–1941, 2023.
  * Lee et al. [2019] Jason Lee, Kyunghyun Cho, and Douwe Kiela.  Countering language drift via visual grounding.  _arXiv preprint arXiv:1909.04499_ , 2019.
  * Li et al. [2023a] Dongxu Li, Junnan Li, and Steven CH Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _arXiv preprint arXiv:2305.14720_ , 2023a.
  * Li et al. [2023b] Lingxiao Li, Yi Zhang, and Shuhui Wang.  The euclidean space is evil: Hyperbolic attribute editing for few-shot image generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22714–22724, 2023b.
  * Liu et al. [2019] Kanglin Liu, Wenming Tang, Fei Zhou, and Guoping Qiu.  Spectral regularization for combating mode collapse in gans.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6382–6390, 2019.
  * Liu et al. [2023] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones: Concept neurons in diffusion models for customized generation.  _arXiv preprint arXiv:2303.05125_ , 2023.
  * Lu et al. [2020] Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin.  Countering language drift with seeded iterated learning.  In _International Conference on Machine Learning_ , pages 6437–6447. PMLR, 2020.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022\.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _arXiv preprint arXiv:2304.03411_ , 2023.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Srivastava et al. [2017] Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton.  Veegan: Reducing mode collapse in gans using implicit variational learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Tao et al. [2022] Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu.  Df-gan: A simple and effective baseline for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16515–16525, 2022.
  * Tao et al. [2023] Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu.  Galip: Generative adversarial clips for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14214–14223, 2023.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023\.
  * Thanh-Tung and Tran [2020] Hoang Thanh-Tung and Truyen Tran.  Catastrophic forgetting and mode collapse in gans.  In _2020 international joint conference on neural networks (ijcnn)_ , pages 1–10. IEEE, 2020.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  _arXiv preprint arXiv:2303.09522_ , 2023.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ , 2023.
  * Yang et al. [2022] Mengping Yang, Zhe Wang, Ziqiu Chi, and Wenyi Feng.  Wavegan: Frequency-aware gan for high-fidelity few-shot image generation.  In _European Conference on Computer Vision_ , pages 1–17. Springer, 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.