# Towards Real-time Text-driven Image Manipulation
with Unconditional Diffusion Models

Nikita Starodubtsev
Yandex
Correspondence to<EMAIL_ADDRESS>Dmitry Baranchuk
Yandex
Valentin Khrulkov
Yandex
Artem Babenko
Yandex

###### Abstract

Recent advances in diffusion models enable many powerful instruments for image
editing. One of these instruments is text-driven image manipulations: editing
semantic attributes of an image according to the provided text description.
Existing diffusion-based methods already achieve high-quality image
manipulations for a broad range of text prompts. However, in practice, these
methods require high computation costs even with a high-end GPU. This greatly
limits potential real-world applications of diffusion-based image editing,
especially when running on user devices.

In this paper, we address efficiency of the recent text-driven editing methods
based on unconditional diffusion models and develop a novel algorithm that
learns image manipulations $4.5{-}10\times$ faster and applies them $8\times$
faster. We carefully evaluate the visual quality and expressiveness of our
approach on multiple datasets using human annotators. Our experiments
demonstrate that our algorithm achieves the quality of much more expensive
methods. Finally, we show that our approach can adapt the pretrained model to
the user-specified image and text description on the fly just for ${\sim}4$
seconds. In this setting, we notice that more compact unconditional diffusion
models can be considered as a rational alternative to the popular text-
conditional counterparts.

##  1 Introduction

Figure 1: Few examples of single image editing. A user provides an image and a
text description of the desired transform. Our diffusion-based approach adapts
the pretrained model to the given image and text and returns the manipulated
image. The entire procedure takes ${\sim}4$ seconds.

Diffusion probabilistic models (DPMs) have recently shown remarkable results
in image generation, producing both realistic and diverse samples [15, 39].
After their initial success, diffusion models have seen use in many related
applications, including object detection [7], semantic segmentation [5], image
processing [37, 26] and many others.

One popular application of diffusion models is text-guided image manipulation,
where DPM is used to edit an existing image, changing its attributes to match
a user-specified text description. This application has become a hot topic
after the appearance of high-quality text-conditional DPMs like Imagen [36],
DALL-E 2 [33], Stable Diffusion [34].

Text-conditional DPMs are trained on large and diverse datasets with rich text
information, such as MS-COCO [24] or LAION5B [38], and applicable to a wide
range of domains at a time. However, some practical applications focus solely
on some simple and narrow domain, e.g., human faces, and often have strict
computational budgets. In these cases, unconditional diffusion models might be
a reasonable alternative since they already provide strong generative
performance on such domains being more efficient than large and diverse text-
conditional models. Moreover, unconditional models avoid tedious text
annotation process that increases the costs and can raise additional legal and
ethical concerns.

Several recent works have learned text-driven image manipulations using only
unconditional generative models. Some methods use unconditional GANs [19, 18],
showing strong results for text-driven domain adaptation [12, 3, 2] and single
image editing [31]. Other works [20, 23] exploit unconditional diffusion
models, significantly outperforming GANs in terms of both image fidelity and
editing quality.

However, compared to GANs, diffusion-based image manipulations require
expensive sequential inversion and generation, limiting their applicability in
resource-limited environments. Moreover, some applications, e.g., single image
editing, also require highly performant online adaptation where current
diffusion-based methods are still not efficient enough in terms of both
adaptation time and memory usage.

In this work, we analyze the performance bottlenecks of the existing text-
guided image manipulation methods based on unconditional diffusion models and
propose a workaround to circumvent them. As a result, we present a fast and
memory-efficient diffusion-based approach that preserves the visual and
editing quality of the prior image manipulation methods.

To sum up, the contributions of the paper are as follows:

  * •

We develop a practical algorithm for text-driven image manipulation based on
fast approximate sampling procedures. Our algorithm learns image manipulations
$4.5{-}10\times$ faster and applies them $8\times$ faster than previous
methods based on unconditional diffusion models. The source code of our
algorithm is available online111https://github.com/quickjkee/eff-diff-edit.

  * •

We conduct a thorough human evaluation on multiple standard image datasets to
compare the produced image manipulations in terms of visual quality and their
correspondence to the text attribute. Our algorithm achieves similar editing
quality as significantly more expensive counterparts.

  * •

We observe an emergent property of our training procedure. When the model is
trained using our protocol, it also learns to improve the perceptual quality
of approximate image estimates without being explicitly enforced to do so. We
reveal that it is an implicit side-effect of the directional CLIP loss [12].
From a practical standpoint, this phenomenon helps us to accelerate model
inference without perceptible loss in image fidelity.

  * •

We demonstrate that unconditional diffusion models can learn text-guided image
manipulations from a single image. This allows users to create their own image
transformations on the fly, using the same image they intend to manipulate.
For this application, our approach can edit images with significantly better
quality than existing GAN-based alternatives. Notably, on narrow domains like
human faces, the proposed method can provide reasonable quality-efficiency
trade-off compared to the state-of-the-art text-driven editing methods based
on Stable Diffusion.

##  2 Background

This section provides an overview of diffusion models, briefly discusses
previous text-driven image manipulation methods and describes the core aspects
of the DiffusionCLIP [20] approach that is mostly leveraged in our work.

###  2.1 Diffusion probabilistic models

Diffusion models [15] are latent variable generative models trained to
approximate the data distribution $x_{0}{\sim}q(x_{0})$ by using forward and
reverse diffusion processes.

The forward diffusion process $q(x_{1:T}|x_{0})$ gradually applies Gaussian
noise with some predefined variance schedule $\beta$ to a real image $x_{0}$
until it converges to isotropic Gaussian distribution $x_{T}{\sim}N(0,I)$. A
latent variable $x_{t}$ can be sampled directly from $x_{0}$ in the closed
form:

$q(x_{t}|x_{0}){=}\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},1-\bar{\alpha}_{t}I),$
(1)

where $\alpha_{t}\coloneqq 1-\beta_{t}$,
$\bar{\alpha}_{t}\coloneqq\prod_{s=1}^{t}\alpha_{s}$.

The reverse process allows generating new samples from $q(x_{0})$ by gradually
transforming $x_{T}{\sim}N(0,I)$ to $x_{0}$, which requires $T$ neural network
forward passes. Often, $T{=}1000$ and hence the generation takes much more
time compared with feed-forward generative models, e.g., GANs [13, 19].

Many recent works have addressed this problem [10, 17, 39, 40, 44, 25] and
demonstrated convincing results within a few dozens of inference steps. DDIM
[39] is one of the most prevalent samplers that generates plausible samples
for $\tau_{dec}\approx 50{-}100$ steps without re-training the initial model.

In more detail, DDIM obtains a sample $x_{t-1}$ from $x_{t}$ in the following
way:

$x_{0}(t;\theta){=}\frac{x_{t}{-}{\sqrt{1{-}\alpha_{t}}}\epsilon_{\theta}(x_{t})}{\sqrt{\alpha_{t}}}$
(2)
$x_{t-1}{=}\sqrt{\alpha_{t-1}}{\cdot}x_{0}(t;\theta){+}\sqrt{1{-}\alpha_{t-1}}\epsilon_{\theta}(x_{t})$
(3)

where $x_{0}(t;\theta)$ is a $x_{0}$ estimate at a time step $t$ and
$\epsilon_{\theta}(x_{t})$ is a prediction of the noise component using a
pretrained diffusion model with parameters $\theta$.

In addition, DDIM is deterministic and serves as a de-facto standard inversion
method for diffusion models with low reconstruction errors. Opposed to (1),
DDIM has to sequentially apply the pretrained model to map a real image
$x_{0}{\sim}q(x_{0})$ into the latent variable $x_{t}$:

$x_{t+1}{=}\sqrt{\alpha_{t+1}}{\cdot}x_{0}(t;\theta){+}\sqrt{1{-}\alpha_{t+1}}\epsilon_{\theta}(x_{t})$
(4)

Thus, the image “inversion” for diffusion models is much more expensive
compared with VAEs [21] or GANs [41, 1].

###  2.2 Text-guided image manipulation

One of the leading directions for semantic image manipulation with generative
models is to use natural language prompts to guide the image generation toward
the desired transformation.

The first popular group of methods considers pretrained text-conditional
diffusion models [34, 33, 36, 30]. Some of them leverage the internal
knowledge of the existing models by manipulating text embeddings [11, 35] or
cross attention weights [14, 28]. Other works propose to finetune the model
parameters on a single image [46, 42] and perform expressive text-driven
manipulations upon it. The recent work [6] finetunes Stable Diffusion on the
large set of GPT-3 powered paired examples. This allows them to achieve both
efficient and high-quality image transforms.

Another research direction is to exploit pretrained unconditional models along
with an external model for text guidance, typically CLIP [32]. In general, the
methods encode real images into the model latent space and then either modify
the latent representations [23, 31] or the model parameters directly [20, 12]
to minimize the CLIP loss.

Although GAN-based methods [31, 12, 3, 2] can provide efficient adaptation and
inference, they still suffer from imperfect image inversion and provide less
expressive and natural image transformations compared with the diffusion
models. On the other hand, DDPM-based methods [20, 23] have to perform costly
sequential forward and reverse processes to edit a single image.

There is also a line of work that suggests training an image generator from
scratch on a single image [22, 4, 43] and then applying the CLIP model for the
text-driven editing. The main limitations of such methods are a time consuming
adaptation and a scarce set of available transformations, e.g., style transfer
and image harmonization.

###  2.3 DiffusionCLIP

DiffusionCLIP [20] is a recent approach for text-driven image manipulation
using unconditional diffusion models. This method adapts $\epsilon_{\theta}$
to a new diffusion model $\epsilon_{\hat{\theta}}$ such that the
transformation of a source image $x_{0}$ into $x_{0}(\hat{\theta})$ is
associated with the text description $y_{tar}$, e.g., “Makeup face”.

In more detail, it encodes the image $x_{0}$ into the latent variable
$x_{t_{0}}$ using DDIM in forward direction (4) with the original diffusion
model $\epsilon_{\theta}$. In practice, the DDIM encoding is applied for some
predefined steps $\tau_{enc}\leq t_{0}$ to obtain $x_{t_{0}}$. The typical
values of $t_{0}$ are within $[300,600]$. Lower $t_{0}$ is useful for
“shallow” image manipulations, e.g., adding makeup. Higher $t_{0}$ is required
for strong transformations that significantly affect the structure of semantic
attributes, e.g., “Person” to “Zombie”.

Then, at each training iteration, DiffusionCLIP transforms $x_{t_{0}}$ to
$x_{0}(\theta)$ using the DDIM generation (3) for $\tau_{dec}$ steps and
backpropagates through the entire decoding process to minimize the following
objective:

$\displaystyle\mathcal{L}(\theta)=\mathcal{L}_{dir}(x_{0}(\theta),y_{tar};x_{0},y_{ref})+\lambda{\cdot}\mathcal{L}_{id}(x_{0}(\theta);x_{0})$
(5)

where $\mathcal{L}_{dir}$ essentially enforces semantic transfer from
$y_{ref}$ to $y_{tar}$ and $\mathcal{L}_{id}$ serves to preserve the
attributes of the original image $x_{0}$. $\lambda$ is a hyperparameter that
controls the regularization strength. Note that $\tau_{enc}$ and $\tau_{dec}$
are hyperparameters of the method allowing us to trade reconstruction quality
for processing speed; in practice, typical values are $\tau_{enc}\in[40,200]$
and $\tau_{dec}\in[6,40]$ [20].

$\mathcal{L}_{dir}$ is a directional CLIP loss [12] which minimizes the cosine
distance between cross-domain directions in the CLIP space. Specifically,
given the CLIP text encoder $E_{T}$, the text direction is defined as
${\Delta}T{=}E_{T}(y_{tar}){-}E_{T}(y_{ref})$, where $y_{ref}$ is a text
prompt that represents the source domain in general, e.g., “Person”. The
direction between the source $x_{0}$ and transformed $x_{0}(\theta)$ images is
defined similarly using the CLIP image encoder $E_{I}$:
${\Delta}I{=}E_{I}(x_{0}(\theta)){-}E_{I}(x_{0})$. The overall
$\mathcal{L}_{dir}$ objective is calculated as follows:

$\mathcal{L}_{dir}=1-\frac{\langle{\Delta}T,{\Delta}I\rangle}{\|{\Delta}T\|_{2}\|{\Delta}I\|_{2}}$
(6)

$\mathcal{L}_{id}$ is typically a reconstruction loss in the pixel space:
$L_{1}{=}\|x_{0}(\theta)-x_{0}\|_{1}$. Optionally, it may include additional
domain specific losses, e.g., $\mathcal{L}_{face}(x_{0}(\theta),x_{0})$ that
attempts to preserve the face identity in the image $x_{0}$ for the source
domain “Person” via a pretrained face recognition network.

Efficiency. The resulting algorithm performs image manipulations with state-
of-the-art quality but requires significantly more compute and memory than the
GAN-based counterparts. When dealing with $256{\times}256$ images,
DiffusionCLIP takes minutes to adapt the model to the target transform and a
few seconds to apply it to a single image.

During training, DiffusionCLIP unrolls several diffusion steps and
backpropagates through the decoding process at each model update step. To
allow this backpropagation, it needs to store intermediate model activations
from every step, taking up extra GPU memory. Thus, even with a batch size $1$,
it consumes ${\sim}21$GiB GPU VRAM and hence cannot fit most end-user GPUs. In
contrast, the GAN-based competitor [12] consumes only ${\sim}3$GiB.

The authors of DiffusionCLIP suggest a possible way to reduce memory
consumption by independently updating the model parameters at each decoding
step. This strategy halves the overall GPU usage but doubles the training time
due to the sequential model update steps.

During inference, the main performance bottleneck is the DDIM encoding that
requires running a forward pass of the diffusion model at each step to apply
noise to the original image. As a result, the algorithm spends more than half
of the total inference time applying noise to the image.

Figure 2: Overview of the proposed approach. Training phase. Diffusion model
parameters $\theta$ are updated to minimize the directional CLIP loss (6) for
a single-step $x_{0}$ estimate that lack high-frequency perceptual details.
Inference phase. A real image $x_{0}$ is mapped to the latent variable
$x_{t_{0}}$ using the forward diffusion process in the closed-form (1). The
edited image is generated for a few steps of the DDIM decoding that
sequentially applies the learned transform and recovers the fine-grained
details.

##  3 Method

In this section, we design a more efficient image manipulation algorithm that
circumvents the expensive multi-step processing in the DiffusionCLIP training
and inference stages. We conjecture that the objective (5) does not need
precise image reconstruction $x_{0}(\theta)$ to successfully learn the text-
driven image transformation. We apply this intuition to speed up training and
inference in Sections 3.1 and 3.2, respectively. Figure 2 summarizes our
approach.

###  3.1 Single step training

As we describe in Section 2.3, each DiffusionCLIP training step starts with
$x_{t_{0}}$, applies the DDIM generating process for $\tau_{dec}$ steps to
obtain $x_{0}(\theta)$, then uses it to estimate $\mathcal{L}(\theta)$ and
backpropagate through the entire computational graph to update parameters
$\theta$.

To avoid the costly multi-step procedure, instead of generating an edited
image $x_{0}(\theta)$ for $\tau_{dec}$ steps, we estimate the expected $x_{0}$
using eq. (2) and denote it as $x_{0}(t_{0};\theta)$. The
$x_{0}(t_{0};\theta)$ estimate lacks some fine-grained details. This is
illustrated in Figure 3 that presents $x(t_{0};\theta)$ predictions for
different $t_{0}$ steps222The predictions are obtained with a diffusion model
pretrained on the CelebA-HQ dataset from
https://github.com/ermongroup/SDEdit.. However, we hypothesize that our
training procedure only needs $x_{0}(t_{0};\theta)$ to preserve the semantic
attributes that are not supposed to be altered during the particular image
manipulation.

Since $x_{0}(t_{0};\theta)$ is a _single-step_ estimate, the method performs
only a single forward and backward pass of the diffusion model per training
iteration. This results in both faster training and significantly less memory
usage since there is no need to store multi-step activations in GPU memory. We
report the exact runtimes and memory usage in Section 4.2.

Figure 3: Visualisation of $x_{0}(t_{0};\theta)$ predictions for different
$t_{0}$. The estimates for $t_{0}\leq 500$ preserve main semantic attributes
of the original images.

Emergent effect of the training procedure. Interestingly, aside from the
intended purpose of faster training, the proposed algorithm improves the
visual quality of the resulting $x_{0}(t_{0};\hat{\theta})$ predictions. This
effect is not obvious since the objective $\mathcal{L}$ does have any terms
that explicitly optimize the image quality. This phenomenon allows us to
reduce the number of DDIM decoding steps $\tau_{dec}$ and further speed up the
inference. In Section 4.1, we explore this emergent property in more detail.

###  3.2 Efficient forward processing

In addition, DiffusionCLIP needs the costly DDIM encoding process to precisely
invert the sampling procedure up to fine-grained details. We argue that this
exact inversion is unnecessary because semantic image manipulations are
performed at the middle steps of the diffusion process where most semantic
attributes have already been formed. Also, the edit typically requires
noticeable image manipulations that override the details generated during the
final steps.

Therefore, we suggest replacing the DDIM encoding (3) with the DDPM forward
process and sample $x_{t_{0}}{\sim}q(x_{t_{0}}|x_{0})$ in the closed-form
using eq.(1). Notably, we use the DDPM forward process only for encoding,
retaining DDIM for the decoding phase.

In Figure 4, we visualize the image reconstructions $x_{0}(\theta)$ obtained
with DDIM and DDPM encoding methods for different $t_{0}\in[300,600]$ values.
We observe that $x_{0}(\theta)$ reconstructed from
$x_{t_{0}}{\sim}q(x_{t_{0}}|x_{0})$ are still consistent with $x_{0}$ for
$t_{0}{\leq}450$. In Appendix C, we provide the mean opinion score that
quantitatively supports this observation.

To sum up, we combine the two modifications above and present the overall
training algorithm in Appendix A.

During the inference, the proposed method encodes an image $x_{0}$ into the
latent variable $x_{t_{0}}{\sim}q(x_{t_{0}}|x_{0})$ and uses the DDIM decoding
for $\tau_{dec}$ steps to gradually apply the learned manipulation and recover
high-frequency details.

Figure 4: Image reconstruction with stochastic (DDPM) and deterministic (DDIM)
encoding methods for different diffusion steps $t_{0}$. The reconstructions
obtained from DDPM encodings for $t_{0}\leq 450$ are still consistent with the
original images and suit well for most image manipulations.

##  4 Experiments

In this section, we first investigate the emergent effect caused by our
training procedure. Then, we evaluate the efficiency of our approach in
comparison with DiffusionCLIP and GAN-based alternatives. Finally, we
demonstrate the qualitative and quantitative results of our approach for the
following settings:

  * •

Prelearned image manipulations — the model is first adapted to the text
description on $50$ images. Then, the learned transform is applied to the
hold-out images.

  * •

Single-image editing — the model is adapted to the user-specified text
description and image on the fly.

###  4.1 Analysis

Quality improvement effect. As we discuss in Section 3.1, the proposed
training procedure demonstrates an interesting phenomenon: optimizing
$\mathcal{L}_{dir}$ improves the perceptual quality of the
$x_{0}(t_{0};\hat{\theta})$ estimate after the model adaptation. Thanks to
this effect, our method can produce samples of the same visual quality for
$\tau_{dec}{=}6$ steps instead of $40$ considered in the best DiffusionCLIP
configuration.

In Figure 5, we measure the dynamics of $x_{0}(t_{0},\hat{\theta})$ quality
w.r.t. different fine-tune iterations using NIQE [27] — the established metric
for the no-reference image quality assessment. Opposed to DiffusionCLIP, we
observe that the quality of the $x_{0}$ estimates indeed significantly
increases over training for our method.

Figure 5: Image quality (NIQE) of $x_{0}(t_{0};\hat{\theta})$ estimate w.r.t.
training iterations. For both methods, the estimates get better but
improvements are way more pronounced for the proposed method.

What causes improvement of $x_{0}(t_{0};\hat{\theta})$? The directional CLIP
loss (6) aims to move the image embedding of the $x_{0}(t_{0};\hat{\theta})$
estimate to make a vector $E_{I}(x_{0}(t_{0};\hat{\theta})){-}E_{I}(x_{0})$
co-directional to the text direction $E_{T}(y_{ref}){-}E_{T}(y_{tar})$ in the
CLIP embedding space.

We hypothesize that the $x_{0}$ transformations along the text direction
weakly correlate to the perceptual quality unless it is assumed by the
manipulation. If so, the quality of the $x_{0}(t_{0};\hat{\theta})$ is
essentially determined by the source image $x_{0}$.

To confirm our assumption, we first consider some text direction, e.g, “Face”
to “Makeup Face” and optimize $\mathcal{L}_{dir}$ for $x_{0}$ and its manually
blurred version $x^{blur}_{0}$, independently. In Figure 6, we visualize
$x_{0}(t_{0};\hat{\theta})$ estimates after the model adaptation. We observe
that the estimates tend to preserve the sharpness of the source images.

Figure 6: Visualization of $x_{0}(t_{0};\hat{\theta})$ estimates after fine-
tuning for $x_{0}$ of different perceptual quality. The estimates tend to
inherit the perceptual details of the source images.

Secondly, we explore the behavior of the semantic direction when the
perceptual quality of the target and source images changes. In more detail, we
consider source images $x_{0}$ and their $x_{0}(t_{0};\hat{\theta})$ estimates
after the model adaptation. In the first setting, we fix $x_{0}$ and apply
Gaussian blur of different degree only to $x_{0}(t_{0};\hat{\theta})$: $\Delta
I(x_{0},x^{blur}_{0}(t_{0};\hat{\theta}))$. In the second setting, we apply
Gaussian blur to both $x_{0}$ and $x_{0}(t_{0};\hat{\theta})$: $\Delta
I(x^{blur}_{0},x^{blur}_{0}(t_{0};\hat{\theta}))$. We consider Gaussian blur
with kernel size $7$ and vary sigma from $0$ to $4$. Then, we measure cosine
between the image direction $\Delta I$ and the text direction $\Delta T$ for
various $y_{tar}$. The results are averaged over $6$ transforms and presented
in Figure 7. We observe that the image direction ${\Delta}I$ does not
noticeably change if both images are equally corrupted.

Figure 7: Cosine between $\Delta T$ and $\Delta I$ directions in the CLIP
space w.r.t. Gaussian blur strength applied to the image $x_{0}$ and its
manipulated estimate $x_{0}(t_{0};\hat{\theta})$. The angle is not affected
much if both $x_{0}$ and $x_{0}(t_{0};\hat{\theta})$ are corrupted equally.
This indicates that the semantic direction induced by $\Delta T$ has weak
correlation with the image quality direction.

###  4.2 Training and inference efficiency

Setting. The proposed method and DiffusionCLIP exploit the same pretrained
models. As a GAN-based baseline, we consider StyleGAN-NADA [12] that is built
upon the StyleGAN2 [19] generator. In this evaluation, all models are
pretrained on CelebA-HQ and operate on $256{\times}256$ images.

All measurements are performed in an isolated environment on a single NVIDIA
A100 GPU and averaged over $10$ independent runs. For both baselines, we
consider the official implementations and the best settings described in the
corresponding papers. We provide the exact hardware and software configuration
in Appendix B.

Figure 8: Comparison of our approach, DiffusionCLIP and StyleGAN-NADA in terms
GPU memory usage (GiB) (Left) and run-times (sec) (Right) for a single
training iteration.

Method | Encoding, sec | Decoding, sec
---|---|---
StyleGAN-NADA | $0.29$ | $0.07$
DiffusionCLIP | $1.53$ | $0.22$
Ours | $0.00$ | $0.22$

Table 1: The time spent to encode and decode a single image of
$256{\times}256$ resolution at the inference stage.

Training efficiency. We start with the evaluation of the training performance
of each method. The training set contains $50$ images.

The first phase is to precompute the latent variables $x_{t_{0}}$ for all
training images $x_{0}$. DiffisionCLIP exploits the DDIM encoding for
$\tau_{enc}{=}40$ steps and spends ${\sim}67$ seconds to encode $50$ images.
On the other hand, our method gets $x_{t_{0}}$ in the closed form. StyleGAN-
NADA is trained on self-generated samples and hence skips this phase.

Then, in Figure 8, we compare the GPU memory consumption (Left) and runtime
(Right) of a single training iteration w.r.t. different batch sizes. As an
additional baseline, we consider the memory efficient (ME) DiffusionCLIP
procedure described in Section 2.3. During the training, DiffusionCLIP
performs $\tau_{gen}{=}6$ decoding steps.

We observe that our approach is significantly faster than both DiffusionCLIP
versions. For example, our method demonstrates $0.18$s against $0.70$s and
$1.42$s for DiffusionCLIP and ME-DiffusionCLIP, respectively. Notice that the
gap increases with a batch size.

In terms of GPU memory usage, our procedure consumes $7$GiB for a batch size
$1$ that is $14$GiB less compared with the original DiffusionCLIP method.
Moreover, DiffusionCLIP exceeds the GPU memory limit ($>80$GiB) for a batch
size $5$ while our method consumes only ${\sim}18$GiB and meets OOM only for a
batch size $27$. ME-DiffusionCLIP consumes similar GPU memory to our method
because both perform a model update only for a single diffusion step at a
time. On the contrary, StyleGAN-NADA is still noticeably more memory efficient
than diffusion-based alternatives, especially for large batch sizes.

Finally, we measure the overall training performance. DiffusionCLIP and our
method perform $1{-}10$ training epochs depending on the target transform.
Thus, for a batch size $1$, our procedure spends from $10.7$ seconds to $1.58$
minutes, while the fastest DiffusionCLIP procedure spends from $101$ seconds
to $7.2$ minutes. Note that the efficiency gains increase for larger batch
sizes.

StyleGAN-NADA requires $50{-}300$ training iterations and operates on a batch
size $2$. In this case, the overall training takes from $8.1$ seconds to
$58.5$ seconds.

In the result, our training procedure is $4.5{-}10{\times}$ faster and
consumes $14$GiB less GPU memory than original DiffusionCLIP and
$8{-}18{\times}$ faster than its memory efficient alternative. In comparison
with StyleGAN-NADA, our adaptation still consumes extra $4$ GiB memory but,
notably, is comparable in terms of the training time.

Figure 9: Visual examples of the image manipulations learned with the
DiffusionCLIP, StyleGAN-NADA and our approaches for the Celeba-HQ-256, LSUN-
church-256, AFHQ-dog-256 and ImageNet-512 datasets.

Inference time. Then, we compare the method performance at the inference
stage. The fastest DiffusionCLIP configuration considers $\tau_{enc}{=}40$
encoding steps and the $\tau_{dec}{=}6$ for decoding. Our approach encodes an
image for free and performs $\tau_{dec}{=}6$ decoding steps. StyleGAN-NADA
utilizes Restyle [1] \+ e4e [41] as an encoding method.

The results are presented in Table 1. As one can see, our approach with
efficient approximate encoding results in $8{\times}$ faster inference than
DiffusionCLIP. Moreover, it also demonstrates ${\sim}50\%$ speedup compared to
StyleGAN-NADA due to the costly image encoder.

###  4.3 Prelearned image manipulations

For this setting, we evaluate the quality of the image manipulations on the
Celeba-HQ-256 [16], AFHQ-dog-256 [8], LSUN-Church-256 [45] and ImageNet-512
[9] datasets. As baseline methods, we consider DiffusionCLIP, StyleGAN-NADA,
Asyrp [23] and HyperDomainNet [3]. For each method, we consider the
hyperparameter values from the corresponding official implementations and
papers. If some hyperparameters are missing, we carefully tune them by
ourselves. For DiffusionCLIP, opposed to the fastest setting with
$\tau_{dec}{=}6$ in Section 4.2, in this experiment, we set $\tau_{dec}{=}40$
during the inference to derive the best editing results. Note that our
approach still performs $\tau_{dec}{=}6$ decoding steps. For comparison, we
use $35$ images from the test sets. $16$ text descriptions are taken for
CelebA-HQ-256 and $6$ for other datasets. The list of textual transforms is in
Appendix D.1.

As a primary quality measure, we consider a side-by-side human evaluation.
Specifically, we provide a source image, text description and images edited
with two different methods and then ask people to answer two questions:
1)Which of the edited images corresponds better to the text description? and
2)Which of the edited images has more artifacts and changes that are not
related to the text?. In total, we collect $9500$ votes. The voting results
are presented in Table 2. Figure 9 provides the visualizations for various
transformations. More visual examples are in Figures 16, 17, 18, 19, 20, 21.
The qualitative comparison with HyperDomainNet [3] is presented in Figure 22.

Compared with DiffusionCLIP, our approach corresponds better to the text
attributes on Celeba-HQ and LSUN-Church. On other datasets, the votes are
distributed equally. In terms of artifacts, we observe parity between both
methods. This means that our method still produces high-fidelity images and
provides as expressive image transforms as DiffusionCLIP. Moreover, despite
using stochastic encoding, our algorithm does not induce noticeable artifacts
in semantic attributes: even for the shallow transformations, e.g., “Makeup
face”, images still preserve important source details.

Against StyleGAN-NADA, Asyrp and HyperDomainNet, the proposed method
significantly outperforms all of them according to both criteria.

Figure 10: Visual examples produced with different single-image editing
approaches. Our method, DiffusionCLIP, StyleCLIP and StyleGAN-NADA represent
the methods using unconditional generative models. InstructPix2Pix and Null-
text Inversion methods are based on Stable Diffusion.

Dataset | Ours, % | Both, % | DiffusionCLIP, %
---|---|---|---
|  Q1. Which one corresponds better to the text?
Celeba-HQ | $\mathbf{54.21}\pm 1.82$ | $5.96\pm 1.23$ | $39.82\pm 2.68$
AFHQ-Dog | $\mathbf{44.11}\pm 3.25$ | $16.74\pm 4.65$ | ${38.85}\pm 2.12$
ImageNet | $35.42\pm 4.11$ | $31.23\pm 1.96$ | $33.23\pm 3.37$
LSUN-Church | ${40.67}\pm 3.85$ | $18.95\pm 1.39$ | ${40.38}\pm 3.22$
|  Q2. Which one has more text-irrelevant changes?
Celeba-HQ | ${40.53}\pm 1.89$ | $20.60\pm 2.05$ | ${38.85}\pm 1.95$
AFHQ-Dog | $40.28\pm 3.53$ | ${21.43}\pm 4.52$ | ${37.70}\pm 3.07$
ImageNet | $39.90\pm 4.19$ | $21.90\pm 4.71$ | $38.19\pm 4.62$
LSUN-Church | ${35.23}\pm 1.78$ | $28.28\pm 1.98$ | ${36.47}\pm 1.98$
Dataset | Ours, % | Both, % | StyleGAN-NADA, %
|  Q1. Which one corresponds better to the text?
Celeba-HQ | $\mathbf{57.26}\pm 1.65$ | $6.65\pm 0.20$ | $35.95\pm 1.61$
LSUN-Church | $\mathbf{69.04}\pm 3.75$ | $9.05\pm 1.85$ | $21.23\pm 2.67$
|  Q2. Which one has more text-irrelevant changes?
Celeba-HQ | ${29.06}\pm 2.06$ | $21.79\pm 2.17$ | $\mathbf{48.89}\pm 0.72$
LSUN-Church | ${18.76}\pm 3.28$ | $19.23\pm 1.82$ | $\mathbf{62.01}\pm 3.71$
Dataset | Ours, % | Both, % | Asyrp, %
|  Q1. Which one corresponds better to the text?
Celeba-HQ | $\mathbf{81.50}\pm 6.6$ | $5.53\pm 1.92$ | $13.05\pm 5.83$
|  Q2. Which one has more text-irrelevant changes?
Celeba-HQ | ${21.50}\pm 5.36$ | $22.06\pm 6.00$ | $\mathbf{57.01}\pm 8.77$
Dataset | Ours, % | Both, % | HyperDomainNet, %
|  Q1. Which one corresponds better to the text?
Celeba-HQ | $\mathbf{56.72}\pm 3.13$ | $2.53\pm 2.02$ | $40.83\pm 1.34$
|  Q2. Which one has more text-irrelevant changes?
Celeba-HQ | ${36.75}\pm 1.56$ | $22.50\pm 4.33$ | $\mathbf{40.81}\pm 2.24$

Table 2: Side-by-side human evaluation of various image manipulations learned
with the proposed, DiffusionCLIP, Asyrp and StyleGAN-NADA approaches. Our
method preserves quality of DiffusionCLIP and significantly outperforms other
methods according to both criteria.

| Ours , % | Both, % | DiffusionCLIP, %
---|---|---|---
Text correspondence | $\mathbf{60.20}\pm 3.80$ | $5.05\pm 2.30$ | $34.74\pm 5.81$
More irrelevant changes | ${36.56}\pm 4.06$ | $26.46\pm 4.39$ | ${36.76}\pm 5.73$
| Ours, % | Both, % | StyleCLIP, %
Text correspondence | $\mathbf{53.33}\pm 6.40$ | $10.41\pm 3.49$ | $36.25\pm 8.89$
More irrelevant changes | $27.50\pm 4.25$ | $23.33\pm 6.89$ | $\mathbf{49.17}\pm 3.86$
| Ours, % | Both, % | StyleGAN-NADA, %
Text correspondence | $\mathbf{71.57}\pm 5.22$ | $7.84\pm 2.70$ | $20.58\pm 3.82$
More irrelevant changes | $35.68\pm 3.59$ | $19.21\pm 3.73$ | $\mathbf{45.09}\pm 2.40$
| Ours, % | Both, % | Null-text Inversion, %
Text correspondence | $\mathbf{68.32}\pm 6.85$ | $5.81\pm 2.00$ | $25.81\pm 3.82$
More irrelevant changes | $42.31\pm 6.71$ | $16.84\pm 8.73$ | ${40.82}\pm 8.62$
| Ours, % | Both, % | InstructPix2Pix, %
Text correspondence | $36.88\pm 4.50$ | $6.25\pm 2.38$ | $\mathbf{56.88}\pm 4.30$
More irrelevant changes | $\mathbf{52.63}\pm 4.47$ | $16.32\pm 1.97$ | ${31.05}\pm 2.86$

Table 3: Side-by-side human evaluation for single-image editing on CelebA-
HQ-256. Our method outperforms DiffusionCLIP and Null-text Inversion according
to text correspondence and GAN-based methods according to both criteria.
InstructPix2Pix provides better editing results for most transforms.

###  4.4 Single-image editing

In this experiment, we make use of a significantly more efficient training
procedure and consider our approach for text-driven single-image editing.

In this setting, we consider CelebA-HQ-256 and compare our method with
DiffusionCLIP, StyleGAN-NADA and StyleCLIP [31] — GAN-based approach that
allows fast single-image editing.

In addition, popular text-conditional models also offer the tools to edit a
single image for the user-specified text description. Therefore, we compare
our method with recent editing methods [28, 6] based on Stable Diffusion [34].

First, we measure the overall time to apply the transform to a source image.
DiffusionCLIP and our approach require ${\sim}20$ training iterations for high
quality adaptation. StyleGAN-NADA starts producing reasonable image transforms
for $150$ training iterations. Overall, our method, DiffusionCLIP, StyleGAN-
NADA and StyleCLIP demonstrate $3.8$, $14.9$, $35.8$ and $1.1$ seconds per
image, respectively. Null-text inversion [28] requires $40{-}120$ seconds to
edit a single image. On the other hand, InstructPix2Pix [6] performs text-
guided editing for a single forward pass of Stable Diffusion which takes
${\sim}9$ seconds.

Then, we evaluate the visual quality of the manipulated images. We consider
$6$ images and $8$ in-domain and $8$ out-of-domain textual transforms. The
full list of textual transforms is in Appendix D.2. The results for some of
them are presented in Figure 10. More examples for various textual transforms
are in Figures 23, 24.

Also, we run the side-by-side comparison collecting $1860$ votes for all
evaluations and present the results in Table 3. Note that StyleCLIP is not
designed to handle out-of-domain manipulations, e.g., “Person” to “Zombie”.
Therefore, for a fair evaluation, we compare our method to StyleCLIP only on
the in-domain transformations.

We observe that our training procedure demonstrates higher text correspondence
compared to DiffusionCLIP under the same level of artifacts and significantly
outperforms all GAN-based methods according to both criteria.

For Null-text Inversion, we also select $6$ out of $16$ transforms for which
it is able to produce reasonable manipulation results. For these transforms,
our method still provides more expressive results under the same level of
text-irrelevant changes.

Our strongest competitor, InstructPix2Pix demonstrates impressive results for
most transforms in our evaluation. However, we notice that there are some
transforms where InstructPix2Pix produces either too weak or too unrealistic
results, e.g., “Smiling person”, “Surprised person” or “Makeup face”.333We
followed the tips provided by the authors but still could not get plausible
results. Therefore, if one wants to add new manipulations, one needs to
reproduce the entire pipeline of InstructPix2Pix training that includes
incredibly expensive data collection and adaptation procedures. In contrast,
our method uses the pretrained diffusion model out-of-the-box and needs only a
single image to learn the desired transform. Notably, our approach provides
reasonable image manipulations for all considered transforms. More visual
examples in comparison with InstructPix2Pix are in Figure 24.

##  5 Conclusion

This work addresses the inefficiency of existing image manipulation approaches
built upon unconditional diffusion models. We observe that image manipulations
can be efficiently learned on inaccurate source image predictions without
noticeable loss in perceptual and editing quality. In the result, we obtain a
highly performant and memory-efficient procedure that can be further
considered for real-world applications looking for expressive and efficient
image manipulation methods.

## References

  * [1] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.
  * [2] Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, and Daniel Cohen-Or.  Third time’s the charm? image and video editing with stylegan3, 2022.
  * [3] Aibek Alanov, Vadim Titov, and Dmitry P. Vetrov.  Hyperdomainnet: Universal domain adaptation for generative adversarial networks.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [4] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European Conference on Computer Vision, pages 707–723. Springer, 2022.
  * [5] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  In International Conference on Learning Representations, 2022.
  * [6] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  arXiv preprint arXiv:2211.09800, 2022.
  * [7] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo.  Diffusiondet: Diffusion model for object detection.  arXiv preprint arXiv:2211.09788, 2022.
  * [8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. IEEE, 2009.
  * [10] Tim Dockhorn, Arash Vahdat, and Karsten Kreis.  GENIE: Higher-Order Denoising Diffusion Solvers.  In Advances in Neural Information Processing Systems, 2022.
  * [11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
  * [12] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators, 2021\.
  * [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [14] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  2022\.
  * [15] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  * [16] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In International Conference on Learning Representations, 2018.
  * [17] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  In Proc. NeurIPS, 2022.
  * [18] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In Proc. NeurIPS, 2021.
  * [19] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In Proc. CVPR, 2020.
  * [20] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [21] Diederik P. Kingma and Max Welling.  An introduction to variational autoencoders.  Foundations and Trends® in Machine Learning, 12(4):307–392, 2019\.
  * [22] Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli.  Sinddm: A single image denoising diffusion model.  arXiv preprint arXiv:2211.16582, 2022.
  * [23] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  In The Eleventh International Conference on Learning Representations, 2023.
  * [24] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll’a r, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  CoRR, abs/1405.0312, 2014.
  * [25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [26] Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In CVPR, pages 11451–11461. IEEE, 2022.
  * [27] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik.  Making a “completely blind” image quality analyzer.  IEEE Signal Processing Letters, 20(3):209–212, 2013.
  * [28] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models, 2022.
  * [29] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162–8171. PMLR, 18–24 Jul 2021.
  * [30] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16784–16804. PMLR, 17–23 Jul 2022.
  * [31] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085–2094, October 2021.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. PMLR, 18–24 Jul 2021.
  * [33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  CoRR, abs/2204.06125, 2022.
  * [34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [35] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  2022\.
  * [36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [37] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  arXiv:2104.07636, 2021.
  * [38] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [39] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In International Conference on Learning Representations, 2021.
  * [40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [41] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  * [42] Dani Valevski, Matan Kalman, Y. Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  ArXiv, abs/2210.09477, 2022.
  * [43] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li.  Sindiffusion: Learning a diffusion model from a single natural image.  arXiv preprint arXiv:2211.12445, 2022.
  * [44] Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan.  Learning to efficiently sample from diffusion probabilistic models, 2022\.
  * [45] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [46] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris Metaxas, and Jian Ren.  Sine: Single image editing with text-to-image diffusion models.  arXiv preprint arXiv:2212.04489, 2022.

Appendix

##  Appendix A Training algorithm

(Red) represents the snippets of the DiffusionCLIP procedure that are replaced
with the proposed modifications in Section 3.1 and Section 3.2 (Green).

Input :
$\epsilon_{\theta},n_{iter},x_{0},y_{ref},y_{tar},\tau_{enc},\tau_{dec}$

Output : $\epsilon_{\hat{\theta}}$

/* Encoding process */

for  _$\tau=0,...,\tau_{enc}$_ do

$x_{0}(\tau;\theta)=x_{\tau}{-}\sqrt{1{-}\alpha_{\tau}}{\cdot}\epsilon_{\theta}(x_{\tau})/\sqrt{\alpha_{\tau}}$

$x_{\tau+1}(\theta)=\sqrt{\alpha_{\tau+1}}{\cdot}x^{\tau}_{0}(\theta){+}\sqrt{1{-}\alpha_{\tau+1}}\epsilon_{\hat{\theta}}(x_{\tau})$

end for

$x_{t_{0}}{=}\sqrt{\bar{\alpha}_{t}}{\cdot}x_{0}+\sqrt{1-\bar{\alpha}_{t}}{\cdot}I$

$\hat{\theta}=\theta$

for  _$i=1,...,n_{iter}$_ do

/* Decoding process and model update */

for  _$\tau=\tau_{dec},...,1$_ do

$x_{0}(\tau;\hat{\theta})=x_{\tau}{-}\sqrt{1{-}\alpha_{\tau}}{\cdot}\epsilon_{\hat{\theta}}(x_{\tau})/\sqrt{\alpha_{\tau}}$

$x_{\tau-1}(\hat{\theta})=\sqrt{\alpha_{\tau-1}}{\cdot}x^{\tau}_{0}(\hat{\theta}){+}\sqrt{1{-}\alpha_{\tau-1}}\epsilon_{\hat{\theta}}(x_{\tau})$

end for

$\hat{\theta}=\hat{\theta}+\nabla_{\hat{\theta}}\mathcal{L}(x_{0}(\hat{\theta}),y_{tar},x_{0},y_{ref}))$

$x_{0}(t_{0};\hat{\theta})=\left(x_{t_{0}}-\sqrt{1-\alpha_{t_{0}}}\epsilon_{\theta}(x_{t_{0}},t_{0})\right)/{\sqrt{\alpha_{\tau}}}$
$\hat{\theta}=\hat{\theta}+\nabla_{\hat{\theta}}\mathcal{L}(x_{0}(t_{0};\hat{\theta}),y_{tar},x_{0},y_{ref}))$

end for

return $\epsilon_{\hat{\theta}}$

Algorithm 1 Training procedure. DiffusionCLIP vs Ours.

##  Appendix B Technical setup

All experiments are performed on a single Tesla A100 GPU. 64 CPU cores are
used. PyTorch version is 1.10.1.

We use the following architectures of the diffusion models: [15] for Celeba-
HQ-256, [29] for LSUN-church-256, AFHQ-dog-256 and ImageNet-512.

##  Appendix C Evaluation of stochastic encoding

In this section, we compare stochastic (DDPM) and deterministic (DDIM)
encoding methods using the mean opinion score. We ask assessors to estimate
how the reconstructed image is similar to the reference one according to the
criteria in Table 4 (Bottom).

According to the human opinion in Table 4 (Top), for $t_{0}=300$, DDPM
encoding does not affect semantic attributes noticeably. Thus, our method
considers lower $t_{0}\leq 300$ for shallow image manipulations, e.g., “Makeup
face”, where preserving most details of the original image is important. On
the other hand, $t_{0}=500$ slightly alters the face attributes. Therefore, we
use $t_{0}{=}350{-}500$ for severe transforms, e.g., “Zombie”, that
significantly affect the semantic attributes.

$t_{0}$ | 100 | 300 | 500
---|---|---|---
DDPM | $4.58$ | $4.08$ | $3.40$
DDIM | $4.75$ | $4.46$ | $4.40$

Criterion
---
5: Looks identical. No visible changes or artifacts.
4: Minor visible changes. All face attributes are fully preserved.
3: Minor changes of the face attributes. The person is the same.
2: Significant changes of the face attributes.
1: Completely different person.

Table 4: (Top) Comparison of stochastic (DDPM) and deterministic (DDIM)
encoding methods for different steps $t_{0}$ in terms of mean opinion score
that measures similarity between reconstructed and original images. (Bottom)
The criteria which were shown to assessors for estimation.

##  Appendix D List of textual transforms

###  D.1 Prelearned image manipulations

For the human evaluation, we use the following established transformations
[20, 12]:

  * •

Celeba-HQ-256 — “Face” $\rightarrow$ “Angry face”, “Face” $\rightarrow$ “Pale
face”, “Face” $\rightarrow$ “Smiling Face”, “Photo” $\rightarrow$ “Painting in
Fernando Botero style”, “Person” $\rightarrow$ “Nicolas Cage”, “Photo”
$\rightarrow$ “Painting in Cubism style”, “Face” $\rightarrow$ “Makeup face”,
“Photo” $\rightarrow$ “Painting in Modigliani style”, “Human” $\rightarrow$
“Neanderthal, “Person” $\rightarrow$ “Old person”, “Human” $\rightarrow$ “3D
rendering in style of Pixar”, “Person” $\rightarrow$ “Surprised”, “Face”
$\rightarrow$ “Tanned face”, “Photo” $\rightarrow$ “Watercolor art”, “Human”
$\rightarrow$ “Zombie”, “Person” $\rightarrow$ “Mark Zuckerberg”.

  * •

LSUN-church-256 — “Church” $\rightarrow$ “Golden church, Church” $\rightarrow$
“Colorful church”, “Church” $\rightarrow$ “Gothic church, “Church”
$\rightarrow$ “Modern architecture”, “Church” $\rightarrow$ “Snow covered
church”, “Church” $\rightarrow$ “Ancient temple”.

  * •

AFHQ-dog-256 — “Dog” $\rightarrow$ “Angry dog”, “Dog” $\rightarrow$ “Anime
dog”, “Dog” $\rightarrow$ “Bear”, “Dog” $\rightarrow$ “Fox”, “Dog”
$\rightarrow$ “Smiling dog”, “Dog to Zombie dog”.

  * •

ImageNet-512 — “Photo” $\rightarrow$ “Painting in cubism style”, “Photo”
$\rightarrow$ “Painting in cubism style”, “Photo” $\rightarrow$ “Painting in
Van Gogh style”, “Photo” $\rightarrow$ “Painting in pointilism style”, “Photo”
$\rightarrow$ “Sketch”, “Photo” $\rightarrow$ “Watercolor art”.

###  D.2 Single image editing

  * •

In-domain transforms — “Face” $\rightarrow$ “Angry face”, “Person”
$\rightarrow$ “Nicolas Cage”, “Face” $\rightarrow$ “Makeup face”, “Person”
$\rightarrow$ “Old person”, “Person” $\rightarrow$ “Surprised”, “Man”
$\rightarrow$ “Woman” (or vice versa), “Face” $\rightarrow$ “Smiling face”,
“Person” $\rightarrow$ “Mark Zuckerberg”.

  * •

Out-of-domain transforms — “Human” $\rightarrow$ “3D rendering in style of
Pixar”, “Human” $\rightarrow$ “Neanderthal”, “Photo” $\rightarrow$ “Painting
in Fernando Botero style”, “Photo” $\rightarrow$ “Painting in Modigliani
style”, “Photo” $\rightarrow$ “Self portrait by Frida Kahlo”, “Photo”
$\rightarrow$ “Sketch”, “Human” $\rightarrow$ “Jocker”, “Human” $\rightarrow$
“Zombie”.

Text description | $lr$ | $\lambda$ | $t_{0}$
---|---|---|---
| CelebA-HQ-256
Angry face | $3e{-}6$ | 0.6 | 300
Pale face | $3e{-}6$ | 0.6 | 300
Smiling face | $3e{-}6$ | 0.6 | 300
Painting in Fernando Botero style | $8e{-}6$ | 0.3 | 450
Nicolas Cage | $7e{-}6$ | 0.1 | 450
Painting in Cubism style | $7e{-}6$ | 0.1 | 450
Makeup face | $1e{-}6$ | 0.9 | 300
Painting in Modigliani style | $5e{-}6$ | 0.4 | 450
Neanderthal | $6e{-}6$ | 0.3 | 450
Old person | $5e{-}6$ | 0.2 | 380
3D rendering in style of Pixar | $3e{-}6$ | 0.1 | 450
Surprised | $4e{-}6$ | 0.8 | 450
Tanned face | $4e{-}6$ | 0.8 | 450
Watercolor art | $6e{-}6$ | 0.5 | 450
Zombie | $2e{-}5$ | 0.1 | 430
Mark Zuckerberg | $2e{-}6$ | 0.3 | 400
| LSUN-Church-256
Golden church | $3e{-}6$ | 0.1 | 380
Colorful church | $3e{-}6$ | 0.0 | 400
Gothic church | $3e{-}6$ | 0.1 | 380
Modern architecture | $2e{-}6$ | 0.2 | 400
Snow covered church | $3e{-}6$ | 0.6 | 400
Ancient temple | $4e{-}6$ | 0.8 | 400
| AFHQ-Dog-256
Angry dog | $6e{-}6$ | 0.8 | 380
Anime dog | $8e{-}6$ | 0.2 | 450
Bear | $8e{-}6$ | 0.7 | 450
Fox | $8e{-}6$ | 0.6 | 450
Smiling dog | $6e{-}6$ | 0.3 | 400
Zombie dog | $6e{-}6$ | 0.2 | 450
| ImageNet-512
Painting in cubism style | $8e{-}6$ | 0.1 | 450
Painting in cubism style | $8e{-}6$ | 0.1 | 450
Painting in Van Gogh style | $8e{-}6$ | 0.1 | 450
Painting in pointilism style | $8e{-}6$ | 0.2 | 450
Sketch | $8e{-}6$ | 0.1 | 450
Watercolor art | $6e{-}6$ | 0.1 | 450

Table 5: Hyperparameter values used to learn text-driven image manipulations.
$\lambda$ represents the coefficient in front of the $\mathcal{L}_{id}$ term.
Figure 11: The $x_{0}$ estimates before and after finetuning using
$\mathcal{L}_{dir}$ and $\mathcal{L}_{id}$ losses, independently. Opposed to
$\mathcal{L}_{dir}$, $\mathcal{L}_{id}$ does not improve the perceptual
quality of the estimates.

##  Appendix E Quality improvement effect

The paper demonstrates that our adaptation procedure improves the perceptual
quality of the approximate $x_{0}$ estimates at a time step $t_{0}$.

In this experiment, we ensure that $\mathcal{L}_{dir}$ (6) is indeed
responsible for these changes but not $\mathcal{L}_{id}$ (5). Figure 11
provides the visual examples of the $x_{0}$ estimates after the model
finetuning independently using one of these losses. We observe that
$\mathcal{L}_{id}$ does not lead to perceptual quality improvements.

Figure 12: The effect of different hyperparameter values for a few transforms.
$\lambda_{\mathcal{L}_{id}}$ corresponds to the coefficient in front of the
regularization term.

##  Appendix F Hyperparameters

The proposed approach is sensitive to hyperparameter values. In Table 5, we
provide the full list of hyperparameter values used for different datasets and
transforms. Figure 12 presents a few examples of image manipulations with
different hyperparameter values. Below, we give a few recommendations to make
the tuning of our method more approachable for users.

  * •

Consider using higher values of the regularization coefficient $\lambda$
(e.g., $5{-}10$) for the shallow transforms like “Red hair”.

  * •

Higher values of the $t_{0}$ allow us to make stronger transforms but can
bring more irrelevant changes. Thus, we recommend to use smaller $t_{0}$
values ($t_{0}=200{-}350$) for the shallow transforms, e.g., “Makeup face”,
“Surprised face”, and higher $t_{0}$ values ($t_{0}=350{-}500$) for the strong
manipulations, e.g., “Zombie”, “Sketch”.

  * •

We also find that a higher learning rate, e.g., $2e{-}5$, allows obtaining
better results for the strong transforms, while for the shallow ones, smaller
values are preferable, e.g., $5e{-}6$.

##  Appendix G Failure cases

Figure 13: Visual examples of the failed image manipulations produced with our
approach. Figure 14: Visual examples of the multi-attribute transfer.
Sequential setting adapts the model to two different text descriptions one by
one. Simultaneous means that we form a single text description that combines
two different attributes.

The proposed approach is limited by the capabilities of the CLIP and
pretrained diffusion models. Sometimes, the former cannot provide the desired
signal to adapt the diffusion model properly. In Figure 13, we provide a few
visual examples of such text descriptions.

##  Appendix H Multi-attribute transfer

Following [20], we also demonstrate that our approach is able to produce
multi-attribute transfers in both sequential and simultaneous regimes. The
visual examples of the learned image manipulations are presented in Figure 14.

Figure 15: Voting results for prelearned image manipulations (Top) and single-
image editing (Bottom). Figure 16: More visual examples of the prelearned
image manipulations on CelebA-HQ-256 (Ours and DiffusionCLIP). Figure 17: More
visual examples of the prelearned image manipulations on CelebA-HQ-256 (Ours
and Asyrp). Figure 18: More visual examples of the prelearned image
manipulations on AFHQ-Dogs-256 (Ours and DiffusionCLIP). Figure 19: More
visual examples of the prelearned image manipulations on ImageNet-512 (Ours
and DiffusionCLIP). Figure 20: More visual examples of the prelearned image
manipulations on LSUN-Church-256 (Ours, DiffusionCLIP and StyleGAN-NADA).
Figure 21: More visual examples of the prelearned image manipulations on
CelebA-HQ-256 (Ours and StyleGAN-NADA). Figure 22: More visual examples of the
prelearned image manipulations on CelebA-HQ-256 (Ours and HyperDomainNet).
Figure 23: More visual examples of the text-guided single-image editing
produced with our method. Figure 24: More visual examples of the text-guided
single-image editing (Ours and InstructPix2Pix).