11institutetext: Dept. of Bio & Brain Engineering, KAIST22institutetext: Kim
Jae Chul AI graduate school, KAIST
22email: {jeongsol, pky3436<EMAIL_ADDRESS>
Equal contribution

# DreamSampler: Unifying Diffusion Sampling and Score Distillation for Image
Manipulation

Jeongsol Kim 1Dept. of Bio & Brain Engineering, KAIST1 * * Geon Yeong Park
1Dept. of Bio & Brain Engineering, KAIST1 * * Jong Chul Ye 2Kim Jae Chul AI
graduate school, KAIST

Equal contribution 2{jeongsol, pky3436<EMAIL_ADDRESS>1Dept. of Bio &
Brain Engineering, KAIST1 * *1Dept. of Bio & Brain Engineering, KAIST1 * *2Kim
Jae Chul AI graduate school, KAIST

Equal contribution 2{jeongsol, pky3436<EMAIL_ADDRESS>
###### Abstract

Reverse sampling and score-distillation have emerged as main workhorses in
recent years for image manipulation using latent diffusion models (LDMs).
While reverse diffusion sampling often requires adjustments of LDM
architecture or feature engineering, score distillation offers a simple yet
powerful model-agnostic approach, but it is often prone to mode-collapsing. To
address these limitations and leverage the strengths of both approaches, here
we introduce a novel framework called DreamSampler, which seamlessly
integrates these two distinct approaches through the lens of regularized
latent optimization. Similar to score-distillation, DreamSampler is a model-
agnostic approach applicable to any LDM architecture, but it allows both
distillation and reverse sampling with additional guidance for image editing
and reconstruction. Through experiments involving image editing, SVG
reconstruction and etc, we demonstrate the competitive performance of
DreamSampler compared to existing approaches, while providing new
applications.

###### Keywords:

Latent diffusion model Generation Score distillation

##  1 Introduction

Figure 1: Upper panel: DreamSampler vs (a) reverse diffusion and (b) score
distillation. Lower panel: DreamSampler can be used for image restoration
using vectorization, editing, inpainting, etc.

Diffusion models [6, 25, 24] have been extensively studied as powerful
generative models in recent years. These models operate by generating clean
images from Gaussian noise through a process termed ancestral sampling. This
involves progressively reducing noise by utilizing an estimated score function
to guide the generation process from a random starting point towards the
distribution of natural images. Reverse diffusion sampling introduces
stochasticity through the reverse Wiener process within the framework of SDE
[25], contributing to the prevention of mode collapse in generated samples
while enhancing the fidelity [3]. Furthermore, the reverse diffusion can be
flexibly regularized by various guidance gradients. For instance, classifier-
guidance [3] applies classifier gradients to intermediate noisy samples,
facilitating conditional image generation. Additionally, DPS [1] utilizes
approximated likelihood gradients to constrain the sampling process, ensuring
data consistency between the current estimated solution and given observation,
thereby solving noisy inverse problems in a zero-shot manner.

On the other hand, another type of approach, called score distillation,
utilizes diffusion model as prior knowledge for image generation and editing.
For example, DreamFusion [20] leveraged 2D text-conditioned diffusion models
for text-guided 3D representation learning via NeRF. Here, the diffusion model
serves as the teacher model, generating gradients by comparing its predictions
with the label noises and guiding the generator as the student model. The
strength of the score distillation method lies in its ability to leverage the
pre-trained diffusion model in a black-box manner without requiring any
feature engineering, such as adjustments to the model architecture.
Unfortunately, the score distillation is more often prone to mode collapsing
compared to the reverse diffusion.

Although both algorithms are grounded in the same principle of diffusion
models, the approaches appear to be different, making it unclearly how to
synergistically combine the two approaches. To address this, we introduce a
unified framework called DreamSampler, that seamlessly integrates two distinct
approaches and take advantage of the both worlds through the lens of
regularized latent optimization. Specifically, DreamSampler is model-agnostic
and does not require any feature engineering, such as adjustments to the model
architecture. Moreover, in contrast to the score distillation, DreamSampler is
free of mode collapsing thanks to the stochastic nature of the sampling.

The pioneering aspect of DreamSampler is rooted in two pivotal insights.
First, we demonstrate that the process of latent optimization during reverse
diffusion can be viewed as a proximal update from the posterior mean by
Tweedie’s formula. This interpretation allows us to integrate additional
regularization terms, such as measurement consistency in inverse problems, to
steer the sampling procedure. Moreover, we illustrate that the loss associated
with the proximal update can be conceptualized as the score distillation loss.
This insight bridges a natural connection between the score-distillation
methodology and reverse sampling strategies, culminating in their harmonious
unification. In subsequent sections, we will explore various applications
emerging from this integrated framework and demonstrate the efficacy of
DreamSampler through empirical evidence.

##  2 Motivations

###  2.1 Related Works

In LDMs, the encoder ${\mathcal{E}}_{\phi}$ and the decoder
${\mathcal{D}}_{\varphi}$ are trained as auto-encoder, satisfying
${\boldsymbol{x}}={\mathcal{D}}_{\varphi}({\mathcal{E}}_{\phi}({\boldsymbol{x}}))={\mathcal{D}}_{\varphi}({\boldsymbol{z}}_{0})$
where ${\boldsymbol{x}}$ denotes clean image and ${\boldsymbol{z}}_{0}$
denotes encoded latent vector. Then, the diffusion process is defined on the
latent space, which is a range space of ${\mathcal{E}}_{\phi}$. Specifically,
the forward diffusion process is

$\displaystyle{\boldsymbol{z}}_{t}=\sqrt{\bar{\alpha}_{t}}{\boldsymbol{z}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}$
(1)

where $\bar{\alpha}_{t}$ denotes pre-defined coefficient that manages noise
scheduling, and $\boldsymbol{\epsilon}\sim{\mathcal{N}}(0,\boldsymbol{I})$
denotes a noise sampled from normal distribution. The reverse diffusion
process requires a score function via a neural network (i.e. diffusion model,
$\boldsymbol{\epsilon}_{\theta}$) trained by denoising score matching [6, 25]:

$\displaystyle\min_{\theta}{\mathbb{E}}_{t,\boldsymbol{\epsilon}\sim{\mathcal{N}}(0,\boldsymbol{I})}\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)\|^{2}_{2}.$
(2)

According to formulation of DDIM [24, 2], the reverse sampling from the
posterior distribution
$p({\boldsymbol{z}}_{t-1}|{\boldsymbol{z}}_{t},{\boldsymbol{z}}_{0})$ could be
described as

$\displaystyle{\boldsymbol{z}}_{t-1}$
$\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\hat{\boldsymbol{z}}_{0|t}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\boldsymbol{\epsilon}}$
(3)

where

$\displaystyle\hat{\boldsymbol{z}}_{0|t}$
$\displaystyle=({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t))/\sqrt{\bar{\alpha}_{t}}$
(4) $\displaystyle\tilde{\boldsymbol{\epsilon}}$
$\displaystyle=\frac{\sqrt{1-\bar{\alpha}_{t-1}-\eta^{2}\beta_{t}^{2}}\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)}{\sqrt{1-\bar{\alpha}_{t-1}}}+\frac{\eta\beta_{t}\boldsymbol{\epsilon}}{\sqrt{1-\bar{\alpha}_{t-1}}}$
(5)

where $\hat{\boldsymbol{z}}_{0|t}$ refers to the denoised latent through
Tweedie’s formula and and $\tilde{\boldsymbol{\epsilon}}$ is the noise term
composed of both deterministic
$\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)$ and stochastic term
$\boldsymbol{\epsilon}\sim{\mathcal{N}}(0,\boldsymbol{I})$. Here, $\eta$ and
$\beta_{t}$ denote variables that controls the stochastic property of
sampling. When $\eta\beta_{t}=0$, the sampling is deterministic.

###  2.2 Key Observations

The first key insight of DreamSampler is that the DDIM sampling can be
interpreted as the solution of following proximal optimization problem:

$\displaystyle{\boldsymbol{z}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\bar{\boldsymbol{z}}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\boldsymbol{\epsilon}},\quad\mbox{where}\quad\bar{\boldsymbol{z}}=\operatorname*{arg\,min}_{\boldsymbol{z}}\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}\|^{2}$
(6)

Although looks trivial, one of the important implications of (6) is that we
can now add an additional regularization term:

$\min_{\boldsymbol{z}}\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}\|^{2}_{2}+\lambda_{reg}{\mathcal{R}}({\boldsymbol{z}})$
(7)

where $\lambda_{reg}$ is scalar weight for the regularization function
${\mathcal{R}}({\boldsymbol{z}})$. For example, we can use data consistency
loss to ensure that the updated variable agrees with the given observation
during inverse problem solving [9].

Second key insight arises from the important connection between (6) and the
score-distillation loss. Specifically, using the forward diffusion to generate
${\boldsymbol{z}}_{t}$ from the clean latent ${\boldsymbol{z}}$:

$\displaystyle{\boldsymbol{z}}_{t}$
$\displaystyle=\sqrt{\bar{\alpha}_{t}}{\boldsymbol{z}}+\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon},$
(8)

the objective function in (6) can be converted to:

$\displaystyle\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}\|^{2}$
$\displaystyle=\left\|\frac{{\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}}{\sqrt{\bar{\alpha}_{t}}}-\frac{{\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)}{\sqrt{\bar{\alpha}_{t}}}\right\|^{2}$
(9)
$\displaystyle=\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)\|^{2},$
(10)

which is equivalent to the score-distillation loss.

Figure 2: Unified framework of DreamSampler. (a) Distillation step where the
gradient is computed from regularized latent optimization problem. (b) Reverse
sampling step where estimated noise by diffusion model is added to the updated
generation.

##  3 DreamSampler

###  3.1 General Formulation

In LDMs, diffusion model offers prior knowledge conditioned on text through
the classifier-free-guidance (CFG) [7] where the estimated noise is computed
by

$\displaystyle\boldsymbol{\epsilon}_{\theta}^{\omega}({\boldsymbol{z}}_{t},t,c_{ref})=\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\phi})+\omega[\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{ref})-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\phi})]$
(11)

where $\omega$ denotes the guidance scale, $c_{\phi}$ refers to the null-text
embedding, and $c_{ref}$ is the conditioning text embedding, which are encoded
by pre-trained text encoder such as CLIP [21]. For simplicity, we will
interchangeably use the terms
$\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)$ and
$\boldsymbol{\epsilon}_{\theta}^{\omega}({\boldsymbol{z}}_{t},t,c_{ref})$
unless stated otherwise.

Suppose that $g(\psi)$ denotes a generated data by an arbitrary generator $g$
and parameter $\psi$. Inspired by the two key insights described in the
previous section, the sampling process of DreamSampler at timestep $t$ is
given by111Here, we omit the encoder ${\mathcal{E}}_{\phi}$ in
${\mathcal{E}}_{\phi}(g(\psi))$ for notational simplicity. The encoder maps
the generated image $g(\psi)$ to the latent space.

$\displaystyle{\boldsymbol{z}}_{t}$
$\displaystyle=\sqrt{\bar{\alpha}_{t}}g(\psi_{t-1})+\sqrt{1-\bar{\alpha}_{t}}\tilde{\boldsymbol{\epsilon}},$
(12)

where the noise $\tilde{\boldsymbol{\epsilon}}$ is as described in (5) and

$\displaystyle\hat{\boldsymbol{z}}_{0|t}$
$\displaystyle=\left({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t)\right)/\sqrt{\bar{\alpha}_{t}}$
(13) $\displaystyle\psi_{t}$
$\displaystyle=\operatorname*{arg\,min}_{\psi}\|g(\psi)-\hat{\boldsymbol{z}}_{0|t}\|^{2}+\lambda_{reg}{\mathcal{R}}(g(\psi))$
(14)

With appropriately defined generator and regularization function, this
generalized latent optimization can induce distinct score distillation
algorithms combined with reverse sampling. In the following sections, we will
provide special cases of DreamSampler.

###  3.2 DreamSampler with External Generators

Similar to DreamFusion [20], for any differentiable generator $g$,
DreamSampler can feasibly update parameters $\psi$ by leveraging the diffusion
model and sharing the same sampling process. To emphasize the distinctions
between the original score distillation algorithms and DreamSampler, we
conduct a line-by-line comparison of the pseudocode in Algorithm 1 and
Algorithm 2.

Algorithm 1 Score Distillation

1:$T$, $\zeta$, $g$, $\psi$, ${\mathcal{E}}_{\phi}$,
{$\bar{\alpha}_{t}\\}_{t=1}^{T}$

2:${\boldsymbol{z}}_{0}\leftarrow{\mathcal{E}}_{\phi}({\boldsymbol{x}}_{0})$

3:for $i=T$ to $1$ do

4: $t\sim U[0,T]$

5: $\tilde{\boldsymbol{\epsilon}}\sim{\mathcal{N}}(0,\boldsymbol{I})$

6:
${\boldsymbol{z}}_{t}\leftarrow\sqrt{\bar{\alpha}_{t}}{\boldsymbol{z}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\tilde{\boldsymbol{\epsilon}}$

7:
$\hat{\boldsymbol{\epsilon}}_{\theta}\leftarrow\boldsymbol{\epsilon}_{\theta}^{\omega}({\boldsymbol{z}}_{t},t,c)$

8:
$\nabla_{\psi}{\mathcal{L}}_{ds}\leftarrow\tilde{\boldsymbol{\epsilon}}-\hat{\boldsymbol{\epsilon}}_{\theta}$

9: $\psi\leftarrow\psi-\zeta\nabla_{\psi}{\mathcal{L}}_{ds}$

10: ${\boldsymbol{z}}_{0}\leftarrow{\mathcal{E}}_{\phi}(g(\psi))$

11:end for

12:return $\psi$

Algorithm 2 DreamSampler

1:$T$, $\zeta$, $g$, $\psi$, ${\mathcal{E}}_{\phi}$,
{$\bar{\alpha}_{t}\\}_{t=1}^{T}$

2:${\boldsymbol{z}}_{0}\leftarrow{\mathcal{E}}_{\phi}({\boldsymbol{x}}_{0}),\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{T+1}):=\boldsymbol{\epsilon}\sim{\mathcal{N}}(0,\boldsymbol{I})$

3:for $i=T$ to $1$ do

4: $t\leftarrow i$, $\boldsymbol{\epsilon}\sim{\mathcal{N}}(0,\boldsymbol{I})$

5:
$\tilde{\boldsymbol{\epsilon}}\leftarrow\frac{\sqrt{1-\bar{\alpha}_{t-1}-\eta^{2}\beta_{t}^{2}}\hat{\boldsymbol{\epsilon}}_{\theta}+\eta\beta_{t}\boldsymbol{\epsilon}}{\sqrt{1-\bar{\alpha}_{t}}}$

6:
${\boldsymbol{z}}_{t}\leftarrow\sqrt{\bar{\alpha}_{t}}{\boldsymbol{z}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\tilde{\boldsymbol{\epsilon}}$

7:
$\hat{\boldsymbol{\epsilon}}_{\theta}\leftarrow\boldsymbol{\epsilon}_{\theta}^{\omega}({\boldsymbol{z}}_{t},t,c)$

8:
$\nabla_{\psi}{\mathcal{L}}_{ds}\leftarrow\tilde{\boldsymbol{\epsilon}}-\hat{\boldsymbol{\epsilon}}_{\theta}$

9:
$\psi\leftarrow\psi-\zeta[\nabla_{\psi}{\mathcal{L}}_{ds}+{\color[rgb]{0,1,1}\lambda_{reg}\nabla_{\psi}{\mathcal{R}}}]$

10: ${\boldsymbol{z}}_{0}\leftarrow{\mathcal{E}}_{\phi}(g(\psi))$

11:end for

12:return $\psi$

First, in DreamSampler the timestep follows the reverse sampling process,
whereas the original distillation algorithms utilize uniformly random sampling
for the timestep. This provides us with a novel potential for further
refinement in utilizing various time schedulers to enhance reconstruction
quality or for accelerated sampling. Second, as DreamSampler is built upon the
proximal optimization approach that enables the use of flexible regularization
functions, the gradient derived from the regularization function is
additionally leveraged for updating the parameters of the generator. For
example, the generator could be constrained to reconstruct the true image that
aligns with the given measurement by defining the regularization function as
the data consistency term.

Figure 2 illustrates the sampling process of DreamSampler. At each timestep,
the generated image $g(\psi)$ is mapped to a noisy manifold by incorporating
the estimated noise from the previous timestep, and new noise is subsequently
estimated by the diffusion model. The distillation gradient is then computed
between these two estimated noises and utilized to update the generator
parameters.

###  3.3 DreamSampler for Image Editing

As DreamSampler is a general framework, we can reproduce other existing
algorithms by properly defining $g(\psi)$, $\hat{\boldsymbol{z}}_{0|t}$, and
${\mathcal{R}}$. As a representative example, here we derive Delta Denoising
Score (DDS) [4] for image editing task and discuss its potential extension
from the perspective of DreamSampler.

The main assumption of DDS is to decompose the SDS gradient [20] into text
component and bias component, where only the text component contains
information to be edited according to given text prompt while the bias
component includes preserved information. To remove the bias component from
distillation gradient, DDS leverages the difference of two conditional
predicted noises,
$\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{tgt})-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{src})$,
where $c_{tgt}$ denotes description of editing direction and $c_{src}$ denotes
description of the original image. Specifically, DDS update reads

$\displaystyle\bar{\boldsymbol{z}}=\hat{\boldsymbol{z}}_{0|t}(c_{src})-\gamma\rho[\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{tgt})-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{src})].$
(15)

The following Theorem 3.1 shows that by defining the regularization function
${\mathcal{R}}({\boldsymbol{z}})$ as Euclidean distance from the posterior
mean conditioned on text, we can reproduce the DDS sampling from DreamSampler.

###### Theorem 3.1

Supposed $c_{src}$ in (15) be defined as the null-text, i.,e.
$c_{src}=c_{\phi}$ and consider text- conditioned posterior mean:

$\displaystyle\hat{\boldsymbol{z}}_{0|t}(c)={\mathbb{E}}[{\boldsymbol{z}}_{0}|{\boldsymbol{z}}_{t},c]=({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c))/\sqrt{\bar{\alpha}_{t}}.$
(16)

Then, DDS update in (15) can be obtained from the following latent
optimization:

$\displaystyle\min_{\boldsymbol{z}}\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{\phi})\|^{2}+{\gamma}R({\boldsymbol{z}}),\quad\mbox{where}\quad
R({\boldsymbol{z}}):=\frac{\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt})\|^{2}}{(1-\gamma)}$
(17)

Furthermore, it is equivalent to Tweedie’s formula with CFG, i.e.:

$\displaystyle\hat{\boldsymbol{z}}_{0|t}^{\gamma}(c_{tgt})$
$\displaystyle:=\frac{{\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon}_{\theta}^{\gamma}({\boldsymbol{z}}_{t},t,c_{tgt})}{\sqrt{\bar{\alpha}_{t}}}$
(18)

where
$\boldsymbol{\epsilon}_{\theta}^{\gamma}({\boldsymbol{z}}_{t},t,c_{tgt})=\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\phi})+\gamma[\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{tgt})-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\phi})]$

Theorem 3.1 reveals that the one step latent optimization (17) by DreamSampler
is equivalent to the DDS update (15). That being said, the main advantages of
DreamSampler stems from the added noise to updated source image
${\boldsymbol{z}}_{0}$. Specifically, in contrast to the original DDS method
that adds newly sampled Gaussian noise to ${\boldsymbol{z}}_{0}$, DreamSampler
adds the estimated noise by $\boldsymbol{\epsilon}_{\theta}$ in the previous
timestep of reverse sampling. With initial noise computed by DDIM inversion,
reverse sampling do not deviate significantly from the reconstruction
trajectory even though source description is not given, because the latent
optimization (17) represents a proximal problem that regulates the sampling
process. Consequently, DreamSampler only requires text prompt that describe
the editing direction for the real imaging editing.

Finally, it is noteworthy that the equivalent interpretation (18) could be
readily extended to localized distillation by computing the solution as

$\displaystyle\hat{\boldsymbol{z}}_{0|t}^{\gamma}=\left({{\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\sum_{i}{{\mathcal{M}}_{i}\odot\boldsymbol{\epsilon}_{\theta}^{\gamma}({\boldsymbol{z}}_{t},t,c_{tgt}^{(i)})}}\right)/{\sqrt{\bar{\alpha}_{t}}}$
(19)

where ${\mathcal{M}}_{i}$ denotes the pixel-wise mask, $\odot$ is element-wise
multiplication, and $c_{tgt}^{(i)}$ denote $i$th mask and corresponding text
prompt pair. The entire process for the real image editing via DreamSampler is
described in Algorithm 3.

Algorithm 3 DreamSampler for Image Editing

source image ${\boldsymbol{x}}$, image encoder ${\mathcal{E}}_{\phi}$, latent
diffusion model $\boldsymbol{\epsilon}_{\theta}$, null-text embedding
$c_{\phi}$, conditioning text embedding $c_{tgt}$.

${\boldsymbol{z}}_{0}\leftarrow{\mathcal{E}}_{\phi}({\boldsymbol{x}})$

${\boldsymbol{z}}_{T}\leftarrow\text{Inversion}({\boldsymbol{z}}_{0})$

for $t\in[T,0]$ do

$\hat{\boldsymbol{\epsilon}}_{\theta}\leftarrow\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\phi})+\gamma[\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{tgt})-\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\phi})]$

$\bar{\boldsymbol{z}}\leftarrow({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\boldsymbol{\epsilon}}_{\theta})/\sqrt{\bar{\alpha}_{t}}$

$\tilde{\boldsymbol{\epsilon}}\leftarrow(\sqrt{1-\bar{\alpha}_{t-1}-\eta^{2}\beta_{t}^{2}}\hat{\boldsymbol{\epsilon}}_{\theta}+\eta\beta_{t}\boldsymbol{\epsilon})/\sqrt{1-\bar{\alpha}_{t-1}}$

${\boldsymbol{z}}_{t}\leftarrow\sqrt{\bar{\alpha}_{t-1}}\bar{\boldsymbol{z}}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\boldsymbol{\epsilon}}$

end for

###  3.4 DreamSampler for Inverse Problems

DreamSampler can leverage multiple regularization terms to precisely constrain
the sampling process to solve inverse problems. For example, we can solve the
text-guided image inpainting task by defining the regularization function as

$\displaystyle{\mathcal{R}}({\boldsymbol{z}})=(1-\gamma)\|{\mathcal{M}}\odot({\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt}))\|^{2}+\gamma\|{\boldsymbol{y}}-{\mathcal{A}}{\mathcal{D}}_{\varphi}({\boldsymbol{z}})\|^{2},$
(20)

where ${\mathcal{M}}$ denotes the operation to create the measurement by
masking out the target region. This regularization term implies that the
sample image satisfies data consistency for regions where the true signal is
preserved, while guiding the masked region to reflect the target text prompt.
When we solve the entire latent optimization problem, we follows two-step
approach of TReg [9]. For the details, refer to the appendix. The main
difference with TReg is that we separate the text-guidance from the data
consistency term. In other words, we initialize the ${\boldsymbol{z}}$ for the
data consistency term as $\hat{\boldsymbol{z}}_{0|t}(c_{\phi})$ while TReg
uses
$(1-\omega)\hat{\boldsymbol{z}}_{0|t}(c_{\phi})+\omega\hat{\boldsymbol{z}}_{0|t}(c_{tgt})$
where the $\omega$ denotes the CFG scale. This difference allows DreamSampler
to solve the inpainting problem with different text-guidance for each masked
region, by combining localized distillation approach introduced in Section
3.3.

##  4 Experimental Results

###  4.1 Image Restoration through Vectorization

Figure 3: Representative results for image vectorization task with image
reconstruction. Figure 4: Qualitative comparison of DreamSampler with baseline
methods. DreamSampler can simultaneously solve the inverse problem and update
SVG parameters via score distillation.

As a novel application that has not been exported by other methods, here we
presents an novel application image vectorization from blurry measurement
using DreamSampler. In this scenario, the generator $g(\cdot)$ corresponds to
a differentiable rasterizer (DiffVG [10]), and the parameters $\psi$ of the
generator consist of path parameters comprising Scalable Vector Graphics
(SVG).

Specifically, we address a text-guided SVG inverse problem, acknowledging that
low-quality, noisy measurements can detract from the detail and aesthetic
quality of vector designs. Here, our goal is to accurately reconstruct SVG
paths for the parameter $\psi$ that, when rasterized, align closely with the
provided measurements ${\boldsymbol{y}}$ and text conditions $c_{\mathbf{y}}$,
related through the forward measurement operator ${\mathcal{A}}$. In this
context, the regularizer ${\mathcal{R}}$ in (14) corresponds to the data
consistency term. Forward measurement operators are specified as follows: (a)
For super-resolution, bicubic downsampling is performed with scale x8. (b) For
Gaussian blur, the kernel has size $61\times 61$ with a standard deviation of
5. Then, the latent optimization framework of the SVG inverse problem is
defined as follows:

$\min_{\psi}(1-\gamma)\lambda_{SDS}\|{\mathcal{E}}(g(\psi))-\hat{\boldsymbol{z}}_{0|t}(c_{{\boldsymbol{y}}})\|^{2}+\gamma\lambda_{DC}\|{\boldsymbol{y}}-{\mathcal{A}}g(\psi)\|^{2},$
(21)

where we found out that $\gamma=\bar{\alpha}_{t}$ works well in practice. SVG
primitives $\psi$ are initialized with radius 20, random fill color, and
opacity uniformly sampled between 0.7 and 1, following [8]. In this paper, we
use closed Bézier curves for iconography artistic style. For the optimization
of (21), we use Adam optimizer with ($\beta_{1},\beta_{2}$)=(0.9, 0.9). For
the text condition $c_{{\boldsymbol{y}}}$, we append a suffix to the
object222e.g. ”{a cute fox}, minimal flat 2d vector icon. lineal color. on a
white background. trending on artstation.”. Additional experimental details
are provided in the appendix.

Comparatively, the proposed framework is evaluated against a multi-stage
baseline approach involving initial rasterized image reconstruction using the
state-of-the-art solver (PSLD [23]) that is based on the latent diffusion
model. Then, we vectorize the reconstruction using the off-the-shelf Layer-
wise Image Vectorization program (LIVE [14]). An optional step includes
refining the SVG output via latent score distillation sampling, exemplified by
VectorFusion [8].

Figure 3 illustrates that the proposed framework achieves high-quality SVG
reconstructions with semantic alignment closely matching the specified text
condition $c_{\mathbf{y}}$. In contrast, Figure 4 highlights the deficiencies
of the multi-stage baseline methods, particularly its inability to retain
detailed fidelity. Errors accumulate during the initial restoration process,
resulting in blurriness and undesirable path overlap in the vector outputs.
While the solver may achieve adequate reconstructions, the subsequent
vectorization step disregards text caption $c_{{\boldsymbol{y}}}$, leading to
the potential loss of details and semantic coherence. Additional VectorFusion
fine-tuning loses consistency with the measurement. Conversely, DreamSampler
effectively restores SVGs by directly utilizing latent-space diffusion and
text conditions for both vectorization and reconstruction, ensuring the
preservation of detail and contextual relevance.

###  4.2 Real Image Editing

Figure 5: Representative results for real image editing via distillation
through reverse sampling. We leverage source images from various domains and
the caption above each image denotes text prompt reflecting the editing
direction. The result demonstrates that distillation could be effectively
conducted during the reverse sampling.

For the real image editing via DreamSampler, we leverage the Stable-Diffusion
v1.5 provided by HuggingFace. We use linear time schedule and set NFE to 200
for both the DDIM inversion and reverse sampling. For the null-text, we use
empty text and does not leverage negative prompts. For the CFG scale $\gamma$,
we found that time-dependent value between $0.1\bar{\alpha}_{t}$ and
$0.3\bar{\alpha}_{t}$ can edit image with robustness. As
$\bar{\alpha}_{t}\rightarrow 1$ when $t\rightarrow 0$, this scale leads early
stage sampling to reconstruct the source image while the later stage sampling
to reflect guided direction, according to (17). For more details, refer to the
appendix.

To demonstrate the ability of DreamSampler on real image editing, we leverage
source images from various domains, including photographs of animals, human
faces and drawings. All results show that DreamSampler accurately reflects the
provided text prompts for editing. Specifically, DreamSampler does not change
bias components, which are intended not to be edited by text prompt. For
instance, in 3rd and 4th rows of Figure 5, features such as braces and
background including striped shirts are well-maintained, while the text
prompts are accurately reflected. Also, DreamSampler effectively reflects
multiple editing directions simultaneously such as "doberman" $+$ "wearing
glasses", "woman" $+$ "wearing glasses" or "photography" $+$ "fox".

Figure 6: Qualitative comparison with DDS for the real image editing. Both
DreamSampler and DDS edit images following target text, but DreamSampler
achieves higher fidelity with preserving bias component.

We also conduct a qualitative comparision with the original DDS algorithm to
show the improvement emerged from integration of distillation into reverse
sampling. As illustrated in Figure 5, DreamSampler is capable of editing
according to text prompt robustly to domain of source images. Furthermore,
DreamSampler achieves better fidelity and better bias component preserving
compared to the original DDS as shown in Figure 6.

###  4.3 Text-guided Image Inpainting

Figure 7: Qualitative comparison for text guided image inpainting task.
DreamSampler can generate more realistic images according to given text
prompt.

To demonstrate the performance of DreamSampler for the image inpainting task,
we generate measurements by masking out two rectangular regions on the eyes
and mouth, where the masked region is determined based on the averaged face of
the FFHQ validation dataset. The size of measurement is 512x512. When we
initialize ${\boldsymbol{z}}_{T}$ with DDIM inversion, the masked regions are
empty, which could hinder the generation of realistic images. Thus, we
substitute the masked region of ${\boldsymbol{z}}_{T}$ with Gaussian noise. To
enhance the consistency of masked region and other regions, we apply DPS [1]
steps during the sampling by following TReg [9]. All experiment in this
section are conducted with Stable-Diffusion v1.5 using 200 NFE. For more
details, refer to the appendix.

Figure 7 shows that DreamSampler fills masked region by reflecting given text
prompt accurately. While the text guidance is applied inside the mask, data
consistency gradients combined with the reconstruction by inversion is applied
outside the mask, which results in superior fidelity of the output image. Note
that we display the image output as is without any post-processing such as
projection333For linear operator ${\mathcal{A}}$ and measurement
${\boldsymbol{y}}$, the projection means
${\mathcal{A}}^{\top}{\mathcal{A}}{\boldsymbol{y}}+(\boldsymbol{I}-{\mathcal{A}}^{\top}{\mathcal{A}}){\boldsymbol{x}}$..
The bottom two rows of Figure 7 depict the solution for inpainting problem
with two different masks and distinct text guidance. Through localized
distillation gradient, DreamSampler generates solutions according to the
provided guidance. Compared to TReg, DreamSampler generates masked regions
with better robustness. Additionally, in the case of multiple masks,
DreamSampler successfully reflects text in the correct regions via localized
distillation gradients, whereas TReg fails to meet one of the conditions.

##  5 Conclusion

We presented DreamSampler, a unified framework of reverse sampling and score
distillation by taking the advantages of each algorithms. Specifically, we
connected two distinct algorithms under the perspective of latent optimization
problem. Consequently, we introduced a generalized optimization framework,
which offers new design space to solve various applications. Especially,
DreamSampler enables to combine various regularization functions to constrain
the sampling process. Additionally, we provided three applications including
image vectorization with reconstruction, real image editing, and image
inpainting. DreamSampler could be extended to other algorithms by defining
appropriate regularization functions. The codebase will be available to public
at https://github.com/DreamSampler/dream-sampler.

Potential negative social impact.  The performance of algorithms established
on DreamSampler heavily depends on the prior of diffusion model. Hence, the
proposed method basically influenced by the potential negative impacts of LDM
itself.

Supplementary material of "DreamSampler: Unifying Diffusion Sampling and Score
Distillation for Image Manipulation"

Jeongsol KimGeon Yeong Park Jong Chul Ye

##  6 Implementation details

In this section, we provide details on the implementation and experimental
settings of DreamSampler.

###  6.1 Image Restoration through Vectorization

For the reverse sampling, we set the CFG scale to 100, and NFE to 1000 for
both super-resolution and Gaussian deblurring. For the Lagrangian coefficient,
we set $\lambda_{SDS}=2.4,\lambda_{DC}=3$ for Guassian deblur and
$\lambda_{SDS}=1,\lambda_{DC}=4$ for super-resolution. Other optimization
configurations, e.g. learning rate, optimization algorithms, paths, etc,
follow [8]. Specifically, a self-intersection regularizer $\mathcal{L}_{xing}$
is used with a weight $0.01$. The learning rate initiates at 0.02 and linearly
escalates to 0.2 across 500 steps, subsequently undergoing a cosine decay to
0.05 upon optimization completion for control point coordinates. Fill colors
are subjected to a learning rate that is 20 times lower than that of control
points, while the solid background color is allocated a learning rate 200
times lower.

###  6.2 Real Image Editing

We utilize the DDIM inversion using the null-text to initialize the latent
${\boldsymbol{z}}_{T}$. The null-text is defined by empty text "", which could
be leveraged in general. For the noise addition steps at each timestep, we
leverage the deterministic sampling by setting $\eta\beta_{t}=0$. As mentioned
in the main manuscript, we set the CFG scale $\gamma$ between
0.1$\bar{\alpha}_{t}$ and 0.3$\bar{\alpha}_{t}$ and set NFE to 200. Here,
$\gamma$ indicates the interpolation coefficient for the latent optimization
problem

$\min_{\boldsymbol{z}}(1-\gamma)\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{\phi})\|^{2}+\gamma\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt})\|^{2},$
(22)

which is analyzed in section 8.1.

###  6.3 Text-guided Image Inpainting

For the inapinting task, we utilize the DDIM inversion with null-text where
the null-text is "out of focus, depth of field" to adopt concept negation for
better generation quality. After solving the latent optimization problem, we
add stochastic noise by setting
$\eta\beta_{t}=\sqrt{\bar{\alpha}_{t}}\sqrt{1-\bar{\alpha}_{t-1}}$. As other
tasks, we set NFE to 200.

From the general formulation of DreamSampler, we can derive the latent
optimization problem for the inpainting task. Suppose that the generator
$g={\mathcal{D}}_{\varphi}$ and $\phi={\boldsymbol{z}}$ so the generator maps
the latent vector to pixel space. Note that ${\mathcal{D}}_{\varphi}$ is a
component of autoencoder that satisfies
${\boldsymbol{z}}={\mathcal{E}}_{\phi}({\mathcal{D}}_{\varphi}({\boldsymbol{z}}))$
called perfect reconstruction constraint. Then, by defining the regularization
function as

$\displaystyle{\mathcal{R}}({\mathcal{D}}_{\varphi}({\boldsymbol{z}}))=\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt})\|^{2}+\|{\boldsymbol{y}}-{\mathcal{A}}{\mathcal{D}}_{\varphi}({\boldsymbol{z}})\|^{2}$
(23)

we can reach to the latent optimization problem of

$\displaystyle\min_{{\boldsymbol{z}},{\boldsymbol{x}}}\gamma_{1}\underbrace{\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt})\|^{2}}_{\text{Score
distillation}}$
$\displaystyle+\gamma_{2}\underbrace{\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{\phi})\|^{2}}_{\text{Proximal
term}}$
$\displaystyle+\gamma_{3}\underbrace{\left[\|{\boldsymbol{y}}-{\mathcal{A}}{\mathcal{D}}_{\varphi}({\boldsymbol{z}})\|^{2}+\|{\boldsymbol{z}}-{\mathcal{E}}_{\phi}({\boldsymbol{x}})\|^{2}\right]}_{\text{Data
consistency}}$ (24)

where the last term comes from the perfect reconstruction constraint, and
$\gamma_{1}+\gamma_{2}+\gamma_{3}=1$. The objective function comprises three
components: data consistency, which ensures alignment between the current
estimate and the given measurement; score distillation, which serves as
textual guidance; and a proximal term, which regulates the solution to remain
close to the inverted trajectory. Following TReg [9], we solve the
optimization problem sequentially. First, using the approximation
${\boldsymbol{x}}={\mathcal{D}}_{\varphi}({\boldsymbol{z}})$, the optimization
problem with respect to ${\boldsymbol{x}}$ becomes

$\displaystyle\min_{x}\|{\boldsymbol{y}}-{\mathcal{A}}{\boldsymbol{x}}\|^{2}+\|{\boldsymbol{z}}-{\mathcal{E}}_{\phi}({\boldsymbol{x}})\|^{2}+\lambda\|{\boldsymbol{x}}-{\mathcal{D}}_{\phi}({\boldsymbol{z}})+{\boldsymbol{\eta}}\|^{2}$
(25)

where the dual variable ${\boldsymbol{\eta}}$ is set to a zero vector for
simplicity. Then, by initializing
${\boldsymbol{z}}=\hat{\boldsymbol{z}}_{0|t}(c_{\phi})$, we have

$\displaystyle\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}})=\operatorname*{arg\,min}_{\boldsymbol{x}}\|{\boldsymbol{y}}-{\mathcal{A}}({\boldsymbol{x}})\|^{2}+\lambda\|{\boldsymbol{x}}-{\mathcal{D}}_{\varphi}(\hat{\boldsymbol{z}}_{0|t}(c_{\phi}))\|^{2}.$
(26)

This could be solved by conjugate gradient (CG) method. Subsequently, using
the approximation ${\boldsymbol{z}}={\mathcal{E}}_{\phi}({\boldsymbol{x}})$
with ${\boldsymbol{\eta}}=\mathbf{0}$, the optimization with respect to
${\boldsymbol{z}}$ becomes

$\displaystyle\min_{z}\gamma_{1}\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt})\|^{2}+\gamma_{2}\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{\phi})\|^{2}+\gamma_{3}\|{\boldsymbol{z}}-{\mathcal{E}}_{\phi}(\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}}))\|^{2},$
(27)

which leads to a closed-form solution

$\displaystyle\bar{\boldsymbol{z}}=\bar{\alpha}_{t}\hat{\boldsymbol{z}}_{0|t}(c_{tgt})+(1-\bar{\alpha}_{t})^{2}\hat{\boldsymbol{z}}_{0|t}(c_{\phi})+\bar{\alpha}_{t}(1-\bar{\alpha}_{t})\hat{\boldsymbol{z}}_{0|t}({\boldsymbol{y}})$
(28)

where
$\hat{\boldsymbol{z}}_{0|t}({\boldsymbol{y}}):={\mathcal{E}}_{\phi}(\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}}))$.
Specifically, we apply localized distillation gradient by leveraging a mask so
the objective function inside/outside of masked region is differ as

$\displaystyle\min_{z}\gamma_{1}\|{\mathcal{M}}\odot({\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{tgt}))\|^{2}+\gamma_{2}\|{\boldsymbol{z}}-\hat{\boldsymbol{z}}_{0|t}(c_{\phi})\|^{2}+\gamma_{3}\|{\boldsymbol{z}}-{\mathcal{E}}_{\phi}(\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}}))\|^{2}$
(29)

where ${\mathcal{M}}$ denotes pixel-wise mask with ones inside the masked
region and zeros elsewhere, while $\odot$ denotes element-wise multiplication.
Hence, the closed form solution is described as

$\displaystyle\bar{\boldsymbol{z}}=\begin{cases}\bar{\alpha}_{t}\hat{\boldsymbol{z}}_{0|t}(c_{tgt})+(1-\bar{\alpha}_{t})^{2}\hat{\boldsymbol{z}}_{0|t}(c_{\phi})+\bar{\alpha}_{t}(1-\bar{\alpha}_{t})\hat{\boldsymbol{z}}_{0|t}({\boldsymbol{y}})&\text{inside
mask}\\\
(1-\bar{\alpha}_{t})\hat{\boldsymbol{z}}_{0|t}(c_{\phi})+\bar{\alpha}_{t}\hat{\boldsymbol{z}}_{0|t}({\boldsymbol{y}})&\text{otherwise}\\\
\end{cases}$ (30)

Also, we use additional DPS step to ensure the consistency between inside and
outside of masked region. By following TReg, we compute the (30) when
$\Gamma=\\{t|t\text{ mod }3=0,t\leq 170\\}$ and apply DPS gradie nt otherwise.
In summary, the pseudocode of DreamSampler for the inpainting task is
described as Algorithm 4

Algorithm 4 DreamSampler for Image Inpainting

measurement ${\boldsymbol{y}}$, image encoder ${\mathcal{E}}_{\phi}$, latent
diffusion model $\boldsymbol{\epsilon}_{\theta}$, null-text embedding
$c_{\emptyset}$, conditioning text embedding $c_{tgt}$.

${\boldsymbol{z}}_{0}\leftarrow{\mathcal{E}}_{\phi}({\boldsymbol{y}})$

${\boldsymbol{z}}_{T}\leftarrow\text{Inversion}({\boldsymbol{z}}_{0})$

for $t\in[T,0]$ do

$\hat{\boldsymbol{\epsilon}}_{\theta}(c_{\emptyset}),\hat{\boldsymbol{\epsilon}}_{\theta}(c_{tgt})\leftarrow\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{\emptyset}),\boldsymbol{\epsilon}_{\theta}({\boldsymbol{z}}_{t},t,c_{tgt})$

$\boldsymbol{\epsilon}\sim{\mathcal{N}}(0,\boldsymbol{I})$

$\tilde{\boldsymbol{\epsilon}}_{t}\leftarrow(\sqrt{1-\bar{\alpha}_{t-1}-\eta^{2}\beta_{t}^{2}}\hat{\boldsymbol{\epsilon}}_{\theta}(c_{\emptyset})+\eta\beta_{t}\boldsymbol{\epsilon})/\sqrt{1-\bar{\alpha}_{t-1}}$

if $t\in\Gamma$ then

$\hat{\boldsymbol{z}}_{0|t}(c_{\emptyset})\leftarrow({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\boldsymbol{\epsilon}}_{\theta}(c_{\emptyset}))/\sqrt{\bar{\alpha}_{t}}$

$\hat{\boldsymbol{z}}_{0|t}(c_{tgt})\leftarrow({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\boldsymbol{\epsilon}}_{\theta}(c_{tgt}))/\sqrt{\bar{\alpha}_{t}}$

$\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}})\leftarrow\operatorname*{arg\,min}_{\boldsymbol{x}}\|{\boldsymbol{y}}-{\mathcal{A}}({\boldsymbol{x}})\|^{2}+\lambda\|{\boldsymbol{x}}-{\mathcal{D}}_{\varphi}(\hat{\boldsymbol{z}}_{0|t}(c_{\emptyset}))\|^{2}$

$\bar{\boldsymbol{z}}_{in}\leftarrow\bar{\alpha}_{t}\hat{\boldsymbol{z}}_{0|t}(c_{tgt})+(1-\bar{\alpha}_{t})^{2}\hat{\boldsymbol{z}}_{0|t}(c_{\emptyset})+\bar{\alpha}_{t}(1-\bar{\alpha}_{t}){\mathcal{E}}_{\phi}(\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}}))$

$\bar{\boldsymbol{z}}_{out}\leftarrow(1-\bar{\alpha}_{t})\hat{\boldsymbol{z}}_{0|t}(c_{\emptyset})+\bar{\alpha}_{t}{\mathcal{E}}_{\phi}(\hat{\boldsymbol{x}}_{0}({\boldsymbol{y}}))$

$\bar{\boldsymbol{z}}\leftarrow{\mathcal{M}}\odot\bar{\boldsymbol{z}}_{in}+(1-{\mathcal{M}})\odot\bar{\boldsymbol{z}}_{out}$

${\boldsymbol{z}}_{t-1}\leftarrow\sqrt{\bar{\alpha}_{t-1}}\bar{{\boldsymbol{z}}}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\boldsymbol{\epsilon}}_{t}$

else

$\hat{\boldsymbol{z}}_{0|t}(c_{\emptyset})=({\boldsymbol{z}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\boldsymbol{\epsilon}}_{\theta}(c_{\emptyset}))/\sqrt{\bar{\alpha}_{t}}$

${\boldsymbol{z}}_{t-1}^{\prime}\leftarrow\sqrt{\bar{\alpha}_{t-1}}\hat{{\boldsymbol{z}}}_{0|t}(c_{\emptyset})+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\boldsymbol{\epsilon}}_{t}$

${\boldsymbol{z}}_{t-1}\leftarrow{\boldsymbol{z}}^{\prime}_{t-1}-\rho_{t}\nabla_{{\boldsymbol{z}}_{t}}\|{\mathcal{A}}({\mathcal{D}}_{{\varphi}}(\hat{\boldsymbol{z}}_{0|t}))-{\boldsymbol{y}}\|$

end if

end for

##  7 Qualitative/Quantitative comparison

In this section, we conduct qualitative and quantitative comparison of the
DreamSampler for image editing and image inpainting tasks, against multiple
state-of-the-art diffusion-based algorithms. To ensure the fair comparison, we
leverage the StableDiffusion v1.5 checkpoint for all tasks.

###  7.1 Real Image Editing

For the real image editing, we compare the performance with diffusion-based
editing algorithms [15, 5, 17], following the experimental setup of [18] and
[19]. For the text prompt, we utilize the frame "a photography of {}", wherein
the class of the source text and the target object is inserted into the empty
space. For the CFG scale of DreamSampler, we use $0.15\bar{\alpha}_{t}$ for
all cases. Table S1 shows that DreamSampler outperforms most baselines which
indicates the efficiency of the DreamSampler in real image editing task. In
the case of "cat $\rightarrow$ cat w/ glasses," DreamSampler tends to generate
glasses smoothly, while baseline algorithms emphasize the generation of
glasses. This tendency makes baseline algorithms more sensitive to the CLIP
encoder’s ability to identify the existence of glasses compared to
DreamSampler, resulting in higher CLIP accuracy. However, as illustrated in
Figure S8, DreamSampler edits source images with higher fidelity while placing
less emphasis on the glasses.

Table S1: Comparison to diffusion-based editing methods. Dist for DINO-ViT
Structure Distance. Baseline results are from [18].

Method | Cat$\rightarrow$Dog | Horse$\rightarrow$Zebra | Cat$\rightarrow$ Cat w/ glasses
---|---|---|---
CLIP-Acc $\uparrow$ | Dist $\downarrow$ | CLIP-Acc $\uparrow$ | Dist $\downarrow$ | CLIP-Acc $\uparrow$ | Dist $\downarrow$
SDEdit + word swap | 71.2% | 0.081 | 92.2% | 0.105 | 34.0% | 0.082
DDIM + word swap | 72.0% | 0.087 | 94.0% | 0.123 | 37.6% | 0.085
prompt to prompt | 66.0% | 0.080 | 18.4% | 0.095 | 69.6% | 0.081
p2p-zero | 92.4% | 0.044 | 75.2% | 0.066 | 71.2% | 0.028
EBCA | 93.7% | 0.040 | 90.4% | 0.061 | 81.1% | 0.052
DreamSampler (Ours) | 90.3% | 0.029 |  95.2% | 0.038 | 48.3% | 0.025

Table S2: Comparison to diffusion-based inpainting solvers. Bold: the best
score, Underline: the second best.

Method | Glasses | Smile
---|---|---
PSNR $\uparrow$ | FID $\downarrow$ | CLIP-sim $\uparrow$ | PSNR $\uparrow$ | FID $\downarrow$ | CLIP-sim $\uparrow$
Stable-Inpaint | 19.82 | 54.26 | 0.281 | 25.33 | 19.22 | 0.249
TReg | 21.97 | 61.04 | 0.288 | 26.71 | 24.48 | 0.249
DreamSampler (Ours) | 24.61 | 27.10 | 0.263 | 27.90 | 24.33 | 0.242

Figure S8: Qualitative comparison for real image editing task. Text prompt "a
photography of cat wearing glasses" is given.

###  7.2 Text-guided Inpainting

For the text-guided inpainting task, we evaluate both the quality of
reconstructions and how well the text guidance is reflected. We compare the
performance with diffusion-based image inpainting algorithms [22, 13] and
text-guided inverse problem solver [9]. Specifically, Stable-Inpaint [22] is
fine-tuned StableDiffusion model for inpainting tasks. We leverage 1k FFHQ
validation images for the evaluation. For the mask, we generate rectangle
masks that cover the eyes and mouth of average face, and solve the inpainting
problem using text prompts "a photography of face wearing glasses" and "a
photography of face with smile". Specifically, we compute PSNR and FID scores
to measure the quality of the reconstructed solution and compute CLIP
similarity to assess how well the text prompt is reflected. The results in
Table S2 show that DreamSampler outperforms the baselines in terms of PSNR and
FID score while achieving comparable CLIP similarity. This implies that
DreamSampler generates the masked region according to the given text prompt
with high fidelity. Stable-Inpaint demonstrates a lower FID score, while
DreamSampler, serving as a zero-shot inpainting solver, achieves a higher
PSNR. This suggests superior reconstruction quality through data consistency
update.

Figure S9: Qualitative comparison for inpainting task with 512x512 FFHQ
dataset. Text prompt "A photography of face wearing glasses" is given. Figure
S10: Qualitative comparison for inpainting task with 512x512 FFHQ dataset.
Text prompt "A photography of face with smile" is given.

##  8 Ablation study

###  8.1 Effect of $\gamma$ in Real Image Editing

From the optimization problem (22), $\gamma$ could be interpreted as the
interpolation weight between ${\boldsymbol{z}}_{0|t}(c_{\phi})$ and
${\boldsymbol{z}}_{0|t}(c_{tgt})$, where $c_{\phi}$ is utilized for DDIM
inversion to initialize ${\boldsymbol{z}}_{T}$. Thus, setting $\gamma$ close
to 1 steers the solution of the problem towards
${\boldsymbol{z}}_{0|t}(c_{tgt})$, while setting $\gamma$ close to 0 directs
the solution towards ${\boldsymbol{z}}_{0|t}(c_{\phi})$. Considering the role
of $\gamma$ as a scale for the CFG, this interpretation aligns well. In this
study, we use time-dependent $\gamma_{t}=C\bar{\alpha}_{t}$ where $C$ denotes
a constant. To demonstrate the effect of $\gamma$ in a real image editing
task, we generated multiple images by varying the value of $C$ from 0.01 to
1.0. The results in Figure S11 demonstrate that when $C$ is smaller, the
generated image closely resembles the reconstruction, while when $C$ is
larger, the generated image adheres closely to the text guidance.

Figure S11: Ablation study on effects of $\gamma$ on real image editing task.

###  8.2 Effect of Distillation in Image Restoration through Vectorization

To examine the significance of text-guided distillation in the context of
restoration, we adjusted the parameter $\lambda_{SDS}=0$ in Equation (21) of
the main manuscript. Figure S12 highlights that relying solely on data
consistency regularization falls short of achieving a sufficient level of
restoration, manifested through the emergence of scattering artifacts within
the SVGs. This suggests that the absence of a text-guided distillation
refinement process may contribute to severe degradation.

Figure S12: Ablation study on effects of text-guided distillation on
restoration task.

##  9 Additional Results

###  9.1 3D representation learning using degraded view

Since the proposed framework is defined in terms of an arbitrary generator $g$
and generic parameter $\psi$, we apply Dreamsampler to the 3D NeRF [16]
representation learning, specifically targeting scenarios with degraded views.
We acknowledge that low-quality, noisy measurements can compromise the detail
of 3D modeling, aligning with the motivation of vectorized image restoration
task. In this context, the parameters $\psi$ of the generator consist of NeRF
MLP, which parameterizes volumetric density and albedo (color). Our goal is to
accurately reconstruct NeRF parameters that, when rendered, closely align with
the provided degraded view ${\boldsymbol{y}}$ and text conditions
$c_{\boldsymbol{y}}$.

For this NeRF inverse problem, we consider Gaussian blur operator
${\mathcal{A}}$ with $5\times 5$ kernel and standard deviation of $10$
following [11]. For a single input image, we introduce novel 16 degraded view
images with SyncDreamer [12] and blurring operator ${\mathcal{A}}$. Then, we
first pre-train NeRF with the data consistency regularization
$\ell({\boldsymbol{y}},{\mathcal{A}}g(\psi))$ for warm-up, where
$\ell:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$ represents a
generic loss function. To facilitate effective representation learning, we
integrate $\ell_{2}$-loss and perceptual similarity loss such as LPIPS [26]
for data consistency. Then the latent optimization framework of the NeRF
inverse problem is defined as follows:

$\min_{\psi}(1-\gamma)\lambda_{SDS}\|{\mathcal{E}}_{\phi}(g(\psi))-\hat{\boldsymbol{z}}_{0|t}(c_{{\boldsymbol{y}}})\|^{2}+\gamma\lambda_{DC}\|{\boldsymbol{y}}-{\mathcal{A}}g(\psi)\|^{2},$
(31)

which is analogous to the SVG inverse problem. We set
$\gamma=\bar{\alpha}_{t}$ as SVG experiments. Following [27], we additionally
adapt pixel-space distillation loss to enhance supervision for high-resolution
images such as

$\|g(\psi)-{\mathcal{D}}_{\varphi}(\hat{\boldsymbol{z}}_{0|t}(c_{{\boldsymbol{y}}}))\|^{2},$
(32)

where
${\mathcal{D}}_{\varphi}(\hat{\boldsymbol{z}}_{0|t}(c_{{\boldsymbol{y}}}))$
represents a recovered image via the decoder ${\mathcal{D}}_{\varphi}$. We
further employ $z$-coordinates regularization and kernel smoothing techniques
to improve sampling in high-density areas and address texture flickering
issues.

The findings, as illustrated in S13, reveal that Dreamsampler successfully
achieves high-quality 3D representation despite the compromised quality of the
degraded views. Thus, Dreamsampler demonstrates its efficacy in learning 3D
representations by directly leveraging degraded views and text conditions,
effectively facilitating both distillation and reconstruction processes.

Figure S13: Novel view synthesis via DreamSampler. Blurry views are given.

###  9.2 More results for Real Image Editing

Figure S14: Results for real image editing. Cat $\rightarrow$ Dog. Best views
are displayed.

Figure S15: Results for real image editing. Horse $\rightarrow$ Zebra. Best
views are displayed.

###  9.3 More results for Text-guided Inpainting

Figure S16: Results for text-guided inpainting with 512x512 FFHQ dataset. Text
prompt "wearing glasses" is given.

Figure S17: Results for text-guided inpainting with 512x512 FFHQ dataset. Text
prompt "smile" is given.

## References

  * [1] Chung, H., Kim, J., Mccann, M.T., Klasky, M.L., Ye, J.C.: Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 (2022)
  * [2] Chung, H., Lee, S., Ye, J.C.: Fast diffusion sampler for inverse problems by geometric decomposition. arXiv preprint arXiv:2303.05754 (2023)
  * [3] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, 8780–8794 (2021)
  * [4] Hertz, A., Aberman, K., Cohen-Or, D.: Delta denoising score. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2328–2337 (2023)
  * [5] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
  * [6] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [7] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
  * [8] Jain, A., Xie, A., Abbeel, P.: Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1911–1920 (2023)
  * [9] Kim, J., Park, G.Y., Chung, H., Ye, J.C.: Regularization by texts for latent diffusion inverse solvers. arXiv preprint arXiv:2311.15658 (2023)
  * [10] Li, T.M., Lukáč, M., Gharbi, M., Ragan-Kelley, J.: Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG) 39(6), 1–15 (2020)
  * [11] Liu, G.H., Vahdat, A., Huang, D.A., Theodorou, E.A., Nie, W., Anandkumar, A.: I2sb: Image-to-image schrodinger bridge. arXiv preprint arXiv:2302.05872 (2023)
  * [12] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023)
  * [13] Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L.: Repaint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11461–11471 (2022)
  * [14] Ma, X., Zhou, Y., Xu, X., Sun, B., Filev, V., Orlov, N., Fu, Y., Shi, H.: Towards layer-wise image vectorization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16314–16323 (2022)
  * [15] Meng, C., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 (2021)
  * [16] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [17] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6038–6047 (2023)
  * [18] Park, G.Y., Kim, J., Kim, B., Lee, S.W., Ye, J.C.: Energy-based cross attention for bayesian context update in text-to-image diffusion models. Advances in Neural Information Processing Systems 36 (2024)
  * [19] Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1–11 (2023)
  * [20] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
  * [21] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [22] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [23] Rout, L., Raoof, N., Daras, G., Caramanis, C., Dimakis, A., Shakkottai, S.: Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems 36 (2024)
  * [24] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
  * [25] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)
  * [26] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
  * [27] Zhu, J., Zhuang, P.: Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint arXiv:2305.18766 (2023)