*.jpg,.png,.PNG

#  Text-driven Visual Synthesis with Latent Diffusion Prior

Ting-Hsuan Liao Songwei Ge Yiran Xu Yao-Chih Lee Badour AlBahar Jia-Bin Huang
University of Maryland, College Park
http://latent-diffusion-prior.github.io

###### Abstract

There has been significant progress in using diffusion models for large-scale
text-to-image synthesis. This has led to versatile downstream applications
such as 3D object synthesis, image editing, and customized image generation.
In this paper, we present a generic approach that uses latent diffusion models
as powerful image priors for various visual synthesis tasks. Existing methods
that use these priors do not fully exploit the models’ capabilities. To
address this issue, we propose 1) a feature matching loss that provides
detailed guidance by comparing features from different decoder layers and 2) a
KL divergence loss that regularizes predicted latent features to stabilize the
training process. We apply our method to three applications: text-to-3D,
StyleGAN adaptation, and layered image editing, and demonstrate its efficacy
through extensive experimentation. Our results show that our method performs
favorably compared to baseline methods.

Text-to-3D

“A high quality photo of an ice cream sundae.”

“A high-quality photo of a colorful parrot, highly detailed.”

“A stuffed animal that is frowning is on a skateboard.”

“A red fire hydrant spraying water.”

Image editing

Input

“golden horse”

“long-mane horse”

“horse with giraffe skin”

StyleGAN adaptation

Source domain (FFHQ)

Target domains

Source domain (AFHQ)

Target domains

Figure 1:  Applications of the proposed latent diffusion prior. We showcase
various visual synthesis applications from the proposed latent diffusion
prior, including Text-to-3D, StyleGAN adaptation, and layered image editing.

##  1 Introduction

Diffusion models have shown impressive image generation capabilities in terms
of photorealism and compositionality [1, 30, 40, 42]. Through guidance control
and embedding techniques, text-to-image diffusion models have been applied to
various visual editing and processing tasks, _e.g_. customized image
generation [41], video generation [52], image and video editing [17, 18],
text-to-3D generation [22, 35], and image generator adaptation [45]. With a
free-form text prompt, these methods utilize a pre-trained text-to-image
diffusion model to guide the synthesis tasks. However, specialized methods are
required for each task to produce good results. While these methods may share
similar objectives, a unified approach is underexplored.

In this paper, we leverage a pretrained diffusion model as a generic image
prior for various visual synthesis applications. This is similar to the recent
line of work that uses a CLIP model [36]. CLIP models are trained to encode a
paired image and its caption and maximize their cosine similarity. Several
works have used a pretrained CLIP model as a prior to facilitate different
text-guided synthesis tasks, _e.g_., image and video editing [2, 33, 47, 53],
generator adaptation [6], text-to-3D synthesis [12, 27]. However, as CLIP
models are not generative models, their contrastive objective may not preserve
the visual information useful for synthesis-oriented tasks.

Diffusion models have recently shown great potential in these synthesis tasks
[17, 18, 22, 35, 45]. They can achieve competitive or even better performance
compared to CLIP-based approaches. However, unlike CLIP models where the
original CLIP objective was uniformly adopted, these works have applied
diffusion models differently under different tasks. In this work, we present a
framework to utilize diffusion models for various tasks.

We build our framework on Score Distillation Sampling [35], which uses an
image diffusion model [42] as prior to train a NeRF model without costly
backpropagating through the diffusion model itself. Other works [22, 25, 45,
49] follow a similar approach but are based on latent diffusion models [40].
The score distillation loss is computed in the latent space, which we term
“latent score distillation (LSD)”. Although the methods using score
distillation with trained diffusion models have shown promising results, the
loss is computed in a limited spatial resolution (_e.g_., $64\times 64$) and
thus cannot provide sufficient detailed guidance.

To achieve detailed guidance, we propose a new _Feature Matching Loss_ (FM)
loss that uses features in multiple decoder layers of the latent diffusion
model to guide the optimization process.

Another limitation of recent works utilizing LSD is the lack of regularization
over the optimized latent code. While minimizing the optimization loss in an
unconstraint manner, these methods are likely to produce out-of-distribution
latent code that is not seen by the decoder during the training, resulting in
a lower-quality output. To mitigate this issue, we propose to use a KL
divergence loss to regularize the optimized latent so that they stay close to
the prior distribution during the training.

We evaluate our method on three text-driven synthesis applications, text-to-3D
generation, layered image editing, and StyleGAN adaptation (as shown in Fig.
1). Our approach achieves competitive results over baselines using the CLIP
model as the image prior and diffusion models with latent score distillation.

We make the following contributions:

  * •

We propose a feature matching loss to extract detailed information from the
decoder to guide text-based visual synthesis tasks.

  * •

We propose a KL divergence loss to regularize the optimized latent,
stabilizing the optimization process.

  * •

We extensively evaluate our method on three downstream tasks and show
competitive results with strong baselines.

Figure 2:  Method overview. Our method aims to guide the generation and
editing process based on a text prompt. We obtain the latent code from a
differentiable renderer under different applications, including the generator
from Text2LIVE [2], StyleGAN-based generator, or a NeRF model. This latent
code $\mathbf{v}$ is perturbed following the latent diffusion model’s
scheduler at a random time step $t$, such that
$F_{t}:\mathbf{z}_{t}=\alpha_{t}\mathbf{v}+\sigma_{t}\mathbf{\epsilon}$. This
perturbed latent code $\mathbf{z}_{t}$ is then passed to the UNet to generate
the predicted noise $\hat{\epsilon}$. We then use the predicted noise
$\hat{\epsilon}$ to derive the _latent score distillation_ gradient. To derive
the _feature matching_ gradient, we input the latent code $v$ and noised
latent code $\mathbf{v}+(\hat{\mathbf{\epsilon}}-\mathbf{\epsilon})$ into the
decoder $G_{\phi_{dec}}(\cdot)$. We compute the difference between the multi-
level features from three different layers of the decoder to compute the
feature matching loss. Finally, both the _latent score distillation_ and
_feature matching_ gradients are backpropagated to the differentiable
renderer.

##  2 Related Work

Text-to-image diffusion models. Diffusion models [10, 16, 44] synthesize
images by denoising independent noises drawn from the standard Gaussian
diffusion. Impressive progress has been made in photorealistic and zero-shot
text-to-image generation [1, 5, 30, 38, 40, 42, 56]. When combined with tricks
such as classifier-free guidance [11] and gradient guidance [24], the
flexibility of the diffusion model makes it readily adapt to different
conditional generation tasks that are not part of its training objectives,
such as image editing [17, 48], and personalized image generation [41]. Our
work capitalizes on the recent progress text-to-image diffusion model and uses
it as image priors for various visual synthesis tasks.

Text-driven 3D generative models. With the guidance from a Contrastive
Language-Image Pre-training (CLIP) model [36], text-guided 3D generation
becomes possible together with a differentiable renderer [12, 13, 27, 43]. By
optimizing the 3D representation through the CLIP objective computed on the
rendered image, they can explicitly control the pose of generated 3D shapes,
and generate creative appearances and shapes from the text freely. More
recently, DreamFusion [35] has brought diffusion models to this task and shown
impressive results. In contrast to [7] backpropagating through the pre-trained
diffusion model itself, DreamFusion leverages noise residual predicted by the
pre-trained diffusion model as gradients for efficient backpropagation.
Follow-up works [25, 49] adapt the method to latent diffusion models [40].
However, the 3D models resulting from these methods still lack details since
the latent score distillation is computed in the latent space with a limited
spatial resolution (_e.g_., $64\times 64$ latent space for a $512\times 512$
image). In this work, we leverage the knowledge embedded in the _latent
diffusion decoder_ to provide more detailed guidance. We show better details
can be achieved in the synthesized 3D models.

Image generator domain adaptation. Some works aim to fine-tune a pre-trained
generator with few-shot or text-guided zero-shot domain adaptation to reduce
the cost of training an image generator. Few-shot adaption aims to use fewer
data (typically hundreds or fewer) to train an image generator. Previous works
control the learnable parameters [26, 31, 39, 51], design regularizers [21,
34, 46], or use auxiliary tasks [23, 55] to improve the quality of the
generator. With the advancement of large fundamental models, some works use
them as guidance to achieve zero-shot adaptation. StyleGAN-NADA [6] uses pre-
trained CLIP model [36] as guidance and shows diverse domain adaption results
on StyleGAN [14]. StyleGANFusion [45] leverages pre-trained StableDiffusion
[40] and follow the score distillation approach proposed by DreamFusion [35]
to achieve even better results. Building upon StyleGANFusion [45], we show
that using our proposed method produces a significantly better FID score and a
competitive CLIP score.

Text-driven image editing. Pioneering image manipulation methods [20, 29]
utilize GANs to achieve editing of appearances while preserving the shape.
However, the training image domains and text expression constraints often
restrict the GAN-based approaches. The advanced methods have leveraged
embeddings from a pretrained CLIP [36]. These embeddings can be applied to
update the generative models with test-time optimization [2, 6, 19, 33, 54].
Recently, text-to-image diffusion models have shown exceptional success in
manipulation tasks [1, 3, 17, 28, 30, 41]. We demonstrate the application of
our diffusion prior to the image editing task. Unlike existing diffusion-based
editors, our method manipulates images using test-time optimization with the
proposed diffusion guidance. Our method produces more detailed results than
the latent diffusion-guided baselines and the CLIP-guided method, Text2LIVE
[2].

##  3 Proposed Method

###  3.1 Background

Score distillation. DreamFusion [35] first proposes using the Imagen [42]
diffusion model for a text-driven 3D generation. Imagen can generate _high-
resolution_ images by cascade diffusion. However, the base model only operates
on a _low-resolution_ $64\times 64$ image space. They build upon the
formulation introduced by [7] and propose a more stable training process.
Their approach optimizes the underlying 3D representation model by using the
training objective of the diffusion model. The resulting approach, score
distillation sampling, involves perturbing the rendered image with a random
noise $\mathbf{\epsilon}$, and then using the pretrained diffusion model to
predict the noise $\hat{\mathbf{\epsilon}}$. The training is based on the
gradient computed from the _noise residual_ between the added random noise and
the predicted one: $\hat{\mathbf{\epsilon}}-\mathbf{\epsilon}$. Note that the
Imagen model [42] only operates on low-resolution image space.

Latent score distillation. Jacobian NeRF [49], Latent NeRF [25] and
StyleGANFusion [45] have recently incorporated score distillation into the
latent diffusion models [40]. A latent diffusion model generally contains two
components, an autoencoder with encoder $E_{\phi_{enc}}$ and decoder
$G_{\phi_{dec}}$, and a diffusion model $\hat{\epsilon}_{\phi_{UNet}}$
operating in the latent space. However, the decoder is not utilized in the
latent score distillation of [25, 45, 49], which can lead to inferior results
since the knowledge of converting low-resolution latent to high-resolution RGB
images is not used. To address this, we propose using feature matching and KL
losses to reintroduce the decoder into the optimization procedure.

###  3.2 Feature matching Loss

We propose a _feature matching_ loss to leverage the generative capacity of
the decoder $G_{\phi_{dec}}$ and provide finer-grained guidance to the
differentiable renderer. We use the decoder of the stable diffusion
autoencoder as shown in Figure 2. We feed the original latent code
$\mathbf{v}$ and the updated latent code
$\mathbf{v}^{\prime}=\mathbf{v}+(\hat{\mathbf{\epsilon}}-\mathbf{\epsilon})$
to the decoder $G_{\phi_{dec}}(\cdot)$, and then compute the _feature
matching_ loss:

$L_{FM}=\sum_{\ell}\|G^{\ell}_{\phi_{dec}}\left(\mathbf{v}\right)-G^{\ell}_{\phi_{dec}}\left(\mathbf{v}+\hat{\mathbf{\epsilon}}-\mathbf{\epsilon}\right)\|_{1},$

where $\ell$ is the specific level in the decoder $G_{\phi_{dec}}$.

Our proposed feature matching loss, denoted as $L_{FM}$, is inspired by the
GAN discriminator feature matching loss proposed in pix2pixHD [50], which aims
to align the features of multiple discriminator layers of real and synthetic
images. Our feature matching loss $L_{FM}$ operates similarly to the latent
score distillation loss $L_{LSD}$, but its signal is _amplified_ through the
use of the decoder $G_{\phi_{dec}}$. Our approach differs from the feature
matching loss in pix2pixHD in two major aspects. First, we measure the
similarity of the extracted features from the pretrained (and fixed) _decoder_
, not from an additional trainable discriminator. Second, we use latent code
_with added noise residual_ and the clean latent as the decoder input, as
opposed to real/fake images.

We denote $\mathbf{\theta}$ as the target parameters in the renderer
$g_{\mathbf{\theta}}(\cdot)$ to optimize, _e.g_. NeRF model for text-to-3D
generation and the StyleGAN model for generator adaptation. The direct
gradient computation involves calculating the UNet Jacobian, which is
computationally expensive [35]. We therefore consider the gradient
backpropagation to the generator $\frac{dL_{FM}}{d\mathbf{\theta}}$ using the
chain rule:

$\displaystyle\frac{dL_{FM}}{d\mathbf{\theta}}$
$\displaystyle=\frac{\partial\mathbf{v}}{\partial\mathbf{\theta}}\frac{\partial
L_{FM}}{\partial\mathbf{v}}$ (1)
$\displaystyle=\frac{\partial\mathbf{v}}{\partial\mathbf{\theta}}\left(\alpha_{t}\frac{\partial\hat{\mathbf{\epsilon}}}{\partial\mathbf{z}_{t}}\frac{\partial\mathbf{x}^{\prime}}{\partial\mathbf{v}^{\prime}}\frac{\partial
L_{FM}}{\partial\mathbf{x}^{\prime}}+\frac{\partial\mathbf{x}^{\prime}}{\partial\mathbf{v}^{\prime}}\frac{\partial
L_{FM}}{\partial\mathbf{x}^{\prime}}+\frac{\partial\mathbf{x}}{\partial\mathbf{v}}\frac{\partial
L_{FM}}{\partial\mathbf{x}}\right),$

where $\frac{\partial\hat{\mathbf{\epsilon}}}{\partial\mathbf{z}_{t}}$ refers
to the UNet Jacobian term, which we ignore due to the high optimization cost.
$\mathbf{x}$ and $\mathbf{x}^{\prime}$ denote the latent feature within the
decoder obtained by the original $\mathbf{v}$ and updated latent code
$\mathbf{v}^{\prime}$. $\mathbf{z}_{t}$ refers to the perturbed latent code
defined as $\mathbf{z}_{t}=\alpha_{t}\mathbf{v}+\sigma_{t}$, where
$\alpha_{t}$ and $\sigma_{t}$ are time-dependent constant defined in DDPM
[10].

The final gradient can be derived as follows:

$\displaystyle\frac{dL_{FM}}{d\mathbf{\theta}}$
$\displaystyle=\frac{\partial\mathbf{v}}{\partial\mathbf{\theta}}\left(\left(1+\alpha_{t}\right)\frac{\partial\mathbf{x}^{\prime}}{\partial\mathbf{v}^{\prime}}\frac{\partial
L_{FM}}{\partial\mathbf{x}^{\prime}}+\frac{\partial\mathbf{x}}{\partial\mathbf{v}}\frac{\partial
L_{FM}}{\partial\mathbf{x}}\right).$ (2)

(a) Text-to-3D

(b) StyleGAN adaptation

(c) Layered image editing

Figure 3:  Applications pipeline. To apply our proposed feature matching loss
$L_{FM}$, KL regularizer loss $L_{KL}$ and score distillation loss $L_{LSD}$,
we first obtain the latent code $\mathbf{v}$ using the differentiable renderer
in each application. As illustrated in the figure, to obtain the image that
produces $\mathbf{v}$ with StableDiffusion encoder $E_{\phi_{enc}}$, in (a)
Text-to-3D, we render from a NeRF model with a random camera viewpoint; (b)
StyleGAN adaptation, we generate the image with a pretrained StyleGAN model;
(c) Layered image editing application, we use the generator of Text2LIVE to
synthesize the edited image, alpha map, and the alpha blending of the initial
and edited images.

###  3.3 Kullback-Leibler divergence regularizer.

When conducting optimization through the latent code in an unconstrained way
[25, 35, 49], the resulting latent code likely drifts away from the original
distribution. Consequently, the decoder must handle an _unseen input_ during
the training and produces poor quality. Our feature matching loss mitigates
this issue by incorporating the gradient from the decoder. However, it is
insufficient in practice as we still observe artifacts or unrealistic styles
in the decoded image. Therefore, we propose further regularizing the latent
with a KL divergence loss.

Stable Diffusion and VAE models both use a KL penalty to regularize the
variance of the latent space. However, in the text-to-3D task, a latent
radiance field is built to directly predict the latent code without an encoder
$E_{\phi_{enc}}$. Thus, different from the training process, we do not compute
the KL penalty on the mean and variance output by the encoder but directly on
the latent sample $v$ as below:

$L_{KL}=\frac{1}{2}\left(\mu_{\mathbf{v}}^{2}+\sigma^{2}_{\mathbf{v}}-log\left(\sigma^{2}_{\mathbf{v}}\right)+1\right),$
(3)

where $\mu_{\mathbf{v}}=\frac{1}{N}\sum_{i}\mathbf{v}_{i}$ and
$\sigma^{2}_{\mathbf{v}}=\frac{1}{N}\sum_{i}(\mathbf{v}_{i}-\mu_{\mathbf{v}})^{2}$
represent the mean and variance of the latent code $\mathbf{v}$, respectively.
And $N$ denotes the number of elements of the latent code $\mathbf{v}$. We
find this to be effective in improving the stability of the optimization
process.

###  3.4 Training procedure

The diffusion prior includes three parts, _latent score distillation_ ,
_feature matching loss_ , and _KL regularizer_.

During the optimization, the latent code $\mathbf{v}$ is first perturbed
following the DDPM [10] scheduler at a random time step $t$, such that
perturbed latent code
$\mathbf{z}_{t}=\alpha_{t}\mathbf{v}+\sigma_{t}\mathbf{\epsilon}$. This
perturbed latent code $\mathbf{z}_{t}$ is then passed to the UNet to generate
the predicted noise $\hat{\mathbf{\epsilon}}$. We then use the predicted noise
$\hat{\mathbf{\epsilon}}$ to derive the _latent score distillation_ gradient.
We define the latent score distillation loss as $L_{LSD}$ in this paper. To
compute the feature matching loss, we input the latent code $\mathbf{v}$ and
updated latent code $\mathbf{v}+(\hat{\mathbf{\epsilon}}-\mathbf{\epsilon})$
into the decoder $G_{\phi_{dec}}(\cdot)$. We use the decoded features at three
different layers from the decoder, to compute the feature matching loss.

We compute the gradient partial to the latent code $\mathbf{v}$, and propagate
back to optimize $g_{\mathbf{\theta}}$ using the final loss $L_{final}$:

$L_{final}=\lambda_{1}L_{FM}+\lambda_{2}L_{KL}+\lambda_{3}L_{LSD},$ (4)

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are the balancing factor for each
loss.

##  4 Experiments

We evaluated three applications using Stable Diffusion [40] v1.4 for Text-
to-3D (Latent NeRF) and StyleGAN adaptation, and Stable Diffusion v1.5 for
Text-to-3D (Jacobian NeRF) and layered image editing as our pretrained
diffusion model. Fig. 3 depicts the overall pipeline for each application and
demonstrates how we acquire the latent code $\mathbf{v}$ from each
differentiable generator used in the three applications. We include the
implementation details and more results in the supplementary material and we
will make the source code publicly available.

###  4.1 Applications

burger111“A high quality photo of a delicious burger.”

koi fish

blue jug222“A high quality photo of a jug made of blue and white porcelain.”

CLIP-Mesh [27]

DreamField [12]

Latent NeRF [25]

Ours (Latent NeRF)

Jacobian NeRF [49]

Ours (Jacobian NeRF)

Figure 4: Comparisons on Text-to-3D. We compare our method with CLIP-Mesh
[27], DreamField [12], Latent NeRF [25] and Jacobian NeRF [49]. Our approach
can provide more details and a better appearance. For instance, the hamburger
appears with clearer layers, while the fish and vase possess a more solid
structure. We show the complete text prompts in the footnote.

Text-to-3D. The text-to-3D task aims to generate a 3D model from a free-form
text description. We evaluate our method on two text-guided 3D generative
models, Jacobian NeRF [49], and Latent NeRF [25] using the Stable Diffusion as
guidance. Both methods learn and predict the latent code within each viewing
point and optimize with the latent score distillation. We applied the proposed
feature matching loss $L_{FM}$ and KL regularizer $L_{KL}$ to both methods. We
also compare our results with two other baselines, CLIP-Mesh [27] and
DreamFields [12], which leverage CLIP [36] as guidance instead. As shown in
Fig. 4, Ours (Latent NeRF) and Ours (Jacobian NeRF) using our proposed losses
upon Latent NeRF and Jacobian NeRF offer better quality with more details.

We further evaluate the generated 3D model with the CLIP-R precision score
[32], comparing with DreamFields [12], CLIP-Mesh [27], Latent NeRF [25] and
Jacodian NeRF [49]. We follow the same experiment setup outline in CLIP-Mesh
[27], which involved generating one 3D model per prompt in 153 text prompts.
We take a rendered image from a random camera pose during the evaluation to
compute the CLIP score with 99 random prompts and the generated text prompt.
We test two different-sized CLIP models for computing the precision. As shown
in Table 1, our method consistently improves over both baseline methods [25,
49]. Note that both DreamFields and CLIP-Mesh [12, 27] _optimize directly with
CLIP loss_ , which can lead to potential overfitting issues as discussed in
[12]. Our focus is _not_ on achieving state-of-the-art text-to-3D results but
showing how our proposed losses complement the commonly used latent score
distillation. The SOTA methods [22, 35] rely on proprietary _image-based_
diffusion models (Imagen [42] and e-Diff [1]) that are not publicly available.
Magic3D [22], however, does use a latent diffusion model in their second stage
of training. We believe that our proposed method can also be applied to
improve the results.

Table 1: R-Precision: the coherence of our model generations with their caption using different CLIP retrieval models. Method | CLIPB/32 | CLIPB/16
---|---|---
GT Images | 77.1 | 79.1
DreamFields [12] | 68.3 | 74.2
CLIP-Mesh [27] | 67.8 | 75.8
Latent NeRF [25] |  29.8 $\pm$ 1.54 |  37.7 $\pm$ 2.74
Ours (Latent NeRF) |  35.1 $\pm$ 1.63 |  39.4 $\pm$ 3.03
Jacobian NeRF [49] |  31.2 $\pm$ 0.82 |  44.0 $\pm$ 2.41
Ours (Jacobian NeRF) |  33.3 $\pm$ 0.53 |  46.0 $\pm$ 1.34
Table 2: We compare our method with two baselines, StyleGANFusion [45] and
StyleGAN-NADA [6], in Cat-to-Animals and report the average FID scores (the
lower the better). Our method outperforms other baselines in most of the
experiments.

| StyleGANFusion | StyleGAN-NADA | Ours
---|---|---|---
*○ Dog | 133.68 | 228.59 | 132.35
*○ Fox | 47.05 | 144.89 | 52.73
*○ Lion | 28.70 | 170.42 | 25.69
*○ Tiger | 45.67 | 173.47 | 39.54
*○ Wolf | 59.43 | 161.47 | 53.82

StyleGAN adaptation. We also apply our proposed method to image generator
adaptation. We conduct experiments on StyleGAN2 [15] with our feature matching
loss $L_{FM}$ and KL loss $L_{KL}$. To demonstrate the effectiveness of our
method, we compare our approach StyleGANFusion [45] and StyleGAN-NADA [6]. We
follow the three metrics used in [45] to evaluate different methods: FID score
[9], CLIP score [37], and LPIPS score [57]. For our method, we apply the
feature matching loss $L_{FM}$ to StyleGANFusion as additional guidance with
KL loss $L_{KL}$.

bear →polar bear

blackswan→whiteswan

horse →zebra

cat→tiger

Input

Text2LIVE [2]

$L_{LSD}$ baseline

Ours ($L_{FM}+L_{LSD}+L_{KL}$)

Figure 5: Comparisons on layered image editing. We compare our method with the
SOTA Text2LIVE [2] and the latent diffusion prior $L_{LSD}$ baseline. The
leftmost column shows the editing text prompts. The zoom-in windows
(highlighted in green) are shown at the right-bottom of the images to allow
comparisons in detail areas. Our method can produce reasonable editing and
more details than the baseline. Table 3: We compare our method with StyleGAN-
NADA [6] and StyleGANFusion [45] on animal experiments. CLIP score is computed
on generated samples and input text prompts to examine the text-image
similarity. LPIPS is computed between all possible pairs in the generated
samples to evaluate the diversity of images.

| StyleGANFusion | StyleGAN-NADA | Ours
---|---|---|---
Cat $\rightarrow$ | CLIP$\uparrow$ | LPIPS$\uparrow$ | CLIP$\uparrow$ | LPIPS$\uparrow$ | CLIP$\uparrow$ | LPIPS$\uparrow$
*○ Dog | 0.285 | 0.584 | 0.288 | 0.457 | 0.289 | 0.594
*○ Hamster | 0.333 | 0.544 | 0.352 | 0.460 | 0.329 | 0.569
*○ Badger | 0.311 | 0.554 | 0.368 | 0.435 | 0.297 | 0.555
*○ Fox | 0.321 | 0.560 | 0.343 | 0.554 | 0.318 | 0.536
*○ Otter | 0.323 | 0.523 | 0.368 | 0.389 | 0.317 | 0.542
*○ Lion | 0.299 | 0.577 | 0.316 | 0.478 | 0.302 | 0.555
*○ Bear | 0.294 | 0.567 | 0.332 | 0.399 | 0.305 | 0.555
*○ Pig | 0.318 | 0.586 | 0.322 | 0.549 | 0.306 | 0.596

Cat $\rightarrow$ fox

prompt 1333“3d cute cat, closeup cute and adorable, cute big circular
reflective eyes, long fuzzy fur, Pixar render, unreal engine cinematic smooth,
intricate detail, cinematic”

Cat $\rightarrow$ hamster

prompt 2444“3d human face, closeup cute and adorable, cute big circular
reflective eyes, Pixar render, unreal engine cinematic smooth, intricate
detail, cinematic”

Cat $\rightarrow$ lion

prompt 3555“photo of a face [SEP] A realistic detailed portrait, single face,
science fiction, artstation, volumetric lighting, octane render”

Source

StyleGANFusion

StyleGAN-NADA

Ours

Source

StyleGANFusion

StyleGAN-NADA

Ours

Figure 6: Comparison with StyleGANFusion [45] and StyleGAN-NADA [6]. We show
uncurated samples from different methods. We test _short prompts_ like “photo
of a/an X” on the left, and _long prompts_ on the right. Compared to
baselines, our approach shows higher fidelity (_e.g_., whiskers) and keeps the
attributes from the source image (_e.g_., pupils). We show the complete text
prompts in the footnote.

We generate 2,000 samples from the adapted generator after the training for
the quantitative evaluations. We first show FID scores of different approaches
on various target domains in Table 2. Groundtruth images are extracted from
AFHQ dataset [4]. Some of the labels are provided by [45]. Following
StyleGANFusion [45], we compare the FID scores on “Cat $\rightarrow$
Dog/Fox/Lion/Tiger/Wolf”. Our FID scores outperform other baselines in most
cases. This indicates that our method can help gain a more similar
distribution to the target domain and improve the quality.

$L_{LSD}$

$L_{LSD}+L_{KL}$

$L_{FM}$

$L_{FM}+L_{KL}$

$L_{FM}+L_{LSD}$

$L_{FM}+L_{LSD}+L_{KL}$

Figure 7:  Ablation study. We test our two component multi-level feature
matching loss $L_{FM}$ and KL regulizier $L_{KL}$on an image-syntheis.

Text-to-3D

StyleGAN adaptation

Layered image editing

View 1

View 2

Sample 1

Sample 2

Input

“rusty car”

Figure 8:  Limitations. Text-to-3D: multi-face, StyleGAN adaptation: out-of-
focus, Layered image editing: over-saturation issue.

CLIP score is used to evaluate the similarity between the text prompt and the
output image. LPIPS score measures all the pairs in the samples to examine the
_diversity_ of the generated samples. Our method achieves competitive CLIP and
LPIPS scores compared to the baselines. For some domains, _e.g_., Dog,
Hamster, Badger, Otter, and Pig, our LPIPS score outperforms other baselines
by a large margin. Note that the results shown here have discrepancies with
the reported results in StyleGANFusion [45]. We consulted with the authors and
were informed that the reported results were obtained from _per-dataset
tuning_. We run the experiments with the default parameters provided by the
authors. The qualitative comparisons of human faces and cats are illustrated
in Fig. 6. As reported in [45], StyleGAN-NADA cannot handle long prompts well
but achieves competitive CLIP scores for short prompts. We show results with
short prompts, like “a photo of X”, and long prompts. Other baselines tend to
have over-smoothed texture (StyleGANFusion) or lower fidelity (StyleGAN-NADA).
Our method can generate sharper textures and better overall image quality.

Layered image editing. We demonstrate the application on text-driven image
editing tasks. We follow the layered-editing approach in Text2LIVE [2], which
utilizes a CNN generator to output an RGBA editing layer blended with the
input image. The generator is trained with CLIP [36] guidance via test-time
optimization. We adopt the same CNN generator $\theta_{CNN}$ and introduce an
additional trainable latent code $\theta_{latent}$, allowing to generate
richer details. The trainable parameter set is
$\theta=\\{\theta_{CNN},\theta_{latent}\\}$. The parameters are updated using
a combination of latent score distillation $L_{LSD}$ and the proposed _feature
matching loss_ $L_{FM}$ along with KL loss $L_{KL}$. Besides, we use object
mask supervision for the blending alpha map to improve editing quality.

We compare our method with Text2LIVE, and the latent-score-distillation-only
(_i.e_., $L_{LSD}$) baseline with the same configuration as our model in Fig.
5. Text2LIVE fails to manipulate the appearance effectively and produces
noticeable artifacts, particularly in the “zebra” case. On the other hand,
while the $L_{LSD}$-only baseline can guide the editing effectively, it fails
to generate fine details. It cannot adequately synthesize the zebra stripes in
Example 2 and the tiger’s face in Example 3. In contrast, our proposed method
can generate more detailed results than the $L_{LSD}$-only baseline, such as
the feathers of the white swan in Example 1. Overall, our proposed method
produces visually better results and richer details than the baselines.

###  4.2 Ablation

We conduct an ablation study in Fig. 7 to assess the effectiveness of the
feature matching loss $L_{FM}$ and the KL regularizer $L_{KL}$, along with the
baseline latent score distillation loss $L_{LSD}$ in a simple image-synthesis
task. The results demonstrate that incorporating $L_{FM}$ and $L_{KL}$
improved the overall image quality performance. Specifically, the feature
matching loss $L_{FM}$ is able to bring more detail to the image, while the KL
regularizer mitigates the color over-saturation problem. For more quantitative
and qualitative results of ablation studies on the three applications, please
refer to the supplementary material.

##  5 Implementation Details

In this section, we elaberate our implemantation detail for each application.

###  5.1 Text-to-3D.

We evaluate our method on two text-guided 3D generative models, Latent NeRF
[25] and Jacobian NeRF [49]. We follow the original setup of using Stable
Diffusion [40] v1.4 for Latent NeRF and Stable Diffusion v1.5 for Jacobian
NeRF. We integrate their method with our $L_{FM}$ and KL regularizer. For
Jacobian NeRF, we run experiments for 10,000 iterations, and for Latent NeRF,
we run experiments for 5,000 iterations with an additional 2,000 refined
iterations, following the default setting. The objective after integrating our
method will be $L=\lambda_{1}L_{FM}+\lambda_{2}L_{KL}+\lambda_{3}L_{LSD}$,
where $\lambda_{1}=10^{-1},\lambda_{2}=10^{-1},\lambda_{3}=1.0$ for Jacobian
NeRF and $\lambda_{1}=10^{-2},\lambda_{2}=10^{-1},\lambda_{3}=1.0$ for Latent
NeRF.

###  5.2 StyleGAN adaptation.

We implement our method on StyleGAN2 [15] with our feature matching loss
$L_{FM}$. Our method is built upon StyleGANFusion [45]. We add our $L_{FM}$
and KL regularizer. We run the experiments for 2,000 training iterations and a
learning rate of 0.002. We generate 2,000 samples from the adapted generator
after the training for the quantitative evaluations. The objective after
integrating our method will be
$L=\lambda_{1}L_{FM}+\lambda_{2}L_{KL}+\lambda_{3}L_{LSD}$, where
$\lambda_{1}=3.0,\lambda_{2}=0,5\times 10^{-1},\lambda_{3}=1.0$.

###  5.3 Layered image editing

We follow the layered image editing approach of Text2LIVE [2]. As illustrated
in Fig. 9, we first feed the input image $I$ to the CNN generator of Text2LIVE
to produce an initial editing layer and alpha map $(I^{\prime}_{CNN},\alpha)$.
The initial edited image can be synthesized by alpha blending
$I^{\prime}=I^{\prime}_{CNN}\cdot\alpha+I\cdot(1-\alpha)$. We then input the
initial edited image $I^{\prime}$ into the frozen encoder $E_{\phi_{enc}}$ of
the latent diffusion model [40] to obtain the initial latent
$\mathbf{v}^{\prime}$. We exploit an additional learnable residual latent
$\theta_{latent}$ to add to the initial latent
$\mathbf{v}^{\prime\prime}=\mathbf{v}^{\prime}+\theta_{latent}$ to obtain
richer details. Lastly, the frozen decoder $G_{\phi_{dec}}$ generates the
final edited image $I^{\prime\prime}$ from the latent
$\mathbf{v}^{\prime\prime}$ followed by the same alpha blending. The parameter
set $\\{\theta_{CNN},\theta_{latent}\\}$ (_i.e_., the parameters of
Text2LIVE’s CNN generator and the additional residual latent) is trained by
the diffusion prior, either the $L_{LSD}$-only baseline or our proposed
$L_{FM}+L_{KL}+L_{LSD}$ full loss. We compute the diffusion-guided losses on
$\mathbf{v}^{\prime\prime}$ and $I^{\prime\prime}$. In addition, to enhance
the editing quality, we use an additional mask supervision loss on learning
the alpha map $\alpha$. The final loss is
$L_{final}=\lambda_{1}L_{FM}+\lambda_{2}L_{KL}+\lambda_{3}L_{LSD}+\|\alpha-M\|_{1}$,
where $\lambda_{1}=10^{-5},\lambda_{2}=10^{-7},\lambda_{3}=10^{-6}$, and $M$
is the mask obtained from MaskRCNN [8].

Figure 9: Layered image editing pipeline.

###  5.4 Image Synthesis

We conduct an ablation study of the two primary components of our proposed
method, the feature matching loss $L_{FM}$ and the KL regularizer $L_{KL}$ on
a straightforward image synthesis task. We begin by initializing a $64\times
64\times 4$ random noise map and then optimizing it with a combination of
different losses for 1000 iterations. The final results are rendered from the
decoder of the pretrained diffusion model. We use the AdamW optimizer with a
learning rate of $10^{-1}$ for optimization. The objective of the full model
is $L=\lambda_{1}L_{FM}+\lambda_{2}L_{KL}+\lambda_{3}L_{LSD}$, where
$\lambda_{1}=3.0,\lambda_{2}=10^{-1},\lambda_{3}=1.0$.

##  6 Limitations

In Text-to-3D task, the learned 3D objects sometimes encounter the multiple
faces Janus problem (Fig. 8 first case), in which the model will appear with
more than one front face. The problem also comes up in DreamFusion, which
might be caused by less knowledge of 3D geometry in the diffusion model. Or as
mentioned in DreamFusion and Latent NeRF, the text prompt often interprets
from the canonical views and might not be a good condition for sampling other
viewpoints. Besides, with the guidance of diffusion score distillation, we
found that few cases would suffer from the issue of color over-saturation and
out-of-focus despite the use of our KL loss $L_{KL}$ (Fig. 8 second and third
case). The over-saturation issue can also be found in previous score
distillation works [25, 35].

##  7 Conclusions

In this paper, we proposed a framework that uses a diffusion model as a prior
for visual synthesis tasks. Our core contributions are a feature-matching loss
to extract detailed information and a KL loss for regularization. Through
extensive experimental evaluations, we show that our method improves the
quality compared to strong baselines on text-to-3D, StyleGAN adaptation, and
layered image editing tasks.

## References

  * [1] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu.  ediff-i: Text-to-image diffusion models with ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  ECCV, 2022.
  * [3] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  arXiv preprint arXiv:2211.09800, 2022.
  * [4] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In CVPR, 2020.
  * [5] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors, 2022\.
  * [6] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  ACM Transactions on Graphics (TOG), 2022.
  * [7] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras.  Diffusion models as plug-and-play priors.  In NeurIPS, 2022.
  * [8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn, 2017.
  * [9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  NeurIPS, 2017.
  * [10] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  arXiv preprint arxiv:2006.11239, 2020.
  * [11] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance, 2022.
  * [12] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In CVPR, 2022.
  * [13] Nikolay Jetchev.  Clipmatrix: Text-controlled creation of 3d textured meshes.  ArXiv, abs/2109.12922, 2021.
  * [14] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [15] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, 2020.
  * [16] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising diffusion restoration models.  In Advances in Neural Information Processing Systems, 2022.
  * [17] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [18] Gyeongman Kim, Hajin Shim, Hyunsu Kim, Yunjey Choi, Junho Kim, and Eunho Yang.  Diffusion video autoencoders: Toward temporally consistent face video editing via disentangled video encoding.  arXiv preprint arXiv:2212.02802, 2022.
  * [19] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  In CVPR, 2022.
  * [20] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr.  Manigan: Text-guided image manipulation.  In CVPR, 2020.
  * [21] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman.  Few-shot image generation with elastic weight consolidation.  In NeurIPS, 2020.
  * [22] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  arXiv preprint arXiv:2211.10440, 2022.
  * [23] Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal.  Towards faster and stabilized gan training for high-fidelity few-shot image synthesis.  In ICLR, 2020.
  * [24] Cheng Ma, Yongming Rao, Yean Cheng, Ce Chen, Jiwen Lu, and Jie Zhou.  Structure-preserving super resolution with gradient guidance.  In CVPR, 2020.
  * [25] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In CVPR, 2023.
  * [26] Sangwoo Mo, Minsu Cho, and Jinwoo Shin.  Freeze the discriminator: a simple baseline for fine-tuning gans.  In CVPR AI for Content Creation Workshop, 2020.
  * [27] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  In SIGGRAPH Asia 2022 Conference Papers, 2022.
  * [28] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  arXiv preprint arXiv:2211.09794, 2022.
  * [29] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.  Text-adaptive generative adversarial networks: manipulating images with natural language.  NeurIPS, 2018.
  * [30] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  CoRR, abs/2112.10741, 2021.
  * [31] Atsuhiro Noguchi and Tatsuya Harada.  Image generation from small datasets via batch statistics adaptation.  In ICCV, 2019.
  * [32] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach.  Benchmark for compositional text-to- image synthesis.  In NeurIPS, 2022.
  * [33] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In ICCV, 2021.
  * [34] Justin NM Pinkney and Doron Adler.  Resolution dependent gan interpolation for controllable image synthesis between domains.  arXiv preprint arXiv:2010.05334, 2020.
  * [35] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In ICLR, 2023.
  * [36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning. PMLR, 2021.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning. PMLR, 2021.
  * [38] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning. PMLR, 2021.
  * [39] Esther Robb, Wen-Sheng Chu, Abhishek Kumar, and Jia-Bin Huang.  Few-shot adaptation of generative adversarial networks.  arXiv preprint arXiv:2010.11943, 2020.
  * [40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [41] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [43] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, and Marco Fumero.  Clip-forge: Towards zero-shot text-to-shape generation.  In CVPR, 2022.
  * [44] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  ICLR, 2021.
  * [45] Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris Metaxas, and Ahmed Elgammal.  Diffusion guided domain adaptation of image generators.  arXiv preprint arXiv:2212.04473, 2022.
  * [46] Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang.  Regularizing generative adversarial networks under limited data.  In CVPR, 2021.
  * [47] Rotem Tzaban, Ron Mokady, Rinon Gal, Amit Bermano, and Daniel Cohen-Or.  Stitch it in time: Gan-based facial editing of real videos.  In SIGGRAPH Asia 2022 Conference Papers, 2022.
  * [48] Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan.  Unitune: Text-driven image editing by fine tuning an image generation model on a single image.  arXiv preprint arXiv:2210.09477, 2022.
  * [49] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In CVPR, 2023.
  * [50] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In CVPR, 2018.
  * [51] Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu.  Transferring gans: generating images from limited data.  In ECCV, 2018.
  * [52] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [53] Yiran Xu, Badour AlBahar, and Jia-Bin Huang.  Temporally consistent semantic video editing.  In ECCV. Springer, 2022.
  * [54] Zipeng Xu, Tianwei Lin, Hao Tang, Fu Li, Dongliang He, Nicu Sebe, Radu Timofte, Luc Van Gool, and Errui Ding.  Predict, prevent, and evaluate: Disentangled text-driven image manipulation empowered by pre-trained vision-language model.  In CVPR, 2022.
  * [55] Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou.  Data-efficient instance generation from instance discrimination.  NeurIPS, 2021.
  * [56] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation, 2022.
  * [57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.