# Efficient Transfer Learning in Diffusion Models via Adversarial Noise

Xiyu Wang1, Baijiong Lin2, Daochang Liu1, Chang Xu1
1 School of Computer Science, Faculty of Engineering, The University of Sydney
2 The Hong Kong University of Science and Technology (Guangzhou)
{xiyuwang.usyd<EMAIL_ADDRESS>
{daochang.liu<EMAIL_ADDRESS>
###### Abstract

Diffusion Probabilistic Models (DPMs) have demonstrated substantial promise in
image generation tasks but heavily rely on the availability of large amounts
of training data. Previous works, like GANs, have tackled the limited data
problem by transferring pre-trained models learned with sufficient data.
However, those methods are hard to be utilized in DPMs since the distinct
differences between DPM-based and GAN-based methods, showing in the unique
iterative denoising process integral and the need for many timesteps with no-
targeted noise in DPMs. In this paper, we propose a novel DPMs-based transfer
learning method, TAN, to address the limited data problem. It includes two
strategies: similarity-guided training, which boosts transfer with a
classifier, and adversarial noise selection which adaptive chooses targeted
noise based on the input image. Extensive experiments in the context of few-
shot image generation tasks demonstrate that our method is not only efficient
but also excels in terms of image quality and diversity when compared to
existing GAN-based and DDPM-based methods.

##  1 Introduction

Generative models, such as GANs [7, 1, 13], VAEs [14, 22], and autoregressive
models [28, 4, 8], have made remarkable successes in various fields across
images [1, 21], text [2], and audio [6, 20] by utilizing vast amounts of
unlabeled data for training. Diffusion probabilistic models (DPMs) [25, 9,
17], which are designed to replicate data distributions by learning to invert
multi-step noise procedures, have recently experienced significant
advancements, enabling the generation of high-definition images with broad
diversity. Although DPMs have emerged as a potent tool for image generation
with remarkable results in terms of both quality and diversity, modern DPMs
heavily rely on extensive amounts of data to train the large-scale parameters
of their networks [3]. This dependency can lead to overfitting and a failure
to generate diverse, high-quality images with limited training data.
Unfortunately, gathering sufficient data is not always feasible in certain
situations.

Transfer learning can be an effective solution to this challenge, as it
applies knowledge from a pre-trained generative model trained on a large
dataset to a smaller one. The fundamental idea is to begin training with a
source model that has been pre-trained on a large dataset, and then adapt it
to a target domain with limited data. Several techniques have been proposed in
the past to adapt pre-trained GAN-based models [29, 11, 30, 15] from large-
scale source datasets to target datasets using a limited number of training
samples. Typically, methods for few-shot image generation either enhance the
training data artificially using data augmentation to prevent overfitting [32,
11], or directly evaluate the distance between the processed image and the
target image [19, 33].

Nevertheless, applying prior GAN-based techniques to DPMs is challenging due
to the differences in training processes between GAN-based and DPM-based
methods. GANs can quickly generate a final processed image from latent space,
while DPMs only predict less noisy images at each step and request large
timesteps to generate a high-quality final image. Such an iterative denoising
process poses two challenges when transferring diffusion models. The first
challenge is that the transfer direction needs to be estimated on noisy
images. The single-pass generation of GANs allows them to directly compare the
generated clean images with the target image [15, 19, 33], which is not easily
applicable to diffusion models. The current DPM-based few-shot method, DDPM
pairwise adaptation (DDPM-PA) [34], substitutes the high-quality real final
image with the predicted blurry final at the intermediate timestep to address
this issue. However, comparing the target image with the blurry image can be
problematic and inaccurate, as the predicted image may not accurately
represent the domain of generated images. It leads to the production of DDPM-
PA final images that are fuzzy and distorted. Moreover, even if the transfer
direction can be available, we still face a more fundamental second challenge
resulting from the noise mechanism in diffusion models. The diffusion and
denoising process utilize fully random Gaussian noise, which is independent of
the input image and makes no assumption of it. We observe that such non-
targeted noise imposes unbalanced effects on different images, leading to
divergent transferring pace in terms of training iteration needed. As
demonstrated in Figure 1, when one image (below) is just successfully
transferred from the source domain to the target domain, another image (above)
may have severely overfit and become too similar to the target image. Such
normally distributed noise may also necessitate an extensive number of
iterations to transfer, especially when the gradient direction is noisy due to
limited images.

Figure 1: Two sets of images generated from corresponding fixed noise inputs
at different stages of fine-tuning DDPM from FFHQ to 10-shot Sunglasses. The
perceptual distance (LPIPS [32]) with the training target image is shown on
each generated image. When the bottom image successfully transfers to the
target domain, the top image has already suffered from overfitting.

In this paper, to handle the challenge of transferring direction estimation
for diffusion models, we propose to leverage a similarity measurement to
estimate the gap between the source and the target, which circumvents the
necessity of comparing individual images. Building upon this, we introduce a
similarity-guided training approach to fine-tune the pre-trained source model
to the target domain. It employs a classifier to estimate the divergence
between the source and target domains, leveraging existing knowledge from the
source domain to aid in training the target domain. This method not only helps
in bridging the gap between the source and target domains for diffusion models
but also addresses the unstable gradient direction caused by limited target
data in the few-shot setting by implicitly comparing the sparse target data
with the abundant source data. More importantly, to tackle the challenge of
non-targeted noise in diffusion models, we propose a novel min-max training
process, i.e., adversarial noise selection, to dynamically choose the noise
according to the input image. The adversarial noise scheme enhances few-shot
transfer learning by minimizing the “worse-case” Gaussian noise which the pre-
trained model fails to denoise on the target dataset. This strategy also
significantly reduces the training iterations needed and largely improves the
efficiency of the transfer learning for diffusion models. Our adversarial
strategy with similarity measurement excels in few-shot image generation
tasks, speeding up training, achieving quicker convergence, and creating
images fitting the target style while resembling source images. Our
Experiments on few-shot image generation tasks demonstrate our method
surpasses existing GAN-based and DDPM-based techniques, offering superior
quality and diversity.

##  2 Related Work

###  2.1 Diffusion Probabilistic Models

Ho et al. [9] has been leveraged as an effective generative model that
circumvents the adversarial training inherent in GANs [7]. DDPMs, by enabling
the diffusion reverse process, are capable of reconstructing images. However,
due to their extensive iterative time steps, DDPMs are subject to the
challenge of high computational time. DDIM [26] addresses this issue by
"implicating" the model, which allows it to function with far fewer iterations
and dramatically reduces the inference time compared to DDPM. Conversely, a
fresh approach to the diffusion model is the score-based model via SDE,
wherein the diffusion and the denoising processes are both modeled by SDEs.
Song and Ermon [27] initially proposed the generation of samples from latent
noise via the Dynamic Langevin Sampling method. For fast high-quality, high-
resolution image generation, Latent Diffusion Models (LDMs) [23] propose an
advanced machine learning methodology by gradually transforming a random noise
into the target image through a diffusion process that uses latent space
representations.

###  2.2 Few-shot Image Generation

Existing practices predominantly adopt an adaptation pipeline where a
foundational model is pre-trained on a large source domain, and then adjusted
to a smaller target domain. In contrast, few-shot image generation strives to
envision new and diverse examples while circumventing overfitting to the
limited training images. CDC [19] introduces a cross-domain consistency loss
and patch-level discrimination to forge a connection between the source and
target domains. DCL [33] uses contrastive learning to distance the generated
samples from actual images and maximize the similarity between corresponding
image pairs in source and target domains. The DDPM-PA [34] adopts a similar
approach to CDC for adapting models pre-trained on extensive source domains to
target domains. GAN-based methods, like CDC and DCL, require the final
generated image during training. In contrast, DPMs’ training process aims to
predict the next stage of noised images and can only yield a blurry predicted
image during the training stage.

##  3 Preliminary

Gaussian diffusion models are used to approximate the data distribution
$x_{0}\sim q(x_{0})$ by $p_{\theta}(x_{0})$. The distribution
$p_{\theta}(x_{0})$ is modeled in the form of latent variable models.
According to [9], the diffusion process from a data distribution to a Gaussian
distribution with variance $\beta_{t}\in(0,1)$ for timestep $t$ can be
expressed as:

$\displaystyle q(x_{t}|x_{0})$
$\displaystyle=\mathcal{N}(x_{t};\bar{\alpha}_{t}x_{0},(1-\bar{\alpha}_{t})\mathbf{I}),$
$\displaystyle x_{t}$
$\displaystyle=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon\
,$

where $\alpha_{t}:=1-\beta_{t}$,
$\bar{\alpha}_{t}:=\prod_{i=0}^{t}\left(1-\beta_{i}\right)$ and
$\epsilon\sim\mathcal{N}(\mathbf{0,I})$. Ho et al. [9] trains a U-Net [24]
model parameterized by $\theta$ to fit the data distribution $q(x_{0})$ by
maximizing the variational lower-bound. The DDPM training loss with model
$\epsilon_{\theta}(x_{t},t)$ can be expressed as:

$\mathcal{L}_{\texttt{sample}}(\theta):=\mathbb{E}_{t,x_{0},\epsilon}\left\|\epsilon-\epsilon_{\theta}(x_{t},t)\right\|^{2}.$
(1)

Based on [26], the reverse process of DDPM and DDIM at timestep $t$ can be
expressed as:

$x_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\underbrace{\left(\frac{x_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}(x_{t},t)}{\bar{\alpha}_{t}}\right)}_{\mathrm{predicted\
x_{0}}}+\underbrace{\sqrt{1-\bar{\alpha}_{t-1}-\sigma_{t}^{2}}\cdot\epsilon_{\theta}(x_{t},t)}_{\mathrm{direction\
pointing\ to\ x_{t}}}+\underbrace{\sigma_{t}\epsilon_{t}}_{\mathrm{random\
noise}},$

where
$\sigma_{t}=\eta\sqrt{(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_{t})}\sqrt{1-\bar{\alpha}_{t}/\bar{\alpha}_{t-1}}$
and $\eta=0$ [26] or $\eta=1$ [9] or
$\eta=\sqrt{(1-\bar{\alpha}_{t})/(1-\bar{\alpha}_{t-1})}$ [9]. Enhance,
Dhariwal and Nichol [5] propose the conditional reverse noise process as:

$\displaystyle
p_{\theta,\phi}(x_{t-1}|x_{t},y)\approx\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t)+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y|x_{t}),\sigma_{t}^{2}\mathbf{I}),$ (2) $\displaystyle\textrm{where
}~{}~{}\mu_{\theta}(x_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right),$
(3)

and $\gamma$ is a hyperparameter for conditional control. For the sake of
clarity in distinguishing domains, this paper uses $\mathcal{S}$ and
$\mathcal{T}$ to represent the source and target domain, respectively.

##  4 Method

###  4.1 Transfer Learning in Diffusion Models via Adversarial Noise

In this subsection, we introduce transfer learning in diffusion models via
Adversarial Noise, TAN, with similarity-guided training and adversarial noise
selection for stronger transfer ability. Given the challenges associated with
generating the final image, we suggest using similarity to measure the gap
between the source and target domains using a noised image $x_{t}$ at timestep
$t$. Drawing inspiration from [5, 16], we express the domain difference
between the source and target in terms of the divergence in similarity
measures.

Initially, we assume a model that can predict noise with both source and
target domains, denoted as $\theta_{(\mathcal{S},\mathcal{T})}$. As Equation
(2), the reverse process for the source and target images can be written as:

$p_{\theta_{(\mathcal{S},\mathcal{T})},\phi}(x_{t-1}|x_{t},y=Y)\approx\mathcal{N}(x_{t-1};\mu_{\theta_{(\mathcal{S},\mathcal{T})}}+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=Y|x_{t}),\sigma_{t}^{2}\mathbf{I})\ ,$ (4)

where $Y$ is $\mathcal{S}$ or $\mathcal{T}$ for source or target domain image
generation, respectively. We can consider
$\mu(x_{t})+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{S}|x_{t})$ as the source model $\theta_{\mathcal{S}}$,
which only synthesize image on the source domain respectively. For brevity, we
denote
$p_{\theta_{\mathcal{S}},\phi}(x_{t-1}^{\mathcal{S}}|x_{t})=p_{\theta_{(\mathcal{S},\mathcal{T})},\phi}(x_{t-1}|x_{t},y=\mathcal{S})$.
We similar define $p_{\theta_{\mathcal{T}},\phi}(x_{t-1}^{\mathcal{T}}|x_{t})$
as above by replace $\mathcal{S}$ with $\mathcal{T}$. Therefore, the KL-
divergence between the output of source model $\theta_{\mathcal{S}}$ and the
target $\theta_{\mathcal{T}}$ with the same input $x_{t}$ at timestep $t$, is
defined as follows:

$\displaystyle\texttt{D}_{\textrm{KL}}\left(p_{\theta_{\mathcal{S}},\phi}(x_{t-1}^{\mathcal{S}}|x_{t}),p_{\theta_{\mathcal{T}},\phi}(x_{t-1}^{\mathcal{T}}|x_{t})\right)$
$\displaystyle=~{}$
$\displaystyle\mathbb{E}_{t,x_{0},\epsilon}\left[\left\|\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{S}|x_{t})-\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})\right\|^{2}\right],$ (5)

where $p_{\phi}$ is a classifier to distinguish $x_{t}$. The detailed
derivation is in Appendix. We consider the $\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{S}|x_{t})$ and $\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})$ as the similarity measures of the given $x_{t}$
in the source and target domains, respectively. Since $x_{t}$ pertains to the
target image during fine-tuning, the term
$p_{\phi}(y=\mathcal{S}|x_{t}^{\mathcal{T}})$ is near zero and its gradient is
large and chaotic. Therefore, we disregard the first term and utilize only
$p_{\phi}(y=\mathcal{T}|x_{t}^{\mathcal{T}})$ to guide the training process.
Specifically, we employ a fixed pre-trained binary classifier that
differentiates between source and target images at time step $t$ to boost the
training process. Similarly with the vanilla training loss in DPMs [9], i.e.,
Equation (1), we use the KL-divergence between the output of current model
$\theta$ and target model $\theta_{\mathcal{T}}$ at time step $t$ as:

$\min_{\theta}\mathbb{E}_{t,x_{0},\epsilon}\left[\left\|\epsilon_{t}-\epsilon_{\theta}(x_{t},t)-\hat{\sigma}_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})\right\|^{2}\right]\ ,$ (6)

where $\epsilon_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$,
$\epsilon_{\theta}$ is the pre-trained neural network on source domain,
$\gamma$ is a hyper-parameter to control the similarity guidance,
$\hat{\sigma}_{t}=\left(1-\bar{\alpha}_{t-1}\right)\sqrt{\frac{\alpha_{t}}{1-\bar{\alpha}_{t}}}$,
and $p_{\phi}$ is the binary classifier differentiating between source and
target images. Equation (6) is defended as similarity-guided DPMs train loss.
The full proof is provided in the Appendix. We leverage the pre-trained
classifier to indirectly compare the noised image $x_{t}$ with both domain
images, subtly expressing the gap between the currently generated image and
the target image. By minimizing the output of the neural network with
corrected noise, we bridge the gap in the diffusion model and bolster transfer
learning. Furthermore, similarity guidance enhances few-shot transfer learning
by avoiding misdirection towards the target image, as $\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})$ acts as an indirect indicator, rather than
straightly relying on the original image.

Despite potentially determining the transfer direction, we still encounter a
fundamental second challenge originating from the noise mechanism in diffusion
models. As mentioned, the model needs to be trained to accommodate the
quantity of noise $\epsilon_{t}$ over many iterations. However, increasing
iterations with limited images may lead to overfitting of the training
samples, thereby reducing the diversity of the generated samples. On the other
hand, training with too few iterations might only successfully transform a
fraction of the generated images into the target domain as Figure 1.

To counter these issues, we propose an adaptive noise selection method. This
approach utilizes a min-max training process to reduce the actual training
iterations required and ensure the generated images closely resemble the
target images. After the model has been trained on a large dataset, it
exhibits a strong noise reduction capability for source datasets. This implies
it only needs to minimize specific types of Gaussian noise with which the
trained model struggles or fails to denoise with the target domain sample. The
first step in this process is to identify the maximum Gaussian noise with the
current model, and then specifically minimize the model using this noise.
Based on Equation (6), this can be mathematically formulated as follows:

$\min_{\theta}\max_{\epsilon}\mathbb{E}_{t,x_{0}}\left[\left\|\epsilon-\epsilon_{\theta}(x_{t},t)-\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})\right\|^{2}\right]\ .$ (7)

Although finding the exact maximum noise is challenging as Equation (7), the
projected gradient descent (PGD) strategy can be used to solve the inner
maximization problem instead. Specifically, the inner maximization of Gaussian
noise can be interpreted as finding the “worse-case” noise corresponding to
the current neural network. Practically, the similarity-guided term is
disregarded, as this term is hard to compute differential and is almost
unchanged in the process. We utilize the multi-step variant of PGD with
gradient ascent, as expressed below:

$\epsilon^{j+1}=\textrm{Norm}\left(\epsilon^{j}+\omega\nabla_{\epsilon^{j}}\left\|\epsilon^{j}-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon^{j},t)\right\|^{2}\right),~{}~{}j=0,\cdots,J-1,$
(8)

where $\omega$ is a hyperparameter that represents the “learning rate” of the
negative loss function, and Norm is a normalization function that
approximately ensures the mean and standard deviation of $\epsilon^{j+1}$ is
$\mathbf{0}$ and $\mathbf{I}$ respectively. The initial value, $\epsilon_{0}$,
is sampled from the Gaussian distribution as
$\epsilon_{0}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. We use this method to
identify this worse-case noise and minimizing the “worse-case” Gaussian noise
is akin to minimizing all Gaussian noises that are “better” than it. By
adaptively choosing this specific noise, we can more accurately correct the
gradient to enhance training with limited data, effectively addressing the
underfitting problem during a limited number of iterations.

###  4.2 Optimization

For time and GPU memory saving, we implement an additional adaptor module,
$\psi^{l}$, [18] to learn the shift gap as Equation (5) based on $x_{t}$ in
practice. During the training, we keep the parameters of $\theta^{l}$ constant
and update the additional adaptor layer parameters $\psi^{l}$. The overall
loss function can be expressed as follows,

$\displaystyle
L(\psi)\equiv\mathbb{E}_{t,x_{0}}\left\|\epsilon^{\star}-\epsilon_{\theta,\psi}(x_{t}^{\star},t)-\sigma_{t}^{2}\gamma\nabla_{x_{t}^{\star}}\log
p_{\phi}(y=\mathcal{T}|x_{t}^{\star})\right\|^{2}$ (9)
$\displaystyle\textrm{s.t.
}\epsilon^{\star}=\arg\max_{\epsilon}\left\|\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t)\right\|^{2}\
\ ,\ \epsilon^{\star}_{\text{mean}}=\mathbf{0}\text{ and
}\epsilon^{\star}_{\text{std}}=\mathbf{I}\ ,$ (10)

where $\epsilon^{\star}$ is the “worse-case” noise, the
$x_{t}^{\star}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon^{\star}$
is the corresponding noised image at the timestep $t$ and $\psi$ is certain
extra parameter beyond pre-trained model. We link the pre-trained U-Net model
with the adaptor layer [10] as
$x_{t}^{l}={\theta^{l}(x_{t}^{l-1})}+{\psi^{l}(x_{t}^{l-1})}$, where
$x_{t}^{l-1}$ and $x_{t}^{l}$ represents the $l$-th layer of the input and
output, and $\theta^{l}$ and $\psi^{l}$ denote the $l$-th layer of the pre-
trained U-Net and the additional adaptor layer, respectively.

Algorithm 1 Training DPMs with TAN

1:binary classifier $p_{\phi}$, pre-trained DPMs $\epsilon_{\theta}$, learning
rate $\eta$

2:repeat

3: $x_{0}\sim q(x_{0})$;

4: $t\sim\textrm{Uniform}(\\{1,\cdots,T\\})$;

5: $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$;

6: for $j=0,\cdots,J-1$ do

7: Update $\epsilon^{j}$ via Eq. (8);

8: end for

9: Compute $L(\psi)$ with $\epsilon^{\star}=\epsilon^{J}$ via Eq. (9);

10: Update the adaptor model parameter: $\psi=\psi-\eta\nabla_{\psi}L(\psi)$;

11:until converged.

The full training procedure of our method, named DPMs-TAN, is outlined in
Algorithm 1. Initially, as in the traditional DDPM training process, we select
samples from target datasets and randomly choose a timestep $t$ and standard
Gaussian noise for each sample. we employ limited extra adaptor module
parameters with the pre-train model. Subsequently, we identify the adaptive
inner max as represented in Equation (8) with the current neural network.
Based on these noises, we compute the similarity-guided DDPM loss as Equation
(6), which bridges the discrepancy between the pre-trained model and the
scarce training samples. Lastly, we execute backpropagation to only update the
adaptor module parameters.

##  5 Experiment

To demonstrate the effectiveness of our approach, we perform a series of few-
shot image generation experiments using a limited set of just 10 training
images with the same setting as DDPM-PA [34]. We compare our method against
state-of-the-art GAN-based and DDPM-based techniques, assessing the quality
and diversity of the generated images through both qualitative and
quantitative evaluations. This comprehensive comparison provided strong
evidence of the superiority of our proposed method in the context of few-shot
image generation tasks.

###  5.1 Visualization on Toy Data

To conduct a quantitative analysis, we trained a diffusion model to generate
2-dimensional toy data with two Gaussian noise distributions. The means of the
Gaussian noise distributions for the source and target are $(1,1)$ and
$(-1,-1)$, and their variances are denoted by $\mathbf{I}$. We train a simple
neural network with source domain samples and then transfer this pre-trained
model to target samples.

Figure 2 (a) illustrates the output layer gradient direction of four different
settings in the first iteration, with the same noise and timestep $t$. The red
line, computed with ten thousand different samples, is considered a reliable
reference direction (close to 45 degrees southwest). For 10-shot samples, we
repeat them a thousand times into one batch to provide a unified comparison
with ten thousand different samples. The dark blue line and the sienna
represent the gradient computed with the traditional DDPM as the baseline and
similarity-guided training in a 10-shot sample, respectively. The orange line
represents our method, DDPM-TAN, in a 10-shot sample. The gradient of our
method is closer to the reliable reference direction, demonstrating that our
approach can effectively correct the issue of the noisy gradient. The red
points in the background symbolize "worse-case" noise, which is generated
through adversarial noise selection. The graphic shows how the noise
distribution transitions from a circle (representing a normal Gaussian
distribution) to an ellipse. The principal axis of this ellipse is oriented
along the gradient of the model parameters. This illustrates the noise
distribution shift under our adversarial noise selection approach, which
effectively fine-tunes the model by actively targeting the “worst-case” noise
that intensifies the adaptation task.

Figure 2: This Figure visualizes gradient changes and heat maps: Figure (a)
shows gradient directions with various settings—the cyan line for the gradient
of 10,000 samples in one step, dark blue for ten samples in one step as
baseline method (trained with traditional DDPM), the sienna for our
similarity-guided training, and the orange for our method DDPM-TAN, while red
points at the background are "worse"-case noises by adversarial noise
selection; Figure (b) and (c) depict heat-maps of the baseline and our method,
with cyan and gold lines representing the generation sampling process value
with the original DDPM and our method, respectively.

Figures 2 (b) and (c) present heatmaps of the baseline and our method in only
one dimension, respectively. The cyan and gold lines denote the values of the
generation sampling process using the original DDPM and our method. The heat-
maps in the background illustrate the distribution of values for 20,000
samples generated by the original DDPM (baseline) and our method. The lighter
the color in the background, the greater the number of samples present. There
is a significantly brighter central highlight in (c) compared to (b),
demonstrating that our method can learn the distribution more quickly than the
baseline method. The gold and cyan lines in the two figures are approximately
parallel, providing further evidence that our method can learn the gap, as per
Equation (5), more rapidly.

###  5.2 Experimental Setup

#### Datasets.

Following [19], we use FFHQ [12] and LSUN Church [31] as source datasets. For
the target datasets, we employe 10-shot Sketches, Babies, Sunglasses, and face
paintings by Amedeo Modigliani and Raphael Peale, which correspond to the
source domain FFHQ. Additionally, we utilize 10-shot Haunted Houses and
Landscape drawings as target datasets corresponding to the LSUN Church source
domain.

#### Configurations.

We evaluate our method not only on the DDPM framework but also on the LDMs.
For this, we employ a pre-trained DDPM similar to DDPM-PA and use the pre-
trained LDMs as provided in [23]. We restrict our fine-tuning to the shift
module of the U-Net, maintaining the pre-trained DPMs and autoencoders in LDMs
as they are. For the $l$-th shift adaptor layer $\psi$, it can be expressed
as: $\psi^{l}(x^{l-1})=f(x^{l-1}W_{down})W_{up}$ [10]. We project the input
downward using $W_{down}$, transforming it from its original dimension
$\mathbb{R}^{w\times h\times r}$ to a lower-dimensional space with a
bottleneck dimension $\mathbb{R}^{\frac{w}{c}\times\frac{h}{c}\times d}$.
Following this, we apply a non-linear activation function $f(\cdot)$ and
execute an upward projection with $W_{up}$. For the DDPMs, we set the
parameters $c=4$ and $d=8$, whereas we set $c=2$ and $d=8$ for the LDMs. To
ensure the adapter layer outputs are initialized to zero, we set all the extra
layer parameters to zero. For similarity-guided training, we establish
$\gamma=5$. In the case of adversarial noise selection, we assign $J=10$ and
$\omega=0.02$ for most transfer learning tasks. We employ a learning rate of
$5\times 10^{-5}$ for the DDPMs or $1\times 10^{-5}$ LDMs for approximately
300 iterations with a batch size of 40 on $\times 8$ NVIDIA A100.

Figure 3: The 10-shot image generation samples on LSUN Church $\to$ Landscape
drawings (top) and FFHQ $\to$ Raphael’s paintings (bottom). When compared with
other GAN-based and DDPM-based methods, our method, TAN, yields high-quality
results that more closely resemble images of the target domain style, with
less blurring.

#### Measurements.

In our evaluation of generation diversity, we utilize Intra-LPIPS and FID as
described in CDC [19]. For Intra-LPIPS, we generate 1,000 images, which each
of them wiil be assigned to the training sample with the smallest LPIPS
distance. The Intra-LPIPS measurement is obtained by averaging the pairwise
LPIPS distances within the same cluster and then averaging these results
across all clusters. FID is a widely use metric for assessing the generation
quality of generative models by calculating the distribution distances between
generated samples and datasets. However, FID may become unstable and
unreliable when applied to datasets with few samples, such as the 10-shot
datasets used in this paper. Following the DDPM-PA approach, we provide FID
evaluations using larger target datasets, such as Sunglasses and Babies, which
consist of 2,500 and 2,700 images, respectively.

#### Baselines.

To adapt pre-trained models to target domains using a limited number of
samples, we compare our work with several GAN-based and DDPMs baselines that
share similar objectives. These include TGAN [29], TGAN+ADA [11], EWC [15],
CDC [19], DCL [33], and DDPM-PA [34]. All these methods are implemented on the
same StyleGAN2 [12] codebase.

Table 1: The Intra-LPIPS ($\uparrow$) results for both DDPM-based strategies
and GAN-based baselines are presented for 10-shot image generation tasks.
These tasks involve adapting from the source domains of FFHQ and LSUN Church.
The “Parameter Rate” column provides information regarding the proportion of
parameters fine-tuned in comparison to the pre-trained model’s parameters. The
best results are marked as bold.

Methods | Parameter | FFHQ $\to$ | FFHQ $\to$ | FFHQ $\to$ | LSUN Church $\to$ | LSUN Church $\to$
---|---|---|---|---|---|---
Rate | Babies | Sunglasses | Raphael’s paintings | Haunted houses | Landscape drawings
TGAN | 100% | 0.510$\pm$0.026 | 0.550$\pm$0.021 | 0.533$\pm$0.023 | 0.585$\pm$0.007 | 0.601$\pm$0.030
TGAN+ADA | 100% | 0.546$\pm$0.033 | 0.571$\pm$0.034 | 0.546$\pm$0.037 | 0.615$\pm$0.018 | 0.643$\pm$0.060
EWC | 100% | 0.560$\pm$0.019 | 0.550$\pm$0.014 | 0.541$\pm$0.023 | 0.579$\pm$0.035 | 0.596$\pm$0.052
CDC | 100% | 0.583$\pm$0.014 | 0.581$\pm$0.011 | 0.564$\pm$0.010 | 0.620$\pm$0.029 | 0.674$\pm$0.024
DCL | 100% | 0.579$\pm$0.018 | 0.574$\pm$0.007 | 0.558$\pm$0.033 | 0.616$\pm$0.043 | 0.626$\pm$0.021
DDPM-PA | 100% | 0.599$\pm$0.024 | 0.604$\pm$0.014 | 0.581$\pm$0.041 | 0.628$\pm$0.029 | 0.706$\pm$0.030
DDPM-TAN (Ours) | 1.3% | 0.592$\pm$0.016 | 0.613$\pm$0.023 |  0.621$\pm$0.068 | 0.648$\pm$0.010 | 0.723$\pm$0.020
LMD-TAN (Ours) | 1.6% |  0.601$\pm$0.018 |  0.613$\pm$0.011 | 0.592$\pm$0.048 |  0.653$\pm$0.010 |  0.738$\pm$0.026

###  5.3 Overall Performance

#### Qualitative Evaluation.

Figure 3 presents samples from GAN-based and DDPM-based methods for 10-shot
LSUN Church $\to$ Landscape drawings (top) and FFHQ $\to$ Raphael’s paintings
(bottom). The samples generated by GAN-based baselines contain unnatural blurs
and artifacts. This illustrates the effectiveness of our approach in handling
complex transformations while maintaining the integrity of the original image
features. Whereas the current DDPM-based method, DDPM-PA (third row), seems to
underfit the target domain images, resulting in a significant difference in
color and style between the generated images and the target images.

Table 2: FID ($\downarrow$) results of each method on 10-shot FFHQ → Babies
and Sunglasses. The best results are marked as bold.

Methods | TGAN | ADA | EWC | CDC | DCL | PA | ADMT
---|---|---|---|---|---|---|---
Babies | 104.79 | 102.58 | 87.41 | 74.39 | 52.56 | 48.92 | 46.70
Sunglasses | 55.61 | 53.64 | 59.73 | 42.13 | 38.01 | 34.75 | 20.06

Our method preserves many source domain shapes and outlines while learning
more about the target style. As demonstrated in Figure 1, our method, TAN,
maintains more details such as buildings (above) and human faces (below) in
the generated images. Moreover, TAN-generated images exhibit a color style
closer to the target domain, especially when compared with DDPM-PA. Compared
to other methods, our approach (based on both DDPMs and LDMs) produces more
diverse and realistic samples that contain richer details than existing
techniques.

#### Quantitative Evaluation.

In Table 1, we display the Intra-LPIPS results for DPMs-TAN under various
10-shot adaptation conditions. DDPM-TAN yields a considerable improvement in
Intra-LPIPS across most tasks when compared with other GAN-based and DDPMs-
based methods. Furthermore, LMD-TAN excels beyond state-of-the-art GAN-based
approaches, demonstrating its potent capability to preserve diversity in few-
shot image generation. The FID results are presented in Table 2, where TAN
also demonstrates remarkable advancements compared to other GAN-based or DPMs-
based methods, especially in FFHQ $\rightarrow$ 10-shot Sunglasses with 20.06
FID. We provide more results for other adaptation scenarios in the Appendix.

Our method can transfer the model from the source to the target domain not
only effectively but also efficiently. Compared to other methods that require
around 5,000 iterations, our approach only necessitates approximately 300
iterations with limited parameter fine-tuning. We use the direct fine-tuning
on DDPM as our baseline, utilizing a batch size of 5 on one GPU. The time cost
of the baseline with 5,000 iterations (same with DDPM-PA) is 7.5 GPU hours,
while our model (DPMs-TAN) with only 300 iterations takes just 3 GPU hours.
Regarding GPU memory consumption, the baseline model requires 20 GB, while
DPMs-TAN only uses 9 GB. Consequently, our method allows for a significant
reduction in both time and GPU usage.

Figure 4: This figure shows our ablation study with all models trained for 300
iterations on a 10-shot sunglasses dataset measured with FID ($\downarrow$):
the first line - baseline (direct fine-tuning model), second line - Adaptor
(fine-tuning only few extra parameters), third line - DPMs-TAN w/o A (only
using similarity-guided training), and final line - DPMs-TAN (our method).

###  5.4 Ablation Analysis

Figure 4 presents an ablation study, with all images synthesized from the same
noise. When compared to directly fine-tuning the entire model (1st row), only
fine-tuning the adaptor layer (2nd row) can achieve competitive FID results
(38.65 vs. 41.88). The DPMs-TAN without adversarial noise selection (DPMs-TAN
w/o A) and all DPMs-TAN (3rd and 4th row) are trained with an extra adaptor
layer to save time and GPU memory, and our analysis focuses on the last three
rows.

The first two columns demonstrate that all methods can successfully transfer
the model to sunglasses, with the TAN containing richer high-frequency details
about sunglasses and background items. The 3rd and 4th columns show that the
similarity-guided method (3rd row) can produce images of people wearing
sunglasses, while the traditional method (2nd row) does not achieve this. The
last two columns highlight the effectiveness of the adaptive noise selection
method in TAN. The step-by-step transformation showcased in the 5th column
provides a clear demonstration of how our method transfers the source face
through an intermediate phase, in which the face is adorned with glasses, to a
final result where the face is wearing sunglasses. This vividly illustrates
the effectiveness of our proposed strategies in progressively boosting the
transfer process. The Frechet Inception Distance (FID) scores further
illustrate the effectiveness of our proposed strategies; it decreases from
41.88 (with direct adaptation) to 26.41 (with similarity-guided training) and
then to 20.66 (with DPMs-TAN), indicating a progressive improvement in the
quality of generated images.

##  6 Conclusion

In conclusion, the application of previous GAN-based techniques to DPMs
encounters substantial challenges due to the distinct training processes of
these models. To overcome this, we introduce TAN to train the DPMs with a
novel adversarial noise selection and the similarity-guided strategy that
improves the efficiency of the transfer learning process. Our proposed method
accelerates training, achieves quicker convergence, and produces images that
fit the target style while resembling the source images. Experimental results
on few-shot image generation tasks demonstrate that our method surpasses
existing state-of-the-art GAN-based and DDPM-based methods, delivering
superior image quality and diversity.

## References

## References

  * Brock et al. [2018] A. Brock, J. Donahue, and K. Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  In _Neural Information Processing Systems_ , 2020.
  * Cao et al. [2022] H. Cao, C. Tan, Z. Gao, G. Chen, P.-A. Heng, and S. Z. Li.  A survey on generative diffusion model.  _arXiv preprint arXiv:2209.02646_ , 2022.
  * Chen et al. [2018] X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel.  Pixelsnail: An improved autoregressive generative model.  In _International Conference on Machine Learning_ , 2018.
  * Dhariwal and Nichol [2021] P. Dhariwal and A. Nichol.  Diffusion models beat gans on image synthesis.  In _Neural Information Processing Systems_ , 2021.
  * Dhariwal et al. [2020] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever.  Jukebox: A generative model for music.  _arXiv preprint arXiv:2005.00341_ , 2020.
  * Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Grill et al. [2020] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al.  Bootstrap your own latent a new approach to self-supervised learning.  In _Neural Information Processing Systems_ , 2020.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  In _Neural Information Processing Systems_ , 2020.
  * Houlsby et al. [2019] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , 2019.
  * Karras et al. [2020a] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila.  Training generative adversarial networks with limited data.  In _Neural Information Processing Systems_ , 2020a.
  * Karras et al. [2020b] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and improving the image quality of stylegan.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020b.
  * Khan et al. [2022] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah.  Transformers in vision: A survey.  _ACM Computing Surveys_ , 54(10s):1–41, 2022\.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li et al. [2020] Y. Li, R. Zhang, J. Lu, and E. Shechtman.  Few-shot image generation with elastic weight consolidation.  _arXiv preprint arXiv:2012.02780_ , 2020.
  * Liu et al. [2023] X. Liu, D. H. Park, S. Azadi, G. Zhang, A. Chopikyan, Y. Hu, H. Shi, A. Rohrbach, and T. Darrell.  More control for free! image synthesis with semantic diffusion guidance.  In _IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023.
  * Nichol and Dhariwal [2021] A. Q. Nichol and P. Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , 2021.
  * Noguchi and Harada [2019] A. Noguchi and T. Harada.  Image generation from small datasets via batch statistics adaptation.  In _IEEE/CVF International Conference on Computer Vision_ , 2019.
  * Ojha et al. [2021] U. Ojha, Y. Li, J. Lu, A. A. Efros, Y. J. Lee, E. Shechtman, and R. Zhang.  Few-shot image generation via cross-domain correspondence.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Oord et al. [2016] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Razavi et al. [2019] A. Razavi, A. Van den Oord, and O. Vinyals.  Generating diverse high-fidelity images with VQ-VAE-2.  In _Neural Information Processing Systems_ , 2019.
  * Rezende et al. [2014] D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _International Conference on Machine Learning_ , 2014.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention_ , 2015\.
  * Sohl-Dickstein et al. [2015] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , 2015.
  * Song et al. [2020] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Song and Ermon [2019] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  In _Neural Information Processing Systems_ , 2019.
  * Van den Oord et al. [2016] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al.  Conditional image generation with pixelcnn decoders.  In _Neural Information Processing Systems_ , 2016.
  * Wang et al. [2018] Y. Wang, C. Wu, L. Herranz, J. Van de Weijer, A. Gonzalez-Garcia, and B. Raducanu.  Transferring gans: generating images from limited data.  In _European Conference on Computer Vision_ , 2018.
  * Wang et al. [2020] Y. Wang, A. Gonzalez-Garcia, D. Berga, L. Herranz, F. S. Khan, and J. v. d. Weijer.  Minegan: effective knowledge transfer from gans to target domains with few images.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Yu et al. [2015] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zhang et al. [2018] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018\.
  * Zhao et al. [2022] Y. Zhao, H. Ding, H. Huang, and N.-M. Cheung.  A closer look at few-shot image generation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Zhu et al. [2022] J. Zhu, H. Ma, J. Chen, and J. Yuan.  Few-shot image generation with diffusion models.  _arXiv preprint arXiv:2211.03264_ , 2022.

## Appendix for “Efficient Transfer Learning in Diffusion Models via
Adversarial Noise”

Table 3: The Intra-LPIPS ($\uparrow$) results for both DDPM-based strategies
and GAN-based baselines are presented for 10-shot image generation tasks. The
best results are marked as bold.

Methods | FFHQ $\to$ | FFHQ $\to$
---|---|---
Sketches | Amedeo’s paintings
TGAN | 0.394$\pm$0.023 | 0.548$\pm$0.026
TGAN+ADA | 0.427$\pm$0.022 | 0.560$\pm$0.019
EWC | 0.430$\pm$0.018 | 0.594$\pm$0.028
CDC | 0.454$\pm$0.017 | 0.620$\pm$0.029
DCL | 0.461$\pm$0.021 | 0.616$\pm$0.043
DDPM-PA | 0.495$\pm$0.024 | 0.626$\pm$0.022
DDPM-TAN (Ours) | 0.544$\pm$0.025 | 0.620$\pm$0.021

##  Appendix A Experiment

In this section, we present Intra-LPIPS evaluation results on two additional
datasets for 10-shot transformations, FFHQ $\rightarrow$ Sketches and FFHQ
$\rightarrow$ Amedeo’s paintings. We also provide supplementary information
detailing our selection process for the hyper-parameters.

###  A.1 Additional Results

#### Quantitative Evaluation.

As depicted in Table 3, our proposed DPMs-TAN method demonstrates superior
performance over contemporary GAN-based and DPMs-based methods in terms of
generation diversity for the given adaptation scenarios in FFHQ $\to$ Sketches
and FFHQ $\to$ Amedeo’s paintings. Especially, we achieve 0.544$\pm$0.025 for
the FFHQ $\to$ Sketches, far more better than other methods.

#### Qualitative Evaluation.

In Figure 5, we provide additional results for GAN-based and DDPM-based
methods for the 10-shot FFHQ $\rightarrow$ Sunglasses and Babies task. When
compared to the GAN-based method (shown in the 2nd and 3rd rows), our approach
(shown in the 5th and 6th rows) generates images of faces wearing sunglasses,
displaying a wide variety of detailed hairstyles and facial features.
Moreover, DPMs-TAN produces samples with more vivid and realistic reflections
in the sunglasses. Notably, our method also manages to generate more realistic
backgrounds.

###  A.2 Hyper-parameters

In this subsection, we delve into our process for selecting key
hyperparameters, including $\gamma$ for the similarity-guided training,
$\omega$ for the adversarial noise selection, and the count of training
iterations. All experiments are conducted using a pre-trained LDM, and for
evaluation purposes, we generate 1000 images for the Intra-LPIPS evaluation
and 10000 images for the FID.

#### Similarity-guided Training Scale $\gamma$.

Table 4: This shows the change in FID ($\downarrow$) and Intra-LPIPS
($\uparrow$)kan results for FFHQ $\to$ Sunglasses as the $\gamma$ value
increases.

$\gamma$ | FID ($\downarrow$) | Intra-LPIPS ($\uparrow$)
---|---|---
1 | 20.75 | 0.641 $\pm$ 0.014
3 | 18.86 | 0.627 $\pm$ 0.013
5 | 18.13 |  0.613 $\pm$ 0.011
7 | 24.12 | 0.603 $\pm$ 0.017
9 | 29.48 | 0.592 $\pm$ 0.017

Table 4 shows the changes in FID ($\downarrow$) and Intra-LPIPS ($\uparrow$)
scores for FFHQ $\to$ Sunglasses as the $\gamma$ (in Equation (7)) increases.
Initially, the FID score decrease, as the generated images gradually become
closer to the target domain. At $\gamma=5$, the FID reaches its lowest value
of 18.13. Beyond this point, the FID score increases as the generated images
become too similar to the target images or become random noise as failed case,
leading to lower diversity and fidelity. The Intra-LPIPS score consistently
decreases with gamma increasing, which further supports the idea that larger
$\gamma$ values lead to overfitting with the target image. Therefore, we
select $\gamma=5$ as a trade-off.

#### Adversarial Noise Selection Scale $\omega$.

As shown in Table 5, the FID ($\downarrow$) and Intra-LPIPS ($\uparrow$)
scores for FFHQ $\to$ Sunglasses vary with an increase in the omega ($\omega$)
value (from Equation (8)). Initially, the FID score decreases as the generated
images gradually grow closer to the target image. When $\omega=0.02$, the FID
reaches its lowest value of 18.13. Beyond this point, the FID score increases
because the synthesized images become too similar to the target image, which
lowers diversity. The Intra-LPIPS score consistently decreases as $\omega$
increases, further supporting that larger $\omega$ values lead to overfitting
with the target image. We also note that the results are relatively stable
when $\omega$ is between $0.1$ and $0.3$. As such, we choose $\omega=0.02$ as
a balance between fidelity and diversity.

Table 5: This shows the change in FID (lower is better) and Intra-LPIPS
(higher is better) results for FFHQ $\to$ Sunglasses as the $\omega$ value
increases.

$\omega$ | FID ($\downarrow$) | Intra-LPIPS ($\uparrow$)
---|---|---
0.01 | 18.42 | 0.616 $\pm$ 0.020
0.02 | 18.13 |  0.613 $\pm$ 0.011
0.03 | 18.42 | 0.613 $\pm$ 0.016
0.04 | 19.11 | 0.614 $\pm$ 0.013
0.05 | 19.48 | 0.623 $\pm$ 0.015

#### Iteration.

As illustrated in Table 6, the FID ($\downarrow$) and Intra-LPIPS ($\uparrow$)
for FFHQ $\to$ Sunglasses vary as training iterations increase. Initially, the
FID value drops significantly as the generated image gradually resembles the
target image, reaching its lowest at 18.13 with 300 training iterations. After
this point, the FID score stabilizes after around 400 iterations as the
synthesized images closely mirror the target image. The Intra-LPIPS score
steadily decreases with an increase in iterations up to 400, further
suggesting that a higher number of iterations can lead to overfitting to the
target image. Therefore, we select 300 as an optimal number of training
iterations, offering a balance between image quality and diversity.

Table 6: This shows the change in FID (lower is better) and Intra-LPIPS
(higher is better) results for FFHQ $\to$ Sunglasses as the number of training
iterations increases.

Iteration | FID ($\downarrow$) | Intra-LPIPS ($\uparrow$)
---|---|---
0 | 111.32 | 0.650 $\pm$ 0.071
50 | 93.82 | 0.666 $\pm$ 0.020
100 | 58.27 | 0.666 $\pm$ 0.015
150 | 31.08 | 0.654 $\pm$ 0.017
200 | 19.51 | 0.635 $\pm$ 0.014
250 | 18.34 | 0.624 $\pm$ 0.011
300 | 18.13 |  0.613 $\pm$ 0.011
350 | 21.17 | 0.604 $\pm$ 0.016
400 | 21.17 | 0.608 $\pm$ 0.019

##  Appendix B Proofs

###  B.1 Source and Target Model Distance

This subsection introduces the detailed derivation of source and target model
distance, Equation (5) as following:

$\displaystyle\texttt{D}_{\textrm{KL}}\left(p_{\theta_{\mathcal{S}},\phi}(x_{t-1}^{\mathcal{S}}|x_{t}),p_{\theta_{\mathcal{T}},\phi}(x_{t-1}^{\mathcal{T}}|x_{t})\right)$
$\displaystyle=~{}$
$\displaystyle\texttt{D}_{\textrm{KL}}\left(p_{\theta_{(\mathcal{S},\mathcal{T})},\phi}(x_{t-1}|x_{t},y=\mathcal{S}),p_{\theta_{(\mathcal{S},\mathcal{T})},\phi}(x_{t-1}|x_{t},y=\mathcal{T})\right)$
$\displaystyle\approx~{}$
$\displaystyle\texttt{D}_{\textrm{KL}}(\mathcal{N}(x_{t-1};\mu_{\theta_{(\mathcal{S},\mathcal{T})}}+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{S}|x_{t}),\sigma_{t}^{2}\mathbf{I}),\mathcal{N}(x_{t-1};\mu_{\theta_{(\mathcal{S},\mathcal{T})}}+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t}),\sigma_{t}^{2}\mathbf{I}))$ $\displaystyle=~{}$
$\displaystyle\mathbb{E}_{t,x_{0},\epsilon}\left[\frac{1}{2\sigma_{t}^{2}}\left\|\mu_{\theta_{(\mathcal{S},\mathcal{T})}}+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{S}|x_{t})-\mu_{\theta_{(\mathcal{S},\mathcal{T})}}-\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})\right\|^{2}\right]$ $\displaystyle=~{}$
$\displaystyle\mathbb{E}_{t,x_{0},\epsilon}\left[C_{1}\left\|\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{S}|x_{t})-\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})\right\|^{2}\right],$ (11)

where $C_{1}=\gamma/2$ is a constant. Since $C_{1}$ is scale constant, we can
ignore this scale constant for the transfer gap and Equation (11) is the same
as Equation (5).

###  B.2 Similarity-Guided Loss

In this subsection, we introduce the full proof how we get similarity-guided
loss, Equation (6). Inspired by [9], training is carried out by optimizing the
typical variational limit on negative log-likelihood:

$\displaystyle\mathbb{E}[-\log p_{\theta,\phi}(x_{0}|y=\mathcal{T})]\leq~{}$
$\displaystyle\mathbb{E}_{q}\left[-\log\frac{p_{\theta,\phi}(x_{0:T}|y=\mathcal{T})}{q(x_{1:T}|x_{0})}\right]$
$\displaystyle=~{}$ $\displaystyle\mathbb{E}_{q}\left[-\log
p(x_{T})-\sum_{t\geq
1}\log\frac{p_{\theta,\phi}(x_{t-1}|x_{t},y=\mathcal{T})}{q(x_{t}|x_{t-1})}\right]:=L\
.$ (12)

According to [9], $q(x_{t}|x_{0})$ can be expressed as:

$q(x_{t}|x_{0})=\mathcal{N}\left(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\right)\
.$ (13)

Training efficiency can be obtained by optimizing random elements of $L$ (12)
using stochastic gradient descent. Further progress is made via variance
reduction by rewriting $L$ (12) with Equation (13) as Ho et al. [9]:

$\displaystyle L=~{}$
$\displaystyle\mathbb{E}_{q}[\underbrace{\texttt{D}_{\textrm{KL}}\left(q(x_{T}|x_{0},p(x_{T}|y=\mathcal{T})\right)}_{L_{T}}+\sum_{t>1}\underbrace{\texttt{D}_{\textrm{KL}}\left(q(x_{t-1}|x_{t},x_{0}),p_{\theta,\phi}(x_{t-1}|x_{t},y=\mathcal{T})\right)}_{L_{t-1}}$
$\displaystyle-\underbrace{\log
p_{\theta,\phi}(x_{0}|x_{1},y=\mathcal{T})}_{L_{0}}]\ .$ (14)

As Dhariwal and Nichol [5], the conditional reverse noise process
$p_{\theta,\phi}(x_{t-1}|x_{t},y)$ is:

$p_{\theta,\phi}(x_{t-1}|x_{t},y)\approx\mathcal{N}\left(x_{t-1};\mu_{\theta}(x_{t},t)+\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y|x_{t}),\sigma_{t}^{2}\mathbf{I}\right)\ .$ (15)

The $L_{t-1}$ with Equation (15) can be rewrited as:

$\displaystyle L_{t-1}:=$
$\displaystyle~{}\texttt{D}_{\textrm{KL}}\left(q(x_{t-1}|x_{t},x_{0}),p_{\theta,\phi}(x_{t-1}|x_{t},y=\mathcal{T})\right)$
$\displaystyle=$
$\displaystyle~{}\mathbb{E}_{q}\left[\frac{1}{2\sigma_{t}^{2}}\left\|\tilde{\mu}_{t}(x_{t},x_{0})-\mu_{t}(x_{t},x_{0})-\sigma_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y|x_{t})\right\|^{2}\right]$ $\displaystyle=$
$\displaystyle~{}\mathbb{E}_{t,x_{0},\epsilon}\left[C_{2}\left\|\epsilon_{t}-\epsilon_{\theta}(x_{t},t)-\hat{\sigma}_{t}^{2}\gamma\nabla_{x_{t}}\log
p_{\phi}(y=\mathcal{T}|x_{t})\right\|^{2}\right]\ ,$ (16)

where
$C_{2}=\frac{\beta^{2}_{t}}{2\sigma^{2}_{t}\alpha_{t}(1-\bar{\alpha}_{t})}$ is
a constant, and
$\hat{\sigma}_{t}=\left(1-\bar{\alpha}_{t-1}\right)\sqrt{\frac{\alpha_{t}}{1-\bar{\alpha}_{t}}}$.
We define the $L_{t-1}$ as similarity-guided DPMs train loss and we will
ignore the $C_{2}$ for better results during training as [9].

##  Appendix C Limitation

In this subsection, we acknowledge some limitations of our method. Given that
our goal is to transfer the image from the source domain to the target domain,
the images we synthesize will feature characteristics specific to the target
domain, such as sunglasses as shown in Figure 4. This can potentially lead to
inconsistency in the generated images and there is a risk of privacy leakage.
For instance, the reflection in the sunglasses seen in the 3rd and 4th columns
of the 4th row in Figure 4 is very similar to the one in the target image.
This could potentially reveal sensitive information from the target domain,
which is an issue that needs careful consideration in applying this method.

##  Appendix D Samples

Figure 5: The 10-shot image generation samples on FFHQ $\to$ Sunglasses and
FFHQ $\to$ Babies.