# A Principled Method for the Creation of Synthetic Multi-fidelity Data Sets

Clyde Fare
IBM Research Europe - Daresbury
UK
<EMAIL_ADDRESS>
&Peter Fenner
IBM Research Europe - Daresbury
UK
<EMAIL_ADDRESS>
&Edward Pyzer-Knapp
IBM Research Europe - Daresbury
UK
<EMAIL_ADDRESS>

###### Abstract

Multifidelity and multioutput optimisation algorithms are of active interest
in many areas of computational design as they allow cheaper computational
proxies to be used intelligently to aid experimental searches for high
performing species. Characterisation of these algorithms involves benchmarks
that typically either use analytic functions or existing multifidelity
datasets. However, analytic functions are often not representative of relevant
problems, while preexisting datasets do not allow systematic investigation of
the influence of characteristics of the lower fidelity proxies. To bridge this
gap, we present a methodology for systematic generation of synthetic
fidelities derived from preexisting datasets. This allows for construction of
benchmarks that are both representative of practical optimisation problems
while also allowing systematic investigation of the influence of the lower
fidelity proxies.

_Keywords_ Multifidelity, Generative Modelling, Gaussian Process,
Coregionalization, Bayesian Optimization

##  1 Introduction

Building and deploying data-derived models has become a ubiquitous activity in
many fields. Generation of vastly complex models is now possible where the
limiting factor within many fields of application is a sufficiently large and
diverse dataset with which one can train models. For many tasks, financial or
time costs limit the collection of data at the desired accuracy, and so
methods which are able to take advantage of multiple disparate sources of data
are beginning to become popular. One example of this is the emerging area of
multifidelity optimization Song et al. (2018); Huang et al. (2006); Kandasamy
et al. (2017) where optimisation algorithms are able to make use of queries of
approximate variants or lower ’fidelities’ of the intended optimisation
target.

For single fidelity blackbox optimisation problems there are well known
benchmark suites Hansen et al. (2021) that allow comparison of different
algorithms. However preparation of such benchmarks for multifidelity
optimisation is challenging due to the need not only to specify diverse and
relevant optimisation problems but also multiple different proxies. Recent
libraries of analytic functions suitable for multifidelity optimisation have
been developed van Rijn and Schmitt (2020); Mainini et al. (2022) as have some
general benchmark suites Wang et al. (2018); Eggensperger et al. (2021).
However the lower fidelity approximations of current benchmarks do not offer
fine grained tools for controlling the behaviour of the low fidelity proxies.
This limits investigation of the behaviour of the optimization algorithms on
the characteristics of the different fidelities and risks biasing development
of novel multifidelity algorithms.

To build better multifidelity tooling there is a need to be able to
systematically investigate how different algorithms perform on different
problems where the number of quality of the lower fidelity proxies can
explicitly varied. To further this goal we present a simple means to generate
sets of synthetic fidelities based on any single or multifidelity dataset.
These synthetic fidelities exhibit a controllable degree of correlation to the
reference and known lower fidelity data and thus can be used to benchmark
multifidelity optimisation algorithms in a systematic fashion.

The creation of such synthetic fidelities falls under generative modelling of
functions for which much prior work exists Blundell et al. (2015); C.E.
Rasmussen and Williams (2004); Gal and Ghahramani (2016); Hernandez-Lobato and
Adams (2015); Harshvardhan et al. (2020) but to our knowledge there has been
little focus on the generative modelling of proxy functions with specific
consideration of the relation between the generated proxies and the known
reference function(s) those proxies are based upon.

##  2 Algorithm Outline

Conceptually our algorithm involves:

  * •

Taking preexisting single or multifidelity data along with a specification for
the desired degree of correlation to that existing fidelity data.

  * •

Fitting a Bayesian model which includes extraction of characteristics common
to the data.

  * •

Modifying the model to include a posterior over an additional synthetic
fidelity (for which there is no data).

  * •

Controlling underlying parameters of the model such that a sample drawn from
the posterior over the synthetic fidelity possesses the desired degrees of
correlation to the preexisting data.

We choose a Gaussian Process (GP) C.E. Rasmussen and Williams (2004) as the
underlying methodology used to model function distributions as GPs provide an
analytically tractable posterior formulation. In particular we make use of a
Multiple Output Gaussian Process (MOGP) and as we are interested in systematic
control of the posterior we choose to use a simple coregionalisation kernel
architecture Bonilla et al. (2008); Álvarez et al. (2012) to model inter
function covariances, which is paired with a typical GP kernel used to model
intra function covariances.

The training regime we adopt is to first take any data associated with the
reference ground truth function and any available lower fidelity proxies and
use this data to fit the hyperparameters for the MOGP via maximisation of the
log marginal likelihood of the data. As these hyperparameters include the
kernel used to model the intra-function variances this process captures
characteristics (lengthscales, spectral mixtures, etc.) common to the
reference function and any given proxy functions. As a consequence samples
drawn from the GP prior using these optimised hyperparameters result in
functions that share these characteristic.

Having constructed a model of the available data we then modify the
coregionalisation kernel structure to introduce an additional function output
which will be used to model the synthetic fidelity. As shown in section 3, a
sample from this expanded coregionalisation GP can be expressed in terms of a
linear combination of the preexisting fidelity data and a sample from the
intra-fidelity prior. We can then solve for the coefficients that define this
linear combination in order to achieve a sample with desired correlation to
the ground truth data and any provided lower fidelity proxies.

A graphical illustration of the steps associated with this algorithm can be
seen in figure 1

Figure 1: Panels showing progress of algorithm. Panel A shows initial data
available for a ground truth function of interest (top blue) and a lower
fidelity proxy (bottom green). Panel B shows the data along side the posterior
of a MOGP using a coregionalisation kernel that has been fit via maximum log
likelihood to the initial data. Panel C shows the prior that uses the fit
hyperparameters for the new synthetic fidelity and a sample of the synthetic
fidelity drawn from that prior. While Panel D shows two different
transformations of the samples and the posterior, chosen to achieve two
different correlations to the existing ground truth and lower fidelity proxy.

##  3 Mathematical Background

###  3.1 Gaussian Process Regression

Gaussian processes express a Gaussian prior over function space where the
prior mean is usually taken to be zero and the prior covariance is specified
by a kernel function. Given an initial set of training points represented by
an $n$ by $d$ matrix $X$ and associated $n$ dimensional target vector
$\mathbf{y}$, where $n$ is the number of data points and $d$ is the
dimensionality of the data, the posterior mean evaluated at a set of test
points represented by an $n^{\prime}$ by $d$ matrix $x_{*}$ is defined by:

$\mathbf{\mu_{\ast}}=K^{T}_{\ast}K^{-1}\mathbf{y}$

where T indicates the transpose operation, $\mathbf{\mu_{*}}$ is an
$n^{\prime}$ dimensional vector of predicted means for the function values at
positions $x_{*}$, $K_{*}$ is the $n$ by $n^{\prime}$ matrix defining the
covariance between $X$ and $x_{*}$ and $K^{-1}$ is the $n$ by $n$ inverse
covariance matrix between $X$ and $X$. The $K_{\ast}$ and $K$ matrices are
computed using the kernel function i.e. $K_{ij}=k_{f}(x_{i},x_{j},\theta)$
where $k_{f}$ is a kernel function, $x_{i}$ and $x_{j}$ are the $i$th and
$j$th training points respectively and $\theta$ are a set of kernel
hyperparameters.

The posterior covariance evaluated at points $x_{*}$ is defined by:

$\sigma_{\ast}=K_{\ast\ast}-K^{T}_{\ast}K^{-1}K_{\ast},$

where $\sigma_{\ast}$ is an $n^{\prime}$ by $n^{\prime}$ posterior covariance
matrix and $K_{\ast\ast}$ is the $n^{\prime}$ by $n^{\prime}$ prior covariance
matrix between $x_{*}$ and $x_{*}$.

Once the posterior is calculated we can sample from this posterior via
transformation of a white noise vector $\mathbf{r}$, where the sample over
positions $x_{*}$ is:

$\mathbf{s}=\mathbf{\mu_{*}}+\sigma_{*}\mathbf{r}$

where $\mathbf{s}$ is a $n^{\prime}$ dimensional vector of the posterior
sample values and $\mathbf{r}$ is an $n^{\prime}$ dimensional Gaussian random
vector sampled from $N(0,\mathbb{1})$

We are interested in an MOGP I.e. rather than expressing a distribution over a
single set of function values, we express a joint distribution over multiple
sets of functional values. To achieve this we augment the domain with an
additional dimension that encodes the function index. This allows us to
express the covariance between values of different functions evaluated at
different points within the domain as a function of points in this augmented
domain. We make use of a coregionalisation kernel to calculate this
covariance.

In this picture the kernel function is equal to a pairwise value which
expresses the relatedness of the fidelities for the two points in question and
a standard (typically stationery) GP kernel function. I.e.

$k^{coreg}_{f}(x^{k}_{i},x^{l}_{j})=k_{f}(x_{i},x_{j})t_{kl}$

where $x^{k}_{i}$ is the $i$th point at fidelity $k$, $x^{l}_{j}$ is the $j$th
point at fidelity $l$, $k_{f}(x_{i},x_{j})$ is a standard GP kernel function
and $t_{kl}$ is a coefficient that specifies the covariance between fidelity
$k$ and $l$.

If the training data consists of the value of each fidelity on the same set of
training points $X$, the full covariance matrix $K$ can be expressed as:

$K^{coreg}=\Sigma_{T}\otimes K_{c}$

where $\Sigma_{T}$ is an $n_{t}$ by $n_{t}$ task matrix made up of the
$t_{kl}$ coefficients that encodes pairwise covariances between the $n_{t}$
different fidelities and $K_{c}$ is the core task independent kernel matrix
between $X$ and $X$.

The values for the $t_{kl}$ are typically fit by maximising the log marginal
likelihood of the training data along with any parameters of the kernel
function.

###  3.2 Generating a Posterior over a synthetic fidelity

We consider the situation where we have a set of points from a domain
represented as a $n_{x}$ by $n_{d}$ sized matrix $X$ where $n_{x}$ is the
number of points in the domain for which values are available for the ground
truth function and any available lower fidelity proxies and $n_{d}$ is the
dimensionality of the domain. The target values are initially represented by
an $n_{x}*n_{t}$ target vector $\mathbf{y}$, which is rearranged to form a
$n_{x}$ by $n_{t}$ matrix $Y$ where $n_{t}$ is the number of fidelities for
which data is available (and is equal to 1 if we only have data from a single
reference ground truth function).

We suppose we have fit the hyperparameters for the prior for a
coregionalisation kernel GP via maximisation of the log marginal likelihood
and as such have extracted parameters $\theta$ for the core kernel function
along with the values of the task matrix $\Sigma_{T}$. We now wish to consider
the case where we will calculate the posterior over an additional task for
which there is no data. This will be the synthetic fidelity which we will
ultimately sample from. To achieve this we will expand the task matrix to
include an additional task. This gives rise to an $n_{t}+1$ by $n_{t}+1$
dimensional task matrix $\Sigma^{\prime}_{T}$, which requires provision of an
additional $n_{t}+1$ parameters which we initially consider unknown. We are
interested in calculating the posterior distribution for this new synthetic
fidelity at the same domain points for which the values of the known
fidelities are available. I.e. $x_{*}=X$. We note that this is an extension of
the situation examined by Bonilla Bonilla et al. (2008).

Inserting the expression for the coregionalisation kernel into the expression
for the posterior mean and setting $x_{*}$ equal to $X$ gives:

$\displaystyle\mathbf{\mu}_{p}^{T}$ $\displaystyle=(\Sigma_{T*}^{T}\otimes
K_{c}^{T})(\Sigma_{T}\otimes K_{c})^{-1}\mathbf{y}$
$\displaystyle=(\mathbf{\Sigma}_{T*}^{T}\Sigma^{-1}_{T})Y^{T}$

where:

  * •

$\mathbf{\mu}_{p}$ is the predicted mean values for the synthetic fidelity
task,

  * •

we have wrapped the target vector $\mathbf{y}$ into a target matrix $Y$ such
that the $i$th column of $Y$ is the values of fidelity $i$ at domain points
$x_{*}$,

  * •

$\mathbf{\Sigma}_{T*}$ is the $n_{t}$ sized vector of task covariances between
the synthetic fidelity and the given fidelity data,

  * •

$\Sigma_{T}$ is the $n_{t}$ by $n_{t}$ matrix of task covariances associated
with the given fidelity data.

The equivalent expression for the posterior covariance is:

$\displaystyle\sigma_{p}$ $\displaystyle=(\Sigma_{T**}\otimes
K_{c})-(\mathbf{\Sigma}_{T*}^{T}\otimes K_{c}^{T})(\Sigma_{T}\otimes
K_{c})^{-1}(\mathbf{\Sigma}_{T*}\otimes K_{c})$
$\displaystyle=(\Sigma_{T**}-\mathbf{\Sigma}_{T*}^{T}\Sigma^{-1}_{T}\mathbf{\Sigma}_{T*})K_{c}$

where $\Sigma_{T**}$ is a scalar defining the task variance associated with
the synthetic fidelity

We note the similarity of the above equations to the standard GP equations
given in section 3.1. The expression for the predictive mean is composed of
the $n_{t}$ sized decorrelated contribution vector of the preexisting
fidelities to the synthetic fidelity
$\mathbf{\Sigma}_{T*}^{T}\Sigma^{-1}_{T}$, which is multiplied by the target
matrix $Y$. Thus it is as if we have a Gaussian Process expressed over tasks
where rather than scalar valued output we have vector valued output.

Similarly the expression for the posterior variance strongly resembles the
posterior variance of a single fidelity GP where again we replace the prior
variance and inverse prior variances over the points in the domain with prior
variance and inverse prior variance over the tasks. We can interpret the
posterior task variance
$\Sigma_{T**}-\mathbf{\Sigma}_{T*}^{T}\Sigma^{-1}_{T}\mathbf{\Sigma}_{T*}$ as
the variance that remains after observing the training (task) data (which can
be zero if the fidelities for which we have data are sufficiently correlated
with the synthetic fidelity). As we are expressing the posterior over a single
task - the synthetic fidelity’s posterior task variance is a scalar. The full
posterior covariance is then equal to this task variance scalar multiple by
the prior covariance.

###  3.3 Generating a synthetic fidelity sample

A sample of the synthetic fidelity from this posterior is thus:

$\displaystyle\mathbf{s}$
$\displaystyle=\mathbf{\mu}_{p}+\mathbf{\sigma}_{p}\mathbf{r}$
$\displaystyle=Y(\Sigma^{-1}_{T}\mathbf{\Sigma}_{T*})+(\Sigma_{T**}-\mathbf{\Sigma}_{T*}^{T}\Sigma^{-1}_{T}\mathbf{\Sigma}_{T*})K_{c}\mathbf{r}$

where as before $\mathbf{r}$ is an $n_{x}$ sized vector of white noise values.

Examining this expression we can see that a synthetic fidelity posterior
sample corresponds to a linear combination of basis vectors $\mathbf{y_{i}}$
and sample coefficients $c_{i}$

$\mathbf{s}=\sum_{i=0}^{i<n_{t}+1}{c_{i}\mathbf{y}_{i}}$

where $\mathbf{y}_{i}$ is column $i$ of $Y$ and $c_{i}$ is the $i$th component
of the vector $(\Sigma^{-1}_{T}\mathbf{\Sigma}_{T*})$ for $i<n_{t}$, and we
additionally define $\mathbf{y}_{n_{t}}=K_{c}\mathbf{r}$ and
$c_{n_{t}}=(\Sigma_{T**}-\mathbf{\Sigma}_{T*}^{T}\Sigma^{-1}_{T}\mathbf{\Sigma}_{T*})$
to account for the posterior variance.

This can be expressed in terms of an expanded $n_{x}$ by $n_{t}+1$ target
matrix $Y^{\prime}$ as:

$\mathbf{s}=Y^{\prime}\mathbf{c}$

where $\mathbf{c}$ is the vector of sample coefficients
$\mathbf{c}_{i}=c_{i}$.

The only dependence of the posterior sample on the task matrix is through the
coefficients $c_{i}$ while the basis vectors $\mathbf{y}_{i}$ are independent
of the task matrix and as such they can be calculated ahead of time regardless
of the desired sample correlation.

###  3.4 Specifying a sample with a specific covariance

Assuming the values of the task matrix $\Sigma_{T}$ have been provided via
maximisation of the log-marginal likelihood, the sample coefficients $c_{i}$
are then specified by the inter-task covariance between the synthetic fidelity
and the given fidelities $\mathbf{\Sigma}_{T*}$. Typically
$\mathbf{\Sigma}_{T*}$ would be specified by maximisation of the log-marginal
likelihood or sampled from an underlying prior; however, in our case we lack
data for this synthetic task and more importantly we wish to generate samples
from an MOGP posterior such that those samples possess a particular
correlation with the preexisting fidelity data. To achieve this we will
directly control the values of the coefficients $c_{i}$ which will implicitly
set inter-task covariances $\mathbf{\Sigma}_{T*}$ such that the generated
sample possess the intended correlation. By transforming a set of intended
correlations between the synthetic fidelity sample and the existing fidelity
data into a set of covariances between the synthetic fidelity sample and the
existing fidelity data and augmenting this with a covariance between the
synthetic fidelity sample and the sample from the task-independent prior we
can formulate a linear problem whose solution will define the sample
coefficients we require.

Thus we suppose we have generated the $\mathbf{y}_{i}$ basis vectors and have
been provided with a vector of Pearson Correlations $\mathbf{P}_{c}$
specifying the degree to which the sample should be correlated to the existing
$n_{t}+1$ sample basis vectors, i.e. to the preexisting fidelities and to the
sample from the intra-fidelity kernel Prior. (note by the required properties
of the full Correlation Matrix, specification of the correlation between the
synthetic fidelity sample and the task independent prior sample is determined
given specification of the correlations to the preexisting fidelities).

By making use of a heuristic defined below we choose a variance
$\mathbf{\sigma}_{h}$ for our intended sample and use this to transform the
vector of Pearson Correlations into a vector of sample covariances
$\mathbf{\sigma}_{s}$:

$\mathbf{\sigma}_{s}=\Sigma^{0.5}_{h}\cdot\mathbf{P}_{c}$

where $\Sigma_{h}^{0.5}$ is a diagonal matrix with diagonal elements
$\sqrt{\sigma_{h}}$.

We define zero mean sample basis vectors as:

$\mathbf{\tilde{y}}_{i}=\mathbf{y}_{i}-E[\mathbf{y}_{i}].$

Similarly we define the zero mean sample as:

$\displaystyle\mathbf{\tilde{s}}$ $\displaystyle=\mathbf{s}-E[\mathbf{s}]$
$\displaystyle=\sum_{i=0}^{i<n_{t}+1}{c_{i}\mathbf{y}_{i}-\sum_{i=0}^{i<n_{t}+1}{c_{i}E[\mathbf{y}_{i}]}})$
$\displaystyle=\sum_{i=0}^{i<n_{t}+1}{c_{i}(\mathbf{y}_{i}-E[\mathbf{y}_{i}]})$
$\displaystyle=\sum_{i=0}^{i<n_{t}+1}{c_{i}\mathbf{\tilde{y}}_{i}}$

i.e.

$\mathbf{\tilde{s}}=\tilde{Y^{\prime}}\mathbf{c}$

where $\tilde{Y^{\prime}}$ is the matrix of normalised sample basis vectors
$\tilde{Y^{\prime}}_{i}=\mathbf{\tilde{y}}_{i}$.

As we have transformed the Pearson Correlations into covariances and the dot
product between two zero mean sample vectors gives the covariance, we can
express the requirement that the covariance between the synthetic fidelity
sample and the sample from fidelity $i$ be equal to $\sigma_{si}$ as:

$\mathbf{\tilde{s}}\cdot\mathbf{\tilde{y}}_{i}=\sigma{{}_{s_{i}}}$

i.e.

$\tilde{Y^{\prime}}^{T}\mathbf{\tilde{s}}=\mathbf{\sigma}_{s}.$

By substituting the expression for $\mathbf{\tilde{s}}$ above we arrive at:

$\tilde{Y^{\prime}}^{T}\tilde{Y^{\prime}}\mathbf{c}=\mathbf{\sigma}_{s}$

which we can solve for the basis coefficients:

$\mathbf{c}=(\tilde{Y^{\prime}}^{T}\tilde{Y^{\prime}})^{-1}\mathbf{\sigma}_{s}$

and hence generate the required sample.

###  3.5 Heuristic for transformation of sample Pearson Correlations to sample
covariances

It is natural to specify desired correlations to a set of preexisting fidelity
data; however, to solve for the desired sample we require transformation of
the correlations into a set of covariances. This can be achieved by
specification of a desired sample variance. We make use of a heuristic to pick
this sample variance that ensures that a correlation vector that specifies
100% correlation to one of the preexisting fidelities will be assigned a
variance equal to the variance of the matching fidelity.

The proposed heuristic is to use the overlap of the vector $\mathbf{P}_{c}$
with the rows of the $n_{t}+1$ by $n_{t}+1$ correlation matrix $C$ associated
with the sample basis vectors defined as.

$\displaystyle
C_{ij}=\Sigma_{i}^{-0.5}\mathbf{\tilde{y}}_{i}\cdot\Sigma_{j}^{-0.5}\mathbf{\tilde{y}}_{j}$

where $\Sigma_{i}$ is a diagonal matrix with diagonal elements equal to the
variance of $\mathbf{\tilde{y}}_{i}$.

We then iteratively determine a set of overlap coefficients by picking the
maximally overlapping correlation matrix row vector, noting the overlap then
projecting out the selected vector from $P_{c}$ and continuing until we have a
set of overlaps associated with each sample basis. I.e.

$\displaystyle\mathbf{v}\leftarrow\mathbf{\mathbb{0}}$
$\displaystyle\mathbf{w}\leftarrow\mathbf{\mathbb{0}}$
$\displaystyle\text{for}\;j<n_{t}+1:$
$\displaystyle\;\;\;\mathbf{P}_{c}\leftarrow\mathbf{P}_{c}-(\mathbf{P}_{c}\cdot\mathbf{v})\mathbf{v}$
$\displaystyle\;\;\;i\leftarrow\text{arg\,max}_{i}(\mathbf{C}_{i}\cdot\mathbf{P}_{c})$
$\displaystyle\;\;\;\mathbf{v}\leftarrow\mathbf{C}_{i}$
$\displaystyle\;\;\;w_{j}\leftarrow\mathbf{C}_{i}\cdot\mathbf{P}_{c}$

We then determine the sample variance by computing the weighted mean of the
sample basis variances. I.e.

$\mathbf{\sigma}_{h}=\mathbf{w}\cdot\text{diag}(\tilde{Y}^{T}\tilde{Y})$

###  3.6 Generating valid Pearson Correlation Vectors

The above assumes provision of a vector of desired Pearson Correlations
$\mathbf{P}_{c}$ to the $n_{t}+1$ prior fidelities and to the sample from the
prior. The constraints on such a vector are well known; however, for
completeness we outline the generation procedure.

We suppose that the elements of the $\mathbf{P}_{c}$ vector are chosen
sequentially and that the bounds for the values of a particular element of the
$\mathbf{P}_{c}$ vector are conditioned on the previous choices. I.e we start
with a free choice of an initial correlation value of the synthetic fidelity
sample to one of the $n_{t}$ fidelities with bounds of $[-1,1]$, then
subsequent choices are constrained such that the combined correlation matrix
as defined above is a valid correlation matrix. I.e. is positive semi-definite
with diagonal elements equal to one.

Given the set of sample basis vectors we calculate the correlation between
them, giving an $n_{t}+1$ by $n_{t}+1$ correlation matrix $C$. Then via
Cholesky decomposition we have:

$C=U^{T}U$

where $U$ is an upper triangular matrix. We now consider the expansion of $C$
by an additional row and column to form an $n_{t}+2$ by $n_{t}+2$ combined
correlation matrix $C^{\prime}$ where this additional row/column will be the
Pearson correlation vector $\mathbf{P}_{c}$. The expansion of $C$ to
$C^{\prime}$ leads to an equivalent expansion of the Cholesky factorised
matrix $U$ to $U^{\prime}$.

Via the Cholesky decomposition, for each $i$ we have

$\mathbf{U}^{\prime}_{i}\cdot\mathbf{U}^{\prime}_{p}=P_{ci}$

where $\mathbf{U}^{\prime}_{i}$ is the $i$th column of $U^{\prime}$ and, for
ease of notation, we let $p=n_{t}+1$ be the index of the last column.

Since $U^{\prime}$ is an upper triangular matrix it follows that:

$\mathbf{U}^{\prime}_{i}\cdot\mathbf{U}^{\prime}_{p}=\mathbf{U}^{\prime}_{[:i],i}\cdot\mathbf{U}^{\prime}_{[:i],p}+U^{\prime}_{ii}U^{\prime}_{ip}$

where $\mathbf{U}^{\prime}_{[:i],j}$ is the vector corresponding to the first
$i$ elements of column $j$ of the Cholesky factor matrix $U^{\prime}$.

We can use this relation to provide bounds on the possible values of
$\mathbf{P}_{c}$, and we do so sequentially by providing bounds on $P_{ci}$ in
terms of the previous values $P_{c0}\ldots P_{c,{i-1}}$.

As the Cholesky factor for a correlation matrix has element
$U^{\prime}_{00}=1$ for initial choice of correlation $P_{c0}$ we have:

$U^{\prime}_{0p}=P_{c0}$

Further, as the correlation of the synthetic fidelity sample with itself must
be 1 it follows that $\mathbf{U}^{\prime}_{p}\cdot\mathbf{U}^{\prime}_{p}=1$.
Then, since

$\mathbf{U}^{\prime}_{p}\cdot\mathbf{U}^{\prime}_{p}=\mathbf{U}^{\prime}_{[:i],p}\cdot\mathbf{U}^{\prime}_{[:i],p}+{U^{\prime}_{ip}}^{2}+\sum_{j=i+1}^{p}{U^{\prime}_{jp}}^{2}$

this means that the bounds on the allowed values for $U^{\prime}_{ip}$ can be
expressed in terms of the previous values $\mathbf{U}^{\prime}{{}_{[:i],p}}$
as follows:

$\mathbf{U}^{\prime}_{[:i],p}\cdot\mathbf{U}^{\prime}_{[:i],p}+{U^{\prime}_{ip}}^{2}\leq
1$

i.e.

$-\sqrt{1-\mathbf{U}^{\prime}_{[:i],p}\cdot\mathbf{U}^{\prime}_{[:i],p}}\leq
U^{\prime}_{ip}\leq+\sqrt{1-\mathbf{U}^{\prime}_{[:i],p}\cdot\mathbf{U}^{\prime}_{[:i],p}}$

Which translates into bounds for the Pearson correlation coefficient $P_{ci}$
of:

$\mathbf{U}^{\prime}_{[:i],i}\cdot\mathbf{U}^{\prime}_{[:i],p}-U^{\prime}_{ii}\sqrt{1-\mathbf{U}^{\prime}_{[:i],p}\cdot\mathbf{U}^{\prime}_{[:i],p}}\leq
P_{ci}\leq\mathbf{U}^{\prime}_{[:i],i}\cdot\mathbf{U}^{\prime}_{[:i],p}+U^{\prime}_{ii}\sqrt{1-\mathbf{U}^{\prime}_{[:i],p}\cdot\mathbf{U}^{\prime}_{[:i],p}}$

##  4 Pseudo-Code

⬇

def draw_s(K, X, Y):

# generate sample from the prior

r = normal(0, ones_like(X))

y_n = (K @ r).t()

# set prior sample std to mean of Y std to make heuristics choice

# of sample std more intuitive

y_n /= y_n.std()

y_n *= Y.std(axis=1).mean()

# create set of reference vectors

Y_ = vstack([Y, y_n])

Y_ -= Y_.mean(axis=1)[:,None]

# calculate reference covariances

fid_cov = (Y_@Y_.t())

fid_stds = diag(fid_cov)**0.5

# calculate reference Pearson Correlations

_Y_ = Y_/fid_stds[:, None]

fid_pcs = _Y_ @ _Y_.t()

pcs = build_pcs(fid_pcs)

heuristic_weights = get_heuristic_weights(pcs, fid_pcs)

synth_fid_std = heuristic_weights @ fid_stds

# using the reference vector stds and the calculated synth_fid std

# generate a set of covariances from the supplied pearson correlations

sigma_s = pcs * fid_stds * synth_fid_std

# using the covariances solve for the coefficients

c = inverse(fid_cov) @ sigma_s

# use the coefficients to generate a sample

s = c[:,None].t() @ Y_

return s

⬇

def get_heuristic_weights(pcs, ref_C):

weights = []

for i in range(len(ref_C)):

overlaps = ref_C @ pcs

ind = argmax(overlaps)

weight = overlaps[ind]

weights.append(weight)

pcs = pcs - weight*ref_C[ind]

return array(weights)

def build_pcs(ref_C):

"""Sequentially build up Pearson correlation values for synthetic sample

against known fidelities and prior sample"""

ref_L = cholesky(ref_C)

# expand cholesky matrix with additional row/column

master_L = zeros([len(ref_C)+1, len(ref_C)+1])

master_L[:-1, :-1] = ref_L

master_L[-1, -1] = 1

pcs = []

for i in range(len(ref_C)+1):

partial_pc_i = (master_L[-1,:i] @ master_L[i,:i])

abs_max_l_i = (1 - master_L[-1,:i]@master_L[-1,:i])**0.5

if i != len(ref_C):

bound_pc_i = (master_L[i,i]*abs_max_l_i)

else:

bound_pc_i = (abs_max_l_i**2)

upper = partial_pc_i + bound_pc_i

lower = partial_pc_i - bound_pc_i

allowed_vals = [lower, upper]

pc = get_input(allowed_vals)

pcs.append(pc)

# convert pc back in l and add to the master_L matrix

_l = (pc - partial_pc_i)/ master_L[i,i]

master_L[-1,i] = _l

return pcs

##  5 Examples

In this section we graphically show examples generated for the Liu Liu et al.
(2018) and Currin Currin et al. (1991) synthetic functions making use of the
ground truth and low fidelity proxy functions within a dual task
coregionalisation kernel MOGP which in turn made use of a spectral mixture
model Parra and Tobar (2017) with 4 mixtures as the intra-fidelity kernel.

Figure 2: The left panel shows reference high and and low fidelity function
data generated using Liu’s functions, where the data points used for training
are shown by the markers and the underlying functions are shown by the dashed
lines. The right panel shows a selection of six synthetic fidelity samples
that have been generated with varying sample correlations to the high
fidelity, predicted values are again shown with markers while the dashed line
shows an interpolation of the data Figure 3: The left panel shows reference
high and low fidelity function data generated using Currin’s functions. Data
points were taken from a 20 x 20 grid. To aid visualisation a contour plot of
the function values is present on the x-y plane in addition to projections of
the function values onto the x-f and y-f planes. Colouring of the function
surface and x-y contour plot reflect the value of the function while colouring
of the function projections reflects the distance of the function value from
the projection plane.The right hand panel shows a selection of six synthetic
fidelity samples that have been generated with varying sample correlation to
the high fidelity.

##  6 Conclusion

In this work we present an initial approach to the problem of systematic
synthesis of datasets for the evaluation of multi-fidelity optimisation
problems. We show that we can construct a multi-output Gaussian process making
use of the coregionalisation kernel based on high fidelity data from a ground
truth function and any lower fidelity proxies that are available and expand
this GP it to include a synthetic fidelity. We show that samples drawn from
the posterior of this synthetic fidelity can be expressed in terms of a linear
combination of the known fidelity data and a sample from the prior and we use
this to generate synthetic samples that are arbitrarily correlated to the
ground truth function and the available lower fidelity proxies. We anticipate
this approach will allow more detailed interrogation into the factors that
influence multifidelity optimisations in particular the relationship between
lower fidelity proxy costs and interfidelity correlation on regret curves.

##  7 Acknowledgement

This work was supported by the Hartree National Centre for Digital Innovation,
a collaboration between STFC and IBM.

## References

  * (1)
  * Álvarez et al. (2012) Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. 2012\.  Kernels for Vector-Valued Functions: A Review.  _Foundations and Trends® in Machine Learning_ 4, 3 (jun 2012), 195–266.  https://doi.org/10.1561/2200000036 arXiv:1106.6251
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015.  Weight uncertainty in neural networks. In _32nd International Conference on Machine Learning, ICML 2015_.  arXiv:1505.05424
  * Bonilla et al. (2008) Edwin V Bonilla, Kian M Chai, and Christopher Williams. 2008\.  Multi-task Gaussian process prediction. In _Advances in neural information processing systems_. 153–160.
  * C.E. Rasmussen and Williams (2004) C.E. Rasmussen and K.I. Williams. 2004.  _Gaussian processes for machine learning._ Vol. 14.  MIT press. 69–106 pages.  https://doi.org/10.1142/S0129065704001899 arXiv:026218253X
  * Currin et al. (1991) Carla Currin, Toby Mitchell, Max Morris, and Don Ylvisaker. 1991.  Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments.  _J. Amer. Statist. Assoc._ 86, 416 (1991), 953–963.
  * Eggensperger et al. (2021) Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. 2021\.  HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO.  https://doi.org/10.48550/ARXIV.2109.06716
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.  _33rd International Conference on Machine Learning, ICML 2016_ 3 (2016), 1651–1660.  arXiv:1506.02142
  * Hansen et al. (2021) Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2021\.  COCO: a platform for comparing continuous optimizers in a black-box setting.  _Optimization Methods and Software_ (2021).  https://doi.org/10.1080/10556788.2020.1808977 arXiv:1603.08785
  * Harshvardhan et al. (2020) GM Harshvardhan, Mahendra Kumar Gourisaria, Manjusha Pandey, and Siddharth Swarup Rautaray. 2020\.  A comprehensive survey and analysis of generative models in machine learning.  _Computer Science Review_ (2020).  https://doi.org/10.1016/j.cosrev.2020.100285
  * Hernandez-Lobato and Adams (2015) Jose Miguel Hernandez-Lobato and Ryan Adams. 2015.  Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks. In _Proceedings of the 32nd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 37)_ , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 1861–1869.  https://proceedings.mlr.press/v37/hernandez-lobatoc15.html
  * Huang et al. (2006) D. Huang, T. T. Allen, W. I. Notz, and R. A. Miller. 2006\.  Sequential kriging optimization using multiple-fidelity evaluations.  _Structural and Multidisciplinary Optimization_ 32, 5 (2006), 369–382.  https://doi.org/10.1007/s00158-005-0587-0
  * Kandasamy et al. (2017) Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. 2017\.  Multi-fidelity Bayesian optimisation with continuous approximations. In _34th International Conference on Machine Learning, ICML 2017_ , Vol. 4. 2861–2878.  arXiv:1703.06240
  * Liu et al. (2018) Haitao Liu, Jianfei Cai, and Yew Soon Ong. 2018.  Remarks on multi-output Gaussian process regression.  _Knowledge-Based Systems_ 144 (2018), 102–121.  https://doi.org/10.1016/j.knosys.2017.12.034
  * Mainini et al. (2022) L Mainini, A Serani, M P Rumpfkeil, E Minisci, D Quagliarella, H Pehlivan, S Yildiz, S Ficini, R Pellegrini, F Di Fiore, D Bryson, M Nikbay, M Diez, and P Beran. 2022.  Analytical Benchmark Problems for Multifidelity Optimization Methods.  https://doi.org/10.48550/ARXIV.2204.07867
  * Parra and Tobar (2017) Gabriel Parra and Felipe Tobar. 2017.  Spectral mixture kernels for multi-output Gaussian processes.  _Advances in Neural Information Processing Systems_ 2017-Decem, Nips (2017), 6682–6691.  arXiv:1709.01298
  * Song et al. (2018) Jialin Song, Yuxin Chen, and Yisong Yue. 2018.  A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes.  (nov 2018), 1–17.  arXiv:1811.00755 http://arxiv.org/abs/1811.00755
  * van Rijn and Schmitt (2020) Sander van Rijn and Sebastian Schmitt. 2020.  MF2: A Collection of Multi-Fidelity Benchmark Functions in Python.  _Journal of Open Source Software_ (2020).  https://doi.org/10.21105/joss.02049
  * Wang et al. (2018) Handing Wang, Yaochu Jin, and John Doherty. 2018.  A Generic Test Suite for Evolutionary Multifidelity Optimization.  _IEEE Transactions on Evolutionary Computation_ 22, 6 (2018), 836–850.  https://doi.org/10.1109/TEVC.2017.2758360


# FineDiffusion: Scaling up Diffusion Models for
Fine-grained Image Generation with 10,000 Classes

Ziying Pan1 Kun Wang4,5 Gang Li2,5, Corresponding author (e-mail:
ucasligang@gmail.com). Feihong He6 Xiwang Li4,5 &Yongxuan Lai1,3
1School of Informatics, Xiamen University
2Institute of Software, Chinese Academy of Sciences
3School of Mathematics and Information Engineering, Longyan University
4Shenyang Institute of Computing Technology of Chinese Academy of Sciences
5University of Chinese Academy of Sciences
6School of Computer Science and Technology, Soochow University

###### Abstract

The class-conditional image generation based on diffusion models is renowned
for generating high-quality and diverse images. However, most prior efforts
focus on generating images for general categories, e.g., 1000 classes in
ImageNet-1k. A more challenging task, large-scale fine-grained image
generation, remains the boundary to explore. In this work, we present a
parameter-efficient strategy, called FineDiffusion, to fine-tune large pre-
trained diffusion models scaling to large-scale fine-grained image generation
with 10,000 categories. FineDiffusion significantly accelerates training and
reduces storage overhead by only fine-tuning tiered class embedder, bias
terms, and normalization layers’ parameters. To further improve the image
generation quality of fine-grained categories, we propose a novel sampling
method for fine-grained image generation, which utilizes superclass-
conditioned guidance, specifically tailored for fine-grained categories, to
replace the conventional classifier-free guidance sampling. Compared to full
fine-tuning, FineDiffusion achieves a remarkable 1.56$\times$ training speed-
up and requires storing merely 1.77% of the total model parameters, while
achieving state-of-the-art FID of 9.776 on image generation of 10,000 classes.
Extensive qualitative and quantitative experiments demonstrate the superiority
of our method compared to other parameter-efficient fine-tuning methods. The
code and more generated results are available at our project website:
https://finediffusion.github.io/.

Figure 1: Examples of images generated by the FineDiffusion model. The top row
shows Bird images at a resolution of 512$\times$512 pixels, generated using
the iNaturalist 2021 mini dataset. The bottom two rows are at a resolution of
256$\times$256 pixels, showing images of Plant, Mammal, Arthropoda, and Ray-
finned fish generated from the same dataset. Each image is from a distinct
fine-grained class, and every three images in the same row belong to one
superclass. The generated results of fine-grained class images under the same
superclass showcase the powerful fine-grained image generation capability of
our method.

##  1 Introduction

Over the past decade, we have witnessed explosive growth in the realm of image
generation due to the emergence of deep generative models. Recently, diffusion
models Ho et al. (2020); Song et al. (2020) have pushed back new frontiers of
image generation, especially in text-to-image generation Nichol et al. (2021);
Rombach et al. (2022); Ramesh et al. (2022); Saharia et al. (2022) and class-
conditional image generation Peebles and Xie (2023); Gao et al. (2023). These
approaches outperform Generative Adversarial Networks (GANs) Goodfellow et al.
(2014) in synthesizing high-quality and diverse images.

Prior efforts for class-conditional image generation focus on generating
images from general categories, e.g., the 1000 classes in ImageNet-1k Peebles
and Xie (2023); Gao et al. (2023); Kim et al. (2022). Meanwhile, training
diffusion models for class-conditional image generation from scratch involves
considerable computational costs, primarily due to their large parameter sizes
and mode-covering behavior Rombach et al. (2022). Consider the recent
Diffusion Transformer (DiT) Peebles and Xie (2023) as an illustration,
specifically the DiT-XL/2 model, the largest variant in the DiT family
renowned for its cutting-edge generative performance on the ImageNet class-
conditional generation benchmark. To elaborate, DiT-XL/2 boasts 640 million
parameters and involves computationally intensive training procedures.
Training DiT-XL/2 on 256$\times$256 images demands 950 V100 GPU days (7
million iterations), and for 512$\times$512 images, it requires 1733 V100 GPU
days (3 million iterations) Xie et al. (2023). Relative to the image
generation of general categories, fine-grained image generation poses a
greater challenge, demanding models to intricately model subtle differences
(e.g., feather texture of birds) in highly similar fine-grained categories.
Therefore, training diffusion models for large-scale fine-grained class-
conditional image generation from scratch requires greater computational
resources and training iterations. We intend to develop an inexpensive method
to fine-tune large pre-trained class-conditional diffusion models (e.g., DiT
models) efficiently for large-scale fine-grained image generation.

Recent advancements in diffusion models, e.g., DiffFit Xie et al. (2023), have
demonstrated that satisfactory performance on downstream tasks can be attained
by fine-tuning a small subset of parameters of the pre-trained model. In this
paper, we further explore efficient fine-tuning strategies extending to large-
scale fine-grained image generation tasks. We propose a novel fine-tuning
method named FineDiffusion, capable of generating fine-grained images in
downstream tasks by fine-tuning only the proposed TieredEmbedder, bias terms,
and the scaling and offset terms of normalization layers. The proposed
TieredEmbedder considers hierarchical class labels, including superclasses and
subclasses, which are underutilized in image generation based on the diffusion
model. Considering that large-scale fine-grained categories comprise various
objects with noticeable appearance differences, knowing the superclass to
which each class belongs can provide additional assistance for high-difficulty
fine-grained image generation. Moreover, we introduce a novel sampling method
as an alternative to the conventional classifier-free guidance sampling for
fine-grained image generation. Specifically, we combine noise estimation from
the subclass-conditioned diffusion model and the jointly trained superclass-
conditioned diffusion model. The guidance provided by the mixing weight
facilitates the image generation toward specific subclasses within the
superclass. By leveraging hierarchical category embedding information and
employing fine-grained classifier-free guidance sampling, our FineDiffusion
achieves finer control compared to unconditional diffusion guidance. It
captures distribution differences among samples from various subclasses within
the same superclass. The results of our fine-grained classifier-free guided
sampling indicate that the diffusion model incorporating hierarchical
information significantly improves the quality and diversity of generated
images. To the best of our knowledge, our work is the pioneering effort in
utilizing diffusion-based techniques for fine-grained image generation
encompassing 10,000 classes. In performance metrics, FineDiffusion achieves an
FID of 9.776 and an LPIPS of 0.721, surpassing the baseline algorithms
substantially.

We summarize our contributions as follows:

  * •

We introduce a novel parameter-efficient fine-tuning method called
FineDiffusion for fine-grained image generation. We first propose a
hierarchical class embedder called TieredEmbedder, which models the data
distribution of both superclass and subclass samples. By only fine-tuning
proposed TieredEmbedder, bias terms, and the scaling and offset terms of
normalization layers, our method significantly speeds up training while
reducing model storage overhead (as shown in Figure 2 and Table 2).

  * •

We introduce a fine-grained classifier-free guidance sampling method that
enhances control over fine-grained image generation through conditional
information from subclasses and superclasses.

  * •

Experimental results demonstrate that FineDiffusion outperforms state-of-the-
art parameter-efficient fine-tuning methods in fine-grained image generation
in terms of FID and LPIPS scores.

Figure 2: The overall FID score is computed for the fine-tuned DiT model in
the iNaturalist 2021 mini dataset. The size of each data point corresponds to
the duration of training, with smaller points indicating faster training
speed. FineDiffusion exhibits outstanding performance by attaining superior
FID results, all the while demanding a reduced computational workload and
fewer parameters.

##  2 Related Work

###  2.1 Diffusion Models

Diffusion models Sohl-Dickstein et al. (2015); Ramesh et al. (2022); Saharia
et al. (2022); Li et al. (2022); He et al. (2024) have become the most
prevalent generative models, offering more stable training and generating
higher-quality visual content compared to earlier mainstream models such as
GANs Goodfellow et al. (2014), VAEs Kingma and Welling (2014), and Flow-based
models Dinh et al. (2015). Denoising diffusion probabilistic models (DDPMs) Ho
et al. (2020) architecture, based on U-Net Ronneberger et al. (2015), has
propelled the diffusion model to significant success in image generation.
Following this, improvements in sampling techniques have further enhanced
DDPM’s performance Song et al. (2020); Ho and Salimans (2022, 2022).
Classifier-free guidance Ho and Salimans (2022) proposes a diffusion guidance
method without additional classifiers by jointly training conditional and
unconditional diffusion models. For its impressive image generation results,
numbers of models have adopted classifier-free guidance, making significant
strides in the image generation field Nichol et al. (2021); Ramesh et al.
(2022); Gafni et al. (2022); Mokady et al. (2023). However, classifier-free
guidance inadequately considers the correlations among different classes in
fine-grained fields. In light of this, we explore a fine-grained classifier-
free guidance approach that effectively utilizes the label hierarchy structure
to improve fine-grained image generation. Additionally, recent works Bao et
al. (2023); Peebles and Xie (2023) have explored the integration of
Transformer architectures into diffusion models. DiT models Peebles and Xie
(2023) have garnered significant attention due to their superior scalability
and simplicity, inspiring numerous subsequent works Xie et al. (2023); Mo et
al. (2024); He et al. (2023); Lu et al. (2024). We follow the DiT Peebles and
Xie (2023) paradigm and further fine-tune the pre-trained models for fine-
grained image generation.

###  2.2 Parameter-efficient Fine-tuning

The traditional full fine-tuning approach Howard and Ruder (2018); Dai and Le
(2015), which requires tuning all parameters, is resource-intensive and time-
consuming. To solve this issue, parameter-efficient fine-tuning has gained
momentum. Some researches introduce additional parameters, such as plug-and-
play modules or the incorporation of side network structures, for fine-tuning
the small subset of parameters introduced additionally Houlsby et al. (2019);
Hu et al. (2021); Chen et al. (2022); He et al. (2022). On the flip side, some
researches have seamlessly extended successful prompt tuning techniques from
the field of NLP (Natural Language Processing) to CV (Computer Vision) Zhou et
al. (2022); Yao et al. (2021); Jia et al. (2022); Rao et al. (2022). Other
researches explore the fine-tuning of only crucial parameters within the model
to enhance the efficiency of the fine-tuning process Zaken et al. (2021); Zhao
et al. (2020); Xu et al. (2021). There are also hybrid fine-tuning methods,
e.g., DiffFit Xie et al. (2023), which fine-tune only specific parameters
related to bias, class embedding, normalization, and additional introduced
scale factors. However, DiffFit is a general image generation model fine-
tuning strategy lacking specific enhancements tailored for fine-grained image
generation. In contrast, our approach possesses specific innovations in this
regard.

##  3 Methodology

Figure 3: The proposed FineDiffusion method involves the preservation of the
majority of parameters within the pre-trained DiT model. We introduce a
specialized TieredEmbedder optimized for generating fine-grained categories.
Notably, we exclusively fine-tune the tiered label embedding component, bias
terms, and normalization terms, and this fine-tuning process affects merely
1.77% of the pre-trained model’s parameters. This strategic approach showcases
an effective means of achieving efficient parameter fine-tuning. (Zoom-in for
the best view.)

###  3.1 Preliminaries

##### Diffusion Models.

Diffusion models are currently one of the most mainstream generative models,
consisting of two processes: forward noise injection and backward denoising.
In the forward process, Gaussian noise is added to the original image $x_{0}$
to produce the noisy image $x_{t}$, where $t\sim[1,T]$. The image $x_{t}$ is
obtained by the formula
$x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\text{
where }\epsilon\sim\mathcal{N}(0,\mathbf{I})\text{ and
}\bar{\alpha}_{t}=\prod_{t=1}^{t}\alpha_{t}$, $\alpha_{t}$ is the pre-defined
hyperparameter related to the time step t. The backward process is defined as
a Markov chain which beginning with $x_{T}\sim\mathcal{N}(0,\mathbf{I})$,
$x_{t-1}$ can be determined by $x_{t}$ according to the following formula:
$p_{\theta}\left(x_{t-1}\mid
x_{t}\right)=\mathcal{N}\left(x_{t-1};\mu_{\theta}\left(x_{t},t\right),\sigma_{t}^{2}\mathbf{I}\right)$.
After T iterations of denoising, the clean image $x_{0}$ is obtained.

##### Diffusion Transformers (DiT).

DiT Peebles and Xie (2023) is a recent notable approach that formulates a
diffusion model using Transformer architecture. Meanwhile, it also adopts the
idea of Latent Diffusion Models (LDM) Rombach et al. (2022) to perform image
denoising within the latent space. DiT consists of two parts for image
generation given sample $x$: (1) Comprising an encoder ($E$) and a decoder
($D$), the Variational Autoencoder (VAE) operates such that the latent code is
represented as $z=E(x)$, and the reconstructed data is denoted as
$\hat{x}=D(z)$. (2) A latent diffusion transformer incorporating patchify,
sequential DiT blocks, and depatchify operations. The combination of embedded
labels and iteration time step embeddings $t$ serves as a condition for the
training of the DiT model, with the latent used as input.

##### Classifier-free Guidance (CFG).

CFG Ho and Salimans (2022) is a sampling strategy independent of the
classifier. Distinct from the classifier-guidance strategy, it significantly
reduces training complexity while ensuring high-quality image generation.
During training, conditional and unconditional diffusion models are jointly
trained, being parameterized using a score estimator $\epsilon_{\theta}$.
During sampling, for the given condition $c$, the conditional diffusion models
follow the formula: $\hat{\epsilon}_{\theta}\left(x_{t}\mid
c\right)=\epsilon_{\theta}\left(x_{t}\mid\varnothing\right)+\omega\cdot\left(\epsilon_{\theta}\left(x_{t}\mid
c\right)-\epsilon_{\theta}\left(x_{t}\mid\varnothing\right)\right)$, where
$\omega$ is the guidance scale to adjust the condition-relatedness of the
results, and the null label $\varnothing$ input denotes the estimation of the
unconditional model. CFG has been widely applied in diffusion models Nichol et
al. (2021); Ramesh et al. (2022); Peebles and Xie (2023) because of the
exceptional performance.

###  3.2 FineDiffusion

####  3.2.1 3.2.1 Parameter-efficient Fine-tuning

In this part, we provide technical details on how FineDiffusion fine-tunes the
DiT model for fine-grained image generation. The overview of our FineDiffusion
is depicted in Figure 3. In the DiT model, the label embedding segment encodes
class labels to be utilized as conditions for class-conditional image
generation. For image generation of a specific class, the corresponding class
embedding vector is concatenated with the noised latent image, serving as the
input for the sampling process. For fine-grained image generation, the
inclusion of superclass information facilitates a more precise capture of
category-specific features, thereby enhancing the overall quality of generated
images. We propose a hierarchical class label encoding strategy that
concurrently encodes superclass and subclass labels. In the DiT model, the
label embedding segment accepts class labels as input and outputs a table with
fixed-dimensional embedding vectors. We introduce embeddings representing the
superclass into the label embedding segment and train superclass embeddings
together with other subclass embeddings, to minimize parameter introduction.
And the novel embedder module responsible for encoding hierarchical class
labels is named TieredEmbedder. After training the proposed TieredEmbedder,
both superclass and subclass labels are input to the diffusion model for
guiding fine-grained image generation. To further improve fine-tuning
effectiveness, we consider adjusting parameters beyond the TieredEmbedder.
Inspired by DiffFit Xie et al. (2023), we simultaneously fine-tune bias and
normalization terms along with the tiered embedder to learn the global
dataset’s distribution characteristics. Through these designs, our proposed
FineDiffusion aims to deepen the model’s understanding of image content. By
integrating higher-level semantic structures and learning dataset-wide,
superclass, and subclass information, it enhances the conditional generation
images’ quality.

During fine-tuning, the majority of parameters in the pre-trained DiT model
are are frozen, and updates are applied only to TieredEmbedder, bias terms,
and normalization terms. FineDiffusion requires fine-tuning only 1.77% of the
parameters in the DiT-XL/2 pre-trained model, achieving a training speed-up of
1.56$\times$ compared to full fine-tuning. Simultaneously, as an efficient
parameter fine-tuning method, the proposed approach enhances adaptability for
fine-grained image generation tasks while effectively mitigating the risk of
catastrophic forgetting. For each dataset fine-tuning, it only needs to store
the small subset of parameters undergoing fine-tuning, which dramatically
reducing the model’s storage requirements.

####  3.2.2 3.2.2 Fine-grained Classifier-free Guidance

Classifier-free guidance has supplanted classifier guided sampling to achieve
a trade-off between the diversity and quality of sampling. While reducing
training complexity, this approach secures high-quality image generation.
Currently, classifier-free guidance is widely utilized in various diffusion
models. We attempt to introduce hierarchical category label information
because knowing which superclass each category belongs to is helpful for
capturing intricate details and enhancing the overall quality of the generated
images. Therefore, we introduce a fine-grained classifier-free guidance
sampling method, yielding improved results under specific conditions.
Integrated with TieredEmbedder, this approach leverages superclass conditional
information to enhance control over generated images. Conventional classifier-
free guidance involves training both conditional and unconditional generative
models. In our approach, a superclass conditional generative model is
introduced to replace the unconditional model. During training, we use the
same neural network to parameterize both the conditional model and the
superclass conditional model, leveraging a score estimator $\epsilon_{\theta}$
for optimization guidance. Subclass labels $c$ are probabilistically replaced
with their corresponding superclass labels $c_{p}$ to achieve the learning of
superclass embeddings. During sampling, trained superclass and subclass
embeddings are input into the DiT network to obtain predictive noise. Our
method combines conditional scores estimation
$\epsilon_{\theta}\left(x_{t}\mid c\right)$ and superclass conditional scores
estimation $\epsilon_{\theta}\left(x_{t}\mid c_{p}\right)$ to guide specific
category image generation, with the specific formula provided below:

$\hat{\epsilon}_{\theta}\left(x_{t}\mid
c\right)=\epsilon_{\theta}\left(x_{t}\mid
c_{p}\right)+\omega\cdot\left(\epsilon_{\theta}\left(x_{t}\mid
c\right)-\epsilon_{\theta}\left(x_{t}\mid c_{p}\right)\right),$

(1)

where $\omega$ is the guidance scale, and $\hat{\epsilon}_{\theta}$ is the
modified score.

By thoroughly exploiting the distributional characteristics of superclasses
and subclasses, our sampling method achieves a finer control than classifier-
free guidance. We can capture commonalities among subclasses within the same
superclass and simultaneously capture distinct distributions between different
superclasses, thereby making the learned distributions of each category more
closely approximate real-world situations.

| Plants | Insects | Birds | Fungi | Reptiles | Mammals
---|---|---|---|---|---|---
Method | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$
Full Fine-tuning | 13.200 | 0.671 | 18.477 | 0.733 | 21.554 | 0.720 | 27.259 | 0.657 | 37.828 | 0.686 | 38.061 | 0.715
BitFit | 12.025 | 0.670 | 19.328 | 0.740 | 23.677 | 0.723 | 29.589 | 0.662 | 41.845 | 0.683 | 39.918 | 0.714
DiffFit | 12.166 | 0.668 | 19.809 | 0.738 | 24.063 | 0.721 | 30.000 | 0.659 | 42.657 | 0.688 | 40.243 | 0.716
FineDiffusion(ours) | 15.752 | 0.693 | 13.871 | 0.741 | 14.713 | 0.727 | 26.805 | 0.695 | 25.266 | 0.709 | 34.567 | 0.716
| Ray-finned Fishes | Amphibians | Mollusks | Arachnids | Animalia | Overall
Method | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$ | FID$\downarrow$ | LPIPS$\uparrow$
Full Fine-tuning | 35.613 | 0.702 | 55.077 | 0.713 | 44.075 | 0.717 | 45.722 | 0.732 | 50.038 | 0.698 | 10.662 | 0.708
BitFit | 41.265 | 0.706 | 66.261 | 0.714 | 44.594 | 0.725 | 45.287 | 0.744 | 51.904 | 0.700 | 10.586 | 0.710
DiffFit | 44.340 | 0.698 | 68.030 | 0.714 | 44.845 | 0.731 | 45.865 | 0.742 | 52.059 | 0.697 | 10.788 | 0.709
FineDiffusion(ours) | 30.348 | 0.716 | 33.827 | 0.717 | 38.590 | 0.735 | 33.214 | 0.751 | 47.353 | 0.704 | 9.776 | 0.721

Table 1: FID and LPIPS metrics comparison on with existing methods full fine-tuning, BitFit Zaken et al. (2021) and DiffFit Xie et al. (2023) on 11 categories of iNaturalist 2021 mini dataset image synthesis. Method | Params.(M) |  | Training Cost
---
(GPU days)
| Training
---
Time
Full Fine-tuning | 685.2(100%) | 15.11 | 1$\times$
BitFit | 12.01(1.75%) | 9.59 | 0.635$\times$
DiffFit | 12.14(1.77%) | 9.65 | 0.638$\times$
FineDiffusion(ours) | 12.15(1.77%) | 9.69 | 0.642$\times$
Table 2: Comparison of the number of parameters and training overhead on iNaturalist 2021 mini image synthesis. Method |  FID$\downarrow$ |  LPIPS $\uparrow$
---|---|---
FineDiffusion-256 |  9.776 |  0.721
FineDiffusion-512 |  9.490 |  0.723
Table 3: Comparison of the overall FID and overall LPIPS metrics for FineDiffusion fine-tuning on the 256$\times$256 and 512$\times$512 resolutions of the DiT/XL-2 model . Method |  FID$\downarrow$ |  LPIPS $\uparrow$
---|---|---
Full Fine-tuning |  13.034 |  0.651
BitFit |  15.022 |  0.654
DiffFit |  15.068 |  0.653
FineDiffusion(ours) |  12.382 |  0.667
Table 4: Comparison of the overall FID and overall LPIPS metrics on VegFru
dataset. Figure 4: Images generated by the FineDiffusion model after fine-
tuning on the VegFru dataset. In each row, every pair of images belongs to the
same superclass, namely: Melon, Eggplant, Citrus fruit, Litchies, Berry fruit,
Drupe, Green-leafy vegetable, Wild vegetable and Collective fruit. Our
approach effectively generates subclasses with visually similar features
within the same superclass.

##  4 Experiments

###  4.1 Datasets

iNaturalist 2021. The iNaturalist dataset Van Horn et al. (2018), in its 2021
version, represents a comprehensive real-world collection featuring 10,000
distinct species across 11 supercategories. In our experiments, we employed
the mini version of the iNaturalist 2021 training dataset, consisting of
500,000 training samples. The official validation set, containing 100,000
images, is utilized for metric testing.

VegFru. The VegFru dataset Hou et al. (2017) comprises 292 fine-grained
categories of vegetables and fruits, organized into 25 superclasses. Its
training set contains 29,200 images, and the test set contains 14,600 images.
We utilize the dataset splits provided by the official source.

Figure 5: Comparison of generated results for several mammalian categories
across different methods. Each method is assigned the same class label input
and the same random seed for sampling. The authentic images of real species
are also presented. This comparison underscores FineDiffusion’s capability to
generate species images aligned with actual categories, and these images are
more photorealistic compared to other methods. (Zoom-in for the best view.)

###  4.2 Implementation Details

We utilize the pre-trained DiT XL/2 model as the backbone of our model,
_i.e.,_ the model size is XL, patchified with the size of 2. The original DiT
model is trained on ImageNet-1k 256$\times$256\. The fine-tuning training
process involves leveraging V100 GPUs, training 200K iterations on the
iNaturalist 2021 mini training set, and training 100K iterations on the VegFru
dataset. The model is optimized with AdamW Loshchilov and Hutter (2017)
optimizer and the learning rate is 1e-4. The sampling procedure adopts the
classifier-free guidance with a scale of 4.0 and uses 250 steps for the
sampling process. When examining the overall results, the generated images are
based on randomly selected sample seeds. To evaluate the convergence speed of
each algorithm, a uniform set of fixed sample seeds was employed across
distinct algorithms, ensuring controlled variables.

###  4.3 Evalution Metrics

We utilize the Fréchet Inception Distance (FID) Heusel et al. (2017) and
Learned Perceptual Image Patch Similarity (LPIPS) Zhang et al. (2018) metrics
to assess the model’s performance. FID quantifies the differences in mean and
covariance of features between real and generated images in the feature space,
and it reflects their similarity through traces. A smaller FID value indicates
a higher degree of similarity between the two image sets. LPIPS is primarily
employed to measure the similarity between image samples, with higher values
indicating greater perceptual dissimilarity between samples. This implies that
the generated fine-grained images exhibit significant diversity, which
correlates with better performance.

Figure 6: Comparison with stable diffusion in the generation of various fine-
grained bird categories. The first two columns depict real images, while the
images in the subsequent columns, organized by rows, showcase generated images
for each fine-grained category. FineDiffusion achieves superior accuracy in
generating avian species that closely resemble those depicted in real images.
(Zoom-in for the best view.) Figure 7: The t-SNE visualization of label
embedding distributions obtained from different fine-tuning methods. Diverse
colors represent unique superclasses within iNaturalist 2021, where
“$\bullet$” represents subclasses in each superclass. “$\star$” represents the
average embeddings of subclasses belonging to the same superclass, while the “

+

” represents the embeddings of the superclass. (Zoom-in for the best view.)

###  4.4 Quantitative Comparison

Results on iNaturalist. The outcomes are presented in Tables 1, 2 and 3,
accompanied by generated samples illustrated in Figure 1. Through quantitative
comparison, we evaluate our algorithm alongside three comparative algorithms:
full fine-tuning, BitFit (fine-tuning only the bias terms), and DiffFit (fine-
tuning bias terms, normalization layers, and additional scaling terms). In the
generation of 11 super categories, the proposed FineDiffusion fine-tuning
based on 256$\times$256 resolution model achieved an exceptional overall FID
of 9.776 with only 1.77% parameter fine-tuning, as shown in Table 1 and 2.
Significantly, FineDiffusion exhibits superior performance, achieving the best
FID scores in 10 out of the 11 categories. In comparison, three baseline
algorithms exhibit similar FID scores. The full fine-tuning approach shows
relatively good FID performance in each category, demonstrating its robust
performance as a baseline method. However, it’s important to note that full
fine-tuning with 100% parameters leads to significant computational resource
wastage and large storage cost for more downstream tasks. BitFit obtained the
second-best overall FID, trailing behind FineDiffusion, through 1.75%
parameter fine-tuning, showing its parameter efficiency. In contrast, DiffFit
exhibited subpar performance compared to several algorithms. This could be
attributed to DiffFit being more suited for generating images of generalized
categories, or there might be a need to explore parameter settings more
compatible with DiffFit. The exceptional FID scores of the FineDiffusion model
highlight the efficacy of fine-tuning tiered embeddings and guided sampling
utilizing superclass information. This strategy effectively preserves the
knowledge acquired by the pre-trained model, facilitating successful transfer
to diverse fine-grained species. In terms of the LPIPS metric, the
FineDiffusion model consistently outperforms other models in all categories,
as well as maintaining the highest overall LPIPS score. This suggests that our
algorithm not only enhances the quality of fine-grained images generation but
also exhibits a high level of diversity.

Table 2 demonstrates that the FineDiffusion model requires only fine-tuning a
minimal amount of parameters, effectively conserving model storage space. It
fine-tunes only a few more parameters than other efficient fine-tuning
algorithms, while resulting in a significant improvement in the generated
results. This not only reduces storage costs but also accelerates training
speed, making it more resource-efficient compared to full fine-tuning,
achieving an astonishing 1.56$\times$ training speed-up. The accelerated
training speed of FineDiffusion bears substantial implications for practical
applications. Encouraged by the stellar performance of FineDiffusion on the
256$\times$256 resolution DiT model, we proceeded to fine-tune a
512$\times$512 resolution DiT/XL-2 model to acquire images of heightened
quality. The experimental results are shown in Table 3, revealing commendable
FID and LPIPS values across various categories. The aggregate FID reach a
noteworthy 9.490, while the LPIPS score attain 0.723. The 512-resolution model
demonstrates a further enhancement over the performance of the 256-resolution
model.

Results on VegFru. Additionally, we assess our method on the VegFru dataset,
and the results are presented in Table 4, accompanied by generated samples in
Figure 4. For brevity, we offer overall results instead of an exhaustive list
of categories of 25 superclasses. Notably, FineDiffusion surpasses all
counterparts with a markedly FID of 12.382 and LPIPS of 0.667, suggesting
superior performance in capturing image distribution nuances. In terms of FID,
it achieves a reduction of approximately 5.0% compared to full fine-tuning,
17.6% compared to BitFit, and 17.8% compared to DiffFit. This indicates the
stable applicability of our methodology across diverse fine-grained image
scenarios, offering valuable supplementation to the experiments conducted on
the iNaturalist dataset.

###  4.5 Qualitative Comparison

In Figure 5, we present the results of mammalian class generation using the
FineDiffusion method and juxtapose them with several baseline fine-tuning
algorithms. In comparison to other methods, our model excels in accurately
capturing the inherent features of each species, producing life-like
depictions of animals with realistic lighting and shadow effects. The
resulting images exhibit a pronounced sense of authenticity and exceptional
quality. Significantly, it is noteworthy that in contrast to other algorithms
which entail more extensive parameter fine-tuning, our approach overcomes
problems such as morphological distortions or image blurriness. This
observation implies that our fine-tuning approach maximally retains the
valuable information of the pre-trained model, and efficiently adapts to fine-
grained generation tasks, yielding visually coherent and accurate generative
outcomes.

We also qualitatively compare with state-of-the-art text-guided image
generation models, e.g., Stable Diffusion Rombach et al. (2022). For our
approach, images are generated based on category labels, whereas Stable
Diffusion generates images using the prompt phrase “a photo of xxx” (where
“xxx” indicates the species name). The comparison results are shown in Figure
6, revealing that FineDiffusion excels in generating accurate images for
specific fine-grained species. In contrast, images generated by Stable
Diffusion exhibit a certain degree of similarity under various fine-grained
category prompts.

###  4.6 Visualization of Class Embeddings

We utilize the t-SNE Van der Maaten and Hinton (2008) technique to visualize
high-dimensional label embedding vectors in a two-dimensional context.
Employing eleven distinct colors, we portrayed 10,000 classes in iNaturalist,
revealing the distribution of embeddings across eleven superclasses, as shown
in Figure 7. Examining the class distributions, we observe that in the
FineDiffusion approach, points within the same superclass tend to cluster
together, while points from different superclasses remain separate. The points
within each superclass are relatively dispersed, and the majority of mean
points are positioned at the center of their respective classes. The
embeddings of superclasses also tend to lie among their respective subclasses,
which indicates the effectiveness of superclass embedding learning. In
contrast, the full fine-tuning method shows less separation among superclass
embeddings, with points from different superclasses mixing. Comparatively,
BitFit and DiffFit effectively distinguish superclasses, while there is some
clustering among points within the same superclass, which is not as well-
dispersed as FineDiffusion. The t-SNE results indicate the superior fine-
tuning effectiveness of FineDiffusion on label embeddings, enhancing
discriminability among distinct classes and even within subclasses of the same
supercategory. This is crucial for generating fine-grained images across a
large scale of 10,000 classes.

##  5 Conclusion

In this work, we make the first attempt to scale up diffusion models for fine-
grained image generation with 10,000 classes. We introduce FineDiffusion, an
efficient parameter-tuning approach that fine-tunes key components, including
tiered label embeddings, bias terms, and normalization terms of pre-trained
models. Our method substantially reduces both training and storage overheads.
Furthermore, we introduce a fine-grained classifier-free guidance sampling
technique, harnessing hierarchical data label information to effectively
enhance the performance of fine-grained image generation. Adequate qualitative
and quantitative results demonstrate the superiority of our method compared to
the other methods.

## References

  * Bao et al. [2023] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu.  All are worth words: A vit backbone for diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22669–22679, June 2023.
  * Chen et al. [2022] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  Advances in Neural Information Processing Systems, 35:16664–16678, 2022.
  * Dai and Le [2015] Andrew M Dai and Quoc V Le.  Semi-supervised sequence learning.  Advances in neural information processing systems, 28, 2015.
  * Dinh et al. [2015] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  In International Conference on Learning Representations (ICLR), 2015.
  * Gafni et al. [2022] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In European Conference on Computer Vision, pages 89–106. Springer, 2022.
  * Gao et al. [2023] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan.  Masked diffusion transformer is a strong image synthesizer.  arXiv preprint arXiv:2303.14389, 2023.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * He et al. [2022] Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao.  Sparseadapter: An easy approach for improving the parameter-efficiency of adapters.  arXiv preprint arXiv:2210.04284, 2022.
  * He et al. [2023] Feihong He, Gang Li, Lingyu Si, Leilei Yan, Shimeng Hou, Hongwei Dong, and Fanzhang Li.  Cartoondiff: Training-free cartoon image generation with diffusion transformer models.  arXiv preprint arXiv:2309.08251, 2023.
  * He et al. [2024] Feihong He, Gang Li, Mengyuan Zhang, Leilei Yan, Lingyu Si, and Fanzhang Li.  Freestyle: Free lunch for text-guided style transfer using diffusion models.  arXiv preprint arXiv:2401.15636, 2024.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * Hou et al. [2017] Saihui Hou, Yushan Feng, and Zilei Wang.  Vegfru: A domain-specific dataset for fine-grained visual categorization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 541–549, 2017.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * Howard and Ruder [2018] Jeremy Howard and Sebastian Ruder.  Universal language model fine-tuning for text classification.  arXiv preprint arXiv:1801.06146, 2018.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In European Conference on Computer Vision, pages 709–727. Springer, 2022.
  * Kim et al. [2022] Dongjun Kim, Yeongmin Kim, Wanmo Kang, and Il-Chul Moon.  Refining generative process with discriminator guidance in score-based diffusion models.  arXiv preprint arXiv:2211.17091, 2022.
  * Kingma and Welling [2014] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In International Conference on Learning Representations (ICLR), 2014.
  * Li et al. [2022] Gang Li, Heliang Zheng, Chaoyue Wang, Chang Li, Changwen Zheng, and Dacheng Tao.  3ddesigner: Towards photorealistic 3d object generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2211.14108, 2022.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * Lu et al. [2024] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai.  Fit: Flexible vision transformer for diffusion model.  arXiv preprint arXiv:2402.12376, 2024.
  * Mo et al. [2024] Shentong Mo, Enze Xie, Ruihang Chu, Lanqing Hong, Matthias Niessner, and Zhenguo Li.  Dit-3d: Exploring plain diffusion transformers for 3d shape generation.  Advances in Neural Information Processing Systems, 36, 2024.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * Rao et al. [2022] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082–18091, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265, 2015.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * Van Horn et al. [2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  * Xie et al. [2023] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li.  Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning.  arXiv preprint arXiv:2304.06648, 2023.
  * Xu et al. [2021] Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang.  Raise a child in large language model: Towards effective and generalizable fine-tuning.  arXiv preprint arXiv:2109.05687, 2021.
  * Yao et al. [2021] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  Cpt: Colorful prompt tuning for pre-trained vision-language models.  arXiv preprint arXiv:2109.11797, 2021.
  * Zaken et al. [2021] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  arXiv preprint arXiv:2106.10199, 2021.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * Zhao et al. [2020] Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze.  Masking as an efficient alternative to finetuning for pretrained language models.  arXiv preprint arXiv:2004.12406, 2020.
  * Zhou et al. [2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, 2022.