# Norm-guided latent space exploration for text-to-image generation

Dvir Samuel1,2, Rami Ben-Ari2, Nir Darshan2, Haggai Maron3,4, Gal Chechik1,4
1Bar-Ilan University, Ramat-Gan, Israel
2OriginAI, Tel-Aviv, Israel
3Technion, Haifa, Israel
4NVIDIA Research, Tel-Aviv, Israel
Correspondence to: Dvir Samuel<EMAIL_ADDRESS>
###### Abstract

Text-to-image diffusion models show great potential in synthesizing a large
variety of concepts in new compositions and scenarios. However, the latent
space of initial seeds is still not well understood and its structure was
shown to impact the generation of various concepts. Specifically, simple
operations like interpolation and finding the centroid of a set of seeds
perform poorly when using standard Euclidean or spherical metrics in the
latent space. This paper makes the observation that, in current training
procedures, diffusion models observed inputs with a narrow range of norm
values. This has strong implications for methods that rely on seed
manipulation for image generation, with applications to few-shot and long-tail
learning tasks. To address this issue, we propose a novel method for
interpolating between two seeds and demonstrate that it defines a new non-
Euclidean metric that takes into account a norm-based prior on seeds. We
describe a simple yet efficient algorithm for approximating this interpolation
procedure and use it to further define centroids in the latent seed space. We
show that our new interpolation and centroid techniques significantly enhance
the generation of rare concept images. This further leads to state-of-the-art
performance on few-shot and long-tail benchmarks, improving prior approaches
in terms of generation speed, image quality, and semantic content.

##  1 Introduction

Text-to-image diffusion models demonstrate an unprecedented ability to
generate new and unique images. They map random samples (seeds) from a high-
dimensional space, conditioned on a user-provided text prompt, to a
corresponding image. Unfortunately, the seed space, and the way diffusion
models map it into the space of natural images are still poorly understood,
directly affecting generation quality. For example, current diffusion models
have difficulty generating images of rare concepts, but performing
optimization in the seed space can alleviate this issue [51]. Our limited
understanding of the seed space is further demonstrated by the fact that
standard operations on seeds, such as interpolating between two seeds or
finding the centroid of a given set of seeds, often result in low-quality
images with poor semantic content (Figure 1 left, first two rows). As a
result, methods based on the exploration and manipulation of seed spaces face
a considerable challenge.

The aim of this paper is to propose simple and efficient tools for exploring
the seed space and to demonstrate how these tools can be used to generate rare
concepts. Our main observation is that a specific property, the norm of the
seed vector, plays a key role in how a seed is processed by the diffusion
model. In more concrete terms, since seeds are sampled from a multidimensional
Gaussian distribution, the norm of the seeds has a $\chi$ distribution. For
high dimensional Gaussian distributions, such as the ones used by diffusion
models, the $\chi$ distribution is strongly concentrated around a specific
positive number. Consequently, diffusion models tend to favor inputs with this
norm, resulting in lower-quality images when the norm of the seed differs from
that value.

To address this issue, we propose to use a prior distribution over the norms
in the seed space based on the $\chi$ distribution to guide exploration. While
prior-based exploration techniques for seed spaces have been proposed before
[4, 2], the advantage of our prior is that it does not rely upon an expensive
estimation of the empirical data distribution nor on complex computations and
can be applied to very high-dimensional latent spaces. Yet, as we show below,
our prior still significantly improves exploration techniques in the seed
space.

As a first step, we propose a novel method for interpolating between two
seeds. In contrast to Linear Interpolation (LERP) or Spherical Linear
Interpolation (SLERP) [56], we formulate this problem as finding a likelihood-
maximizing path in seed space according to the aforementioned prior. In
addition to providing us with an interpolating path, we also demonstrate that
the optimal value of this optimization problem defines a new non-Euclidean
metric structure over the seed space. Figure 1 compares our interpolation
paths to two other frequently used interpolation methods in 2D and in image
space. The improvement of the image quality along the path is evident.
Specifically, the 2D example (right panel) illustrates that LERP and SLERP
paths cross low-probability areas whereas our path maintains a high
probability throughout. The same phenomenon is shown for images (left panel)
where it is apparent that intermediate points in the paths generated by the
baseline methods have a significantly lower quality.

As a next step, we build on our newly defined metric to define a generalized
centroid for a set of seeds. We extend the standard definition … and define
the standard definition of the centroid in Euclidean spaces, and define the
centroid as the point that minimizes the distances to the seeds according to
the new distance function (also known as the Fréchet mean for that given
metric). We show how to discretize the two optimization problems above and
solve them using a simple and efficient optimization scheme. We call our
approach NAO for Norm-Aware Optimization.

We evaluate NAO extensively. First, we directly assess the quality of images
generated by our methods, showing higher quality and better semantic content.
Second, we use our seed space interpolation and centroid finding methods in
two tasks: (1) Generating images of rare concepts, and (2) Augmenting semantic
data for few-shot classification and long-tail learning. For these tasks, our
experiments indicate that seed initialization with our prior-guided approach
improves SoTA performance and at the same time has a significantly shorter
running time (up to X10 faster) compared to other approaches.

Figure 1: The interplay between interpolation and input distribution. (Left)
Visual comparison of three interpolation methods between a pair of seeds in
the high-dimensional space of Stable Diffusion [46]. Interpolation paths
computed by linear (LERP) and spherical SLERP [56] interpolation produce seeds
that result in images that are flat or just noise, whereas our approach is
capable of generating images that are meaningful (quantified in Table 1).
(Right) Paths found using linear (LERP), spherical (SLERP), and likelihood-
based interpolation methods in 2D space, where the norm of samples has a
$\chi$ distribution. Colors denote the log of the probability distribution
function. The interpolation path computed by NAO traverses areas that have the
desired norm (yellow, high likelihood).

##  2 Related Work

Text-guided diffusion models. Text-guided diffusion models involve mapping
random seed (noise) $z_{T}$ and textual condition $P$ to an output image
$z_{0}$ through a denoising process [43, 49, 7]. The inversion of this process
can be achieved using a deterministic scheduler (e.g. DDIM [58]), allowing for
the recovery of the latent code $z_{T}$ from a given image $z_{0}$. See a
detailed overview in the supplemental.

Rare concept generation with text-to-image models. Diffusion models excel in
text-to-image generation [43, 49, 7], but struggle with rare fine-grained
objects (e.g.payphone or tiger-cat in StableDiffusion [46]) and compositions
(e.g.shaking hands) [51, 34]. Techniques like pre-trained image classifiers
and text-driven gradients have been proposed to improve alignment with text
prompts, but they require pre-trained classifiers or extensive prompt
engineering [18, 27, 40, 49, 35, 39, 68]. Other approaches using segmentation
maps, scene graphs, or strengthening cross-attention units also face
challenges with generating rare objects [5, 21, 75, 20, 10]. SeedSelect [51]
is a recent approach that optimizes seeds in the noise space to generate rare
concepts. However, it suffers from computational limitations and long
generation times. This paper aims to address these limitations by developing
efficient methods that significantly reduce generation time while improving
the quality of generated images.

Latent space interpolation. Interpolation is a well-studied topic in computer
graphics [60, 24, 57]. Linear Interpolation (LERP) is commonly used for smooth
transitions by interpolating between two points in a straight line. Spherical
Linear Interpolation (SLERP) [56], on the other hand, offers computing
interpolation along the arc of a unit sphere, resulting in smoother
transitions along curved paths. Image interpolations in generative models are
obtained by three main approaches: (1) Linear or spherical interpolation
between two latent vectors [1, 55, 44, 77], (2) Image-to-Image translation
approaches [37] and (3) Learning an interpolation funciton or metrics based on
the data [11, 55, 30, 4]. [3] observed that linearly traveling a normally
distributed latent space leads to sub-optimal results and proposed an
interpolation based on a Riemannian metric. [2] further proposed a methodology
to approximate the induced Riemannian metric in the latent space with a
locally conformally flat surrogate metric that is based on a learnable prior.
Note, that as opposed to our approach, these priors do not have a closed-form
solution, they work on a relatively low dimensional latent space of a VAE
(constrained and compact latent space), and they learn the metric from the
data itself. In this paper, we do not assume any of the above. We introduce a
novel interpolation approach that effectively use the inherent structure of
the latent space to achieve correct interpolation without any additional data,
on the high-dimensional seed space of a diffusion model.

Data Augmentation via Latent Space Exploration. Previous methods, such as [36,
17] and [6], have proposed techniques for semantic data augmentation using the
latent space of generative models. These methods involve imposing uniform
latent representations and applying linear interpolation or learning mappings
to sample specific areas in the latent space. However, these approaches
require training generative models from scratch. In contrast, this paper
demonstrates a more efficient approach by utilizing the latent space of a pre-
trained diffusion model for creating data augmentations without the need for
additional model fine-tuning.

Figure 2: (a) Gradual changes to the norm of a fixed seed around a middle
value with a norm of $\sqrt{d}=128$. Images are generated by Stable Diffusion
[46]. The visual quality of generated images degrades as the norm diverges
away from $\sqrt{d}$ at the left and right end. (b) Mean per-class FID of the
generated images in relation to seed norm (lower is better). (c) Mean per-
class accuracy of the generated images, as determined by a state-of-the-art
pre-trained classifier, as a function of seed norm. Plots (b) and (c) show
that image quality degrades as the seed norm moves away from $\sqrt{d}$.

##  3 A norm-based prior over seeds

We start with reviewing statistical properties of samples in a seed latent
space $z_{T}\in\mathbb{R}^{d}$, with $d$ denoting the dimension of the space.

In diffusion models, it is common to sample $z$ from a high-dimensional
standard Gaussian distribution $z_{T}\sim\mathcal{N}(0,I_{d})$. For such
multivariate Gaussians, the $L_{2}$ norm of samples has a $\chi$ (Chi)
distribution:
$||z_{T}||=\sqrt{\sum^{d}_{i=0}{{z^{i}}}^{2}}\sim\chi^{d}=||z_{T}||^{d-1}e^{-||z_{T}||^{2}/2}/(2^{d/2-1}\Gamma(\frac{d}{2}))$
where $\Gamma(\cdot)$ is the Gamma function and $\|\cdot\|$ is the standard
Euclidean norm.

Importantly, as the dimension grows, the distribution of the norm tends to be
highly concentrated around the mean, since at a high dimension, the variance
approaches a constant 0.5. This strong concentration is illustrated in the
inset figure on the right, for $d=16384=128^{2}$, the seed dimension used by
Stable Diffusion [46]. At this dimension, the mean is also very close to the
mode of the distribution, and both approach $\sqrt{d}=128$. This property
means that samples drawn from a multi-variate high-dimensional Gaussian
distribution are concentrated around a specific value
$r=\text{mode}(\chi^{d})\approx\sqrt{d}$. Our key observation is that
diffusion models are trained with inputs sampled from the above normal
distribution, and therefore the models are only exposed to inputs with norm
values close to $r$ during training. We hypothesize that this causes the model
to be highly biased toward inputs with similar norm values.

We conducted several experiments to validate this bias. First, we inspected
the visual quality of images generated with different norm values, all sharing
the same direction of the seed vector. Figure 2(a) visually illustrates the
sensitivity of the Stable Diffusion model to the input norm, showing that
quality degrades as the norm drifts away from the mode. Second, we conducted a
systematic quantitative experiment and measured the impact of seed norm on
image quality in terms of FID and classification accuracy scores using an
ImageNet1k pre-trained classifier. Figures 2(b)-2(c), show again that image
quality depends on the seed having a norm close to the mode. Full details of
these experiments are given in Section 5 and supplemental material.

We conclude that the norm of the seed constitutes a key factor in the
generation of high-quality images. Below we describe how this fact can be used
for seed optimization and interpolation.

##  4 Norm-guided seed exploration

Based on the above results, we define a prior over the seed space as
$\mathcal{P}(z_{T}):=\chi^{d}(||z_{T}||)$. This probability density function
represents the likelihood of a seed with norm $||z_{T}||$ to be drawn from the
Gaussian distribution. We now describe simple and efficient methods for seed
interpolation and centroid finding using that prior.

###  4.1 Prior induced interpolation between two seeds

We first tackle the task of finding an interpolation path between the seeds of
two images. The derivation of this interpolation path illustrates the
advantages of using the prior in a simple setup, and will also be used later
for finding centroids for sets of seeds. As seen in Figure 1 (see also Figure
3), a linear interpolation path between seeds consists of seeds that yield
low-quality images. Instead, we define a better path
$\gamma:[0,1]\rightarrow\mathbb{R}^{d}$ as the solution to the following
optimization problem: Given two images, $I_{1}$ and $I_{2}$, and their
corresponding inversion seeds, $z^{1}_{T}$ and $z^{2}_{T}$, derived by
inversion techniques (e.g. DDIM Inversion [58]), we aim to maximize the log-
likelihood of that path under our prior, defined as the line-integral of the
log-likelihood of all points on the path.

Equivalently, we minimize the negative log-likelihood of the path, which is
strictly positive, yielding

$\displaystyle\underset{\mathbf{\gamma}}{\inf}~{}-\int_{\gamma}\log{\mathcal{P}(\gamma)}ds\quad\text{s.t.}\quad\gamma(0)=z^{1}_{T},\gamma(1)=z^{2}_{T}.$
(1)

Here, the infimum is taken with respect to all differentiable curves $\gamma$
and $\int_{\gamma}W(\gamma)ds$ denotes the line integral of a function
$W:\mathbb{R}^{d}\rightarrow\mathbb{R}$ over the curve $\gamma$ 000When
$\gamma$ is differentiable, the integral can be calculated using the following
formula: $\int_{0}^{1}W(\gamma(t))\|\gamma^{\prime}(t)\|dt$. We denote the
optimal value obtained for optimization problem (1) as
$f(z^{1}_{T},z^{2}_{T})$. It turns out that $f$ defines a (non-euclidean)
distance on the seed space. This is stated formally in the following
proposition.

######  Proposition 1.

For any distribution yielding strictly positive negative log-likelihood, and
specifically when $\mathcal{P}$ is the $\chi^{d}$ distribution, then
$f(\cdot,\cdot)$ is a distance function on $\mathbb{R}^{d}$.

See the supplementary material for proof. It is important to note that the
optimization problem does not only provide us with a path that maximizes the
log-likelihood of our prior, but it also defines a new metric structure on the
seed space that will prove useful in Section 4.2.

To approximate the solution to problem (1) in practice, we discretize the path
into a sequence of piece-wise linear segments, that connect a series of points
$z^{1}_{T}=x_{0},\dots,x_{n}=z^{2}_{T}$ and replace the integral with its
corresponding Riemann sum over that piece-wise linear path:

$\displaystyle\underset{x_{0},\dots,x_{n}}{\text{minimize}}$
$\displaystyle-\sum_{i=1}^{n}\log{\mathcal{P}\bigg{(}\frac{x_{i}+x_{i-1}}{2}\bigg{)}\|x_{i}-x_{i-1}\|}$
(2) $\displaystyle\text{s.t.}\quad
x_{0}=z^{1}_{T},x_{n}=z^{2}_{T},\quad\mathcal{C}(x)=|x_{i}-x_{i-1}\|-\delta\leq
0,i\in\\{1,\ldots,n\\}$

To facilitate a good approximation of the continuous integration, we also
constrain consecutive path points to be close (see implementation at the end
of Sec. 4.2).

Figure 3: Qualitative comparison among different interpolation methods between
two image seeds. "Jeep" is a common concept while "Tiger cat" is a rare
concept. Images generated using SD [46].

Figures 1-3 illustrate paths resulting from the optimization of the
discretized optimization problem. Our optimized path consistently produces
higher-quality images compared to other methods. A quantitative evaluation is
given in Section 5.

###  4.2 Prior induced centroid

Figure 4: Comparing different centroid finding methods in 2D space on the
contour of the $\chi$ distribution (log) PDF.

Having defined a new metric structure in seed space, we are now ready to
tackle the problem of finding a centroid of multiple seeds. To this end, we
assume to have a set of images $\\{I_{1},I_{2},\ldots,I_{k}\\}$ with their
inversions $\\{z^{1}_{T},z^{2}_{T},\ldots,z^{k}_{T}\\}$ and we wish to find
the centroid of these images. For example, one possible use of such a centroid
would be to find a good initialization for a seed associated with a rare
concept based on a few images of that concept. This enables us to merge
information between seeds and generate a more reliable fusion of the images
that would not be achievable through basic interpolation between two seeds.

Recall that the centroid of a set of points is defined as the point that
minimizes the sum of distances between all the points. Perhaps the simplest
way to define a centroid in our case is by using the Euclidean distance, where
the centroid definition boils down to a simple formula - the average of the
seeds. Unfortunately, as we show in Section 3 this definition of a centroid is
not suitable for our purposes due to the incompatibility of the centroid’s
norm with the diffusion model. Instead, we propose to use a generalization of
the Euclidean centroid, called the Fréchet mean, which is induced by the
distance function $f$ defined above. To achieve this, we formulate an
optimization problem that seeks the centroid $c\in\mathbb{R}^{d}$ that
minimizes the distances $f(c,z^{i}_{T})$ to all the seeds:

$c^{*},\gamma^{*}_{1},\ldots,\gamma^{*}_{k}=\underset{c,\gamma_{1},\dots,\gamma_{k}}{\text{argmin}}\left({-\sum_{l=1}^{k}\int_{\gamma_{i}}\log{\mathcal{P}(\gamma_{i})}ds}\right),$
(3)

where $\gamma_{i}$ are paths between the common centroid $c$ and the inversion
$z^{i}_{T}$. This optimization problem can be discretized in the same way we
discretized Equation (1), i.e. by defining all the paths as piecewise linear
paths defined by a sequence of points and adding constraints on the distances
between successive points in the path. See the supplemental for a discretized
equation.

Figure 5: Comparing different centroid optimization approaches on common and
rare concepts of ImageNet1k, w.r.t strong baselines. We further initialized
SeedSelect [51] with the centroids and run it for up to 5 iterations ($\sim$20
sec on a single A100 GPU). NAO manages to generate correct images for rare
concepts (a cat in tiger-cat, a clear oxygen mask and payphone), even without
SeedSelect. With SeedSelect the images are further improved. The effect is
less emphasized on common concepts w.r.t to strong baselines.

Figure 4 illustrates an example of a centroid in 2D found with our approach.
Figure 5 shows images resulting from the centroids found in seed space.
Section 5 provides quantitative results.

Application to seed optimization methods. Diffusion models often encounter
significant imbalances in the distribution of concepts within their training
data [51]. This imbalance presents a challenge for standard pre-trained
diffusion models, leading to difficulties in accurately generating rare
concepts. One proposed solution, as suggested by [51], involves employing a
seed optimization technique. SeedSelect begins with a randomly generated seed
that produces an incorrect image and progressively optimizes the seed until a
plausible image is generated. However, the method described in [51] is time-
consuming, requiring up to 5 minutes to generate a single image. Our approach
can serve as an initial point for SeedSelect to reduce substantially its
optimization time by generating initial seeds that result in realistic and
quality images.

In our case, given few images, generating new data can be obtained in the
following manner: First, find a centroid $c^{*}$ and the interpolation paths
$\gamma^{*}_{i}$ between image inversion in the seed space. Next, new data is
generated by sampling points along the paths from the givens seeds to the
centroid, and using them as initializations for SeedSelect.

Implementation We implemented a simple optimization algorithm that optimizes
the discretized problems using Pytorch and the Adam optimizer. To speed up
convergence we initialize the optimization variables: the centroid is
initialized with the Euclidean centroid and path variables are initialized as
the values of the linear path between the points and the centroid. We
implement the constraints $\mathcal{C}(x)=|x_{i}-x_{i-1}|-\delta\leq 0$ using
a soft penalty term in the optimization, by the form
$\alpha\cdot\text{ReLU}(\mathcal{C}(x))$ where $\alpha$ is a hyper-parameter.
Note that there is a penalty for positive $\mathcal{C}(x)$, when the
constraint is not satisfied. We note that in practice there is no guarantee
that this optimization scheme converges to the optimal value of the
optimization problem, however, we see in practice that high-quality paths are
obtained.

##  5 Experiments

We evaluate our approach in terms of image quality and also consider
generation time. We start by studying the quality and semantic content of
images generated with our seeding approach and evaluate them in three
applications: (1) rare concept generation and (2) semantic data augmentation,
to be tested at enhancing few-shot classification, long-tail learning, and
few-shot image recognition. (3) We further tested our model on video frame
interpolation, showing the results in the supplementary material. An ablation
study can also be found in the supplementary material.

Direct evaluation of interpolation and centroid finding:  Table 1 compares FID
scores and the accuracy of images generated using different interpolation and
centroid finding methods. For the interpolation experiment, we randomly
selected a class from ImageNet1k, and obtained a pair of images and their
corresponding seeds through inversion [58]. For interpolation methods that
require path optimization, we used paths with 10 sampled points. We then
select three seeds along the path (also for LERP and SLERP), with uniform
intervals, and feed them into StableDiffusion [46], to generate 3 new images
per pair. We repeated the process above to obtain 100 images per class. For
the centroid experiment, we used 3-25 seed points obtained from the inversion
of additional images (randomly selected) from the train set. We repeated this
process for 50 random ImageNet1k classes. Mean FID scores were then calculated
(against real ImageNet1k images), along with mean per-class accuracy using a
pre-trained classifier. See supplementary material for more details. Our
optimized path consistently produces higher-quality images compared to other
methods.

| Acc | FID
---|---|---
Interpolation methods |  |
LERP | 0.0 | 50.59
SLERP [56] | 30.41 | 18.44
NAO-path (ours) | 51.59 | 6.78
Centroid computation methods |  |
Euclidean | 0.0 | 54.88
Normalized Euclidean | 27.95 | 37.04
Sphere Projection | 40.81 | 14.28
NAO-centroid (ours) | 67.24 | 5.48

Table 1: Comparing FID and accuracy of images generated by SD through sampling
from different interpolation and centroid computation methods.

We compared our approach to the following baselines: Euclidean is the standard
Euclidean centroid calculated as the mean of the seeds. Normalized Euclidean
is the same as Euclidean, but the centroid is projected to the sphere induced
by the $\chi$ distribution. Sphere Projection first normalizes the seed to a
sphere with radius $r=Mode(\chi)$, then finds the centroid on a sphere by
optimizing a point that minimizes the sum of geodesic paths between the seeds
to the centroid, as presented in [9]. See supplemental for more details. NAO-
path and NAO-centroid are our methods presented in sections 4.1 and 4.2,
respectively. The high accuracy and low FID levels of NAO in Table 1
demonstrate that our interpolation approach outperforms other baselines in
terms of image quality and content. We further put these results to test in
downstream tasks (in Sec. 5.1-5.3).

###  5.1 Rare-concept generation

Following [51] we compared different centroid optimization strategies in rare-
concept generation.

Dataset. The evaluation is performed on ImageNet1k classes ordered by their
prevalence in the LAION2B dataset [53]. LAION2B [53] is a massive "in the
wild" dataset that is used for training foundation diffusion models (e.g.
Stable Diffusion [46]).

Compared Methods. We conducted a comparative evaluation between different
centroid estimation strategies. SeedSelect [51] is a baseline method where a
seed is randomly sampled and then optimized to produce correct rare concepts.
Here, no centroid is calculated. Other baselines were presented at the
beginning of Section 5.

| ImageNet1k in LAION2B
---|---
Methods | Many | Med | Few | Total Acc | FID | $\hat{T}_{Init}$ | $\hat{T}_{Opt}$
| n=235 | n=509 | n=256 |  |  | (sec) | (sec)
| #>1M | 1M>#>10K | 10K># |  |  |  |
SeedSelect [51] | 97.8 | 95.8 | 76.1 | 91.3 | 6.5 | 0 | 298
Euclidean | 0.0 | 0.0 | 0.0 | 0.0 | 51.1 | 0 | -
Euclidean + SeedSelect | 0.0 | 0.0 | 0.0 | 0.0 | 65.7 | 0 | inf
Normalized Euclidean | 45.3 | 39.8 | 20.1 | 36.0 | 45.8 | 0 | -
Normalized Euclidean + SeedSelect | 88.1 | 84.8 | 70.9 | 82.0 | 19.2 | 0 | 285
Sphere projection | 52.0 | 48.9 | 26.5 | 44.0 | 12.3 | 0.1 | -
Sphere projection + SeedSelect | 97.1 | 95.1 | 74.2 | 90.2 | 8.6 | 0.1 | 72
NAO-centroid (ours) | 56.8 | 55.6 | 41.1 | 52.2 | 9.8 | 25 | -
NAO-centroid + SeedSelect (ours) | 98.5 | 96.9 | 85.1 | 94.3 | 6.4 | 25 | 29

Table 2: Image generation quality measured by the accuracy of a pre-trained
classifier. Average per-class accuracy is reported separately for the head,
tail, and middle classes. NAO produces the best performance in terms of image
quality and is also $\times 10$ faster.

Experimental Protocol. For every class in ImageNet, we randomly sampled
subsets of training images, calculated their centroid in seed space, and
generated an image using Stable Diffusion directly or as input to SeedSelect
[51]. The class label was used as the prompt. This process is repeated until
100 images are generated for each class. We then used a SoTA pre-trained
classifier [62] to test if the generated images are from the correct class or
not (more details can be found in the supplementary). We use this measure to
evaluate the quality of the generated image, verifying that a strong
classifier correctly identifies the generated image class. We also report Mean
FID score between the generated images and the real images, mean centroid
initialization time $\hat{T}_{Init}$ and mean SeedSelect optimization time
until convergence $T_{Opt}$ on a single NVIDIA A100 GPU. The results
summarized in Table 2 show that our NAO method substantially outperforms other
baselines, both in accuracy and in FID score. Furthermore, NAO gives a better
initialization point to SeedSelect [51], yielding significantly faster
convergence without sacrificing accuracy or image quality.

Next, we evaluate NAO as a semantic data augmentation method on two learning
setups: (1) Few-shot classification, and (2) Long-tail classification. We aim
to show that our approach not only achieves faster generation speed but also
attains state-of-the-art accuracy results on these benchmarks.

(a) | miniImageNet | CIFAR-FS
---|---|---
Model |  | 5-way 5-shot |  | 5-way 5-shot
Label-Halluc [28] |  | $67.04\pm 0.7$ |  | 89.37$\pm$0.6
FeLMi [47] |  | 86.08$\pm$0.4 |  | 89.47$\pm$0.5
SEGA [70] $\dagger$ |  | 79.03$\pm$0.2 |  | 86.00$\pm$0.2
SVAE [69] $\ddagger$ |  | 80.70$\pm$0.2 |  | 78.89$\pm$0.3
Textual Inversion [22]* $\ddagger$ |  | 85.44$\pm$3.9 |  | -
Stable Diffusion [46] $\ddagger$ |  | 85.05$\pm$0.5 |  | 90.87$\pm$0.5
DiffAlign [48] $\ddagger$ |  | 88.63$\pm$0.3 |  | 91.96$\pm$0.5
SeedSelect [51] |  | 92.08$\pm$0.7 |  | 94.87$\pm$0.4
NAO-centroid (ours) |  | 93.21$\pm$0.6 |  | 94.85$\pm$0.5

(b) | CUB
---|---
Model |  | 5-way 5-shot
TriNet [12] |  | 84.10$\pm$0.4
FEAT [71] |  | 82.90$\pm$0.2
DeepEMD [72] |  | 88.69$\pm$0.5
MultiSem [54] $\dagger$ |  | 82.9$\pm$n/a
SEGA [70] $\dagger$ |  | 90.85$\pm$0.2
Stable Diffusion* [46] $\ddagger$ |  | 90.61$\pm$0.5
SeedSelect [51] |  | 96.01$\pm$0.4
NAO-centroid (ours) |  | 97.92$\pm$0.2

(c) | ImageNet-LT |
---|---|---
Model |  |
CE | 41.6 |
MetaSAug [33] | 47.4 |
smDragon [50] | 47.4 |
CB LWS [29] | 47.7 |
DRO-LT [52] | 53.5 |
Ride [67] | 55.4 |
PaCO [13] | 53.5 |
DRAGON [50] $\dagger$ | 57.0 |
VL-LTR [61] $!$ | 70.1 |
Stable Diffusion [51] $\ddagger$ | 56.4 |
SeedSelect [51]$\ddagger$ | 74.9$\pm$0.5 |
NAO-centroid (ours) | 78.9$\pm$0.1 |

Table 3: (a)-(b) Few-shot recognition. Comparing NAO to prior work on few-shot
learning benchmarks. We report our results with 95% confidence intervals on
meta-testing split of the dataset. (c) Long-tail recognition. Comparing NAO to
prior work on long-tail learning benchmark. Values are accuracy, obtained with
a ResNet-50 backbone. * denote results provided by [51]. $\dagger$ for multi-
modal methods that use class labels as additional information, $\ddagger$ for
multi-modal methods that were also pre-trained on external datasets, and $!$
for multi-modal methods that further finetuned foundation models. Our approach
achieves the best results on all benchmarks.

| miniImageNet | CIFAR-FS | CUB
---|---|---|---
| $\bar{T}_{Init}$ | $\bar{T}_{Opt}$ | $\bar{T}_{Total}$ | $\bar{T}_{Init}$ | $\bar{T}_{Opt}$ | $\bar{T}_{Total}$ | $\bar{T}_{Init}$ | $\bar{T}_{Opt}$ | $\bar{T}_{Total}$
SeedSelect [51] | - | 276 | 276 sec | - | 228 sec | 228 sec | - | 312 sec | 312 sec
NAO-centroid (ours) | 28 sec | 25 sec | 53 sec | 29 sec | 21 sec | 50 sec | 31 sec | 29 sec | 60 sec

Table 4: Comparing convergence time between SeedSelect [51] alone and
SeedSelect initialized with seeds found using NAO. Note the five-fold
reduction in runtime for our method.

###  5.2 Few-shot learning

Few-shot benchmarks typically provide a limited number of samples per class,
along with the corresponding class labels. NAO is used as a semantic data
augmentation approach: We use the given samples and generate many more samples
from that class, which are then used to train a classifier.

Datasets. We evaluated NAO on three common few-shot classification benchmarks:
(1) CUB-200 [66]: A fine-grained dataset comprising 11,788 images of 200 bird
species. The classes are divided into three sets, with 100 for meta-training,
and 50 each for meta-validation and meta-testing. (2) miniImageNet [64]: A
modified version of the standard ImageNet dataset [15]. It contains a total of
100 classes, with 64 classes used for meta-training, 16 classes for meta-
validation, and 20 classes for meta-testing. The dataset includes 50,000
training images and 10,000 testing images, with an equal number of images
distributed across all classes. (3) CIFAR-FS [8]: Created from CIFAR-100 [32]
by using the sampling criteria as miniImageNet. Has 64 classes for meta-
training, 16 classes for meta-validation, and 20 classes for meta-testing;
each class containing 600 images.

Following all previous baselines, we report classification accuracy as the
metric. We report our results with 95% confidence intervals on the meta-
testing split of the dataset.

Compared Methods. We conducted a comparative evaluation of our approach with
several state-of-the-art methods for few-shot learning. These methods fall
into three categories based on their approach. (A) Methods that do not use
pre-training nor use class labels during training: Label-Hallucination [28]
and FeLMi [47]; (B) Methods that use class labels during training: SEGA [70];
and (C) Methods that utilize a classifier pre-trained on external data and
also use class labels during training: SVAE [69], Vanilla Stable Diffusion
(Version 2.1) [46], Textual Inversion [22], DiffAlign [48], and SeedSelect
[51]. The last four methods are semantic data augmentation methods.

Experimental Protocol. For a fair comparison with prior work, we follow the
training protocol in [48] and [51]. We generated 1,000 additional samples for
each novel class using SeedSelect. It was initialized with seeds found with
NAO using the centroid and interpolation samples of the few-shot images
provided during meta-testing and prompted it with the corresponding class
name. We used a ResNet-12 model for performing N-way classification and
trained it using cross-entropy loss on both real and synthetic data.

Results. Tables 3a and 3b compare NAO with SoTA approaches on few-shot
classification benchmarks: CUB, miniImageNet, and CIFAR-FS. NAO consistently
outperforms all few-shot methods on CUB [66] and miniImageNet [64], and
reaches comparable results to SeedSelect [51] on CIFAR-FS [8]. Table 4 further
compares the mean run time of SeedSelect with and without NAO on these
datasets, on a single NVIDIA A100 GPU. The results highlight the competence of
our approach in generating rare and fine-grained classes, to reach top
accuracy with a five-fold reduction in the runtime.

###  5.3 Long-tail learning

Datasets. We further evaluated NAO on long-tailed recognition task using the
ImageNet-LT [38] benchmark. ImageNet-LT [38] is a variant of the ImageNet
dataset [16] that is long-tailed. It was created by sampling a subset of the
original dataset using the Pareto distribution with a power value of
$\alpha=6$. The dataset contains 115,800 images from 1,000 categories, with
the number of images per class ranging from 5 to 1,280.

Compared Methods. We compared our approach with several state-of-the-art long-
tail recognition methods. These methods fall into three categories based on
their approach. (A) Long-tail learning methods that do not use any pretraining
nor employ class labels for training: CE (naive training with cross-entropy
loss), MetaSAug [33], smDragon [50], CB LWS [29], DRO-LT [52], Ride [67] and
Paco [13]. (B) Methods that use class labels as additional information during
training: DRAGON [50]. (C) Methods that were pre-trained on external datasets
and use class labels as additional information during training: VL-LTR [61],
Vanilla Stable Diffusion (Version 2.1) [46] and SeedSelect [51]. MetaSAug
[33], Vanilla Stable Diffusion and SeedSelect [51] are semantic augmentations
methods. Note that VL-LTR [61], compared to other models, further fine-tuned
the pre-trained model (CLIP [42]) on the training sets.

Experimental Protocol. Following previous methods, we use a ResNet-50 model
architecture, train it on real and generated data, and report the top-1
accuracy over all classes on class-balanced test sets.

Results. Tables 3c evaluates our approach compared to long-tail recognition
benchmarks. NAO reaches SoTA results albeit simple, compared to other complex
baselines.

###  5.4 CLIP finetuning with Synthetic data for image classification

We follow [51, 76, 74] and further examine the advantage of using our method
in the generation of synthetic images, as semantic augmentation, for the
downstream task of few shot CLIP recognition. In order to deal with rare
concepts we use our method together with SeedSelect showing SoTA results in
this task [51]. To this end we use NAO as seed initialization for SeedSelect.
In the context of this task, we are provided with a limited number of real
training samples per class, along with their corresponding class names, and
the goal is to fine-tune CLIP.

Experimental Setup. For a fair comparison we follow the same experimental
protocol of [51]. Specifically, we generated 800 samples for each class using
NAO+SeedSelect from the limited real training samples. We then fine-tune a
pre-trained CLIP-RN50 (ResNet-50) through mix-training, incorporating both
real and generated images.

Compared Methods. We compare our approach with the following baselines: Zeor-
shot CLIP: Applying the pre-trained CLIP classifier without fine-tuning; CooP
[76]: Fine-tuning a pre-trained CLIP via learnable continuous tokens while
keeping all model parameters fixed; Tip Adapter [74]: Fine-tuning a
lightweight residual feature adapter; CT & SD: Classifier tuning with images
generated with SD. Textual Inversion [22]: Classifier tuning with images
generated using personalized concepts (See Supp for implementation detail).
Results for CT & SD were reproduced by us on SDv2.1 using the code published
by the respective authors.

Figure 6: Results for few-shot image recognition, comparing NAO+SeedSelect to
previous approaches. Fine-tuning a CLIP classifier on SeedSelect generated
images initialized with NAO consistently achieves SOTA results across all shot
levels.

Results. Figure 6 shows results for the few-shot image recognition task. It
demonstrates that initiating seeds using NAO improves SeedSelect performance
on all shots.

##  6 Conclusion

This paper proposes a set of simple and efficient tools for exploring the seed
space of text-to-image diffusion models. By recognizing the role of the seed
norm in determining image quality based on the $\chi$ distribution as prior,
we introduce a novel method for seed interpolation and define a non-Euclidean
metric structure over the seed space. Furthermore, we redefine the concept of
a centroid for a set of seeds and present an optimization scheme based on the
new distance function. Experimental results demonstrate that these
optimization schemes, biased toward the $\chi$ distribution mode, generate
higher-quality images compared to other approaches. Despite the simplicity and
effectiveness of our approach, there are several limitations to be aware of.
Firstly, compared to standard interpolation and centroid calculation, it
involves an additional optimization step. Secondly, our centroid and/or the
samples along our interpolation paths may not produce plausible and
semantically correct images on their own, necessitating the use of SeedSelect
optimization. Lastly, although our method is expected to be applicable to all
diffusion models, we specifically evaluated it with the open-source Stable
Diffusion [46] model in this study.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  ICCV, 2019.
  * [2] Georgios Arvanitidis, Bogdan Georgiev, and Bernhard Schölkopf.  A prior-based approximate latent riemannian metric.  In AISTAT, 2021.
  * [3] Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg.  Latent space oddity: on the curvature of deep generative models.  ICLR, 2018.
  * [4] Georgios Arvanitidis, Søren Hauberg, Philipp Hennig, and Michael Schober.  Fast and robust shortest paths on manifolds learned from data.  ArXiv, abs/1901.07229, 2019.
  * [5] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin.  Spatext: Spatio-textual representation for controllable image generation.  CVPR, 2023.
  * [6] Idan Azuri and Daphna Weinshall.  Learning from small data through sampling an implicit conditional generative latent optimization model.  ICPR, 2020.
  * [7] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [8] Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi.  Meta-learning with differentiable closed-form solvers.  ICLR, 2019.
  * [9] Samuel R. Buss and Jay P. Fillmore.  Spherical averages and applications to spherical splines and interpolation.  ACM Trans. Graph., 2001.
  * [10] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  arXiv preprint arXiv:2301.13826, 2023.
  * [11] Ying-Cong Chen, Xiaogang Xu, Zhuotao Tian, and Jiaya Jia.  Homomorphic latent space interpolation for unpaired image-to-image translation.  CVPR, 2019.
  * [12] Z. Chen, Yanwei Fu, Yinda Zhang, Yu-Gang Jiang, X. Xue, and Leonid Sigal.  Multi-level semantic feature augmentation for one-shot learning.  IEEE Transactions on Image Processing, 28:4594–4605, 2018.
  * [13] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia.  Parametric contrastive learning.  ICCV, 2021.
  * [14] Duolikun Danier, Fan Zhang, and David Bull.  Ldmvfi: Video frame interpolation with latent diffusion models.  arXiv preprint arXiv:2303.09508, 2023.
  * [15] J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In cvpr, 2009.
  * [16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [17] Terrance Devries and Graham W. Taylor.  Dataset augmentation in feature space.  ICLR Workshop, 2017.
  * [18] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  NeurIPS, 2021.
  * [19] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  NeurIPS, 2021.
  * [20] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Training-free structured diffusion guidance for compositional text-to-image synthesis.  ICLR, 2023.
  * [21] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  In ECCV, 2022.
  * [22] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  ICLR, 2023.
  * [23] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  CVPR, 2021.
  * [24] Dianyuan Han.  Comparison of commonly used image interpolation methods.  In ICCSEE, 2013.
  * [25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In NIPS, 2017.
  * [26] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NeurIPS, 2020.
  * [27] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  NeurIPS workshop on Deep Generative Models and Downstream Applications, 2021.
  * [28] Yiren Jian and Lorenzo Torresani.  Label hallucination for few-shot classification.  In AAAI, 2022.
  * [29] B. Kang, S. Xie, M. Rohrbach, M. Yan, A. Gordo, J. Feng, and Y. Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  ICLR, 2020.
  * [30] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  ICLR, 2018.
  * [31] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  NeurIPS, 2022.
  * [32] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [33] Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng.  Metasaug: Meta semantic augmentation for long-tailed visual recognition.  CVPR, 2021.
  * [34] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In ECCV, 2022.
  * [35] Vivian Liu and Lydia B Chilton.  Design guidelines for prompt engineering text-to-image generative models.  In ACM SIGCHI, 2022.
  * [36] Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang, Site Li, Ping Jia, and Jane You.  Data augmentation via latent space interpolation for image classification.  In ICPR, 2018.
  * [37] Yahui Liu, E. Sangineto, Yajing Chen, Linchao Bao, Haoxian Zhang, Niculae Sebe, Bruno Lepri, and Marco De Nadai.  Smooth image-to-image translations with latent space interpolations.  ArXiv, 2022.
  * [38] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, 2019.
  * [39] Gary Marcus, Ernest Davis, and Scott Aaronson.  A very preliminary analysis of dall-e 2.  arXiv preprint arXiv:2204.13807, 2022.
  * [40] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  NeurIPS, 2022.
  * [41] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In CVPR, 2016.
  * [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [44] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  CVPR, 2020.
  * [45] Eitan Richardson and Yair Weiss.  On gans and gmms.  NeurIPS, 2018.
  * [46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [47] Aniket Roy, Anshul Shah, Ketul Shah, Prithviraj Dhar, Anoop Cherian, and Rama Chellappa.  FeLMi : Few shot learning with hard mixup.  In NeurIPS, 2022.
  * [48] Aniket Basu Roy, Anshul B. Shah, Ketul Shah, Anirban Roy, and Ramalingam Chellappa.  Diffalign : Few-shot learning using diffusion based synthesis and alignment.  ArXiv, abs/2212.05404, 2022.
  * [49] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  NeurIPS, 2022.
  * [50] Dvir Samuel, Yuval Atzmon, and Gal Chechik.  From generalized zero-shot learning to long-tail with class descriptors.  In WACV, 2021.
  * [51] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik.  It is all about where you start: Text-to-image generation with seed selection.  ArXiv, abs/2304.14530, 2023.
  * [52] Dvir Samuel and Gal Chechik.  Distributional robustness loss for long-tail learning.  ICCV, 2021.
  * [53] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  NeurIPS, 2022.
  * [54] Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex Bronstein.  Baby steps towards few-shot learning with multiple semantics.  Pattern Recognition Letters, 2022.
  * [55] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  CVPR, 2019.
  * [56] Ken Shoemake.  Animating rotation with quaternion curves.  In SIGGRAPH, 1985.
  * [57] Wan-Chi Siu and Kwok-Wai Hung.  Review of image interpolation and super-resolution.  In Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 2012.
  * [58] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  ICLR, 2021.
  * [59] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  CVPR, 2015.
  * [60] Philippe Thévenaz, Thierry Blu, and Michael Unser.  Image interpolation and resampling.  Handbook of medical imaging, processing and analysis, 2000.
  * [61] Changyao Tian, Wenhai Wang, Xizhou Zhu, Xiaogang Wang, Jifeng Dai, and Y. Qiao.  Vl-ltr: Learning class-wise visual-linguistic representation for long-tailed visual recognition.  In European Conference on Computer Vision, 2021.
  * [62] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  ECCV, 2022.
  * [63] Vishaal Udandarao, Ankush Gupta, and Samuel Albanie.  Sus-x: Training-free name-only transfer of vision-language models.  ArXiv, 2022.
  * [64] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In NeurIPS, 2016.
  * [65] Vikram Voleti, Alexia Jolicoeur-Martineau, and Chris Pal.  Mcvd-masked conditional video diffusion for prediction, generation, and interpolation.  Neurips, 2022.
  * [66] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [67] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu.  Long-tailed recognition by routing diverse distribution-aware experts.  ICLR, 2021.
  * [68] Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau.  Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models.  arXiv preprint arXiv:2210.14896, 2022.
  * [69] Jingyi Xu and Hieu Le.  Generating representative samples for few-shot classification.  In CVPR, 2022.
  * [70] Fengyuan Yang, Ruiping Wang, and Xilin Chen.  Sega: semantic guided attention on visual prototype for few-shot learning.  In WACV, 2022.
  * [71] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Few-shot learning via embedding adaptation with set-to-set functions.  In CVPR, 2020.
  * [72] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In CVPR, 2020.
  * [73] Qinsheng Zhang and Yongxin Chen.  Fast sampling of diffusion models with exponential integrator.  ICLR, 2023.
  * [74] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Jiao Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  ECCV, 2022.
  * [75] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal.  Image generation from layout.  In CVPR, 2019.
  * [76] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 2021.
  * [77] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  ECCV, 2020.

Supplemental Material

##  Appendix A Proof for proposition 1

######  Proposition 2.

Let $W:\mathbb{R}^{d}\rightarrow\mathbb{R}$ be a strictly positive continuous
function. Given $x,y\in\mathbb{R}^{d}$, define $f(x,y)$ as the infimum over
all path integrals from $x$ to $y$ where the infimum is taken over all
piecewise differentiable curves.

$\displaystyle
f(x,y):=\underset{\mathbf{\gamma}}{\text{inf}}\int_{\gamma}W(\gamma)ds$
$\displaystyle\quad\text{s.t.}\quad\gamma(0)=x,\gamma(1)=y.$ (4)

Then $f$ is a distance function

###### Proof.

We need to show that $f$ satisfies three properties: non-negativity, symmetry,
and the triangle inequality.

First, $f(x,y)\geq 0$ for all $x,y\in\mathbb{R}^{d}$. This follows from the
fact that any line integral of a positive function is non-negative, and from
the fact that the infimum of a set of non-negative numbers is itself non-
negative. In addition, we have to show that $f(x,y)=0$ iff $x=y$. If $x=y$
then $W$ has zero line integral on the constant curve $\gamma\equiv x$ which
implies $f(x,y)\leq 0$, which combined with the non-negativity of $f$ implies
$f(x,y)=0$.

In the other direction assume that $x\neq y$ so $\|x-y\|\geq 0$. We will show
that for any such $x,y$ there is a constant $c>0$ such that the line integral
over $\gamma$ is strictly larger than $c$ which implies $f(x,y)\geq c>0$. Let
$K$ be a closed ball that contains $x,y$ in its interior. Let $\epsilon>0$ be
the minimal distance between $x$ and $\partial K$. let $m_{K}>0$ be the
minimal value of $W$ on $K$ that exists due to compactness of $K$, and the
positivity and continuity of $W$. finally, Let $\gamma$ be a path from $x$ to
$y$. We split into two cases:

  1. 1.

if $Im(\gamma)\subset K$ then we have $f(x,y)\geq m_{k}\|x-y\|>0$ since the
value of any line integral from $x$ to $y$ is bounded by the distance between
the points times the minimal value of the integrand.

  2. 2.

Otherwise, $\gamma$ has to intersect with $\partial K$ at some point
$x^{\prime}$ and a similar argument shows that $f(x,y)\geq
m_{k}\|x-x^{\prime}\|\geq m_{k}\epsilon>0$.

This implies that for any $\gamma$ the value of the line integral is larger
than $c=\min(m_{k}\epsilon,m_{k}\|x-y\|)>0$ and concludes this part of the
proof.

Next, it is easy to see that $f(x,y)=f(y,x)$ for all $x,y\in\mathbb{R}^{d}$.
This follows from the fact that the reverse path gives rise to the same line
integral value as the original path.

Finally, we show that $f(x,z)\leq f(x,y)+f(y,z)$ for all
$x,y,z\in\mathbb{R}^{d}$. Let $\epsilon>0$. Following the infimum definition
there is a path $\gamma_{1}$ from $x$ to $y$ such that
$\int_{\gamma_{1}}W(\gamma_{1})ds<f(x,y)+\epsilon/2$, and a path $\gamma_{2}$
from $y$ to $z$ such that
$\int_{\gamma_{2}}W(\gamma_{2})ds<f(y,z)+\epsilon/2$. Then, the path $\gamma$
obtained by concatenating $\gamma_{1}$ and $\gamma_{2}$ is a piecewise
differentiable curve from $x$ to $z$. Therefore,

$\displaystyle f(x,z)$ $\displaystyle\leq\int_{\gamma}W(\gamma)ds$
$\displaystyle=\int_{\gamma_{1}}W(\gamma_{1})ds+\int_{\gamma_{2}}W(\gamma_{2})ds$
$\displaystyle<f(x,y)+f(y,z)+\epsilon.$

Since this is true for any $\epsilon>0$ we get $f(x,z)\leq f(x,y)+f(y,z)$ as
needed. Thus, we have shown that $f$ satisfies all three properties of a
distance function, and therefore $f$ is a distance function. ∎

To prove Proposition 1 in the main paper, we use the proposition above and the
fact that the $\\-\log(\chi^{d}(x))$ is positive for all $x\in\mathbb{R}^{d}$.

##  Appendix B Background and preliminaries

Diffusion models: Text-guided diffusion models aim to map a random noise
$z_{t}$ and textual condition $P$ to an output image $z_{0}$, which
corresponds to the given conditioning prompt. The mapping between $z_{t}$ to
$z_{0}$ using $P$, also called the "denoising process", is performed
sequentially by a network $\varepsilon_{\theta}$ which is trained to predict
noise by minimizing the loss:
$\mathcal{L}=\mathbb{E}_{z_{0},y,\varepsilon\sim\mathcal{N}(0,1),t}\left[||\varepsilon-\varepsilon_{\theta}(z_{t},t,c(y))||_{2}^{2}\right]$.
Here, $z_{t}$ is a noised sample, and $c(P)$ is the embedding of the text
condition, where noise is added to the sampled data $z_{0}$ according to a
timestamp $t$. Essentially, the role of the denoising network
$\epsilon_{\theta}$ is to accurately eliminate the added noise $\epsilon$ from
the latent code $z$ at every time step $t$, based on the input of the noisy
latent code $z_{t}$ and the encoding of the conditioning $c(P)$. At inference,
$z_{T}$ is sampled from a standard multivariate Gaussian distribution
$\mathcal{N}(0,I)$ and the noise is iteratively removed by the trained
$\epsilon_{\theta}$ for T steps, resulting $z_{0}$.

Although our method is universally applicable to all diffusion models, in this
study we specifically employed the open-source Stable Diffusion [46]. Here,
the diffusion process is applied on a latent image encoding
$z_{0}=\mathcal{E}(x_{0})$ and an image decoder is employed at the end of the
diffusion backward process $x_{0}=\mathcal{D}(z_{0})$.

Sampling and Inversion: The process of mapping an image to noise is a Markov
chain starting from $z_{0}$, and gradually adding noise to obtain latent
variables $z_{1},z_{2},\dots,z_{T}$. The sequence of latent variables follows
$q(z_{1},z_{2},\ldots,z_{T})=\Pi^{t}_{i=1}{q(z_{t}|z_{t-1}})$. A step in this
process is a Gaussian transition
$q(z_{t}|z_{t-1}):=\mathcal{N}(z_{t},\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I)$
parameterized by a schedule $\beta_{0},\beta_{1},\ldots,\beta_{T}\in(0,1)$.
Note that $z_{t}$ can be expressed as a linear combination of noise and
$z_{0}$: $z_{t}=\sqrt{\alpha_{t}z_{0}}+\sqrt{1-\alpha_{t}}w$, where
$w\sim\mathcal{N}(0,I)$ and $\alpha_{t}=\Pi^{t}_{i=1}(1-\beta_{i})$.

Reversing the process is not immediately obvious and thus several schedulers
were proposed [58, 31, 26, 23]. In this paper, we employ DDIM [58] scheduler,
a popular deterministic scheduler. Other deterministic scheduler would be
suitable, and we show in section I below that our method performs well with
other schedulers. For DDIM, [58] proposed denoising in the following way:
$z_{t-1}=\sqrt{\frac{\alpha_{t-1}}{\alpha_{t}}}z_{t}+(\sqrt{\frac{1}{\alpha_{t-1}}-1}-\sqrt{\frac{1}{\alpha_{t}}-1})\cdot\varepsilon_{\theta}(z_{t},t,c(y))$.

Since DDIM is deterministic, its inversion can be obtained based on the
assumption that the Ordinary Differential Equation (ODE) process can be
reversed in the limit of small steps.
$z_{t+1}=\sqrt{\frac{\alpha_{t+1}}{\alpha_{t}}}z_{t}+(\sqrt{\frac{1}{\alpha_{t+1}}-1}-\sqrt{\frac{1}{\alpha_{t}}-1})\cdot\varepsilon_{\theta}(z_{t},t,c(y))$.
See [19, 58] for more details. Using this technique we can inverse a real
image $z_{0}$ to its latent $z_{t}$ in the seed space.

##  Appendix C Analysis of Approximation and Optimization Errors

In this section, we delve into the approximation and optimization aspects of
our proposed methodology, particularly with regard to Equation (1) and
Equation (2). Solving the optimization problem stated in Equation (2) yields
an approximation to its continuous counterpart, Equation (1). There are two
primary error components:

(i) Approximation Error: This error arises from the discreteness of the
integral in Equation (2) as compared to the continuous integral in Equation
(1). However, we assert that due to the smoothness of the function f(x), the
minimizer of Equation (1) is expected to exhibit smooth behavior.
Consequently, we can use piecewise linear paths in Equation (2) to approximate
it with arbitrarily small error by increasing the number of points.

(ii) Optimization Error: This error stems from the optimization process
applied to Equation (2). In line with common practice in deep learning, we
employ first-order optimization techniques to optimize a non-smooth function.
We show that our experimental results and 2D visualizations support the
assertion that our optimization procedure converges to a satisfactory
solution.

From a practical standpoint, despite the potential lack of numerical
exactness, our approach consistently yields high-quality results, as indicated
by the Frechet Inception Distance (FID) measures, which serve as a metric for
evaluating the quality of generated images.

##  Appendix D Discretizing the centroid-finding problem

To approximate the solution to problem Eq. 3 in practice, we discretize the
paths to the centroid in a similar fashion to Equation (5) by representing
them as a sequence of piece-wise linear segments. We then replace the integral
with its corresponding Riemann sum over these piece-wise linear paths:

$\displaystyle\underset{c,x^{l}_{0},\dots,x^{l}_{n}\forall
l}{\text{minimize}}$
$\displaystyle-\log{\mathcal{P}(c)}-\sum_{l=1}^{k}\sum_{i=1}^{n}\log{\mathcal{P}\bigg{(}\frac{x^{l}_{i}+x^{l}_{i-1}}{2}\bigg{)}\|x^{l}_{i}-x^{l}_{i-1}\|}$
(5) $\displaystyle\text{s.t.}\quad
x^{l}_{0}=c,x^{l}_{n}=z^{l}_{T},\quad\|x^{l}_{i}-x^{l}_{i-1}\|\leq\delta,l\in\\{1,\ldots,k\\},i\in\\{1,\ldots,n\\}$

where $i$ indexes the points along a path between the centroid and an
inversion point, and $l$ indexes the paths. The minimization drives the
optimization toward a centroid $c$ that has a minimum overall distance to all
reference points in terms of our new metrics.

##  Appendix E FID and per-class accuracy with a pre-trained classifier

Figures 2(b)-2(c) in the main paper provide FID and per-class accuracy of a
pre-trained classifier for images generated by Stable Diffusion. Tables 2,1
further evaluate different interpolation and centroid evaluations with these
metrics. Here we provide more details about these experiments.

FID (Fréchet inception distance) [59, 25] was calculated between generated
images (typically 100 images per class) and real ImageNet1k test images (50
images for each class). To calculate the FID score, we used the features of a
pre-trained ImageNet1k classifier. Specifically, we used InceptionV3 with
$64$-dim feature vectors.

For per-class accuracy, we follow [51] and used a SoTA pre-trained ImageNet
classifier. Specifically, we used MaxViT image classification model [62] pre-
trained on ImageNet-21k (21843 Google specific instance of ImageNet-22k) and
fine-tuned on ImageNet-1k. It achieves a top-1 of 88.53% accuracy and 98.64
top-5 accuracy on the test-set of ImageNet. This classifier measures the
correctness of generated images.

##  Appendix F A Spherical Projection baseline

The "Spherical Projection" baseline [9] in the main paper tries to find a
centroid on a sphere. Specifically, it optimizes a point that minimizes the
sum of geodesic distances between all input points (inversions). This can be
formulated as follows. Given $z^{1}_{T},z^{2}_{T},\ldots,z^{k}_{T}$ points
(projected) on a sphere with radius $R$. We would like to find a point $c$
such that

$c^{*}=~{}\underset{c}{\text{argmin}}{R\sum^{k}_{i=1}{\arccos(c\cdot
z^{i}_{T}})}\quad,$ (6)

where $R\cdot\arccos(c\cdot z^{i}_{T})$ the "arc length" and the geodesic
distance between the centroid $c$ and another point $z^{i}_{T}$. These extrema
can be solved using Lagrangian multipliers, with $c$ on the sphere, as the
constraint: $L(c,\lambda)=\sum_{i=1}^{k}R\cdot\arccos\left(c\cdot
z^{i}_{T}\right)+\lambda(R-c\cdot c)$. Taking partial derivatives results
with:

$\frac{\partial L}{\partial
C_{j}}(c,\lambda)=-R\sum_{i=1}^{k}\frac{z^{i}_{T}(j)}{\sqrt{1-(c\cdot
z^{i}_{T})^{2}}}-2\lambda c_{j}$ (7) $\frac{\partial
L}{\partial\lambda}(c,\lambda)=R^{2}-c\cdot c\quad,$ (8)

where $z^{i}_{T}(j)$ is the $j-$th element of $z^{i}_{T}$. The extrema is
then:

$c=m\sum_{i=1}^{k}\frac{z^{i}_{T}}{\sqrt{1-(c\cdot z^{i}_{T})^{2}}},$ (9)

where $m$ a constant selected such that the second normalization constraint
holds. An iterative equation was obtained, and the centroid is determined
through iterative processes until convergence is achieved.

Fig. S 1: In the video frame interpolation task, NAO can produce intermediate
frames through interpolating between seed inversions of two provided frames
(I1 and I2). The red overlaid lines are at fixed locations and were added for
sake of visualization. The generated intermediate frames from seed
interpolation are highly close to the ground truth, and contain the subtle
temporal changes.

| PSNR | SSIM
---|---|---
| (Higher is better) | (Lower is better)
MCVD [65] | 18.646 | 0.705
LDMVFI [14] | 25.541 | 0.833
NAO-path (ours) | 25.413 | 0.813

Table S 1: Results of NAO for video frame interpolation task on the DAVIS [41]
dataset. NAO achieves comparable results to existing methods.

##  Appendix G Video frame interpolation

We further applied our approach to the task of video frame interpolation,
aiming to generate an intermediate frame between existing consecutive frames
(I1 and I2) in a video sequence. Using NAO, we interpolated between the seed
inversions of two given frame images, optimizing 50 points, and generating an
image from the middle points. Example images in Figure S1 illustrate NAO’s
ability to create intermediate frames. To further demonstrate NAO’s
capability, we conducted also a quantitative evaluation. To this end, we
followed the experiment outlined in [14, 65], focusing on the DAVIS dataset
[41]: a widely acknowledged benchmark for Video Frame interpolation tasks. The
evaluation of predicted frames against the ground truth was carried out using
established metrics like PSNR and SSIM. Table S1 demonstrates that our
interpolation approach achieves comparable results to existing methods
specifically tailored for video interpolation, despite solely using a pre-
trained text-to-image model.

##  Appendix H Experiments on few-shot and long-tail learning

We provide here additional information and implementation details regarding
experiments on Few-shot and Long-tail learning.

#### Few-shot learning:

We followed the training protocol of [48] and evaluated NAO for 600 episodes.
At each episode, base and novel classes are randomly selected. We generated
1000 additional samples for each novel class using NAO and SeedSelect [51].
Specifically, SeedSelect was initialized with centroids and samples from the
interpolation paths found by NAO using the 5-shot images provided during meta-
testing. The mean accuracy is calculated for each episode. Finally, we
reported the mean accuracies along all episodes with 95% confidence intervals.

For NAO + SeedSelect to generate 1000 total images for each class, we
optimized paths with a length of 200 points between the centroid of the 5-shot
training samples and the samples themselves.

To ensure a fair comparison, we employed the ResNet-12 architecture as the
backbone network. On top of the feature extractor, we incorporate a two-layer
MLP for N-way classification. A stochastic gradient descent (SGD) optimizer is
used with a momentum of 0.9. The learning rates for the backbone and
classifier are respectively set to 0.025 and 0.05, accompanied by a weight
decay of 5e-4. During training, standard data augmentations such as color
jittering, random crop, and horizontal flips are applied within a minibatch
consisting of 250 images.

#### Long-tail learning:

NAO was further evaluated as a semantic augmentation approach on the ImageNet-
LT [38] benchmark. Here, we generated samples for each class until the
combined total of real and generated samples equaled the count of the class
with the highest number of samples in the dataset, resulting in a uniform data
distribution. Similarly to the approach used for few-shot benchmarks, we
generated images for each class by initializing SeedSelect with centroids and
samples obtained from interpolation paths discovered between training samples
to their centroid. We optimized paths ranging in length from 10 to 200,
adjusting based on the available number of samples for each class. We measure
the accuracy for each class and report the mean for all classes.

For a fair comparison, we follow prior work [38, 50, 29] and train a ResNet-50
backbone. The backbone was trained on real and generated data using a
stochastic gradient descent (SGD) optimizer with a momentum of 0.9.
Additionally, we used the cosine learning-rate schedule to train the network.
During training, standard data augmentations such as color jittering, random
crop, and horizontal flips are applied within a minibatch consisting of 250
images.

#### Hyper-parameter tuning:

We determined the number of training epochs (early-stopping), and the learning
rate of the backbone network using the validation set provided.

Fig. S 2: Images generated by different schedulers on the same interpolation
path between two seeds optimized by NAO. Images were generated with 20
denoising steps. Our approach results in high-quality images regardless of the
scheduler used in the method.

##  Appendix I Sensitivity to scheduler

In the main paper, we used DDIM scheduler both for the diffusion forward
process (seed to image) and inversion process (image to seed). Here we show
that our method can be also used with another deterministic scheduler. Figure
S2 shows images generated by sampling from NAO-path using DDIM [58], DEIS [73]
Determenistic Euler Scheduler [31] and Heun [31] schedulers. Note that while
Euler and Heun schedulers rescale the input latent, they still initiate their
process with a sample from a multi-variant Gaussian distribution. This
characteristic allows our approach to be potentially applicable to all
schedulers.

##  Appendix J Ablation study

###  J.1 Dependence on the number of samples (“shots”)

Fig. S 3: Effect of number of few-shot samples on accuracy of a pre-trained
ImageNet-1k classifier.  Fig. S 4: Additional qualitative comparison for
different interpolation methods between two image seeds.  Fig. S 5: Additional
qualitative comparison for different centroid optimization approaches on
common and rare concepts of ImageNet1k. We further initialized SeedSelect [51]
with the centroids and run it for up to 3 iterations ($\sim$15 sec on a single
A100 GPU). While alternative methods demonstrate satisfactory performance with
common concepts, they frequently lack visual credibility when employed with
uncommon objects.

Figure S3 illustrates the comparison between our suggested method NAO and
SeedSelect [51] regarding the impact of varying the number of training samples
(#shots) on generation quality. The evaluation was performed on tail classes
of ImageNet1k [51]. We tested per-class tail accuracy for changing the number
of training samples using a pre-trained ImageNet1k classifier (see Section E).
The increase in the number of training samples leads to the generation of more
semantically correct images and it is evident that NAO achieves superior
accuracy results compared to SeedSelect [51].

###  J.2 Diversity

We follow [51] and analyze the diversity of images generated by NAO. To
analyze the diversity of our generated images, we used two metrics: The number
of statistically different beans (NDB) [45] and the entropy across clusters.
While NDB tries to locate mode collapse, the entropy metric makes sure that
the generated images are diverse.

Specifically, we divided the test set of each ImageNet class into 50 clusters
(one for each test sample). Subsequently, we allocated the images generated by
NAO to these clusters and evaluated the NDB and entropy within the clusters.
On average, the entropy across all classes was 3.9 bits, whereas the test set
clusters had an entropy of 5 bits. Furthermore, the NDB value was 2.21. These
findings indicate that the generated images possess a level of diversity
comparable to real images, demonstrating a lack of mode collapse.

We acknowledge that there are potential alternative methods for quantifying
diversity [63]. The exploration of diversity is still in its early stages and
has not been thoroughly examined. However, this area holds significant
potential for future research and investigation, offering valuable
opportunities for further exploration.

Fig. S 6: Analysis examining the influence of various $\delta$ values on the
FID of images generated along the interpolation trajectory. The investigation
demonstrates that extremely low or high delta values detrimentally impact FID
scores. Such values either result in excessive proximity between points,
causing overlap or overly disperse them, steering the path into regions of low
likelihood.

###  J.3 $\delta$ analysis

In the experiments done in the main paper, we set $\delta$ to be
$||z_{1}-z_{2}||/n$, where $z_{1}$ and $z_{2}$ are seed inversions of real
images ($x_{1}$ and $x_{2}$), and $n$ is the number of interpolation points to
be optimized. We further conducted an in-depth analysis to investigate the
impact of different delta values on the FID of images generated along the
interpolation path. Figure S6 presents the results of this analysis. The
findings reveal that excessively low or high delta values adversely affect the
FID, as they either constrain the points too closely together, causing
overlap, or spread them too far apart, leading the path into low-likelihood
regions.

##  Appendix K Additional qualitative results

Figure S4 provides qualitative analysis and compares NAO to LERP and SLERP
[56]. Figure S5 further provides a qualitative comparison between NAO and
other centroid finding approaches. It is evident from the figures that our
approach samples better paths resulting in higher quality images, particularly
in rare concepts.