# Diversified in-domain synthesis with efficient fine-tuning for few-shot
classification

Victor G. Turrisi da Costa 1 Nicola Dall’Asen††footnotemark:  1,2 Yiming Wang3
Nicu Sebe1 Elisa Ricci1,3
1 University of Trento 2 University of Pisa 3 Fondazione Bruno Kessler Victor
G. Turrisi da Costa and Nicola Dall’Asen contributed equally.

###### Abstract

Few-shot image classification aims to learn an image classifier using only a
small set of labeled examples per class. A recent research direction for
improving few-shot classifiers involves augmenting the labelled samples with
synthetic images created by state-of-the-art text-to-image generation models.
Following this trend, we propose Diversified In-domain Synthesis with
Efficient Fine-tuning (DISEF), a novel approach which addresses the
generalization challenge in few-shot learning using synthetic data. DISEF
consists of two main components. First, we propose a novel text-to-image
augmentation pipeline that, by leveraging the real samples and their rich
semantics coming from an advanced captioning model, promotes in-domain sample
diversity for better generalization. Second, we emphasize the importance of
effective model fine-tuning in few-shot recognition, proposing to use Low-Rank
Adaptation (LoRA) for joint adaptation of the text and image encoders in a
Vision Language Model. We validate our method in ten different benchmarks,
consistently outperforming baselines and establishing a new state-of-the-art
for few-shot classification. Code is available at
https://github.com/vturrisi/disef.

##  1 Introduction

Few-shot classification aims to develop models that can categorize new
samples, _i.e._ the _query_ set, into a set of classes by only learning from a
very limited number of labeled samples of each class, _i.e._ the _support_
set. This is especially relevant in application domains where collecting
extensive labeled datasets is expensive or unfeasible. The main challenge in
few-shot classification lies in how to learn generalizable representation from
such a limited support set. To address this issue, over the years, researchers
have proposed different approaches, e.g. based on meta-learning [13], transfer
learning [51], metric learning [17, 20] and, more recently, on fine-tuning
vision and language models (VLM) [49, 50, 7, 40], complemented by data
augmentation techniques.

Figure 1: Radar chart comparing our proposed method, DISEF (with and without
synthetic data) against other fine-tuning methods with Vision Language Models,
namely Classifier Tuning [9], Text Prompt Tuning (TPT) [48], Visual Prompt
Tuning (VPT) [16] and a combination of VPT and TPT. Different angles
correspond to the ten different benchmark datasets.

Generative models and, in particular, text-to-image diffusion models [42, 12,
36] have reached a significant level of maturity that enables the synthesis of
highly photo-realistic images. These advances have spurred a new research
trend that investigates the use of synthetic data in image recognition tasks
[43, 39, 50]. However, existing work mostly focuses on investigating novel
generative models [50] to enrich the data with access to the distribution of
the whole dataset, or on providing recipes to improve model pre-training [43].

Surprisingly, how to use synthetic data to enhance classification models in
data-scarce scenarios, _i.e_. the few-shot learning setting, has not been well
studied. He _et al_. [9] explored the idea of adopting synthetically generated
data to increase the diversity of the support set, in order to learn
classification models which are more robust and able to generalize better. Lin
_et al_. [22] explored the potential of generated images to improve object
detection in a few-shot scenario. However, when adopting text-to-image
generation models as in [9, 22, 41], the process for creating synthetic data
consists of simply requesting a photo of a [CLASS] without constraints on
details. This may yield out-of-domain synthetic images, _e.g_. images of the
correct class but with a very distinct viewpoint or visual style. Such
diverse, but out-of-domain generation can degrade the classification
performance [41].

Another aspect that so far has been overlooked in the literature of learning
from limited real (and synthetic) data is related to strategies for fine-
tuning pre-trained models. These strategies are fundamental to maximally boost
the generalization capability of classifiers, especially in a few-show
setting. The emergence of large vision and language models (VLM), such as CLIP
[33] has opened up different possibilities for performing model fine-tuning on
downstream tasks. Notable approaches include text prompt tuning [48, 49, 40],
visual prompt tuning [16] or multi-modal prompt tuning [18]. Recently,
parameter-efficient fine-tuning (PEFT) methods [35, 14, 15, 21, 47, 8, 23]
have attracted attention, becoming the de facto strategy for fine-tuning
models in the text domain. These approaches mainly consist of adding (smaller)
adapters [35, 14], performing prompt tuning [21, 47, 8] or learning low-rank
update matrices to the parameters [15]. Nevertheless, these techniques, apart
from prompt tuning, have seen limited adoption in computer vision, with only a
few exceptions [16, 3].

In this paper, we propose to tackle the problem of few-shot classification
with synthetic data by innovating key recipes in data synthesis and parameter-
efficient model fine-tuning. Our approach, Diversified In-domain Synthesis
with Efficient Fine-tuning (DISEF), brings two main contributions. First, we
propose a novel text-to-image augmentation pipeline which encourages in-domain
data synthesis and promotes sample diversity. Precisely, we leverage state-of-
the-art captioning models, such as LLaVA [25], to produce textual descriptions
of support images that are rich in semantic details. Such descriptions are
then exploited as anchors in a cross-sample manner to promote in-domain
synthesis with diversity. We also incorporate real samples of the support set
in the noise injection procedure of the diffusion-based generative model, so
that the generated images have a consistent visual appearance to the real
images of the same class. Second, we demonstrate that effective model fine-
tuning is a key factor in few-shot recognition and we, for the first time,
leverage Low-Rank Adaptation (LoRA) [15] for jointly adapting the text and
vision encoders of a VLM in the few-shot scenario. Our approach provides the
flexibility to choose which components of the VLM are updated without
modifying the original architecture or the input of the networks, as we do not
rely on learnable prompts in any of the modalities. This represents a powerful
yet simple adaptation strategy when learning in data-constrained scenarios. We
validate our proposed method on ten benchmarks for few-shot classification
(see also Fig. 1), consistently outperforming baseline methods and setting a
new state-of-the-art.

The Contributions of our work can be summarized as below:

  * •

We introduce DISEF, a new framework for few-shot classification that leverages
synthetic data and parameter-efficient fine-tuning.

  * •

For generating synthetic images, we propose a novel augmentation pipeline that
leverages both support images and their captions for producing diverse but in-
domain training samples.

  * •

For fine-tuning, we shed new light on the importance of model fine-tuning in
the context of few-shot classification with VLMs and propose to leverage LoRA
[15] for adapting both the vision and text encoders.

  * •

We achieve the new state-of-the-art for few-shot image classification on
extensive benchmarks, proving the effectiveness of our proposed method.

##  2 Related work

We review related works on few-shot classification based on VLMs, the use of
generative models for image data augmentation, and parameter-efficient fine-
tuning methods.

Few-shot recognition with VLMs. Several recent works have shown the
effectiveness of VLMs when applied to few-shot classification. For instance,
Zhou _et al_. [48] proposed an adaptation method for VLMs that consists of
adding learnable prompts to the text encoder, similar to a simple Text Prompt
Tuning (TPT). In [49], this approach was further extended to condition the
learnable text prompts on the input image. Gao _et al_. [7] introduced CLIP-
Adapter, which adds learnable linear layers to the output of both the vision
and language encoders, freezing CLIP’s parameters. Differently, Katthak _et
al_. [18] employed learnable prompts at multiple layers of the text and vision
encoders, conditioning the visual prompts by linearly projecting the text
prompts. Shi _et al_. [40] proposed LoGoPrompt, a method derived from the
observations that images with written class names and natural images activate
the same neurons in CLIP. However, none of these works focused on enhancing
the performance of few-shot classifiers derived from VLMs by using synthetic
data.

Synthetic data as additional training data. In the last few years, diffusion
models [42, 12] have emerged as a powerful approach for generating highly
realistic images. By driving the generation process with text [36] or with any
other conditioning signals [46] (depth maps, pose, semantic maps, etc.)
diffusion models have also demonstrated high flexibility. These recent
advances in image generation have encouraged researchers to investigate the
use of synthetic images within recognition tasks, with the purpose of
enriching the original data distribution. For instance, Tian _et al_. [43]
showed that it is possible to pre-train large models in a self-supervised way
by using only synthetic data generated by Stable Diffusion [36], achieving
competitive performance with models pre-trained with real data. Zhou _et al_.
[50] introduced Diffusion Inversion, an approach that uses a pre-trained
Stable Diffusion model to create synthetic datasets. It ensures coverage of
the original data manifold while producing novel samples that complete the
training domain by generating variations of real samples, thus facilitating
generalization. Azizi _et al_. [1] found that fine-tuning Imagen [38] on
ImageNet leads to synthetic data that better matches the training data
distribution. They also showed that by jointly using real and synthetic data
the accuracy of a model on ImageNet can be improved. Shipard _et al_. [41]
showed how a recognition model trained only on synthetic data can generalize
well on real data. More recently, He _et al_. [9] proposed to leverage GLIDE
[30] to generate synthetic data in a few- and zero-shot scenario. They also
implemented a filtering scheme to eliminate samples from one class that
exhibit close proximity to another class in the feature space. This work is
the most closely related to ours, as they specifically used VLMs and a
synthetic data generation approach for improving few-shot classification
performance. However, their language enhancement pipeline, a fine-tuned T5
model [34], does not provide semantically rich captions, but only
grammatically correct sentences. Differently from them, we start from captions
obtained by an image captioning model, thus providing the text-to-image
pipeline with a grounded and detail-rich description. Additionally, the use of
GLIDE, a two-stage pipeline, as the generative backbone makes the model
impractical for larger datasets, especially given the fact they require up to
800 images per class. These reasons, united with the controllable guidance
coming from real images, make our generation pipeline more suitable for the
few-shot scenario.

Parameter efficient fine-tuning methods. The advent of large deep learning
models and their widespread use in several applications has prompted numerous
endeavors to develop techniques for fine-tuning these models on downstream
tasks. For instance, in [35] and [14] small trainable MLPs are added within
the layers of a frozen pre-trained model. LLaMA-Adapter [47] and LLaMA-Adapter
v2 [8] used learnable prompts at different layers of a transformer
architecture, with the effect of progressively injecting information into the
main model using a zero-initialized attention mechanism. Jia _et al_. [16]
introduced Visual Prompt Tuning (VPT), a method inspired by the prompt tuning
literature from the text domain. It adds learnable visual prompts in either a
shallow way, where the prompts are added only at the input of the model, or a
deep way, where multiple different prompts are employed in each layer. Hu _et
al_. [15] introduced LoRA, a method which learns low-rank update matrices to
adapt a pre-trained model. Chavan _et al_. [3] proposed GLoRA, a modification
of LoRA which uses a different set of learnable parameters per layer and
leverages an evolutionary algorithm to select which learnable parameters to
add per layer. In this work, we explore for the first time the use of LoRA to
concurrently adapt the text and vision encoders of a VLM in the context of
few-shot recognition. Unlike previous approaches, we show that we do not need
to adapt only a single modality, nor do we need to restrict the adaptation to
simple learnable prompts. This offers additional flexibility in terms of what
can be used to adapt VLMs in the few-shot scenario, hopefully opening an
avenue to connect fine-tuning methods for large language models (LLM) to fine-
tuning methods for VLMs in data-constrained scenarios.

##  3 Proposed approach

Figure 2: Proposed method for few-shot learning. At the top, we present our
adaptation strategy. Starting from one of the few-shot images available, we
generate additional training data by applying our Synthetic Augmentation
Pipeline (SAP). Then, we treat both the real images and the synthetic images
in the same manner. Considering an image $x$, we forward it through our vision
encoder ($V$) to produce visual features $f_{v}$. In parallel, we forward all
the class labels, combined with a pre-defined prompt template through the text
encoder ($T$), generating text features $f_{t}$. Then, we compute the
$logits=sim(f_{v},f_{t})$, where $sim$ is the cosine similarity function, and
the cross-entropy loss $L_{ce}$. Finally, instead of updating all the
parameters of our model, we modify the original model by adding LoRA layers in
the query ($\mathcal{Q}$) and value ($\mathcal{V}$) embeddings of the self-
attention ($\mathcal{SA}$) layers in both the text and vision encoders At the
bottom, we show the SAP procedure. Starting from the set of images
$\mathcal{X}$, we caption them with an image captioning model
($\mathcal{ICM}$), while also projecting them in the Stable Diffusion latent
space. We inject noise into the latent vectors and run the reverse diffusion
process with shuffled, per class, captions, obtaining synthetic images
$\mathcal{X}^{\prime}$. Lastly, we filter $\mathcal{X}^{\prime}$ with CLIP to
retain only synthetic images which are classified as their intended class.

We consider the support set $\mathcal{X}=\\{(x,y)\\}^{K,N}$, containing tuples
in the format of an image $x$ and its label $y$, where $K$ is the number of
images per class (_i.e_. the number of shots), and $N$ the number of classes.
The goal is to learn a function ${\mathcal{F}}_{\theta}(x)\rightarrow y$ that
maps the image $x$ to its corresponding label $y$. Usually,
$\mathcal{F}_{\theta}(\cdot)$ is represented by a neural network parametrized
by $\theta$. Moreover, methods addressing few-shot image classification often
exploit a pre-trained model and restrain the training to just a small subset
of parameters [48, 49, 18, 40]. Following a similar philosophy, we build our
method DISEF (as shown in Figure 2) on top of a pre-trained VLM, where we
fine-tune only a small fraction of the model with a novel application of
parameter-efficient fine-tuning and a new Synthetic Augmentation Pipeline
(SAP).

SAP is built on top of Stable Diffusion [36] with the objective of promoting
diversified in-domain sample generation. Different from [9], which only uses
one image and an augmented prompt at a time to generate synthetic data, SAP
generates a set of synthetic images $\mathcal{X}^{\prime}$ by leveraging the
whole support set together with their captions, which are rich in semantic
details. Specifically, given the support subset $\mathcal{X}_{y}$
corresponding to a class label $y$, we extract its image captions with an off-
the-shelf image captioning model $\mathcal{ICM}$, obtaining a set of captions
$C_{y}$. To encourage in-domain generation, we start the generation by
embedding each real sample $x_{i}$ in the latent space of Stable Diffusion.
Then, we inject noise to perturb the low-level details while maintaining high-
level class semantics. Meanwhile, to encourage diversity, we condition the
generation with the caption of another sample $x_{j}\in\mathcal{X}_{y}$. In
this way, we can generate diverse samples that are semantically correct and
visually in-domain. Furthermore, to reduce the number of incorrect images, we
filter them based on their similarity to the textual representation of their
desired class.

At adaptation time, we treat both synthetic and real samples in a similar way,
and we use the joint set
$\hat{\mathcal{X}}=\mathcal{X}\cup\mathcal{X}^{\prime}$ to fine-tune the
model. Specifically, we add LoRA [15] layers to the query ($\mathcal{Q}$) and
value ($\mathcal{V}$) embeddings on the self-attention ($\mathcal{SA}$) layers
of both vision and the text encoders. This allows us to adapt a pre-trained
VLM for both modalities, while at the same time being efficient.

###  3.1 Synthetic Augmentation Pipeline

SAP augments the support set $\mathcal{X}$ with additional synthetic data
points that are interpolated within the domain inferred by $\mathcal{X}$. The
in-domain synthesis requires the synthesized images not only to belong to the
same semantic class but also to exhibit similar visual patterns as the real
images. On the other hand, diversity is needed for the data to be useful for
training and requires the synthesized images to present different semantic
details. Our proposed SAP encourages such in-domain diversity from two novel
perspectives: first, by manipulating the visual representation of real images
during the diffusion process, and second, by leveraging semantically detailed
captions provided by a captioning model.

Condition on real images. For a real image $x_{i}\in\mathcal{X}_{y}$, we
leverage a pre-trained VAE [36] to project the image into the latent space,
obtaining a latent vector $z_{i}$. Then, Gaussian noise $\epsilon^{t}$ is
added to perturb the low-level visual details without changing the semantic
class, obtaining a noisy latent vector $z_{i}^{t}$, for a given step $t$,
following the diffusion process. Note that the amount of noise injected can be
dataset-dependant as it correlates to the granularity among classes, i.e. the
more coarse-grained the classes are, the more noise we can inject without
modifying the semantic class.

Condition on semantic-detailed captions. Given the support set
$\mathcal{X}_{y}$ of class $y$, we first generate their captions using the
image captioning model $\mathcal{ICM}$ obtaining a set of captions $C_{y}$.
These add more details such as size, color, composition, and action, that are
far richer than the standard CLIP prompt of a photo of a [CLASS].

Synthetic image generation. We input $z_{i}^{t}$ to the generator
$\mathcal{G}$, together with a randomly sampled caption $p_{j}\in{C}_{y},j\neq
i$. The generation follows the classifier-free guidance procedure of diffusion
models [11], where we obtain a reconstructed latent vector
$\hat{z_{i}}=\mathcal{G}(z_{i}^{t},p_{j})$. We then decode $\hat{z_{i}}$ to
obtain the synthetic image. By using $p_{j}$ and the noisy latent vector
$z_{i}^{t}$ to condition the generation, we are more likely to generate
visually in-domain samples corresponding to the same class, with different,
yet plausible, semantic details.

Synthetic image filtering. We further apply a filtering strategy to ensure the
generated samples are aligned to the semantic class. More formally, we
leverage the zero-shot classifier $W_{zs}\in\mathbb{R}^{N\times d}$ from CLIP
for a particular dataset, where $d$ is the CLIP dimensionality. First, we
compute the latent representation for the generated images by forwarding them
through CLIP’s vision encoder, obtaining $f_{v}=V(x^{\prime})$ for each
$x^{\prime}\in\mathcal{X}^{\prime}$. We obtain the predicted class for each
sample $\hat{y}=\operatorname*{arg\,max}(f_{v}\,W_{zs}^{\intercal})$, where
$f_{v}$ and $W_{zs}$ are $L_{2}$ normalized. Lastly, we remove from
$\mathcal{X}^{\prime}$ the samples whose $\hat{y}$ is not the correct class.

At each generation, we choose a random real image in the support set
$\mathcal{X}$ and a random caption corresponding to a different image of the
same class, followed by the sample filtering procedure. We repeat the
generation for each class until we obtain a fixed number of synthetic samples
$K_{syn}$ for each class, forming the final synthetic set
$\mathcal{X}^{\prime}$.

###  3.2 Parameter-efficient VLM fine-tuning

Inspired by recent advances in LLM fine-tuning, we propose to integrate LoRA
[15], originally proposed for the text domain, to fine-tune a VLM for few-shot
learning. Specifically, LoRA proposes to add small, low-rank update matrices
to adapt a pre-trained model. Given a pre-trained dense layer $o=W\,i$, where
$W\in\mathbb{R}^{m\times n}$ is the weight matrix, $i$ is the input and $o$ is
the output, LoRA modifies the layer by adding an update matrix $\Delta W$,
that can be further decomposed, as:

$o=W\,i+\Delta W\,i=W\,i+B\,A\,i,$ (1)

where $B\in\mathbb{R}^{m\times r}$, $A\in\mathbb{R}^{r\times n}$ are two low-
rank matrices, $m$ and $n$ are the number of rows and columns in the original
weight matrix $W$, and $r$ is the rank of the update matrix, with
$r\ll\min(m,n)$. This decomposition greatly reduces the number of trainable
parameters to just a fraction of the original layer with no impact on
inference time, as it is possible to merge $W$ and $B\,A$ during inference.
$A$ is initialized by sampling from a Gaussian distribution and $B$ is
initialized with zeros to avoid disrupting the model so that it produces the
same results as the original model at the beginning of the fine-tuning
process. Lastly, $\Delta W\,i$ is further scaled with a weight
$\frac{\alpha}{r}$, where $\alpha$ is a hyperparameter.

As the domain shift between the pre-training data and the few-shot task is not
specific to a single modality, we propose to add LoRA layers to both the
vision encoder $V$ and the text encoder $T$, instead of only the text encoder.
Specifically, we integrate the LoRA layers to the query ($\mathcal{Q}$) and
value ($\mathcal{V}$) embeddings of the self-attention ($\mathcal{SA}$)
layers, the most effective placement as demonstrated in [15].

###  3.3 Training

Considering a single image $x$, we first forward it through the vision encoder
$V$ producing visual features $f_{v}=V(x)$. In parallel, we forward each class
label $c$ with a default prompt through the text encoder $T$, producing
$f_{t}=\\{T(c)\\}^{N}$. This default prompt is selected on a per-dataset
basis, borrowing the same prompts in CLIP [33]. Then, we compute the logits
as:

$logits=sim(f_{v},f_{t}),$ (2)

where $sim(\cdot)$ is the cosine similarity function,
$f_{t}=\\{f_{t_{1}},f_{t_{2}},...,f_{t_{N}}\\}$ and $N$ is the total number of
classes.

Lastly, a cross-entropy loss $L_{ce}$ is used to compute the gradients of the
model. As our batch is composed of both real and synthetic images, we compute
a separate loss for each part, $L_{real}$ and $L_{syn}$, respectively. We
compute the final loss as the weighted average of the two cross-entropy losses
$L_{real}$ and $L_{syn}$:

$L_{ce}=\lambda L_{real}+(1-\lambda)L_{syn},\text{ where }\lambda\geq 0.5$ (3)

##  4 Experiments

Table 1: 16-shot results on all the datasets averaged across 3 seeds. We
highlight best and second best results.

| Caltech101 | DTD | EuroSAT | FGVC Aircraft | ImageNet | Oxford Pets | Stanford Cars | SUN397 | Food 101 | Flowers 102 | Avg
---|---|---|---|---|---|---|---|---|---|---|---
Classifier Tuning [9] | 96.01 | 73.64 | 87.13 | 46.73 | 73.41 | 92.81 | 82.55 | 76.16 | 87.28 | 86.52 | 80.22
Visual Prompt Tuning [16] | 95.40 | 66.06 | 92.33 | 36.21 | 69.57 | 91.84 | 69.01 | 70.47 | 86.99 | 90.95 | 66.31
Text Prompt Tuning [48] | 95.23 | 70.73 | 87.05 | 45.50 | 67.97 | 89.89 | 81.40 | 72.95 | 83.65 | 97.62 | 79.20
VPT + TPT | 95.75 | 72.02 | 90.42 | 48.03 | 67.34 | 88.87 | 81.99 | 72.58 | 83.70 | 98.12 | 79.88
DISEF w/o synth | 96.71 | 74.31 | 94.25 | 62.09 | 73.64 | 93.88 | 88.17 | 77.01 | 87.66 | 98.77 | 84.65
DISEF | 96.94 | 75.36 | 94.31 | 63.89 | 73.94 | 94.33 | 88.63 | 77.43 | 87.11 | 98.85 | 85.08

We evaluate our proposed method DISEF in comparison with state-of-the-art
methods using ten benchmark datasets under two scenarios. We describe the
experimental setup followed by a discussion of the main comparisons. Moreover,
we ablate our main design choices regarding SAP and the PEFT adaptation and
present both qualitative and quantitative results to show their effectiveness.

Datasets. We consider ten commonly used datasets for few-shot image
classifier, namely, ImageNet [37] and Caltech101 [6] for generic object
classification, SUN397 [44] for scene understanding, DTD [4] for texture
classification, FGVC Aircraft [28], Oxford Pets [32], Stanford Cars [19], Food
101 [2] and Flowers 102 [31] for fine-grained classification and EuroSAT [10]
for satellite image classification.

Evaluation protocol. We evaluate our method in two scenarios. In the first
(default) scenario, following [9], we train and evaluate our model on all
classes. In the second (base/new) scenario, we use half of the classes for
training and evaluation (base classes) and the other half for evaluation only
(new classes), following the split in [49]. We report only the top-1 accuracy
in the default scenario while, for the base/new scenario, we report the top-1
accuracy for the base classes (Base), the new classes (New), and their
harmonic mean (H).

Baseline methods. In the default scenario, we compare our method against a
list of PEFT methods including the Classifier Tuning from [9], TPT (CoOp from
[48]); VPT [16], and a combination of both VPT and TPT. For the base/new
scenario, we compare our method with the state-of-the-art methods including
CoOp [48], CoCoOp [49], MaPLe [18] and LoGoPrompt [40]. We did not include
Classifier Tuning as it consists of training a linear classifier that needs to
know all classes a priori, thus not being applicable to unseen classes.

Implementation details. We build our method on top of CLIP [33] with ViT-B/16
as vision encoder. Our model is trained for $50$ epochs in all datasets, using
AdamW [26] as optimizer, a cosine learning rate scheduler without warmup, and
a weight decay of $1\times 10^{-3}$. Hyperparameters such as the learning rate
and batch size are tuned on a per-dataset basis, as
$lr\in\\{2^{-i},i\in[8,15]\\}$ and $bs\in\\{16,32,64,128,256\\}$. We apply
traditional data augmentation techniques, such as RandomResizedCrop,
RandAugment [5], MixUp [45], CutMix [45] and label smoothing [29], depending
on the dataset. In experiments involving only real data, we set LoRA’s $r=16$,
$\alpha=32$, and $dropout=0.1$ for both the vision and text encoders, while
for experiments with real and synthetic data, we set $r=64$ and
$\alpha\in\\{32,64\\}$ for the vision encoder depending on the dataset. With
the introduction of synthetic data during fine-tuning, we notice that a higher
rank for the vision encoder can be beneficial as there is more visual data to
leverage. We set $\lambda\in\\{0.5,0.6,0.7,0.8\\}$ when using synthetic data.
For SAP, we use LLaVA 1.5 [24] for captioning and a CLIP with ViT-H as vision
encoder for filtering. We choose as diffusion model a fine-tuned version of
Stable Diffusion 1.5 [36] on realistic images. As sampler, we use DPM-Solver++
[27] with 20 steps. We fixed the classifier-free guidance factor to 8. For the
noising procedure, we perform a per-dataset choice of either 25% or 75% of the
noising schedule. We set the number of synthetic samples per class
$K_{syn}=64$. More details are available in the Supplementary Material.

##  5 Results

Table 2: 16-shot results on all datasets for base/new classes averaged across
3 seeds.

(a) Average across datasets

| Base | New | H
---|---|---|---
CLIP [33] | 69.22 | 73.89 | 71.48
CoOp [48] | 82.02 | 63.94 | 72.08
CoCoOp [49] | 80.28 | 71.51 | 75.65
MaPLe [18] | 82.21 |  74.75 | 78.33
LoGoPrompt [40] | 84.29 | 74.35 | 79.02
DISEF w/o synth | 85.16 | 74.07 | 78.95
DISEF |  86.71 | 74.53 |  79.74

(b) Caltech101

| Base | New | H
---|---|---|---
CLIP [33] | 96.84 | 94.00 | 95.40
CoOp [48] | 98.00 | 89.81 | 93.73
CoCoOp [49] | 97.96 | 93.81 | 95.84
MaPLe [18] | 97.74 |  94.36 | 96.02
LoGoPrompt [40] | 98.19 | 93.78 | 95.93
DISEF w/o synth |  98.58 | 92.98 | 95.70
DISEF | 98.49 | 93.85 |  96.12

(c) DTD

| Base | New | H
---|---|---|---
CLIP [33] | 53.24 | 59.90 | 56.37
CoOp [48] | 79.44 | 41.18 | 54.24
CoCoOp [49] | 77.01 | 56.00 | 64.85
MaPLe [18] | 80.36 | 59.18 | 68.16
LoGoPrompt [40] | 82.87 | 60.14 | 69.70
DISEF w/o synth | 82.21 |  65.62 |  72.98
DISEF |  83.57 | 64.37 | 72.70

(d) EuroSAT

| Base | New | H
---|---|---|---
CLIP [33] | 56.48 | 64.05 | 60.03
CoOp [48] | 92.19 | 54.74 | 68.69
CoCoOp [49] | 87.49 | 60.04 | 71.21
MaPLe [18] | 94.07 | 73.23 | 82.35
LoGoPrompt [40] | 93.67 | 69.44 | 79.75
DISEF w/o synth | 97.72 |  73.44 |  83.86
DISEF |  97.97 | 72.86 | 83.53

(e) FGVC Aircraft

| Base | New | H
---|---|---|---
CLIP [33] | 27.19 |  36.29 | 31.09
CoOp [48] | 40.44 | 22.30 | 28.75
CoCoOp [49] | 33.41 | 23.71 | 27.74
MaPLe [18] | 37.44 | 35.61 | 36.50
LoGoPrompt [40] | 45.98 | 34.67 | 39.53
DISEF w/o synth | 48.50 | 32.23 | 38.71
DISEF |  55.94 | 34.33 |  42.53

(f) ImageNet

| Base | New | H
---|---|---|---
CLIP [33] | 72.43 | 68.14 | 70.22
CoOp [48] | 76.47 | 67.88 | 71.92
CoCoOp [49] | 75.98 | 70.43 | 73.10
MaPLe [18] | 76.66 | 70.54 | 73.47
LoGoPrompt [40] | 76.74 | 70.83 | 73.66
DISEF w/o synth | 77.64 | 69.98 | 73.61
DISEF |  78.34 |  71.04 |  74.51

(g) Oxford Pets

| Base | New | H
---|---|---|---
CLIP [33] | 91.17 | 97.26 | 94.12
CoOp [48] | 93.67 | 95.29 | 94.47
CoCoOp [49] | 95.20 | 97.69 | 96.43
MaPLe [18] | 95.43 |  97.76 | 96.58
LoGoPrompt [40] | 96.07 | 96.31 | 96.18
DISEF w/o synth | 96.19 | 94.98 | 95.58
DISEF |  96.40 | 97.67 |  97.03

(h) Stanford Cars

| Base | New | H
---|---|---|---
CLIP [33] | 63.37 |  74.89 | 68.65
CoOp [48] | 78.12 | 60.40 | 68.13
CoCoOp [49] | 70.49 | 73.59 | 72.01
MaPLe [18] | 72.94 | 74.00 | 73.47
LoGoPrompt [40] | 78.36 | 72.39 | 75.26
DISEF w/o synth | 80.98 | 68.28 | 74.09
DISEF |  84.07 | 68.75 |  75.63

(i) SUN397

| Base | New | H
---|---|---|---
CLIP [33] | 69.36 | 75.35 | 72.23
CoOp [48] | 80.60 | 65.89 | 72.51
CoCoOp [49] | 79.74 | 76.86 | 78.27
MaPLe [18] | 80.82 |  78.70 | 79.75
LoGoPrompt [40] | 81.20 | 78.12 | 79.63
DISEF w/o synth | 81.87 | 78.46 | 80.13
DISEF |  83.14 | 78.22 |  80.61

(j) Food 101

| Base | New | H
---|---|---|---
CLIP [33] | 90.10 | 91.22 | 90.66
CoOp [48] | 88.33 | 82.26 | 85.19
CoCoOp [49] | 90.70 | 91.29 | 90.99
MaPLe [18] | 90.71 |  92.05 |  91.38
LoGoPrompt [40] | 90.82 | 91.41 | 91.11
DISEF w/o synth |  91.00 | 90.51 | 90.75
DISEF | 90.56 | 91.48 | 91.02

(k) Flowers 102

| Base | New | H
---|---|---|---
CLIP [33] | 72.08 |  77.80 | 74.83
CoOp [48] | 97.60 | 59.67 | 74.06
CoCoOp [49] | 94.87 | 71.75 | 81.71
MaPLe [18] | 95.92 | 72.46 | 82.56
LoGoPrompt [40] |  99.05 | 76.52 |  86.34
DISEF w/o synth | 96.96 | 74.23 | 84.09
DISEF | 98.64 | 72.72 | 83.72

In Table 1, we present the results for the default scenario for all compared
methods and ours with and without synthetic data. First, we can see that TPT
is better than VPT on average, although for EuroSAT, ImageNet, Oxford Pets,
and Food 101, VPT outperforms TPT. Nonetheless, DISEF significantly
outperforms all compared methods, with and without synthetic data, on almost
all datasets, except for Food 101 where using synthetic data reduces
performance. We suspect that this might be due to the limited generation
capability in certain very fine-grained classes. Although classifier tuning is
competitive with DISEF in Caltech101, ImageNet, and Flowers 102, it is limited
to only known classes, thus inappropriate for the base/new scenario, which
involves unseen classes.

In addition, Table 2(k) presents the full results on the base/new scenario. We
can see that DISEF without synthetic data can improve the Base accuracy while
being competitive with previous methods in New accuracy and H. This shows that
our fine-tuning strategy is effective without compromising the
generalizability of the model to new classes. Nonetheless, when adding our
synthetic data, we further boost the base accuracy by +1.55% on average while
also improving New accuracy by +0.46%, leading to a new state-of-the-art with
an improvement of +0.72% compared to the previous best-performing method
LoGoPrompt [40] in H.

###  5.1 Ablation

In this section, we demonstrate the effectiveness of our design choices both
in the augmentation pipeline and the PEFT part. For SAP, we ablate the effect
of using the detail-rich captions obtained by $\mathcal{ICM}$ (LLaVA in our
case), and the use of real samples as anchors during generation. Additionally,
we also demonstrate the effectiveness of the generated data when using it as
augmentation with other fine-tuning methods, namely Classifier Tuning, VPT,
TPT, and VPT + TPT. For PEFT, we demonstrate that applying LoRA in both the
image and text encoders of CLIP is important for reaching state-of-the-art
results. To demonstrate the effectiveness of our design choices, we select
four representative datasets featuring different image recognition tasks:
scene recognition (SUN397), fine-grained classes (FGVC Aircraft), satellite
images (EuroSAT), and texture classification (DTD).

How to generate the synthetic data? We show the effect of using captions that
are rich in semantic details and using real images as anchors for the
generation. We ablate the individual contribution of each component in Table
3. We can see that the optimal choice of which components to use is dataset-
dependant, correlated with the nature of the data and the domain shift from
the datasets diffusion models are trained on (e.g. EuroSAT is a challenging
domain to generate). On the other hand, we can see that when both components
are used together, SAP is most beneficial for model generalization by bringing
consistent improvement in all the datasets, leaving the amount of noise to
inject as the only parameter to choose on a per-dataset basis.

Table 3: Ablation of the SAP components.

LLaVA caption |  Real images guidance | EuroSAT | DTD |  FGVC Aircraft | SUN397
---|---|---|---|---|---
✗ | ✗ | 93.98 | 74.43 | 63.64 | 74.40
✗ | ✓ | 94.36 | 74.59 | 63.60 | 74.90
✓ | ✗ | 94.08 | 76.04 | 64.42 | 75.22
✓ | ✓ | 94.31 | 75.36 | 63.89 | 77.43

Does SAP generalize to different fine-tuning methods? We experiment with
different tuning methods augmented with the synthetic data generated by SAP,
demonstrating that SAP is generally effective, regardless of fine-tuning
methods. We choose the same baselines as Table 1 and train them in the default
scenario using the same data used for DISEF. From Table 4, we see that all the
baseline methods are able to achieve better results by including our synthetic
data, proving that the images we generate are effective regardless of the
method used to fine-tune CLIP. The only notable exception is classifier
tuning, where the use of synthetic data for FGVC Aircraft marginally decreases
the performance. We deem this result to be the saturation of the learnable
parameters of the classifier and the difficulty of this recognition task,
which makes adding more data non-beneficial given the low-parameter regime.

Table 4: Effect of using synthetic data on the baselines in the default
scenario.

| EuroSAT | DTD |  FGVC Aircraft | SUN397
---|---|---|---|---
Classifier Tuning [9] | 87.13 | 73.64 | 46.73 | 76.16
+synthetic | 88.14 | 73.78 | 46.47 | 76.90
Visual Prompt Tuning [16] | 92.33 | 66.06 | 36.21 | 70.47
\+ synthetic | 93.01 | 68.62 | 39.86 | 71.70
Text Prompt Tuning [48] | 87.05 | 70.73 | 45.50 | 72.95
\+ synthetic | 86.68 | 71.41 | 46.81 | 73.90
VPT + TPT [16, 48] | 90.42 | 72.02 | 48.03 | 72.58
\+ synthetic | 92.88 | 72.62 | 50.55 | 73.79
Ours (LoRA [15]) | 94.25 | 74.31 | 62.09 | 77.01
\+ synthetic (DISEF) | 94.31 | 75.36 | 63.89 | 77.43

Which encoder should we adapt? We also ablate the effect of adding LoRA only
to the vision encoder, only the text encoder, or both. As shown in Table 5, we
can see that depending on the dataset, there are different effects of adapting
only one of the modalities. This depends mostly on how large the domain shift
is for each modality w.r.t. the pre-training datasets of CLIP. For example, in
datasets such as EuroSAT, where the text modality is already well-separated by
CLIP but the visual aspects of the data greatly differ from the pre-training
data, adapting only the vision encoder is enough to bring results close to our
best. It is also interesting to see that adding LoRA only to the vision
encoder is a better choice than adding it only to the text encoder. This is
different from adding learnable prompts, as we saw that TPT achieved better
results than VPT. When using LoRA, we can properly adapt the vision encoder,
which results in higher gains than adapting only the text encoder.
Nonetheless, both encoders are complementary, therefore adapting the text
encoder contributes to further improvements.

Table 5: Effect of LoRA for adaptation.

LoRA | DTD | EuroSAT |  FGVC Aircraft | SUN397
---|---|---|---|---
Vision Only | 72.79 | 93.88 | 60.03 | 72.76
\+ synthetic | 73.78 | 93.30 | 62.50 | 75.18
Text Only | 70.92 | 86.97 | 46.68 | 76.04
\+ synthetic | 71.93 | 86.91 | 47.06 | 75.90
Both (Ours) | 74.31 | 94.25 | 62.09 | 77.01
\+ synthetic | 75.36 | 94.31 | 63.89 | 77.43

###  5.2 Qualitative results

We present some generated samples in Figure 3 by our SAP with both the
semantically rich captions and the guidance from the real samples in
$\mathcal{X}$, by GLIDE in [9] and a Stable Diffusion [36] model with standard
CLIP prompts as input. We can see that the samples generated by GLIDE in [9]
generally present lower quality with fewer details, _e.g_. they miss the
basket in the basketball court in the third column. On the other hand,
generation from scratch with a generic prompt (second row) leads to generally
good-looking samples, however also lacks details, _e.g_. the missing front
part of the airplane or the general absence of barrels in the barrel storage.
Instead, our SAP generation (the last row) exhibits a higher degree of detail
with the presence of the correct and complete object in the images.

He et al. [9] |  |  |
---|---|---|---
Stable Diffusion [36] |  |  |
SAP (Full) |  |  |
Figure 3: He et al. [9] generation pipeline compared with our SAP and Stable
Diffusion [36]. The generated classes are, from left to right, ”a Boeing
737-200”, ”a wine cellar barrel storage” and ”an outdoor basketball court”.
Images from [9] exhibit wrong proportions, a general lack of details, or
missing fundamental objects. Images naively generated from scratch with a
generic prompt exhibit cropped objects or wrong semantics. Our SAP, which
leverages real data and rich captions, does not exhibit such drawbacks.

##  6 Conclusion

We presented DISEF, a novel method for few-shot learning with two main
contributions: a novel design of synthetic augmentation pipeline for in-domain
diversity and a novel application of a parameter-efficient fine-tuning method
to VLMs for effective adaptation. For the synthetic data generation, we
propose to apply noise to the real samples and use them as a starting point
for the diffusion process. Additionally, we leverage textual information, in
the form of captions, to enrich the details of the generated data. Finally, we
leveraged LoRA for fine-tuning both the vision and the text encoder, allowing
us to effectively adapt a pre-trained VLM with limited samples. Our method was
evaluated against previous approaches in two scenarios on ten benchmark
datasets, achieving the new state-of-the-art in few-shot image classification.

Limitations. Using a pre-trained diffusion model as the generative backbone
might limit the semantics of the generated images for domains on which the
model was not explicitly trained, e.g. medical images. Furthermore, since we
are adapting a pre-trained CLIP model, if the domain between the pre-training
data and the few-shot data differs too much, fine-tuning might not suffice.

Broader Societal Impacts. Although few-shot image classification has already
alleviated the need for massive training data, data privacy still remains a
concern in case the support set contains sensitive information. The potential
biases can be exacerbated by limited training samples, leading to unfair
outcomes.

## References

  * Azizi et al. [2023] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  _Transactions on Machine Learning Research (TMLR)_ , 2023.
  * Bossard et al. [2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _Proceedings of the IEEE/CVF European Conference on Computer vision (ECCV)_ , 2014.
  * Chavan et al. [2023] Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen.  One-for-all: Generalized lora for parameter-efficient fine-tuning.  _arXiv preprint arXiv:2306.07967_ , 2023.
  * Cimpoi et al. [2014] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014.
  * Cubuk et al. [2020] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Fei-Fei et al. [2006] Li Fei-Fei, Robert Fergus, and Pietro Perona.  One-shot learning of object categories.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2006.
  * Gao et al. [2023a] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _International Journal of Computer Vision (IJCV)_ , 2023a.
  * Gao et al. [2023b] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao.  Llama-adapter v2: Parameter-efficient visual instruction model.  _arXiv preprint arXiv:2304.15010_ , 2023b.
  * He et al. [2023] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2023.
  * Helber et al. [2019] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Hospedales et al. [2022] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2022.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2019.
  * Hu et al. [2022] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2022.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  _arXiv preprint arXiv:2203.12119_ , 2022.
  * Jung et al. [2022] Deunsol Jung, Dahyun Kang, Suha Kwak, and Minsu Cho.  Few-shot metric learning: Online adaptation of embedding for retrieval.  In _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , 2022.
  * Khattak et al. [2023] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) workshops_ , 2013.
  * Li et al. [2023] Xiaoxu Li, Xiaochen Yang, Zhanyu Ma, and Jing-Hao Xue.  Deep metric learning for few-shot image classification: A review of recent developments.  _Pattern Recognition_ , 2023.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the Association for Computational Linguistics (ACL)_ , 2021.
  * Lin et al. [2023] Shaobo Lin, Kun Wang, Xingyu Zeng, and Rui Zhao.  Explore the power of synthetic data on few-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Liu et al. [2022] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023b.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * Lu et al. [2023] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2023.
  * Maji et al. [2013] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Müller et al. [2019] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton.  When does label smoothing help?  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2022.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _Proceedings of the IEEE Indian Conference on Computer Vision, Graphics & Image Processing_, 2008.
  * Parkhi et al. [2012] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_ , 2020.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  _International Journal of Computer Vision (IJCV)_ , 2015.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Sariyildiz et al. [2023] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.  Fake it till you make it: Learning transferable representations from synthetic imagenet clones.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Shi and Yang [2023] Cheng Shi and Sibei Yang.  Logoprompt: Synthetic text images can be good visual prompts for vision-language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Shipard et al. [2023] Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, and Clinton Fookes.  Diversity is definitely needed: Improving model-agnostic zero-shot classification via stable diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshops_ , 2023.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _Proceedings of the International Conference of Machine Learning (ICML) workshops_ , 2015.
  * Tian et al. [2023] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.  Stablerep: Synthetic images from text-to-image models make strong visual representation learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2023.
  * Xiao et al. [2010] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba.  SUN database: Large-scale scene recognition from abbey to zoo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2010.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023a.
  * Zhang et al. [2023b] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_ , 2023b.
  * Zhou et al. [2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision (IJCV)_ , 2022a.
  * Zhou et al. [2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022b.
  * Zhou et al. [2023] Yongchao Zhou, Hshmat Sahak, and Jimmy Ba.  Training on thin air: Improve image classification with generated data.  _arXiv preprint arXiv:2305.15316_ , 2023.
  * Zhuang et al. [2019] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 2019.

Supplementary Material

In the supplementary material, we provide additional results. In Section A, we
provide an ablation on the number of synthetic images used during training. In
Section B, we show the effect of using less real samples for DISEF and all the
baselines in the default scenario. In Section C, we provide a complete
overview of all the parameters used for our DISEF for all the components,
divided by dataset. Finally, in Section D, we show additional results
generated by our SAP, proving the effectiveness of our augmentation pipeline.

##  Appendix A Effect of using less synthetic images

We study the effect of the number of synthetic data used in the fine-tuning
process. In Table 6, we present these results on the four representative
datasets for our ablation studies. We adopt the 16-shot default scenario and
vary the number of synthetic images from 4 to 64 (our default value). We can
see that a higher number of synthetic images leads to better performance in
all the datasets. Nonetheless, we also notice that the performance achieved by
64 synthetic samples is only marginally better than the ones obtained by 32
synthetic samples on all datasets, indicating that performance is saturating
at 64. Such saturation may be due to the limited amount of diversity the 16
original samples can provide for generating synthetic data. Additionally,
there is an increased computational burden on generating a significantly
higher number of synthetic images.

##  Appendix B Fewer shots

In this section, we ablate the effect of training with fewer shots, _i.e_.,
fewer real images per class. More specifically, we evaluate the baseline
methods, classifier tuning [9], VPT [16], TPT [48] and VPT+TPT, on the default
scenario, using a ViT-B/16 as vision encoder. We consider the same four
ablation datasets, EuroSAT, DTD, FGVC Aircraft, and SUN397, and conduct
experiments with 1, 2, 4, and 8 shots. Since our synthetic data generation is
conditioned on the images and their captions, for each fewer-shot experiment,
we re-generate the data such that only those images are used for conditioning,
and we still generate 64 synthetic images. We present these results in Figure
4. First, it is interesting to see that VPT performs much better than TPT with
less real data in DTD, FGVC Aircraft, and SUN397, although it converges to a
much lower value. Nonetheless, we can see that DISEF without synthetic data
performs better than all methods regardless of the number of shots. When
coupled with synthetic data, our method improves on all datasets and with
different shots, with the exception of EuroSAT for 4-shot. Even in the most
extreme case 1-shot where we use only one image and its caption as a starting
point, our generation is still powerful and versatile enough to achieve the
most competitive accuracy.

EuroSAT | DTD
---|---
|
FGVC Aircraft | SUN397
|

Figure 4: Fewer shots for the default scenario for Classifier Tuning [9], VPT
[16], TPT [48], VPT + TPT, DISEF w/o synthetic data and DISEF. Table 6: Effect
of reducing the number of synthetic images.

Number of synthetic shots | EuroSAT | DTD | FGVC Aircraft | SUN397
---|---|---|---|---
4 | 92.42 | 69.98 | 57.69 | 76.08
8 | 92.90 | 73.07 | 61.93 | 77.14
16 | 94.16 | 74.05 | 63.33 | 77.39
32 | 94.13 | 75.00 | 63.79 | 77.41
64 (ours) | 94.31 | 75.36 | 63.89 | 77.43

##  Appendix C Hyperparameters per dataset

We perform a grid search on the hyperparameters of both the fine-tuning and
generation process on a per-dataset basis and present these values in Table 8.
For the fine-tuning hyperparameters, we experimented with different batch
sizes, learning rates, LoRA’s $r$ and $\alpha$ for the vision encoder,
$\lambda$, cutmix, mixup and label-smoothing. Batch size and learning rate
were searched in the ranges of $\\{16,32,64,128,256\\}$ and
$\\{2^{-i},i\in[8,15]\\}$. For CutMix, MixUp and label-smoothing we used
either $[0.0,0.0,0.0]$, $[0.1,0.1,0.1]$ or $[0.8,1.0,0.1]$ for each respective
parameter. LoRA’s $r$ and $\alpha$ for the vision encoder were searched in the
ranges of $\\{16,32,64\\}$ and $\\{32,64\\}$.
$\text{Augmentation}=\text{True}$ means using RandAugment [5] with default
parameters followed by RandomResizedCrop, from the official Pytorch
implementation. For the synthetic parameters, we search only for the step to
start generation from, in $\\{5,15\\}$, with $5$ meaning an initial sample
closer to Gaussian noise, while $15$ implies a sample closer to the initial
image.

##  Appendix D Synthetic samples generated by SAP

We show more images generated by our SAP in Figure 5 and Figure 6. Figure 5
shows the in-domain synthesis capabilities of our SAP, while Figure 6 shows
randomly-picked samples for all the datasets, from left to right, Caltech101,
DTD, Eurosat, FGVC Aircraft, ImageNet, Oxford Pets, Stanford Cars, SUN397,
Food 101, and Flowers 102.

In Figure 5, in the first row (highlighted with a red box), we should a real
image from the 16 available shots for that class. Below them, we show images
of the same class generated by our SAP. We can notice two different behaviors
depending on the amount of noise injected. For the datasets where classes are
more well-separated, _e.g_. DTD and SUN397, we inject a higher amount of
noise, as reported in Table 8. This leads to more diverse samples w.r.t. the
original image, while the generated images maintain their semantic properties.
For datasets where the classes are more overlapping in the visual domain,
_e.g_. EuroSAT or FGVC Aircraft, we prefer fidelity over diversity, as
injecting too much noise might destroy semantic-relevant information,
therefore for these datasets we choose to stop at step $5$ of the diffusion
process. This leads to images which closely resemble original samples with
variations in the details, _e.g_. the livery of the plane.

In Figure 6, we show different classes (top to bottom) for each dataset (left
to right), providing the corresponding class names in Table 7. Overall, the
generated images exhibit high fidelity and a high degree of realism, even for
the dataset where we inject a high degree of noise, _e.g_. Caltech101, DTD,
ImageNet, SUN397, and Food 101 (we refer to Table 8 for more details).

A detailed inspection of the images highlights some hallucinations in the
generation, _e.g_. the shape of the helicopter in the eleventh row of
Caltech101, the missing hole in the donut in the eleventh row of DTD, the
hallucinated small plane in the fifth row of FGVC Aircraft, or the weird hand
posture in the sixth row of ImageNet. Although these would constitute errors
in traditional generative tasks, for the recognition tasks, we deem these
inconsistencies as not influential as the semantics of the original class are
preserved. Nevertheless, our SAP can fully exploit the rich LLaVA captions to
generate high-quality, realistic, and detail-rich images of the desired class.
Moreover, the use of real samples as the starting point for the generation
allows the model to better preserve the semantics of each class, _e.g_. the
proportion of the planes in FGVC Aircraft. Additionally, our generation is
able to distinguish and generate very similar classes, _e.g_. it can
distinguish between bolognese spaghetti and carbonara spaghetti (fourth and
fifth row of Food 101, although one could argue basil does not belong on
carbonara).

Figure 5: Qualitative example of diverse in-domain synthetis of our SAP. In red box samples from the ground truth 16 shots, below different synthesis results from our model. EuroSAT | DTD | FGVC Aircraft | SUN397
---|---|---|---
AnnualCrop | Banded | Supermarine Spitfire | Bar
|  |  |
|  |  |
|  |  |
|  |  |
|  |  |
Figure 6: Randomly selected images generated by our SAP for the ten datasets. Class names can be found in Table 7. Caltech101 | DTD | EuroSAT | FGVCA | ImageNet | OxfordPets | Cars | SUN397 | Food 101 | Flowers102
---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |
Table 7: Class names of the generated images in Figure 6.

Caltech | DTD | EuroSAT | FGVC Aircraft | ImageNet | Oxford Pets | Stanford Cars | SUN397 | Food 101 | Flowers 102
---|---|---|---|---|---|---|---|---|---
Anchor | Banded | AnnualCrop | Airbus A300 | Tench | Abyssinian | 1991 Volkswagen Golf | Balcony Exterior | Apple Pie | Japanese Anemone
Ant | Blotchy | Forest | Antonov An-12 | Brambling | American Bulldog | 1993 Geo Metro | Balcony Interior | Baby Back Ribs | Thorn Apple
Barrel | Braided | Herbaceous Vegetation | ATR-72 | Palace | American Pitbull | 1998 Nissan 240SX | Bow Window Indoor | Baklava | Azalea
Beaver | Bubbly | Highway | Beechcraft 1900 | Bench | Birman | 2001 Lamborghini Diablo | Bow Window Outdoor | Bolognese | Balloon Flower
Binoculars | Chequered | Industrial | Boeing 737-200 | Planetarium | Bombay | 2007 BMW Serie 6 | Car Interior Backseat | Carbonara | Camelia
Bonsai | Cobwebbed | Pasture | Bombardier Aerospace Global Express | Camera | Chihuahua | 2007 Cadillac Escalade | Car Interior Frontseat | Spring Rolls | Desert Rose
Buddha | Cracked | Permanent Crop | Canadair Challenger 600 | Fridge | Great Pyrenees | 2007 Ford F-150 | Cathedral Indoor | Tiramisu | Fire Lily
Butterfly | Honeycombed | Residential | Cessna 172 | Scale | Pomeranian | 2008 Audi RS4 | Wine Barrel Storage | Pizza | Giant White Arum Lily
Cougar | Interlaced | River | Dassault Aviation Falcon 900 | Sport Car | Russian Blue | 2009 Bentley Arnage | Wine Bottle Storage | Paella | Globe Flower
Electric Guitar | Marbled | SeaLake | de Havilland DH-82 | Totem Pole | Samoyed | 2009 Hummer H2 | Volcano | Macarons | Bearded Iris
Helicopter | Sprinkled | River | Ilyushin Il-76 | Vault | Shiba Inu | 2012 Porsche Panamera | Vineyard | Lasagna | Primula
Laptop | Veined | Highway | Supermarine Spitfire | Window Shade | Yorkshire Terrier | 2012 Volkswagen Beetle | Baseball Field | Ice Cream | Sunflower

Table 8: Per-dataset parameters of DISEF

| Caltech101 | DTD | EuroSAT | FGVC Aircraft | ImageNet | Oxford Pets | Stanford Cars | SUN397 | Food 101 | Flowers 102
---|---|---|---|---|---|---|---|---|---|---
Model | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16 | ViT-B/16
Optimizer | AdamW | AdamW | AdamW | AdamW | AdamW | AdamW | AdamW | AdamW | AdamW | AdamW
Batch size $bs$ | 128 | 16 | 16 | 16 | 256 | 128 | 16 | 128 | 64 | 32
Learning rate $lr$ | $2^{-12}$ | $2^{-15}$ | $2^{-12}$ | $2^{-11}$ | $2^{-15}$ | $2^{-15}$ | $2^{-13}$ | $2^{-14}$ | $2^{-14}$ | $2^{-12}$
Weight decay | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
$lr$ scheduler | Cosine | Cosine | Cosine | Cosine | Cosine | Cosine | Cosine | Cosine | Cosine | Cosine
Real/synth weight $\lambda$ | 0.8/0.2 | 0.5/0.5 | 0.8/0.2 | 0.8/0.2 | 0.8/0.2 | 0.7/0.3 | 0.8/0.2 | 0.8/0.2 | 0.5/0.5 | 0.8/0.2
Vision Encoder LoRA $r$ | 64 | 64 | 16 | 64 | 64 | 16 | 64 | 64 | 64 | 64
Vision Encoder LoRA $\alpha$ | 64 | 64 | 32 | 32 | 64 | 32 | 64 | 32 | 32 | 32
Vision Encoder LoRA dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Text Encoder LoRA $r$ | 16 | 16 | 16 | 16 | 16 | 16 | 16 | 16 | 16 | 16
Text Encoder LoRA $\alpha$ | 32 | 32 | 32 | 32 | 32 | 32 | 32 | 32 | 32 | 32
Text Encoder LoRA dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Augmentation | True | True | True | True | True | True | True | True | True | True
Cutmix | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.0 | 0.1 | 0.1 | 0.1 | 0.1
Mixup | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.0 | 0.1 | 0.1 | 0.1 | 0.1
Label-smoothing | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.0 | 0.1 | 0.1 | 0.1 | 0.1
Diffusion Sampler | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++ | DPM-Solver++
CFG Strength | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8
Number of diffusion steps | 20 | 20 | 20 | 20 | 20 | 20 | 20 | 20 | 20 | 20
Number of noising steps | 5 | 5 | 15 | 15 | 5 | 15 | 15 | 5 | 15 | 15


N(r)|R|_{X^{r}_{\mu}}|w|_{X^{r}_{\mu}}.$

Gathering these estimates, together with (5.7), we observe

$\frac{d}{dt}\tilde{E}_{r}(\mathbb{W})\lesssim\mu^{2}({\varepsilon}+\beta)|R|_{X^{r}_{\mu}}(\tilde{E}_{r}(\mathbf{W}))^{\frac{1}{2}}+N(r+1)\tilde{E}_{r}(\mathbf{W}).$

Now, a simple application of Grönwall’s inequality and (5.7) yields

$|(\eta,w)|_{Y^{r}_{\mu}}\lesssim\mu^{2}({\varepsilon}+\beta)t\>|R|_{X^{r}_{\mu}}e^{N(r+1)t}.$
(7.2)

Finally, to conclude we use that $Y^{r}_{\mu}({\mathbb{R}})\subset
H^{r}({\mathbb{R}})\hookrightarrow L^{\infty}({\mathbb{R}})$ for
$r>\frac{3}{2}$, and (7.2) to get

$\displaystyle|\mathbf{U}-\mathbf{U}^{\text{\tiny
WGN}}|_{L^{\infty}([0,t];{\mathbb{R}})}$
$\displaystyle\lesssim|(\eta,w)|_{L^{\infty}([0,t];Y^{r}_{\mu}({\mathbb{R}}))}$
$\displaystyle\lesssim\mu^{2}({\varepsilon}+\beta)t\>|R|_{X_{\mu}^{r}}e^{N(r+1)t}.$

To conclude, we let $s$ be large enough such that $r+1<s$ to get that

$\displaystyle|\mathbf{U}-\mathbf{U}^{\text{\tiny
WGN}}|_{L^{\infty}([0,t];{\mathbb{R}})}\lesssim\mu^{2}({\varepsilon}+\beta)t,$

for all $t\in[0,\frac{\min\\{\tilde{T},T\\}}{\max\\{{\varepsilon},\beta\\}}]$.

∎

## Acknowledgements

This research was supported by a Trond Mohn Foundation grant. It was also
supported by the Faculty Development Competitive Research Grants Program
2022-2024 of Nazarbayev University: Nonlinear Partial Differential Equations
in Material Science, Ref. 11022021FD2929.

## References

  * [1] Borys Alvarez-Samaniego and David Lannes.  A Nash-Moser theorem for singular evolution equations. Application to the Serre and Green-Naghdi equations.  Indiana Univ. Math. J., 57(1):97–131, 2008.
  * [2] Mathias Nikolai Arnesen.  Existence of solitary-wave solutions to nonlocal equations.  Discrete Contin. Dyn. Syst., 36(7):3483–3510, 2016.
  * [3] J. L. Bona and R. Smith.  The initial-value problem for the Korteweg-de Vries equation.  Philos. Trans. Roy. Soc. London Ser. A, 278(1287):555–601, 1975\.
  * [4] John D. Carter.  Bidirectional equations as models of waves on shallow water.  Wave Motion, 82:51–61, 2018.
  * [5] F. Chazel, M. Benoit, A. Ern, and S. Piperno.  A double-layer Boussinesq-type model for highly nonlinear and dispersive waves.  Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci., 465(2108):2319–2346, 2009.
  * [6] F. Chazel, D. Lannes, and F. Marche.  Numerical simulation of strongly nonlinear and dispersive waves using a Green-Naghdi model.  J. Sci. Comput., 48(1-3):105–116, 2011.
  * [7] M. W. Dingemans.  Comparison of computations with Boussinesq-like models and laboratory measurements.  Technical report, Deltares, Delft, 1994.
  * [8] E. Dinvay and D. Nilsson.  Solitary wave solutions of a Whitham-Boussinesq system.  Nonlinear Anal. Real World Appl., 60:Paper No. 103280, 24, 2021\.
  * [9] Evgueni Dinvay.  On well-posedness of a dispersive system of the Whitham–Boussinesq type.  Applied Mathematics Letters, 88:13–20, 2019.
  * [10] Evgueni Dinvay.  Well-posedness for a Whitham-Boussinesq system with surface tension.  Math. Phys. Anal. Geom., 23(2):Paper No. 23, 27, 2020.
  * [11] Evgueni Dinvay, Denys Dutykh, and Henrik Kalisch.  A comparative study of bi-directional systems.  Appl. Numer. Math., 141:248–262, 2019.
  * [12] Evgueni Dinvay, Sigmund Selberg, and Achenef Tesfahun.  Well-posedness for a dispersive system of the Whitham-Boussinesq type.  arXiv:1902.09438v3, 2019.
  * [13] V. Duchêne, S. Israwi, and R. Talhouk.  A new class of two-layer Green-Naghdi systems with improved frequency dispersion.  Stud. Appl. Math., 137(3):356–415, 2016.
  * [14] Vincent Duchêne.  Many models for water waves.  A unified theoretical approach. Open Math Notes, OMN:202109.111309, 2021\.
  * [15] Vincent Duchêne and Samer Israwi.  Well-posedness of the Green-Naghdi and Boussinesq-Peregrine systems.  Ann. Math. Blaise Pascal, 25(1):21–74, 2018.
  * [16] Vincent Duchêne, Dag Nilsson, and Erik Wahlén.  Solitary Wave Solutions to a Class of Modified Green–Naghdi Systems.  J. Math. Fluid Mech., 20(3):1059–1091, 2018.
  * [17] Mats Ehrnström, Mark D. Groves, and Erik Wahlén.  On the existence and stability of solitary-wave solutions to a class of evolution equations of Whitham type.  Nonlinearity, 25(10):2903–2936, 2012.
  * [18] Mats Ehrnström, Mathew A. Johnson, and Kyle M. Claassen.  Existence of a highest wave in a fully dispersive two-way shallow water model.  Arch. Ration. Mech. Anal., 231(3):1635–1673, 2019.
  * [19] Mats Ehrnström and Henrik Kalisch.  Traveling waves for the Whitham equation.  Differential Integral Equations, 22(11-12):1193–1210, 2009.
  * [20] Mats Ehrnström and Erik Wahlén.  On Whitham’s conjecture of a highest cusped wave for a nonlocal dispersive equation.  Ann. Inst. H. Poincaré Anal. Non Linéaire, 36(6):1603–1637, 2019.
  * [21] Louis Emerald.  Rigorous derivation from the water waves equations of some full dispersion shallow water models.  SIAM J. Math. Anal., 53(4):3772–3800, 2021.
  * [22] Louis Emerald.  Rigorous derivation of the Whitham equations from the water waves equations in the shallow water regime.  Nonlinearity, 34(11):7470–7509, 2021.
  * [23] Louis Emerald.  Local well-posedness result for a class of non-local quasi-linear systems and its application to the justification of whitham-boussinesq systems, 2022.
  * [24] Maurício F. Gobbi, James T. Kirby, and Ge Wei.  A fully nonlinear Boussinesq model for surface waves. II. Extension to $O(kh)^{4}$.  J. Fluid Mech., 405:181–210, 2000.
  * [25] Vera Mikyoung Hur.  Wave breaking in the Whitham equation.  Adv. Math., 317:410–437, 2017.
  * [26] Vera Mikyoung Hur and Mathew A. Johnson.  Modulational instability in the Whitham equation with surface tension and vorticity.  Nonlinear Anal., 129:104–118, 2015.
  * [27] Vera Mikyoung Hur and Ashish Pandey.  Modulational instability in a full-dispersion shallow water model.  Studies in Applied Mathematics, 08 2016.
  * [28] Samer Israwi.  Large time existence for 1D Green-Naghdi equations.  Nonlinear Anal., 74(1):81–93, 2011.
  * [29] Mathew A. Johnson and J. Douglas Wright.  Generalized solitary waves in the gravity-capillary Whitham equation.  Stud. Appl. Math., 144(1):102–130, 2020.
  * [30] Carlos E. Kenig, Gustavo Ponce, and Luis Vega.  Well-posedness and scattering results for the generalized Korteweg-de Vries equation via the contraction principle.  Comm. Pure Appl. Math., 46(4):527–620, 1993.
  * [31] Lannes.  The water waves problem: mathematical analysis and asymptotics.  Mathematical surveys and monographs; volume 188. American Mathematical Society, Rhode Island, United-States, 2013.
  * [32] Felipe Linares and Gustavo Ponce.  Introduction to nonlinear dispersive equations.  Universitext. Springer, New York, second edition, 2015.
  * [33] P. A. Madsen, H. B. Bingham, and Hua Liu.  A new Boussinesq method for fully nonlinear waves from shallow to deep water.  J. Fluid Mech., 462:1–30, 2002.
  * [34] Dag Nilsson and Yuexun Wang.  Solitary wave solutions to a class of Whitham-Boussinesq systems.  Zeitschrift für angewandte Mathematik und Physik, 70(70), 2019\.
  * [35] Ashish Pandey.  The effects of surface tension on modulational instability in full-dispersion water-wave models.  European Journal of Mechanics - B/Fluids, 77:177–182, 2019.
  * [36] Martin Oen Paulsen.  Long time well-posedness of Whitham-Boussinesq systems.  Nonlinearity, 35(12):6284–6348, 2022.
  * [37] Nathan Sanford, Keri Kodama, John D. Carter, and Henrik Kalisch.  Stability of traveling wave solutions to the Whitham equation.  Phys. Lett. A, 378(30-31):2100–2107, 2014.
  * [38] Jean-Claude Saut and Yuexun Wang.  The wave breaking for Whitham-type equations revisited.  SIAM J. Math. Anal., 54(2):2295–2319, 2022.
  * [39] Atanas Stefanov and J. Douglas Wright.  Small amplitude traveling waves in the full-dispersion Whitham equation.  J. Dynam. Differential Equations, 32(1):85–99, 2020.
  * [40] Tien Truong, Erik Wahlén, and Miles H. Wheeler.  Global bifurcation of solitary waves for the Whitham equation.  Math. Ann., 383(3-4):1521–1565, 2022.
  * [41] R.M. Vargas-Magaña and P. Panayotaros.  A Whitham–Boussinesq long-wave model for variable topography.  Wave Motion, 65:156–174, 2016.
  * [42] Ge Wei, James T. Kirby, Stephan T. Grilli, and Ravishankar Subramanya.  A fully nonlinear Boussinesq model for surface waves. I. Highly nonlinear unsteady waves.  J. Fluid Mech., 294:71–92, 1995.
  * [43] G. B. Whitham.  Variational methods and applications to water waves.  Royal Society, London, 299:6–25, 1967.