# Collaboration of Pre-trained Models Makes Better Few-shot Learner

Renrui Zhang1,2, Bohao Li3, Wei Zhang1, Hao Dong4,
Hongsheng Li2, Peng Gao1, Yu Qiao1

###### Abstract

Few-shot classification requires deep neural networks to learn generalized
representations only from limited training images, which is challenging but
significant in low-data regimes. Recently, CLIP-based methods have shown
promising few-shot performance benefited from the contrastive language-image
pre-training. Based on this point, we question if the large-scale pre-training
can alleviate the few-shot data deficiency and also assist the representation
learning by the pre-learned knowledge. In this paper, we propose CoMo, a
Collaboration of pre-trained Models that incorporates diverse prior knowledge
from various pre-training paradigms for better few-shot learning. Our CoMo
includes: CLIP’s language-contrastive knowledge, DINO’s vision-contrastive
knowledge, and DALL-E’s language-generative knowledge. Specifically, CoMo
works in two aspects: few-shot data expansion and diverse knowledge ensemble.
For one, we generate synthetic images via zero-shot DALL-E to enrich the few-
shot training data without any manpower. For the other, we introduce a
learnable Multi-Knowledge Adapter (MK-Adapter) to adaptively blend the
predictions from CLIP and DINO. By such collaboration, CoMo can fully unleash
the potential of different pre-training methods and unify them to perform
state-of-the-art for few-shot classification. We conduct extensive experiments
on 11 datasets to demonstrate the superiority and generalization ability of
our approach.

## Introduction

Convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012) and
transformers (Vaswani et al. 2017) have attained great success on a wide range
of vision tasks, which is based on thorough training with abundant datasets
(Deng et al. 2009). Instead, for some data-deficient and resource-finite
scenarios, few-shot learning (Vinyals et al. 2016; Snell, Swersky, and Zemel
2017) also becomes a research hotspot that learns a well-performed network
under limited images with annotations. Many previous works have been proposed
in this field to enhance model’s generalization capability by meta learning
(Finn, Abbeel, and Levine 2017a; Wang and Hebert 2016), metric learning (Zhang
et al. 2020), and data augmentation (Hariharan and Girshick 2017; Yang, Liu,
and Xu 2021). Recently, CLIP (Radford et al. 2021) pre-trained by large-scale
language-image pairs shows favorable zero-shot transfer ability for open-
vocabulary visual recognition. The follow-up CoOp (Zhou et al. 2021), CLIP-
Adapter (Gao et al. 2021) and Tip-Adapter (Zhang et al. 2021) further extend
it for few-shot classification and achieve superior performance on various
downstream datasets. This indicates that, even if the few-shot training data
is insufficient, the large-scale pre-training has endowed the network with
strong representation ability, which highly benefits the few-shot learning on
downstream domains. Now that there exist various self-supervisory paradigms,
could we adaptively integrate their pre-learned knowledge and collaborate them
to be a better few-shot learner?

Figure 1: The Collaboration Paradigm of CoMo. We adaptively incorporate the
knowledge from three types of pre-training methods and achieve a strong few-
shot learner.

To tackle this issue, we propose CoMo, a Collaboration of Models blending the
knowledge from multiple pre-training paradigms for few-shot image
classification. As shown in Figure 1, we integrate CLIP (Radford et al. 2021),
DINO (Caron et al. 2021), and DALL-E (Ramesh et al. 2021) to provide three
types of prior knowledge for CoMo. Therein, CLIP (Radford et al. 2021) is pre-
trained to produce paired features in the embedding space for every image and
its descriptive text. Guided by texts with different categorical semantics,
CLIP (Radford et al. 2021) can clearly classify the images and contain the
language-contrastive knowledge. DINO follows contrastive self-supervised
learning (Caron et al. 2021) to match the representations between two
transformations of one same image, which is expert at distinguishing different
images with vision-contrastive knowledge. Similar to CLIP (Radford et al.
2021), DALL-E (Ramesh et al. 2021) is also pre-trained by image-text pairs but
learns to predict the encoded image tokens based on the text tokens.
Conditioned on the input text, DALL-E (Ramesh et al. 2021) could leverage the
language-generative knowledge to create high-quality synthetic images in a
zero-shot manner. Therefore, the three models have distinctive pre-training
goals and can provide complementary knowledge to assist the few-shot visual
recognition.

In detail, we make them collaborate from two perspectives: few-shot data
expansion and diverse knowledge ensemble. Firstly, we adopt DALL-E (Ramesh et
al. 2021) to generate additional training images for different categories
based on the domain-specific texts, which enlarges the few-shot training data
but costs no extra manpower for collection and annotation. Then, we propose a
Multi-Knowledge Adapter (MK-Adapter) that adaptively incorporates the
predictions from both CLIP (Radford et al. 2021) and DINO (Caron et al. 2021).
Referring to Tip-Adapter (Zhang et al. 2021), we build a cache model but with
two kinds of keys respectively for the two pre-trained models, which can
produce two classification logits by vision-contrast for the input image.
Regarding CLIP’s zero-shot logits of language-contrast as the baseline, we
reweight the two adapters’ logits via distribution similarity and ensemble all
the three logits as the final output. By fine-tuning only the lightweight MK-
Adapter via expanded training data, CoMo can learn to effectively fuse the
prior knowledge of the three types of pre-training and leverage their
complementary characteristics for better few-shot visual recognition.

Our main contributions are summarized as follows:

  1. 1.

We propose CoMo to incorporate the prior knowledge learned from various pre-
training paradigms for better few-shot learning.

  2. 2.

By collaborating the pre-trained CLIP, DINO, and DALL-E, CoMo enriches the
limited few-shot training data and adaptively ensembles diverse predictions
via the MK-Adapter.

  3. 3.

We conduct thorough experiments on 11 datasets for few-shot classification,
where our CoMo achieves state-of-the-art performance without using extra
annotated data.

## Related Work

### Pre-training of Visual Models

With the breakthroughs in deep learning models (He et al. 2016; Dosovitskiy et
al. 2020; Liu et al. 2021), most modern vision models are based on the
paradigm of pre-training on ImageNet and fine-tuning on downstream tasks (He,
Girshick, and Dollar 2019). Pre-trained models have shown promising
adaptability for various downstream tasks, such as fine-tuning classification
(Kornblith, Shlens, and Le 2019), object detection (Lin et al. 2017) and
semantic segmentation (Chen et al. 2018), etc.

Figure 2: (a) We adopt DALL-E (Ramesh et al. 2021) to generate synthetic
images to expand the limited few-shot training samples. (b) We construct the
Multi-Knowledge Adapter with two kinds of keys to adaptively fuse the
knowledge from CLIP (Radford et al. 2021) and DINO (Caron et al. 2021).

To improve the representation capability by overcoming the constraints of
annotation, unsupervised/self-supervised pre-training has raised wide concerns
by propagating from small annotated samples to large scale unlabeled datasets
(Jing and Tian 2021). Self-supervised learning is initialized as pretext
tasks, such as image restoration from corruption (Vincent et al. 2008; Pathak
et al. 2016; He et al. 2022), pseudo labels (Doersch, Gupta, and Efros 2015;
Noroozi and Favaro 2016) and clustering (Caron et al. 2018). Recently,
contrast learning, which learns representations by contrasting positive pairs
against negative pairs, has gotten well studied for diverse self-supervised
visual representation learning (He et al. 2020; van den Oord, Li, and Vinyals
2018; Chen et al. 2020b, a; Grill et al. 2020a; Caron et al. 2021). Besides,
language-supervised visual pre-training emerges as a novel paradigm closer to
natural visual understanding (Miech et al. 2019; Rampf, Villone, and Frisch
2015; Sharma et al. 2018; Antol et al. 2015), among which CLIP (Radford et al.
2021) obtains powerful zero-shot transferability by contrastive pre-training
on image-text pairs collected from the Internet. In addition, vision-language
pre-training can also promote the zero-shot image generation from text. Open
generative models, such as DALL-E (Ramesh et al. 2021) and CogView (Ding et
al. 2021) pre-trained on large scale image-text pairs are able to generate
images with complex structures and diverse contents conditioned on a given
text.

### Few-shot Adaptation

Few-shot learning highly relies on the transferability of the trained neural
networks. From the perspective of distance measurement, some metric learning
methods learn a metric space by computing the distances from the instances to
novel categories (Snell, Swersky, and Zemel 2017; Sung et al. 2018; Vinyals et
al. 2016). Also, meta-learning is proposed to improve the few-shot adaptation
ability of the models by finding a set of initialized parameters that can
rapidly adapt to novel domains (Finn, Abbeel, and Levine 2017b; Jamal and Qi
2019; Chen et al. 2021; Li et al. 2021). More recently, with the vision-
language pre-training model CLIP (Radford et al. 2021) exhibiting strong zero-
shot adaptation performance, several efforts have started to find efficient
strategies to adapt it to downstream few-shot datasets. Following the recent
research trend of NLP in prompt learning, context optimization (CoOp) (Zhou et
al. 2021) is proposed as a prompt tuning adaptation method by optimizing a set
of learnable prompt tokens. Subsequently, to address the harm of
generalization ability brought by CoOp (Zhou et al. 2021), that is, the
recognition of unknown categories becomes poor, CoCoOp (Zhou et al. 2022)
proposes to train a meta-network to generate image tokens as conditional
inputs for the textual vectors. Further, referring to adapters (Houlsby et al.
2019) in NLP, CLIP-Adapter (Gao et al. 2021) is introduced to fine-tune CLIP
by applying lightweight residual-style bottleneck layers as the adapter. Tip-
Adapter (Zhang et al. 2021) is then proposed as a training-free adaption
method with a constructed key-value cache model. It can also be regarded as a
better initialization of CLIP-Adapter with much faster convergence when fine-
tuning.

Our work proposes the collaboration of multiple pre-trained visual models as a
unified framework and well leverage their strength to adapt CLIP for few-shot
learning: the vision-language model CLIP (Radford et al. 2021) for zero-shot
image classification, the self-supervised visual transformer DINO (Caron et
al. 2021) to provide distinctive visual representations, and the text-to-image
generative model DALL-E (Ramesh et al. 2021) for zero-shot data generation.

## Collaboration of Pre-trained Models

In this section, we first briefly revisit three types of pre-training
paradigms for CoMo. Then, we specifically introduce how our CoMo collaborates
them for few-shot classification.

### Different Pre-training Paradigms

#### Contrastive Vision-Language Pre-training.

The series (Chen et al. 2020a) of contrastive learning between vision and
language learn to map the two modalities into the same embedding space via a
contrastive loss. Driven by web-scale datasets, e.g., 400 million for CLIP
(Radford et al. 2021) and 1.8 billion for ALIGN (Jia et al. 2021), the basic
pre-training target is to minimize the embedding distances of images and their
textual descriptions, while maximize those unpaired ones. By the cross-modal
alignment, we can discriminate images of different categorizes by the texts
with different semantics. We denote such learned prior as language-contrastive
knowledge and adopt CLIP as the representative model for such pre-training
method.

#### Contrastive Vision Pre-training.

As the traditional self-supervised learning methods, vision-contrastive models
(Chen et al. 2020a) focus on the discrimination between different images.
Normally, the positive pairs to be drawn close are two transformations of the
same image, while the optimization of negative pairs (Grill et al. 2020b) is
optional, which can be replaced by a momentum encoder (He et al. 2020) or
cluster assignments (Caron et al. 2020). Recent works reveal that we can learn
self-supervised features without negative pairs between images (Grill et al.
2020a; Caron et al. 2021). Given the strong linear classification capacity,
the pre-trained DINO (Caron et al. 2021) is adopted here to provide vision-
contrastive knowledge for collaboration.

#### Generative Vision-Language Pre-training.

Learned from millions of image-caption pairs, the DALL-E series can generate
language-conditioned images in a zero-shot manner. They are pre-trained to
autoregressively predict the encoded image tokens from the textual tokens of
the captions. With such language-generative knowledge, the pre-trained DALL-E
can be viewed as a free lunch to enlarge the training data without any
manpower. Considering publicity, we select DALL-E-mini (Dayma et al. 2021) as
the representative among DALL-E models.

Figure 3: Adaptive Ensemble of DINO and CLIP via Multi-knowledge Adapter. We
first regard the test image as the query and retrieves CLIP and DINO’s
knowledge from the corresponding two keys. Then, we calculate the distribution
similarities between different classification logits for adaptive ensemble.

### Few-shot Data Expansion

Under the $N$-way $K$-shot settings, we have the few-shot training images
$I_{N,K}$ with labels $L_{N,K}$ that contain $K$ samples for each $N$
categories. Via the zero-shot DALL-E (Dayma et al. 2021), we generate
synthesis images to enrich our limited training data, as shown in Figure 2
(a). For different categories, we adopt a unified simple template, e.g., “a
photo of a [CLASS].”, and put the category name into [CLASS] as the input for
DALL-E. After the generation, we utilize CLIP to filter the top-$K^{\prime}$
best-quality images as the newly-expanded training samples for each category.
Then, we obtain the $N$-category ($K+K^{\prime}$)-sample training images,
formulated as

$\displaystyle I_{N,(K+K^{\prime})}=\\{\text{DALL-E}(T_{N}),\ \ I_{N,K}\\},,$
(1)

where $T_{N}$ denotes the $N$-category textual inputs. We keep $K^{\prime}$
comparable with $K$ to ensure the synthesis quality and also preserve the low-
data regimes. By the pre-trained language-generative knowledge, the data
expansion is totally zero-shot, which not only requires no manpower to collect
or annotate the data, but also alleviates the data deficiency issue inherently
for few-shot learning nearly at no cost.

### Diverse Knowledge Ensemble.

Besides DALL-E for expanding few-shot data, we introduce a Multi-Knowledge
Adapter, termed as MK-Adapter, which constructs a cache model to adaptively
collaborate the pre-trained CLIP (Radford et al. 2021) and DINO (Caron et al.
2021).

#### Adapter Construction.

MK-Adapter adopts the cache-based adapter that retrieves knowledge from
constructed key-value cache model. Different from Tip-Adapter (Zhang et al.
2021) only adapting CLIP, our cache model contains the pre-learned knowledge
from both CLIP and DINO by caching two kinds of keys. Specifically in Figure 3
(b), we first utilize CLIP and DINO to independently extract visual features
of the few-shot training images, formulated as

$\displaystyle F_{\text{CLIP}}$
$\displaystyle=\text{CLIP${}_{vis}$}(I_{N,(K+K^{\prime})});$ (2)
$\displaystyle F_{\text{DINO}}$
$\displaystyle=\text{DINO}(I_{N,(K+K^{\prime})}),$ (3)

where CLIPvis denotes the CLIP’s visual encoder and
$F_{\text{CLIP}},F_{\text{DINO}}\in\mathbb{R}^{N(K+K^{\prime})\times C}$.
Besides the two keys, we convert the few-shot training labels into one-hot
encodings $L_{N,(K+K^{\prime})}\in\mathbb{R}^{N(K+K^{\prime})\times N}$, and
regard them as the only values for both keys. During training, we follow Tip-
Adapter that only enables the cached keys in the adapter to be learnable and
keeps the pre-trained models frozen.

#### Adaptive Ensemble.

For a test image in Figure 3, we first extract its two visual features
$f_{\text{CLIP}},f_{\text{DINO}}\in\mathbb{R}^{1\times C}$ and regard them as
queries to retrieve diverse knowledge from the MK-Adapter. Then, we could
acquire three predicted classification logits
$p_{\text{ZS}},p_{\text{CLIP}},p_{\text{DINO}}\in\mathbb{R}^{1\times N}$,
which are respectively from CLIP’s zero-shot textual alignment and the two
keys of MK-Adapter. We formulate them as

$\displaystyle p_{\text{ZS}}$ $\displaystyle=f_{\text{CLIP}}\
\text{CLIP${}_{tex}$}(T_{N})^{T};$ (4) $\displaystyle p_{\text{CLIP}}$
$\displaystyle=\varphi(f_{\text{CLIP}}F_{\text{CLIP}}^{T})\
L_{N,(K+K^{\prime})};$ (5) $\displaystyle p_{\text{DINO}}$
$\displaystyle=\varphi(f_{\text{DINO}}F_{\text{DINO}}^{T})\
L_{N,(K+K^{\prime})},$ (6)

where CLIPtex represents CLIP’s textual encoder, and
$f_{\text{CLIP}}\&F_{\text{CLIP}}^{T}$ denotes the query-key affinity matrix
of CLIP’s adapter, analogous to DINO’s. $\varphi(x)=\exp(-\beta\cdot(1-x))$
serves as a non-linear modulator to control the sharpness of affinity matrix.
As the language-contrastive $p_{\text{ZS}}$ is pre-trained by 400 million data
and can perform strong zero-shot transfer ability, we regard $p_{\text{ZS}}$
as the prediction baseline and calculate the weights of
$p_{\text{CLIP}},p_{\text{DINO}}$ for ensemble based on their distribution
similarity with $p_{\text{ZS}}$. By this, we can suppress some obviously false
category possibilities in $p_{\text{CLIP}},p_{\text{DINO}}$ and also amplify
the moderately correct ones during ensemble. Firstly, we respectively
normalize the scales of three classification logits into -1$\sim$1 by their
each mean and standard deviation. We then calculate the distribution
similarities as the ensemble weights for the two logits of the adapter as

$\displaystyle w_{\text{CLIP}}=p_{\text{CLIP}}\ p_{\text{ZS}}^{T};\ \
w_{\text{DINO}}=p_{\text{DINO}}\ p_{\text{ZS}}^{T}.$ (7)

Finally, we adopt the softmax function to normalize the weights and obtain the
final ensemble logits as

$\displaystyle
p_{en}=p_{\text{ZS}}+\sum_{i}p_{i}\cdot\text{softmax($w_{i}$)},$ (8)

where $i\in\\{\text{CLIP},\text{DINO}\\}$. By such similarity-based ensemble,
$p_{en}$ can adaptively fuse the prior knowledge learned by CLIP and DINO’s
pre-training and achieve stronger few-shot image classification.

Figure 4: Performance (%) Comparison on ImageNet. We compare our CoMo with
other methods for different few-shot settings, where “CoMo w/o D.” denotes
CoMo without DALL-E’s generated images.

## Experiments

### Settings

#### Datasets.

We conduct few-shot experiments on 11 publicly available datasets: ImageNet
(Deng et al. 2009), StandfordCars (Krause et al. 2013), UCF101 (Soomro, Zamir,
and Shah 2012), Caltech101 (Fei-Fei, Fergus, and Perona 2004), Flowers102
(Nilsback and Zisserman 2008), SUN397 (Xiao et al. 2010), DTD (Cimpoi et al.
2014), EuroSAT (Helber et al. 2019), FGVCAircraft (Maji et al. 2013),
OxfordPets (Parkhi et al. 2012), and Food101 (Bossard, Guillaumin, and Gool
2014). We follow (Zhang et al. 2021) to train our CoMo with 1, 2, 4, 8, 16
shots and test on the full test set. As we could adopt DALL-E to generate
training images in a zero-shot manner, we also report CoMo’s zero-shot
performance without the downstream few-shot training set.

#### Implementation.

Our CoMo integrates the knowledge from pre-trained CLIP (Radford et al. 2021),
DINO (Caron et al. 2021), and DALL-E (Ramesh et al. 2021). For CLIP, we
utilize ResNet-50 (He et al. 2016) as the visual encoder and a transformer as
the textual encoder. To align with the visual representation from CLIP, we
also adopt DINO pre-trained upon ResNet-50. For DALL-E, we adopt different
domain-specific textual templates as the input for different datasets, which
correspond to the textual prompts for CLIP’s textual encoder. During training,
we only set the two kinds of keys in MK-Adapter to be learnable and utilize
the data augmentation following Tip-Adapter. We train CoMo using batch size 64
only for 20 epochs, and adopt AdamW optimizer with the initial learning rate
0.0001 and a cosine scheduler.

Few-shot Setting | 0 | 1 | 2 | 4 | 8 | 16
---|---|---|---|---|---|---
CLIP | 60.33 | - | - | - | - | -
Linear-probe CLIP | - | 22.17 | 31.90 | 41.20 | 49.52 | 56.13
CoOp | - | 57.15 | 57.81 | 59.99 | 61.56 | 62.95
CLIP-Adapter | - | 61.20 | 61.52 | 61.84 | 62.68 | 63.59
Tip-Adapter-F | - | 61.32 | 61.69 | 62.52 | 64.00 | 65.51
CoMo w/o D. | - | 61.39 | 62.55 | 63.96 | 65.78 | 68.08
CoMo | 60.77 | 63.24 | 63.84 | 65.23 | 66.41 | 68.42
Table 1: Quantative Performance (%) Comparison on ImageNet. We compare our CoMo with other methods for different few-shot settings, where “CoMo w/o D.” denotes CoMo without DALL-E’s generated images. Models | Epochs | Time | Accuracy | Gain
---|---|---|---|---
Zero-shot CLIP | 0 | 0 | 60.33 | -
Linear-probe CLIP | - | 13min | 56.13 | -4.20
CoOp | 200 | 14h 40min | 62.95 | +2.62
CLIP-Adapter | 200 | 50min | 63.59 | +3.26
Tip-Adapter-F | 20 | 5min | 65.51 | +5.18
CoMo | 20 | 10min | 68.42 | +8.09
Table 2: Efficiency Comparison on ImageNet. We test the training time with a
single A100 GPU under 16-shot setting.

(a)  (b)  (c)  (d)  (e)

(f)  (g)  (h)  (i)  (j)

Figure 5: Performance (%) Comparison on 10 Datasets. Our method shows state-
of-the-art performance for all few-shot settings on different datasets.

### Performance

#### On ImageNet.

We compare CoMo with other CLIP-based adaption methods on the most
representative ImageNet: Linear-probe CLIP (Radford et al. 2021), CoOp (Zhou
et al. 2021), CLIP-Adapter (Gao et al. 2021), and Tip-Adapter-F (Zhang et al.
2021). All these methods are based on the pre-trained CLIP (Radford et al.
2021) with ResNet-50 visual encoders. As reported in Figure 4 and Table 1, our
CoMo surpasses all existing methods for different shot settings. Remarkably,
CoMo with 1 shot even outperforms the 8-shot Linear-probe CLIP and CoOp, and
CoMo with 8 shots is better than all methods with 16 shots. Since our method
benefits from the zero-shot generation of DALL-E, we also compare the results
without DALL-E, denoted as “CoMo w/o D.”, which still remains state-of-the-
art. In Table 2, we present the efficiency comparison of CoMo and other
methods concerning training epochs and time. Our CoMo achieves the best
performance-efficiency trade-off with 68.42% accuracy and only 10 minutes
training.

#### On Other Datasets.

To further assess the robustness in different scenarios, we test CoMo on the
extra 10 datasets in Figure 5. For different semantic domains including real-
world scenes, detailed textures, and satellite-captured landscapes, CoMo
consistently shows leading performance and indicates excellent robustness via
the collaboration of diverse knowledge. Also on some datasets, e.g.,
Caltech101 and OxfordPets, the zero-shot CoMo perform even comparably to other
methods with 4 shots, demonstrating the effectiveness of zero-shot DALL-E for
few-shot data expansion.

Datasets | Source | Target
---|---|---
ImageNet | -V2 | -Sketch
Zero-shot CLIP | 60.33 | 53.27 | 35.44
Linear-probe CLIP | 56.13 | 45.61 | 19.13
CoOp | 62.95 | 54.58 | 31.04
CLIP-Adapter | 63.59 | 55.69 | 35.68
Tip-Adapter-F | 65.51 | 57.11 | 36.00
CoMo | 68.42 | 57.99 | 39.43
Table 3: Distribution Shift (%) Comparison. We train the models on “Source”
dataset and test on “target” datasets.

#### Distribution Shift.

We further evaluate our CoMo’s robustness to distribution shift by training on
“Source” dataset and testing on “Target” datasets. In Table 3, we select the
“Source” as ImageNe with the “Target” as ImageNet-V2 (Recht et al. 2019) and
ImageNet-Sketch (Hendrycks et al. 2021). As shown, by incorporating diverse
knowledge from pre-trained models, we achieve the best out-of-distribution
performance on the two “Target” datasets.

Pre-trained Models | Shot
---|---
CLIP | DINO | DALL-E | 1 | 2 | 4 | 8 | 16
✓ |  |  | 61.32 | 61.69 | 62.52 | 64.00 | 65.51
| ✓ |  | 34.14 | 34.44 | 40.47 | 44.32 | 53.27
✓ | ✓ |  | 61.39 | 62.55 | 63.96 | 65.78 | 68.08
✓ |  | ✓ | 61.36 | 61.78 | 62.83 | 64.04 | 65.53
| ✓ | ✓ | 34.13 | 34.44 | 41.12 | 45.01 | 53.63
✓ | ✓ | ✓ | 63.24 | 63.84 | 65.23 | 66.41 | 68.42
Table 4: Ablation Study (%) of Collaboration Models. We ensemble different
pre-trained models on ImageNet.

### Ablation study

#### Collaboration Models.

In Table 4, we explore how each pre-trained model contributes to the
collaboration on different shots of ImageNet. Therein, “CLIP” denotes the
zero-shot CLIP with MK-Adapter containing only CLIP’s keys, and “DINO” denotes
only the MK-Adapter with DINO’s keys. As shown in the first three rows, the
CLIP’s language-contrastive knowledge performs stronger than DINO’s vision-
contrastive knowledge, which might benefit from millions of pre-training data.
Their adaptive ensemble by MK-Adapter can bring larger improvement when the
shot number increases. For the next two rows, DALL-E can boost both CLIP and
DINO for nearly all shots with the generated synthetic images. The last row
represents our final solution, CoMo that incorporates all three pre-trained
models with the best performance for all shots.

#### Zero-shot Generation of DALL-E.

We utilize DALL-E to generate synthetic images as the expanded few-shot
training data. In Table 6, we explore the best synthetic number $K^{\prime}$
for each category of different shots on ImageNet. We observe that the larger
$K^{\prime}$ does not lead to better few-shot performance. As we adopt pre-
trained CLIP to select the top-$K^{\prime}$ generated images, which are scored
by the similarities between CLIP-encoded images and category texts, the larger
$K^{\prime}$ would contain more low-quality images and adversely affect the
MK-Adapter. Furthermore, the amount of expanded data is comparable to the
original $K$ shots and thus preserves the characteristic of few-shot learning.

#### Adaptive Ensemble.

In Table 5, we ablate different ensemble methods of CLIP and DINO’s
predictions on ImageNet. The first two rows represent the MK-Adapter with one
type of keys respectively for two pre-trained models without ensemble. Then,
we adopt average and maximum pooling between the two predictions and ensemble
the result with $p_{\text{ZS}}$. However, such naive integration without
adaptive weights causes accuracy degradation. In the last three rows, we
calculate the distribution similarities for adaptive ensemble and respectively
select the three logits as the baseline. As shown, using $p_{\text{ZS}}$ as
the distribution baseline performs the best, since $p_{\text{ZS}}$ itself
shows strong transfer ability and can effectively suppress the wrong
predictions of other logits.

Figure 6: t-SNE Visualization. Different colors represent different categories on 16-shot ImageNet. Method | Shot
---|---
1 | 2 | 4 | 8 | 16
CLIP | 61.36 | 61.78 | 62.83 | 64.04 | 65.53
DINO | 34.13 | 34.44 | 41.12 | 45.01 | 53.63
Average | 60.70 | 60.72 | 60.99 | 61.47 | 61.97
Maximum | 61.64 | 62.45 | 62.95 | 63.60 | 64.97
$p_{\text{CLIP}}$ Base. | 62.36 | 63.22 | 64.11 | 65.50 | 67.40
$p_{\text{DINO}}$ Base. | 62.61 | 63.39 | 64.31 | 65.83 | 67.73
$p_{\text{ZS}}$ Base. | 63.24 | 63.84 | 65.23 | 66.41 | 68.42
Table 5: Ablation Study (%) of Adaptive Ensemble. We conduct different
ensemble methods of Multi-Knowledge Adapter on ImageNet with different shots.

#### CLIP’s Visual Encoders.

We also conduct CoMo with different CLIP’s visual encoders for comparison with
other methods. As shown in Table 7, CoMo consistently achieves leading
performance with different visual backbones, indicating our generalizability
to network architectures.

Figure 7: Learning Curves of Test Accuracy (%) for different combinations of pre-trained models during 20 epochs on 16-shot ImageNet. DALL-E | 1 | 2 | 4 | 8 | 16
---|---|---|---|---|---
1 | 62.55 | 63.35 | 64.71 | 66.13 | 68.24
2 | 62.94 | 63.84 | 64.84 | 66.29 | 68.28
4 | 63.24 | 63.76 | 65.02 | 66.35 | 68.42
8 | 63.20 | 63.82 | 65.23 | 66.35 | 68.37
16 | 62.88 | 63.53 | 64.86 | 66.41 | 67.96
Table 6: Ablation Study (%) of Zero-shot Generation of DALL-E. We conduct CoMo with different number of generated images on ImageNet. Models | RN50 | RN101 | ViT-B/32 | ViT-B/16
---|---|---|---|---
Zero-shot CLIP | 60.33 | 62.53 | 63.80 | 68.73
CoOp | 62.95 | 66.60 | 66.85 | 71.92
CLIP-Adapter | 63.59 | 65.39 | 66.19 | 71.13
Tip-Adapter-F | 65.51 | 68.56 | 68.65 | 73.69
CoMo | 68.42 | 70.58 | 70.68 | 74.48
Table 7: Ablation Study (%) of CLIP’s Visual Encoders. RN50 denotes ResNet-50,
and ViT-B/32 denotes ViT-Base with patch size 32. We experiment on the 16-shot
ImageNet.

### Visualization

#### t-SNE.

We present the t-SNE visualization of our CoMo and the second-best Tip-
Adapter-F in Figure 6. As shown, CoMo shows more contrastive distribution of
category clusters than Tip-Adapter-F and can well mitigate some aliasing
between simliar classes.

#### Learning Curves.

In Figure 7, we visualize the 20-epoch learning curves of test accuracy on
16-shot ImageNet. Compared to the single CLIP, collaborating with DINO and
DALL-E significantly improves the convergence speed and classification
accuracy on test set.

## Conclusion

We propose CoMo, a collaboration of models that comprehends diverse knowledge
learned from different pre-training paradigms. Based on CLIP, we incorporate
the language-generative DALL-E to expand the few-shot training data and
adaptively fuse the vision-contrastive DINO via a Multi-Knowledge Adapter. By
collaboration, CoMo achieves state-of-the-art performance for few-shot
learning on 11 datasets. Although CoMo has unified three types of pre-
training, our future direction will focus on integrating more existing pre-
trained knowledge, such as the mask-generated knowledge of MAE (He et al.
2022) and the 3D-contrastive knowledge of CrossPoint (Afham et al. 2022).

## References

  * Afham et al. (2022) Afham, M.; Dissanayake, I.; Dissanayake, D.; Dharmasiri, A.; Thilakarathna, K.; and Rodrigo, R. 2022.  Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9902–9912.
  * Antol et al. (2015) Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015.  VQA: Visual Question Answering.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Bossard, Guillaumin, and Gool (2014) Bossard, L.; Guillaumin, M.; and Gool, L. V. 2014.  Food-101–mining discriminative components with random forests.  In _European conference on computer vision_ , 446–461. Springer.
  * Caron et al. (2018) Caron, M.; Bojanowski, P.; Joulin, A.; and Douze, M. 2018.  Deep Clustering for Unsupervised Learning of Visual Features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Caron et al. (2020) Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; and Joulin, A. 2020\.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ , 33: 9912–9924.
  * Caron et al. (2021) Caron, M.; Touvron, H.; Misra, I.; Jégou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021.  Emerging Properties in Self-Supervised Vision Transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 9650–9660.
  * Chen et al. (2018) Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2018.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(4): 834–848.
  * Chen et al. (2020a) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020a.  A Simple Framework for Contrastive Learning of Visual Representations.  In III, H. D.; and Singh, A., eds., _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , 1597–1607. PMLR.
  * Chen et al. (2020b) Chen, X.; Fan, H.; Girshick, R.; and He, K. 2020b.  Improved Baselines with Momentum Contrastive Learning.
  * Chen et al. (2021) Chen, Y.; Liu, Z.; Xu, H.; Darrell, T.; and Wang, X. 2021.  Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 9062–9071.
  * Cimpoi et al. (2014) Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; and Vedaldi, A. 2014.  Describing textures in the wild.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3606–3613.
  * Dayma et al. (2021) Dayma, B.; Patil, S.; Cuenca, P.; Saifullah, K.; Abraham, T.; Le Khac, P.; Melas, L.; and Ghosh, R. 2021.  DALL·E Mini.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Ding et al. (2021) Ding, M.; Yang, Z.; Hong, W.; Zheng, W.; Zhou, C.; Yin, D.; Lin, J.; Zou, X.; Shao, Z.; Yang, H.; and Tang, J. 2021.  CogView: Mastering Text-to-Image Generation via Transformers.  In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., _Advances in Neural Information Processing Systems_ , volume 34, 19822–19835. Curran Associates, Inc.
  * Doersch, Gupta, and Efros (2015) Doersch, C.; Gupta, A.; and Efros, A. A. 2015.  Unsupervised Visual Representation Learning by Context Prediction.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2020.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
  * Fei-Fei, Fergus, and Perona (2004) Fei-Fei, L.; Fergus, R.; and Perona, P. 2004.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _2004 conference on computer vision and pattern recognition workshop_ , 178–178. IEEE.
  * Finn, Abbeel, and Levine (2017a) Finn, C.; Abbeel, P.; and Levine, S. 2017a.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , 1126–1135. PMLR.
  * Finn, Abbeel, and Levine (2017b) Finn, C.; Abbeel, P.; and Levine, S. 2017b.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.  In Precup, D.; and Teh, Y. W., eds., _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , 1126–1135. PMLR.
  * Gao et al. (2021) Gao, P.; Geng, S.; Zhang, R.; Ma, T.; Fang, R.; Zhang, Y.; Li, H.; and Qiao, Y. 2021\.  CLIP-Adapter: Better Vision-Language Models with Feature Adapters.
  * Grill et al. (2020a) Grill, J.-B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.; Buchatskaya, E.; Doersch, C.; Avila Pires, B.; Guo, Z.; Gheshlaghi Azar, M.; Piot, B.; kavukcuoglu, k.; Munos, R.; and Valko, M. 2020a.  Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 21271–21284. Curran Associates, Inc.
  * Grill et al. (2020b) Grill, J.-B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.; Buchatskaya, E.; Doersch, C.; Avila Pires, B.; Guo, Z.; Gheshlaghi Azar, M.; et al. 2020b.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in Neural Information Processing Systems_ , 33: 21271–21284.
  * Hariharan and Girshick (2017) Hariharan, B.; and Girshick, R. B. 2017.  Low-Shot Visual Recognition by Shrinking and Hallucinating Features.  In _IEEE ICCV_ , 3037–3046.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked Autoencoders Are Scalable Vision Learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 16000–16009.
  * He et al. (2020) He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum Contrast for Unsupervised Visual Representation Learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * He, Girshick, and Dollar (2019) He, K.; Girshick, R.; and Dollar, P. 2019.  Rethinking ImageNet Pre-Training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Helber et al. (2019) Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(7): 2217–2226.
  * Hendrycks et al. (2021) Hendrycks, D.; Zhao, K.; Basart, S.; Steinhardt, J.; and Song, D. 2021.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15262–15271.
  * Houlsby et al. (2019) Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , 2790–2799. PMLR.
  * Jamal and Qi (2019) Jamal, M. A.; and Qi, G.-J. 2019.  Task Agnostic Meta-Learning for Few-Shot Learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Jia et al. (2021) Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q.; Sung, Y.-H.; Li, Z.; and Duerig, T. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International Conference on Machine Learning_ , 4904–4916. PMLR.
  * Jing and Tian (2021) Jing, L.; and Tian, Y. 2021.  Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(11): 4037–4058.
  * Kornblith, Shlens, and Le (2019) Kornblith, S.; Shlens, J.; and Le, Q. V. 2019.  Do Better ImageNet Models Transfer Better?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Krause et al. (2013) Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE international conference on computer vision workshops_ , 554–561.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  ImageNet Classification with Deep Convolutional Neural Networks.  In Pereira, F.; Burges, C.; Bottou, L.; and Weinberger, K., eds., _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc.
  * Li et al. (2021) Li, B.; Yang, B.; Liu, C.; Liu, F.; Ji, R.; and Ye, Q. 2021.  Beyond max-margin: Class margin equilibrium for few-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7363–7372.
  * Lin et al. (2017) Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; and Dollar, P. 2017.  Focal Loss for Dense Object Detection.  In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Liu et al. (2021) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021\.  Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 10012–10022.
  * Maji et al. (2013) Maji, S.; Rahtu, E.; Kannala, J.; Blaschko, M.; and Vedaldi, A. 2013.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_.
  * Miech et al. (2019) Miech, A.; Zhukov, D.; Alayrac, J.-B.; Tapaswi, M.; Laptev, I.; and Sivic, J. 2019\.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Nilsback and Zisserman (2008) Nilsback, M.-E.; and Zisserman, A. 2008.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing_, 722–729. IEEE.
  * Noroozi and Favaro (2016) Noroozi, M.; and Favaro, P. 2016.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.  In Leibe, B.; Matas, J.; Sebe, N.; and Welling, M., eds., _Computer Vision – ECCV 2016_ , 69–84. Cham: Springer International Publishing.  ISBN 978-3-319-46466-4.
  * Parkhi et al. (2012) Parkhi, O. M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C. 2012.  Cats and dogs.  In _2012 IEEE conference on computer vision and pattern recognition_ , 3498–3505. IEEE.
  * Pathak et al. (2016) Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; and Efros, A. A. 2016.  Context Encoders: Feature Learning by Inpainting.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 8748–8763. PMLR.
  * Ramesh et al. (2021) Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021.  Zero-Shot Text-to-Image Generation.  In Meila, M.; and Zhang, T., eds., _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , 8821–8831. PMLR.
  * Rampf, Villone, and Frisch (2015) Rampf, C.; Villone, B.; and Frisch, U. 2015.  How smooth are particle trajectories in a CDM Universe?
  * Recht et al. (2019) Recht, B.; Roelofs, R.; Schmidt, L.; and Shankar, V. 2019.  Do imagenet classifiers generalize to imagenet?  In _International Conference on Machine Learning_ , 5389–5400. PMLR.
  * Sharma et al. (2018) Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2556–2565. Melbourne, Australia: Association for Computational Linguistics.
  * Snell, Swersky, and Zemel (2017) Snell, J.; Swersky, K.; and Zemel, R. 2017.  Prototypical networks for few-shot learning.  _Advances in neural information processing systems_ , 30.
  * Soomro, Zamir, and Shah (2012) Soomro, K.; Zamir, A. R.; and Shah, M. 2012.  UCF101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_.
  * Sung et al. (2018) Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P. H.; and Hospedales, T. M. 2018\.  Learning to Compare: Relation Network for Few-Shot Learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * van den Oord, Li, and Vinyals (2018) van den Oord, A.; Li, Y.; and Vinyals, O. 2018.  Representation Learning with Contrastive Predictive Coding.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017.  Attention is All you Need.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vincent et al. (2008) Vincent, P.; Larochelle, H.; Bengio, Y.; and Manzagol, P.-A. 2008.  Extracting and Composing Robust Features with Denoising Autoencoders.  In _Proceedings of the 25th International Conference on Machine Learning_ , ICML ’08, 1096–1103. New York, NY, USA: Association for Computing Machinery.  ISBN 9781605582054.
  * Vinyals et al. (2016) Vinyals, O.; Blundell, C.; Lillicrap, T.; kavukcuoglu, k.; and Wierstra, D. 2016\.  Matching Networks for One Shot Learning.  In Lee, D.; Sugiyama, M.; Luxburg, U.; Guyon, I.; and Garnett, R., eds., _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc.
  * Wang and Hebert (2016) Wang, Y.; and Hebert, M. 2016.  Learning to Learn: Model Regression Networks for Easy Small Sample Learning.  In _ECCV_ , 616–634.
  * Xiao et al. (2010) Xiao, J.; Hays, J.; Ehinger, K. A.; Oliva, A.; and Torralba, A. 2010.  Sun database: Large-scale scene recognition from abbey to zoo.  In _2010 IEEE computer society conference on computer vision and pattern recognition_ , 3485–3492. IEEE.
  * Yang, Liu, and Xu (2021) Yang, S.; Liu, L.; and Xu, M. 2021.  Free lunch for few-shot learning: Distribution calibration.  _arXiv preprint arXiv:2101.06395_.
  * Zhang et al. (2020) Zhang, C.; Cai, Y.; Lin, G.; and Shen, C. 2020.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 12203–12213.
  * Zhang et al. (2021) Zhang, R.; Fang, R.; Gao, P.; Zhang, W.; Li, K.; Dai, J.; Qiao, Y.; and Li, H. 2021\.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  _arXiv preprint arXiv:2111.03930_.
  * Zhou et al. (2021) Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2021.  Learning to Prompt for Vision-Language Models.
  * Zhou et al. (2022) Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022.  Conditional Prompt Learning for Vision-Language Models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 16816–16825.

##  Appendix A Analysis by Other Metrics

To evaluate our approach with other metrics, we employ the negative log-
likelihood (NLL) and the area under the risk-coverage curve (AURC) to measure
different ensemble models on 16-shot ImageNet (Deng et al. 2009). Lower NLL
and AURC imply the better classification quality. In Table 8, we observe that
collaborating either DINO (Caron et al. 2021) or DALL-E (Ramesh et al. 2021)
with CLIP (Radford et al. 2021) could improve the classification of each pre-
trained model. However, different from CLIP, integrating DINO and DALL-E would
slightly harm the performance. This might because the ImageNet-pre-trained
DINO is unable to well recognize the synthetic images generated by DALL-E, but
CLIP with much stronger transfer ability can be benefited from the expanded
training set. In addition, the collaboration of all three pre-trained models
contributes to the largest performance boost, which indicates the significance
of their complementary characteristics.

Pre-trained Models | 16-shot ImageNet
---|---
CLIP | DINO | DALL-E | NLL $\downarrow$ | AURC $\downarrow$ | Acc. $\uparrow$
✓ |  |  | 1.29 | 141.9 | 65.51
| ✓ |  | 2.96 | 245.0 | 53.27
✓ | ✓ |  | 1.27 | 132.0 | 68.08
✓ |  | ✓ | 1.29 | 141.6 | 65.53
| ✓ | ✓ | 3.45 | 248.6 | 53.63
✓ | ✓ | ✓ | 1.22 | 121.2 | 68.42
Table 8: NLL and AURC ($\times 10^{3}$) of Different Models. NLL, AURC, and
Acc. (%) denote the negative log-likelihood, area under the risk-coverage
curve, and classification accuracy, respectively.

##  Appendix B Additional Ablation Study

#### Zero-shot CoMo.

As we leverage the pre-trained DALL-E to generate the supplementary few-shot
training set in a zero-shot manner, our CoMo can be evaluated under zero-shot
settings the same as CLIP, for which none of the human-annotated training
images is given. In Table 10, we report the best generated image number
$K^{\prime}$ of DALL-E for zero-shot CoMo. The number “0” denotes Zero-shot
CLIP. For different datasets, the best number varies ranging from 1$\sim$16,
and the larger number normally cannot get the better result, probably due to
the low-quality synthetic images. On Caltech101 (Fei-Fei, Fergus, and Perona
2004) and EuroSAT (Helber et al. 2019), zero-shot CoMo largely surpasses CLIP
by +3.53% and +4.87%, indicating our superiority under zero-shot settings.

#### Hyperparameter $\beta$.

In Formula 5 and 6, we utilize a non-linear modulator
$\varphi(x)=\exp(-\beta\cdot(1-x))$ for the affinity matrix of CLIP and DINO
in the MK-Adapter, where $\beta$ controls the matrix sharpness. In Table 9, we
experiment CoMo with different $\beta$ on 16-shot ImageNet and observe 0.6
performs the best.

##  Appendix C Additional Visualization

In Figure 8, we visualize the synthetic images generated by DALL-E on ImageNet
(Deng et al. 2009), OxfordPets (Parkhi et al. 2012), and Caltech101 (Fei-Fei,
Fergus, and Perona 2004) datasets. As shown, benefited from the pre-trained
DALL-E, the generated images can well highlight the semantics of target
category and effectively expand the few-shot training set in low-data regimes.

Sharpness $\beta$ | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 1.0
---|---|---|---|---|---|---
CoMo |  68.29 |  68.38 |  68.42 |  68.36 |  68.33 |  68.29
Table 9: Ablation Study (%) of Hyperparameter $\beta$. DALL-E |  ImageNet |  Caltech101 |  Flower102 |  Food101 |  DTD |  EuroSAT |  OxfordPets |  SUN397 |  StanfordCars |  UCF101 |  FGVCAircraft
---|---|---|---|---|---|---|---|---|---|---|---
0 | 60.33 | 86.29 | 66.14 | 77.20 | 42.32 | 37.56 | 85.77 | 58.52 | 55.61 | 61.46 | 17.28
1 | 60.61 | 89.05 | 66.34 | 77.24 | 44.98 | 42.43 | 87.27 | 60.62 | 57.18 | 62.20 | 17.70
2 | 60.76 | 88.72 | 66.50 | 77.27 | 45.04 | 40.58 | 86.89 | 59.61 | 56.86 | 62.54 | 17.58
4 | 60.77 | 89.13 | 66.26 | 77.24 | 45.15 | 40.01 | 87.16 | 59.94 | 56.97 | 62.44 | 17.70
8 | 60.75 | 89.37 | 66.46 | 77.28 | 44.86 | 40.68 | 87.35 | 59.97 | 57.12 | 62.41 | 17.64
16 | 60.74 | 89.82 | 66.38 | 77.27 | 44.86 | 39.81 | 87.00 | 59.73 | 58.11 | 62.94 | 17.49
Table 10: Ablation study (%) of Zero-shot CoMo via DALL-E on Different
Dataset. We leverage DALL-E to generate different numbers of synthetic images
for zero-shot recognition. Figure 8: Visualizations of Generated Images by
DALL-E.