# Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Renrui Zhang∗1, Rongyao Fang∗2, Wei Zhang∗1, Peng Gao†1, Kunchang Li1
Jifeng Dai3, Yu Qiao1, Hongsheng Li2
1Shanghai AI Laboratory
2The Chinese University of Hong Kong 3SenseTime Research
{zhangrenrui, gaopeng<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new
paradigm for learning visual representations by using large-scale contrastive
image-text pairs. It shows impressive performance on zero-shot knowledge
transfer to downstream tasks. To further enhance CLIP’s few-shot capability,
CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and
significantly improves the performance for few-shot classification. However,
such a process still needs extra training and computational resources. In this
paper, we propose Training-Free CLIP-Adapter (Tip-Adapter), which not only
inherits CLIP’s training-free advantage but also performs comparably or even
better than CLIP-Adapter. Tip-Adapter does not require any back propagation
for training the adapter, but creates the weights by a key-value cache model
constructed from the few-shot training set. In this non-parametric manner,
Tip-Adapter acquires well-performed adapter weights without any training,
which is both efficient and effective. Moreover, the performance of Tip-
Adapter can be further boosted by fine-tuning such properly initialized
adapter for only a few epochs with super-fast convergence speed. We conduct
extensive experiments of few-shot classification on ImageNet and other 10
datasets to demonstrate the superiority of proposed Tip-Adapter. The code will
be released at https://github.com/gaopengcuhk/Tip-Adapter.

††∗ indicates equal contributions, and $\dagger$ indicates corresponding
author. Figure 1: A Comparison of CLIP-Adapter vs. the proposed Training-free
CLIP-Adapter (Tip-Adapter). CLIP-Adapter and Tip-Adapter share similar
architectures, namely, two linear layers and a residual connection blending
updated visual features with pre-trained CLIP features. CLIP-Adapter is
trained with Stochastic Gradient Descent (SGD), while Tip-Adapter is training-
free, whose weights of linear layers are initialized from Cache Model.

##  1 Introduction

Vision and language are two modalities for humans to perceive the surrounding
world and perform diverse interactions with the environment. The accuracy of
vision tasks, such as classification [37, 24, 28, 14, 61, 45], detection [53,
5, 19] and segmentation [69], has been boosted significantly thanks to better
neural architecture designs (e.g. ResNet [24], Transformer [62], etc.) and
delicately designed frameworks (e.g., Faster R-CNN [53], RetinaNet [40], DETR
[5], etc.). Language generation and understanding have also been improved a
lot due to large-scale self-supervised tasks, including mask prediction [11]
and language model pre-training on web-scale dataset [52]. As vision and
language usually contain complementary information, joint learning of vision
and language representations has been proven to be quite effective on multi-
modality tasks, such as Visual Question Answering [2, 1, 33, 18], Image
Captioning [67, 29], and Referring Expression [68]. Different from previous
multi-modality interaction methods that learn vision and language
representations independently on each dataset [1, 43, 59], CLIP [51] proposed
to learn transferable visual features from natural language supervisions and
successfully demonstrate the amazing zero-shot classification ability of pre-
trained model. Due to the interplay between language and vision, the encoded
visual representations can be used in open-vocabulary classification without
re-training. Following the research direction of prompt design [4, 41], CoOp
[70] proposed to fine-tune a pre-trained CLIP model for few-shot
classification via learnable continuous tokens and achieved strong performance
on few-shot image classification. Recently, CLIP-Adapter [17] was also
proposed to tackle the few-shot classification problem by introducing a
feature adapter that generates the adapted features and then combines it with
the original CLIP feature using a residual connection, which incorporates the
new knowledge from the few-shot training set and prior knowledge encoded in
CLIP. CLIP-Adapter demonstrates promising performance on few-shot
classification without utilizing prompt designs.

Although CoOp [70] and CLIP-Adapter [17] show strong performance on few-shot
classification benchmarks, in comparison with CLIP [51] and linear probe CLIP
[51], they generally require much computational resources to fine-tune the
large-scale vision-language model due to the slow convergence of Stochastic
Gradient Descent (SGD) [34, 42] and huge GPU memory consumption [54]. Thus, we
ask the following question: Can we achieve best of both worlds of CLIP and
CLIP-Adapter, which does not only has the advantage of training-free property
of CLIP but also enjoys the strong performance of CLIP-Adapter for few-shot
classification?

To achieve the goal, we propose a novel Training-free CLIP-Adapter (Tip-
Adapter), which adopts the architecture design of CLIP-Adapter. It appends
CLIP model with an adapter of two-layer Multi-layer Perceptron (MLP) and a
residual connection [24] combining pre-trained features with the updated
features. Different from CLIP-Adapter, Tip-Adapter does not require SGD to
train the adapter but constructs a query-key cache model [32, 48, 20] from
few-shot supervisions to obtain the weights of adapter. Specifically, Tip-
Adapter extracts visual features of few-shot training images by CLIP’s visual
encoder and transforms their corresponding labels into one-hot encoding. On
top of that, a cache model is created which contains visual features and one-
hot labels of the few-shot training set as key-value pairs.

Based on the constructed cache model, the weights of Tip-Adapter could be
obtained in a training-free non-parametric manner. In detail, the two linear
layers of the CLIP-Adapter are set as the cached visual features and their
corresponding one-hot labels from the few-shot training set. Thus, the
processing of the adapter with such weights can be regarded as retrieving the
few-shot knowledge from the key-value cache model. By this approach, Tip-
Adapter exhibits great efficiency compared to CLIP-Adapter’s SGD training.
During inference, the test image’s adapted feature is combined with its
original CLIP-encoded feature. In this way, the Tip-Adapter is able to exploit
knowledge from both the pre-trained CLIP and the few-shot training dataset.
Surprisingly, Tip-Adapter with such constructed weights could perform
comparably to the fully fine-tuned CLIP-Adapter. Furthermore, if we unfreeze
the first linear layer of the Tip-Adapter and further fine-tune it, Tip-
Adapter’s performance could be significantly boosted with just a few training
epochs. It only requires 20 epochs in comparison with CLIP-Adapter’s 200
epochs.

The contributions of our paper are summarised below:

  * •

We propose Training-free CLIP-Adapter (Tip-Adapter), which has strong
performance on few- classification via directly setting the weights of adapter
with a cache model to avoid the conventional SGD fine-tuning.

  * •

The performance of Tip-Adapter can be further improved by fine-tuning based on
such constructed weights with super-fast convergence.

  * •

We evaluate Tip-Adapter on 11 few-shot classification datasets and conduct
extensive ablation studies to demonstrate its characteristics. Tip-Adapter
achieves competitive performance with state-of-the-art methods and leads to
significant reduction of training time.

##  2 Related Work

#### Data-efficient Transfer Learning.

The capability of deep neural networks is revealed with the assistance of
large-scale datasets [37]. However, collecting large-scale dataset is
challenging due to actual data’s long-tail distribution, noisy annotations and
the increasing labeling costs. Thus, transfer learning is proposed to reduce
such costly requirements and has become a popular research field. Supervised
pre-training on large-scale image classification tasks (e.g. ImageNet [9]) and
transferring to downstream tasks (e.g. detection [53] and segmentation [23])
have been widely adopted. Self-supervised learning, such as MoCo [22] and BYOL
[21], further alleviates the need of large-scale supervised pre-training and
converts the problem into a self-supervised form. From the view of language,
VirTex [10] verifies the data-efficiency of language supervision via
captioning on learning high-quality visual representations. Recently, CLIP
[51], DeCLIP [39] and ALIGN [30] have demonstrated that simple contrastive
learning between vision-language pairs is able to learn zero-shot transferable
features over diverse image classification datasets without further training.
Furthermore, CoOp [70], CLIP-Adapter [17] and WiSE-FT [64] indicate that the
performance of CLIP can be significantly improved with either abundant or
limited training data by fine-tuning weight-fixed CLIP with adapter or prompt
optimization. In contrast, our proposed Tip-Adapter aims at directly infusing
few-shot supervisions into the pre-trained CLIP model with a training-free
manner. Thus, Tip-Adapter is quite efficient at time and computation, as it
only calculates and caches the few-shot training set once, and requires no
more training.

Figure 2: The Pipeline of Tip-Adapter. Given a $K$-shot $N$-class training
set, we construct the weights of the two-layer adapter by creating a cache
model from the few-shot training set. It contains few-shot visual features
$F_{train}$ encoded by CLIP’s visual encoder and few-shot ground-truth labels
$L_{train}$. $F_{train}$ and $L_{train}$ can be used as the weights for the
first and second layers in the adapter.

#### Transformer.

Transformer [62] introduces the key-query attention mechanism for encoding
high-quality features and is superior to many previous language and vision
models, such as Long Short-term Memory (LSTM) [26] and Convolution Neural
Network (CNN) [38]. For natural language processing, transformer achieves
leading performance on machine translation [62, 49], natural language
understanding [11] and language generation [52, 4]. Also for vision,
transformer-based architectures have shown great capabilities and potentials
on image classification [13], object detection [5], and semantic segmentation
[69]. Transformer utilizes multi-head key-query multiplication to implement
highly-efficient message passing between all pairs of input tokens. In this
paper, we analyse the relationship between Tip-Adapter’s cache model and
transformer’s attention mechanism. This similar relations have also been
discussed in persistent memory [57] and non-parametric attention [35].

#### Cache Model.

A cache model stores features of training images and their labels as a key-
value database. During inference, the feature generated from a test example
would be treated as query and aggregate information from the database by
retrieval [62], which is similar to attention mechanism in transformer [62].
It is important that the whole process is non-parametric [35] and requires no
parameter update. The cache model has been adopted for improving language
generation in kNN-LMs [32], Unbounded Cache [20] and Pointer Sentinel Mixture
Models [46]. Although simple cache model [48] has shown some promising
results, it needs to store a huge memory of training dataset for test-time
inference. Therefore, approximate kNN and high-optimized similarity search
system [31] are proposed, but the whole pipeline is generally verbose, error-
prone and slow. Different from previous cache models’ setup, we explore it
with CLIP and adopt the few-shot settings. In detail, we generate cached
features from CLIP’s visual encoder so that strong label-related information
is naturally encoded in the key-value pair. Due to the few-shot setup with
limited training samples, the total cache is small and can be efficiently
calculated by two cascades of matrix multiplication. Moreover, the cache model
in Tip-Adapter is able to dynamically update via Stochastic Gradient Descent
(SGD), which further improves its performance.

##  3 Method

In this section, we introduce the proposed Training-free CLIP-Adapter (Tip-
Adapter) for significantly improving CLIP’s zero-shot classification accuracy.
In Section 3.1, we first briefly revisit CLIP-Adapter. In Section 3.2, we
present the designs of Tip-Adapter and the further fine-tuned version for
achieving stronger performance. Finally in Section 3.3, we discuss the
relationship with previous cache-based methods.

###  3.1 A Revisit of CLIP-Adapter

Adapter [27] is a lightweight neural module to conduct parameter-efficient
fine-tuning of pre-trained models, such as BERT [11] in natural language
processing. CLIP learns visual representations by exploiting contrastive
learning on large-scale image-text pairs, which achieves inspirational
performance in zero-shot classification. Following the spirit of adapter, to
transfer CLIP’s knowledge for achieving few-shot classification, CLIP-Adapter
[17] appends a lightweight two-layer Multi-layer Perceptron (MLP) to the pre-
trained fixed-weight CLIP model and predicts the adapted feature residuals for
each input image. The residual connection in the adapter helps to fuse the
prior visual knowledge encoded by CLIP and the updated features learned from
the few-shot training set. Compared with zero-shot and linear-probe CLIP [51],
CLIP-Adapter achieves significant performance improvement on multiple image
classification datasets. It is optimized by minimizing the cross-entropy loss
between predictions and ground truths with stochastic gradient descent (SGD).
Thus, although CLIP-Adapter achieves great performance boost, it still
requires additional training, which is slow and costly compared with the
original training-free zero-shot CLIP.

Specifically, for an input image, its L2 normalized feature $f_{c}$ is first
obtained by CLIP’s pre-trained visual encoder. An adapter of two-layer MLP,
with parameters $W_{1},b_{1},W_{2},b_{2}$, is appended to obtain the updated
feature $f_{a}$,

$\displaystyle\begin{split}f_{a}=\varphi(f_{c}W_{1}^{T}+b_{1})W_{2}^{T}+b_{2},\end{split}$
(1)

where $\varphi$ denotes the activation function in the MLP. Then, the adapted
feature $f_{a}$ is linearly combined with the pre-trained feature $f_{c}$ with
a hyper-parameter $\alpha\in[0,1]$ to output the final classification logits.
In this way, the prior knowledge of CLIP on the input image is updated by the
adapter in an additive manner,

$\displaystyle\begin{split}\mathrm{logits}=\alpha
f_{a}W_{c}^{T}+f_{c}W_{c}^{T},\end{split}$ (2)

where $W_{c}$ is the weights of the text classifier. To construct $W_{c}$,
following zero-shot CLIP, CLIP-Adapter places each category name into the pre-
defined prompt template and encodes them by CLIP’s pre-trained textual
encoder.

###  3.2 Training-free CLIP-Adapter

We propose Tip-Adapter, which is a training-free and non-parametric extension
of CLIP-Adapter [17] but performs comparably and even better than it. To
achieve this goal, it adopts the same architecture as CLIP-Adapter, but we
construct a key-value cache model from the few-shot training set and transform
the cache into the weights of the adapter MLP in a non-parametric manner
without fine-tuning. Surprisingly, with weights constructed by a properly
designed approach, Tip-Adapter without fine-tuning can achieve comparable
performance as CLIP-Adapter with fine-tuning. In addition, if fine-tuning is
allowable, further fine-tuning with such weights as network initialization is
able to achieve much higher accuracy with super-fast convergence speed.

#### Cache Model Construction.

Given the pre-trained CLIP [51] model and a $K$-shot $N$-class training set
for few-shot classification, there are $K$ annotated images in each of the $N$
categories, denoted as $\mathrm{I}_{K}$ with their labels $\mathrm{L}_{N}$.
The goal is to conduct image classification on the $N$ classes. We create a
key-value cache model and convert it to obtain the weights of the proposed
Tip-Adapter. For each training image, we utilize the CLIP’s visual encoder to
extract its $C$-dimensional L2 normalized visual feature, and convert its
ground-truth label into an $N$-dimensional one-hot vector. For all $NK$
training samples, we denote their visual features and corresponding label
vectors as $\mathbf{F}_{\rm train}\in\mathbb{R}^{NK\times C}$ and
$\mathbf{L}_{\rm train}\in\mathbb{R}^{NK\times N}$,

$\displaystyle\mathbf{F}_{\rm train}=\mathrm{VisualEncoder}(\mathrm{I}_{K}),$
(3) $\displaystyle\mathbf{L}_{\rm train}=\mathrm{OneHot}(\mathrm{L}_{N}).$ (4)

To create the key-value cache, the CLIP-encoded representations
$\mathbf{F}_{\rm train}$ are treated as keys, while the one-hot ground-truth
vectors $\mathbf{L}_{\rm train}$ are used as their values. In this way, the
key-value cache contains all the new knowledge extracted from the few-shot
training set, which can be converted to the weights of the adapter MLP to
update the prior knowledge encoded in the pre-trained CLIP.

#### Tip-Adapter.

After constructing the cache model, the adapter weights are determined
according to cached key-value pairs. Specifically, the test image’s normalized
visual feature $f_{\mathrm{test}}\in\mathbb{R}^{1\times C}$ is first extracted
by the CLIP’s visual encoder and serves as a query for retrieving from the
key-value cache. The affinities between the test query and keys of the cached
few-shot training set can be then estimated as $A\in\mathbb{R}^{1\times NK}$,

$\displaystyle A=\exp{(-\beta(1-f_{\mathrm{test}}\mathbf{F}^{T}_{\rm
train}))},$ (5)

where $\beta$ stands for a modulating hyper-parameter. Since both query and
key features are L2 normalized, the term
$(1-f_{\mathrm{test}}\mathbf{F}^{T}_{\rm train})$ is equivalent to calculating
the Euclidean distances between the test feature $f_{\mathrm{test}}$ and all
few-shot training images’ features $\mathbf{F}^{T}_{\rm train}$. The
exponential function is adopted to convert query-key Euclidean distances to
non-negative affinities $A$ with $\beta$ modulating its sharpness. Afterwards,
the retrieved value from the cache model can be obtained via the
multiplication between the query-key affinities and the cached values as
$A\mathbf{L}_{\rm train}$.

The predicted logits of the test image by the Tip-Adapter is then calculated
as

$\displaystyle\mathrm{logits}$ $\displaystyle=\alpha
A\mathbf{L}_{\mathrm{train}}+f_{\mathrm{test}}W_{c}^{T}$
$\displaystyle=\alpha\varphi(f_{\mathrm{test}}\mathbf{F}^{T}_{\rm
train})\mathbf{L}_{\mathrm{train}}+f_{\mathrm{test}}W_{c}^{T},$ (6)

where $W_{c}$ represents CLIP’s text classifier, $\alpha$ denotes the residual
ratio, and we define $\varphi(x)=\exp(-\beta(1-x))$. Intuitively, Tip-
Adapter’s class prediction contains two terms: predictions according to values
retrieved from the cache model and predictions from the pre-trained CLIP. The
former term adaptively summarizes information from the few-shot training set.
The values $\mathbf{L}_{\mathrm{train}}$ (class predictions of the cached
samples) in the cache are linearly combined according to the query-key
affinities $A$. The latter term preserves the prior knowledge from the
original CLIP by directly using the pre-trained classifier $W_{c}^{T}$ to
process the test image feature $f_{\mathrm{test}}$. The two terms are balanced
by the weight $\alpha$. Empirically, $\alpha$ is set to be large if the domain
gap between pre-trained and downstream tasks is large, since more knowledge
from the few-shot set is required, and small otherwise.

Comparing Eqs. (2) and (3.2), the proposed Tip-Adapter can be seen as a
special form of CLIP-Adapter with the following weights,

$\displaystyle
W_{1}=\mathbf{F}_{\mathrm{train}},W_{2}=\mathbf{L}^{T}_{\mathrm{train}},\
b_{1}=0,\ b_{2}=0,$ (7)
$\displaystyle\varphi(x)=\operatorname{exp}(-\beta(1-x)),\ \text{ where }\
x\in[0,1].$ (8)

The differences between CLIP-Adapter and Tip-Adapter can be summarized as
follows. Firstly, CLIP-Adapter randomly initializes $W_{1}$ and $W_{2}$ and
learns them via SGD, while Tip-Adapter directly sets $W_{1}$ as cached
training features $\mathbf{F}_{\mathrm{train}}$ and $W_{2}$ as the transposed
one-hot encoding of the ground-truth labels $\mathbf{L}_{\mathrm{train}}$,
which are non-parametric and training-free. Secondly, the bottleneck dimension
of Tip-Adapter is equal to $NK$, while CLIP-Adapter selects a low-dimensional
bottleneck to prevent the risk of over-fitting. This indicates that with such
proper initialization, the over-fitting problem on few-shot datasets is much
alleviated, which further releases the fitting power of high-dimensional
linear layers. Thirdly, Tip-Adapter introduces a new activation function
denoted in Eq. (7). As its inputs are the distances in the normalized feature
space, it is bounded between 0 and 1. However, for CLIP-Adapter, the common
activation function, ReLU$(\cdot)$, is chosen to handle unbounded inputs.
Aided by the new activation $\varphi(\cdot)$, the calculated distances can be
well modulated and help the performance. In summary, Tip-Adapter could acquire
the well-performed adapter weights without training. In other words, it is
more efficient and effective on few-shot classification.

Tip-Adapter can greatly improve the classification performance of CLIP by
incorporating new knowledge provided by the few-shot training set and be
implemented as a CLIP-Adapter with cached $W_{1}$ and $W_{2}$. However, given
more shots, there still exists slight performance gap between Tip-Adapter and
CLIP-Adapter. To mitigate the gap, we can treat the cache model as a good
initialization point and continue to fine-tune the Tip-Adapter via SGD to
surpass the randomly initialized CLIP-Adapter.

For fine-tuning, we supervise the Tip-Adapter’s predictions with few-shot
training data and the cross-entropy loss, during which the weights in the
cache model are updated via SGD. Specifically, we unfreeze the weights of keys
$W_{1}=\mathbf{F}_{\mathrm{train}}$, but still fix the weights of value
$W_{2}=\mathbf{L}_{\mathrm{train}}$ and two encoders of the CLIP model. The
intuition is that updating the keys in the cache model can adaptively boost
the estimation of affinities, namely, the distances calculation between
training and testing images in the embedding space. In contrast, values in the
cache are one-hot encodings representing ground-truth annotations and shall be
kept frozen to accurately memorize the memory information. Thanks to the good
initialization provided by the cache model, Tip-Adapter only needs to be fine-
tuned for a small number of epochs and can surpass the CLIP-Adapter with much
longer training scheme (20 epochs versus 200 epochs). It demonstrates that the
proposed Tip-Adapter can achieve strong performance with fast convergence and
limited resources.

###  3.3 Relationship with Cache-based Networks

Constructing a cache model from few-shot training data has been explored by
many previous methods, including Matching Network [63], Prototypical Networks
[55], MAML [16], Relation Network [58] and others [12, 7, 60, 6]. Different
from all previous methods, Tip-Adapter adopts the residual architecture
following CLIP-Adapter but proposes to use cached features for properly
setting the weights of the CLIP-adapter. By performing further fine-tuning,
Tip-Adapter significantly improves the performance of few-shot learning.

From another perspective, our Tip-Adapter can be regarded as a heterogenous
cache model with both visual and textual features extracted by CLIP.
Specifically, the weights $W_{c}$ of CLIP’s linear classifier can be treated
as cached textual features. Therefore, the final classification of the test
image $f_{\mathrm{test}}$ is jointly inferred by multi-modality features,
which fully exploit vision-language prior knowledge encoded in CLIP. From this
perspective, the two terms in Eq. (3.2) can be reinterpreted as distance
calculation with cached visual and textual features, respectively, and their
importance is balanced by $\alpha$.

##  4 Experiments

###  4.1 Training Settings

We conduct experiments for Tip-Adapter on 11 image classification datasets:
ImageNet [9], StandfordCars [36], UCF101 [56], Caltech101 [15], Flowers102
[47], SUN397 [66], DTD [8], EuroSAT [25], FGVCAircraft [44], OxfordPets [50],
and Food101 [3]. Performance comparison is conducted between Zero-shot CLIP
[51], Linear-probe CLIP [51], CLIP-Adapter [17] and CoOp [70].

As mentioned in the previous section, our Tip-Adapter has two versions. The
first version is training-free, which directly sets the adapter’s MLP weights
following Eq. (7). The second version allows further fine-tuning of the
adapter initialized by the properly set weights. The two version are denoted
as Tip-Adapter and Tip-Adapter-F in this section, respectively. Each model is
trained with 1, 2, 4, 8, and 16 few-shot training sets, and tested on the full
test sets. For the CLIP backbone, we utilize ResNet-50 [24] as the visual
encoder and a transformer [14] as the textual encoder. In terms of prompt
design, we adopt prompt ensembling in [51], which inputs 7 templates into the
CLIP textual encoder and then averages them as the final prompt. The 7
templates are: “itap of a [CLASS].”, “a bad photo of the [CLASS].”, “a origami
[CLASS].”, “a photo of the large [CLASS].”, “a [CLASS] in a video game.”, “art
of the [CLASS].” and “a photo of the small [CLASS].”. To fine-tune Tip-
Adapter-F, we train it with a batch size of $256$, and use Stochastic Gradient
Descent (SGD) [34, 42] with a learning rate $0.001$ and a cosine scheduler. In
contrast to the 200-epoch training in CoOp and CLIP-Adapter, Tip-Adapter-F
only requires 20 epochs for fine-tuning and has super-fast convergence speed,
saving much computational cost for training.

There are two image pre-processing methods adopted by existing methods. The
first one is adopted by CLIP and the second one is reported in CoOp and CLIP-
Adapter. We denote them as CLIP-style and CoOp-style pre-processing,
respectively. Both of them are composed of random cropping, resizing, and
random horizontal flip. They are different in resizing. The CLIP-style pre-
processing resizes the cropped image’s short side to 224 while keeping its
original aspect ratio, while the CoOp-style resizes image’s both sides to 224.
We follow CLIP-style pre-processing by default, since it preserves cropped
images’ original aspect ratios.

###  4.2 Comparison on ImageNet

We compare our proposed Tip-Adapter and Tip-Adapter-F with Zero-shot CLIP
[51], Linear-probe CLIP [51], CoOp [70], and CLIP-Adapter [17]. The zero-shot
CLIP uses no extra training sample and conducts classification purely by
calculating similarities between visual features of test images and textual
features of the hand-crafted prompts. Linear-probe CLIP fine-tunes an
additional linear classifier after the original frozen CLIP on the few-shot
training set. To make prompts learnable, CoOp learns to generate different
designs for the format of prompts, which we select its best-performance
variant for comparison – placing the class token at the end of the 16-token
prompts and sharing the context among all classes. CLIP-Adapter appends a
feature adapter [27] on top of CLIP’s visual encoder and textual encoder to
narrow the domain gap between the pre-trained and transferred features, which
helps to achieve better few-shot classification performance. Likewise, we
compare Tip-Adapter with the best-performing variant of CLIP-Adapter with only
learnable visual adapter. Moreover, for fair comparison, we adopt prompt
ensembling of 7 templates for zero-shot CLIP and CLIP-Adapter, the same as
Tip-Adapter.

We report other models with both CLIP-style pre-process and CoOp-style pre-
process in Table 2 for fair comparison, but in Table 1, Figure 3 and the
following pages, only the results with CLIP-style pre-processing are
presented. Also, the far-behind performance of linear-probe CLIP are only
presented in Figure 3 for conciseness.

#### Performance Analysis.

As shown in Figure 3 and Table 2, both versions of our Tip-Adapter show
outstanding performances over compared methods. Similarly without training,
Tip-Adapter consistently surpasses zero-shot CLIP. Compared to linear-probe
CLIP and CoOp with time-consuming fine-tuning, Tip-Adapter greatly exceeds
them when the numbers of training samples are limited. For instance, in 1-shot
and 2-shot settings, Tip-Adapter surpasses linear-probe CLIP by 38.53$\%$ and
29.06$\%$, and CoOp by 13.08$\%$ and 10.08$\%$. They demonstrate the
superiority of our training-free cache-based method. However, Tip-Adapter
still falls behind the well-tuned CLIP-Adapter. With further fine-tuning, Tip-
Adapter-F, updates the weights initialized from the cache model and achieves
the best performance over all methods in all few-shot settings. Compared to
the training-free version, the performance gain resulted from fine-tuning
becomes larger as the number of training samples increases, from 1-shot’s
+0.62$\%$ to 16-shot’s +3.44$\%$. This accords with the intuition that more
training samples provide the model with a more robust cache.

Figure 3: Classification accuracy of Tip-Adapter, Tip-Adapter-F and other
models under different few-shot settings with CLIP-style pre-processing.

Models RN50 RN101 ViT/32 ViT/16 RN50$\times$16 Zero-shot CLIP[51] 60.33 62.53
63.80 68.73 70.94 CoOp[70] 62.95 66.60 66.85 71.92 - CLIP-Adapter[17] 63.59
65.39 66.19 71.13 - Tip-Adapter 62.03 64.78 65.61 70.75 72.95 Tip-Adapter-F
65.51 68.56 68.65 73.69 75.81

Table 1: Performances ($\%$) of different models on various vision backbones.
RN50 denotes ResNet-50, and ViT/32 denotes ViT-Base with 32 $\times$ 32 patch
size, and RN50$\times$16 denotes ResNet-50 with 16 times more computations
[51].

In Table 1, we also implement Tip-Adapter with different visual encoders over
ResNet [24] and Vision Transformer [14] backbones. The leading performances of
Tip-Adapter-F fully demonstrate its superior cabability for visual
recognition.

#### Efficiency Comparison.

In Table 3, we show the 16-shot training time of different models and the
experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU. Tip-Adapter
and CLIP-Adapter first cache the textual features from the CLIP model and load
them during training, but CoOp adopts learnable prompts so that the textual
features are required to be calculated online in every iteration. Linear-probe
CLIP utilizes Logistic Regression [65] to train the final linear layer, so it
cannot be measured by epoch. From the comparison we can observe that CoOp
takes most training time for learning prompts and has a 2.26$\%$ performance
gain over zero-shot CLIP. CLIP-Adapter significantly reduces the training time
with a larger improvement. Tip-Adapter gains 1.70$\%$ boost but requires no
extra training time, which makes it a good trade-off between performance and
efficiency. Tip-Adapter-F further reaches a higher accuracy with only $1/10$
of CLIP-Adapter’s epochs, achieving best of both worlds.

Few-shot Setup 1 2 4 8 16 CoOp-style Pre-process, Zero-shot CLIP∗[51]: 57.81
Linear-probe CLIP∗[51] 21.14 31.67 40.33 47.12 51.59 CoOp∗[70] 53.12 53.23
56.47 57.58 60.46 CLIP-Adapter∗[17] 58.17 58.58 59.40 60.39 61.33 CLIP-style
Pre-process, Zero-shot CLIP[51]: 60.33 Linear-probe CLIP[51] 22.17 31.90 41.20
49.52 56.13 CoOp[70] 47.62 50.88 56.22 59.93 62.95 CLIP-Adapter[17] 61.20
61.52 61.84 62.68 63.59 Tip-Adapter 60.70 60.96 60.98 61.45 62.03 Tip-
Adapter-F 61.32 61.69 62.52 64.00 65.51 _by fine-tuning_ +0.62 +0.73 +1.54
+2.55 +3.48

Table 2: Classification accuracy ($\%$) of models under both CoOp-style and
CLIP-style pre-processing in few-shot settings. Models following the CoOp-
style pre-process method are labeled by $*$. The last row in blue records the
performance gain brought by further fine-tuning over Tip-Adapter.

Models Epochs Time Accuracy Gain Zero-shot CLIP[51] 0 0 60.33 0 Linear-probe
CLIP[51] - 13min 56.13 -4.20 CoOp[70] 200 14h 40min 62.95 +2.62 CLIP-
Adapter[17] 200 50min 63.59 +3.26 Tip-Adapter 0 0 62.03 +1.70 Tip-Adapter-F 20
5min 65.51 +5.18

Table 3: Fine-tuning time and classification accuracy of 16-shot learning with
different methods. The last column in blue records the performance gain
relative to zero-shot CLIP.

Figure 4: Results of few-shot classification on 10 datasets. Tip-Adapter
exceeds zero-shot ClIP by a large margin, and Tip-Adapter-F further surpasses
all compared methods by few-epoch fine-tuning.

###  4.3 Performances on Other Datasets

Figure 4 shows Tip-Adapter’s performances on other 10 datasets: StandfordCars
[36], UCF101 [56], Caltech101 [15], Flowers102 [47], SUN397 [66], DTD [8],
EuroSAT [25], FGVCAircraft [44], OxfordPets [50], and Food101 [3]. Comparing
with Zero-shot CLIP[51], CLIP-Adapter[17] and CoOp[70], we observe that our
Tip-Adapter significantly boosts the classification performance over zero-shot
CLIP in different types of datasets. Although on some datasets, Tip-Adapter
falls behind CLIP-Adapter when there are more shots for training, Tip-
Adapter-F with a few-epoch fine-tuning eliminates this gap and further
surpasses all other models. The consistent superiority over 10 datasets fully
demonstrates the effectiveness and generality of Tip-Adapter.

###  4.4 Ablation Studies

In this section, we conduct several ablation studies about Tip-Adapter on
ImageNet with the CLIP-style pre-processing. All experiments adopt the 16-shot
setting and our training-free Tip-Adapter is adopted by default.

#### Residual Ratio $\alpha$.

The hyper-parameter $\alpha$ combines newly adapted features from cache model
with pre-trained CLIP encoder’s features, or in other words, weighing the
importance of the visual and textual cache. As formulated above, larger
$\alpha$ denotes using more knowledge from the few-shot training set and less
otherwise. We vary $\alpha$ from 0.0 to 5.0, and set the hyper-parameter
$\beta$ as 5.5. When $\alpha$ equals 0.0, the model is equivalent to zero-shot
CLIP without using new knowledge from the training set. From the top part of
Table 4, we observe that the classification accuracy is improving as $\alpha$
increases from 0.0 to 1.0, achieving the best 62.03$\%$ at 1.0 , but further
increasing the weight of the adapted features hurts the performance. This
indicates that both the prior knowledge from CLIP and adapted features from
the few-shot training set are equally important.

Ablation Studies on Tip-Adapter Residual Ratio $\alpha$ 0.0 0.5 1.0 2.0 3.0
4.0 60.33 61.44 62.03 61.41 60.36 59.14 Sharpness Ratio $\beta$ 1.5 3.5 5.5
7.5 9.5 11.5 61.82 61.91 62.03 61.76 61.62 61.40 Cache Size 0 1 2 4 8 16 60.33
61.45 61.71 61.79 61.83 62.03 More Shots than 16 Shot Setup 16 32 64 128 Tip-
Adapter 62.03 62.51 62.88 63.15 Tip-Adapter-F 65.47 66.58 67.96 69.74

Table 4: Four ablation studies of the proposed Tip-Adapter on ImageNet, from
top to bottom: residual ratio $\alpha$, sharpness ratio $\beta$, the size of
cache model and the performance given more shots with cache size 16.

#### Sharpness Ratio $\beta$.

In Eq. (5), $\beta$ in the activation function $\varphi$ controls the
sharpness of the affinities. When $\beta$ is large, only the nearby training
samples to the test image have large influences to its class prediction and
vice versa. The results of different $\beta$ are presented on the second part
of Table 4, in which $\alpha$ is set as 1.0. We observe that the variation of
$\beta$ has limited impact. A moderate 5.5 for $\beta$ leads to the best-
performing Tip-Adapter.

Figure 5: t-SNE Visualization of $W_{1}$ in Tip-Adapter. Dots in different
colors stand for embeddings of different categories. From left to right, three
distributions indicate the variation of $W_{1}$ during fine-tuning.

#### Size of the Cache Model.

Here, we explore the influence of the cache model’s size in Tip-Adapter. The
largest setting caches 16 samples for each class, and the smallest one caches
only one sample per class. We experiment with $K$ equaling 0, 1, 2, 4, 8 and
16. Cache size 0 is equivalent to zero-shot CLIP without the cache. When it is
more than 0 but less than 16, taking 8 as an example, we randomly divide 16
samples into 8 uniform groups and obtain 8 prototypes by averaging features of
2 samples in each group. Considering such random division of samples might
influence the performance, we experiment for 5 times and report the average
scores. The results from the third part of Table 4 illustrate that the more
samples we cache, the higher accuracy Tip-Adapter can achieve.

#### Scaling Up to More Shots.

One might concern if more shots are given, the increasing size of the cache
model along with higher-dimensional adapter would become a burden for both
memory and computation. Thus, we here explore a way to fully utilize the
training set with more than 16 shots, but fix the cache size as 16. Taking 64
shots as an example, following the division strategy in the above paragraph,
we obtain 16 prototypes from 4 groups to construct the cache model. The
results in the final part of Table 4 show that even if the cache size is
restrained to 16, it still well contains the information of 32, 64 and 128
training samples per category. We also observe that the growth rate of
performance gradually slows down when more samples are provided. This
indicates a possible limit load for cache size 16, but Tip-Adapter-F solves it
by fine-tuning and more shots result in larger performance boost.

Figure 6: Performances of compared methods with three prompt designs: Single
Prompt in cyan, Prompt Ensembling in orange and Learnable Prompt in purple.

#### Prompt Design.

For zero-shot CLIP, CLIP-Adapter and Tip-Adapter, the experiments above are
all based on prompt ensembling of 7 templates from [51]. In this study, we
test only using a single prompt: “a photo of a [CLASS].” to see how this
influences the performance. As shown in Figure 6, the performance drops
resulted from using a single prompt are smaller for Tip-Adapter-F and CLIP-
Adapter, whose classification scores are relatively high, but larger for Tip-
Adapter and Zero-shot CLIP. In general, the better-performing models are less
affected by prompt variations.

Visual Enc. Textual Enc. Adapter $W_{1}$ Adapter $W_{2}$ Score($\%$) - - - -
62.03 - - ✓ - 65.51 - - - ✓ 60.90 - - ✓ ✓ collapse ✓ - - - 62.84 - ✓ - - 63.15
✓ ✓ - - 51.22

Table 5: Ablations of Tip-Adapter fine-tuning different modules. ✓denotes
fine-tuning the module and symbol - denotes freezing. Visual Enc. and Textual
Enc. stand for visual encoder and textual encoder in pre-trained CLIP.

#### Fine-tuning Settings.

Tip-Adapter-F only fine-tunes adapter’s $W_{1}$ in the cache model, but
freezes $W_{2}$, CLIP’s visual encoder and textual encoder. Here, we explore
whether other modules in Tip-Adapter shall be fine-tuned. In Table 5, we
conduct 7 fine-tuning experiments on unfreezing different modules of Tip-
Adapter. Note that we set the learning rate of the two CLIP encoders as 1/1000
of the learning rate of $W_{1}$. The first and second rows represent training-
free Tip-Adapter’s 62.03$\%$ and Tip-Adapter-F’s 65.5$\%$. If we fine-tune
$W_{2}$ in the adapter, the performance would fall to 60.90$\%$ or even
collapse, which accords with our assumption that the one-hot ground-truth
labels in $W_{2}$ shall not be updated. Furthermore, we fix all weights in the
adapter and fine-tune the pre-trained CLIP’s weights. If the visual encoder or
textual encoder is independently tuned, the performance could be improved to
62.84$\%$ and 63.15$\%$, but when both encoders are jointly fine-tuned, the
classification accuracy would significantly drop because of severe over-
fitting, indicating that it is harmful to fine-tune such a huge-parameter
model with few-shot training set.

###  4.5 Visualizations of Tip-Adapter

In order to ease the understanding of cache-initialized Tip-Adapter and the
variation during fine-tuning process, we utilize t-SNE [51] to visualize
$W_{1}$, the first linear layer in the adapter or the cached training
features. As shown in Figure 5, dots in different colors denote 10 categories
of ImageNet, and their relative distances here reflect the high-dimensional
distributions of category embeddings. We conduct the visualization under the
16-shot settings, so there are 16 embeddings per category. From left to right
in Figure 5, the three sub-figures represent the initialized Tip-Adapter and
the Tip-Adapter-F after fine-tuning, respectively. It could be observed that
before training, such distribution has shown good discrimination thanks to the
properly designed approach. During fine-tuning, embeddings of the same
category gradually converges together and different clusters become more
contrastive and separate with each other, contributing to stronger
classification capability. This clustering process in feature space clearly
demonstrates the performance improvement brought by Tip-Adapter-F.

##  5 Conclusion

We propose Tip-Adapter, the non-parametric extension of CLIP-Adapter, which
obtains the weights of adapter not by SGD training but from a cache model
constructed by few-shot training set. In this way, the feature extraction by
adapter could also be viewed as retrieving the few-shot knowledge from the
key-value cache. Surprisingly, Tip-Adapter with such training-free
initialization achieves on par or even better results than CLIP-Adapter with
training. In addition, Tip-Adapter could be further enhanced by fine-tuning
for just a few epochs, a good trade-off between efficiency and performance.
Considering limitations, although it is marginal, Tip-Adapter still requires
20-epoch fine-tuning to learn the best-performed weights. Our future work will
focus on exploring new methods for adapter’s weights construction to fully
unleash its power for visual representation.

## References

  * [1] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In European conference on computer vision, pages 446–461. Springer, 2014.
  * [4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv preprint arXiv:2005.14165, 2020.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [6] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  arXiv preprint arXiv:1904.04232, 2019.
  * [7] Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell.  A new meta-baseline for few-shot learning.  arXiv preprint arXiv:2003.04390, 2020.
  * [8] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [10] Karan Desai and Justin Johnson.  Virtex: Learning visual representations from textual annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11162–11173, 2021.
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [12] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto.  A baseline for few-shot image classification.  arXiv preprint arXiv:1909.02729, 2019.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [15] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004.
  * [16] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135. PMLR, 2017.
  * [17] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  arXiv preprint arXiv:2110.04544, 2021.
  * [18] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li.  Dynamic fusion with intra-and inter-modality attention flow for visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6639–6648, 2019.
  * [19] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of detr with spatially modulated co-attention.  arXiv preprint arXiv:2101.07448, 2021.
  * [20] Edouard Grave, Moustapha Cissé, and Armand Joulin.  Unbounded cache model for online language modeling with open vocabulary.  arXiv preprint arXiv:1711.02604, 2017.
  * [21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [23] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [25] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
  * [26] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [27] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, 2019.
  * [28] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [29] Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei.  Attention on attention for image captioning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4634–4643, 2019.
  * [30] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [31] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  IEEE Transactions on Big Data, 2019.
  * [32] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models.  arXiv preprint arXiv:1911.00172, 2019.
  * [33] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang.  Bilinear attention networks.  arXiv preprint arXiv:1805.07932, 2018.
  * [34] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [35] Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Tom Rainforth, and Yarin Gal.  Self-attention between datapoints: Going beyond individual input-output pairs in deep learning.  arXiv preprint arXiv:2106.02584, 2021.
  * [36] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NIPS, 2012.
  * [38] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [39] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  arXiv preprint arXiv:2110.05208, 2021.
  * [40] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [41] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  arXiv preprint arXiv:2107.13586, 2021.
  * [42] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [43] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  arXiv preprint arXiv:1908.02265, 2019.
  * [44] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [45] Mingyuan Mao, Renrui Zhang, Honghui Zheng, Peng Gao, Teli Ma, Yan Peng, Errui Ding, and Shumin Han.  Dual-stream network for visual recognition.  arXiv preprint arXiv:2105.14734, 2021.
  * [46] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  arXiv preprint arXiv:1609.07843, 2016.
  * [47] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [48] A Emin Orhan.  A simple cache model for image recognition.  arXiv preprint arXiv:1805.08709, 2018.
  * [49] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli.  Scaling neural machine translation.  arXiv preprint arXiv:1806.00187, 2018.
  * [50] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  * [51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [52] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * [53] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [54] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation.  Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
  * [55] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  arXiv preprint arXiv:1703.05175, 2017.
  * [56] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [57] Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin.  Augmenting self-attention with persistent memory.  arXiv preprint arXiv:1907.01470, 2019.
  * [58] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
  * [59] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [60] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 266–282. Springer, 2020.
  * [61] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, 2021.
  * [62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [63] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29:3630–3638, 2016.
  * [64] Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt.  Robust fine-tuning of zero-shot models.  arXiv preprint arXiv:2109.01903, 2021.
  * [65] Raymond E Wright.  Logistic regression.  1995\.
  * [66] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010.
  * [67] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo.  Image captioning with semantic attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659, 2016.
  * [68] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg.  Mattnet: Modular attention network for referring expression comprehension.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307–1315, 2018.
  * [69] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6881–6890, 2021.
  * [70] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  arXiv preprint arXiv:2109.01134, 2021.


further developed in [18] by the introduction of the notion of stable
functorial choice of path objects. We adapt this definition to the right maps
of an awfs as follows.

######  Definition 89.

A _functorial factorisation of the diagonal_ on a category ${\mathcal{C}}$ is
a functor
$P=(r,\rho):\mathbf{Ar}({\mathcal{C}})\to\mathbf{Ar}({\mathcal{C}})\times_{{\mathcal{C}}}\mathbf{Ar}({\mathcal{C}})$
such that $\rho f.rf=\delta_{f}$ for any map $f:A\to B$ in ${\mathcal{C}}$,
where $\delta_{f}:A\to A\times_{B}A$ is the diagonal morphism. Such a
factorisation is called _stable_ if its right leg $\rho$ preserves pullback
squares.

######  Definition 90.

A _stable functorial choice of path objects_ (sfpo) on an awfs
$(\mathbb{L},\mathbb{R})$ is a lift $\mathbf{P}$ of a stable functorial
factorisation of the diagonal $P$:

${\mathbb{R}_{1}}$${\mathbb{L}_{1}\times_{\mathcal{C}}\mathbb{R}_{1}}$${\mathbf{Ar}({\mathcal{C}})}$${\mathbf{Ar}({\mathcal{C}})\times_{\mathcal{C}}\mathbf{Ar}({\mathcal{C}})\hbox
to0.0pt{ .\hss}}$$\scriptstyle{\mathbf{P}}$$\scriptstyle{P}$

######  Proposition 91.

The comprehension category $\mathbb{R}_{1}\to\mathbf{Ar}({\mathcal{C}})$
associated with an awfs with an sfpo admits a pseudo-stable choice of
$\mathrm{Id}$-types.

###### Proof.

The choice of $\mathrm{Id}$-type for an $\mathbb{R}$ map $\mathbf{f}$ is given
by the right leg $\bm{\rho}\mathbf{f}$ of the sfpo
$\mathbf{P}=(\bm{r},\bm{\rho})$; this assignment is functorial and preserves
cartesian morphisms by definition. The rest of the choices of structure are
made as in [5, Proposition 4.9]. ∎

We thus obtain the desired analog of [5, Theorem 4.11].

######  Theorem 92.

A $\pi$-awfs with an sfpo induces a comprehension category with strictly
stable choices of $\Sigma$-, $\Pi$-, and $\mathrm{Id}$-types.

###### Proof.

By Propositions 86, 88, and 91 the comprehension category associated with the
awfs has pseudo stable choices of these types, and so its right adjoint
splitting has strictly stable choices by [5, Theorem 2.6]. ∎

###  8.2. The groupoid model

Recall from Example 60 that the double categories
$(\mathbb{S}\mathbf{plRef}({\bf Cat}),\mathbb{S}\mathbf{plFib}({\bf Cat}))$ on
${\bf Cat}$ of split reflections and split fibrations of categories form an
awfs. This pair does not readily admit a Frobenius structure satisfying the
Beck–Chevalley condition—for that we need the right class to consist of split
_op_ fibrations, as shown in Proposition 81. To remedy this, we lift this awfs
on ${\bf Cat}$ along the inclusion ${\bf Gpd}\to{\bf Cat}$ to one on ${\bf
Gpd}$, because for groupoids the notions of fibration and opfibration
coincide.

######  Proposition 93.

The pair of double categories $(\mathbb{S}\mathbf{plRef}({\bf
Gpd}),\mathbb{S}\mathbf{plFib}({\bf Gpd}))$ over $\mathbb{S}\mathbf{q}({\bf
Gpd})$, together with the lifting operation of Example 37, is a $\pi$-awfs on
${\bf Gpd}$.

###### Proof.

This is the projective/injective lift of the awfs
$(\mathbb{S}\mathbf{plRef}({\bf Cat}),\mathbb{S}\mathbf{plFib}({\bf Cat}))$
along the inclusion ${\bf Gpd}\to{\bf Cat}$, see [3, Section 4.5]. Fibrations
of groupoids are also opfibrations, so there is an isomorphism
$\mathbb{S}\mathbf{plFib}({\bf Gpd})\cong\mathbb{S}\mathbf{plOpFib}({\bf
Gpd})$. Hence, this awfs gets its Frobenius structure from Propositions 71,
which satisfies the Beck–Chevalley condition as per Proposition 81. ∎

######  Proposition 94.

The awfs $(\mathbb{S}\mathbf{plRef}({\bf Gpd}),\mathbb{S}\mathbf{plFib}({\bf
Gpd}))$ admits an sfpo.

###### Proof.

A straightforward adaptation of [5, Proposition 3.5]. ∎

This gives us the following analog of [5, Theorem 5.5].

######  Theorem 95.

The right adjoint splitting of $\mathbf{SplRef}({\bf Gpd})\to\mathbf{Ar}({\bf
Gpd})$ yields a model of dependent type theory with $\Sigma$-, $\Pi$-, and
$\mathrm{Id}$-types.

###### Proof.

By Propositions 93, 94, and Theorem 92. ∎

##  9\. Conclusion

In this paper we have obtained a Frobenius equivalence and formulated a
suitable version of the Beck–Chevalley condition for algebraic weak
factorisation systems. We have shown how these notions can be used to obtain
models of type theory with dependent function types, as illustrated by split
fibrations of groupoids, the basis for the groupoid model of Hofmann and
Streicher [11].

In this way our work is similar to [14, 5], where the authors also utilize an
analogical statement of the Frobenius equivalence for awfss, which is stated
and proven in [6]. However, their version differs from the one we propose here
in several ways. First of all, they develop a version of the Frobenius
equivalence for awfss which are cofibrantly generated by a category; we make
no such assumption. In addition, their works take the “object view” rather
than the “arrow view”, in that they primarily view pullback and pushforward as
operations on objects rather than arrows of slice categories. We hope to have
demonstrated here that it is conceptually simpler to take the arrow view.
Finally, the works of [6, 14, 5] use an interpretation of type theory in which
the dependent types are interpreted using the algebras of the pointed
endofunctor associated with an awfs, rather than the algebras of the monad
associated with an awfs. In that way their framework targets a different class
of examples and does not cover examples like split fibrations of groupoids.

In future work we hope to find more examples. In particular, we would like to
show how our ideas apply to the _effective Kan fibrations_ from [17]. These
are intended to be a good constructive analogue of the Kan fibrations from
simplicial homotopy theory and to lead to a constructive account of
Voevodsky’s model of homotopy type theory in simplicial sets [13]. In [19] the
second author has shown together with Freek Geerligs how the effective Kan
fibrations appear as the right class in an awfs, so our current framework is
the appropriate one for these maps.

## References

  * [1] Steve Awodey and Michael A. Warren  “Homotopy theoretic models of identity types”  In _Mathematical Proceedings of the Cambridge Philosophical Society_ 146.1, 2009, pp. 45–55  DOI: 10.1017/S0305004108001783
  * [2] John Bourke  “An orthogonal approach to algebraic weak factorisation systems”  In _Journal of Pure and Applied Algebra_ 227.6, 2023, pp. 107294  DOI: 10.1016/j.jpaa.2022.107294
  * [3] John Bourke and Richard Garner  “Algebraic weak factorisation systems I: Accessible AWFS”  In _Journal of Pure and Applied Algebra_ 220.1, 2016, pp. 108–147  DOI: 10.1016/j.jpaa.2015.06.002
  * [4] P. J. Freyd and G. M. Kelly  “Categories of continuous functors, I”  In _Journal of Pure and Applied Algebra_ 2.3, 1972, pp. 169–191  DOI: 10.1016/0022-4049(72)90001-1
  * [5] Nicola Gambino and Marco Federico Larrea  “Models of Martin-Löf type theory from algebraic weak factorisation systems”  In _The Journal of Symbolic Logic_ 88.1, 2023, pp. 242–289  DOI: 10.1017/jsl.2021.39
  * [6] Nicola Gambino and Christian Sattler  “The Frobenius condition, right properness, and uniform fibrations”  In _Journal of Pure and Applied Algebra_ 221.12, 2017, pp. 3027–3068  DOI: 10.1016/j.jpaa.2017.02.013
  * [7] Richard Garner  “Understanding the small object argument”  In _Applied Categorical Structures_ 17.3, 2009, pp. 247–285  DOI: 10.1007/s10485-008-9137-4
  * [8] Marco Grandis and Walter Tholen  “Natural weak factorization systems”  In _Archivum Mathematicum_ 42.4 Department of Mathematics, Faculty of Science of Masaryk University, Brno, 2006, pp. 397–408
  * [9] Sina Hazratpour and Emily Riehl  “A 2-categorical proof of Frobenius for fibrations defined from a generic point”  arXiv, 2024  arXiv:2210.00078 [math]
  * [10] Philip S. Hirschhorn  “Model categories and their localizations” 99, Mathematical Surveys and Monographs  American Mathematical Society, 2003  DOI: 10.1090/surv/099
  * [11] Martin Hofmann and Thomas Streicher  “The groupoid interpretation of type theory”  In _Twenty Five Years of Constructive Type Theory_ Oxford University Press, 1998, pp. 83–111  DOI: 10.1093/oso/9780198501275.003.0008
  * [12] Andre Joyal  “Notes on clans and tribes”  arXiv, 2017  arXiv:1710.10238 [math]
  * [13] Krzysztof Kapulkin and Peter LeFanu Lumsdaine  “The simplicial model of univalent foundations (after Voevodsky)”  In _Journal of the European Mathematical Society_ 23.6, 2021, pp. 2071–2126  DOI: 10.4171/jems/1050
  * [14] Marco Federico Larrea  “Models of dependent type theory from algebraic weak factorisation systems”, 2018
  * [15] Paige Randall North  “Type theoretic weak factorization systems”, 2017  DOI: 10.17863/CAM.11207
  * [16] The Univalent Foundations Program  “Homotopy type theory: Univalent foundations of mathematics”, 2023  DOI: 10.48550/arXiv.1308.0729
  * [17] Benno van den Berg and Eric Faber  “Effective trivial Kan fibrations in simplicial sets”  Cham: Springer International Publishing, 2022  DOI: 10.1007/978-3-031-18900-5˙8
  * [18] Benno van den Berg and Richard Garner  “Topological and simplicial models of identity types”  In _ACM Transactions on Computational Logic_ 13.1, 2012, pp. 3:1–3:44  DOI: 10.1145/2071368.2071371
  * [19] Benno van den Berg and Freek Geerligs  “Examples and cofibrant generation of effective Kan fibrations”  arXiv, 2024  arXiv:2402.10568 [math]
  * [20] Wijnand Koen van Woerkom  “Algebraic models of type theory”, 2021