# CoAPT: Context Attribute words for Prompt Tuning

Gun Lee1 Subin An1 Sungyong Baik2 Soochahn Lee1
Kookmin University1, Hanyang University2
{leegun4488, nesquiq<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose a novel prompt tuning method called _CoAPT (Context Attribute words
in Prompt Tuning)_ for few/zero-shot image classification. The core motivation
is that attributes are descriptive words with rich information about a given
concept. Thus, we aim to enrich text queries of existing prompt tuning
methods, improving alignment between text and image embeddings in CLIP
embedding space. To do so, _CoAPT_ integrates attribute words as additional
prompts within learnable prompt tuning and can be easily incorporated into
various existing prompt tuning methods. To facilitate the incorporation of
attributes into text embeddings and the alignment with image embeddings, soft
prompts are trained together with an additional meta-network that generates
input-image-wise feature biases from the concatenated feature encodings of the
image-text combined queries. Our experiments demonstrate that _CoAPT_ leads to
considerable improvements for existing baseline methods on several few/zero-
shot image classification tasks, including base-to-novel generalization,
cross-dataset transfer, and domain generalization. Our findings highlight the
importance of combining hard and soft prompts and pave the way for future
research on the interplay between text and image latent spaces in pre-trained
models.

##  1 Introduction

Vision-language pre-trained models have fueled remarkable progress in
few/zero-shot image classification [38, 17, 61, 60]. Models such as CLIP[38]
or ALIGN[17] are trained by contrastive learning of interrelations between
pairs of text and image, and demonstrate generalization to various downstream
tasks. For image classification as a downstream task, class-specific text
embeddings given by the text encoder from manually crafted prefixes, such as
“a photo of a {class name}.” are used to generate queries that are
subsequently matched with the image embeddings processed through the image
encoder [61, 60].

Transfer learning by fine-tuning is perhaps the most direct approach to adapt
the pre-trained model to the particular downstream task [38]. But this can be
computationally intensive and risks degrading pre-trained knowledge of the
model [58]. Recent works [61, 60] have introduced supplementary prompts to
existing vision language models (VLM), with a fixed backbone encoder. The
additional prompts enrich the context to enhance the correlation of the
embedded tokens of corresponding text and image pairs.

In few-shot classification, these prompts can be categorized into two types:
hard and soft. Hard prompts are text tokens encoded from manually crafted word
sets [41]. Soft prompts are token vectors in the high-dimensional, continuous
latent space, optimized through a learning process [61]. By expanding the
model’s dimensions, soft prompts may offer a more flexible and potentially
more nuanced way for the model to adapt to the given task without the need for
labor-intensive manual text crafting.

The positive impact of soft prompts depends on the design of the prompt and
its optimization process. This has led to many works [61, 60, 25] exploring
various scenarios for initialization, optimization, and configuration of soft
prompts for few/zero-shot classification.  However, such methods increase the
training difficulties and hinder the model explainability. Thus, we turn our
attention back to designing hard prompts that can better facilitate soft
prompt learning, without a significant increase in human effort or
compromising explainability.

Figure 1: Comparative overview of CoAPT. Existing soft prompt tuning methods,
such as CoOp [61], do not fully utilize the text encoder input. The empty
slots in the text query can be enhanced by integrating additional hard
prompts. For the sample classes from the OxfordPets dataset[35], CoAPT
achieves better classification accuracy for both base and new classes.

In this paper, we propose a novel method, termed CoAPT (Context Attribute
words in Prompt Tuning), for incorporating hard prompts with soft prompt
tuning. The core of CoAPT is the addition of _attribute words_ as hard prompts
in tuning soft prompts, as depicted in Figure 1. We believe that our work
provides important contributions:

  * •

CoAPT is simple, easy to implement, and can be integrated with many existing
prompt tuning methods.

  * •

CoAPT demonstrates consistent empirical improvement upon baselines across
base-to-novel generalization, cross-dataset evaluation and domain
generalization tasks.

  * •

We present comprehensive ablative evaluations and analysis of different
variations of CoAPT, with emphasis on the process and configurations for
generating relevant hard prompts.

##  2 Related Work

Vision-Language Models (VLMs). VLMs, including CLIP [38], ALIGN [17], LiT
[55], Filip [52], and Florence [53], have demonstrated capabilities for joint
learning of images and natural language to generate multi-modal latent
representation embeddings. The core technology behind VLMs includes large-
scale training data and contrastive learning methods. Dual encoders for vision
[7, 13] and text [48, 4, 23] are jointly trained on large scale datasets of
paired images and text descriptions. By aligning the embedding vectors of
image and text from the respective encoders during training, VLMs learn the
interrelations between two different modalities. As the number of aligned
pairs increase with larger datasets, the learned interrelations become more
sophisticated, enabling more relevant cross-modal generations, as demonstrated
by GPT-4-vision [1] and Gemini-Pro [44].

Prompt Learning in Natural Language Processing originated from the idea of
appending prefixes as language instructions to the input text. This method
enabled pre-trained LLMs like GPT-3,4[4, 1], Llama[46] to adapt to downstream
tasks in few-shot or zero-shot settings. Subsequent research focused on
constructing more effective prompts. Researchers have explored generating
optimal prompts through methods like text mining[19], paraphrasing[19, 12,
54], and gradient-based approaches[28, 27, 41, 59, 37, 11]. These advances
demonstrate the potential of prompt learning for robust transfer learning.

Prompt Learning in Vision-Language Model. In CLIP [38], pre-defined prefixes
are used as hard prompts to evaluate the pre-trained model on several zero-
shot tasks. Subsequent methods [38, 42, 10] employed a global soft prompt
together with the input for the target tasks. Further methods were proposed
using sets of soft prompts, trained per input class or task [61, 20, 40], or
trained to generate input-dependent prompts [60, 21]. These methods can also
be categorized by whether the prompts are for text or image features. In CoOp
[61] and CoCoOp [60], the soft prompts are defined for the text encoder. In
MaPLe [24], VPT [18], and PromptSRC [25], trainable soft prompts are defined
for the image encoder.

Augmenting Prompt Tuning in Few/Zero-Shot Classification. The research
community has explored various methods to generate task-specific text prompts,
enhancing model adaptability in few/zero-shot scenarios. In [57, 33], they
utilize pre-existing metadata like EXIF of images or scientific name of class
as their text prompt. Studies such as [45, 36, 22, 29, 8, 31] explored
leveraging the inherent knowledge within foundation models. Their methods
incorporated automated prompt generation per task, class, or instance, with
pre-trained foundation models. In [36], LLMs are instructed to generate
descriptions of the class object, which are directly used as prompts without
soft prompts. In [45], LLM responses go through additional prompt mining
processes, including manual curation, for better alignment with input image
features. In [2], the use of CLIP inferred per-image attributes such as
background or orientation as query context is explored.

Figure 2: Overview of CoAPT method with baseline prompt learning model. The
CoAPT method consists of two steps. First, attribute words are generated using
a language model, which is a one-time process. Second, during prompt learning,
these words are combined with the soft prompt and class token. Inputs generate
queries processed by a Meta-network, adding a bias term to the text queries.
The combined image-text queries are then used to maximize the score for the
ground-truth class.

##  3 Method

###  3.1 Preliminaries

Prompt Engineering. CLIP [38] has provided a text and image encoder pair that
is highly correlated with an open vocabulary. Naturally, CLIP has been
successfully applied for image classification. Given an image dataset
comprising $|C|$ classes and a specified class name word embedding denoted as
$\mathbf{c}_{i}$, $i=\\{1,...,|C|\\}$, a text prompt
$\mathbf{y}_{i}=\\{\mathbf{w}_{SOS},\mathbf{w}_{1},\mathbf{w}_{2},\cdots,\mathbf{c}_{i},\mathbf{w}_{EOS}\\}$
is crafted as the embedding of the context phrase with the class name, such as
“a photo of a {class name}”. Here, $\mathbf{w}_{j}$ are the embeddings of the
phrase words, with $\mathbf{w}_{SOS}$ and $\mathbf{w}_{EOS}$ denoting the
start and end of the sentence respectively.

This prompt is then processed by the CLIP text encoder, denoted as $g_{T}$, to
produce a class-specific text feature encoding
$\mathbf{t}_{i}=g_{T}(\mathbf{y}_{i})$. Simultaneously, an image $\mathbf{x}$
to be classified is passed through the model’s image encoder, $g_{V}$, to
generate an image-specific feature, $\mathbf{f}=g_{V}(\mathbf{x})$. The
classification of the image into one of the $|C|$ categories is then
determined by computing a probability distribution:

$P(y=i|\mathbf{x})=\frac{\exp\left(\cos\left(\mathbf{f},\mathbf{t}_{i}\right)/\tau\right)}{\sum_{j=1}^{|C|}\exp\left(\cos\left(\mathbf{f},\mathbf{t}_{j}\right)/\tau\right)},$
(1)

over the class labels $y=\\{1,2,\cdots,|C|\\}$, based on the similarities
between per-class $\mathbf{t}^{i}$ and $\mathbf{f}$. The parameter $\tau$
represents the temperature in the CLIP model, controlling the range of logits
in the softmax function.

Soft Prompt Learning. Adaptation of CLIP models can be challenging because the
original information can be lost as training progresses. Additionally, the
model’s large number of parameters can lead to overfitting, especially when
using a small dataset. Using prompt learning, the model can be readily adapted
to various settings with only a small portion of additional parameters, while
preserving pre-trained weights.

Soft prompts are trainable continuous tokens, which can be defined for both
the text encoder and image (vision) encoder, denoted here as
$\mathbf{P}^{T}=\\{\mathbf{p}^{T}_{m}\\}_{m=1}^{M}$,
$\mathbf{P}^{V}=\\{\mathbf{p}^{V}_{m}\\}_{m=1}^{M}$, respectively. By learning
the best soft prompts for the given dataset, they can be effective for
classification by complementing the handcrafted prompts $\mathbf{y}_{i}$. They
can be easily concatenated for both text as
$\tilde{\mathbf{y}}_{i}=\\{\mathbf{w}_{SOS},\mathbf{p}^{T}_{1},\cdots,\mathbf{p}^{T}_{M},\mathbf{c}_{i},\mathbf{w}_{EOS}\\}$,
and image as
$\tilde{\mathbf{x}}=\\{\mathbf{x},\mathbf{p}^{V}_{1},\mathbf{p}^{V}_{2},\cdots,\mathbf{p}^{V}_{M}\\}$.
This results in augmented feature embeddings
$\tilde{\mathbf{t}}_{i}=g_{T}(\tilde{\mathbf{y}}_{i})$ and
$\tilde{\mathbf{f}}=g_{V}(\tilde{\mathbf{x}})$. The probability distribution
over the class labels $l_{i}$ is also computed by Eq.(1), but with input
$\tilde{\mathbf{t}}_{i}$ and $\tilde{\mathbf{f}}$ in place of
${\mathbf{t}}_{i}$ and ${\mathbf{f}}$.

###  3.2 Context Attribute Words for Prompt Tuning

Context Attribute Words. Whether the contexts are phrase words or soft
prompts, they are invariably less than the total permitted context length in
many existing methods [61, 60, 24, 25, 34, 56], resulting in suboptimal
prompts and thus failing to fully exploit the capability of VLMs, as described
in Figure 1. To provide rich relevant information instead of empty slots and
thereby take full advantage of prompt and VLM capability, we fill these empty
context slots with words that are highly correlated with a given class:
namely, attributes of the class. While there may be many different ways to
collect these words, in this work, we apply a large language model (LLM) to
generate these words to minimize manual human efforts. We design a template
instruction phrase that is customized with the dataset and class names, on
which LLM is conditioned to generate the attribute words
$\mathbf{A}_{i}=\\{\mathbf{a}^{n}_{i}\\}^{N}_{n=1}$. As attribute words are
generated for each class, they are denoted as $\mathbf{a}^{n}_{i}$. Then, the
$i_{th}$ class text prompt is augmented with $\mathbf{A}_{i}$, as
$\tilde{\mathbf{y}}^{a}_{i}=\\{\mathbf{w}_{SOS},\mathbf{p}^{T}_{1},\cdots,\mathbf{p}^{T}_{M},\mathbf{c}_{i},\mathbf{a}^{1}_{i},\cdots,\mathbf{a}^{N}_{i},\mathbf{w}_{EOS}\\}$.
The attribute words provide enhanced context, improving discriminative
representation from soft prompt tuning. We show sample attribute words in the
supplemental material.

Input Image Specific Query Adaptation. Depending on the resulting tuned soft
prompts, the attribute-enhanced context in soft prompt tuning may contain too
much information about general information about a given class, hindering the
alignment with a given image feature. To better align with a given image
feature and still exploit context attributes, we adapt the encoded query text
feature $\tilde{\mathbf{t}}^{a}_{i}=g_{T}(\tilde{\mathbf{y}}^{a}_{i})$ with an
input-image-text-specific bias term
$\boldsymbol{\beta}_{i}=g_{M}(\tilde{\mathbf{t}}^{a}_{i},\tilde{\mathbf{f}})$.
Here, $g_{M}$ denotes a simple meta-network comprising multiple linear layers
coupled with activation functions. This process enhances the alignment between
modalities by ensuring that the relevant class text features are better
aligned with the corresponding image features. The class probability after
adaptation is then represented as:

$P(y=i|\mathbf{x})=\frac{\exp\left(\cos(\tilde{\mathbf{f}},\tilde{\mathbf{t}}^{a}_{i}+\boldsymbol{\beta}_{i})/\tau\right)}{\sum_{j=1}^{|C|}\exp\left(\cos(\tilde{\mathbf{f}},\tilde{\mathbf{t}}^{a}_{j}+\boldsymbol{\beta}_{j})/\tau\right)}.$
(2)

Inference Only Vocabulary Ensemble. If we ask the LLM the same question
multiple times, the answer may slightly vary each time. Thus, attribute words
generated from the LLM may vary for each sample output, leading to nontrivial
performance variations. During training, we just fix the vocabulary to a
particular sample output. But for inference on novel classes, we use three
sample sets of $\mathbf{A}^{k}_{i}$, $k=\\{1,2,3\\}$. We then use the ensemble
score from the average of each score from $\mathbf{A}^{k}_{i}$, as follows:

$P_{novel}(y=i|\mathbf{x})=\sum_{k=1}^{K}P(y=i|\mathbf{x},\mathbf{A}^{k}_{i})$
(3)

##  4 Experiments

###  4.1 Settings

Following the seminal work [60], the proposed CoAPT and other models are
evaluated for three different few-shot classification tasks: base-to-novel
generalization, cross-dataset transfer, and domain generalization.

Base-to-Novel Generalization. The base-to-novel generalization evaluation aims
to assess the generalization of learning methods. To do so, each dataset is
first partitioned into two disjoint subsets: base and novel classes. Then,
examples of base classes are further divided into support sets for training
and query sets for evaluation. In particular, after training on few examples
(support sets) of base classes, methods are assessed with respect to the
performance on unseen examples (query sets) of seen base classes (base-to-
base); the performance on unseen novel classes (base-to-novel); and the
harmonic mean of the two.

Cross-Dataset Transfer. To further evaluate the generalization of learning
algorithms and models to unseen task distributions, they are first trained on
a single dataset and then evaluated on diverse datasets with different
classes, thus different task distributions.

Domain Generalization. Domain generalization evaluation has a similar goal to
cross-dataset transfer evaluation: evaluating the generalization to unseen
distributions. The difference is while cross-dataset transfer evaluation
focuses on different task distribution (different classes), domain
generalization evaluation looks at same task but different domains (e.g.,
styles).

Implementation Details. Following the settings from [60], we employ the
ViT-B/16 architecture of CLIP as a backbone architecture, and all experimental
results are obtained by averaging results across three different random seeds.
For each seed, our method generates three attribute word vocabulary from a
large language model. The results of baselines and other methods are
reproduced with their respective open-source code.

Method | Avg. over 11 datasets | ImageNet | Caltech101 | OxfordPets
---|---|---|---|---
Base |  Novel |  HM |  Base |  Novel | HM |  Base |  Novel | HM |  Base |  Novel | HM
CoOp [61] |  82.69 |  63.22 |  71.66 |  76.47 |  67.88 | 71.92 |  98.00 |  89.81 | 93.73 |  93.67 |  95.29 | 94.47
+CoAPT |  82.70+0.01 |  72.79+9.57 |  77.43+5.77 |  76.32 |  68.96 | 72.45 |  98.28 |  92.83 | 95.48 |  94.74 |  97.58 | 96.13
CoCoOp [60] |  80.47 |  71.69 |  75.83 |  75.98 |  70.43 | 73.10 |  97.96 |  93.81 | 95.84 |  95.20 |  97.69 | 96.43
+CoAPT |  82.66+2.19 |  74.45+2.76 |  78.34+2.51 |  75.93 |  68.32 | 71.92 |  98.15 |  93.09 | 95.55 |  94.49 |  97.18 | 95.82
KgCoOp [51] |  80.73 |  73.60 |  77.00 |  75.83 |  69.96 | 72.78 |  97.72 |  94.39 | 96.03 |  94.65 |  97.76 | 96.18
+CoAPT |  82.62+1.89 |  74.62+1.02 |  78.42+1.42 |  76.67 |  70.19 | 73.28 |  98.15 |  93.63 | 95.83 |  94.15 |  97.76 | 95.92
MaPLe [24] |  82.28 |  75.14 |  78.55 |  76.66 |  70.54 | 73.47 |  97.74 |  94.36 | 96.02 |  95.43 |  97.76 | 96.58
+CoAPT |  83.77+1.49 |  75.13-0.01 |  79.21+0.66 |  77.10 |  68.73 | 72.67 |  98.43 |  93.16 | 95.72 |  95.41 |  97.71 | 96.54
PromptSRC [25] |  84.26 |  76.10 |  79.97 |  77.60 |  70.73 | 74.01 |  98.10 |  94.03 | 96.02 |  95.33 |  97.30 | 96.30
+CoAPT |  84.74+0.48 |  77.07+0.97 |  80.72+0.75 |  77.87 |  69.88 | 73.66 |  98.43 |  93.81 | 96.06 |  95.59 |  97.71 | 96.63
ProMetaR [34] |  84.39 |  76.93 |  80.49 |  77.76 |  70.75 | 74.09 |  98.11 |  94.29 | 96.16 |  95.57 |  97.43 | 96.49
+CoAPT$\dagger$ |  84.00-0.39 |  77.73+0.80 |  80.74+0.25 |  77.54 |  70.63 | 73.92 |  98.41 |  94.61 | 96.47 |  95.62 |  97.69 | 96.64
DePT [56] |  85.19 |  76.17 |  80.43 |  78.20 |  70.27 | 74.02 |  98.57 |  94.10 | 96.28 |  95.43 |  97.33 | 96.37
+CoAPT |  84.92-0.27 |  77.26+1.09 |  80.91+0.48 |  76.88 |  67.20 | 71.72 |  98.77 |  94.76 | 96.72 |  94.67 |  97.52 | 96.07
Method | StanfordCars | Flowers102 | Food101 | FGVCAircraft
Base |  Novel |  HM |  Base |  Novel | HM |  Base |  Novel | HM |  Base |  Novel | HM
CoOp [61] |  78.12 |  60.40 |  68.13 |  97.60 |  59.67 | 74.06 |  88.33 |  82.26 | 85.19 |  40.44 |  22.30 | 28.75
+CoAPT |  76.44 |  70.83 |  73.53 |  97.37 |  74.04 | 84.09 |  89.89 |  90.73 | 90.31 |  40.86 |  38.47 | 39.63
CoCoOp [60] |  70.49 |  73.59 |  72.01 |  94.87 |  71.75 | 81.71 |  90.70 |  91.29 | 90.99 |  33.41 |  23.71 | 27.74
+CoAPT |  74.98 |  71.40 |  73.14 |  97.37 |  75.44 | 85.01 |  89.74 |  90.53 | 90.13 |  40.40 |  38.73 | 39.54
KgCoOp [51] |  71.76 |  75.04 |  73.36 |  95.00 |  74.73 | 83.65 |  90.50 |  91.70 | 91.09 |  36.21 |  33.55 | 34.83
+CoAPT |  75.14 |  72.51 |  73.80 |  97.88 |  77.21 | 86.30 |  89.70 |  90.46 | 90.08 |  40.20 |  39.25 | 39.70
MaPLe [24] |  72.94 |  74.00 |  73.47 |  95.92 |  72.46 | 82.56 |  90.71 |  92.05 | 91.38 |  37.44 |  35.61 | 36.50
+CoAPT |  77.59 |  72.41 |  74.90 |  97.40 |  77.68 | 86.43 |  89.91 |  90.71 | 90.31 |  41.16 |  38.31 | 39.67
PromptSRC [25] |  78.27 |  74.97 |  76.58 |  98.07 |  76.50 | 85.95 |  90.67 |  91.53 | 91.10 |  42.73 |  37.87 | 40.15
+CoAPT |  79.73 |  72.67 |  76.03 |  97.94 |  78.53 | 87.17 |  90.72 |  91.56 | 91.14 |  45.72 |  38.63 | 41.87
ProMetaR [34] |  78.32 |  75.18 |  76.72 |  98.13 |  77.66 | 86.70 |  90.80 |  91.89 | 91.34 |  42.02 |  38.63 | 40.25
+CoAPT† |  76.80 |  74.64 |  75.70 |  97.69 |  80.14 | 88.05 |  90.76 |  91.75 | 91.25 |  40.94 |  38.31 | 39.56
DePT [56] |  80.80 |  75.00 |  77.79 |  98.40 |  77.10 | 86.46 |  90.87 |  91.57 | 91.22 |  45.70 |  36.73 | 40.73
+CoAPT |  81.35 |  74.20 |  77.61 |  98.64 |  79.36 | 87.95 |  90.45 |  91.58 | 91.01 |  46.66 |  38.17 | 41.98
Method | SUN397 | DTD | EuroSAT | UCF101
Base |  Novel |  HM |  Base |  Novel | HM |  Base |  Novel | HM |  Base |  Novel | HM
CoOp [61] |  80.60 |  65.89 |  72.51 |  79.44 |  41.18 | 54.24 |  92.19 |  54.74 | 68.69 |  84.69 |  56.05 | 67.46
+CoAPT |  81.03 |  74.56 |  77.66 |  81.17 |  54.87 | 65.48 |  89.35 |  59.44 | 71.30 |  84.23 |  78.40 | 81.20
CoCoOp [60] |  79.74 |  76.86 |  78.27 |  77.01 |  56.00 | 64.85 |  87.49 |  60.04 | 71.21 |  82.33 |  73.45 | 77.64
+CoAPT |  80.45 |  75.68 |  77.99 |  81.83 |  58.21 | 68.03 |  91.18 |  70.78 | 79.02 |  84.76 |  79.65 | 82.12
KgCoOp [51] |  80.29 |  76.53 |  78.36 |  77.55 |  54.99 | 64.35 |  85.64 |  64.34 | 73.48 |  82.89 |  76.67 | 79.65
+CoAPT |  81.03 |  76.57 |  78.74 |  80.63 |  53.06 | 63.99 |  90.45 |  71.10 | 79.34 |  84.87 |  79.07 | 81.86
MaPLe [24] |  80.82 |  78.70 |  79.75 |  80.36 |  59.18 | 68.16 |  94.07 |  73.23 | 82.35 |  83.00 |  78.66 | 80.77
+CoAPT |  81.55 |  76.27 |  78.82 |  82.41 |  61.55 | 70.35 |  95.33 |  71.50 | 81.65 |  85.18 |  78.38 | 81.61
PromptSRC [25] |  82.67 |  78.47 |  80.52 |  83.37 |  62.97 | 71.75 |  92.90 |  73.90 | 82.32 |  87.10 |  78.80 | 82.74
+CoAPT |  82.64 |  78.42 |  80.47 |  84.18 |  63.49 | 72.36 |  93.11 |  81.67 | 86.99 |  86.18 |  81.41 | 83.72
ProMetaR [34] |  82.70 |  79.02 |  80.82 |  83.02 |  64.05 | 72.31 |  94.94 |  77.44 | 85.30 |  86.97 |  79.84 | 83.25
+CoAPT† |  82.36 |  79.22 |  80.76 |  82.79 |  64.73 | 72.65 |  95.41 |  81.24 | 87.75 |  85.68 |  82.06 | 83.83
DePT [56] |  83.27 |  78.97 |  81.06 |  84.80 |  61.20 | 71.09 |  93.23 |  77.90 | 84.88 |  87.73 |  77.70 | 82.46
+CoAPT |  82.05 |  77.58 |  79.75 |  84.76 |  65.62 | 73.97 |  93.60 |  82.00 | 87.38 |  86.31 |  81.92 | 84.06

Table 1: Comparison of accuracy w/ or w/o CoAPT for base-to-novel
generalization task. We adapt the CoAPT method with 7 baseline models that do
not fully utilize text encoder input. HM refers to harmonic mean. Bold letters
indicate better performance between the baseline and CoAPT. Underlined text
signifies the highest accuracy across methods. $\dagger$ denotes models
without the meta-net $g_{M}$, only adding context attribute words during
training and inference.

Datasets. We follow the procedure from [60] for constructing datasets. For
base-to-novel generalization and cross-dataset transfer tasks, we collect 11
different datasets, each constructed for different purposes and hence
containing different data distributions. These include two generic-object
datasets (ImageNet [6] and Caltech101 [9]); five fine-grained datasets
(OxfordPets [35], StanfordCars [26], Flowers102 [32], Food101 [3], and
FGVCAircraft [30]); scene recognition dataset (SUN397 [50]); action
recognition dataset (UCF101 [43]); texture analysis dataset (DTD [5]); and
satellite imagery dataset (EuroSAT [14]). For the domain generalization task,
we take ImageNet as the primary source dataset, while taking its four variants
as target datasets: ImageNetV2 [39], ImageNet-Sketch [49], ImageNet-A [16],
and ImageNet-R [15]. For all classes in all datasets, the support set
comprises 16 shots, as sampled in [60].

| Source | Target |
---|---|---|---
|  ImageNet |  Caltech101 |  OxfordPets |  StanfordCars |  Flowers102 |  Food101 |  FGVCAircraft |  SUN397 |  DTD |  EuroSAT |  UCF101 |  Average
CoOp | 71.51 | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88
+CoAPT | 71.51 | 93.50 | 91.05 | 62.96 | 70.73 | 84.71 | 24.11 | 65.60 | 43.68 | 36.43 | 67.27 | 64.00+0.12
CoCoOp | 71.02 | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 | 65.74
+CoAPT | 70.39 | 93.79 | 92.07 | 64.34 | 71.01 | 84.98 | 24.51 | 67.13 | 46.04 | 39.02 | 68.46 |  65.14-0.60
KgCoOp | 70.66 | 93.92 | 89.83 | 65.41 | 70.01 | 86.36 | 22.51 | 66.16 | 46.35 | 46.04 | 68.50 | 65.51
+CoAPT | 71.04 | 94.00 | 92.17 | 66.57 | 71.47 | 85.75 | 25.74 | 67.11 | 46.77 | 37.81 | 69.40 | 65.68+0.17
MaPLe | 70.72 | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 | 66.30
+CoAPT | 71.87 | 93.09 | 92.11 | 64.28 | 70.02 | 84.86 | 25.22 | 66.11 | 46.73 | 47.37 | 67.82 |  65.76-0.54
PromptSRC | 71.27 | 93.60 | 90.25 | 65.70 | 70.25 | 86.15 | 23.90 | 67.10 | 46.87 | 45.50 | 68.75 | 65.81
+CoAPT | 71.35 | 94.23 | 92.18 | 65.29 | 70.82 | 86.06 | 25.63 | 67.22 | 47.87 | 53.38 | 68.91 |  67.16+1.35
ProMetaR | 71.29 | 93.74 | 90.59 | 65.83 | 71.13 | 86.39 | 24.78 | 67.41 | 47.08 | 45.02 | 69.50 | 66.15
+CoAPT† | 70.77 | 93.93 | 91.81 | 64.57 | 70.83 | 85.96 | 24.91 | 67.27 | 48.31 | 48.50 | 69.08 | 66.52+0.37
DePT | 71.60 | 93.80 | 90.13 | 66.00 | 70.93 | 86.27 | 24.30 | 67.23 | 46.60 | 45.83 | 69.10 | 66.02
+CoAPT | 71.73 | 93.67 | 91.36 | 64.62 | 70.09 | 85.79 | 24.68 | 67.07 | 48.58 | 51.45 | 69.06 | 66.64+0.62
Table 2: Comparison of accuracy w/ or w/o CoAPT for cross-dataset transfer
task. Bold letters indicate better performance between the baseline and CoAPT.
Underlined text signifies the highest accuracy across methods. $\dagger$
denotes model without the meta-net $g_{M}$.

###  4.2 Quantitative Evaluation

To show the effectiveness, applicability, and flexibility of our CoAPT, we
apply our CoAPT on top of several baselines: namely, CoOp [61], CoCoOp [60],
KgCoOp [51], MaPLe [24], PromptSRC [25], ProMetaR [34] and DePT [56].  As
ProMetaR already employs a meta-network to each text and image feature, we do
not apply $g_{m}$ to ProMetaR to avoid redundancy.

Base-to-Novel Generalization results of baselines and its CoAPT-enhanced
version are presented in Table 1 across 11 datasets. We maintain all attribute
words $\mathbf{A}_{i}$ and the meta-net architecture consistent across all
baseline methods and datasets. On average, CoAPT leads to improvements in the
harmonic mean of the base and novel class accuracies for all baseline models.
CoAPT leads to improvements for both base and novel class classification,
except for MaPLe, ProMetaR and DePT. The results demonstrate that combining
enhanced context in hard prompts with soft prompt tuning can lead to better
dataset adaptation and generalization.

Cross-Dataset Transfer results are presented in Table 2, evaluating the
transferability of knowledge learned by models trained on ImageNet to various
other datasets consisting of different classes. CoAPT generally improved the
performance of baselines, especially achieving the largest performance
improvement and thus the best results with PromptSRC as a baseline. It seems
that the regularization process of PromptSRC, involving multiple template
phrases as textual augmentation, can complement CoAPT more effectively than
other baselines. However, a slight decrease in performance is observed after
applying CoAPT on CoCoOp and Maple. Our conjecture is that this is due to
their additional layers, such as meta-net architectures, which may introduce
overfitting of the soft prompts.

| Source | Target |
---|---|---|---
| ImageNet | -V2 | -S | -A | -R | Avg.
CoOp | 71.51 | 64.20 | 47.99 | 49.71 | 75.21 | 59.28
+CoAPT | 71.51 | 64.18 | 48.25 | 50.53 | 75.28 | 59.56+0.28
CoCoOp | 71.02 | 64.07 | 48.75 | 50.63 | 76.18 | 59.91
+CoAPT | 70.39 | 63.38 | 48.65 | 50.35 | 76.11 |  59.62-0.29
KgCoOp | 71.20 | 64.10 | 48.97 | 50.69 | 76.70 | 60.11
+CoAPT | 71.04 | 64.27 | 49.14 | 51.08 | 76.87 | 60.34+0.23
MaPLe | 70.72 | 64.07 | 49.15 | 50.90 | 76.98 | 60.27
+CoAPT | 71.87 | 64.72 | 49.01 | 47.78 | 76.43 |  59.49-0.78
PromptSRC | 71.27 | 64.35 | 49.55 | 50.90 | 77.80 | 60.65
+CoAPT | 71.35 | 64.75 | 49.83 | 50.95 | 77.65 |  60.79+0.14
ProMetaR | 71.29 | 64.39 | 49.55 | 51.25 | 77.89 | 60.77
+CoAPT$\dagger$ | 70.77 | 64.32 | 49.52 | 51.41 | 77.59 |  60.71-0.06
DePT∗ | 71.58 | 64.32 | 48.82 | 49.74 | 76.76 | 59.91
+CoAPT | 71.73 | 64.18 | 47.56 | 48.20 | 75.11 |  58.76-1.15
Table 3: Comparison of accuracy w/ or w/o CoAPT for domain generalization
task. Bold letters indicate better performance between the baseline and CoAPT.
Underlined text signifies the highest accuracy across methods. $\dagger$
denotes model without the meta-net $g_{M}$. ∗ denotes our reproduced results.

Domain Generalization results are presented in Table 3, evaluating the
robustness of models to domain changes. CoAPT is shown to bring improvements
upon CoOp, KgCoOp, and PromptSRC. In particular, PromptSRC + CoAPT surpasses
all other baselins and its CoAPT variants, showing the effectiveness of CoAPT.
However, for other baselines, CoAPT is observed to fail to bring improvements.
The accuracy drop may be due to the fact that attribute words tend to contain
semantic information, which may not help to cope with different domains, such
as styles of images and textures.

Figure 3: Ablative evaluation of base-to-novel generalization on number of
context attribute words with CoAPT integrated into PromptSRC [25]. Plots are
accuracy of base classes (a), novel classes (b), and harmonic mean (c),
averaged over 11 datasets.

###  4.3 Ablation Study

In this section, we perform ablation studies to assess the effectiveness of
each component of our proposed CoAPT. We perform all ablation studies on
PromptSRC as a main baseline in base-to-new generalization task.

Number of Attribute Words. We analyze the influence of the number of context
attribute words on the performance, as shown in Figure 3. No (zero) attribute
word corresponds tosing the meta-net architecture without any attribute word.
With respect to all base-to-base accuracy, base-to-novel accuracy, and the
harmonic mean of the two, a general trend of accuracy increase with the number
of attribute words is observed. In particular, we observe the highest
improvement in the harmonic mean with 32 attribute words, which is the maximum
number within the maximum context length.

CoAPT Subcomponents. To evaluate the contribution of each component of CoAPT
to its overall performance, we sequentially add each component one at a time
and assessed the impact on accuracy, as presented in Table 4. The positive
effect of each component is evident, with the biggest portion coming from the
combination of the context attributes with input specific query adaptation
from the meta-net. We also demonstrate the flexibility of CoAPT by integrating
with the DePT [56] method, also added to a PromptSRC baseline, to achieve the
maximum accuracy.

Method | Base Cls. | Novel Cls. | HM
---|---|---|---
0: PromptSRC: reported in [25] | 84.26 | 76.10 | 79.97
1: PromptSRC: Reproduced | 84.24 | 75.81 | 79.80
2: + Context Attribute Words | 84.56 | 75.95 |  80.03+0.23
3: + Input Specific Query Adaptation | 84.74 | 76.99 |  80.68+0.65
4: + Inference Vocabulary Ensemble | 84.74 | 77.07 |  80.72+0.04
5: + DePT [56] | 84.92 | 77.26 |  80.91+0.19
Table 4: Ablative evaluation of CoAPT components for base-to-novel
generalization. Accuracy values are averaged over 11 datasets. HM refers to
harmonic mean.

Meta-net Configurations. Table 5 presents the ablative evaluations on the
meta-net configurations of CoAPT: namely, the output of the meta-net and where
meta-net outputs are applied. In particular, we assess two options for the
meta-net output: just a bias vector or affine transformation parameters (scale
and bias vectors). As for the location of application of the meta-net outputs,
the outputs are either applied to text encoding
$\tilde{\mathbf{t}}^{a}_{i}=g_{T}(\tilde{\mathbf{y}}^{a}_{i})$ or soft prompts
$\mathbf{P}^{T}$.

Our experiments indicate that adding bias to the text encoding
$\tilde{\mathbf{t}}^{a}_{i}$ yields the highest performance. Interestingly,
while the accuracy difference between bias and affine transformations when
applied to $\mathbf{P}^{T}$ is small, the difference is much larger when
applied to $\tilde{\mathbf{t}}^{a}_{i}$, by a margin of more than 2% points.
Perhaps the expansion of the controllable dimensions from the affine
transformation may result in overfitting, rather than generalizability of soft
prompts.

Output type | Applied to | Base Cls. | Novel Cls. | HM
---|---|---|---|---
Affine | $\mathbf{P}^{T}$ | 84.53 | 76.97 | 80.57
Affine | $\tilde{\mathbf{t}}^{a}_{i}$ | 84.70 | 72.91 | 78.36
Bias | $\mathbf{P}^{T}$ | 84.32 | 77.25 | 80.63
Bias | $\tilde{\mathbf{t}}^{a}_{i}$ | 84.74 | 77.07 | 80.72
Table 5: Ablative evaluation of meta-net configuration for base-to-novel
generalization. $\tilde{\mathbf{t}}^{a}_{i}$ and $\mathbf{P}^{T}$ denote the
text query encoding and soft prompt, respectively. Accuracy values are
averaged over 11 datasets. HM refers to harmonic mean. Figure 4: Qualitative
comparative analysis of attribute words from GPT4-Language and GPT4-Vision.
Color denotes change in accuracy (scaled in range $[0,1]$) when replacing
GPT4-Language generated word (x-axis) with that of GPT4-Vision (y-axis).

LLMs for Attribute Word Generation. In Table 6, we analyze how the performance
is affected by a language model used to generate attribute words. We choose
LLaMa2-7B, LLaMa2-13B [47], GPT4-Language, and GPT4-Vision [1] as a possible
candidate language model. Serving as a pseudo oracle, when using GPT4-Vision
as the attribute word generator, we provide images from the few-shot support
set with customized instruction based on the class name for both base and
novel classes.

We observe that the accuracy increases with the capacity of the large model,
especially for novel classes. This indicates that the context attribute words
indeed can positively enhance generalizability to novel classes, and that
larger models tend to provide more relevant attribute words.

Another notable observation is the improvement from using GPT4-Vision, which
is additionally provided with images of a support set to generate context
attribute words. While the use of the support set for novel classes violates
the assumption of zero-shot classification, serving as pseudo oracle, it
demonstrates that better attribute words can be generated with better input
instructions. We believe that both these observations warrant further research
in the attribute word generation process.

Vocabulary | Base Cls. | Novel Cls. | HM
---|---|---|---
LLaMa2-7B | 84.62 | 75.29 | 79.68
LLaMa2-13B | 84.71 | 75.49 | 79.83
GPT4-Language | 84.74 | 77.07 | 80.72
GPT4-Vision | 84.88 | 79.85 | 82.28
Table 6: Ablative evaluation of attribute word generators for base-to-novel
generalization. Accuracy values are averaged over 11 datasets. HM refers to
harmonic mean.

Pairwise comparison of attribute words from GPT4-Language and GPT4-Vision. To
investigate which specific attribute word may enhance or hinder classification
performance, we perform qualitative analysis by replacing each attribute word
generated using GPT4-Language with one that was generated using GPT4-Vision (a
pseudo oracle), in Figure 4. Here, we exclude the meta-net for ease of
comparison, as well as to exclusively examine the direct differences between
wordsppwo. We guide the reader to focus on the red and orange colors in Figure
4, which correspond to considerable improvements from the GPT-Vision generated
attribute words.

For the Eurosat dataset, the classification accuracy of the class “sea or
lake” considerably improves by inserting the words “blue”, “deep”, and “deep-
sea” into the context. In the DTD dataset, the classification of the “meshed”
class benefits from the words “copper”, “nets”, “latticed”, “chain-link”, and
“grid”. In the UCF101 dataset, the classification of the “Nunchucks” class is
enhanced with the words “flipping”, “performance”, and “spinning”. We can also
see from the vertical patterns that the replacement of GPT4-Language word
“Nunchucks”, which is a duplicate of the class name, and words “training” and
“rotation” with almost any GPT4-Vision word results in improvement. While it
is not clear-cut what the common characteristics between these words are, we
can generally note that the GPT4-Vision words have more visual
characteristics, in that they may be more readily visualized as an image. For
example, the words “nets”, “latticed”, and “grid” for the “meshed” class can
be correlated to unambiguous visual structures, more so compared to
GPT4-Language words “rough”, “smooth”, “hard” and “soft”, to which many
different visual descriptions can be associated.

##  5 Conclusion

We introduce CoAPT, a novel method that effectively integrates attribute words
into the query context within soft prompt tuning in to enhance text queries in
the CLIP embedding space. CoAPT is a plug-and-play method that can be
seamlessly incorporated into various existing soft prompt tuning methods to
improve the alignment between text and image embeddings. A meta-network is
incorporated to compute feature biases that act as adaptation for the image-
text combined queries.

Our experiments have shown that CoAPT can considerably improve accuracy across
all few/zero-shot image classification tasks. The results highlight the
importance of using appropriate hard prompts within the input query context.

The improvements from GPT4-Vision demonstrate that there are more effective
attribute words. We believe our analysis on generating the optimal context
attribute words may guide further research into the interactions between text
and image latent spaces in pre-trained models.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Bang An, Sicheng Zhu, Michael-Andrei Panaitescu-Liess, Chaithanya Kumar Mummadi, and Furong Huang.  Perceptionclip: Visual classification by inferring and conditioning on contexts.  In International Conference on Learning Representations, 2024.
  * [3] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.
  * [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [8] Reza Esfandiarpoor and Stephen Bach.  Follow-up differential descriptions: Language models resolve ambiguities for image classification.  In The Twelfth International Conference on Learning Representations, 2023.
  * [9] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004.
  * [10] Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP 2021, pages 3816–3830. Association for Computational Linguistics (ACL), 2021.
  * [11] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May.  Warp: Word-level adversarial reprogramming.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4921–4933, 2021.
  * [12] Adi Haviv, Jonathan Berant, and Amir Globerson.  Bertese: Learning to speak to bert.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3618–3623, 2021.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
  * [15] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021.
  * [16] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021.
  * [17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  * [18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In European Conference on Computer Vision, pages 709–727. Springer, 2022.
  * [19] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig.  How can we know what language models know?  Transactions of the Association for Computational Linguistics, 8:423–438, 2020.
  * [20] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie.  Prompting visual-language models for efficient video understanding.  In European Conference on Computer Vision, pages 105–124. Springer, 2022.
  * [21] Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song.  Generating instance-level prompts for rehearsal-free continual learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11847–11857, 2023.
  * [22] Prannay Kaul, Weidi Xie, and Andrew Zisserman.  Multi-modal classifiers for open-vocabulary object detection.  arXiv preprint arXiv:2306.05493, 2023.
  * [23] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [24] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023.
  * [25] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan.  Self-regulating prompts: Foundational model adaptation without forgetting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15190–15200, 2023.
  * [26] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [27] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
  * [28] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021.
  * [29] Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Sebe, and Elisa Ricci.  Democratizing fine-grained visual recognition with large language models.  In The Twelfth International Conference on Learning Representations, 2023.
  * [30] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [31] Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel E O’Connor.  Enhancing clip with gpt-4: Harnessing visual descriptions as prompts.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 262–271, 2023.
  * [32] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE, 2008.
  * [33] Shubham Parashar, Zhiqiu Lin, Yanan Li, and Shu Kong.  Prompting scientific names for zero-shot species recognition.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9856–9861, 2023.
  * [34] Jinyoung Park, Juyeon Ko, and Hyunwoo J Kim.  Prompt learning via meta-regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26940–26950, 2024.
  * [35] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  * [36] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi.  What does a platypus look like? generating customized prompts for zero-shot image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691–15701, 2023.
  * [37] Guanghui Qin and Jason Eisner.  Learning how to ask: Querying lms with mixtures of soft prompts.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, 2021.
  * [38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [39] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In International conference on machine learning, pages 5389–5400. PMLR, 2019.
  * [40] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E Gonzalez, Kurt Keutzer, and Trevor Darrell.  Multitask vision-language prompt tuning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5656–5667, 2024.
  * [41] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, 2020.
  * [42] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao.  Test-time prompt tuning for zero-shot generalization in vision-language models.  Advances in Neural Information Processing Systems, 35:14274–14289, 2022.
  * [43] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [45] Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang.  Argue: Attribute-guided prompt tuning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28578–28587, 2024.
  * [46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [49] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  Advances in Neural Information Processing Systems, 32, 2019.
  * [50] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010.
  * [51] Hantao Yao, Rui Zhang, and Changsheng Xu.  Visual-language prompt tuning with knowledge-guided context optimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6757–6767, 2023.
  * [52] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  In International Conference on Learning Representations, 2021.
  * [53] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [54] Weizhe Yuan, Graham Neubig, and Pengfei Liu.  Bartscore: Evaluating generated text as text generation.  Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
  * [55] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022.
  * [56] Ji Zhang, Shihan Wu, Lianli Gao, Hengtao Shen, and Jingkuan Song.  Dept: Decoupled prompt tuning.  arXiv preprint arXiv:2309.07439, 2023.
  * [57] Chenhao Zheng, Ayush Shrivastava, and Andrew Owens.  Exif as language: Learning cross-modal associations between images and camera metadata.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6945–6956, 2023.
  * [58] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You.  Preventing zero-shot transfer degradation in continual learning of vision-language models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19125–19136, 2023.
  * [59] Zexuan Zhong, Dan Friedman, and Danqi Chen.  Factual probing is [mask]: Learning vs. learning to recall.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, 2021.
  * [60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022.
  * [61] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, 2022.

## Appendix

##  Appendix A Attribute Word Samples Generated by Various Foundation Models

We leverage several foundation models [1, 47] to generate attribute words. Due
to the inherent randomness and varying parameters of these models, the
generated attribute words can differ. To maintain consistency in the attribute
word results, we design a template instruction and apply it uniformly across
all foundation models. The general instruction is as follows:

“Please provide only {Number of words} descriptive English words for the
{Dataset name} {Class name} dataset. Format your response as follows:
$\backslash$nword$\backslash$nword$\backslash$nword$\backslash$nword”

We note that we have included explicit instructions to the model to generate
English words only, as the model may generate words in other languages or even
nonverbal outputs such as emoji. We also include explicit instruction on the
output format, for seamless data loading in downstream processes.

For the GPT-4 Vision model, instructions are given with the few-shot sample
images, making the instructions slightly different, as follows:

“Please take a look at this(these) image file(s) and provide only {Number of
words} descriptive English words for the {Dataset name} {Class name} dataset.
Format your response as
follows:$\backslash$nword$\backslash$nword$\backslash$nword$\backslash$nword”

Tables 7 and 8 presents samples of attribute words for the base-to-novel
dataset [6, 9, 35, 26, 32, 3, 30, 50, 5, 14, 43] generated by different
foundation models. Full attribute words set and code available in :
https://github.com/LeeGun4488/CoAPT

DB | Class | Model |  Attribute word set
---|---|---|---
ImageNet |  Goldfish | GPT4-V |  Goldfish Aquarium Orange Fins Swimming Carassius Auratus Water
GPT4-L |  Goldfish Aquatic Fish Ornamental Pet Swimming Water Bowl
LLaMa2-13B |  Scales Fins Eyes Mouth Body Tail Colors Shapes
LLaMa2-7B |  Fish Gold Fishy Aquarium Swim School Tank Fin
Castle | GPT4-V |  Fortified Historic Medieval Towers Battlements Moat Stone Architecture
GPT4-L |  Medieval Fortress Palace Towers Gothic Stone Historic Majestic
LLaMa2-13B |  Majestic Medieval Towering Fortified Grandiose Historic Picturesque Sprawling
LLaMa2-7B |  Medieval Tower Moat Battlements Crenellations Drawbridge Curtain wall
Caltech101 |  Beaver | GPT4-V |  Beaver Rodent Mammal Tail Fur Brown Water Aquatic
GPT4-L |  Animal Beaver Wildlife Nature Water Dam Canadian Aquatic
LLaMa2-13B |  Fur Tail Dam Pond Log Tree Branch Water
LLaMa2-7B |  Object Image Animal Beaver Water Scene Nature Aquatic
Laptop | GPT4-V |  Laptop Portable Computer Notebook Keyboard Screen Display Trackpad
GPT4-L |  Electronics Technology Computing Screen Hardware Laptop Gray Black
LLaMa2-13B |  Sleek Modern Slim Lightweight High-performance Powerful Efficient Fast
LLaMa2-7B |  Laptop Computer Device Gadget Technology Tool Machinery Equipment
OxfordPets |  Beagle | GPT4-V |  Beagle Dog Pet Animal Canine Tricolored Brown White
GPT4-L |  Adorable Alert Athletic Brown Curious Cute Domestic Energetic
LLaMa2-13B |  Fluffy Adorable Playful Curious Energetic Friendly Loyal Cuddly
LLaMa2-7B |  Adorable Playful Loyal Friendly Curious Energetic Affectionate Intelligent
Bombay | GPT4-V |  Bombay Cat Black Feline Glossy Yellow Eyes Domestic
GPT4-L |  Furry Shiny Playful Cute Sleek Affectionate Tuxedo Intelligent
LLaMa2-13B |  Furry Cute Playful Fluffy Adorable Friendly Curious Energetic
LLaMa2-7B |  Dog Cat Puppy Kitten Paw Fur Whiskers Tail
StanfordCars |  2012 Ferrari 458 Italia Coupe | GPT4-V |  Ferrari 458 Italia Coupe Red Yellow Luxury Sports
GPT4-L |  Ferrari 458 Italia Coupe 2012 Luxury Sports Speed
LLaMa2-13B |  Sleek Sporty Luxurious Red Shiny Powerful Aerodynamic High-performance
LLaMa2-7B |  Sleek Sporty Luxury Powerful Agile Aerodynamic Stylish Speedster
2012 Audi S5 Coupe | GPT4-V |  Audi S5 Coupe 2012 Automobile Vehicle Luxury Performance
GPT4-L |  Luxurious Sleek Sporty Sophisticated Compact Powerful Stylish Modern
LLaMa2-13B |  Sleek Sporty Luxurious High-performance Powerful Agile Stylish Advanced
LLaMa2-7B |  Coupe Audi S5 Luxury Sports Sedan German Engine
Flowers102 |  Passion Flower | GPT4-V |  Passiflora Exotic Floral Radial Purple White Green Stigma
GPT4-L |  Passionflower Exotic Twining Climbing Tropical Subtropical Vibrant Multicolored
LLaMa2-13B |  Delicate Intricate Vibrant Exotic Colorful Intriguing Beautiful Detailed
LLaMa2-7B |  Petal Flower Vine Intricate Delicate Beauty Nature Bloom
Rose | GPT4-V |  Blooming Floral Petals Roses Gardening Botany Cultivation Fragrance
GPT4-L |  Rose Blooming Pink Red Floral Petals Garden Thorns
LLaMa2-13B |  Delicate Petal Velvety Red Full Bloom Soft Vibrant
LLaMa2-7B |  Beauty Bloom Colorful Delicate Elegant Fragrant Graceful Lovely
Food101 |  Chocolate Mousse | GPT4-V |  Chocolate Mousse Dessert Creamy Garnish Berries Whipped Cream
GPT4-L |  Tasty Chocolatey Sweet Creamy Rich Decadent Whipped Velvety
LLaMa2-13B |  Rich Creamy Decadent Smooth Silky Velvety Dark Bittersweet
LLaMa2-7B |  Rich Creamy Velvety Smooth Luxurious Decadent Dense Chocolatey
Hamburger | GPT4-V |  Beef Bun Sesame Lettuce Tomato Pickle Cheese Grilled
GPT4-L |  Bun Beef Cheese Lettuce Tomato Onion Pickle Mustard
LLaMa2-13B |  Juicy Savory Soft Meaty Tasty Cheesy Crispy Fresh
LLaMa2-7B |  Juicy Beef Bun Cheese Lettuce Tomato Onion Pickles
Table 7: Sample attribute words per class in various datasets(Imagenet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101) DB | Class | Model |  Attribute word set
---|---|---|---
FGVCAircraft |  Global Express | GPT4-V |  Airplane Jet Wing Landing-gear Tail Fuselage Cockpit Windows
GPT4-L |  Global Express Aircraft Dataset FGVCAircraft Aerial Imagery Remote
LLaMa2-13B |  Aircraft Global Express Flight Travel Airline Passenger Journey
LLaMa2-7B |  Jet Aircraft Global Express Luxury Private Plane Flying
Falcon 900 | GPT4-V |  Falcon 900 Aircraft Private Jet Trijet Landing Gear
GPT4-L |  Aircraft Aerospace Technology Jet Twin-engine Luxury Travel Transportation
LLaMa2-13B |  Luxurious Private Jet Plane Business Travel High-end Leather
LLaMa2-7B |  Jet Luxury Private Charter Flight Business Travel Aircraft
SUN397 |  Aquarium | GPT4-V |  Aquarium Fish Water Coral Visitors Children Tunnel Marine
GPT4-L |  Aquatic Colorful Coral Fishes Tropical Marine Underwater Biodiversity
LLaMa2-13B |  Fish Tank Water Coral Seaweed Anemone Clownfish Goby
LLaMa2-7B |  Underwater Fish Tank Coral Reef Shark Sea Creatures
Cemetery | GPT4-V |  Tombstones Graves Cemetery Monuments Crosses Memorial Peaceful Trees
GPT4-L |  Grave Tombstone Cemetery Memorial Solemn Tranquil Peaceful Death
LLaMa2-13B |  Graveyard Tombstones Headstones Epitaphs Memorials Monuments Cenotaphs Obelisks
LLaMa2-7B |  Ancient Graves Headstones Tombstones Epitaphs Memorials Gravestones Markers
DTD |  Dotted | GPT4-V |  Polka-dots Circles Patterns Textured Colorful Variegated Spotted Repetitive
GPT4-L |  Spotted Punctuated Speckled Dotted Pinpoint Stippled Freckled Mottled
LLaMa2-13B |  Bumpy Rough Dotted Speckled Spotted Splotchy Mottled Flecked
LLaMa2-7B |  Rough Smooth Bumpy Ridgy Prickly Silky Gritty Sticky
Woven | GPT4-V |  Interwoven Basketweave Textured Fabric Material Craftsmanship Handwoven Detailed
GPT4-L |  Rough Soft Patterned Threaded Grainy Coarse Smooth Ribbed
LLaMa2-13B |  Soft Rough Bumpy Smooth Fuzzy Velvety Shiny Dense
LLaMa2-7B |  Bumpy Rough Smooth Soft Silky Sandy Gritty Sticky
EuroSAT |  Highway or Road | GPT4-V |  Aerial Highway Road Infrastructure Transportation Vehicles Asphalt Traffic
GPT4-L |  Asphalt Motorway Vehicles Transportation Urban Rural Traffic Suburban
LLaMa2-13B |  Road Highway Lane Path Pavement Asphalt Concrete Shoulder
LLaMa2-7B |  Road Highway Street Lane Junction Intersection Bridge Tunnel
River | GPT4-V |  Water River Aerial Satellite Imagery Curvature Meandering Flow
GPT4-L |  River Water Flow Stream Current Brook Rapids Outdoor
LLaMa2-13B |  River Water Stream Creek Watershed Channel Current Flow
LLaMa2-7B |  River Water Waters Flow Current Channel Shore Bank
UCF101 |  Playing Violin | GPT4-V |  Violin Performance Musicians Concert Classical Instrument Stringed Bow
GPT4-L |  Musical Violin Bow Strings Melodic Concentration Performance Instrument
LLaMa2-13B |  Skilled Gestures Movements Fingers Bowing Strings Posture Precision
LLaMa2-7B |  Violin Music Performance Notes Melody Rhythm Bowing Strings
Surfing | GPT4-V |  Surfing Waves Ocean Beach Surfers Surfboards Balance Riding
GPT4-L |  Surfing Ocean Waves Beach Surfboard Balance Splash Surfers
LLaMa2-13B |  Waves Ocean Beach Sun Sand Surf Board Ripples
LLaMa2-7B |  Wave Surf Ocean Beach Catch Ride Wiped Spray
Table 8: Sample attribute words per class in various datasets(FGVCAircraft,
SUN397, DTD, EuroSAT, UCF101)