# LLMs as Visual Explainers: Advancing Image Classification with
Evolving Visual Descriptions

Songhao Han Le Zhuo11footnotemark: 1 Yue Liao Si Liu
Beihang University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contribution.Corresponding author.

###### Abstract

Vision-language models (VLMs) offer a promising paradigm for image
classification by comparing the similarity between images and class
embeddings. A critical challenge lies in crafting precise textual
representations for class names. While previous studies have leveraged recent
advancements in large language models (LLMs) to enhance these descriptors,
their outputs often suffer from ambiguity and inaccuracy. We identify two
primary causes: 1) The prevalent reliance on textual interactions with LLMs,
leading to a mismatch between the generated text and the visual content in
VLMs’ latent space - a phenomenon we term the “explain without seeing”
dilemma. 2) The oversight of the inter-class relationships, resulting in
descriptors that fail to differentiate similar classes effectively. To address
these issues, we propose a novel image classification framework combining VLMs
with LLMs, named Iterative Optimization with Visual Feedback. In particular,
our method develops an LLM-based agent, employing an evolutionary optimization
strategy to refine class descriptors. Crucially, we incorporate visual
feedback from VLM classification metrics, thereby guiding the optimization
process with concrete visual data. Our method leads to improving accuracy on a
wide range of image classification benchmarks, with $3.47\%$ average gains
over state-of-the-art methods. We also highlight the resulting descriptions
serve as explainable and robust features that can consistently improve the
performance across various backbone models. Code is available here.

##  1 Introduction

In recent years, a plethora of vision-language models (VLMs) [29, 1, 16, 35]
has emerged, showcasing impressive transfer learning capabilities across
diverse visual tasks. These models, by pretraining on large datasets, learn to
align images and text within a shared embedding space. Unlike conventional
models, VLMs classify images by computing the similarity between the input
image and textual descriptions. Notably, CLIP [29] achieves outstanding
results in zero-shot image classification tasks across various datasets. This
is achieved by employing a combination of class names and pre-defined
templates as input prompts and then matching images to the most similar
prompt.

Figure 1: Schematic of the method. (a) Previous methods use an LLM to generate
descriptive prompts for each class directly. (b) Our method optimizes class
descriptions through an evolutionary process. We utilize a VLM (such as CLIP
[29]) to obtain visual feedback,  _e.g._ , the confusion matrix, assessing the
quality of current descriptions. Upon building the visual feedback, an LLM
generates refined category descriptions, iterating multiple times to achieve
the final optimal category descriptions.

This paradigm, though effective, is highly dependent on the quality of class
prompts. For instance, in datasets with abstract or ambiguous class names,
such as CUB [34] and Flowers102 [25], CLIP struggles to effectively
distinguish images using class names as the sole prompts. The advent of Large
Language Models (LLMs) [2, 27, 3, 30] has prompted research [24, 31, 26, 28]
into enhancing class descriptions through LLMs. These methods exploit LLMs’
extensive world knowledge to generate more detailed and semantically rich
descriptions for each category, thereby enriching the class prompts. Despite
these advancements, current approaches exhibit several drawbacks. LLMs,
trained exclusively on text, lack a nuanced understanding of visual concepts.
Consequently, when provided only with textual class names, LLMs tend to
produce ambiguous or inaccurate descriptions [24],  _e.g._ ,  _“short stubby
beak”_ for Cactus Wren, which actually has curved and relatively long beaks.
Evidence from WaffleCLIP [31] also suggests replacing LLM-generated class
descriptions with random, meaningless characters does not hurt the overall
classification performance, questioning the effectiveness of these methods.
Moreover, the fundamental goal of these methods is to approximate the global
optimal centroids within the CLIP embedding space for all classes using
descriptive texts generated by LLMs. Achieving this necessitates considering
inter-category relationships and engaging in an iterative optimization
process. Current methodologies, as depicted in Figure 1, are tailored to
generate descriptions for individual classes in a single iteration. As a
result, the generated descriptions tend to be overly general, with multiple
categories sharing similar phrases,  _e.g._ ,  _“various colors”_ in bird
classification. This generality hinders the ability to effectively
discriminate between similar categories.

In light of these limitations, a key question arises: How can we design an
automated pipeline that empowers LLMs to discover globally optimal class
descriptions, thereby improving the overall visual classification performance?
In this paper, we introduce a novel approach, named Iterative Optimization
with Visual Feedback, which demonstrates how an LLM agent can collaborate with
VLMs, employing the feedback of visual classification to progressively refine
class descriptions (Figure 1). Our method formulates this task as a
combinatorial optimization problem - identifying the combination of class
descriptions for each category that maximizes VLM image classification
performance. Given the problem’s infinitely complex search space, we develop
an LLM agent integrated with a Genetic Algorithm, where descriptions are
evolved toward better solutions. Within each iterative cycle, the agent first
conducts mutation based on the last round’s descriptions and then performs
crossover among various candidates to produce optimized concepts. This dual
process of mutation and crossover allows the agent to explore the solution
space both locally and globally, searching for the most effective visual
concepts. We further introduce the concept of visual feedback to reduce
variance across different results and computational resources, using image
classification metrics from CLIP. Visual feedback can serve as both reward and
memory for our agent, steering the LLM towards rational optimization and
mitigating random-walk behavior during the process.

Extensive experiments conducted across nine image classification benchmark
datasets reveal that our approach significantly outperforms current LLM-based
methods as well as vanilla CLIP. We demonstrate that our LLM agent is able to
iteratively discover highly descriptive visual descriptions that are conducive
to image classification. We highlight another key insight that the final
optimized class descriptions serve as a robust representation with strong
interpretability, which can consistently improve model performance across
different backbones.

In summary, our contributions are: 1) We identify the limitations in existing
methods and propose a novel paradigm for LLM-augmented visual classification
using a genetic algorithm-inspired agent for iterative class description
refinement with visual feedback. 2) Our approach consistently outperforms
existing methods across different datasets. Without any parameter update, we
achieve an increase of up to $19.45\%$ absolute improvements over vanilla CLIP
performance ($6.40\%$ on average). 3) We demonstrate the explainability and
transferability of our optimized class descriptions.

##  2 Related Work

Figure 2: Illustration of iterative optimization with visual feedback. (a)
Given raw class names as input, we first prompt the LLM to generate an
initialization of class descriptors. These descriptors undergo an iterative
optimization comprising three stages: mutation, where diverse new candidates
are generated; crossover, involving mixing and matching across different
candidates to produce better candidates; and natural selection, selecting the
most suitable candidate based on a fitness function. The final optimized set
of class descriptors is used to evaluate image classification performance. (b)
In each iteration, we compute visual metrics including classification accuracy
and confusion matrix for current class descriptors. We further use these
metrics to construct visual feedback, update memory banks, and pick the best
candidate in natural selection. (c) Through this iterative optimization, the
LLM progressively identifies the most effective class descriptors, thereby
enhancing the differentiation between ambiguous classes.

Large Language Models. Large Language Models (LLMs) have exhibited remarkable
proficiency and sophisticated reasoning skills, significantly influencing
various domains within artificial intelligence [7, 30, 2, 27, 33, 22, 3].
These models have been proven capable of solving complex tasks, once thought
to be solely within human capability, such as mathematical reasoning [36, 13],
drug discovery [17, 21], and decision makeing [41, 23]. Their success in these
areas underscores the planning and reasoning capabilities of LLMs.
Furthermore, LLMs have shown immense potential in the multimodal domain [11,
45, 20, 43, 37]. Most researchers align well-trained encoders from various
modalities with LLMs through instruction tuning, equipping these models to
interpret multimodal inputs. In contrast, our approach leverages a gradient-
free method to integrate visual knowledge into LLMs without any need for fine-
tuning. Nevertheless, the inherent limitations of LLMs in comprehending
alternate modalities exacerbate the phenomenon of “hallucination” in
multimodal contexts [40, 19, 5], leading to inaccurate or even erroneous
outputs. Thus, addressing this issue of hallucinations in LLMs represents a
pivotal challenge.

Prompt Engineering. Originating in NLP, prompt engineering significantly
impacts the performance of VLMs in downstream tasks, leading many extensive
research into identifying the optimal prompt. Prompt tuning [14, 12, 42, 9,
44, 18], a method of parameter-efficient fine-tuning, involves introducing
learnable parameters before the input text or image, which are then optimized
through gradient updates. For instance, CoOp [9] improves class descriptions
by incorporating a set of parameters to represent dataset context, optimizing
prediction accuracy via cross-entropy loss minimization. While prompt tuning
notably increases accuracy, it necessitates additional training. Our method
achieves comparable results without any training and serves as a complementary
approach to prompt tuning, offering further precision improvements when
applied subsequently.

Using LLMs for Prompt Engineering. Recent advancements have seen the emergence
of methods that employ LLMs to generate semantically richer descriptions for
improving class prompts [24, 31, 28, 26, 39]. [24] initially demonstrated that
ensembling class-dependent descriptions generated by LLMs can improve
classification accuracy. Building on this, WaffleCLIP [31] incorporated high-
level concepts related to the dataset to mitigate ambiguities in class names.
CuPL [28], meanwhile, utilizes a series of hand-crafted prompt templates to
enable LLMs to produce diverse descriptions for each class. However, we
observed that descriptions generated by these LLM-based methods often suffer
from inaccuracies and ambiguities. Consequently, we propose a method involving
iterative optimization to continuously refine descriptions, integrating visual
feedback within the optimization process to enable LLMs to maximize
differentiation between distinct categories.

##  3 Method

In this section, we elucidate the motivation and design space of our proposed
method. We first introduce the standard setting of LLM-augmented visual
classification in Section 3.1. In Section 3.2, we describe the components of
our method in detail, explaining how to use the LLM agent to iteratively
optimize class descriptors with visual feedback. The complete framework is
illustrated in Figure 2.

###  3.1 Classification with Descriptor Ensembling

CLIP [29] consists of an image encoder and a text encoder, which has been
trained on 400M image-text pairs to learn a joint embedding space. Given a
query image $x$ and a predefined set of classes
$\mathcal{C}=\\{c_{1},c_{2},c_{3},...,c_{n}\\}$ in natural language, CLIP
performs zero-shot image classification by first encoding both image and class
names into the shared embedding space, then computing the cosine similarity
between the image and each class, finally selecting the one with highest
similarity as the predicted class,

$\tilde{c}=\operatorname*{arg\,max}_{c\in\mathcal{C}}\cos(\phi_{I}(x),\phi_{T}(f(c))),$
(1)

where $\phi_{I},\phi_{T}$ are the image encoder and text encoder, and $f(c)$
is the prompt template like "A photo of a {c}".

Prior work [24] proposed a simple yet effective method to augment the class
names $\mathcal{C}$ using LLMs. They prompt LLMs to generate a set of
descriptors for each category, _e.g._ , Hen: two legs; red, brown, or white
feathers; a small body. With these descriptive texts, they improve image
classification accuracy by computing a comprehensive similarity score for each
category:

$\tilde{c}=\operatorname*{arg\,max}_{c\in\mathcal{C}}\frac{1}{|\mathcal{D}(c)|}\sum_{d\in\mathcal{D}(c)}\cos(\phi_{I}(x),\phi_{T}(d)),$
(2)

where $\mathcal{D}(c)$ indicates the descriptors for class $c$, and we use
$\mathcal{D}$ for short. By averaging the scores of all class descriptors,
which achieves prompt ensembling, we argue that it reduces the noise of class
name embedding and leads to more robust visual classification.

###  3.2 Iterative Optimization with Visual Feedback

A key drawback of existing LLM-based methods is that they generate class
descriptors in a single run, where LLM is frozen and not updated. By contrast,
the ways humans recognize new objects always involve a dynamic learning
process,  _i.e._ , we gradually update our knowledge base of objects via
interaction with the environment, remembering useful features and forgetting
useless features. Inspired by that, we identify two fundamental points:
interaction with the environment and iterative optimization, which are missing
in existing methods. We formulate the problem of finding the optimal class
descriptors as a combinatorial optimization problem. Further, we propose a
novel method to dynamically optimize the set of class descriptors in an
iterative manner integrated with visual feedback from CLIP.

Taking advantage of extended world knowledge and remarkable reasoning skills
showcased by LLMs [27, 36], we empower LLMs to act as prompt-optimization
agents to search for the best combination of class descriptors. Considering
the complex solution space of possible combinations of class descriptors, we
introduce an evolutionary process to search for optimal class descriptors.
Classic genetic algorithms have been proven superior performance in solving
complex optimization problems. The evolution usually starts with a population
of randomly generated samples. In each iteration, it generates the next
generation via two predefined operators,  _mutation_ and  _crossover_ , then
selects the most promising offspring based on some fitness function that
evaluates the quality of an individual. We illustrate the detailed process of
each iteration in Algorithm 1, which will be explained in the following
paragraphs.

Initialization. Since the size of class labels can be large, especially for
datasets like Imagenet [6], we first design a splitting strategy to group all
classes into clusters based on the similarity of their names. We extract the
text embedding of class names and then employ the K-means algorithm to cluster
them into groups, where each group represents semantically similar classes. To
make sure that LLM always focuses on confusing classes, the clustering step is
dynamically conducted not only at initialization but also at the start of each
iteration, where we compute the average embedding of all descriptors for each
class.

After clustering, we condition the LLM to generate initial class descriptors
$\mathcal{D}_{0}$. Specifically, we instruct the LLM to generate $n_{0}$
descriptors for each class, providing task description, output formatting, and
some design tips. We provide more implementation details about our prompts in
Supplementary Material. The initialization step is similar to previous methods
[24, 31, 28]. LLM can generate plausible descriptors usually related to
colors, shapes, textures, etc. On the first try, however, LLM sometimes
generates features that are ambiguous and irrelevant to visual classification
due to the high degree of diversity and lack of specific pre-training data for
visual understanding. Therefore, it is necessary to introduce both visual
feedback from CLIP and iterative optimization to mitigate these issues.

Algorithm 1 Interative Optimization with Visual Feedback

0: class labels $\mathcal{C}$, LLM LLM, prompt prompt, visual feedback $V$

0: iterations $N$, mutation sample size $K$

1: Set $\mathcal{C}\leftarrow\text{K-means}(\mathcal{C})$

2: for $i=1$ to $|\mathcal{C}|$ do

3: $\mathcal{D}_{0,i}=\texttt{LLM}(c_{i},\texttt{prompt})$

4: end for

5: Set $\mathcal{M}\leftarrow\emptyset$,
$\tilde{\mathcal{D}}\leftarrow\mathcal{D}_{0}$

6: for $i=1$ to $N$ do

7: Set $\mathcal{D}_{i-1}\leftarrow\text{K-means}(\mathcal{D}_{i-1})$

8: for $j=1$ to $|\mathcal{D}_{i-1}|$ do

9:
$\mathcal{D}_{i,j}^{0},...,\mathcal{D}_{i,j}^{K}=\texttt{LLM}(\texttt{prompt},\mathcal{D}_{i-1,j},\mathcal{M},V)$

10:
$\mathcal{D}_{i,j}^{K+1}=\texttt{LLM}(\texttt{prompt},\mathcal{D}_{i-1,j},\mathcal{D}_{i,j}^{0},...,\mathcal{D}_{i,j}^{K},V)$

11:
$\mathcal{D}_{i,j}=\operatorname*{arg\,max}_{\mathcal{D}\in\\{\mathcal{D}_{i,j}^{0},...,\mathcal{D}_{i,j}^{K+1}\\}}V(\mathcal{D})$

12: end for

13:
$\mathcal{M}=\texttt{Update}(\mathcal{M},\mathcal{D}_{i-1},\mathcal{D}_{i},V)$

14:
$\tilde{\mathcal{D}}=\operatorname*{arg\,max}_{\mathcal{D}\in\\{\mathcal{D}_{i}\,\tilde{\mathcal{D}}\\}}V(\mathcal{D})$

15: end for

15: $\tilde{\mathcal{D}}$

Visual Feedback. The key idea of visual feedback is grounding LLMs with visual
knowledge in VLMs to better distinguish between ambiguous classes during the
optimization process. Instead of directly updating the model parameters via
instruction tuning in recent multimodal LLMs, we propose a gradient-free
method to inject visual knowledge into LLMs. Specifically, given a current set
of class descriptors $\mathcal{D}$, we construct visual feedback
$V(\mathcal{D})$ through task-related evaluation metrics for CLIP, _e.g._ ,
top-1 overall accuracy, class-wise accuracy, and confusion matrix in image
classification. These metrics offer a holistic evaluation of model performance
given current class descriptors.

Apart from conventional metrics, we propose an improved version of the
confusion matrix to more effectively capture intricate relationships within
classes. We define a confusing threshold $\lambda$ and categorize each
prediction as a positive sample based on its cosine similarity score compared
to $\lambda$ times the cosine similarity score of the ground-truth label. In
this approach, the improved confusion matrix, denoted as $\tilde{M}$, is
computed by aggregating positive sample indicators. Another challenge is we
find that LLMs sometimes struggle to interpret the raw confusion matrix with
shape $|\mathcal{D}|*|\mathcal{D}|$, particularly as the number of classes
$|\mathcal{D}|$ increases. Hence, we refine this process by extracting the
top-$m$ classes from each row of $\tilde{M}$, representing the most confusing
classes for CLIP. The visual feedback using the improved confusion matrix is
formulated as follows:

$\displaystyle\text{Positive}(x,d)$ $\displaystyle=\begin{cases}1&\text{if
}\cos(x,d)>\lambda\cos(x,d_{gt})\\\ 0&\text{otherwise}\end{cases},$ (3)
$\displaystyle\tilde{M}_{dd^{\prime}}$
$\displaystyle=\sum_{x\in\mathcal{X}}\text{Positive}(x,d^{\prime})\quad\text{for
}d,d^{\prime}\in\mathcal{D},$ (4) $\displaystyle V(\mathcal{D})$
$\displaystyle=\bigcup_{d\in\mathcal{D}}\text{Top-}m\left(\tilde{M}_{d*}\right),$
(5)

where we fix $\lambda=0.9,m=3$ in all experiments and $\mathcal{X}$ indicates
the set of images to compute classification matrics in visual feedback.

Though this metric-based visual feedback is simple to construct, it serves an
important role in estimating the divergence between LLM-generated descriptors
and optimal classification centroids in CLIP latent space. It has three major
applications in our optimization process. First, the visual feedback
$V(\mathcal{D})$ can be converted into natural language through a customized
tostring() function. During mutation and crossover, the textual version of
$V(\mathcal{D})$ can be integrated into input prompts to help LLM distinguish
the target category from confusing classes. Second, we adopt $V(\mathcal{D})$
as the fitness function to evaluate sample quality and perform natural
selection. Finally, we introduce the idea of memory banks $M$ consisting of
positive and negative history class descriptors. We dynamically update the
memory banks based on $V(\mathcal{D})$ at the end of each iteration.

Iterative Optimization. We first define the mutation and crossover operators
to generate the next generation of class descriptors based on the previous
version. For the $i$-th iteration, providing the previous set of class
descriptors $\mathcal{D}_{i-1}$, $V(\mathcal{D}_{i-1})$, and memory banks $M$,
we query LLM to pick the top $n_{i}$ most useless descriptors in the current
set and replace them with $n_{i}$ new descriptors to emphasize its distinct
visual features, representing the mutation operation. We generate $K$
independent candidates
$\\{\mathcal{D}_{i}^{0},\mathcal{D}_{i}^{1},...,\mathcal{D}_{i}^{K}\\}$ from
LLM in each iteration to ensure sufficient genetic diversity for optimization.
As for the crossover operation, we provide $K$ generated candidates in the
mutation operation and their corresponding visual feedback as inputs then
prompt LLM to perform mix and match between different samples, then output a
new sample $\mathcal{D}_{i}^{K+1}$. The key idea for crossover is to ensemble
different useful descriptors of different samples and produce an offspring
with overall better performance.

At the end of each iteration, we select the best performance candidate as
$\mathcal{D}_{i}$ to update the current descriptor set among the population of
generated class descriptors denoted as
$\\{\mathcal{D}_{i}^{0},\mathcal{D}_{i}^{1},...,\mathcal{D}_{i}^{K+1}\\}$. We
adopt the overall accuracy in our visual feedback as the fitness score for
natural selection. This fitness-based process ensures our method always
chooses the best candidate as the starting point of the next iteration and
produces class descriptors that better discriminate different categories, thus
gradually moving towards the global optimal. To figure out the impact of
descriptors and update the memory banks $M$, we compare the difference of
$\mathcal{D}_{i}$ and $\mathcal{D}_{i-1}$ in detail, resulting in three groups
of descriptors,  _i.e._ , unchanged, deleted, and added descriptors. Then we
compute the visual feedback for these descriptors and compare their overall
accuracy. If the accuracy of $\mathcal{D}_{i}$ is greater than that of
unchanged descriptors, it indicates the added descriptors are beneficial for
CLIP and we add them to the positive memory bank, otherwise, we add them to
the negative one. Similarly, if the accuracy of $\mathcal{D}_{i-1}$ is greater
than that of unchanged descriptors, it indicates the deleted descriptors are
beneficial for CLIP and we add them to the negative memory bank, otherwise, we
add them to the positive one.

##  4 Experiments

Method | Dataset | Average
---|---|---
ImageNet | EuroSAT | UCF101 | SUN | Caltech | DTD | CIFAR-10 | Flowers102 | CUB
CLIP [29] | 61.80 | 36.83 | 61.01 | 61.51 | 91.24 | 42.73 | 84.28 | 63.59 | 52.26 | 61.69
DCLIP [24] | 63.00 | 49.70 | 61.46 | 62.51 | 91.68 | 43.38 | 85.23 | 67.21 | 53.25 | 64.16
WaffleCLIP [31] | 62.83 | 49.69 | 60.88 | 63.65 | 89.29 | 43.97 | 85.61 | 66.58 | 53.47 | 64.00
CuPL [28] | 64.02 | 48.05 | 63.26 | 64.74 | 91.72 | 46.04 | 85.29 | 65.25 | 53.21 | 64.62
Ours | 64.53 | 56.28 | 67.01 | 66.22 | 92.70 | 51.42 | 86.33 | 72.19 | 56.13 | 68.09
$\Delta$ CLIP | $+2.73$ | $+19.45$ | $+6.00$ | $+4.71$ | $+1.46$ | $+8.69$ | $+2.05$ | $+8.60$ | $+3.87$ | $+6.40$
$\Delta$ SOTA | $+0.51$ | $+6.58$ | $+3.75$ | $+1.48$ | $+0.98$ | $+5.38$ | $+0.72$ | $+4.98$ | $+2.66$ | $+3.47$

Table 1: Performance comparison with vanilla CLIP [29] and three state-of-the-
art methods on nine image classification datasets. All data in the table
represent top-1 accuracy (%) on the test set, where bold figures indicate the
highest accuracy (our method achieves the highest in all cases), and
underlined figures indicate the second-highest accuracy of the current
methods. $\Delta$ CLIP and $\Delta$ SOTA denote the absolute improvements of
our method compared to the vanilla CLIP and the best performance of LLM-based
methods, respectively.

###  4.1 Experimental Setup

Implementation Details. In our method, there are four main hyperparameters,
including the number of iterations $N$, the number of descriptors at
initialization $n_{0}$, the number of descriptors to change in mutation and
crossover $n_{i}$, and the number of mutated candidates in each iteration $K$.
Higher values of these hyperparameters are more conducive to finding better
solutions, but they also increase the convergence time and computational cost.
We set $N=10,n_{0}=30,n_{i}=15,K=4$ for all datasets. As for the number of
groups in K-means clustering, we set it to
$\texttt{round}(\frac{|\mathcal{C}|}{10})$ to make sure there are roughly $10$
classes for each group. Unless specified, we adopt GPT-4 [27] (with
temperature fixed at 1.0) to construct our LLM agent and CLIP ViT-B/32
backbone [29] to extract the image and text embeddings.

Datasets. Our experiments leverages the dataset partitioning introduced by
CoOp [9] on 9 different image classification benchmarks, including: ImageNet
[6], EuroSAT [10], UCF101 [32], Scene UNderstanding (SUN) [38], Caltech [8],
Describable Textures Dataset (DTD) [4], CIFAR-10 [15], Flowers102 [25], and
CUB [34].

Compared Methods. We compare our work with vanilla CLIP and three state-of-
the-art methods using LLM to augment class descriptions. To ensure a fair
comparison, we re-implement these methods following their setting and conduct
evaluations on the same test set. CLIP [29] sets a simple template as "A photo
of a {class name}" as input prompt. DCLIP [24] improves the classification
performance of CLIP using LLM-generated class descriptors with a few in-
context examples. They build the prompt in the format of "{classname}, which
(is/has/etc) {descriptor}". WaffleCLIP [31] further improves DCLIP by
introducing high-level concepts at the beginning of the prompt and replacing
class descriptors with random characters. The extended prompt is "A photo of a
{concept}: a {classname}, which (is/has/etc) {random_sequence}". CuPL [28]
designs hand-crafted prompts for LLMs to generate descriptive sentences from
various aspects.

Figure 3: Ablation on iterative optimizations. X-axis: iteration rounds, where
iteration $0$ denotes initialization, Y-axis: Accuracy (%). Red stars
represent the accuracy of vanilla CLIP. As iteration grows, the accuracy
consistently increases across different datasets.

###  4.2 Main Result

As shown in Table 1, our method consistently outperforms both vanilla CLIP and
existing LLM-based methods by a large margin across all examined datasets.
Notably, our method achieves an average increase of over $6\%$ compared to
CLIP, while other LLM-based methods show average improvements of only about
$2\%$. More specifically, our method excels in relatively abstract, visually
challenging datasets, such as the texture classification dataset, DTD, and the
satellite image classification dataset, EuroSAT, with $8.69\%$ and $19.45\%$
absolute improvements, respectively. For datasets with finer category
granularity, like Flowers102, CUB, UCF101, and SUN, our method also
outperforms CLIP by over $3.5\%$, with a remarkable $8.6\%$ increase on the
Flowers102 dataset. However, other LLM-based methods’ best performances on
these datasets are limited. This highlights our method’s superiority in
finding the global optimal descriptions of various datasets, where multi-round
iterations and visual feedback are essential in distinguishing closely related
and confusing categories. This is not achievable in single-category
optimization methods like DCLIP, WaffleCLIP, and CuPL. Without abundant visual
knowledge in pretrained VLMs, these methods generate ambiguous and inaccurate
descriptions in a single run, providing no discriminative information,  _e.g._
, “black bill” for most classes of birds in CUB. On datasets like CIFAR-10,
Caltech, and ImageNet, the margins between our approach and other LLM-based
baselines are relatively smaller. We attribute this to the fact that these
datasets already include distinct classes, which are easier for LLMs to
recognize and understand through class names.

| ImageNet | EuroSAT | Caltech | Flowers102
---|---|---|---|---
CLIP Architecture | Ours | CLIP | $\Delta$ | Ours | CLIP | $\Delta$ | Ours | CLIP | $\Delta$ | Ours | CLIP | $\Delta$
ViT-B/32 | 64.53 | 61.80 | +2.73 | 56.28 | 36.83 | +19.45 | 92.70 | 91.24 | +1.46 | 72.19 | 63.59 | +8.60
RN101 | 63.36 | 60.65 | +2.71 | 36.30 | 32.02 | +4.28 | 91.81 | 89.49 | +2.32 | 70.85 | 61.39 | +9.46
ViT-B/16 | 69.51 | 66.63 | +2.88 | 52.09 | 42.96 | +9.13 | 94.48 | 92.54 | +1.94 | 75.48 | 66.46 | +9.02
ViT-L/14 | 76.11 | 72.85 | +3.26 | 67.40 | 52.86 | +14.54 | 96.80 | 94.04 | +2.76 | 81.73 | 75.96 | +5.77

Table 2: Ablation on transferring optimized descriptors to different model
architectures. We adopt ViT-B/32 as the original backbone model and transfer
the optimized class descriptions to different architectures, including RN101,
ViT-B/16, and ViT-L/14.

###  4.3 Ablation Study

Iterative Optimization. First, we discuss the impact of iterative
optimization. Figure 3 shows that as the number of iterations increases, both
fitness score and test accuracy exhibit an initial growth followed by stable
oscillations. After iterations of optimization, the final results
significantly surpass the performance of CLIP baseline and single-turn methods
at iteration $0$. Experimental results indicate that adopting our genetic
algorithm-inspired optimization effectively moves the class descriptors toward
better solutions, advancing the overall classification accuracy. Furthermore,
the trends of the two lines in each graph are closely aligned, showcasing that
there are no “overfitting” issues during optimization. This emphasizes the
potential of our final descriptors as a universal textual representation
across different model backbones, which is proven in the following text.

Figure 4: Ablation on visual feedback. X-axis: iteration rounds, where
iteration $0$ indicates initialization, Y-axis: Accuracy (%). “iCM” indicates
improved confusion matrix, “CM” indicates confusion matrix, “wo Memory”
indicates without memory banks, and “wo Feedback” indicates without all
components related to visual feedback.

Visual Feedback. Next, we study the importance of each component of visual
feedback in our method. As illustrated in Figure 4, we design four baselines
for ablations: 1) Ours with improved confusion matrix (iCM). This is our
standard setting, where we adopt both the improved confusion matrix as visual
feedback and memory banks to enhance descriptor optimization described in
Section 3.2. 2) Ours with confusion matrix (CM). This baseline simply replaces
the improved confusion matrix with the conventional confusion matrix. 3)
Without Memory. We remove the memory bank recording iteration history, but the
improved confusion matrix is kept. 4) Without Feedback. In this setting, we
remove all components related to visual feedback, including both the confusion
matrix and memory banks.

In Figure 4, solid lines represent the complete setup, while dashed lines
indicate scenarios where components of visual feedback are omitted. Comparing
the two solid lines, the improved confusion matrix demonstrates overall
superior performance to the standard confusion matrix. The conventional
confusion matrix not only contains excessive redundant information, which may
hamper the understanding of LLMs, but also discards certain critical
information to discriminate related classes since it relies solely on the
top-1 accuracy for construction. Our improved version effectively addresses
these issues. Examining the dashed line for “without memory” alongside the two
solid lines showcases that removing the memory bank leads to significant
fluctuations in accuracy. Although occasionally reaching relatively high
accuracy, such a model contains high variances but lacks robustness. Hence,
the memory bank significantly enhances the stability and robustness during the
optimization process. Regarding the setting without feedback, which is the
removal of both the confusion matrix and memory banks, we notice a minimal
increase or even drop in accuracy, reaffirming the crucial role of visual
feedback.

Transferability. To verify the generalization and transferability of our
method, we tested the optimized descriptions obtained using CLIP ViT-B/32 on
different backbones, including RN101, ViT-B/16, and ViT-L/14 across the
ImageNet, EuroSAT, Caltech, and Flowers102 datasets. Our results, presented in
Table 2, demonstrate that our optimized class descriptions consistently
surpass the baseline CLIP model in terms of accuracy. A notable observation is
the significant performance enhancement on the Flowers102 dataset, with
increases exceeding $9\%$ when transferred to RN101 and ViT-B/16. These
results emphasize our method generates more generalizable natural language
prompts, avoiding overfitting to a specific architecture.

Figure 5: Examples on interpretability generated by our method and CuPL [28].
We select two categories from EuroSAT and Flowers102 and list the top-$3$ and
last-$3$ descriptions for each category, ranked by their similarity scores
averaging across class test samples.

###  4.4 Interpretability and Analysis

Figure 5 visualizes LLM-generated descriptions that contribute the most and
least for classifying images of different classes. Specifically, we select our
results at the first and last iteration, as well as results from CuPL [28] for
visualization, with class Residential Building in EuroSAT and Prince of Wales
Feathers in Flowers102. To compare the effectiveness of descriptions, we
compute the similarity score for each description using the average of all
test images in that category. After sorting them in descending order, we
visualize the top-$3$ and last-$3$ descriptions and their similarity scores in
each setting.

Due to the high degree of diversity in CuPL, the quality of generated
descriptions greatly varies, and it is prone to generate vague and overlapping
descriptions,  _e.g._ , “structures where people live” and “building where
people live”. In contrast, our method generates more concise and relevant
descriptions at initialization and continues to refine them during
optimization, showing consistent growth in similarity scores of all
descriptions. Our final descriptions reveal strong class-dependant semantic
information which is beneficial for visual classification. In addition, for
the confusing class name “Prince of Wales Feathers” in Flowers102, CuPL
mistaken it as the heraldic badge of the Prince of Wales instead of an annual
herb, generating completely irrelevant descriptions. We further find this
issue exists in all LLM-based methods including DCLIP [24] and WaffleCLIP
[31]. Instead, our method successfully generates correct descriptions since we
provide the information of related classes and leverage feedback from CLIP for
visual grounding, emphasizing the importance of visual feedback.

##  5 Conclusion and Limitations

In this work, we present a novel approach for image classification that
leverages LLMs to iteratively refine class descriptors with visual feedback
from VLMs to guide the optimization process. The incorporation of a genetic
algorithm-inspired framework ensures efficient exploration of the complex
solution space, while visual feedback provides a robust mechanism for aligning
textual descriptions with visual features essential for classification. We
validate the effectiveness of our method across 9 benchmark datasets,
showcasing superior performance with multiple benefits including
interpretability and transferability. In the current scope, our focus has been
primarily confined to the realm of image classification. However, extending
this methodology to tackle more intricate visual tasks, _e.g._ , semantic
segmentation and object detection, warrants the development of sophisticated
visual feedback systems and the formulation of more advanced strategies for
iterative updates.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _NeurPS_ , 35, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In _NeurIPS_ , 2020.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Cimpoi et al. [2014] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In _CVPR_ , 2014.
  * Cui et al. [2023] Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao.  Holistic analysis of hallucination in gpt-4v(ision): Bias and interference challenges, 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  _CVPR Workshop_ , 2004.
  * Gao et al. [2021] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_ , 2021.
  * Helber et al. [2019] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.
  * Huang et al. [2023] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv:2302.14045_ , 2023.
  * Huang et al. [2022] Tony Huang, Jack Chu, and Fangyun Wei.  Unsupervised prompt learning for vision-language models.  _arXiv preprint arXiv:2204.03649_ , 2022.
  * Imani et al. [2023] Shima Imani, Liang Du, and Harsh Shrivastava.  Mathprompter: Mathematical reasoning using large language models.  _arXiv preprint arXiv:2303.05398_ , 2023.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _ECCV_ , 2022.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , 2023.
  * Liang et al. [2023] Youwei Liang, Ruiyi Zhang, Li Zhang, and Pengtao Xie.  Drugchat: towards enabling chatgpt-like capabilities on drug molecule graphs.  _arXiv preprint arXiv:2309.03907_ , 2023.
  * Liao et al. [2022] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu.  Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.  In _CVPR_ , 2022.
  * Liu et al. [2023a] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Aligning large multi-modal model with robust instruction tuning.  _arXiv preprint arXiv:2306.14565_ , 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.
  * Liu et al. [2023c] Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao.  Chatgpt-powered conversational drug editing using retrieval and domain feedback.  _arXiv preprint arXiv:2305.18090_ , 2023c.
  * Longpre et al. [2023] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_ , 2023.
  * Ma et al. [2023] Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Eureka: Human-level reward design via coding large language models.  _arXiv preprint arXiv: Arxiv-2310.12931_ , 2023.
  * Menon and Vondrick [2023] Sachit Menon and Carl Vondrick.  Visual classification via description from large language models.  In _ICLR_ , 2023.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _Indian conference on computer vision, graphics & image processing_, 2008.
  * Novack et al. [2023] Zachary Novack, Saurabh Garg, Julian McAuley, and Zachary Lipton.  Chils: Zero-shot image classification with hierarchical label sets.  In _ICML_ , 2023.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Pratt et al. [2023] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi.  What does a platypus look like? generating customized prompts for zero-shot image classification.  In _ICCV_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 2020.
  * Roth et al. [2023] Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata.  Waffling around for performance: Visual classification with random words and broad concepts, 2023.
  * Soomro et al. [2012] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wah et al. [2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. [2022] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  _TMLR_ , 2022.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  _NeurIPS_ , 2022.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023.
  * Xiao et al. [2016] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.  Sun database: Exploring a large collection of scene categories.  _IJCV_ , 2016.
  * Yan et al. [2023] An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley.  Learning concise and descriptive attributes for visual recognition.  In _ICCV_ , 2023.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen.  Woodpecker: Hallucination correction for multimodal large language models.  _arXiv preprint arXiv:2310.16045_ , 2023.
  * Yu et al. [2023] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al.  Language to rewards for robotic skill synthesis.  _arXiv preprint arXiv:2306.08647_ , 2023.
  * Zhang et al. [2022] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  In _ECCV_ , 2022.
  * Zhao et al. [2023] Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang.  Graphtext: Graph reasoning in text space.  _arXiv preprint arXiv:2310.01089_ , 2023.
  * Zhou et al. [2022] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _CVPR_ , 2022.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

Supplementary Material

##  A Full Prompts

In this section, we provide full prompts for our proposed method of iterative
optimization with visual feedback. There are three components in this
framework: initialization, mutation, and crossover. Each component has a
system prompt and a user prompt.

Listing 1: Initialization system prompt.

⬇

You are a prompt engineer trying to optimize the text d-

escription of class labels for image classification. Th-

e CLIP model performs zero-shot image classification by

computing the cosine similarities between input images

and class labels. You are given original class labels a-

nd some hints of confusion classes that are indistingui-

shable to CLIP for each class. Your goal is to generate

a list of visual concepts to improve the description of

current class labels to maximize their distinctions for

CLIP to better recognize.

The output class names should be a Python list of strin-

gs in the following format:

“‘

[”original_class_name1: concept1, concept2, concept3,

…”, ”original_class_name2: concept1, concept2, concep-

t3, …”, …]

“‘

Some helpful tips for optimizing the class descriptions:

1. You should generate {n_concepts_init} different

high-level concept words for each class to emphasiz-

e the distinct visual features of this class but not

appear in other classes, split them with ”,”.

2. DO NOT give me non-visual words!

3. The class description must also be general enough

to cover most training images of this class.

4. Most importantly, always focus on the visual fea-

tures of all classes, since there are only images i-

nput. Do not produce text describing invisible feat-

ures, e.g., voice, mental character, etc.

5. Do not include any other class names in the desc-

ription of one class, which means do not use text l-

ike ”not like a cat” in dogs or ”distinct from birds

”in plans. Only focus on the features of the class

itself.

6. The concept words for each class should be diver-

seand not too similar to each other.

Listing 2: Initialization user prompt.

⬇

The original class names are:

{current_class}

The confusion classes for CLIP are:

{visual_feedback}

Note, classes in the front are more prone to be confused

than the ones in the back.

Write the optimized class names leveraging the hints of

confusion classes list above. Directly generate your an-

swer in the given format without any extra information.

Listing 3: Mutation system prompt.

⬇

You are a prompt engineer trying to optimize the text d-

escription of class labels for image classification. Th-

e CLIP model performs zero-shot image classification by

computing the cosine similarities between input images

and class labels. You are given current class labels, s-

ome hints of confusion classes that are indistinguishab-

le to CLIP and lists of good and bad history words for

each class. Your goal is to optimize the description of

current class labels to maximize their distinctions for

CLIP to better recognize images.

The output class names should be a Python list of strin-

gs in the following format:

“‘

[”original_class_name1: concept1, concept2, concept3,

…”, ”original_class_name2: concept1, concept2, concep-

t3, …”, …]

“‘

Some helpful tips for optimizing the class descriptions:

1. Base on all the provided information, you should

identify {n_concepts} bad high-level concept words

in each class and replace them with better ones to

emphasize the distinct visual feature of this class

but not appear in others’, split them with ”,”.

2. DO NOT give me non-visual words!

3. The class description must also be general enough

to cover most training images of this class.

4. Most importantly, always focus on the visual fea-

tures of all classes, since there are only images i-

nput. Do not produce text describing invisible feat-

ures, e.g., voice, mental character, etc.

5. Do not include any other class names in the desc-

ription of one class, which means do not use text l-

ike ”not like a cat” in dogs or ”distinct from bird-

s” in plans. Only focus on the features of the clas-

s itself.

6. The concept words for each class should be diver-

se and not too similar to each other.

Listing 4: Mutation user prompt.

⬇

The current class names are:

{current_class}

The confusion classes for CLIP are:

{visual_feedback}

Note, classes in the front are more prone to be confused

than the ones in the back.

History list class descriptions that will improve class-

ification accuracy are:

{positive_list}

History list records for class descriptions that will d-

ecrease classification accuracy are:

{negative_list}

Write the optimized class names leveraging the hints of

confusion classes and two lists above. Directly generat-

e your answer in the given format without any extra inf-

ormation.

Listing 5: Crossover system prompt.

⬇

You are a prompt engineer trying to optimize the text d-

escription of class labels for image classification. Th-

e CLIP model performs zero-shot image classification by

computing the cosine similarities between input images

and class labels. You are given {n_samples} versions of

class labels with different text descriptions, and clas-

sification metrics for each of them. Your task is to fi-

nd the best combination of given text description for e-

ach class label based on the classification metrics to

achieve the best overall classification performance.

The output class names should be a Python list of strin-

gs in the following format:

“‘

[”original_class_name1: concept1, concept2, concept3,

…”, ”original_class_name2: concept1, concept2, concep-

t3, …”, …]

“‘

Some helpful tips for optimize the class descriptions:

1. You should only select from the existing concept

words in class descriptions. You cannot create new

words.

2. For each class, you should combine various conce-

pts in different versions of descriptions, and gene-

rate the description that is most conducive to dist-

inguishing this class.

3. You should leverage the classification metrics f-

or your selection. The overall accuracy indicates t-

heglobal performance, while the class-wise accuracy

indicates the performance for each class.

4. Keep the number of concepts in each class descri-

ption unchanged, i.e., {n_concepts_init} concepts f-

oreach class.

Listing 6: Crossover user prompt.

⬇

The {n_samples} versions of class descriptions and corr-

esponding classification metrics are:

{class_samples}

Output the optimized class descriptions according to th-

e classification metrics above. Directly generate your

answer in given format without any extra information.

##  B Result Example

In this section, we showcase some examples generated by our method. We select
a subset of class labels from two datasets for demonstration. For the complete
set of labels, please refer to the .txt file in our Supplementary Material.

Listing 7: Examples of generated descriptions for Flowers102.

⬇

{

”tiger lily”: [

”erect perennial growth”,

”pollen-covered anthers”,

”intense orange hue”,

”lance-shaped leaf structure”,

”vibrant orange-red coloring”,

”summer garden fixture”,

”bold tiger-striped petals”,

”straight growth form”,

”midsummer peak”,

”lanceolate leaf shape”

],

”giant white arum lily”: [

”robust flora”,

”monochromatic palette”,

”tropically adapted”,

”arrow-shaped leaves”,

”swamp inhabitant”,

”stately appearance”,

”massive spoon-like spathe”,

”lush marsh foliage”,

”imposing white florescence”,

”prominent central spadix”,

”large ovate bracts”,

”waxy texture”,

”South African native”,

”water-adjacent growth”

],

”fire lily”: [

”south african endemic”,

”scorching colors”,

”wild habitat specialist”,

”volcanic color palette”,

”excessive pollen attractor”,

”specific African habitat”,

”nectar-abundant flower”,

”curved petal silhouettes”

],

”orange dahlia”: [

”horticultural pride”,

”summer crescendo”,

”elegant growth habit”,

”fiery orange inflorescence”,

”lush verdant foliage”,

”decorative cutting flower”,

”opposite leaf arrangement”,

”compacted petal rows”,

”bold spherical blooms”,

”mid-summer grandeur”,

”fiery”,

”ornamental”,

”foliage”

],

”pink-yellow dahlia”: [

”cut flower favorite”,

”standout horticultural beauty”,

”large petal-packed inflorescence”,

”bedding plant”,

”luminous garden feature”,

”autumn flowering”,

”ornamental garden treasure”,

”delicate pink-yellow gradient petals”,

”soft petal curvature”,

”opulent look”,

”gradient”,

”autumn”,

”opulent”

],

”cautleya spicata”: [

”woodland understory preference”,

”shade-loving”,

”soft yellow tones”,

”Himalayan forest dweller”,

”orchid resemblance”,

”clusters of golden flowers”,

”semi-evergreen”,

”rhizomes rooting”,

”delicate mountainous ginger”,

”forest dweller”

],

”japanese anemone”: [

”tall asiatic perennial”,

”serene woodland flower”,

”east-asian native flower”,

”stoloniferous growth”,

”silky pink petal radiance”,

”woodland garden favorite”,

”east-asian perennial blossom”,

”late-season floral display”,

”stately forest edge flower”,

”tranquil late bloomer”,

”carpel rich center”,

”long-flowering autumnal star”,

”tall asiatic ornamental beauty”,

”delicate purplish-pink blossom array”,

”windflower delicate beauty”

],

”black-eyed susan”: [

”rugged hairy rudbeckia”,

”coneflower-like black-eyed plant”,

”central black cone flower”,

”pollinator-friendly rudbeckia”,

”golden-yellow wildflower”,

”distinct black-eyed blossom”,

”sunny field susan”,

”daisy family rudbeckia”,

”showpiece planting”,

”fuzzy-stemmed rudbeckia”,

”brown-domed center”,

”golden petal surround”,

”easy-growing rudbeckia species”,

”sunny open site favorite”,

”bright American wildflower”

]

…

}

Listing 8: Examples of generated descriptions for SUN.

⬇

{

”abbey”: [

”religious architecture”,

”peaceful prayer courtyard”,

”historic ecclesiastical complex”,

”monastic buildings”,

”carved stone reliefs”,

”silent contemplative gardens”,

”clerical chambers”,

”ancient liturgical hall”,

”sequestered religious refuge”,

”spiritual monastic center”,

”high-altitude abbey setting”,

”medieval landmark”,

”picturesque monastery”,

”tranquil cloister enclosure”,

”secluded pilgrimage destination”,

”ancestral cloister”,

”spirituality center”,

”parochial complex”,

”ecclesiastical estate”,

”tranquil abbey gardens”,

”monastic cellar vaults”,

”placid cloister quarters”,

”benedictine architecture”,

”clerical stonework”,

”Gregorian chant resonance”,

”secluded spiritual refuge”,

”illuminated manuscript repository”,

”divine service chapel”,

”clerestory window designs”,

”ancestral cloister”

],

”airplane cabin”: [

”aeronautical window shape”,

”fuselage cross-section”,

”overhead bin latch”,

”cabin window shade”,

”recessed cabin lighting”,

”seatback tray table”,

”emergency exit handle”,

”aircraft cabin”,

”cabin crew interphone”,

”cabin altitude sign”,

”air nozzle”,

”avionics panel”,

”in-flight magazine pouch”,

”fuselage interior design”,

”cabin class divider”

],

”airport terminal”: [

”terminal retail stores”,

”digital flight board”,

”entrance to arrivals”,

”airport lounge”,

”tax-free goods shop”,

”automated check-in spot”,

”security screening zone”,

”airport seating arrangements”,

”passenger check-in desks”,

”departure lounge seating”,

”airport check-in island”,

”gate podiums”,

”flight information boards”

],

”engine room”: [

”turbine vibration control zone”,

”mechanical performance monitoring space”,

”heavy machinery operational platform”,

”maritime technical engineering station”,

”powertrain thermal regulation sector”,

”central power distribution chamber”,

”diesel generator maintenance location”,

”kinetic propulsion system area”,

”ship engine operation center”,

”marine engine service station”,

”mechanical room vibration analysis”,

”power distribution control centre”,

”engine diagnostic and repair shop”,

”power regeneration equipment area”

],

”indoor escalator”: [

”dynamic stairway”,

”motorized belt”,

”visible step cleats”,

”longitudinal motion path”,

”smooth balustrade glass”,

”public indoor traversal”,

”automated staircase”,

”shopping center feature”,

”kinetic stair mechanism”,

”dynamic railing”,

”continuous movement”,

”moving handrail”,

”glass side panels”,

”flat escalator landing”

],

”excavation”: [

”archaeological dig layers”,

”artefact excavation pits”,

”ancient civilization studies”,

”soil strata analysis”,

”dig site grid system”,

”stratigraphy profiles”,

”excavation site documentation”,

”earth sifting screens”,

”fossil dig uncovering”,

”sedimentary layer analysis”,

”archeological site”,

”dig zone”,

”soil screening”,

”historical excavation”,

”artifact discovery”,

”stratigraphy”,

”archeological survey”,

”relic preservation”

],

”fairway”: [

”expansive fairway views”,

”tree-obstacle layout”,

”picturesque water features”,

”distinctive golf hole design”,

”strategic golf hole locations”,

”bunker-guarded green”,

”rough grass edges”,

”undulating golfer’s terrain”,

”serpentine cart paths”,

”slope-contoured play lane”

]

…

}

Figure 6: Iterative Optimization Visualization. From left to right, the
sequence is as follows: the 0th round of our method, the 4th round of our
method, the 9th round of our method, and the CuPL [28] method. The darker the
color in the heatmap, the higher the corresponding value in the confusion
matrix.

##  C Iterative Optimization Visualization

To further demonstrate the superiority of iterative optimization, in this
section, we present the confusion matrix as heatmaps, offering a more
intuitive visualization of the improvements brought about by the optimization
process and a comparison with a previous state-of-the-art method. As shown in
Figure 6, the three images on the left respectively represent the initial,
mid-stage, and final-round confusion matrix heatmaps of our method, while the
image on the far right is the heatmap of the confusion matrix generated using
the CuPL [28] method for description generation. In our approach, as seen in
the three randomly selected areas, the color intensity in the non-diagonal
regions lightens as the iterations increase. This indicates a reduction in the
number of categories confused with the diagonal class over time, further
underscoring the effectiveness of iterative optimization. Comparing our method
with CuPL, we notice many rows in the heatmap are much darker, indicating a
severe blurring of class distinction along the diagonals. This also reveals
that the strategy of using LLMs to generate detailed categories in previous
methods has limited impacts on optimizing classification effectiveness.