# Towards Realistic Zero-Shot Classification via Self Structural Semantic
Alignment

Sheng Zhang1 Muzammal Naseer1 Guangyi Chen1,2 Zhiqiang Shen1
Salman Khan1,3 Kun Zhang1,2 Fahad Khan1,4

###### Abstract

Large-scale pre-trained Vision Language Models (VLMs) have proven effective
for zero-shot classification. Despite the success, most traditional VLMs-based
methods are restricted by the assumption of partial source supervision or
ideal vocabularies, which rarely satisfy the open-world scenario. In this
paper, we aim at a more challenging setting, Realistic Zero-Shot
Classification, which assumes no annotation but instead a broad vocabulary. To
address this challenge, we propose the Self Structural Semantic Alignment
(S3A) framework, which extracts the structural semantic information from
unlabeled data while simultaneously self-learning. Our S3A framework adopts a
unique Cluster-Vote-Prompt-Realign (CVPR) algorithm, which iteratively groups
unlabeled data to derive structural semantics for pseudo-supervision. Our CVPR
process includes iterative clustering on images, voting within each cluster to
identify initial class candidates from the vocabulary, generating
discriminative prompts with large language models to discern confusing
candidates, and realigning images and the vocabulary as structural semantic
alignment. Finally, we propose to self-learn the CLIP image encoder with both
individual and structural semantic alignment through a teacher-student
learning strategy. Our comprehensive experiments across various generic and
fine-grained benchmarks demonstrate that the S3A method offers substantial
improvements over existing VLMs-based approaches, achieving a more than $15\%$
accuracy improvement over CLIP on average. Our codes, models, and prompts will
be publicly released.

## Introduction

In recent years, large-scale pre-trained Vision Language Models (VLMs) such as
CLIP (Radford et al. 2021; Ren et al. 2022), ALIGN (Li et al. 2021), and BLIP
(Li et al. 2022, 2023) have garnered significant attention for their
remarkable zero-shot generalization ability on multifarious downstream tasks,
particularly in recognizing unseen categories (Zhang et al. 2023). The common
practice to leverage this ability is packing category names into a textual
prompt (e.g., “A photo of a [CLS]”) and aligning image embeddings with text
embeddings of filled prompts in VLM joint embedding space for classification.
To adapt pre-trained VLMs to downstream unseen data, existing prevailing
methods (Wu et al. 2023; Zang et al. 2022; Ghiasi et al. 2021) usually assume
the access to source labeled data (e.g., in zero-shot learning (Zhou et al.
2022a; Gao et al. 2021)), target label distribution (e.g., in unsupervised
prompt tuning (Kahana, Cohen, and Hoshen 2022)), or an ideal vocabulary that
exactly matches the ground-truth label set or with very few open words (e.g.,
in open-vocabulary learning (Wu et al. 2023; Zang et al. 2022; Ghiasi et al.
2021)). However, this ideal vocabulary is unattainable without exhaustive
annotation of all unseen data; whereas, human annotations are exorbitant and
difficult to scale. Therefore, both assumptions are restrictive and
impractical in open-world scenarios with diverse and dynamic nature.

In this paper, we embark on a journey towards Realistic Zero-Shot
Classification (RZSC), a more challenging yet practical problem compared with
conventional zero-shot learning due to its realistic conditions. Here, we term
Realistic as the realistic nature of RZSC which aims to recognize categories
on unseen datasets without annotation and ideal vocabulary, but with a vast,
comprehensive vocabulary with more than 20K category names encompassing all
common classes (Sariyildiz et al. 2021; Ridnik et al. 2021). However, it is
challenging since the vast vocabulary can lead to alignment confusion among
fine-grained options; as we witness the consistent and dramatic CLIP (Radford
et al. 2021) performance drops and reduced neighborhood ranges in Fig. 1.

To confront this challenge, we introduce the Self Structural Semantic
Alignment (S3A) framework, which iteratively discovers structural semantic
alignment from unlabeled data for joint self-learning. This is orchestrated
through our unique Cluster-Vote-Prompt-Realign (CVPR) algorithm, a principled
process comprising four key steps: (1) Clustering unearths inherent grouping
structures within image embeddings, producing meaningful image semantics. (2)
Voting associates each cluster with initial category candidates, representing
potential pseudo-alignments. These two steps can be executed iteratively to
obtain more reliable candidates. (3) Prompting leverages the power of large
language models (LLMs) to discern nuanced candidates by augmenting prompts
with discriminative attributes. (4) Re-alignment represents calibrating the
cluster-vocabulary alignment with LLM-augmented prompts as pseudo structural
semantic alignment labels. Incorporating our CVPR algorithm, our S3A framework
self-trains a student model based on extracted individual and structural
semantic alignment labels from a stable teacher. Simultaneously, the teacher
is updated by student weights to produce more reliable pseudo semantic
alignments.

We extensively evaluate our S3A framework across multiple setups, spanning a
variety of generic and fine-grained benchmarks. The results show that S3A not
only consistently outperforms previous adapted State-of-The-Arts (SOTAs) under
the RZSC setting on all benchmarks, but excels in out-of-vocabulary
evaluation, where category names can fall outside the S3A vocabulary. It is
worth noting the our S3A remarkably enhance naive CLIP by $>10\%$ top-1
classification accuracy on average. Comprehensive evaluations evidence our S3A
framework paves the way to this realistic challenging problem.

Our contributions include: (1) We propose a Self Structural Semantic Alignment
(S3A) framework, to address the challenging Realistic Zero-Shot Classification
problem, which jointly extracts and self-learns on the individual and
structural semantic alignment. (2) We propose a Cluster-Vote-Prompt-Realign
algorithm to reliably derive reliable structural semantic alignments between
images and the large vocabulary. (3) S3A achieves SOTA performance on various
generic and fine-grained benchmarks, remarkably boosting CLIP by over $10\%$
accuracy, and even in the out-of-vocabulary scenarios.

## Related Work

Setting |  Vocab. | Annotation | Train
---|---|---|---
Zero-Shot Transfer | G.T. | ✗ | $\mathcal{Y}_{tgt}$
Zero-Shot Classification | G.T. | ✓ | $\mathcal{Y}_{base}$
Open-Vocabulary Learning |  $<$2K | ✓ | $\mathcal{Y}_{base}$
Unsupervised Fine-tuning | G.T. | ✗ | $\mathcal{Y}_{tgt}$
RZSC |  $>$20K | ✗ | $\mathcal{Y}_{tgt}$
Table 1: Our realistic zero-shot classification and other related settings.
Here, following (Wu et al. 2023), we denote $\mathcal{Y}_{base}$ and
$\mathcal{Y}_{tgt}$ as the sets of base training classes and target testing
classes, which satisfies $\mathcal{Y}_{base}\bigcap\mathcal{Y}_{tgt}=\phi$.
The shared learning goal of all settings is to recognize $\mathcal{Y}_{tgt}$
in test data. Ground-truth is abbreviated as G.T.

Zero-Shot Learning/Open-Vocabulary Learning with VLMs. Traditional
(Generalized) Zero-Shot Classification (ZSC) aims to categorize novel classes
in unseen test data with training on annotated base/seen classes with or
without unlabeled target novel classes (Wang et al. 2019; Pourpanah et al.
2022). However, they usually assume auxiliary semantic information of both
seen and unseen classes, e.g., category attributes (Lampert, Nickisch, and
Harmeling 2009), knowledge graph (Akata et al. 2015), textual keywords (Lei Ba
et al. 2015; Cappallo, Mensink, and Snoek 2016). Recently, large-scale pre-
trained VLMs have been introduced to alleviate these assumptions (Jia et al.
2021; Radford et al. 2021; Zhang et al. 2023). Furthermore, Open-Vocabulary
Learning (OVL) (Wu et al. 2023; Zhou et al. 2023; Zhou, Loy, and Dai 2022;
Karazija et al. 2023) are presented, which train the models on some annotated
data (base classes (Xu et al. 2023) or large-scale image-text pairs (Shin,
Albanie, and Xie 2023)) and test them on open novel classes. Our RZSC setting
differs from conventional ZSC and OVL in not requiring any labeled training
data, and not assuming an ideal vocabulary with a ground-truth target label
set or with few open words (Wu et al. 2023; Xu et al. 2023; Karazija et al.
2023).

Figure 1: (a) Performance comparison between CLIP w/ an ideal vocabulary
(Green) and w/ a large vocabulary of 20K categories (Pink). (b) Distribution
plot of text-to-text average 3-Nearest Neighbors cosine similarity of each
text embedding for three types of vocabulary: with ImageNet-100, ImageNet-1K,
and 20K category names.

Zero-Shot Transfer/Unsupervised Fine-tuning of VLMs. Both Zero-Shot Transfer
(ZST) and Unsupervised Fine-tuning (UF) assume no annotation of target
datasets. ZST (Radford et al. 2021; Ren et al. 2022) directly uses the pre-
trained VLMs for zero-shot prediction without updating models, e.g., SCD (Han
et al. 2023) iteratively refines CLIP inference results with a heuristic
algorithm with a WordNet vocabulary (Miller 1995). More related to us, UF
further transductively adapts pre-trained models with task-specific training,
e.g. adapting VLMs with self-training or prompt tuning (Li, Savarese, and Hoi
2022; Kahana, Cohen, and Hoshen 2022; Shin, Albanie, and Xie 2023). However,
existing methods assume the known ground-truth target label set or label
distribution (Kahana, Cohen, and Hoshen 2022; Li, Savarese, and Hoi 2022). In
this paper, we aim to alleviate the dependence on such assumptions and propose
a new setting, RZSC.

Discussion on Zero-Shot Settings. Here, we summarize the main differences
between our RZSC setting and others in Table 1. Previous related settings
adopt restrictive assumptions including an ideal vocabulary, the target label
distribution, and labeled known classes. Unlike related settings with
restrictive assumptions such as an ideal vocabulary, the target label
distribution, and known labeled classes, our RZSC seeks to learn to categorize
an unlabeled dataset with a huge vocabulary with over 20K classes. An expanded
huge vocabulary presents significant challenges for RZSC problem, as evidenced
by the consistent and dramatic CLIP performance drop (in Fig. 1 a) on all
datasets when the vocabulary scales up. The primary challenge arises from
increased open words, complicating fine-grained category discrimination for
pre-trained VLMs. As displayed in Fig. 1 b, the averaged cosine similarity
between a query text embedding and its 3-nearest text neighbors grows with the
vocabulary size. We include more related works in our appendix B.

Figure 2: Illustration of our Self Structural Semantic Alignment (S3A)
framework, which fine-tunes pre-trained CLIP encoder with a teacher-student
architecture. The teacher is updated by the student with the exponentially
moving average technique. The student is guided by on-the-fly one-hot instance
alignment predicted by the teacher; meanwhile, it self-learns the structural
semantic alignment labels generated by our per-epoch CVPR algorithm on all
teacher image embeddings.

## Methodology

### Problem: Realistic Zero-Shot Classification

Existing methods that adapt pretrained VLMs to unseen data usually hinge on
specific knowledge of target datasets, such as prior distributions or an ideal
vocabulary. These conditions are often challenging to fulfill in real-world
environments. In this paper, we explore a more practical task, Realistic Zero-
Shot Classification, abbreviated as RZSC.

RZSC is formally defined as follows: Let us consider an unlabeled dataset
$\mathcal{D}_{u}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N}\subset\mathcal{X}\times\mathcal{Y}$
with $N$ instances defined on category set $\mathcal{Y}$, and a pretrained VLM
such as CLIP, equipped with image and text encoders $f_{I}$ and $f_{T}$,
respectively. Then, we assume no information of $\mathcal{Y}$ and instead with
a comprehensive vocabulary that contains more than 20,000 distinct category
names, i.e., $|\mathcal{Y}|\ll|\mathcal{W}|$. We build our vocabulary from all
visual categories from ImageNet1K (Deng et al. 2009) and ImageNet21K (Ridnik
et al. 2021) datasets since they are annotated with expert taxonomic knowledge
(Miller 1995) and encompasses most commonly-seen visual categories in the real
world. The goal of the RZSC task is to refine the ${f_{I}}$ to predict the
correct category:

$\hat{y_{i}}=\arg\max_{w_{j}\in\mathcal{W}}{\mathbf{z_{i}}\cdot\mathbf{h}_{j}},$
(1)

where $\mathbf{z}_{i}=f_{I}(\mathbf{x}_{i})$ denotes the image embedding while
text embedding $\mathbf{h}_{j}=f_{T}(w_{j})$ are obtained with a text prompt,
e.g., “a photo of [CLS]”, and the category name $w_{j}$. Here, $\cdot$ denotes
the cosine similarity.

### Overview: Self Structural Semantic Alignment

RZSC presents a more formidable challenge than previous tasks, primarily owing
to the absence of label information and an increased vocabulary size. As
illustrated in part (a) of Fig. 1, the performance of CLIP declines sharply as
the vocabulary size increases. This decline can be attributed to the
introduction of open redundant words as hard negative samples, which creates
noise and confuses the CLIP model, hindering its ability to accurately
identify pseudo labels.

It motivates us to propose the Self Structural Semantic Alignment (S3A)
framework, aiming at discovering the structural semantics through interactive
self-alignment between visual images and textual vocabulary. As shown in Fig.
2, S3A deploys a Cluster-Vote-Prompt-Realign (CVPR) algorithm to explore
structural semantics as pseudo labels, and utilizes a self-training procedure
to iteratively refine both models and pseudo supervision. Our CVPR algorithm
and the self-training procedure mutually enhance each other: as training
progresses in adapting representations, the teacher model can provide
increasingly reliable pseudo alignments in subsequent iterations of the CVPR
algorithm. Concurrently, the CVPR algorithm contributes structural semantics
as part of the supervisory signal for self-training. We elaborate on all
components in the sections that follow.

Figure 3: An illustrative toy example for our CVPR algorithm, which comprises
four steps: (1) We clustering all image embeddings; (2) We conduct 1-nearest
neighbor voting on all text prototypes of the large vocabulary for each
cluster. Since the results of the naive assignment in this step are
susceptible to the noise of text embeddings, we generate cluster-wise
candidate categories instead. (3) We augment CLIP text prompts with visual
discriminative descriptions from the large language model to discern nuanced
candidates. (4) with augmented prompts, the cluster-vocabulary alignment is
calibrated and refined.

### Cluster-Vote-Prompt-Realign

The Cluster-Vote-Prompt-Realign (CVPR) algorithm lies at the heart of the S3A
framework, representing an innovative approach to uncovering structural
semantics in data. As illustrated in Fig. 3, the CVPR algorithm consists of
four key stages, each contributing to the alignment and identification of
structural relationships between visual images and textual vocabulary,
including discovering semantic clusters, voting category names on large
vocabulary, prompting LLM to discriminate the nuanced candidates, and refine
the cluster-vocabulary alignment. Each step is explained in detail in the
subsequent paragraphs. Below we delineate these stages and their functions
within the algorithm.

Clustering. Based on existing evidence (Radford et al. 2021) and our
observation, the pre-trained CLIP excels at grouping instances with the same
or similar semantic labels in the image embedding space. We thus produce the
pseudo supervision by semantic clustering and aligning the clusters with
vocabulary. Specifically, given image embeddings $\bf{z}_{i}$ in
$\mathcal{D}_{u}$, we apply KMeans (Arthur and Vassilvitskii 2007) to obtain
the $K$ clusters, $\Gamma=\\{\Gamma_{k}\\}_{k=1}^{K}$, where $\Gamma_{k}$
denotes the $k$-th set of image embeddings.

Voting. Given the semantic cluster results $\Gamma$, we compute a vocabulary
voting distribution matrix $M\in\mathbf{R}^{K\times|\mathcal{W}|}$, where
$M_{k,j}$ represents the normalized frequency of the prototype of category
$w_{j}$ being the nearest neighbor to all instances in the $k$-th cluster.
Specifically, it is computed as

$M_{k,j}=\frac{1}{K|\Gamma_{k}|}\sum_{\mathbf{z}\in\Gamma_{k}}{\mathbb{I}(w_{j}=\arg\max_{w}\mathbf{z}\cdot\mathbf{h})}$
(2)

where $\mathbb{I}$ is an indicator function, and $|\Gamma_{k}|$ denotes the
size of the $k$-th cluster. Rather than naively assigning each cluster the
argmax from the vocabulary, we assign the top-$m$ frequent words to each
cluster as potential candidates which are treated equally. It indicates that
for each row $M_{k}=(M_{k,j})_{j=1}^{|\mathcal{W}|}$, we maintain the highest
$m$ elements as 1 and others as 0.

Nonetheless, the initial clustering and voting may introduce noise, leading to
low-quality pseudo-labels. To mitigate this issue, we iteratively refine the
previous clusters based on the current voting outcomes. In particular, we
utilize the Hungarian matching (Kuhn 1955) for textual embeddings and clusters
to establish a prototype for each cluster. Subsequently, we reassign the image
embeddings, using these prototypes as the updated cluster centers (Han et al.
2023). Additional details are provided in our appendix C.

Prompting. Through our empirical studies, we observed that CLIP’s
representation struggles to differentiate nuanced candidates effectively. This
observation spurred our efforts to refine the embeddings of textual
candidates. We speculate that the challenge in distinguishing fine-grained
options arises from the presence of noisy or imprecise image-caption
alignments during CLIP’s pre-training.

To address this, our approach is to enhance the basic CLIP prompts by
accentuating the subtle semantic differences. We do this by integrating
auxiliary textual knowledge drawn from a large language model, renowned for
its expertise in knowledge retrieval (Dale 2021; Yang et al. 2021).

To achieve this, we feed $m$ candidate category words from the $k$-th cluster
into the LLM prompt template, each accompanied by their specific definition.
Furthermore, to extract nuanced visual attributes from the LLM, we append an
instruction to the main prompt, directing the model to emphasize and describe
these specific attributes. Our prompt template is structured as follows:

Prompt: Given visual concepts: [CLS-1]: [DEF-1], ..., [CLS-m]: [DEF-m].

Instruction: Goal: to discriminate these visual concepts in a photo. Please
list all possible visual descriptive phrases for each visual concept.

In this template, [CLS] represents the concept name, and [DEF] stands for its
definition from WordNet (Miller 1995). The LLM then generates a roster of
distinctive attributes for each category, such as ‘red-and-black tail’. Each
(category, attribute) pair is filled into an ensemble of CLIP prompts, e.g.,
“A photo of a {category} with {attribute}.”, and forms an augmented text
embedding. This forms an enhanced text embedding. For practicality and
clarity, and to sidestep ambiguity arising from the polysemy phenomenon, we
utilize all synsets and their definitions from WordNet (Miller 1995) for a
single category as the input visual concepts for the LLM prompt.

Re-alignment. During the re-alignment phase, our goal is to enhance the
structural semantic alignments outlined in Eq. 2. We achieve this by re-
establishing alignments between image clusters and the enriched text
embeddings from the earlier step. The refined structural semantic alignment,
$\tilde{M}\in\mathbf{R}^{K\times|\mathcal{W}|}$, is derived by casting votes
on all augmented text embeddings produced in the prompting stage.

To be precise, the alignment relationship between the $k$-th cluster and word
$w_{j}$ is determined by the normalized sum of occasions where the category
$w_{j}$ ranks within the top-3 closest matches for all image embeddings
$\mathbf{z}\in\Gamma_{k}$, w.r.t. all augmented text embeddings
$\mathcal{A}_{k}$. The process can be described by the equation:

$\tilde{M}_{k,j}=\frac{1}{3K|\Gamma_{k}|}\sum_{\mathbf{z}\in\Gamma_{k}}{\mathbb{I}(w_{j}\in\arg\mathop{\texttt{top3}}_{w}(\mathbf{z},\mathcal{A}_{k}))}$
(3)

where, top3 identifies the three closest words to the query image embedding
$\mathbf{z}$, determined using cosine distance. Meanwhile, $\arg$ extracts the
category name linked with the enriched text embedding. In the matrix
$\tilde{M}_{k,j}$, each row signifies the voting probability of the top-3
vocabulary terms for the $k$-th cluster. To establish a clear relationship
between clusters and words, we employ the maximum Hungarian matching technique
(Kuhn 1955) on a bipartite graph, using the weight matrix $\tilde{M}$. This
approach ensures that no category is linked with multiple clusters.
Consequently, every cluster is paired with a unique pseudo category label.

### Self Training with Semantic Alignment

In this section, we present our S3A self-training method, as depicted in Fig.
2. The self-training process leverages both instance-wise and structural
alignment pseudo labels which are produced by using an exponentially moving
averaged (EMA) teacher model. Throughout this procedure, we refine the CLIP
image encoder to improve its representational capabilities, and fix the text
encoder instead.

Structural Semantic Alignment. To incorporate the structural semantic
alignments, computed by our CVPR algorithm, into online self-training, one
challenge needs to be addressed. It is critical to determine the uniform
interval for computing pseudo labels from our CVPR across all datasets. For
quality pseudo-labels with our CVPR algorithm, using consistent model
embeddings is crucial, but collecting them is computationally costly, and
determining the optimal execution interval across datasets is challenging. To
mitigate these issues, we introduce a slowly updated Exponentially Moving
Average (EMA) teacher model. This model executes the CVPR algorithm once per
epoch to yield stable and reliable pseudo labels, which then guide the self-
training of a student model (Grill et al. 2020).

We define the structural semantic alignment loss as the cross-entropy between
the predictions of the student model and the pseudo labels generated by the
teacher model. Formally, this loss for the $i$-th instance can be expressed
as:

$L_{str}(\mathbf{x}_{i})=-\mathbf{\hat{p}}^{T}_{T}(i)\log\mathbf{p}_{S}(\mathbf{x}_{i}).$
(4)

In this equation, $\mathbf{\hat{p}}_{T}(i)$ represents the one-hot pseudo
label for the $i$-th instance, which is inferred from the teacher model during
the last epoch. On the other hand, $\mathbf{p}_{S}$ denotes the softmax
prediction of the student model over the entire vocabulary, computed for the
input $\mathbf{x}_{i}$.

Individual Semantic Alignment. In addition to the structural semantic
alignment loss, we also guide our model with instance-wise pseudo alignments,
which are generated on-the-fly by the EMA teacher model. Without this
guidance, our model would likely converge to suboptimal solutions rapidly. We
formulate the individual semantic alignment loss for the $i$-th instance as
follows:

$L_{in}(\mathbf{x}_{i})=-\mathbb{I}(\mathbf{\tilde{p}}_{T}(\mathbf{x}_{i})>\tau)\mathbf{\tilde{p}}^{T}_{T}(\mathbf{x}_{i})\log\mathbf{p}_{S}(\mathbf{x}_{i}).$
(5)

In this equation, $\mathbf{\tilde{p}}_{T}$ represents the one-hot sharpened
pseudo label produced by the teacher model at each iteration. The symbol
$\tau$ denotes a confidence threshold, which ensures that the loss is computed
only for samples for which the teacher model has a high level of confidence.

To strike a balance between the structural and instance alignment losses, we
introduce a weighted combination of both. This approach not only helps the
model escape from rapid convergence to local optima but also refines the
representations used for future pseudo alignment generation. Consequently, our
total loss function for the $i$-th instance is formulated as:

$L(\mathbf{x}_{i})=L_{str}(\mathbf{x}_{i})+\gamma L_{in}(\mathbf{x}_{i}).$ (6)

Here, $\gamma$ represents a balancing factor that weights the contribution of
the instance alignment loss relative to the structural alignment loss. This
total loss is computed at each iteration, based on our CVPR algorithm which is
executed once per epoch on the teacher model.

We present entire details of our S3A training algorithm together with
implementation details in our appendix C.

## Experiments

Methods | SDogs | IN100 | IN1K | LV17 | NL26 | ET13 | ET30 | Avg
---|---|---|---|---|---|---|---|---
CLIP (Ideal) | 57.59/58.07 | 84.67/84.90 | 65.38/65.53 | 75.24/75.53 | 80.03/80.05 | 78.43/78.50 | 77.99/78.03 | 74.19/74.37
DINO+KMeans | -/45.99 | -/75.16 | -/55.27 | -/72.52 | -/62.81 | -/67.37 | -/64.69 | -/63.40
CLIP | 53.43/55.43 | 29.39/38.54 | 32.21/39.77 | 37.31/47.24 | 33.35/38.96 | 30.09/40.00 | 31.23/39.90 | 35.29/42.83
CLIP (Group) | 19.37/55.92 | 40.62/77.68 | 26.41/56.92 | 38.33/68.81 | 41.09/70.51 | 32.85/71.08 | 36.36/70.78 | 33.57/67.38
MUST | 57.20/60.61 | 33.37/52.56 | 28.97/37.00 | 31.71/49.35 | 35.30/48.68 | 38.46/58.25 | 33.41/47.08 | 36.92/50.50
SCD | 52.63/55.93 | 48.89/77.39 | 37.06/57.00 | 43.33/68.81 | 52.18/71.84 | 40.46/71.25 | 46.29/70.89 | 45.83/67.59
S3A (Our) | 58.94/62.19 | 52.08/82.76 | 42.43/63.15 | 48.34/75.57 | 56.20/75.97 | 45.21/76.92 | 50.41/76.14 | 50.51/73.24
Table 2: Transductive evaluation on seven benchmarks. Top-1 classification accuracy scores (left of ‘/‘) and clustering accuracy scores (right of ‘/‘) are reported in percentage. We highlight the highest scores except for the upper bound. #Row | Prompt | S.T. | $L_{str}$ | ImageNet-100 | Living17
---|---|---|---|---|---
Acc | Cluster | Acc | Cluster
1 | ✗ | ✗ | ✗ | 48.89 | 77.39 | 43.33 | 68.81
2 | ✓ | ✗ | ✗ | 51.81 | 79.38 | 44.60 | 68.69
3 | ✗ | ✓ | ✗ | 46.23 | 81.49 | 46.28 | 74.60
4 | ✗ | ✓ | ✓ | 49.00 | 82.08 | 46.55 | 73.04
5 | ✓ | ✓ | ✓ | 52.08 | 82.76 | 48.34 | 75.57
Table 3: Top-1 accuracy and Clustering results for our method ablations on
ImageNet-100 and BREEDS-Living17. We conduct ablations on our discriminative
prompt augmentation (Prompt), self-training stage (S.T.), and structural
semantic alignment loss ($L_{str}$).

### Evaluation

#### Datasets.

We evaluate S3A on two generic and five fine-grained benchmarks, i.e., the
generic benchmarks of sampled ImageNet-100 (IN100) and ImageNet-1K (IN1K)
(Deng et al. 2009), and fine-grained benchmarks of StanfordDogs (SDogs)
(Khosla et al. 2011), Living17 (LV17), Nonliving26 (NL26), Entity13 (ET13),
and Entity30 (ET30) in BREEDS (Santurkar, Tsipras, and Madry 2020)).
Furthermore, we evaluate our S3A on three benchmarks for the out-of-vocabulary
evaluation (containing categories out of our vocabulary), i.e., Oxford-IIIT
Pet (Pet) (Parkhi et al. 2012), CIFAR100 (Krizhevsky, Hinton et al. 2009), and
Caltech101(Clatech) (Fei-Fei, Fergus, and Perona 2004). The profile of
datasets is listed in our appendix A.

#### Metrics

We adopt the top-1 classification accuracy and clustering accuracy (following
SCD (Han et al. 2023) and defined below) for the evaluation.

$\textsc{Acc}_{clu}=\frac{1}{N}\sum_{i=0}^{N}{\max_{\rho}{\mathbb{I}(y_{i}=\rho(\hat{y_{i}}))}},$
(7)

where $\rho$ is a permutation assignment of cluster indices. $y_{i}$ and
$\hat{y}_{i}$ are ground-truth predicted categories. Meanwhile, we adopt
Intersection-over-Union (IoU) score as an auxiliary metric in ablations to
inspect the overlap between our predictions $\mathcal{Y}_{pred}$ and the
ground-truth label set $\mathcal{Y}_{gt}$, i.e.,
$\frac{|\mathcal{Y}_{pred}\bigcap\mathcal{Y}_{gt}|}{|\mathcal{Y}_{pred}\bigcup\mathcal{Y}_{gt}|}$.
In the out-of-vocabulary experiments, some class names cannot be found in the
vocabulary. Thus, we instead apply a soft accuracy score, defined as the
similarity between the predicted word (in vocabulary) and the ground truth
label. Inspired by BertScore (Zhang et al. 2019), we adopt a language model,
Sentence-Bert (Reimers and Gurevych 2019), to calculate the similarity.

#### Baselines

RZSC is a new setting in which few baselines are ready-to-use. Thus, we
evaluate the baseline methods by reproducing them with officially released
codes in our setting. Specifically, we consider CLIP as the naive baseline,
and two state-of-the-art methods in ZST and UF, i.e., SCD (Han et al. 2023)
and MUST (Li, Savarese, and Hoi 2022). In summary, the following baselines are
included for performance comparisons:

  * •

DINO+KMeans (Caron et al. 2021): DINO is an contrastive self-supervised
learning method. We include it here for clustering quality comparisons. We
report only its clustering accuracy as it cannot classify.

  * •

CLIP (Radford et al. 2021): a large-scale VLM pre-trained with massive image-
caption pairs conducts zero-shot prediction given our vocabulary.

  * •

CLIP (Group) (Radford et al. 2021): We sequentially conduct clustering,
voting, and Hungarian matching on CLIP image embeddings for structural zero-
shot transfer, using S3A vocabulary.

  * •

CLIP (Ideal) (Radford et al. 2021): it denotes zero-shot transfer with pre-
trained CLIP but given an ideal vocabulary, showcasing the upper bound
performance of CLIP representation.

  * •

MUST (Li, Savarese, and Hoi 2022): it is an unsupervised ZSC method leveraging
instance-wise unsupervised self-training jointly with self-supervised masked-
image prediction. We adapt it with our huge vocabulary.

  * •

SCD (Han et al. 2023): it is an unsupervised/semi-supervised zero-shot
transfer method with WordNet vocabulary. Its iterative algorithm aligns each
cluster with one category name. We adapt it with our S3A vocabulary. During
the inference, we classify images with $K$ category prototypes discovered at
the training phase.

### Main Results

To validate the effectiveness of our proposed method, S3A, we conducted an
extensive evaluation under the Real Zero-Shot Classification (RZSC) setting.
We compared S3A with various baselines on both fine-grained and generic
datasets. The results, including classification and clustering accuracies, are
reported in Table 2.

Our method, S3A, consistently achieves SOTA results, outperforming CLIP by a
substantial margin—specifically, over $15\%$ improvement in classification
accuracy. Furthermore, S3A notably excels over our adapted SOTA baselines,
with improvements of nearly $5\%$ in top-1 accuracy and $6\%$ in clustering
accuracy. Impressively, it even surpasses CLIP with an ideal vocabulary on the
Stanford Dogs dataset. Despite CLIP (Group) being augmented with the same
clustering information as S3A, it encounters significant alignment issues when
working with large vocabularies under low-quality clustering, as seen in
ImageNet-1K and Stanford Dogs. We posit this superior performance of S3A may
be attributed to its capacity to dynamically calibrate noisy clustering during
the self-learning process. It is noteworthy that the existing SOTA in
unsupervised zero-shot classification, MUST, at times fails to improve upon
the naive CLIP baseline when using our realistic vocabulary. This underlines
the suboptimality of naive self-training methods for RZSC.

### Ablations and Analysis

Method Ablations. To validate the contribution of each component of our S3A,
we conduct method ablation experiments on one generic and fine-grained
dataset, i.e., ImageNet-100 and LV17. We present the results in Table 3. The
last row represents our full method. When we only keep the initial iterative
clustering in our CVPR (the 1st row), our method is equivalent to SCD (Han et
al. 2023). The 2nd row denotes our CVPR without all self-training-related
components; while, the 3rd row conducts self-training only with instance-wise
semantic alignment, similar to MUST (Li, Savarese, and Hoi 2022). The 4th row
indicates our S3A without LLM knowledge guidance. Based on the results, we can
conclude that: (1) Comparing Row 4 & 5, although the clustering quality
remains comparable without our discriminative prompt augmentation, the
semantic alignment degrades witnessed by the drop in the top-1 accuracy. (2)
Comparing Row 1, 2 & 3, self-training dominates the contribution in
representation adaptation, witnessed by the cluster performance boosts. (3)
Comparing Rows 3 & 4, we observe that the structural alignment loss yields
greater improvements on generic datasets, while its effect is less pronounced
on fine-grained datasets. This observation is consistent with our
expectations, as Hungarian matching proves to be more challenging for fine-
grained tasks, due to the potential confusion between closely related
categories. In summary, all components of our S3A enhance the performance.

Performance on Estimated $K$. The clustering algorithm requires the number of
classes $K$. In Table 2, we used the ground truth $K$. In addition, we also
offer an iterative algorithm using pre-trained CLIP to enable the data itself
to determine an appropriate $K$, based on the semantic structure of data
represented by CLIP (Ouldnoughi, Kuo, and Kira 2023; Han et al. 2023), where
the detailed algorithm is shown in the appendix C. We present our experimental
results on the estimated $K$ in Table 4 on four fine-grained datasets. The
estimated $K$ are listed beside the dataset title, which is close to the
ground-truth values. We show that the estimated $K$ can also bring comparable
results with ground truth ones. Beyond this experiment, we also perform
ablation studies to assess the impact of various estimation error scales on
$K$. Detailed results and discussions are in our appendix D.

Methods | LV17 (73) | NL26 (101) | ET13 (252) | ET30 (206)
---|---|---|---|---
DINO+KMeans | -/72.68 | -/62.93 | -/67.41 | -/63.22
CLIP | 37.31/47.24 | 33.35/38.96 | 30.09/40.00 | 31.23/39.90
MUST | 31.71/49.35 | 35.30/48.68 | 38.46/58.25 | 33.41/47.08
SCD | 40.70/69.17 | 52.63/70.21 | 40.12/71.37 | 45.03/69.14
S3A (Est-K) | 49.83/76.23 |  57.10/75.66 | 45.54/77.23 | 47.86/72.75
S3A (GT-K) | 48.34/75.57 | 56.20/75.97 | 45.21/76.92 | 50.41/76.14
Table 4: Transductive evaluation on four fine-grained benchmarks with estimated cluster numbers (Acc/Cluster). The estimated number is behind the dataset title. Methods | Caltech (0.34) | CIFAR100 (0.12) | Pet (0.62)
---|---|---|---
CLIP (Ideal) | 91.25/90.96 | 81.54/81.12 | 90.87/92.39
CLIP | 50.59/49.66 | 41.62/41.65 | 55.60/57.96
MUST | 51.20/50.80 | 42.93/42.96 | 58.32/55.83
SCD | 54.08/54.46 | 42.62/41.64 | 58.57/57.58
S3A (Our) | 55.29/55.55 | 46.10/46.40 | 59.00/60.57
Table 5: Tranductive and inductive evaluation on out-of-vocabulary benchmarks (Train Acc/Test Acc). The out-of-vocabulary ratios for each dataset are provided alongside their respective names. Performance is reported by cosine similarity of generic pre-trained Sentence-BERT, upscaled by one hundred. #Row | Methods | IN1K | ET13 | ET30
---|---|---|---|---
1 | CLIP (Ideal) | 65.38/96.58 | 78.43/99.61 | 77.99/99.58
2 | CLIP | 32.21/96.49 | 30.09/97.31 | 31.23/95.83
3 | SCD | 37.06/35.09 | 40.46/32.31 | 46.29/39.94
4 | CHiLS∗ | 36.23/34.46 | 41.13/33.33 | 46.09/39.94
5 | Our (WordNet) | 18.69/18.42 | 21.82/16.85 | 20.38/15.94
6 | Our (Single) | 37.40/35.74 | 41.13/33.00 | 47.09/40.76
7 | Our (ChatGPT) | 37.69/36.11 | 42.65/36.84 | 47.43/41.18
8 | Our (GPT-4) |  37.95/36.48 |  44.98/37.56 |  48.37/42.01
Table 6: Ablations on prompt augmentation techniques. Performance is reported
by cosine similarity of generic pre-trained Sentence-BERT, upscaled by one
hundred. a/b denotes the results of Acc/IoU. Figure 4: Qualitative results in
IN100 without finetuning (SCD (Han et al. 2023) and our CVPR).

On Out-Of-Vocabulary (OOV) Scenarios. Considering the scenarios in which
target datasets have category names out of our S3A vocabulary, we further
conduct an out-of-vocabulary evaluation on three benchmarks, i.e., Caltech101
(Fei-Fei, Fergus, and Perona 2004), CIFAR100 (Krizhevsky, Hinton et al. 2009),
and Oxford-IIIT Pet (Parkhi et al. 2012). The out-of-vocabulary ratios of
datasets and results are presented in Table 5. We can conclude that S3A still
achieves SOTA performance in this challenging setup on both inductive and
transductive evaluation.

On Effectiveness of S3A Prompt Augmentation. In this ablation experiment, we
analyze the effect of the proposed LLM-guided discriminative prompt
augmentation in our CVPR algorithm. We compare with four augmentation setups
in Table 6: (1) using WordNet definition for augmentation (5th row); (2)
reduce prompt semantic discriminativeness by requesting visual attributes for
only a single category name in each LLM prompt (6th row); (3) our prompt
augmentation guided by ChatGPT (7th row); (4) our prompt augmentation guided
by GPT-4. Besides, we also compare with a recent SOTA, CHiLS (Novack et al.
2023), in prompt augmentation for zero-shot prediction. We use their prompt to
generate ten subcategories for each class. We can draw the following
conclusions: first, semantic distinctiveness in prompts aids fine-grained
differentiation; second, incorporating WordNet linguistic knowledge may hinder
performance; third, our approach outperforms CHiLS, making it preferable for
RZSC tasks; fourth, CLIP’s focus on instance-wise alignment leads to low ACC
but high IoU; finally, our method benefits from advanced LLMs.

Qualitative Examples We present qualitative examples from IN100 in Fig. 4 to
illustrate the superiority of our semantic structural alignment. The results
demonstrate that our offline CVPR algorithm can effectively correct category
misrecognitions and precisely focus on salient categories.

## Conclusion

In this work, we have tackled the challenging task of Realistic Zero-Shot
Classification, without assuming partial source supervision or ideal
vocabularies. We have proposed the Self Structural Semantic Alignment (S3A)
framework, anchored by the innovative Cluster-Vote-Prompt-Realign (CVPR)
algorithm for structural semantic mining and a self-training process for
iterative semantic alignment. This CVPR algorithm effectively employs
unlabeled data to generate structural semantic information, enabling the model
to further self-learn and continuously refine its understanding. Our
experiments clearly demonstrate the potency of S3A, achieving a significant
accuracy improvement relative to baseline methods on various generic and fine-
grained benchmarks.

## References

  * Akata et al. (2015) Akata, Z.; Perronnin, F.; Harchaoui, Z.; and Schmid, C. 2015.  Label-embedding for image classification.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(7): 1425–1438.
  * Arthur and Vassilvitskii (2007) Arthur, D.; and Vassilvitskii, S. 2007.  K-means++ the advantages of careful seeding.  In _Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms_ , 1027–1035.
  * Bao et al. (2021) Bao, H.; Dong, L.; Piao, S.; and Wei, F. 2021.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_.
  * Cappallo, Mensink, and Snoek (2016) Cappallo, S.; Mensink, T.; and Snoek, C. G. 2016.  Video stream retrieval of unseen queries using semantic memory.  _arXiv preprint arXiv:1612.06753_.
  * Caron et al. (2021) Caron, M.; Touvron, H.; Misra, I.; Jégou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 9650–9660.
  * Cubuk et al. (2020) Cubuk, E. D.; Zoph, B.; Shlens, J.; and Le, Q. V. 2020.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 702–703.
  * Dale (2021) Dale, R. 2021.  GPT-3: What’s it good for?  _Natural Language Engineering_ , 27(1): 113–118.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Fei-Fei, Fergus, and Perona (2004) Fei-Fei, L.; Fergus, R.; and Perona, P. 2004.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories.  _2004 Conference on Computer Vision and Pattern Recognition Workshop_ , 178–178.
  * Gao et al. (2021) Gao, P.; Geng, S.; Zhang, R.; Ma, T.; Fang, R.; Zhang, Y.; Li, H.; and Qiao, Y. 2021\.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_.
  * Ghiasi et al. (2021) Ghiasi, G.; Gu, X.; Cui, Y.; and Lin, T.-Y. 2021.  Open-vocabulary image segmentation.  _arXiv preprint arXiv:2112.12143_.
  * Grill et al. (2020) Grill, J.-B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.; Buchatskaya, E.; Doersch, C.; Avila Pires, B.; Guo, Z.; Gheshlaghi Azar, M.; et al. 2020.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33: 21271–21284.
  * Han et al. (2023) Han, K.; Li, Y.; Vaze, S.; Li, J.; and Jia, X. 2023.  What’s in a Name? Beyond Class Indices for Image Recognition.  _arXiv preprint arXiv:2304.02364_.
  * Jia et al. (2021) Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q.; Sung, Y.-H.; Li, Z.; and Duerig, T. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , 4904–4916. PMLR.
  * Kahana, Cohen, and Hoshen (2022) Kahana, J.; Cohen, N.; and Hoshen, Y. 2022.  Improving Zero-Shot Models with Label Distribution Priors.  _arXiv preprint arXiv:2212.00784_.
  * Karazija et al. (2023) Karazija, L.; Laina, I.; Vedaldi, A.; and Rupprecht, C. 2023.  Diffusion Models for Zero-Shot Open-Vocabulary Segmentation.  _arXiv preprint arXiv:2306.09316_.
  * Khosla et al. (2011) Khosla, A.; Jayadevaprakash, N.; Yao, B.; and Li, F.-F. 2011.  Novel dataset for fine-grained image categorization: Stanford dogs.  In _Proc. CVPR workshop on fine-grained visual categorization (FGVC)_. Citeseer.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Krizhevsky, Sutskever, and Hinton (2017) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2017.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6): 84–90.
  * Kuhn (1955) Kuhn, H. W. 1955.  The Hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2): 83–97.
  * Lampert, Nickisch, and Harmeling (2009) Lampert, C. H.; Nickisch, H.; and Harmeling, S. 2009.  Learning to detect unseen object classes by between-class attribute transfer.  In _2009 IEEE conference on computer vision and pattern recognition_ , 951–958. IEEE.
  * Lei Ba et al. (2015) Lei Ba, J.; Swersky, K.; Fidler, S.; et al. 2015.  Predicting deep zero-shot convolutional neural networks using textual descriptions.  In _Proceedings of the IEEE international conference on computer vision_ , 4247–4255.
  * Li et al. (2021) Li, D.; Li, J.; Li, H.; Niebles, J. C.; and Hoi, S. C. H. 2021.  Align and Prompt: Video-and-Language Pre-training with Entity Prompts.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 4943–4953.
  * Li et al. (2023) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Li et al. (2022) Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , 12888–12900. PMLR.
  * Li, Savarese, and Hoi (2022) Li, J.; Savarese, S.; and Hoi, S. C. 2022.  Masked unsupervised self-training for zero-shot image classification.  _arXiv preprint arXiv:2206.02967_.
  * Lin et al. (2022) Lin, Y.; Li, C.; Cao, Y.; Zhang, Z.; Wang, J.; Wang, L.; Liu, Z.; and Hu, H. 2022\.  A Simple Approach and Benchmark for 21,000-Category Object Detection.  In _European Conference on Computer Vision_ , 1–18. Springer.
  * Liu et al. (2023) Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9): 1–35.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Miller (1995) Miller, G. A. 1995.  WordNet: a lexical database for English.  _Communications of the ACM_ , 38(11): 39–41.
  * Novack et al. (2023) Novack, Z.; McAuley, J.; Lipton, Z. C.; and Garg, S. 2023.  Chils: Zero-shot image classification with hierarchical label sets.  In _International Conference on Machine Learning_ , 26342–26362. PMLR.
  * Ouldnoughi, Kuo, and Kira (2023) Ouldnoughi, R.; Kuo, C.-W.; and Kira, Z. 2023.  CLIP-GCD: Simple Language Guided Generalized Category Discovery.  _arXiv preprint arXiv:2305.10420_.
  * Parkhi et al. (2012) Parkhi, O. M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C. V. 2012.  Cats and dogs.  _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , 3498–3505.
  * Pourpanah et al. (2022) Pourpanah, F.; Abdar, M.; Luo, Y.; Zhou, X.; Wang, R.; Lim, C. P.; Wang, X.-Z.; and Wu, Q. J. 2022.  A review of generalized zero-shot learning methods.  _IEEE transactions on pattern analysis and machine intelligence_.
  * Pratt, Liu, and Farhadi (2022) Pratt, S.; Liu, R.; and Farhadi, A. 2022.  What does a platypus look like? Generating customized prompts for zero-shot image classification.  _ArXiv_ , abs/2209.03320.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Ren et al. (2022) Ren, S.; Li, L.; Ren, X.; Zhao, G.; and Sun, X. 2022.  Rethinking the Openness of CLIP.  _arXiv preprint arXiv:2206.01986_.
  * Ren et al. (2023) Ren, S.; Zhang, A.; Zhu, Y.; Zhang, S.; Zheng, S.; Li, M.; Smola, A.; and Sun, X. 2023.  Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognition.  _arXiv preprint arXiv:2304.04704_.
  * Ridnik et al. (2021) Ridnik, T.; Ben-Baruch, E.; Noy, A.; and Zelnik-Manor, L. 2021.  Imagenet-21k pretraining for the masses.  _arXiv preprint arXiv:2104.10972_.
  * Rousseeuw (1987) Rousseeuw, P. J. 1987.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.  _Journal of computational and applied mathematics_ , 20: 53–65.
  * Santurkar, Tsipras, and Madry (2020) Santurkar, S.; Tsipras, D.; and Madry, A. 2020.  Breeds: Benchmarks for subpopulation shift.  _arXiv preprint arXiv:2008.04859_.
  * Sariyildiz et al. (2021) Sariyildiz, M. B.; Kalantidis, Y.; Larlus, D.; and Alahari, K. 2021.  Concept generalization in visual representation learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 9629–9639.
  * Shao et al. (2023) Shao, Z.; Yu, Z.; Wang, M.; and Yu, J. 2023.  Prompting Large Language Models with Answer Heuristics for Knowledge-based Visual Question Answering.  _ArXiv_ , abs/2303.01903.
  * Shin, Albanie, and Xie (2023) Shin, G.; Albanie, S.; and Xie, W. 2023.  Zero-shot Unsupervised Transfer Instance Segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4847–4857.
  * Vaze et al. (2022) Vaze, S.; Han, K.; Vedaldi, A.; and Zisserman, A. 2022.  Generalized Category Discovery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 7492–7501.
  * Wang et al. (2019) Wang, W.; Zheng, V. W.; Yu, H.; and Miao, C. 2019.  A survey of zero-shot learning: Settings, methods, and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 10(2): 1–37.
  * Wu et al. (2023) Wu, J.; Li, X.; Yuan, S. X. H.; Ding, H.; Yang, Y.; Li, X.; Zhang, J.; Tong, Y.; Jiang, X.; Ghanem, B.; et al. 2023.  Towards Open Vocabulary Learning: A Survey.  _arXiv preprint arXiv:2306.15880_.
  * Xu et al. (2023) Xu, J.; Liu, S.; Vahdat, A.; Byeon, W.; Wang, X.; and De Mello, S. 2023.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2955–2966.
  * Yang et al. (2021) Yang, Z.; Gan, Z.; Wang, J.; Hu, X.; Lu, Y.; Liu, Z.; and Wang, L. 2021.  An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA.  _ArXiv_ , abs/2109.05014.
  * Zang et al. (2022) Zang, Y.; Li, W.; Zhou, K.; Huang, C.; and Loy, C. C. 2022.  Open-vocabulary detr with conditional matching.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IX_ , 106–122. Springer.
  * Zhang et al. (2023) Zhang, J.; Huang, J.; Jin, S.; and Lu, S. 2023.  Vision-language models for vision tasks: A survey.  _arXiv preprint arXiv:2304.00685_.
  * Zhang et al. (2019) Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhou, Loy, and Dai (2022) Zhou, C.; Loy, C. C.; and Dai, B. 2022.  Extract free dense labels from clip.  In _European Conference on Computer Vision_ , 696–712. Springer.
  * Zhou et al. (2022a) Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022a.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9): 2337–2348.
  * Zhou et al. (2022b) Zhou, X.; Girdhar, R.; Joulin, A.; Krähenbühl, P.; and Misra, I. 2022b.  Detecting twenty-thousand classes using image-level supervision.  In _European Conference on Computer Vision_ , 350–368. Springer.
  * Zhou et al. (2023) Zhou, Z.; Lei, Y.; Zhang, B.; Liu, L.; and Liu, Y. 2023.  Zegclip: Towards adapting clip for zero-shot semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11175–11185.
  * Zhu et al. (2022) Zhu, X.; Zhang, R.; He, B.; Zeng, Z.; Zhang, S.; and Gao, P. 2022.  PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning.  _ArXiv_ , abs/2211.11682.

## Towards Realistic Zero-Shot Classification via Self Structural Semantic
Alignment – Supplementary Material

##  Appendix S-1 A Dataset Profile

Dataset | StanfordDogs | ImageNet-100 | ImageNet-1K | Living17 | Nonliving26 | Entity13 | Entity30
---|---|---|---|---|---|---|---
#Train | 12K | 127K | 1M | 88K | 132K | 334K | 307K
#Test | 8K | 5K | 50K | 3K | 5K | 13K | 12K
#Category | 120 | 100 | 1000 | 68 | 104 | 260 | 240
Table S-7: The profile of seven benchmarks for performance comparisons. Here, we report the training set size, test set size, and category name. Dataset | Caltech101 | CIFAR100 | Pet
---|---|---|---
#Train | 6K | 50K | 3K
#Test | 1K | 10K | 3K
#Category | 100 | 100 | 37
%OOV | 34 | 12 | 62
Table S-8: The profile of three benchmarks for out-of-vocabulary evaluation.
Here, we report the training set size, test set size, total category number,
and out-of-vocabulary ratio of category names.

We report the details of our benchmarks in our main results in Table S-7.
Additionally, we also present the out-of-vocabulary benchmark details in Table
S-8.

##  Appendix S-2 B Additional Related Work

Prompt Augmentation. Prompt augmentation has proven conducive to multi-modal
zero-shot transfer by providing supplementary information in addition to raw
category names to better capture the training data distribution (Liu et al.
2023; Radford et al. 2021). The simplest practice is ensembling prompts with
handcrafted templates (Radford et al. 2021). Further, driven by the powerful
knowledge retrieval capability of emergent LLMs (Dale 2021; Shao et al. 2023),
some recent works (Novack et al. 2023; Pratt, Liu, and Farhadi 2022; Zhu et
al. 2022) propose to augment text-prompt templates by querying LLM for class-
specific semantic information (e.g., attributes) or incorporating dataset-
specific domain knowledge. Based on our empirical comparisons, our proposed
VTA augmentation is better at distinguishing between fine-grained semantic
categories.

Learning with Large Vocabulary. Recently, large vocabulary vision tasks grow
popular in the research community. POMP (Ren et al. 2023) proposes to utilize
few-shot examples in ImageNet-21K (Ridnik et al. 2021) to enhance the
generalization capability of CLIP on downstream tasks/datasets. Detic (Zhou et
al. 2022b) proposes to employ massive image-labeled images from ImageNet-21K
to tune a detector classifier of over 20K categories for open-vocabulary
object detection. Meanwhile, (Lin et al. 2022) also proposes a simple two-
stage baseline for the object detection task, which jointly learns from large-
vocabulary classification data and small-vocabulary detection. However, their
methods are orthogonal to ours, since they treat images from a large number of
categories as labeled source data to benefit generalization towards downstream
datasets; while our learning goal is to categorize an unseen unlabeled dataset
with no supervision.

##  Appendix S-3 C S3A Training Algorithm

### More Details on CVPR Algorithm

Iterative Clustering. In the clustering step of our CVPR, we incorporate
additional iterative clustering steps (Han et al. 2023), which refine the
previous cluster results with voting results on textual prototypes. Suppose
the clustering partition at $t$-th step is $\Gamma^{t}$, and its top-1 voting
matrix is $M^{t}\in\mathcal{R}^{K\times|\mathcal{W}|}$, we assign each cluster
with a distinct category name by Hungarian matching, i.e., solving maximum-
weight bipartite matching between $K$ clusters and $|\mathcal{W}|$ words based
on the weight matrix $M^{t}$. Formally, this optimization problem of bipartite
matching is formulated as:

$\begin{array}[]{llcl}\max_{A}&tr(M^{t}A^{T})&&\\\
\text{s.t.}&\displaystyle\sum_{j=1}^{|\mathcal{W}|}A_{kj}=1,&\forall
k\in\\{1,\ldots,K\\}&\\\ &\displaystyle\sum_{k=1}^{K}A_{kj}\leq 1,&\forall
j\in\\{1,\ldots,|\mathcal{W}|\\}&\\\ &A_{kj}\in\\{0,1\\},&\forall
k,j&\end{array}$ (8)

where $A\in\mathcal{R}^{K\times\mathcal{|W|}}$ is the 0-1 assignment matrix.
Here, $A_{k,j}=1$ denotes the $j$-th word is assigned to the $k$-th cluster,
and $A_{k,j}=0$ otherwise. We omit the superscript $t$ for $A$ for notation
simplicity.

After $A$ has been resolved, the textual prototypes of $K$ assigned words are
considered as common argmax (pseudo) classifiers which cluster all image
embeddings into the updated partition $\Gamma^{t+1}$. Then, the image clusters
$\Gamma^{t+1}$ follow the same previous process. This iteration terminates
until the partition $\Gamma^{t+1}$ does not change any further.

Method | LV17 | NL26 | ET13 | ET30
---|---|---|---|---
Ground-truth | 68 | 104 | 260 | 240
Single-pass | 52 | 87 | 323 | 201
Iterative (Our) | 73 | 101 | 252 | 206
Table S-9: Different methods to estimate $K$.

Estimation of $K$ using CLIP. We propose a simple iterative estimation
technique to estimate the class number of an unlabeled dataset based on CLIP
image embeddings. GCD (Vaze et al. 2022) utilizes the elbow algorithm to
determine the optimal cluster count $K$ by pinpointing the inflection in
cluster scores across a predefined $K$ range. This ’elbow’ or inflection
arises when there is a noticeable deceleration in the reduction of clustering
scores. For our clustering metric, we employ the Silhouette score (Rousseeuw
1987). Relying solely on a single-pass elbow algorithm can make the estimated
$K$ vulnerable to noise. To address this, our iterative approach consists of
three passes: first, we scan the range [$\text{LB}_{0}$, $\text{UB}_{0}$],
then [$\text{LB}_{0}$, $\text{S}_{1}$], and finally [$\text{S}_{2}$,
$\frac{\text{S}_{2}+\text{S}_{1}}{2}$], each time applying the elbow
algorithm. Here, $\text{S}_{1}$ and $\text{S}_{2}$ denote the solution of the
first and second pass. In practice, we consistently set $\text{LB}_{0}=50$ and
$\text{UB}_{0}=2000$ across all datasets, and our experiments show that the
iterative method offers improved precision over the single-pass approach (see
Table S-9).

Params | IN/BREEDS | SDogs | Caltech | CIFAR | Pet
---|---|---|---|---|---
Init. EMA | 0.999 | 0.999 | 0.999 | 0.999 | 0.99
EMA Iter. | 2000 | 2000 | 2000 | 500 | 100
$\tau$ | 0.5 | 0.5 | 0.5 | 0.3 | 0.7
Epoch | 30 | 30 | 30 | 30 | 60
Table S-10: Hyperparameters for all benchmarks, including the initial EMA weight decay value, EMA warmup iterations, confidence parameter $\tau$, and training epoch number. Methods | IN100 | LV17 | NL26 | ET13 | ET30 | Avg
---|---|---|---|---|---|---
CLIP (Ideal) | 89.09 | 80.17 | 84.38 | 83.87 | 82.33 | 83.77
CLIP | 34.82 | 43.59 | 36.95 | 35.62 | 36.44 | 37.48
SCD | 50.61 | 54.10 | 55.23 | 48.82 | 49.44 | 51.64
S3A-CVPR (Our) | 52.51 | 55.31 | 56.02 | 51.31 | 52.78 | 53.59
Table S-11: Transductive evaluation on five benchmarks for backbone ablations.
Top-1 classification accuracy scores are reported in percentage. We highlight
the highest scores except for the upper bound. The evaluation is conducted for
the CLIP ViT-L/14 backbone.

Training Algorithm. In this part, we delineate our S3A training algorithm,
which is presented in algo. 1. At each training epoch, we first conduct our
CVPR algorithm on all extracted image embeddings from the teacher model to
obtain the semantic structural alignment labels $\mathbf{\hat{Y}}$ (from line
6 to line 18). During the training (from line 19 to line 24), we compute two
loss functions: (1) instance semantic alignment loss between one-shot pseudo-
label from the teacher model, i.e., the nearest neighbor prediction of each
instance on our vocabulary; (2) structural semantic alignment loss between
student predictions and our structural pseudo labels $\mathbf{\hat{Y}}$.
Finally, the teacher is EMA updated with EMA decay $\eta$ by student
parameters.

Typo Fix: Methodology/Cluster-Vote-Prompt-Realign/Voting/Line10: Change into
It indicates that for each row $M_{k}=(M_{k,j})_{j=1}^{|\mathcal{W}|}$, we
maintain the highest $m$ elements as 1 and others as 0.

Figure S-5: Assessing sensitivity to errors in $K$ estimation. We present
results corresponding to different percentages of estimation inaccuracies
relative to the true value of $K$. Figure S-6: Ablation study on the
structural semantic alignment loss weight $\beta$. By default, we set
$\beta=0.25$.

### Implementation Details of S3A

For our method, we construct our S3A huge taxonomic vocabulary from all
distinct class names in ImageNet-21K (Ridnik et al. 2021) and ImageNet-1K
(Krizhevsky, Sutskever, and Hinton 2017) datasets, which is comprehensive
enough to cover all common classes in downstream datasets (Sariyildiz et al.
2021; Ridnik et al. 2021). We fix $m=3$ and $\gamma=0.25$ across all datasets.
The class number $K$ of the target dataset is assumed known unless specified.
Considering requesting LLM is time-demanding, we only compute prompting at the
first epoch.

We apply strong augmentations for the student, i.e., RandomResizedCrop,
RandomFlip, and RandAug (Cubuk et al. 2020), and weak augmentations for the
teacher, i.e., Resize, and RandomCrop. We adopt ViT-B/16 (Dosovitskiy et al.
2020) as our CLIP backbone for main evaluation. Following (Li, Savarese, and
Hoi 2022), we utilize AdamW (Loshchilov and Hutter 2017) optimizer with a
learning rate of $1e^{-5}$, a batch size of $128$, and a weight decay of
$0.05$. The cosine learning rate schedule without warmup is adopted. Following
(Bao et al. 2021), we also adopt layer-wise learning rate decay of $0.65$.
Notice that we train for 30K iterations at maximum across all datasets.
Besides, we present details on other hyperparameters in Table S-10.
Specifically, we linearly warmup the EMA decay parameter from the specified
initial EMA value to $0.9998$ within specified iterations in Table S-10,
following (Li, Savarese, and Hoi 2022). We observe negligible random variances
in final results since our S3A is deterministic and our randomness only comes
from the optimizer. For backbone ablations, we increase the initial learning
rate to $2e^{-5}$ for ViT-L/14 backbone. During the inference, we adopt the
teacher model for inference with the weak data augmentation on our entire S3A
vocabulary for predictions. All experiments are conducted on a single RTX
A6000 GPU.

##  Appendix S-4 D Additional Experimental Results

Methods | Train Acc | Cluster Acc | Test Acc
---|---|---|---
CLIP (Ideal) | 83.87 | 83.21 | 83.20
CLIP (ViT-L/14) | 35.62 | 40.00 | 33.91
SCD (ViT-L/14) | 48.82 | 76.24 | 46.91
MUST (ViT-L/14) | 43.29 | 58.25 | 37.63
S3A (ViT-B/16) | 45.21 | 76.92 | 45.65
S3A (ViT-L/14) | 52.61 | 81.00 | 53.32
Table S-12: Transductive evaluation on BREEDS-ET13 benchmarks for backbone ablations. We highlight the highest scores except for the upper bound. The evaluation is conducted for the CLIP ViT-L/14 backbone. Method | SDogs | IN100 | ET13 | ET30 | Pet
---|---|---|---|---|---
BLIP-2 | 43.49 | 47.57 | 54.60 | 52.96 | 54.05
BLIP-2 (Group) | 45.01 | 51.65 | 59.60 | 58.17 | 55.12
S3A (Our) | 78.45 | 75.77 | 70.73 | 73.86 | 59.00
Table S-13: Additional evaluation for comparing with BLIP-2 LMM. The scores
are reported with soft accuracies based on Sentence-BERT similarity.

Ablations on Different Backbones. Besides the main results in Table 2, we
conduct additional ablations on our offline CVPR algorithm with different CLIP
backbones, i.e., ViT-L/14, and report our results in Table S-11. We can
observe that, even without self-alignment fine-tuning, our method can
considerably surpass our adapted SOTA, i.e., SCD, with nearly $2\%$ in the
top-1 accuracy. Moreover, we present our results in Table S-12 on a randomly
chosen benchmark, i.e., BREEDS-Entity13, to compare our entire method with
other baselines. Further, we conclude that our entire method with self
semantic alignment tuning can achieve significant performance advancement with
$\sim 4\%$ in classification and $\sim 5\%$ in clustering w.r.t. existing SOTA
methods.

Sensitivity to $K$. To determine the underlying effects of estimation error on
$K$ on our S3A method, we further assess the sensitivity of S3A performance
w.r.t. this estimation error. In Fig. S-5, we showcase results for values of
$K$ varying from a decrease of $20\%$ to an increase of $30\%$ from its true
value. From our analysis, it is evident that our S3A demonstrates substantial
resilience to over-fragmentation. However, with an equivalent degree of under-
fragmentation error, there is a noticeable performance drop due to a larger
increase in the number of instances inaccurately assigned.

Sensitivity to $\beta$. We conduct ablation study on the robustness of our S3A
towards the weight $\beta$ of the structural semantic alignment loss. The
results are presented in Table S-6. In this experiment, we vary the $\beta$ by
$0.1$ and $0.2$ respectively on both sides. From the findings, we deduce that
our S3A remains robust with variations in $\beta$, unless $\beta$ is
exceedingly small. As $\beta$ increases, there’s a marginal decline in overall
performance.

Additional Comparisons with the Large-Multimodal Model (LMM). We further
present additional results for comparisons between our S3A and the recently
proposed powerful LMM, i.e., BLIP-2 (Li et al. 2023), on several benchmarks to
showcase the effectiveness of our method. To ensure fair comparisons, we adopt
pre-trained BLIP-2 with ViT-B/16 backbone and reproduce the method based on
their official code. Our study includes two baselines: (1) In the naive CLIP-2
baseline, we prompt with what is the category name of the object in the photo?
$c_{0}$, $c_{1}$, or $c_{2}$? (where $c_{0},c_{1},c_{2}$ are candidate outputs
from SCD (Han et al. 2023)). Here, we treat the generated names as
predictions. (2) For the CLIP-2 (Group) baseline, the category name with the
highest voting frequency in each cluster (determined using KMeans (Arthur and
Vassilvitskii 2007) clustering algorithm) is designated as the prediction for
all images within that specific cluster. Given that the LMM output can
sometimes fall outside of the expected vocabulary, we utilize soft accuracy
scores for the evaluation. The experimental results are displayed in Table
S-13. We can conclude that our S3A consistently outperforms the powerful
BLIP-2 (Li et al. 2023), showcasing its superiority in handling generic, fine-
grained, and out-of-vocabulary benchmarks. We speculate that the low
performance of BLIP-2 might originate from its unpredictable text generation
tendencies evidenced by its occasional production of unrelated words.

1

Input: Unlabeled Training dataset $\mathcal{D}_{u}$, a pre-trained VLM (CLIP)
with image encoder $f_{I}(\cdot|\theta)$ and frozen text encoder
$f_{T}(\cdot)$, large vocabulary $\mathcal{W}$.

2

Output: Trained adapted model $f_{I}(\cdot|\theta)$.

3

4Initialize teacher encoder $f_{I,T}(\cdot|\theta_{S})$ from the student model
$f_{I}$.

5 Precompute textual embeddings $\mathbf{H}=f_{T}(\mathcal{W})$.

6

7for  _each epoch $e$=1…$E_{1}$ _ do

8 // Offline: CVPR Algorithm

9 $\mathbf{Z}=\phi,\mathbf{\tilde{Y}}=\phi$

10 for  _each batch $\mathbf{X}\in\mathcal{D}_{u}$_ do

$\mathbf{Z}=\left[\mathbf{Z};f_{I,T}(\mathbf{X})\right]$ // collect features

11

$\mathbf{\tilde{Y}}=\left[\mathbf{\tilde{Y}};\text{Nearest}(\mathbf{Z},\mathbf{H})\right]$
// collect predictions on vocabulary

12

13

14 // Iterative Clustering

$\Gamma^{0}=\text{KMeans}(\mathbf{Z}),t=0$ // initial clustering

15 for  _$t\leq\text{max\\_iter}$_ do

Compute $M^{t}$ from $\Gamma^{t}$ with eq. 2 ($m=1$) and $\mathbf{\tilde{Y}}$.
// voting

Compute cluster-category assignment $A$ from $M^{t}$ with eq. 8. // Hungarian
matching

16 Compute new cluster partition $\Gamma^{t+1}$ by assigning instances
$\mathbf{Z}$ to $K$ prototypes from
$\\{\mathbf{H}_{j}|\sum_{k=1}^{K}{A_{k,j}}=1\\}$.

17 $t=t+1$.

18

19

20 // Voting

21 Let $\Gamma=\Gamma^{t}$. Compute $M$ from $\Gamma$ with eq. 2 ($m=3$) and
$\mathbf{\tilde{Y}}$.

Obtain $\\{\mathcal{A}_{k}\\}_{k=1}^{K}$ with prompt augmentation. //
prompting

Compute $\tilde{M}$ with eq. 3 and obtain structural semantic alignment
$\mathbf{\hat{Y}}$ from Hungarian solution $A$ on $\tilde{M}$. // realigning

22 // Online: Self Training with Semantic Alignment

23 for  _each batch $(\mathbf{X},\mathbf{\hat{Y}})\in\mathcal{D}_{u}$_ do

$\mathbf{\tilde{Y}}=\text{Nearest}(f_{I,T}(\mathbf{X}),\mathbf{H})$. //
forward teacher

$\mathbf{\tilde{Y}}_{S}=f_{I}(\mathbf{X})$. // forward student

24 // Individual Semantic Alignment Loss

25 Compute cross entropy between $\mathbf{\tilde{Y}}_{S}$ and
$\mathbf{\tilde{Y}}$ threshold by confidence $\tau$.

26 // Structural Semantic Alignment Loss

27 Compute cross entropy between $\mathbf{\tilde{Y}}_{S}$ and
$\mathbf{\hat{Y}}$.

28 Update teacher parameters $\theta_{S}=\eta\theta_{S}+(1-\eta)\theta$.

29

30

31return _$f_{I,S}(\cdot|\theta_{S})$_

Algorithm 1 S3A Training Algorithm.