# The Curse of Dense Low-Dimensional Information Retrieval for Large Index
Sizes

Nils Reimers and Iryna Gurevych
Ubiquitous Knowledge Processing Lab (UKP-TUDA)
Department of Computer Science, Technical University of Darmstadt
www.ukp.tu-darmstadt.de

###### Abstract

Information Retrieval using dense low-dimensional representations recently
became popular and showed out-performance to traditional sparse-
representations like BM25. However, no previous work investigated how dense
representations perform with large index sizes. We show theoretically and
empirically that the performance for dense representations decreases quicker
than sparse representations for increasing index sizes. In extreme cases, this
can even lead to a tipping point where at a certain index size sparse
representations outperform dense representations. We show that this behavior
is tightly connected to the number of dimensions of the representations: The
lower the dimension, the higher the chance for false positives, i.e. returning
irrelevant documents.

##  1 Introduction

Information retrieval traditionally used sparse representations like TF-IDF or
BM25 to retrieve relevant documents for a given query. However, these
approaches suffer from the lexical gap problem Berger et al. (2000).

To overcome this issue, dense representations have been proposed Gillick et
al. (2018): Queries and documents are mapped to a dense vector space and
relevant documents are retrieved e.g. by using cosine-similarity. Out-
performance over sparse lexical approaches has been shown for various datasets
Gillick et al. (2018); Guo et al. (2020); Guu et al. (2020); Gao et al.
(2020).

Previous work showed the out-performance for fixed, rather small indexes. The
largest dataset where it has been shown is the MS Marco Bajaj et al. (2018)
passage retrieval dataset, where retrieval is done over an index of 8.8
million text passages. However, in production scenarios, index sizes quickly
reach 100 millions of documents.

We show in this paper, that the performance for dense representations can
decrease quicker for increasing index sizes than for sparse representations.
For a small index of e.g. 100k documents, a dense approach might clearly
outperform sparse approaches. However, with a larger index of several million
documents, the sparse approach can outperform the dense approach.

We show theoretically and empirically that this effect is closely linked to
the number of dimensions for the representations: Using fewer dimensions
increases the chances for false positives. This effect becomes more severe
with increasing index sizes.

##  2 Related Work

A common choice for dense retrieval is to fine-tune a transformer network like
BERT Devlin et al. (2018) on a given training corpus with queries and relevant
documents Guo et al. (2020); Guu et al. (2020); Gao et al. (2020); Karpukhin
et al. (2020); Luan et al. (2020). Recent work showed that combining dense
approaches with sparse, lexical approaches can further boost the performance
Luan et al. (2020); Gao et al. (2020). While the approaches have been tested
on various information and question answering retrieval datasets, the
performance was only evaluated on fixed, rather small indexes. Guo et al.
(2020) evaluated approaches for eight different datasets having index sizes
between 3k and 454k documents.

We are not aware of previous work that compares sparse and dense approaches
for increasing index sizes and the connection to the dimensionality. The only
work we are aware of that systematically studies the encoding size for dense
approaches is Luan et al. (2020), but they only studied the connection to the
document length.

##  3 Theory

Dense retrieval approaches map queries and documents111We use document as a
cover-term for text of any length. to a fixed size dense vector. The most
relevant documents for a given query can then be found using cosine-
similarity.

Using as few dimensions as possible is desirable, as it decreases the memory
requirement to store (an index) of millions of vectors and leads to faster
retrieval. However, as we show, lower-dimensional representations can have
issues with large indices.

Given a query vector $q\in\mathbb{R}^{k}$, we search our index of document
vectors $d_{1},...,d_{n}\in\mathbb{R}^{k}$ for the documents that maximizes:

$\text{cossim}(q,d_{i})=\text{cos}(\theta)=\frac{q\cdot
d_{i}}{\left\|q\right\|\left\|d_{i}\right\|}$

Note: In the following we just show the case for cosine similarity. The proof
extends to other similarity functions like dot-product and any p-norm
(Manhatten, Euclidean) as long as the vector space is finite. A finite
$n$-dimensional vector space can be mapped to an $n+1$-dimensional vectors
space with vectors of unit length. In that case, dot-product in $n$ dimensions
is equivalent to cosine-similarity in $n+1$ dimensions. Similar, any $p$-norm
in $n$ dimensions can be re-written as cosine-similarity in $n+1$ dimensions.

Theorem: The probability for false positives (I) increases with the index size
$n$ and (II) with the decreasing dimensionality $k$.

Proof (I): Given a query $q$ and the relevant document $d_{r}$. For
simplicity, we assume only a single relevant document. If multiple documents
are relevant, we consider only the one with the highest cosine similarity. In
order that no false positive is returned, $\text{cossim}(q,d_{r})$ must be
greater than $\text{cossim}(q,d_{i})$ for all $i\neq r$. Assume the possible
vectors are independent. Then, the probability for a false positive is

$P(\text{false positive})=1-(1-P(\text{false positive}_{i}))^{n-1}$

for an index with $n-1$ negative elements and $P(\text{false positive}_{i})$
the probability that a single element is a false positive, i.e.
$\text{cossim}(q,d_{i})>\text{cossim}(q,d_{r})$.

Proof (II): While the previous proof is straightforward, that the chance of
false positives increases with larger index sizes, the more interesting aspect
is the relation to the dimensionality, i.e., what is the probability
$P(\text{false positive}_{i})$ =
$P(\text{cossim}(q,d_{i})>\text{cossim}(q,d_{r}))$ for a random $d_{i}$? We
show that this probability decreases with more dimensions.

Without loss of generality, we assume that the vectors are of unit length. The
vectors are then on an $k$-dimensional sphere with radius 1. A false positive
happens if $\text{cossim}(q,d_{i})>\text{cossim}(q,d_{r})$, or, equivalent if
$1-\text{cossim}(q,d_{i})<1-\text{cossim}(q,d_{r})$. I.e., we intersect the
sphere in $k$ dimensions with a hyperplane in $k-1$ dimensions. The area of
the cut-off portion is defined by $1-\text{cossim}(q,d_{r})$. All vectors
within the cut-off portion (i.e. spherical cap) are false positives. The
probability that a random vector will be returned as false positive is:

$P(\text{false positive}_{i})=A_{cap}/A_{sphere}$

with $A_{cap}$ the surface area of the spherical cap and $A_{sphere}$ the
surface area of the sphere in $k$ dimensions. Define the surface area of the
sphere in $k$ dimensions as $A_{k}$, then the surface area of $A_{cap}$ is Li
(2011):

$A_{cap}=\frac{1}{2}A_{k}I_{sin^{2}\theta}\left(\frac{k-1}{2},\frac{1}{2}\right)$

with $I_{x}(a,b)$ the regularized incomplete beta function and $\theta$ the
polar angle, i.e. the angle between $q$ and the relevant document $d_{r}$.
Hence:

$\displaystyle P(\text{false
positive}_{i})=\frac{1}{2}I_{sin^{2}\theta}\left(\frac{k-1}{2},\frac{1}{2}\right)$
(1)

For constant cosine similarity between query $q$ and relevant document
$d_{r}$, $I_{sin^{2}\theta}\left(\frac{k-1}{2},\frac{1}{2}\right)$ is a
monotonically decreasing function with increasing dimension $k$. In
conclusion, more dimensions decrease the probability for false positives.

Combining (I) and (II) shows that a low dimensional representation might work
well for small index sizes. However, with more indexed documents, the
probability of false positives increases faster for low dimensional
representations than for higher dimensional representations. Hence, at some
index size, higher dimensional representations might outperform the lower-
dimensional representation.

##  4 Empirical Investigation

In the proof, we have assumed that vectors are independent and uniformly
distributed over the space, which gives us a lower bound on the false positive
rate. However, in practice, dense representations are neither independent nor
uniformly distributed. As shown in Ethayarajh (2019); Li et al. (2020), dense
representations derived from pre-trained Transformers like BERT map to an
anisotropic space, i.e., the vectors occupy only a narrow cone in the vector
space. This drastically increases the chance that an irrelevant document is
closer to the query embedding than the relevant document. Hence, we study how
actual dense models are impacted by increasing index sizes and lower-
dimensional representations.

###  4.1 Dataset

We conduct our experiments on the MS MARCO passage dataset Bajaj et al.
(2018). It consists of over 1 million unique real queries from the Bing search
engine, together with 8.8 million paragraphs from heterogeneous web sources.
Most of the queries have only 1 passage judged as relevant, even though more
can exist. The development set consists of 6980 queries and the performance is
evaluated using mean reciprocal rank MRR@10.

To better compare the relative performance differences, we compute a rank-
aware error rate:

$\text{Err}=\frac{1}{n}\sum_{i=1}^{n}\left(1-\frac{1}{\text{rank}_{i}}\right)$

with $\text{rank}_{i}$ being the rank of the relevant document for the $i$-th
query. To be compatible with MRR@10, we set $\text{rank}_{i}=\infty$ for
$\text{rank}_{i}>10$. We then define the relative error rate as
$\text{Err}_{\text{Dense}}/\text{Err}_{\text{BM25}}$. A relative error rate of
$50\%$ indicates that the dense approach makes only 50% of the errors compared
to BM25 retrieval.

###  4.2 Model

For sparse, lexical retrieval, we use ElasticSearch, which is based on BM25.
For dense retrieval, we use a DistilRoBERTa-base model Sanh et al. (2020) as a
bi-encoder: The query and the passage are passed independently to the
transformer model and the output is averaged to create fixed-sized
representations. We train this using InfoNCE loss van den Oord et al. (2018):

$L=-\log\frac{\exp(\tau\cdot\text{cossim}(q,p_{+}))}{\sum_{i}\exp(\tau\cdot\text{cossim}(q,p_{i}))}$

with $q$ the query, $p_{+}$ the relevant passage. We use in-batch negative
sampling and use the other passages in a batch as negative examples. We found
that $\tau=20$ performs well. We train the model in two setups: 1) only with
random (in-batch) negatives, and 2) we provide for each query additionally one
hard-negative passage. We use the hard-negative passages provided by the MS
MARCO dataset, which were retrieved using lexical search. Models are trained
with a batch size of 128 with Adam optimizer and a learning rate of $2e-5$.

DistilRoBERTa produces representations with 768 dimensions. We also experiment
with lower-dimensional representations. There, we added a linear projection
layer on-top of the mean pooling operation to down-project the representation
to either 128 or 256 dimensions. Dense retrieval is performed using cosine
similarity with exact search.

Models were trained using the SBERT framework Reimers and Gurevych
(2019).222https://www.SBERT.net

##  5 Experiments

First, we study the impact of increasing index sizes with real text passages.
Then, we study the performance when random noise is added.

###  5.1 Increasing Index Size

In the first experiment, we start with an index that only contains the 7433
relevant passages for the 6980 queries. Then, we add step-wise randomly
selected passages from the MS MARCO corpus to the index until all 8.8 million
passages are indexed.

Model | 10k | 100k | 1M | 8.8M
---|---|---|---|---
BM25 | 79.93 | 63.88 | 40.14 | 17.56
Trained without hard negatives
128 dim | 87.50 | 68.63 | 39.76 | 15.71
256 dim | 88.82 | 70.79 | 41.74 | 17.08
768 dim | 88.99 | 71.06 | 42.24 | 17.34
Trained with hard negatives
128 dim | 90.32 | 77.92 | 54.45 | 27.34
256 dim | 91.10 | 78.90 | 55.51 | 28.16
768 dim | 91.48 | 79.42 | 56.05 | 28.55
Table 1: Dev performance (MRR@10 $\times 100$) on MS MARCO passage dataset with different index sizes. Higher score = better. Model | 10k | 100k | 1M | 8.8M
---|---|---|---|---
Trained without hard negatives
128 dim | 62.3 | 86.8 | 100.6 | 102.2
256 dim | 55.7 | 80.9 | 97.3 | 100.6
768 dim | 54.9 | 80.1 | 96.5 | 100.3
Trained with hard negatives
128 dim | 48.2 | 61.1 | 76.1 | 88.1
256 dim | 44.3 | 58.4 | 74.3 | 87.1
768 dim | 42.5 | 57.0 | 73.4 | 86.7
Table 2: Relative error rate (%) of dense approaches in comparison to BM25
retrieval. Lower score = better.

Table 1 shows the MRR@10 performance for the different systems. Increasing the
index naturally decreases the performance for all systems, as retrieving the
correct passages from a larger index is more challenging. The dense approach
trained without hard negatives clearly outperforms BM25 for an index with 10k
- 1M entries, but with all 8.8 million passages it performs worse than BM25.

Table 2 shows the relative error rate in comparison to BM25 retrieval. For
small index sizes, we observe that dense approaches drastically reduce the
error rate compared to BM25 retrieval. With increasing index sizes, the gap
closes.

###  5.2 Index with Random Noise

MS MARCO is sparsely labeled, i.e., there is usually only a single passage
labeled as relevant even though multiple passages would be considered as
relevant by humans Craswell et al. (2020). To avoid that the drop in
performance is due to the retrieval of relevant, but unlabeled passages, we
perform an experiment where we add random irrelevant noise to the index. Our
index consists only of the relevant passages and a large fraction of
irrelevant, randomly generated strings.333Strings are generated randomly using
lowercase characters a-z and space.

We also evaluate the popular DPR system by Karpukhin et al. (2020), which is a
BERT-based dense retriever trained on the Natural Questions (NQ) dataset
Kwiatkowski et al. (2019). We chose the NQ dev set, consisting of 1772
questions from Google search logs. DPR encodes the passage as Title [SEP]
Paragraph. We create a random string for the paragraph and combine it with 1)
a randomly generated string as title, 2) selecting randomly one of the over 6
Million real Wikipedia article titles, 3) selecting randomly one of the 1772
article titles found in the NQ dev set.

Model | 100k | 1M | 10M | 100M
---|---|---|---|---
BM25 | 0.00% | 0.00% | 0.00% | 0.00%
Dense without hard negatives - MS MARCO
128 dim | 2.71% | 4.41% | 6.69% | 9.73%
256 dim | 2.39% | 4.03% | 6.16% | 9.04%
768 dim | 2.13% | 3.72% | 5.77% | 8.52%
Dense with hard negatives - MS MARCO
128 dim | 2.87% | 4.20% | 6.00% | 8.11%
256 dim | 2.45% | 3.72% | 5.59% | 7.38%
768 dim | 2.12% | 3.32% | 5.09% | 7.03%
DPR Karpukhin et al. (2020) \- Natural Questions
rnd title | 0.17% | 0.28% | 0.34% | 0.51%
all titles | 2.48% | 5.59% | 9.31% | 12.08%
dev titles | 4.18% | 5.36% | 6.66% | 8.01%
Table 3: Percentage of queries for which a random string passage is ranked
higher than the relevant passage. 100k/1M/10M/100M indicates the number of
random passages in the index.

We count for how many queries a random string is ranked higher than the
relevant passage. The results are shown in Table 3. We observe that BM25 does
not rank any randomly generated passage higher than the relevant passage for
the MS MARCO dataset. The chance that a random passage contains words matching
the query is small.

For the dense retrieval models, we observe for quite a large number of queries
that a random string passage is ranked higher than the relevant passage. As
proven in Section 3, the error increases with larger index sizes and fewer
dimensions.

For DPR, we observe an extreme dependency on the title. Having 100 million
entries in the index with a real Wikipedia article title and a random
paragraph, results in the retrieval of those for about 12.08% of all questions
at the top position.

The error numbers far exceed the estimation from equation (1), confirming that
the representations are not uniformly distributed over the complete vector
space and are concentrated in a small space. In the appendix (Figure 1), we
plot the representations for the queries, the relevant passages, and the
random strings.

##  6 Conclusion

We have proven and shown empirically that the probability for false positives
in dense information retrieval depends on the index size and on the
dimensionality of the used representations. These approaches can even retrieve
completely irrelevant, randomly generated passages with high probability. It
is important to understand the limitations of dense retrieval:

1) Dense approaches work better for smaller, clean indexes. With increasing
index size the difference to sparse approaches can decreases.

2) Evaluation results with smaller indexes cannot be transferred to larger
index sizes. A system that is state-of-the-art for an index of 1 million
documents might perform badly on larger indices.

3) The false positive rate increases with fewer dimensions.

4) The empirically found error rates far exceeded the mathematical lower-bound
error rates, indicating that only a small fraction of the available vector
space is effectively used.

## Acknowledgments

This work has been supported by the German Research Foundation through the
German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU
798/17-1) and has been funded by the German Federal Ministry of Education and
Research and the Hessian Ministry of Higher Education, Research, Science and
the Arts within their joint support of the National Research Center for
Applied Cybersecurity ATHENE.

## References

  * Bajaj et al. (2018) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.  _arXiv preprint arXiv:1611.09268 v3_.
  * Berger et al. (2000) Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. 2000.  Bridging the Lexical Chasm: Statistical Approaches to Answer-Finding.  In _Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’00, page 192–199, New York, NY, USA. Association for Computing Machinery.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020.  Overview of the TREC 2019 deep learning track.  _arXiv preprint arXiv:2003.07820 v2_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_.
  * Ethayarajh (2019) Kawin Ethayarajh. 2019.  How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 55–65, Hong Kong, China. Association for Computational Linguistics.
  * Gao et al. (2020) Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2020.  Complementing Lexical Retrieval with Semantic Residual Embedding.  _arXiv preprint arXiv:2004.13969_.
  * Gillick et al. (2018) Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018.  End-to-End Retrieval in Continuous Space.  _arXiv preprint arXiv:1811.08008_.
  * Guo et al. (2020) Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2020.  MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models.  _arXiv preprint arXiv:2005.02507_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  REALM: Retrieval-Augmented Language Model Pre-Training.  _arXiv preprint arXiv:2002.08909_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Transactions of the Association of Computational Linguistics_.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020.  On the Sentence Embeddings from Pre-trained Language Models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9119–9130, Online. Association for Computational Linguistics.
  * Li (2011) S. Li. 2011.  Concise formulas for the area and volume of a hyperspherical cap.  _Asian Journal of Mathematics and Statistics_ , 4:66–70.
  * Luan et al. (2020) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2020.  Sparse, Dense, and Attentional Representations for Text Retrieval.  _arXiv preprint arXiv:2005.00181_.
  * McInnes et al. (2018) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018.  Umap: Uniform manifold approximation and projection.  _The Journal of Open Source Software_ , 3(29):861.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation Learning with Contrastive Predictive Coding.  _arXiv preprint arXiv:1807.03748_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Sanh et al. (2020) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108 v4_.

##  Appendix A Plot of Random Noise Index

Figure 1 shows a two-dimensional plot of the 6980 development queries in the
MS MARCO passage dataset, together with the 7433 passages that are marked as
relevant and 7433 representations for randomly generated strings (using
lowercase characters and space with a random length between 20 and 150
characters). The representation for the random strings are concentrated, but
we still observe a significant overlap with the region for queries and
relevant documents. This explains why random strings are retrieved for certain
queries (Table 3). We use the dense model that was trained with hard negatives
with 768 dimensions. UMAP McInnes et al. (2018) is used for dimensionality
reduction to 2 dimensions.

Figure 1: Plot of queries (blue), the relevant document (green) and
representations from randomly generated strings (red). Dimensionality
reduction via UMAP McInnes et al. (2018). Model with hard negatives, 768
dimensions.


#  HGCLIP: Exploring Vision-Language Models with Graph Representations for
Hierarchical Understanding

Peng Xia1,3,4 Xingtong Yu6 Ming Hu1,3,5 Lie Ju1,3,5 🖂
Zhiyong Wang7 Peibo Duan4 Zongyuan Ge1,2,3,4 🖂

1Monash Medical AI Group 2AIM Lab, Monash University 3Airdoc-Monash Research
4Faculty of IT, Monash University 5Faculty of Engineering, Monash University
6University of Science and Technology of China 7The University of Sydney
<EMAIL_ADDRESS>
###### Abstract

Object categories are typically organized into a multi-granularity taxonomic
hierarchy. When classifying categories at different hierarchy levels,
traditional uni-modal approaches focus primarily on image features, revealing
limitations in complex scenarios. Recent studies integrating Vision-Language
Models (VLMs) with class hierarchies have shown promise, yet they fall short
of fully exploiting the hierarchical relationships. These efforts are
constrained by their inability to perform effectively across varied
granularity of categories. To tackle this issue, we propose a novel framework
(HGCLIP) that effectively combines CLIP with a deeper exploitation of the
Hierarchical class structure via Graph representation learning. We explore
constructing the class hierarchy into a graph, with its nodes representing the
textual or image features of each category. After passing through a graph
encoder, the textual features incorporate hierarchical structure information,
while the image features emphasize class-aware features derived from
prototypes through the attention mechanism. Our approach demonstrates
significant improvements on both generic and fine-grained visual recognition
benchmarks. Our codes are fully available at https://github.com/richard-peng-
xia/HGCLIP.

##  1 Introduction

(a)

(b)

Figure 1: An illustration of the graph representation based on class
hierarchy. (a) The class hierarchy is presented in a tree structure. (b) The
hierarchical structure is constructed into a graph, with nodes representing
the textual/image features of each category. The graph is fed into a graph
encoder, where the nodes update their parameters by aggregating the messages
from their neighboring nodes. Thus, the class features are fused with
hierarchical structural information via graph representation learning.

Hierarchical image classification [38, 13] aims to enhance classification
accuracy by identifying objects at various levels of granularity and capturing
subtle relationships among them. Specifically, all the classes are organized
into a multi-granularity taxonomic hierarchy (see in Figure 1(a)), where the
top-level nodes represent broader categories (“Mammal”), while the lower-level
nodes encompass finer-grained subcategories (“Dog”). The inherently
hierarchical nature of the task compounds its complexity, as models must
exhibit a keen understanding of semantic hierarchies, balancing the trade-off
between capturing fine-grained details for subclasses while maintaining a
broad understanding of superclasses [4]. Previous works [3, 13] mainly focus
on enhancing image features according to the hierarchy of multiple branch
outputs. These uni-modal methods only focus on the image modality, leading to
certain limitations in complex scenarios, such as the inability to effectively
utilize the textual descriptions of hierarchical labels and adapt to new
classes or datasets. Therefore, leveraging multi-modal models (e.g., VLMs) to
address hierarchical image classification presents stronger potential,
offering richer information and greater scalability.

Given the powerful generalization capabilities of VLMs [36, 19, 52]
demonstrated on downstream tasks, harnessing their capabilities to address
hierarchical image classification tasks presents a highly valuable
exploration. These models are pre-trained on large-scale text-image pairs to
align features from the image and text modalities in a shared latent embedding
space. The predicted probabilities are obtained by calculating the similarity
between image features and text features.

Recently, some works have explored improving accuracy based on VLMs via class
hierarchies. Specifically, CHiLS [32] employs hierarchical mappings to
transform each class into a list of subcategories. However, this approach has
significant drawbacks when applied to fine-grained datasets, as the
subcategories of these labels tend to be specialized and rare, resulting in an
overly detailed and contextually sparse representation. Utilizing these
specific labels as prompts may overwhelm the model, lacking broader contextual
relevance. Hierarchy-CLIP [11] proposes a label augmentation method that
leverages the WordNet [9] label hierarchy, enriching each class with its
parent and child classes. This method aims to provide a richer semantic
expansion of class descriptions. It enhances surface-level semantic
associations rather than delving into the deeper and more structured
connections inherent in a hierarchical structure. This limitation becomes
apparent in scenarios requiring classification across multiple hierarchical
levels, where a nuanced understanding of these relationships is crucial.
Moreover, these methods are both training-free. While this offers the
advantage of simplicity and direct application, it lacks the capacity for
further model adaptation to specific datasets. Additionally, these methods do
not fully exploit the potential of VLMs to adapt to the diverse and complex
nature of hierarchical understanding.

Hence, the limitations of these approaches give rise to a new question: How
can models leverage the class hierarchy thoroughly to simultaneously improve
the prediction accuracy of categories at different semantic granularity
levels?

To address this issue, we first introduce prompt learning [55, 54, 22] as an
efficient method to adapt VLMs to downstream tasks. HGCLIP introduces prompt
tokens within the multi-modal branches of CLIP to facilitate the learning of
hierarchical contextual representations. More importantly, as demonstrated in
Figure 1(b), HGCLIP explores the integration of CLIP with graph
representations for hierarchical image classification. Specifically,
hierarchical relationships are modeled as a graph, given that they inherently
form a tree-like structure. Based on this graph, we employ a graph encoder
[24, 15, 43] to encode text features, enabling them to incorporate
hierarchical structural information. Moreover, since image features represent
features of individual patches/pixels rather than categories, we utilize
prototype learning to represent image features for each category. Similarly, a
graph encoder is leveraged to allow the prototypes to learn hierarchical
relationships, and subsequently utilize the attention mechanism to enable the
spatial feature map of images to focus more on the class-aware features
derived from prototypes. On hierarchical image classification, HGCLIP
outperforms existing CLIP-based approaches across both generic and fine-
grained datasets. In scenarios where hierarchical labels are unavailable,
HGCLIP also improves accuracy when utilizing class hierarchies queried by
ChatGPT [33]. Further, HGCLIP demonstrates favorable generalization ability
and robustness in domain generalization and subpopulation shift settings,
resulting in consistent improvements over existing methods. To sum up, the
main contributions of this work include:

  * •

We propose HGCLIP, a SoTA method in hierarchical image classification for
adaptation of CLIP.

  * •

To better utilize label hierarchies, we explore the graph representations to
incorporate hierarchical structural information into vision-language feature
representations for effective hierarchical understanding.

  * •

Our approach exhibits new state-of-the-art (SoTA) performance across eight
hierarchical image classification benchmarks and performs commendably on eight
extra datasets with distribution shifts.

##  2 Related Work

Figure 2: The pipeline of HGCLIP for adapting CLIP to hierarchical image
classification. We introduce multi-modal hierarchical prompt to learn
contextual representations. Then we construct the label hierarchy into a
graph, with its nodes representing the textual or image features of each
class. Features integrate hierarchical structure information through message
passing in the graph encoder. Textual features directly combine hierarchical
representations, while image features focus on class-aware prototypes through
the attention mechanism.

Prompt Learning in Vision-Language Models: VLMs leverage information from both
image and text modalities to encode multimodal representations. VLMs, _e.g._ ,
CLIP [36], ALIGN [19], and LiT [52] are pre-trained on large-scale image-text
pairs and have demonstrated remarkable representation abilities on various
downstream tasks [10, 57, 7]. However, efficiently adapting them to downstream
tasks is still a major challenge. Prompt learning [28, 27], as a parameter-
efficient technique, is well-suited for utilizing the representation capacity
of pre-trained VLMs to boost performance, instead of the resource-intensive
process of full fine-tuning. Many works [55, 22, 46, 40, 56, 14] have
demonstrated powerful performance on specific downstream tasks by combining
VLMs and prompt tuning.
Hierarchical Image Classification: Hierarchical image classification [38, 13]
aims to categorize images into a hierarchical structure that organizes classes
into a tree-like taxonomy. It acknowledges the inherent hierarchical nature of
visual concepts, allowing for more nuanced and contextually rich image
categorization. Prior research has explored various methodologies, including
model architectures tailored for hierarchical classification [13, 3, 4], and
exploiting the relationship of the categories in the hierarchy [29].
Furthermore, the development of hierarchical classification has spurred some
works [21, 49] that harness the class hierarchy across diverse domains, as
these models tend to focus more on fine-grained and semantically relevant
features. Recently, hierarchical labels are integrated with VLMs [32, 11].
Nonetheless, these methods roughly overlook the hierarchical relationships
among labels. Our work comprehensively leverages the hierarchical
relationships among labels, resulting in performance improvements on both
generic and fine-grained datasets.
Graph Representation Learning: Modern graph analysis methods rely on graph
representation learning, encompassing graph embedding, graph neural networks
(GNNs), and transformers. Early graph embedding techniques [35, 41, 12]
typically map nodes into a low-dimensional space, capturing structural
information like proximity between nodes [2]. Recently, GNNs [24, 15, 42, 47]
have become the mainstream technique in graph representation learning. They
rely on a message-passing framework where each node refines its representation
by recursively aggregating messages from its neighbors [45]. Moreover, some
recent approaches have also explored transformer-based architectures [51, 18,
50]. Hierarchical graph representations are employed in this work to enrich
multi-modal features, thus improving the model performance and generalization.

##  3 Preliminaries

In this work, our goal is to learn hierarchical textual and visual knowledge
via graph encoder based on CLIP. We introduce related concepts and definitions
in the following.

###  3.1 Revisiting CLIP

We denote the CLIP image and text encoder as $\mathcal{I}(\cdot)$ and
$\mathcal{T}(\cdot)$. The dataset contains $K$ categories, i.e.,
$\\{C_{1},\cdots,C_{K}\\}$. CLIP leverages a structured approach by inserting
all category names into a predefined textual template represented by the
[CLASS] token, e.g., creating expressions like “a photo of a [CLASS].”. This
results in the generation of textual inputs denoted as $T_{K}$. Subsequently,
textual features, represented as $\mathbf{F}_{t}\in\mathbb{R}^{K\times D}$,
are extracted. Each input image $I$ is divided into $M$ fixed-sized patches,
and each patch is embedded into $D$-dimensional latent space. Then CLIP
derives its spatial feature map $\mathbf{F}_{s}\in\mathbb{R}^{H\times W\times
D}$ and computes the global visual representations
$\mathbf{f}_{v}\in\mathbb{R}^{1\times D}$ through pooling operations, where
$H$ and $W$ denote the height and width of the feature map. The integration of
features from both encoders is achieved through cosine similarity measures,
ultimately yielding classification $logits\in\mathbb{R}^{1\times K}$. This
comprehensive process can be summarized as follows

$\mathbf{F}_{t}=\mathcal{T}(T_{K}),$ (1)
$\mathbf{f}_{v}=\textsc{Pooling}(\mathbf{F}_{s}),\quad\textbf{F}_{s}=\mathcal{I}(I),$
(2) $\textit{logits}=\mathbf{f}_{v}{\mathbf{F}_{t}}^{T}.$ (3)

The matrix multiplication operation between $\mathbf{f}_{v}$ and
$\mathbf{F}_{t}$ is equivalent to calculating cosine similarities, assumed to
be $L_{2}$-normalized features. logits signifies the computed probabilities
for all $K$ categories, and CLIP identifies the category with the maximum
output probability $\mathrm{argmax}_{C_{K}}(\textit{logits})$ as its final
prediction.

Figure 3: t-SNE plots of image embeddings in previous SoTA prompting method
CoCoOp, MaPLe, and HGCLIP on two datasets with distinct semantic
granularities. HGCLIP shows better separability in both fine-grained and
coarse-grained levels.

###  3.2 Graph Encoder

Graph. A graph is represented as $G=(V,E)$, with $V$ denoting the set of nodes
and $E$ the set of edges. Equivalently, the graph can be represented by an
adjacency matrix $A$, such as $A_{ij}=1$, if $(v_{i},v_{j})\in E$, for any
$v_{i},v_{j}\in V$. Graph Encoder. GNNs are popular choices of graph encoder,
most of which employ a message-passing mechanism [45]. Specifically, each node
in the graph aggregates messages (i.e., input features or embeddings) from its
neighboring nodes to update its own embedding. Multiple layers of neighborhood
aggregation can be stacked, facilitating recursive message passing across the
graph. Formally, in the $l$-th GNN layer, the embedding of node $v$, denoted
by $\mathbf{f}^{l}_{v}$, is calculated based on the embeddings in the previous
layer, as follows

$\mathbf{f}^{l}_{v}=\textsc{Aggr}(\mathbf{f}^{l-1}_{v},\\{\mathbf{f}^{l-1}_{u}:u\in\mathcal{N}_{v}\\};\theta^{l}),$
(4)

where $\mathcal{N}_{v}$ is the set of neighboring nodes of $v$, $\theta^{l}$
is the learnable GNN parameters in layer $l$. $\textsc{Aggr}(\cdot)$ is the
neighborhood aggregation function and can take various forms, ranging from the
simple mean pooling [24, 15] to advanced neural networks such as neural
attention [42] or multi-layer perceptrons [47]. Note that the initial node
embedding $\mathbf{f}^{0}_{v}$ is simply given by the input feature. We
abstract the multi-layer encoding process as

$\displaystyle\mathbf{f}_{v}=\textsc{GraphEncoder}(\mathbf{f}_{v}^{0},\mathcal{N}_{v};\Theta),$
(5)

where $\Theta=(\theta^{1},\ldots,\theta^{L})$ is the collection of weights
across the layers. Note that graph embedding methods [35, 41, 12] and graph
transformers [51, 18, 50] could also serves as GraphEncoder.

##  4 Methodology

In this section, as shown in Figure 2, we present our proposed method, i.e.,
HGCLIP for adapting pre-trained VLMs for hierarchical understanding. Our
approach aims to enhance the capacity for understanding multiple semantic
levels. Most prior approaches focus on single-label classification, whereas
hierarchical classification necessitates that the model attends to features
relevant to multi-granularity hierarchies. To this end, HGCLIP entails: a)
introducing learnable prompt tokens within multiple transformer blocks in both
the visual and textual branches to learn hierarchical contextual
representations; b) employing a graph encoder to encode textual features,
integrating them with hierarchical structural information; c) utilizing
prototype learning to represent image features of each category and similarly
modeling them utilizing a graph encoder, thereafter employing the attention
mechanism to enable the spatial feature map of images to focus more on class-
aware and hierarchy-guided image features.

###  4.1 Hierarchy Setting

The ground truth class hierarchy currently available in a dataset is usually
obtained by querying a WordNet [9]-like dictionary, but in the real world, our
dataset may have no available class hierarchy. In this case, we turn to LLMs,
i.e., ChatGPT [33] (GPT-3.5-Turbo), to approximate the hierarchy diagram.
Specifically, given some label set size $K$, semantic granularity levels $h$,
class names, and optional context, we query ChatGPT with the prompt:

Generate $h$-tier hierarchical labels for the following $K$ categories:
$\\{C_{1},\cdots,C_{K}\\}$.

###  4.2 Multi-modal Hierarchical Prompt

In order to comprehensively and efficiently leverage the capabilities of
pretrained VLMs, we explore the potential of multi-modal prompt, encompassing
both textual and visual prompt. As highlighted in [22], the acquisition of
prompt at deeper transformer layers is crucial, as it progressively models
hierarchical feature representations. Learnable tokens are introduced at
multiple transformer blocks of both textual and visual branches of VLMs, given
as textual prompt
$\mathbf{P}^{T}=\\{\mathbf{p}^{T}_{1},\cdots,\mathbf{p}^{T}_{t}\\}$ and visual
prompt $\mathbf{P}^{V}=\\{\mathbf{p}^{V}_{1},\cdots,\mathbf{p}^{V}_{v}\\}$,
respectively. Therefore, the image encoder processes the input tokens added
visual prompt $\mathbf{P}^{V}$ to generate prompted spatial feature map
represented as $\tilde{\mathbf{F}_{s}}\in\mathbb{R}^{(HW+v)\times D}$ and
prompted global visual representations
$\tilde{\mathbf{f}_{v}}\in\mathbb{R}^{1\times D}$. Similarly, textual prompt
$\mathbf{P}^{T}$ are incorporated into the input tokens for encoding, and
textual features are obtained as $\tilde{\mathbf{F}_{t}}\in\mathbb{R}^{K\times
D}$. These hierarchical prompt tokens leverage the knowledge encoding
capabilities of VLMs to effectively learn task-relevant contextual
representations across different semantic levels.

###  4.3 Delving into Graph Representations

The hierarchical structure among labels naturally forms a tree structure,
hence we leverage graph representations to model the hierarchy and integrate
it into multi-modal features. In Figure 3, we visualize and compare the image
embeddings of HGCLIP with those of previous SoTA CoCoOp and MaPLe. It is worth
noting that the image embeddings of CLIP, CoOp, CoCoOp, and KgCoOp would be
identical, as they do not learn prompts in the visual branch. The
visualization reveals that the image embeddings of HGCLIP are more separable,
indicating that incorporating hierarchical information can better adapt CLIP.
Encoding Text: Clearly, textual features
$\tilde{\mathbf{F}_{t}}=\\{\tilde{\mathbf{f}_{n}^{t}}\\}_{n=1}^{K}$ can be
directly employed as input for a graph encoder, as they possess corresponding
$D$-dimensional textual features for each category. The class hierarchy is
constructed into a graph, where vertices and edges represent individual
classes and pairs of classes with hierarchical relationships, respectively. As
a result, each node $n$ of the text-attributed graph is associated with text
features of the corresponding category $\tilde{\mathbf{f}_{n}^{t}}$. The graph
encoder approaches node classification by using the structural interactions
between nodes. The textual features
$\hat{\mathbf{F}_{t}}=\\{\hat{\mathbf{f}_{n}^{t}}\\}_{n=1}^{K}$ integrating
hierarchical information are encoded as follows:

$\\{\hat{\mathbf{f}_{n}^{t}}\\}_{n=1}^{K}=\textsc{GraphEncoder}(\tilde{\mathbf{f}_{n}^{t}},\mathcal{N}_{n};\Theta_{t}),$
(6)

where $\Theta_{t}$ denotes the parameters of the graph encoder for textual
modality, $\mathcal{N}_{n}$ denotes the neighbor nodes of $n$.
Encoding Image: In contrast to textual features, the spatial feature map
represents the features of each patch, and the global visual representations
characterize the image holistically, rather than representing features for
each category. Therefore, the image features of each image cannot be directly
input into the graph encoder.

To address this issue, as shown in Figure 4, we first leverage prototype
learning to represent the image features for each category. The global
features
$\mathbf{F}_{v}^{*}=\\{\mathbf{f}_{n}^{v*}\\}_{n=1}^{K}\in\mathbb{R}^{K\times
1\times D}$ of all images $\\{I_{n}\\}_{n=1}^{K}$ (only in the training set)
belonging to each class are extracted as prototypes for all categories. These
prototypes can then be utilized by the graph encoder to be encoded. The
procedure is as follows

$\mathbf{F}_{v}^{*}=\textsc{Pooling}(\mathbf{F}_{s}^{*}),\quad\mathbf{F}_{s}^{*}=\mathcal{I}(I_{K}).$
(7)

In the image-attributed graph, each node $n$ is associated with image features
$\\{\mathbf{f}_{n}^{v*}\\}_{n=1}^{K}$, while the rest is consistent with the
text-attributed graph. Similarly, the image features $\hat{\mathbf{F}_{v*}}$
are encoded as follows

$\displaystyle\\{\hat{\mathbf{f}_{v*}^{n}}\\}_{n=1}^{K}=\textsc{GraphEncoder}(\mathbf{f}_{v*},\mathcal{N}_{n};\Theta_{v}),$
(8)

where $\Theta_{v}$ denotes the parameters of the graph encoder for visual
modality. After the visual graph encoder effectively leverages structural
knowledge, we then employ the attention mechanism to obtain the attention
weights of visual features $\tilde{\mathbf{F}_{s}}$ with respect to the
prototypes $\hat{\mathbf{F}_{v*}}$. The calculation of attention weights is as
follows

$\psi=\tilde{\mathbf{F}_{s}}{{\hat{\mathbf{F}_{v*}}}^{T}}\in\mathbb{R}^{(HW+v)\times
K},$ (9)

where $\psi$ denotes the attention map. Each element of $\psi$ represents the
attention weight, namely, the feature similarity between a class prototype and
one image pixel/site. Based on $\psi$, we update the spatial feature map as
follows

$\hat{\mathbf{F}_{s}}=\mathrm{SoftMax}(\psi/\alpha)\hat{\mathbf{F}_{v*}},$
(10)

where $\alpha$ modulates the attention magnitude. Weighted by the attention
scores representing similarity, the image features incorporate structured
information from the prototypes. As the prototypes $\hat{\mathbf{F}_{v*}}$
encode $K$-category visual knowledge, the signals of classes appearing in the
image would be more notable. Meanwhile, the spatial feature map provides
pixel-level fine-grained information for the interaction, contributing to the
thorough integration of class-aware features from the prototypes into the
image features.
Classification Logits: Finally, we obtain the attention-interacted global
visual feature by pooling and output the classification logits as

Figure 4: Semantic prototypes are constructed to guide the learning of
hierarchical semantics of images.

$\hat{\mathbf{f}_{v}}=\textsc{Pooling}(\hat{\mathbf{F}_{s}})\in\mathbb{R}^{1\times
D},$ (11)
$\textit{logits}=\lambda_{1}\cdot\tilde{\mathbf{f}_{v}}{\hat{\mathbf{F}_{t}}}^{T}+\lambda_{2}\cdot\hat{\mathbf{f}_{v}}{\hat{\mathbf{F}_{t}}}^{T},$
(12)

where $\lambda_{1}$ and $\lambda_{2}$ denote hyper-parameters to control the
weight assigned to the logits that incorporate structured image features.

For hierarchical image classification, the model is required to simultaneously
predict several labels at different granularities. Consequently, it is
necessary to partition the predicted logits into their respective hierarchical
categories $\textit{logits}_{i}$, with each level corresponding to the ground
truth labels $GT_{i}$, where $i=1,\cdots,h$. The overall loss function can be
defined as follows

$\mathcal{L}=\sum_{i=1}^{h}w_{i}\cdot\mathcal{L}_{CE}({GT}_{i},\textit{logits}_{i}),$
(13)

where $w_{i}$ denotes the weights for learning features at different
hierarchical levels and $\mathcal{L}_{CE}(\cdot,\cdot)$ represents a cross-
entropy loss. A higher $w_{i}$ prioritizes the learning of features at the
$i$-th level, and vice versa.

##  5 Experiment

###  5.1 Benchmark Setting

Hierarchical Image Classification: We consider both generic (CIFAR-100 [26]
and Caltech-101 [8]) and fine-grained (FGVC-Aircraft [30], StanfordCars [25],
Food-101 [1], Fruits-360 [31], OxfordPets-37 [34] and ETHEC [6]) visual
classification tasks to demonstrate our method under general situations of
data diversity, where the label hierarchical levels range from two to four.
Domain Generalization: We evaluate the robustness of our approach on out-of-
distribution datasets. The source distributions correspond to the original
ImageNet [5]. The task is to classify images from the target datasets
(ImageNetV2 [37], ImageNet-Sketch [44], ImageNet-A [17] and ImageNet-R [16]),
which consist of images that contain various types of domain shifts. It is
important to note that due to the inconsistent semantic granularity at each
hierarchical level in the original ImageNet, we only select all the categories
from ImageNet-A and ImageNet-R (a total of 314 categories) for our
experiments, opting only for categories from two adjacent hierarchical levels.
Subpopulation Shift: We also examine HGCLIP’s robustness to subpopulation
shift within a dataset. The source and target distributions, though
encompassing identical class categories, feature distinct subpopulations
within those classes. Our empirical investigations were executed on the four
BREEDS [39] ImageNet subsets: Living17, Nonliving26, Entity13, and Entity30.
Implementation Details: We use top-1 accuracy to evaluate the prediction
performance. We adopt CLIP ViT-B/16 as the visual encoder and use the
corresponding CLIP Transformer as the text encoder, where $D=512$. We set
$\lambda_{1}$ = 1 and $\lambda_{2}$ = 0.2 to weight the proportion of
hierarchical structural information. For hierarchical classification, we use
deep prompting with $v=t=4$ in the first 9 transformer layers and train the
models for 50 epochs. For distribution shift benchmarks, we use the same
number of prompt tokens in the first 3 layers and train the models for 20
epochs. All models are trained with a batch size of 64 and a learning rate of
3e-4 via SGD optimizer, and decay by the cosine annealing rule during
training. All the experiments are implemented on PyTorch 1.13.0 using an
NVIDIA GeForce RTX 4090 GPU. For additional benchmark setting and
implementation details, see Appendix A.

###  5.2 Hierarchical Image Classification

Table 1 displays the performance of HGCLIP on 8 hierarchical classification
datasets, including 2 generic and 6 fine-grained. We compare its performance
with zero-shot CLIP and recent works on prompt learning. In the case of CLIP,
we utilize handcrafted specific prompts designed for each dataset. In
comparison with state-of-the-art MaPLe [22] and PromptSRC [23], HGCLIP
exhibits improved performance across all levels on all the datasets, with the
exception of a slight decline in performance on Fruits-360. With the
contribution of graph representations, as opposed to MaPLe and PromptSRC,
HGCLIP demonstrates superior generalization across multiple hierarchical
categories on all the datasets, achieving an absolute average gain of 2.1% and
6.3% respectively.

Dataset |  |  CLIP [36]
---
ICML’21
|  CoOp [55]
---
IJCV’22
|  CoCoOp [54]
---
CVPR’22
|  VPT [20]
---
ECCV’22
|  MaPLe [22]
---
CVPR’23
|  KgCoOp [48]
---
CVPR’23
|  PromptSRC [23]
---
ICCV’23
| HGCLIP
---
(Ours)
CIFAR-100* | $l_{1}$ | 43.22 | 83.76 | 82.60 | 88.75 | 90.67 | 78.65 | 88.18 | 91.87
$l_{2}$ | 66.57 | 76.81 | 75.73 | 83.94 | 85.81 | 73.49 | 82.24 | 86.55
Caltech-101 | $l_{1}$ | 58.47 | 96.19 | 96.95 | 96.62 | 98.06 | 93.50 | 95.70 | 98.50
$l_{2}$ | 69.01 | 95.12 | 94.18 | 95.06 | 97.38 | 93.56 | 95.57 | 97.51
$l_{3}$ | 84.56 | 95.88 | 95.85 | 96.12 | 96.88 | 94.81 | 95.51 | 97.03
FVGC- Aircraft* | $l_{1}$ | 31.08 | 54.30 | 54.80 | 56.81 | 70.79 | 53.31 | 55.99 | 79.24
$l_{2}$ | 35.49 | 51.59 | 50.42 | 53.00 | 68.87 | 50.38 | 50.20 | 70.70
$l_{3}$ | 24.69 | 37.74 | 36.10 | 35.00 | 52.58 | 35.56 | 34.21 | 61.33
Stanford Cars* | $l_{1}$ | 61.75 | 82.39 | 83.31 | 83.46 | 83.35 | 82.78 | 82.85 | 83.61
$l_{2}$ | 63.59 | 73.35 | 73.86 | 76.53 | 76.92 | 69.01 | 73.24 | 77.84
Food-101 | $l_{1}$ | 61.38 | 87.04 | 89.13 | 89.28 | 88.16 | 85.18 | 87.37 | 91.12
$l_{2}$ | 85.53 | 86.94 | 87.10 | 88.46 | 88.04 | 86.00 | 86.89 | 88.73
Fruits-360 | $l_{1}$ | 75.43 | 93.73 | 99.21 | 99.46 | 99.59 | 90.95 | 94.41 | 99.71
$l_{2}$ | 34.40 | 86.87 | 96.46 | 98.65 | 98.31 | 76.43 | 94.60 | 98.51
$l_{3}$ | 23.55 | 85.90 | 96.58 | 97.12 | 97.78 | 72.80 | 92.13 | 97.73
Oxford Pets-37 | $l_{1}$ | 99.97 | 99.86 | 99.86 | 99.92 | 99.91 | 99.89 | 99.89 | 99.92
$l_{2}$ | 88.14 | 91.94 | 91.97 | 91.81 | 92.14 | 91.81 | 92.19 | 92.31
ETHEC* | $l_{1}$ | 31.12 | 89.45 | 89.61 | 90.87 | 92.17 | 86.03 | 90.64 | 95.76
$l_{2}$ | 2.65 | 85.10 | 86.07 | 86.79 | 89.60 | 83.11 | 87.91 | 93.40
$l_{3}$ | 17.94 | 74.67 | 75.02 | 75.81 | 78.48 | 71.33 | 77.46 | 82.98
$l_{4}$ | 1.52 | 49.48 | 51.27 | 51.99 | 55.73 | 47.62 | 55.75 | 60.39
Table 1: Top-1 accuracy (%) comparison on hierarchical image classification of HGCLIP with previous CLIP-based prompt tuning methods. The best result is bold and the second best is underlined. * denotes that the dataset is with available class hierarchy, and hierarchies of others are queried through ChatGPT [33]. $l_{i}$ represents the classification accuracy at the $i$-th hierarchical level, where a smaller $i$ indicates a coarser granularity level, and vice versa. Method |  Acc. %
---|---
$l_{1}$ |  $l_{2}$
CLIP [36] ICML’21 |  43.22 |  66.57
Linear probe |  75.60 |  71.27
CLIP-Adapter [10] IJCV’23 |  83.91 |  77.03
Tip-Adapter [53] ECCV’22 |  84.57 |  81.42
CALIP [14] AAAI’23 |  77.51 |  74.28
HGCLIP (Ours) |  91.87 |  86.55
Table 2: Comparision with CLIP-based feature adaption methods. | Source | Target
---|---|---
Method | ImageNet* | -V2* | -Sketch* | -A | -R
CLIP | 86.11 | 80.19 | 72.13 | 46.11 | 70.85
CoOp | 85.44 | 79.71 | 70.85 | 45.63 | 70.91
CoCoOp | 85.03 | 79.49 | 72.96 | 46.99 | 72.47
MaPLe | 91.22 | 85.12 | 74.19 | 49.28 | 73.79
PromptSRC | 89.89 | 83.79 | 75.04 | 49.37 | 74.22
HGCLIP | 92.19 | 86.24 | 77.40 | 50.38 | 76.07
Table 3: Domain generalization. Comparison with CLIP-based methods on
robustness to domain shifts. * denotes that 314 categories out of 1,000
categories are selected for the dataset.

Module | CIFAR-100 | FVGC-Aircraft
---|---|---
TP | TG | VP | VG | $l_{1}$ | $l_{2}$ | $l_{1}$ | $l_{2}$ | $l_{3}$
✗ | ✗ | ✗ | ✗ | 43.22 | 66.57 | 31.08 | 35.49 | 24.69
✓ | ✗ | ✗ | ✗ | 84.21 | 77.22 | 54.96 | 52.67 | 38.72
✗ | ✗ | ✓ | ✗ | 84.21 | 77.22 | 56.81 | 53.00 | 35.00
✓ | ✓ | ✗ | ✗ | 87.42 | 81.24 | 61.56 | 57.90 | 42.83
✗ | ✗ | ✓ | ✓ | 87.18 | 80.87 | 61.52 | 58.13 | 43.17
✓ | ✗ | ✓ | ✗ | 90.67 | 85.81 | 70.79 | 68.87 | 52.58
✓ | ✗ | ✓ | ✓ | 91.28 | 86.04 | 74.61 | 69.27 | 55.66
✓ | ✓ | ✓ | ✗ | 91.43 | 85.96 | 75.37 | 69.28 | 57.50
✓ | ✓ | ✓ | ✓ | 91.87 | 86.55 | 79.24 | 70.70 | 61.33

TP and VP serve as textual and visual prompts. TG and VG denote graph encoder
for textual and visual modality respectively.

Table 4: Component Analysis of HGCLIP.

Moreover, in Table 2, we compare HGCLIP with prior feature adaption methods
based on CLIP. CLIP-Adapter [10] learns two residual-style adapters after
CLIP. Tip-Adapter [53] constructs a key-value cache model by extracting
features from few-shot data through CLIP, then views the cache model as a
well-performing initialization and fine-tunes the cache keys. CALIP [14] is
proposed to boost CLIP performance via a parameter-free attention module
between multi-modal representations. In comparison with these feature adaption
approaches, HGCLIP exhibits excellent feature representation capabilities,
with an accuracy on CIFAR-100 that is 8.7%, 6.2%, and 13.3% higher than
theirs, respectively.

###  5.3 Distribution Shifts

To validate the generalization and robustness of our proposed HGCLIP, we
perform experiments on the two benchmarks with distributional shifts, namely
domain shifts and subpopulation shifts. This allows us to comprehensively
assess the efficacy of HGCLIP in generalizing to out-of-distribution datasets.
Domain Generalization: Table 3 summarizes the results of HGCLIP and prior
approaches on out-of-distribution datasets. We verify the transferability of
models trained on ImageNet to various ImageNet variants with domain shifts. On
the target datasets, the performance of HGCLIP surpasses previous SoTA
methods. This achievement underscores the efficacy of hierarchical graph
representations to enhance multi-modal features. Such an integration improves
the generalization capabilities, enabling it to perform well across varying
domains. This indicates that HGCLIP not only captures the intricate
relationships within the data but also adapts effectively to new and unseen
domains.
Subpopulation Shift: We further evaluate the generalizability of HGCLIP on
four datasets from BREEDS (subsets of ImageNet) that exhibit subpopulation
shifts and provide available class hierarchies. Figure 5 depicts the
performance of HGCLIP and previous methods on this subpopulation shift
benchmark. The models are trained only on base classes and tested on novel
classes. The results suggest that HGCLIP possesses strong generalization
capabilities even when confronted with feature-level shifts, underscoring the
efficacy of hierarchical structure information in enhancing model
generalizability. This success demonstrates the robustness of HGCLIP in
handling variations within subpopulations, ensuring consistent accuracy and
reliability across diverse and shifting data landscapes.

Figure 5: Results for different methods on the four BREEDS datasets [39] to
measure robustness to subpopulation shift. Figure 6: Example decisions from
our model, MaPLe and CLIP.

###  5.4 Ablative Analysis

Components Analysis: HGCLIP primarily consists of multi-modal prompts and
graph encoders. In Table 4, we ablate on the performance of each module. The
prompts facilitate the model in learning hierarchical contextual features,
while the graph encoders effectively integrate hierarchical structure
information into the feature representations. This enables the model to
achieve impressive results across multiple granularities of hierarchical
categories.
Calculating Logits for Hierarchical Classification: In this context, there are
two approaches for calculating hierarchical logits. The first approach
involves calculating the logits for each category separately based on the
hierarchical relationships. The second approach is bottom-up: logits for only
the finest-grained categories are initially computed. Logits for the next
higher level of categories are then derived by summing up the logits of their
corresponding subordinate categories. We compare the performance of these two
computation methods in zero-shot experiments and find that while maintaining
consistent prompt templates, different hierarchical computation methods
yielded varying results at the coarse level, with the former reaching 43.22%
and the latter only achieving 6.13% on CIFAR-100. Refer to Appendix B for
additional illustration.
Noisy Hierarchies Queried by Large Language Models: It is important to note
that LLMs may output sub-optimal hierarchical labels. LLMs produce
inconsistent hierarchical labels based on a set of input category names or
generate hierarchical labels of different levels, leading to certain biases in
the model performance. However, even when utilizing the noisy hierarchy,
HGCLIP still enhances accuracy within the original categories in the dataset.
For more analysis of sub-optimal hierarchical labels, see Appendix C.

###  5.5 Qualitative Analysis

Figure 6 presents illustrative cases showcasing the predicted probabilities of
the models at different semantic granularity levels. CLIP shows
inconsistencies in classification results at different levels, indicating that
CLIP does not grasp the semantic relationship between different hierarchical
levels. MaPLe improves prediction accuracy via learning hierarchical feature
representation. However, it still displays inconsistencies when predicting
classifications across different levels. Our method largely mitigates this
issue, leveraging hierarchical graph representation to bolster the learning of
inter-level class features.

###  5.6 Graph Encoder Analysis

We further conduct experiments to analyze the impact of various graph
encoders. We apply three of the most commonly used graph learning models: GCN
[24], GAT [43], and GraphSAGE [15] to HGCLIP, the results are illustrated in
Figure 7. First, for both hierarchical levels, GAT consistently exhibits
superior performance, particularly at the fine level, where GAT markedly
surpasses the other two graph encoders. Second, with the increase of layer
depth of the graph encoder, the accuracy initially rises. Upon reaching a peak
(3 layers), the accuracy begins to gradually decline with further increase in
layer depth. Therefore, we use a 3-layer GAT as graph encoder in our
experiments.

Figure 7: The impact of different GNN variants and the number of layers on the
performance of hierarchical classification. We report accuracy results on two
hierarchical levels.

##  6 Conclusion

In this work, we propose a novel view that combines VLMs with graph
representations. Deep prompts are incorporated into the multi-modal branches,
allowing VLMs to better learn hierarchical representations. The graph-based
hierarchical relationships are encoded into features, strengthening the
connection of features across multiple granularities. When integrating image
features with graph representation, given that image features are
pixel/region-level, prototype learning is employed for class-level image
features, which are then fused with the original image features through the
attention mechanism. HGCLIP achieves SoTA results on several hierarchical
image classification benchmarks and demonstrates robustness and generalization
on multiple datasets with distribution shifts.

## References

  * Bossard et al. [2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_ , pages 446–461. Springer, 2014.
  * Cai et al. [2018] Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang.  A comprehensive survey of graph embedding: Problems, techniques, and applications.  _IEEE Transactions on Knowledge and Data Engineering_ , 30(9):1616–1637, 2018.
  * Chang et al. [2021] Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo.  Your” flamingo” is my” bird”: fine-grained, or not.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11476–11485, 2021.
  * Chen et al. [2018] Tianshui Chen, Wenxi Wu, Yuefang Gao, Le Dong, Xiaonan Luo, and Liang Lin.  Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding.  In _Proceedings of the 26th ACM international conference on Multimedia_ , pages 2023–2031, 2018.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dhall et al. [2020] Ankit Dhall, Anastasia Makarova, Octavian Ganea, Dario Pavllo, Michael Greeff, and Andreas Krause.  Hierarchical image classification using entailment cone embeddings.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 836–837, 2020.
  * Ding et al. [2022] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11583–11592, 2022.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _2004 conference on computer vision and pattern recognition workshop_ , pages 178–178. IEEE, 2004.
  * Fellbaum [2010] Christiane Fellbaum.  Wordnet.  In _Theory and applications of ontology: computer applications_ , pages 231–243. Springer, 2010.
  * Gao et al. [2023] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _International Journal of Computer Vision_ , pages 1–15, 2023.
  * Ge et al. [2023] Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao.  Improving zero-shot generalization and robustness of multi-modal models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11093–11101, 2023.
  * Grover and Leskovec [2016] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In _KDD_ , pages 855–864, 2016.
  * Guo et al. [2018] Yanming Guo, Yu Liu, Erwin M Bakker, Yuanhao Guo, and Michael S Lew.  Cnn-rnn: a large-scale hierarchical image classification framework.  _Multimedia tools and applications_ , 77(8):10251–10271, 2018.
  * Guo et al. [2023] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui.  Calip: Zero-shot enhancement of clip with parameter-free attention.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 746–754, 2023.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30, 2017.
  * Hendrycks et al. [2021a] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8340–8349, 2021a.
  * Hendrycks et al. [2021b] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15262–15271, 2021b.
  * Hu et al. [2020] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun.  Heterogeneous graph transformer.  In _Proceedings of the web conference 2020_ , pages 2704–2710, 2020.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _European Conference on Computer Vision_ , pages 709–727. Springer, 2022.
  * Ju et al. [2023] Lie Ju, Zhen Yu, Lin Wang, Xin Zhao, Xin Wang, Paul Bonnington, and Zongyuan Ge.  Hierarchical knowledge guided learning for real-world retinal disease recognition.  _IEEE Transactions on Medical Imaging_ , 2023.
  * Khattak et al. [2023a] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19113–19122, 2023a.
  * Khattak et al. [2023b] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan.  Self-regulating prompts: Foundational model adaptation without forgetting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15190–15200, 2023b.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations_ , 2016.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE international conference on computer vision workshops_ , pages 554–561, 2013.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, 2021.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, 2021.
  * Liu et al. [2022] Yang Liu, Lei Zhou, Pengcheng Zhang, Xiao Bai, Lin Gu, Xiaohan Yu, Jun Zhou, and Edwin R Hancock.  Where to focus: Investigating hierarchical attention relationship for fine-grained visual classification.  In _European Conference on Computer Vision_ , pages 57–73. Springer, 2022.
  * Maji et al. [2013] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Mureşan and Oltean [2017] Horea Mureşan and Mihai Oltean.  Fruit recognition from images using deep learning.  _arXiv preprint arXiv:1712.00580_ , 2017.
  * Novack et al. [2023] Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg.  Chils: Zero-shot image classification with hierarchical label sets.  In _International Conference on Machine Learning_ , pages 26342–26362. PMLR, 2023.
  * OpenAI [2023] OpenAI, 2023.  https://chat.openai.com.
  * Parkhi et al. [2012] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In _2012 IEEE conference on computer vision and pattern recognition_ , pages 3498–3505. IEEE, 2012.
  * Perozzi et al. [2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: Online learning of social representations.  In _KDD_ , pages 701–710, 2014.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Salakhutdinov et al. [2011] Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum.  Learning to share visual appearance for multiclass object detection.  In _CVPR 2011_ , pages 1481–1488. IEEE, 2011.
  * Santurkar et al. [2021] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry.  Breeds: Benchmarks for subpopulation shift.  In _International Conference on Learning Representations_ , 2021.
  * Shu et al. [2022] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao.  Test-time prompt tuning for zero-shot generalization in vision-language models.  _Advances in Neural Information Processing Systems_ , 35:14274–14289, 2022.
  * Tang et al. [2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  Line: Large-scale information network embedding.  In _WWW_ , pages 1067–1077, 2015.
  * Veličković et al. [2018a] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In _ICLR_ , 2018a.
  * Veličković et al. [2018b] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In _International Conference on Learning Representations_ , 2018b.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wu et al. [2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  _IEEE transactions on neural networks and learning systems_ , 32(1):4–24, 2020.
  * Xia et al. [2023] Peng Xia, Di Xu, Lie Ju, Ming Hu, Jun Chen, and Zongyuan Ge.  Lmpt: Prompt tuning with class-specific embedding loss for long-tailed multi-label visual recognition.  _arXiv preprint arXiv:2305.04536_ , 2023.
  * Xu et al. [2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  In _ICLR_ , 2019.
  * Yao et al. [2023] Hantao Yao, Rui Zhang, and Changsheng Xu.  Visual-language prompt tuning with knowledge-guided context optimization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6757–6767, 2023.
  * Yi et al. [2022] Kai Yi, Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny.  Exploring hierarchical graph representation for large-scale zero-shot image classification.  In _European Conference on Computer Vision_ , pages 116–132. Springer, 2022.
  * Ying et al. [2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu.  Do transformers really perform badly for graph representation?  _Advances in Neural Information Processing Systems_ , 34:28877–28888, 2021.
  * Yun et al. [2019] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim.  Graph transformer networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Zhai et al. [2022] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18123–18133, 2022.
  * Zhang et al. [2022] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free adaption of clip for few-shot classification.  In _European Conference on Computer Vision_ , pages 493–510. Springer, 2022.
  * Zhou et al. [2022a] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16816–16825, 2022a.
  * Zhou et al. [2022b] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  _International Journal of Computer Vision_ , 130(9):2337–2348, 2022b.
  * Zhou et al. [2023] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu.  Zegclip: Towards adapting clip for zero-shot semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11175–11185, 2023.
  * Zhu et al. [2022] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang.  Prompt-aligned gradient for prompt tuning.  _arXiv preprint arXiv:2205.14865_ , 2022.

Supplementary Material

Figure 8: Illustrations of GPT-generated hierarchical labels.

The following section contains supplemental information and encompasses more
implementation details, results comparison, and a thorough ablative analysis
of HGCLIP. The contents are organized in the following order.

  * •

Additional Implementation Details (Appendix A)

  * •

Additional Method Introduction (Appendix B)

  * •

Additional Ablative Analysis (Appendix C)

  * •

Additional Results Presentation (Appendix D)

##  Appendix A Implementation Details

Hierarchical classes number: Table 5 presents the number of categories at each
hierarchical level for all the datasets utilized in the experiments.
Fruits-360 [31]: From the perspective of class names, its original form is
somewhat incomplete, as some classes are distinguished only by numerical
indices (e.g., “Apple Golden 1” and “Apple Golden 2”). Therefore, we manually
rename the classes using the structure listed in Table 6, which leads to a
reduction in the number of subclasses from 131 to 113. This makes the
evaluation of model performance more reasonable and accurate.
BREEDS [39]: Since our method necessitates the use of class hierarchies, we
opt for the ’good’ split of subpopulations within a given superclass into
source and target domains, signifying that subpopulations in the source and
target domains share a common parent. In contrast, the other splits i.e.,
’bad’ and ’random’ would result in a greater degree of separation between the
subpopulations of the source and target domains in terms of hierarchical
structure distance. However, such an arrangement would not effectively
validate the efficacy of our method. Although these two ways of partitioning
the subpopulations would lead to stronger adversarial conditions, it would
cause the model to learn the wrong label hierarchy. Thus, they are not
considered in our analysis.

>
>     # Import Necessary Libraries
>     from robustness.tools.breeds_helpers
>     import BreedsDatasetGenerator
>
>     # Creating BREEDS datasets
>     DG = BreedsDatasetGenerator(info_dir)
>
>     # Example: Balanced dataset from level=2 with
>     # subclasses partitioned in "good"
>     ret = DG.get_superclasses(level=2,
>                           Nsubclasses=None,
>                           split="good",
>                           ancestor=None,
>                           balanced=True)
>     superclasses, subclass_split, label_map = ret
>

Textual diversity: Table 7 displays the hand-crafted prompts used for textual
prompt initialization.
Implementation Details: We use a publically available ViT-B/16 CLIP model with
$D$ = 512 and use a learning rate of 3e-4 which is fixed for all experiments
in all benchmarks. We set $\lambda_{1}$ = 1 and $\lambda_{2}$ = 0.2 to weight
the proportion of hierarchical structural information. The corresponding
hyperparameters are fixed across all datasets and benchmark. The respective
epochs are fixed across all datasets. For hierarchical classification, we use
deep prompting with $v=t=4$ in the first 9 transformer layers and train for 50
epochs. Since the classification of coarse-grained categories is noticeably
less challenging than that of fine-grained ones, we set $w_{h}$ in Eq. 13 to 2
and the remaining $\\{w_{i}\\}_{i=1}^{h-1}$ to 1, in order to improve the
model performance on fine-grained classification. For distribution shift
benchmarks, we use the same number of prompt tokens in the first 3 layers and
train for 20 epochs. All models are trained using SGD optimizer and utilize a
single NVIDIA GeForce RTX 4090 GPU.

Dataset |  Classes
---|---
CIFAR-100 [26] |  $l_{1}$ |  20
$l_{2}$ |  100
[1pt/1pt] Caltech-101 [8] |  $l_{1}$ |  11†
$l_{2}$ |  51†
$l_{3}$ |  101
[1pt/1pt] FVGC-Aircraft [30] |  $l_{1}$ |  30
$l_{2}$ |  70
$l_{3}$ |  100
[1pt/1pt] StanfordCars [25] |  $l_{1}$ |  9
$l_{2}$ |  196
[1pt/1pt] Food-101 [1] |  $l_{1}$ |  18†
$l_{2}$ |  101
[1pt/1pt] Fruits-360 [31] |  $l_{1}$ |  4†
$l_{2}$ |  69†
$l_{3}$ |  113
[1pt/1pt] OxfordPets-37 [34] |  $l_{1}$ |  2†
$l_{2}$ |  37
[1pt/1pt] ETHEC [6] |  $l_{1}$ |  6
$l_{2}$ |  21
$l_{3}$ |  135
$l_{4}$ |  561
ImageNet* [5] |  $l_{1}$ |  328
$l_{2}$ |  314
Living17 [39] |  $l_{1}$ |  17
$l_{2}$ |  34
[1pt/1pt] Nonliving26 [39] |  $l_{1}$ |  26
$l_{2}$ |  52
[1pt/1pt] Entity13 [39] |  $l_{1}$ |  13
$l_{2}$ |  130
[1pt/1pt] Entity30 [39] |  $l_{1}$ |  30
$l_{2}$ |  120
Table 5: The number of categories at each hierarchical level for all datasets. ${\dagger}$ indicates that the categories for that level are queried through ChatGPT. $l_{i}$ represents the number of classes at the $i$-th hierarchical level, where a smaller $i$ indicates a coarser granularity level, vice versa. * denotes that we only select all the categories from ImageNet-A and ImageNet-R (a total of 314 categories) for our experiments, opting only for categories from two adjacent hierarchical levels due to the inconsistent semantic granularity at each hierarchical level in the original ImageNet. Original Class |  Cleaned Subclass
---|---
Apple Golden 1 |  Apple Golden
Apple Golden 2 |  Apple Golden
Apple Golden 3 |  Apple Golden
Apple Red 1 |  Apple Red
Apple Red 2 |  Apple Red
Apple Red 3 |  Apple Red
Apple Red Yellow 1 |  Apple Red Yellow
Apple Red Yellow 2 |  Apple Red Yellow
Cantaloupe 1 |  Cantaloupe
Cantaloupe 2 |  Cantaloupe
Cherry 1 |  Cherry
Cherry 2 |  Cherry
Cucumber Ripe 2 |  Cucumber Ripe
Grape White 2 |  Grape White
Grape White 3 |  Grape White
Grape White 4 |  Grape White
Peach 2 |  Peach
Pear 2 |  Pear
Plum 2 |  Plum
Plum 3 |  Plum
Tomato 1 |  Tomato
Tomato 2 |  Tomato
Tomato 3 |  Tomato
Tomato 4 |  Tomato
Table 6: Mapping from original class names to new class names for Fruits-360. Dataset | Hand-crafted prompt
---|---
CIFAR-100 | a photo of a [CLASS].
Caltech-101 | a photo of a [CLASS].
FVGC-Aircraft | a photo of an aircraft [CLASS].
StanfordCars | a photo of a car [CLASS].
Food-101 | a photo of a food [CLASS].
Fruits-360 | a photo of fruits [CLASS].
OxfordPets-37 | a photo of pets [CLASS].
ETHEC | a photo of a butterfly [CLASS].
ImageNet | a photo of a [CLASS].
ImageNetV2 [37] | a photo of a [CLASS].
ImageNet-Sketch [44] | a photo of a [CLASS].
ImageNet-A [17] | a photo of a [CLASS].
ImageNet-R [16] | a photo of a [CLASS].
Living17 | a photo of a [CLASS].
Nonliving26 | a photo of a [CLASS].
Entity13 | a photo of a [CLASS].
Entity30 | a photo of a [CLASS].
Table 7: Hand-crafted prompt for different datasets.

##  Appendix B Method Introduction

Calculating Logits for Hierarchical Classification: The calculation of the
predictive probabilities for each hierarchical level primarily involves three
methods [6]. The first method, as depicted in Figure 9, describes a classifier
that functions on a per-level basis, designated as $h$ individual
$\\{K_{i}\\}_{i=1}^{h}$-way classifiers. The model utilizes information about
the label hierarchy by explicitly predicting a single label per level for a
given image. However, this computational approach cannot be applied to methods
based on CLIP. This is because VLMs determine the predicted class by
calculating the similarity between the features of the image and those of each
text prompt, and the model does not possess multiple linear layers to
correspond to the prediction results of each hierarchical level separately.
Second, Figure 10 illustrates a model configured as a hierarchy-agnostic
classifier. This particular multi-label classifier operates without
incorporating any knowledge of an explicit hierarchical structure within the
labels. Our experiments are based on this computational method to calculate
the predictive probabilities at each hierarchical level. Third, Figure 11
presents a model schematic for the marginalization approach, wherein, rather
than assigning a discrete label to each level, the model generates a
probabilistic distribution across the terminal nodes of the hierarchical
structure. The probability associated with non-terminal nodes is computed by
marginalizing across their immediate offspring. This marginalization method
not only captures the connections of the various nodes within the hierarchy
but also acknowledges the presence of $h$ levels within the label hierarchy.
However, employing this method to compute the predictive probabilities of
superclasses yields poor results. Detailed comparative results are presented
in Table 8.

Figure 9: Model schematic for the per-level classifier. Figure 10: Model schematic for the hierarchy-agnostic classifier. Figure 11: Model schematic for the marginalization method. Method | Prompt | CIFAR-100 | ETHEC
---|---|---|---
$l_{1}$ | $l_{2}$ | $l_{1}$ | $l_{2}$ | $l_{3}$ | $l_{4}$
Marginalization | a photo of a [CLASS](, a type of butterfly). | 6.13 | 66.61 | 10.76 | 1.65 | 0.1 | 1.5
80 Prompt Templates Ensemble | 6.35 | 66.97 | 10.76 | 1.65 | 0.1 | 1.31
a photo of a [CLASS${}_{l_{2}}$], which is a kind of [CLASS${}_{l_{1}}$]. | 5.57 | 67.99 | 10.76 | 1.65 | 0.1 | 1.42
Multi-Label | a photo of a [CLASS](, a type of butterfly). | 43.22 | 66.57 | 31.12 | 2.65 | 17.94 | 1.52
80 Prompt Templates Ensemble | 44.85 | 66.97 | 20.82 | 2.29 | 16.58 | 1.29
a photo of a [CLASS${}_{l_{1}}$], possibly a [CLASS${}_{l_{2}}$]. | 57.83 | 66.57 | 27.58 | 11.60 | 17.88 | 1.52
a photo of a [CLASS${}_{l_{2}}$], which is a kind of [CLASS${}_{l_{1}}$]. | 43.22 | 68.00 | 31.12 | 1.65 | 0.1 | 1.42
Table 8: Comparative results of the probability calculations for categories
arcoss different hierarchical levels using multiple prompt templates.

##  Appendix C Ablative Analysis

Hyperparameter Settings: In Figure 12, We conduct ablation on $\lambda_{1}$
and $\lambda_{2}$ hyper-parameters of logits with structured features for
CIFAR-100 dataset. Overall, an increase in $\lambda_{1}$ can enhance
performance across multiple hierarchical levels, indicating that the textual
structured information is rich and effective. On the other hand, with
variations in $\lambda_{2}$, the performance across various levels peaks
around $\lambda_{2}$ = 0.2. This reflects that structured image features
provide more effective information, which in turn improves model performance.
Analysis of Noisy Hierarchies Caused by LLMs: Taking Food-101 as an example,
we conduct two types of comparative experiments utilizing hierarchical labels
obtained from the LLMs (i.e., ChatGPT [33]). The first experiment involves
repeating the same input and testing the various hierarchical labels generated
by the LLMs. Despite LLMs generating suboptimal hierarchical labels, in Table
10, our method still enhances model performance based on these noisy
hierarchical labels. This demonstrates the strong robustness and
generalizability of HGCLIP. The second experiment involves generating
hierarchical labels of different depths and conducting experiments based on
these hierarchies. In Table 10, across varying numbers of levels in label
hierarchies, HGCLIP improves model performance, even though some categories in
the label hierarchies may be overly broad or detailed. Through the validation
of these two types of experiments, our approach effectively utilizes the label
hierarchy structure, thus improving hierarchical classification performance.

Figure 12: Ablation on $\lambda_{1}$ and $\lambda_{2}$ on CIFAR-100 across
coarse and fine level.

Method | Acc. %
---|---
$l_{1}$ | $l_{2}$
CLIP | 61.38 | 85.53
HGCLIP | 91.12 | 88.73
[1pt/1pt] CLIP | 59.22 | 85.53
HGCLIP | 90.67 | 87.20
Table 9: Performance based on hierarchical labels obtained from LLMs given the
same input.

Method | Acc. %
---|---
$l_{1}$ | $l_{2}$ | $l_{3}$
CLIP | 61.38 | 85.53 | -
HGCLIP | 91.12 | 88.73 | -
[1pt/1pt] CLIP | 29.49 | 54.54 | 85.53
HGCLIP | 90.06 | 93.35 | 88.56
Table 10: Comparative results based on hierarchical labels of different
numbers of levels queried by LLMs.

##  Appendix D Results Presentation

Detailed results on subpopulation shift benchmark: In Table 12, we present the
detailed results for the subpopulation shifts benchmark illustrated in Figure
5.
t-SNE visualization: In Figure 3, we present the visualization of image
embeddings, which demonstrates that our method effectively separates
categories across multiple granularities. However, it is hard to observe
whether the features of certain categories with hierarchical relationships are
separable. To intuitively understand the role played by the graph
representations leveraging class hierarchy, we visualize the features of one
single coarse category and its corresponding five fine-grained categories
randomly selected from CIFAR-100 in Figure 13. It is evident that the features
of the fine-grained categories from HGCLIP are more distinct compared to both
CLIP and MaPLe [22]. This can be attributed to the utilization of graph
representations that integrate hierarchical relationships between categories.
Similarly, the features at the coarse-grained level also benefit. CLIP does
not focus on the intra-class arrangement of samples within the feature space,
as long as these samples are proximate to each other and are separable from
samples of different categories. This can lead to a chaotic distribution of
subcategories within the same category region. In contrast, HGCLIP not only
retains the attribute features of the coarse categories but also reduces the
intra-class variance, achieving enhanced generalization.
Analysis with CHiLS [32] and Hierarchy-CLIP [11]: These two works are among
the most recent and earliest to combine label hierarchies with VLMs.
Specifically, CHiLS utilizes the subcategories of each label to enrich the
class name itself, which is effective in cases where datasets may contain rich
underlying structures but have information-poor class labels. However, this
approach experiences a notable performance decline when applied to some fine-
grained datasets. This is because the class labels in fine-grained datasets
are already very specific, and forcibly adding more specialized and specific
subcategories can overwhelm the model, leading to an inability to match images
with the correct text. Building upon this, Hierarchy-CLIP enriches class
labels to a greater extent by adding both the superclass and subclasses to the
class name after initially identifying the top-5 class candidates. This
effectively compensates for the shortcomings of CHiLS. Nevertheless, this
method is still training-free and cannot be maximally adapted to downstream
datasets. Fine-tuning methods based on CLIP, whether prompt tuning [54, 55,
22, 20] or feature adaptation [10, 53, 14], do not integrate label
hierarchies. Therefore, the introduction of HGCLIP effectively addresses these
limitations. It is trainable and can be easily transferred to datasets of
different semantic granularities (regardless of the availability of label
hierarchies), and it also enhances the classification accuracy of each
semantic hierarchical category in the dataset. In Table 11, we compare the
performance of our method with that of these two approaches. Regarding the
CHiLS experiment, as their paper indicates that it is tested only on the 20
classes of CIFAR-20 and 101 classes of Food-101, to maximize the
representation of its effect, we follow its setup and do not test it on the
remaining hierarchical categories. The results show an improvement on the
generic image dataset, but a decline on the fine-grained food dataset,
consistent with our previous analysis. In the Hierarchy-CLIP experiment, it
also exhibits commendable performance, showing improvements across various
hierarchical levels. This is largely attributed to its use of both
superclasses and subclasses of labels for expansion. Our method not only
enhances classification performance at each level but also can be conveniently
transferred to downstream datasets.

Figure 13: Learned embedding tSNE visualization. Method | CIFAR-100 | Food-101
---|---|---
$l_{1}$ | $l_{2}$ | $l_{1}$ | $l_{2}$
CLIP [36] ICML’21 | 43.22 | 66.57 | 61.38 | 85.53
CHiLS [32] ICML’23 | 52.58 | - | - | 83.18
Hierarchy-CLIP [11] CVPR’23 | 58.61 | 67.39 | 63.74 | 86.98
HGCLIP (Ours) | 91.87 | 86.55 | 91.12 | 88.73
Table 11: Performance analysis with CHiLS and Hierarchy-CLIP. Method | Entity13 | Entity30 | Living17 | NonLiving26
---|---|---|---|---
Base | Novel | Base | Novel | Base | Novel | Base | Novel
CLIP | 68.55 | 68.62 | 70.32 | 67.37 | 95.00 | 88.24 | 73.04 | 76.42
CoOp | 94.22 | 85.99 | 89.83 | 78.35 | 95.59 | 88.77 | 85.23 | 76.69
CoCoOp | 95.06 | 85.23 | 91.52 | 78.65 | 96.35 | 87.77 | 91.65 | 77.15
MaPLe | 97.03 | 84.95 | 93.75 | 78.63 | 97.47 | 87.24 | 93.62 | 74.85
PromptSRC | 93.95 | 85.59 | 89.30 | 79.35 | 95.53 | 89.00 | 86.85 | 79.77
HGCLIP | 97.91 | 86.10 | 93.99 | 79.67 | 98.12 | 88.93 | 94.62 | 79.86
Table 12: Detailed performance (%) of different methods on the four BREEDS
datasets [39] to measure robustness to subpopulation shift.