11institutetext: National Key Laboratory for Novel Software Technology,
Nanjing University, China 22institutetext: School of Artificial Intelligence,
Nanjing University, China 33institutetext: Ant Group 44institutetext:
Institute of AI Industry Research (AIR), Tsinghua University, China 44email:
<EMAIL_ADDRESS>
# The Devil is in the Few Shots: Iterative Visual Knowledge Completion for
Few-shot Learning

Yaohui Li∗ 1122 Qifeng Zhou∗ 1122 Haoxing Chen 33 Jianbing Zhang† 1122
Xinyu Dai 1122 Hao Zhou 44

###### Abstract

Contrastive Language-Image Pre-training (CLIP) has shown powerful zero-shot
learning performance. Few-shot learning aims to further enhance the transfer
capability of CLIP by giving few images in each class, aka “few shots”. Most
existing methods either implicitly learn from the few shots by incorporating
learnable prompts or adapters, or explicitly embed them in a cache model for
inference. However, the narrow distribution of few shots often contains
incomplete class information, leading to biased visual knowledge with high
risk of misclassification. To tackle this problem, recent methods propose to
supplement visual knowledge by generative models or extra databases, which can
be costly and time-consuming. In this paper, we propose an Iterative Visual
Knowledge CompLetion (KCL) method to complement visual knowledge by properly
taking advantages of unlabeled samples without access to any auxiliary or
synthetic data. Specifically, KCL first measures the similarities between
unlabeled samples and each category. Then, the samples with top confidence to
each category is selected and collected by a designed confidence criterion.
Finally, the collected samples are treated as labeled ones and added to few
shots to jointly re-estimate the remaining unlabeled ones. The above
procedures will be repeated for a certain number of iterations with more and
more samples being collected until convergence, ensuring a progressive and
robust knowledge completion process. Extensive experiments on 11 benchmark
datasets demonstrate the effectiveness and efficiency of KCL as a plug-and-
play module under both few-shot and zero-shot learning settings. Code is
available at https://github.com/Mark-Sky/KCL

###### Keywords:

Vision-language model Few-shot learning Visual knowledge completion

††∗ indicates equal contributions, $\dagger$ indicates corresponding author

##  1 Introduction

Vision-language pre-training has significantly impacted the machine learning
community, with various Vision-Language Models (VLMs) [33, 22, 19, 23, 49]
demonstrating exceptional transfer performance across tasks such as
classification [20, 3], object detection [43, 25], and segmentation [44, 25,
54]. The most representative study, Contrastive Language-Image Pretraining
(CLIP) [33], is trained on a substantial dataset of image-text pairs using
contrastive loss, have emerged as the standard for transferring knowledge to
downstream tasks in computer vision. CLIP rebuilds the paradigm of zero-shot
transfer, enabling classification on previous unseen categories according to
similarities between image-text pairs. CLIP converts class labels into textual
prompts, such as ‘ _a photo of a <class>_.’, where _< class>_ represents the
ground-truth textual label for each class. It then calculates similarities
between all class prompts and the test image, ultimately selecting the class
with the highest similarity score as the predicted label.

(a) Feature visualization of 1-shot ImageNet [8].

(b) Performance comparison on 11 datasets.

Figure 1: Effectiveness demonstration of KCL. (a) t-SNE visualization of three
categories in ImageNet [8] before and after iterative visual knowledge
completion based on 1-shot MaPLe [20]. (b) Classification results before and
after using iterative visual knowledge completion on zero-shot CLIP [33] and
1-shot MaPLe [20].

Building upon the remarkable zero-shot transfer performance, recent studies
have delved deeper and focus on enhancing CLIP within few-shot learning
scenarios. Under such settings, the model is tasked with further learning from
a limited number of images, aka few shots, for each class to improve transfer
capability towards novel tasks or domains. Existing methods mainly follow two
paradigms to extract and utilize knowledge from few shots: Implicit modeling
and explicit modeling. Specifically, implicit modeling-based methods mainly
apply small number of learnable parameters to learn from few shots
efficiently. Differently, explicit modeling-based methods do not need
learnable parameters but follow the idea of cache models [50], which store the
features and labels of few shots, and then conduct inference by similarity-
based retrieval.

Despite the significant improvements made by existing few-shot learning
approaches, they mainly focus on drawing more knowledge from the given few
shots. However, the fundamental problem caused by data scarcity is rarely
taken seriously, namely, the narrow-size distribution of few shots only
contains incomplete and biased category information. As is shown in Fig. 1(a),
we can find that the given few shots are very likely to be distributed near
classification boundaries and far away from the optimal class centers,
especially in 1-shot scenario. In this case, the biased knowledge extracted
from few shots can easily cause confusions with other nearby categories,
leading to misclassification. Obviously, this situation also exists in zero-
shot settings where no visual data is available [40], which can be regarded as
a special case of few-shot learning.

Recently, C2A [36] and SuS-X [40] propose using generative models (e.g. Stable
Diffusion [35, 4]) and searching in large-scale external databases (e.g.
LAION-5B [37]) to alleviate visual data scarcity. However, generative models
always involve large amount of computations, which can greatly affects
efficiency. Similarly, auxiliary databases require additional storage space
and searching through massive data is time-consuming as well. Besides, they
both have ability bottlenecks and are prone to failure when facing previously
unseen concepts [40]. Therefore, we ask the following question:

_Is it possible to complement the biases arising from data scarcity while also
maintaining an efficient and concise process, without relying on generative
models or auxiliary databases that may incur heavy costs and limited
coverage?_

To achieve this goal, we propose an iterative visual knowledge completion
method, termed as KCL, which can alleviate the bias caused by data scarcity
during inference time without additional training. In contrast to existing
methods that involve high-computation operations such as generation or
searching in external databases, KCL offers an efficient solution to visual
knowledge completion by incorporating high-confidence unlabeled samples with a
designed confidence rule. Specifically, KCL first calculates the similarity
matrix between each test samples and all categories by integrating the
similarity matrices from both textual and visual feature spaces. Then, the
most confident samples belonging to each class are selected as candidate
samples with a design confidence criterion. Finally, the candidate samples are
merged into few shots as supplements, refining potential class centers. The
complemented knowledge will be used to re-estimate the remaining unlabeled
data. We repeat the above procedures for a certain number of iterations with
growing number of samples being merged into few shots, guaranteeing a gradual
and robust visual knowledge complementing process. As is shown in Fig. 1(a),
the potential class centers after knowledge completion are much closer to the
optimal ones.

We further extend KCL to zero-shot settings by modifying the first similarity
matrix to only take textual knowledge from original CLIP, which can be viewed
as a special case of few-shot learning. Since KCL conducts adaptation at
inference time, it can function as a plug-and-play module, seamlessly
integrating with existing CLIP transfer methods without additional
modifications. Therefore, we choose mainstream few-shot transfer methods from
different types as few-shot baselines and original CLIP as zero-shot baseline.
With the assistance of KCL, all these methods achieve remarkable improvements,
which will be presented in Sec. 4. As is shown in Fig. 1(b), we can see that
KCL improves the performance of original methods (zero-shot CLIP [33] and
MaPLe [20]) by large margins, demonstrating its effectiveness in both few-shot
and zero-shot settings.

Overall, our contributions are summarized as follows:

  * 1.

We figure out the fundamental factor: narrow-size distribution of few shots
results in biased visual knowledge, that limits the transfer capability of
CLIP.

  * 2.

We propose KCL, an iterative visual knowledge completion method which
complements the biased visual knowledge without access to auxiliary and
synthetic data. It takes advantages of a designed confidence criterion based
on mutual nearest neighbors theory to mine useful knowledge from high-
confidence test samples to supplement few shots iteratively.

  * 3.

We further extend KCL to zero-shot settings where no labeled data is available
and enhance zero-shot CLIP by incorporating high-confidence visual knowledge
from unlabeled data.

  * 4.

We conduct extensive experiments on 11 benchmark datasets, which demonstrate
the effectiveness of KCL as a training-free plug-and-play module for both
zero-shot and few-shot learning tasks.

##  2 Related Works

Vision-Language Models (VLMs) [33, 22, 19, 23, 49, 47, 18, 13, 38, 10] have
emerged as a novel paradigm in recent years. CLIP [33], a groundbreaking
study, trained on 400 million web-scale image-text pairs, showcasing
remarkable performance on downstream tasks. In this work, we propose to
enhance the transfer capability of CLIP, thus we will introduce existing
transfer methods for CLIP with limited data [52, 51, 12, 2, 20, 48, 45, 50,
55, 32]. Generally, we divide existing methods into two types depending on
whether knowledge completion is adopted.

###  2.1 Methods without Knowledge Completion

Existing methods mainly focus on knowledge extraction other than completion.

Implicit Modeling Methods. Implicit modeling methods mainly apply small number
of learnable parameters to perform fine tuning, such as prompt tuning and
feature adaptation. Unlike traditional methods that rely solely on pre-defined
prompts, CoOp [52] leverages back-propagation on few-shot datasets to
dynamically generate text prompts tailored to specific tasks. On top of CoOp,
CoCoOp [51] leverages visual features to regularize the learning process of
text prompts, enabling model to capture both linguistic and visual cues
simultaneously. Beyond textual prompt tuning, MaPLe [20] propose a multi-modal
prompt tuning strategy for both vision and language branches to improve
alignment between visual and textual representations. Meanwhile, CLIP-Adapter
[12] involves learning a visual and a textual adapter, which are modules
designed to refine the representations of the original CLIP specifically.

Explicit Modeling Methods. Explicit modeling methods follow the paradigm of
metric learning and cache modeling. Tip-Adapter [50] first-time introduces the
idea of explicit modeling for CLIP by directly constructing an adapter with a
key-value cache model, which takes training-free advantage of zero-shot CLIP
while also achieves remarkable performance compared to few-shot transfer
methods. On top of Tip-Adapter, APE [55] proposes to use a channel-wise
feature selection strategy, which maintains the class-relevant features and
wipes out class-irrelevant features. Recently, [45] proposes using Gaussian
discriminant analysis to estimate parameters for constructing cache models.

###  2.2 Methods with Knowledge Completion

In general, existing transfer methods for CLIP mainly focus on how to extract
more useful knowledge from few shots. However, the narrow-size distribution of
few shots always contains incomplete class information, which may lead to
naturally biased visual knowledge. CuPL [32] proposes to augment textual
prompts by external knowledge bases (e.g. GPT-3) to enhance the understanding
towards categories with limited visual data. Recent works SuS-X [40] and C2A
[36] propose using text-image and image-image generation [35] or auxiliary
database LAION-5B [37] to tackle visual data scarcity, which may involve large
amount of computation and storage space. Differently, in this work, we propose
to alleviate data scarcity by incorporating high-confidence test samples
iteratively without using auxiliary or synthetic data.

##  3 Methodology

To better illustrate our iterative visual knowledge completion method, we
first briefly revisit zero-shot CLIP [33] and few-shot Tip-Adapter [50]
methods.

Zero-shot CLIP. Given a classification task with $C$ categories, CLIP
transforms the category names into textual prompts. The encoded prompt vectors
can be interpreted as a weight matrix for the classifier, termed as
$\bm{W}\in\mathbb{R}^{C\times d}$. Then, CLIP’s visual encoder $E_{v}$ is
applied to extract features from $N$ test samples
$\\{x_{1},x_{2},\cdots,x_{N}\\}$:

$f_{i}=E_{v}(x_{i}),i\in[1,N],f_{i}\in\mathbb{R}^{1\times d}.$ (1)

Finally, the cosine similarity matrix $\mathcal{A}_{clip}$ between $N$ test
samples and $C$ categories can be computed by:

$\bm{A}_{clip}=\bm{fW}\in\mathbb{R}^{N\times C},$ (2)

where $\bm{f}$ is the concatenation of $N$ image features
$\\{f_{1},f_{2},\cdots,f_{N}\\}$.

Tip-Adapter. Under the setting of few-shot learning, zero-shot CLIP is
supplemented with $M$ labeled samples for each category with one-hot label
matrix $\bm{l}\in\mathbb{R}^{CM\times C}$. We also extract features of these
$CM$ labeled samples $\\{y_{1},y_{2},\cdots,y_{CM}\\}$:

$F_{i}=E_{v}(y_{i}),i\in[1,CM],F_{i}\in\mathbb{R}^{1\times d}.$ (3)

Then the visual similarity matrix between the concatenated visual features
$\bm{f}$ and $\bm{F}$ is computed by
$\text{exp}(-\beta(1-\bm{f}\bm{F}^{\top})$, where $\beta$ is a dataset-
specific hyper-parameter. Finally, the overall similarity matrix
$\mathcal{A}_{tip}$ is given by:

$\bm{A}_{tip}=\alpha\text{exp}{(-\beta(1-\bm{fF}^{\top}))}\bm{l}+\bm{fW}\in\mathbb{R}^{N\times
C},$ (4)

where $\alpha$ is a hyper-parameter controlling the ratio between visual and
textual similarity matrices.

Methodology of KCL. Motivated by C2A [36] and SuS-X [40] which augment visual
data by utilizing generative diffusion models or large-scale external
databases. In this work, we aim to complement the biased visual knowledge of
few shots by taking advantages of unlabeled samples without accessing
auxiliary or synthetic data.

Following [36] and [40], we also build our KCL module upon Tip-Adapter. Eq. 4
can be viewed as two parts, respectively standing for the image-to-image and
image-to-text similarities between $N$ test features
$\bm{f}=\\{f_{1},f_{2},\cdots,f_{N}\\}$ and $C$ categories
$\bm{S}=\\{S_{1},S_{2},\cdots,S_{C}\\}$. As is discussed before, the knowledge
in few-shot visual features $\bm{F}$ is biased due to data scarcity and we
propose to complement $\bm{F}$ by using high-confidence features in $\bm{f}$.
Therefore, the confidence criterion used to distinguish the most confident
samples becomes a key issue.

In this work, we design a confidence criterion based on mutual nearest
neighbors theory [14, 5], which means only when a test sample and the category
are nearest neighbors of each other, will it be regarded as a high-confidence
sample and selected. Specifically, given the $i$-th category $S_{i}$ with
label $l_{i}$, we can define its mutual nearest neighbors (MNNs) in $\bm{f}$
as:

$\mathcal{M}_{K_{1},K_{2}}(S_{i})=\\{f_{j}\in\bm{f}|f_{j}\in\mathcal{N}_{K_{1}}^{\bm{f}}(S_{i})\wedge
S_{i}\in\mathcal{N}_{K_{2}}^{\bm{S}}(f_{j})\\},$ (5)

where $\mathcal{N}_{K}^{\bm{B}}(a)$ represents a subset of $\bm{B}$ consisting
of the $K$ nearest neighbors of $a$ in $\bm{B}$. Due to the uniqueness of
classification, we fix $K_{2}=1$ under all situations and will further discuss
the influence of $K_{1}$ towards effectiveness and efficiency in Sec. 4.4 and
Sec. 5. After seeking out MNNs for all categories, we get
$\\{\mathcal{M}_{K_{1},K_{2}}(S_{i})\\}_{i=1}^{C}\in\bm{f}$, then we label
them with corresponding categories and collect them into a knowledge
completion set $\bm{D}$ with label set $\bm{l}_{D}$, completing one round of
knowledge completion. The knowledge completion set $\bm{D}$ will be utilized
to jointly re-estimate the similarity matrix between the remaining test data
$\bm{f}^{\prime}=\bm{f}-\bm{D}\in\mathbb{R}^{N^{\prime}\times d}$ and $C$
categories ($N^{\prime}$ denotes the number of remaining samples in
$\bm{f}^{\prime}$).

$\bm{A}_{kcl}^{fs}=\lambda\text{exp}{(-\mu(1-\bm{f^{\prime}D}^{\top}))}\bm{l_{D}}+\alpha\text{exp}{(-\beta(1-\bm{f^{\prime}F}^{\top}))}\bm{l}+\bm{f^{\prime}W}\in\mathbb{R}^{N^{\prime}\times
C}.$ (6)

Note that $\lambda$ and $\mu$ are hyper-parameters for $\bm{D}$, which will be
discussed in Sec. 4.3. Besides, $\alpha$ and $\beta$ are hyper-parameters in
Eq. 4, which are fixed from beginning. In practice, we repeat the above
procedures for a certain number of iterations to gradually increases the
number of samples taken from $\bm{f}$ into $\bm{D}$, ensuring a robust and
progressive knowledge completion process. The specific number of iterations or
the number of unlabeled samples required for visual knowledge completion will
be discussed in Sec. 4.4 and Sec. 5.

We further extend KCL to zero-shot learning scenarios where $\bm{F}=\emptyset$
and the first-step similarity matrix is given by Eq. 2. Then the similarity
matrix after confidence selection is modified to:

$\bm{A}_{kcl}^{zs}=\lambda\text{exp}{(-\mu(1-\bm{f^{\prime}D}^{\top}))}\bm{l_{D}}+\bm{f^{\prime}W}\in\mathbb{R}^{N^{\prime}\times
C}.$ (7)

Note that we keep the same hyper-parameter searching mechanism as few-shot
learning and also repeat a certain number of iterations until convergence.

Note that, when reaching convergence, either few-shot or zero-shot
classification results will be given by $\bm{l}_{D}$ and the last-round
similarity matrix based on the nearest neighbor theory.

##  4 Experiments

###  4.1 Experimental Settings

Datasets. In this work, following the common practice in CLIP transfer
scenario, we adopt 11 benchmark datasets spanning a wide range of object,
scene and fine-grained categories for a comprehensive evaluation: ImageNet
[8], UCF [39], DTD [6], FGVC [27], Flowers [30], Cars [21], Eurosat [17], SUN
[46], Pets [31], Caltech [11] and Food [1].

Implementation details. For few-shot transfer, we follow [52] and conduct
experiments on 1, 2, 4, 8, 16 few-shot classification tasks across 11
datasets. Since KCL acts as a plug-and-play module, we follow the backbone
settings in original papers of each method. Specifically, for experiments on
zero-shot CLIP, CoOp, CLIP-Adapter, Tip-Adapter and APE, we choose ResNet-50
[16] as backbone; while for experiments on MaPLe, we choose ViT-B/16 [9] as
visual backbone structure. For textual prompts, we utilize CuPL [32] for all
experimental results in this work for fair comparisons between different
approaches. Besides, we follow the training details of CoOp [51], CLIP-Adapter
[12] and MaPLe [20] in original papers. The hyper-parameter settings of this
work are as follows: for $\alpha$ and $\beta$, we follow [50] and search on
validation set at the first round, which will be fixed in following
procedures; while for newly introduced $\lambda$ and $\mu$, we search on
validation set for the best numerical values in each iteration. All
experiments in this work are conducted on a single NVIDIA Tesla V100-32G GPU.

###  4.2 Baseline Methods

Since KCL can function as a plug-and-play module, we are able to embed it to
any type of approaches. Therefore, to demonstrate its effectiveness, we embed
KCL to representative methods for CLIP-based few-shot learning of different
types as well as zero-shot CLIP.

Few-shot learning. CoOp [52] is the most representative method in the field of
textual prompt tuning for CLIP transfer, which proposes to replace fixed
prompts with learnable text tokens. Note that we initialize the learnable
tokens of CoOp with CuPL [32] in this work. CLIP-Adapter [12] is the
representative method in the field of feature adapters for CLIP transfer,
which adds MLPs to the two branches of CLIP and adapts by gradient decent on
these two MLPs. We follow the settings of the original paper for CLIP-Adapter
in this paper. Tip-Adapter [50] is a representative approach in the field of
cache model-based CLIP transfer, which explicitly encodes few-shot data into a
cache model and propagate knowledge to test samples. We also follow the
original settings of Tip-Adapter to construct the key-value cache model. MaPLe
[20] is a representative method of multi-modal prompt tuning, which combines
learnable textual and visual prompts to learn better visual and textual
representations for few-shot transfer. We also include APE [55] in
supplementary materials, which extends Tip-Adapter with channel-wise feature
refinement. We attach KCL to above methods and improvements are shown in Table
1.

Zero-shot learning. Zero-shot CLIP  [33]: we use prompt ensembling strategy
with seven prompt templates following [40]. CALIP [15] proposes using a cross
attention mechanism to obtain more discriminative representations. CuPL [32]
augments textual prompts of original CLIP by consulting GPT-3. SuS-X [40]
proposes to complement visual knowledge of zero-shot CLIP by diffusion-based
image generation or retrieval in large-scale databases. Experimental results
under zero-shot settings are shown in Table 2.

Table 1: Experimental results (%) of few-shot learning on 11 benchmark
datasets. The overall results including 8-shot and 16-shot are shown in
supplementary materials.

Method |  ImageNet |  UCF |  DTD |  FGVC |  Flowers |  Cars |  Eurosat |  SUN |  Pets |  Caltech |  Food |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
1-shot
CoOp [52] | 55.70 | 64.92 | 53.25 | 21.18 | 81.81 | 53.89 | 62.56 | 60.68 | 81.60 | 90.06 | 73.42 | 63.55
$\bm{\Delta}\textbf{+KCL}$ | +2.84 | +4.73 | +4.73 | +0.18 | +4.22 | +3.40 | +6.75 | +2.77 | +3.11 | +0.49 | +2.76 | +3.27
CLIP-Adapter [12] | 61.94 | 64.58 | 51.60 | 21.27 | 72.03 | 58.77 | 61.23 | 63.83 | 85.61 | 90.10 | 76.80 | 64.34
$\bm{\Delta}\textbf{+KCL}$ | +1.11 | +6.90 | +5.97 | +1.47 | +10.63 | +1.67 | +0.99 | +1.83 | +1.17 | +1.06 | +0.69 | +3.04
Tip-Adapter [50] | 61.83 | 63.86 | 51.12 | 20.94 | 72.63 | 58.60 | 57.68 | 54.08 | 85.25 | 89.25 | 77.57 | 63.89
$\bm{\Delta}\textbf{+KCL}$ | +0.71 | +6.24 | +5.44 | +0.63 | +9.79 | +2.74 | +7.05 | +1.50 | +1.20 | +1.34 | +0.61 | +3.38
MaPLe [20] | 69.67 | 69.94 | 41.31 | 27.18 | 77.30 | 66.99 | 60.75 | 70.05 | 91.55 | 93.06 | 84.12 | 68.35
$\bm{\Delta}\textbf{+KCL}$ | +1.11 | +3.57 | +6.56 | +2.73 | +12.87 | +3.92 | +6.78 | +1.29 | +0.27 | +0.37 | +0.22 | +3.60
2-shot
CoOp [52] | 57.36 | 68.44 | 55.85 | 25.11 | 84.94 | 60.17 | 65.49 | 62.03 | 84.19 | 89.49 | 73.58 | 66.05
$\bm{\Delta}\textbf{+KCL}$ | +1.15 | +4.97 | +0.53 | -0.18 | +3.61 | +1.03 | +7.52 | +3.17 | +1.36 | +0.17 | +2.28 | +2.32
CLIP-Adapter [12] | 61.84 | 66.88 | 56.97 | 21.87 | 73.12 | 58.92 | 62.23 | 64.11 | 87.33 | 90.83 | 77.34 | 65.58
$\bm{\Delta}\textbf{+KCL}$ | +1.00 | +2.93 | +3.49 | +0.84 | +3.25 | +1.12 | +5.89 | +1.21 | +0.62 | +0.08 | +0.54 | +1.90
Tip-Adapter [50] | 62.01 | 66.56 | 54.85 | 23.10 | 76.49 | 60.87 | 59.07 | 65.49 | 84.25 | 89.49 | 77.35 | 65.32
$\bm{\Delta}\textbf{+KCL}$ | +0.84 | +4.57 | +2.36 | +0.48 | +8.12 | +1.40 | +9.10 | +1.16 | +2.56 | +0.53 | +0.45 | +2.87
MaPLe [20] | 70.43 | 73.96 | 53.66 | 28.98 | 79.54 | 63.06 | 57.89 | 70.35 | 91.09 | 93.87 | 86.33 | 69.92
$\bm{\Delta}\textbf{+KCL}$ | +1.17 | +3.91 | +4.79 | +5.22 | +12.75 | +5.10 | +20.64 | +2.43 | +0.46 | +0.25 | +0.58 | +5.20
4-shot
CoOp [52] | 58.86 | 70.71 | 61.64 | 26.37 | 89.77 | 64.71 | 72.70 | 65.82 | 87.24 | 91.48 | 74.91 | 69.47
$\bm{\Delta}\textbf{+KCL}$ | +1.64 | +4.28 | +0.18 | 0.00 | +1.50 | +0.19 | +4.29 | +0.95 | +0.39 | -0.04 | +1.20 | +1.32
CLIP-Adapter [12] | 62.24 | 69.15 | 57.98 | 22.20 | 82.01 | 60.81 | 73.27 | 66.74 | 87.52 | 91.16 | 77.82 | 68.26
$\bm{\Delta}\textbf{+KCL}$ | +0.98 | +2.94 | -0.12 | +1.71 | +6.42 | +2.57 | +5.15 | +1.16 | +0.35 | +0.73 | +0.41 | +2.02
Tip-Adapter [50] | 62.16 | 68.46 | 59.22 | 23.34 | 82.34 | 62.82 | 69.27 | 65.57 | 84.79 | 90.34 | 77.25 | 67.77
$\bm{\Delta}\textbf{+KCL}$ | +0.49 | +3.18 | +1.60 | -0.09 | +4.26 | +1.02 | +2.74 | +1.13 | +1.61 | +0.49 | +0.49 | +1.53
MaPLe [20] | 70.85 | 76.26 | 58.51 | 31.80 | 84.49 | 69.71 | 62.40 | 72.00 | 92.94 | 94.44 | 86.27 | 72.69
$\bm{\Delta}\textbf{+KCL}$ | +1.44 | +7.14 | +5.44 | +4.11 | +9.26 | +4.44 | +16.59 | +2.08 | +0.03 | +1.02 | +0.14 | +4.69

###  4.3 Experimental Results

(a) Analysis of knowledge completion capability.

(b) Analysis of hyper-parameter sensitivity.

Figure 2: Further analyses under few-shot settings. (a) The averaged
performance on 11 datasets compared with C2A [36]. (b) The averaged
performance of hyper-parameter sensitivity analysis on 11 datasets based on
MaPLe [20]. Note that † means hyper-parameters $\lambda$ and $\mu$ of KCL
method are both fixed to 1.

Few-shot learning results. To estimate the few-shot learning capability of KCL
as a plug-and-play module, we apply it to several kinds of CLIP-based few-shot
learning methods, and a part of the experimental results are shown in Table 1
(please refer tosupp for overall results). We can see that KCL significantly
improves the average performances of existing methods on 11 datasets.
Specifically, for 1-shot setting, KCL can achieve 3.04%-3.60% improvements;
for 2-shot setting, KCL can gain 1.90%-5.20% improvements; for 4-shot setting,
KCL achieves 1.32%-4.69% improvements over original methods; for 8-shot tasks,
KCL can surpass the initial methods by 0.83%-3.88%; for 16-shot tasks, KCL
achieves an average improvement of 1.11%. Generally, we can find that KCL
achieves relatively larger improvements on fewer-shot settings. This
phenomenon is in line with our motivations for proposing KCL, which aims to
complement biased visual knowledge caused by data scarcity. Naturally, the
fewer the shots, the greater the knowledge bias, and thus the larger the
improvements made by KCL. Furthermore, we notice that KCL can achieve larger
improvements when embedded to relatively stronger base models. This is because
stronger base method can better distinguish unlabeled samples with high
confidence.

Besides, we also compare KCL with existing visual knowledge completion methods
C2A [36], which takes advantages of stable diffusion to generate images for
knowledge completion. Since C2A method complements visual knowledge based on
captions made by BLIP [22], Tip-Adater-F [50] model and feature adapters, we
compare it with KCL modules embedded in CoOp and CLIP-Adapter. As is shown in
Fig. 2(a), we can find that KCL module based on single CLIP-Adapter can
achieve competitive performance against C2A. Besides, KCL module embedded in
CoOp can surpass C2A by almost 1% improvements under all settings. This
undoubtedly proves the superiority of KCL which does not require additional
generative models.

Furthermore, we also conduct hyper-parameter sensitivity analysis on 11
datasets based on MaPLe. As is shown in Fig. 2(b), we can find that hyper-
parameter searching has a small impact on final classification performance,
which reveals the robustness and simplicity of the KCL as a plug-and-play
module.

Table 2: Experimental results (%) of zero-shot learning on 11 benchmark
datasets. We present the results reported in [40].

Method |  ImageNet |  UCF |  DTD |  FGVC |  Flowers |  Cars |  Eurosat |  SUN |  Pets |  Caltech |  Food |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
CLIP [33] | 60.31 | 55.56 | 41.01 | 16.71 | 62.89 | 56.33 | 26.83 | 59.11 | 81.82 | 85.92 | 74.11 | 56.41
CALIP [15] | 60.31 | 55.61 | 41.01 | 16.71 | 63.01 | 56.32 | 26.96 | 59.10 | 81.84 | 86.20 | 74.13 | 56.47
CuPL [32] | 61.45 | 58.97 | 48.64 | 19.59 | 65.44 | 57.28 | 38.38 | 62.55 | 84.84 | 89.29 | 76.94 | 60.30
CuPL-e [32] | 61.64 | 61.45 | 47.45 | 19.26 | 65.93 | 57.23 | 37.06 | 62.99 | 85.09 | 89.41 | 77.52 | 60.45
VisDesc [28] | 59.68 | 58.47 | 41.96 | 16.26 | 65.37 | 54.76 | 37.60 | 59.84 | 82.39 | 88.11 | 76.80 | 58.30
SuS-X$\bm{}{SD}$ [40] | 61.84 | 61.54 | 50.59 | 19.47 | 67.72 | 57.27 | 45.57 | 62.95 | 85.34 | 89.53 | 77.58 | 61.72
SuS-X$\bm{}{LC}$ [40] | 61.79 | 61.43 | 49.47 | 20.34 | 67.60 | 57.06 | 37.16 | 62.94 | 85.22 | 89.61 | 77.58 | 60.93
KCL (Ours) | 62.60 | 67.06 | 56.21 | 20.91 | 72.19 | 59.68 | 52.69 | 64.98 | 86.15 | 90.79 | 77.84 | 64.65
$\bm{\Delta}$ vs CLIP | +2.29 | +11.50 | +15.20 | +4.20 | +9.30 | +3.35 | +25.86 | +5.87 | +4.33 | +4.87 | +3.73 | +8.23
$\bm{\Delta}$ vs SuS-X | +0.76 | +5.52 | +5.62 | +0.57 | +4.47 | +2.41 | +7.12 | +2.03 | +0.81 | +1.18 | +0.26 | +2.82

Zero-shot learning results. To evaluate KCL on zero-shot learning tasks, we
also conduct experiments on 11 benchmark datasets. As is shown in Table 2, we
can observe that KCL achieves state-of-the-art performance on 11 benchmark
datasets. Specifically, when compared with zero-shot CLIP, KCL gains
2.29%-25.86% improvements and 8.23% improvements in average, demonstrating the
effectiveness of iterative visual knowledge completion for CLIP. Besides, when
compared with SuS-X [40] that complements visual knowledge with generative
models and large-scale auxiliary databases, KCL also achieves 2.82%
improvements in average. This demonstrates the superiority of KCL against
SuS-X, which can replenish visual knowledge by utilizing unlabeled samples
without accessing auxiliary or synthetic data.

###  4.4 Ablation Study

In this section, we dive into the details of KCL method and analyze different
choices of $K_{1}$ values (hereafter denoted as $K$) in Eq. 5, the number of
unlabeled samples required for knowledge completion, the effectiveness of our
confidence criterion and modalities used to calculate similarity matrix.

The choice of $\bm{K}$ values.

(a) UCF

(b) Cars

(c) FGVC

Figure 3: Ablation study on $K$ values based on MaPLe [20]. Note that we
incorporate all unlabeled samples in test set for fair comparison.

Since KCL iteratively select the top $K$ mutual nearest neighbors from
unlabeled samples for knowledge completion, the choice of $K$ value of becomes
a key issue. Here we try different $K$ values ranging from 1 to full size of
test set and present the corresponding classification results in Fig. 3. We
can find that when the total number of unlabeled samples remains unchanged,
the smaller $K$, the better the performance. In general, when $K$ is small,
the performance decrease is not significant; when $K$ becomes large, the
performance decreases quickly. This is because smaller $K$ value leads to a
smoother and robust knowledge completion process while larger $K$ leads to a
rugged one. Therefore, in this paper, we report all experimental results based
on $K$=1.

Number of unlabeled samples required for completion.

(a) ImageNet

(b) UCF

(c) Cars

Figure 4: Ablation study on number of unlabeled samples required for knowledge
completion. Note that we conduct experiments based on MaPLe [20] under few-
shot settings.

In this work, we propose KCL to take advantages of unlabeled samples for
knowledge completion. However, in real applications, unlabeled samples can be
infinite thus it is time-consuming to make use of all unlabeled samples for
knowledge completion. Therefore, in this section, we take a close look at the
number of unlabeled samples required for convergence. Specifically, we conduct
experiments on three datasets to observe the relationships between
classification performance and number of unlabeled samples (note that we
choose $K$=1 for all datasets). From Fig. 4 we can see that as the number of
unlabeled samples increases, classification performance increases rapidly at
beginning and then levels off till the end. Moreover, we can notice that on
relatively simpler fine-grained datasets like Cars, only 5-10 unlabeled
samples per category are enough for visual knowledge completion; while on
relatively harder datasets like ImageNet and UCF, nearly 20-25 samples per
category are needed for complete visual knowledge. This phenomenon indicates
that KCL has no need to take all unlabeled data into account, instead, only 25
samples per category are enough for achieving remarkable improvements,
ensuring a high efficiency of KCL method.

Effectiveness of the designed confidence criterion.

(a) UCF

(b) Cars

(c) FGVC

Figure 5: Ablation study on the designed confidence criterion based on MaPLe
[20].

In this paper, we select and incorporate unlabeled samples according to a
designed confidence criterion based on mutual nearest neighbor theory [14],
which is a core content of KCL. Besides the remarkable improvements made by
KCL, we would like to fully reveal the contribution of the designed confidence
criterion. Specifically, we compare KCL with two types of degraded nearest
neighbor (NN) strategies: (1) image-to-category NN strategy chooses the most
similar samples to each category for knowledge completion, which is given by
modifying Eq. 5 to
$\overline{\mathcal{M}}_{K}(S_{i})=\\{f_{j}\in\bm{f}|f_{j}\in\mathcal{N}_{K}^{\bm{f}}(S_{i})\\}$,
termed as KCL▽; (2) category-to-image NN strategy makes choices following
modified Eq. 5:
$\widehat{\mathcal{M}}_{K}(S_{i})=\\{f_{j}\in\bm{f}|S_{i}\in\mathcal{N}_{K}^{\bm{S}}(f_{j})\\}$,
termed as KCL△. As is shown in Fig. 5, we conduct ablation study on three
datasets based on MaPLe [20] and we can observe that KCL always achieves the
best performance among different criteria. This demonstrates the superiority
of the MNNs-based confidence criterion, which can give the confidence
selection process a double guarantee.

Modalities for computing similarity matrix.

(a) ImageNet

(b) UCF

(c) FGVC

Figure 6: Ablation study on modalities chosen for computing similarity matrix.
Note that we report results based on MaPLe [20] under both zero-shot and few-
shot settings.

As is shown in Eq. 6 and Eq. 7, we combine a visual similarity matrix and a
textual similarity matrix as a multi-modal similarity matrix. To prove the
effectiveness of visual and textual knowledge for predicting high-confidence
unlabeled samples, we conduct comparative experiments on visual-only and text-
only settings. As is shown in Fig. 7, we can find that multi-modal similarity
matrix achieves the best performance in all cases. Besides, performance under
text-only setting is better than that under visual-only setting in most cases
except for 8-shot and 16-shot FGVC tasks. This may be because FGVC is a
professional fine-grained dataset of various rare aircrafts, which may exceed
the scope of CLIP’s generalization capabilities, thus requiring more
assistance from visual information.

##  5 Discussion

Figure 7: Classification performance (%) with different $K$ and iteration
steps on ImageNet [8].

Method Completion Time Accuracy (%) CLIP∗ [33] – 60.32 VisDesc∗ [28]
$\sim$3.0s 59.68 CuPL ∗[32] $\sim$3.0s 61.64 SuS-X-SD∗[40] $\sim$60.0s 61.84
SuS-X-LC∗[40] $\sim$2.0s 61.89 KCLK=1,Step=2$\ddagger$ $\sim$0.8s 61.87
KCLK=1,Step=4$\ddagger$ $\sim$1.5s 61.94 KCLK=3,Step=18$\ddagger$ $\sim$41.4s
62.60

Table 3: Classification performance (%) and completion time cost for one
ImageNet [8] class. $\ddagger$ denotes our implementation on single V100-32G
GPU and ∗ denotes results reported in [40] with single A100-80G GPU.

Q: KCL requires selecting and incorporating high-confidence samples
iteratively, can it stay efficient compared to other methods? A: Yes.

In this work, we propose to complement visual knowledge by repeatedly
selecting and incorporating unlabeled samples with high confidence, which may
brings complex operations and high computational costs. We conduct in-depth
experiments on zero-shot ImageNet [8] to prove the efficiency of KCL. As is
shown in Fig. 7, we first present classification accuracy under different $K$
and iteration steps. We can see that when the number of unlabeled samples
($K\times\text{step}$) get larger, visual knowledge completion can reach
convergence faster. Moreover, we also compare the time costs of KCL with other
knowledge completion methods on zero-shot tasks, in which VisDesc [28] and
CuPL [32] make use of external knowledge bases (e.g. GPT-3) to complement
textual knowledge; while SuS-X [40] utilizes generative models and auxiliary
databases to complement visual knowledge. From Table 7 we can observe that KCL
can achieve higher classification accuracy with less completion time cost,
achieving a good balance between performance and efficiency.

Q: KCL takes advantage of unlabeled data, does it duplicate with semi-
supervised learning or transductive inference? A: No.

Semi-supervised learning [53, 34] is one of the important approaches for few-
shot and zero-shot learning, which takes advantages of external unlabeled
databases by pseudo labeling. Differently, KCL does not require access to
auxiliary databases, which leverages the power of pre-trained CLIP and
directly makes use of test data with high confidence. Besides, the core
concept of transductive inference [42, 26, 24] is to predict test data as a
whole without learning a rule first like common inductive inference.
Differently, KCL goes beyond transduction and inherits an advanced concept of
_selective inference_ [41], which selects among the unlabeled test samples the
most confident ones. KCL takes it further by utilizing the selected samples to
conduct visual knowledge completion for alleviating the bias caused by data
scarcity. Therefore, KCL is better prepared for difficult test samples with
low confidence than conventional transductive inference, especially in low-
shot tasks. As is shown in Table 4, KCL outperforms transductive label
propagation (TLP) [26, 24], which is a common practice for transductive
inference, demonstrating the above points.

Table 4: Classification results (%) of KCL and transductive label propagation
(TLP).

Method | UCF | FGVC
---|---|---
1-shot | 2-shot | 4-shot | 8-shot | 16-shot | 1-shot | 2-shot | 4-shot | 8-shot | 16-shot
MaPLe+TLP | 71.03 | 75.10 | 78.43 | 81.34 | 82.98 | 28.14 | 31.05 | 34.05 | 35.19 | 39.57
MaPLe+KCL | 73.51 | 77.87 | 83.40 | 84.14 | 84.09 | 29.91 | 34.20 | 35.91 | 41.28 | 44.94

Q: Any limitations or subsequent improvements for KCL? A: Yes.

Despite the remarkable performance achieved by KCL, there are still some
limitations need to be improved in the future: (1) The knowledge completion
process of KCL highly dependents on pre-trained CLIP models, which assumes
CLIP has been exposed to everything during pre-training stage. When facing
something extremely rare, our method may fail to transfer effectively, which
is also a common problem in existing low-shot learning methods [52, 20, 40].
(2) Following most existing approaches for CLIP-based transfer learning [15,
51, 55], KCL still targets at solving closed-set low-shot classification
problems, which is not consistent with real application scenarios where open-
set and incremental classes exist [29]. KCL may take a high risk of failure
under such real settings, being unable to identify out-of-distribution samples
or catastrophically forget previously learned concepts.

##  6 Conclusion

In this work, we figure out the fundamental factor that constrains few-shot
learning performance of CLIP: the biased visual knowledge caused by narrow
distribution of few shots. To tackle this problem, we propose an iterative
visual knowledge completion method (KCL) to make use of unlabeled test data
properly with a designed confidence criterion and further extend it into zero-
shot learning tasks. Unlike existing methods, KCL conducts visual knowledge
completion without access to auxiliary or synthetic data. Extensive
experiments on 11 benchmark datasets demonstrate the effectiveness of KCL in
both zero-shot and few-shot settings, acting as a plug-and-play module to
existing state-of-the-arts. Ablation study and further analyses also prove the
efficiency and rationality of KCL method. Our future work will focus on
adapting KCL to real applications with open-set and continuous categories.

#### Supplementary Materials

##  Appendix 0.A Dataset Details

We provide the information of validation and testing splits for 11 benchmark
datasets in Table 5. The introductions of all datasets are presented as
follows:

Table 5: Brief introduction to 11 benchmark datasets.

Dataset | Classes | Val | Test | Introduction
---|---|---|---|---
ImageNet | 1,000 | 50,000 | 50,000 | Images organized according to WordNet hierarchy
Caltech | 100 | 1,649 | 2,465 | Dataset contains pictures of objects
Pets | 37 | 736 | 3,669 | Dataset consists of different pets
Cars | 196 | 1,635 | 8,041 | Cars images taken from the rear
Flowers | 102 | 1,633 | 2,463 | Images of flowers commonly occurring in UK
Food | 101 | 20,200 | 30,300 | Images of different food organized by types
FGVC | 100 | 3,333 | 3,333 | Benchmark dataset for categorization of aircraft.
SUN | 397 | 3,970 | 19,850 | Benchmark dataset in scene understanding
DTD | 47 | 1,128 | 1,692 | Collection of textural images in the wild
EuroSAT | 10 | 5,400 | 8,100 | Sentinel-2 satellite images covering 13 spectral bands
UCF | 101 | 1,898 | 3,783 | Video clips collected from Youtube

  1. 1.

ImageNet: In this work, we use one of the most highly used subset of ImageNet
namely the "ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
2012-2017 image classification and localization dataset" which is also
referred to in the research literature as ImageNet-1K or ILSVRC2017,
reflecting the original ILSVRC challenge that involved 1,000
classes.111https://en.wikipedia.org/wiki/ImageNet

  2. 2.

UCF: UCF is an action recognition data set of realistic action videos,
collected from YouTube with 101 action categories. The videos in 101 action
categories are grouped into 25 groups, where each group can consist of 4-7
videos of an action. 222https://www.crcv.ucf.edu/data/UCF101.php Following
CoOp[52], we consider the middle frame of each video as our image sample.

  3. 3.

DTD: DTD is a texture database, consisting of 5640 images, organized according
to a list of 47 categories with 120 images per category. The images were
collected from Google and Flickr by entering the proposed attributes and
related terms as search queries. The images were annotated using Amazon
Mechanical Turk in several iterations.
333https://www.robots.ox.ac.uk/~vgg/data/dtd/

  4. 4.

FGVC: FGVC contains 100 images for each of different aircraft model variants,
most of which are airplanes. The (main) aircraft in each image is annotated
with a tight bounding box and a hierarchical airplane model label. Aircraft
models are organized in a four-levels hierarchy: model, variant, family and
manufacturer. 444https://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/

  5. 5.

Flowers: Oxford Flower is an image classification dataset consisting of 102
flower categories. The flowers chosen to be flower commonly occurring in the
United Kingdom. Each class consists of 40-258 images.
555https://paperswithcode.com/dataset/oxford-102-flower

  6. 6.

Cars: The Stanford Cars dataset consists of 196 classes of cars with a total
of 16,185 images. 666https://paperswithcode.com/dataset/stanford-cars

  7. 7.

Eurosat[17]: Eurosat is a dataset and deep learning benchmark for land use and
land cover classification. The dataset is based on Sentinel-2 satellite images
covering 13 spectral bands, consisting of 10 classes with in total 27,000
labeled images.

  8. 8.

SUN: The database contains 108,753 images of 397 categories, used in the Scene
UNderstanding (SUN) benchmark. The number of images varies across categories,
but there are at least 100 images per category.
777https://paperswithcode.com/dataset/sun397

  9. 9.

Pets[21]: The Oxford-Pet Dataset has 37 categories with roughly 200 images in
each class and each image has an associated ground truth annotation of breed,
head ROI, and pixel level segmentation.

  10. 10.

Caltech: Caltech consists of pictures of objects belonging to 101 classes as
well as a background clutter class. Each image is labelled with a single
object. Each class contains roughly 40-800 samples with totally 9,000 images.
888https://www.tensorflow.org/datasets/catalog/caltech101Note that we discard
the "BACKGROUND Google" and "Faces easy classes" from Caltech101 dataset as
described in [52].

  11. 11.

Food: This dataset consists of 101 food categories, with totally 101,000
images. All images were rescaled to have a maximum side length of 512 pixels.
999https://www.tensorflow.org/datasets/catalog/food101

##  Appendix 0.B Overall Experimental Results of Few-shot Learning

As is shown in Table 6, we present the overall improvements achieved by KCL on
five representative methods under 1, 2, 4, 8 and 16-shot settings, including
APE [55]. We can find that, as a training-free plug-and-play module, KCL can
significantly boost existing methods from different types in almost all cases,
which demonstrates its effectiveness in few-shot learning scenarios.

Table 6: Overall experimental results of few-shot learning on 11 benchmark
datasets.

Methods |  Shots |  ImageNet |  UCF |  DTD |  FGVC |  Flowers |  Cars |  Eurosat |  SUN |  Pets |  Caltech |  Food |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
CoOp [52] | 1 | 55.70 | 64.92 | 53.25 | 21.18 | 81.81 | 53.89 | 62.56 | 60.68 | 81.60 | 90.06 | 73.42 | 63.55
$\bm{\Delta}\textbf{+KCL}$(Ours) | +2.84 | +4.73 | +4.73 | +0.18 | +4.22 | +3.40 | +6.75 | +2.77 | +3.11 | +0.49 | +2.76 | +3.27
CLIP-Adapter [12] | 61.94 | 64.58 | 51.60 | 21.27 | 72.03 | 58.77 | 61.23 | 63.83 | 85.61 | 90.10 | 76.80 | 64.34
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.11 | +6.90 | +5.97 | +1.47 | +10.63 | +1.67 | +0.99 | +1.83 | +1.17 | +1.06 | +0.69 | +3.04
Tip-Adapter [50] | 61.83 | 63.86 | 51.12 | 20.94 | 72.63 | 58.60 | 57.68 | 54.08 | 85.25 | 89.25 | 77.57 | 63.89
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.71 | +6.24 | +5.44 | +0.63 | +9.79 | +2.74 | +7.05 | +1.50 | +1.20 | +1.34 | +0.61 | +3.38
APE [55] | 62.00 | 63.86 | 52.90 | 20.97 | 79.33 | 59.89 | 61.15 | 64.18 | 84.88 | 90.22 | 77.27 | 65.15
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.20 | +6.03 | +4.37 | +2.10 | +3.01 | +2.61 | +10.71 | +1.44 | +1.68 | +0.21 | +0.91 | +3.11
MaPLe [20] | 69.67 | 69.94 | 41.31 | 27.18 | 77.30 | 66.99 | 60.75 | 70.05 | 91.55 | 93.06 | 84.12 | 68.35
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.11 | +3.57 | +6.56 | +2.73 | +12.87 | +3.92 | +6.78 | +1.29 | +0.27 | +0.37 | +0.22 | +3.60
CoOp [52] | 2 | 57.36 | 68.44 | 55.85 | 25.11 | 84.94 | 60.17 | 65.49 | 62.03 | 84.19 | 89.49 | 73.58 | 66.05
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.15 | +4.97 | +0.53 | -0.18 | +3.61 | +1.03 | +7.52 | +3.17 | +1.36 | +0.17 | +2.28 | +2.32
CLIP-Adapter [12] | 61.84 | 66.88 | 56.97 | 21.87 | 73.12 | 58.92 | 62.23 | 64.11 | 87.33 | 90.83 | 77.34 | 65.58
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.00 | +2.93 | +3.49 | +0.84 | +3.25 | +1.12 | +5.89 | +1.21 | +0.62 | +0.08 | +0.54 | +1.90
Tip-Adapter [50] | 62.01 | 66.56 | 54.85 | 23.10 | 76.49 | 60.87 | 59.07 | 65.49 | 84.25 | 89.49 | 77.35 | 65.32
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.84 | +4.57 | +2.36 | +0.48 | +8.12 | +1.40 | +9.10 | +1.16 | +2.56 | +0.53 | +0.45 | +2.87
APE [55] | 62.40 | 66.22 | 57.04 | 22.71 | 83.64 | 60.91 | 63.37 | 65.88 | 83.95 | 90.02 | 77.61 | 66.79
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.00 | +4.62 | +3.89 | -0.51 | +4.75 | +1.89 | +4.78 | +0.77 | +2.42 | +1.18 | +0.64 | +2.31
MaPLe [20] | 70.43 | 73.96 | 53.66 | 28.98 | 79.54 | 63.06 | 57.89 | 70.35 | 91.09 | 93.87 | 86.33 | 69.92
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.17 | +3.91 | +4.79 | +5.22 | +12.75 | +5.10 | +20.64 | +2.43 | +0.46 | +0.25 | +0.58 | +5.20
CoOp [52] | 4 | 58.86 | 70.71 | 61.64 | 26.37 | 89.77 | 64.71 | 72.70 | 65.82 | 87.24 | 91.48 | 74.91 | 69.47
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.64 | +4.28 | +0.18 | 0.00 | +1.50 | +0.19 | +4.29 | +0.95 | +0.39 | -0.04 | +1.20 | +1.32
CLIP-Adapter [12] | 62.24 | 69.15 | 57.98 | 22.20 | 82.01 | 60.81 | 73.27 | 66.74 | 87.52 | 91.16 | 77.82 | 68.26
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.98 | +2.94 | -0.12 | +1.71 | +6.42 | +2.57 | +5.15 | +1.16 | +0.35 | +0.73 | +0.41 | +2.02
Tip-Adapter [50] | 62.16 | 68.46 | 59.22 | 23.34 | 82.34 | 62.82 | 69.27 | 65.57 | 84.79 | 90.34 | 77.25 | 67.77
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.49 | +3.18 | +1.60 | -0.09 | +4.26 | +1.02 | +2.74 | +1.13 | +1.61 | +0.49 | +0.49 | +1.53
APE [55] | 62.51 | 68.76 | 60.53 | 24.61 | 87.18 | 64.11 | 71.57 | 66.99 | 86.45 | 91.63 | 77.40 | 69.33
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.10 | +5.60 | +1.23 | +0.71 | +2.26 | +1.40 | +4.31 | +0.47 | -0.40 | -1.04 | +0.84 | +1.41
MaPLe [20] | 70.85 | 76.26 | 58.51 | 31.80 | 84.49 | 69.71 | 62.40 | 72.00 | 92.94 | 94.44 | 86.27 | 72.69
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.44 | +7.14 | +5.44 | +4.11 | +9.26 | +4.44 | +16.59 | +2.08 | +0.03 | +1.02 | +0.14 | +4.69
CoOp [52] | 8 | 60.30 | 74.76 | 64.78 | 30.06 | 93.34 | 69.83 | 78.60 | 68.03 | 86.67 | 92.13 | 75.92 | 72.22
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.96 | +3.85 | +0.65 | +0.57 | +0.45 | -0.17 | +1.17 | +0.49 | +0.49 | -0.08 | +0.97 | +0.85
CLIP-Adapter [12] | 62.83 | 73.25 | 62.12 | 23.85 | 80.43 | 65.18 | 80.93 | 67.97 | 88.28 | 92.09 | 78.23 | 70.46
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.88 | +2.14 | +1.18 | +3.60 | +9.83 | +1.38 | +1.17 | +0.43 | +0.22 | +0.24 | +0.40 | +1.95
Tip-Adapter [50] | 62.42 | 70.58 | 60.99 | 27.69 | 87.66 | 64.68 | 69.85 | 66.74 | 86.45 | 91.08 | 77.72 | 69.62
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.49 | +2.75 | +1.01 | +0.48 | +1.26 | -0.12 | +2.78 | +0.07 | +0.47 | -0.37 | +0.40 | +0.83
APE [55] | 62.76 | 71.64 | 64.78 | 28.86 | 90.35 | 66.74 | 72.60 | 68.01 | 86.63 | 91.08 | 78.10 | 71.05
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.85 | +4.81 | +0.59 | +0.09 | +0.72 | +0.34 | +2.51 | -0.29 | +0.75 | +0.48 | +0.16 | +1.00
MaPLe [20] | 71.42 | 79.25 | 62.23 | 32.31 | 90.38 | 71.27 | 76.05 | 72.61 | 93.40 | 94.60 | 87.17 | 75.51
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.11 | +4.89 | +5.80 | +8.97 | +4.34 | +4.74 | +9.76 | +2.43 | +0.11 | +0.45 | +0.13 | +3.88
CoOp [52] | 16 | 61.12 | 78.61 | 68.68 | 36.03 | 95.66 | 76.26 | 85.58 | 70.02 | 88.25 | 93.67 | 76.91 | 75.52
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.77 | +1.25 | +0.17 | +0.18 | +0.04 | -0.24 | +0.21 | +0.48 | +0.52 | -0.24 | +0.86 | +0.36
CLIP-Adapter [12] | 63.24 | 76.53 | 66.55 | 29.01 | 92.94 | 71.51 | 84.67 | 69.79 | 88.53 | 92.66 | 78.59 | 74.00
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.93 | +4.07 | +0.03 | +4.02 | +0.73 | +0.89 | +0.53 | +0.46 | +0.19 | 0.00 | +0.49 | +1.12
Tip-Adapter [50] | 63.00 | 72.16 | 63.42 | 30.75 | 89.81 | 66.73 | 74.83 | 68.13 | 86.48 | 91.08 | 77.81 | 71.29
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.04 | +1.72 | +0.82 | +0.03 | +0.24 | +1.04 | –0.05 | +1.04 | +0.96 | -0.25 | +0.14 | +0.52
APE [55] | 63.45 | 74.57 | 66.19 | 31.80 | 92.00 | 70.48 | 77.54 | 69.30 | 87.23 | 91.68 | 78.44 | 72.94
$\bm{\Delta}\textbf{+KCL}$(Ours) | +0.40 | +3.20 | +1.30 | +0.18 | +0.08 | +0.12 | +1.68 | +0.13 | -0.42 | +0.04 | -0.22 | +0.59
MaPLe [20] | 71.83 | 81.10 | 67.91 | 35.55 | 93.59 | 73.15 | 86.02 | 74.56 | 94.22 | 95.54 | 87.27 | 78.24
$\bm{\Delta}\textbf{+KCL}$(Ours) | +1.58 | +2.99 | +5.20 | +9.39 | +1.86 | +6.93 | +4.34 | +1.83 | -0.57 | +0.16 | +0.27 | +3.08

##  Appendix 0.C Ablation Study on Visual Backbones

In this section, we conduct ablation study on four different visual backbone
structures, i.e. ResNet-50 [16], ResNet-101 [16], ViT-B/32 [9] and ViT-B/16
[9]. As is shown in Table 7, we present the full performance with different
backbone structures under 1, 2, 4, 8 and 16-shot settings. We can find that
KCL still achieves remarkable improvements based on various backbones.

Table 7: Ablation study of visual backbones on zero-shot CLIP [33] and CoOp
[52].

Zero-shot
---
Methods | CLIP | CLIP+KCL
Datasets | Shots | RN50 | RN101 | ViT/16 | ViT/32 | RN50 | RN101 | ViT/16 | ViT/32
UCF | 0 | 59.69 | 55.46 | 67.06 | 63.57 | 67.06 | 63.23 | 72.98 | 67.88
Eurosat | 0 | 38.02 | 30.51 | 54.16 | 48.30 | 52.69 | 33.36 | 56.64 | 50.56
Cars | 0 | 57.34 | 61.41 | 65.99 | 61.09 | 59.68 | 65.08 | 66.48 | 61.73
DTD | 0 | 48.52 | 49.88 | 53.31 | 50.06 | 56.21 | 53.31 | 56.80 | 54.79
Few-shot
Methods | CoOp | CoOp+KCL
Datasets | Shots | RN50 | RN101 | ViT/16 | ViT/32 | RN50 | RN101 | ViT/16 | ViT/32
UCF | 1 | 64.92 | 67.22 | 72.83 | 70.02 | 69.65 | 73.22 | 80.28 | 77.21
2 | 68.44 | 72.91 | 76.92 | 74.12 | 73.41 | 75.63 | 81.55 | 76.29
4 | 70.71 | 75.31 | 79.80 | 77.43 | 74.99 | 78.19 | 82.13 | 80.44
8 | 74.76 | 79.65 | 83.16 | 80.49 | 78.61 | 81.26 | 84.96 | 82.66
16 | 78.61 | 81.34 | 84.99 | 82.87 | 79.86 | 82.18 | 85.75 | 83.40
Eurosat | 1 | 62.56 | 57.17 | 58.80 | 61.44 | 69.31 | 63.70 | 65.44 | 69.67
2 | 65.49 | 64.30 | 69.54 | 67.79 | 73.01 | 72.36 | 79.91 | 76.00
4 | 72.70 | 73.62 | 82.17 | 80.42 | 76.99 | 76.70 | 84.64 | 81.86
8 | 78.60 | 78.20 | 86.05 | 81.70 | 79.77 | 78.44 | 87.09 | 82.10
16 | 85.58 | 80.52 | 87.48 | 84.17 | 85.79 | 82.19 | 87.78 | 84.96
Cars | 1 | 53.89 | 63.57 | 66.83 | 59.05 | 57.29 | 64.84 | 67.55 | 61.25
2 | 60.17 | 68.00 | 71.17 | 63.09 | 61.20 | 68.74 | 71.57 | 63.54
4 | 64.71 | 71.05 | 74.74 | 66.55 | 64.90 | 71.30 | 74.88 | 67.03
8 | 69.83 | 76.76 | 80.70 | 73.44 | 69.66 | 76.87 | 80.74 | 73.47
16 | 76.26 | 81.68 | 83.36 | 78.61 | 76.02 | 81.71 | 83.67 | 78.77
DTD | 1 | 53.25 | 55.50 | 57.74 | 55.44 | 57.98 | 58.22 | 60.93 | 57.86
2 | 55.85 | 58.98 | 60.64 | 58.27 | 56.38 | 59.34 | 62.94 | 62.53
4 | 61.64 | 63.06 | 64.72 | 63.30 | 61.82 | 64.01 | 65.84 | 65.13
8 | 64.78 | 67.55 | 70.39 | 66.55 | 65.43 | 67.73 | 69.27 | 66.55
16 | 68.68 | 70.33 | 73.17 | 69.74 | 68.85 | 70.33 | 73.23 | 69.33

##  Appendix 0.D Analysis of Knowledge Completion Capability

In Fig. 2a of main paper, we compare the knowledge completion performance of
KCL with generation-based C2A [36]. C2A, building up on the combination of
CLIP-Adapter [12] and Tip-Adapter [50], takes advantages of stable diffusion
[35] and BLIP [22] to generate images as visual knowledge completion. Here we
present the overall results on 11 datasets in Fig. 8. We can find that KCL
module embedded in CLIP-Adapter can achieve competitive performance with C2A,
while KCL module embedded in CoOp can surpass C2A in most cases. Note that KCL
has no need for generative models with high computational costs.

Figure 8: Knowledge completion capability comparison with C2A [36].

##  Appendix 0.E Hyper-parameter Sensitivity Analysis

In Fig. 2b of main paper, we show the averaged results of hyper-parameter
sensitivity analysis. In this section, we show the results of on 11 datasets
based on MaPLe in Fig. 9. $\dagger$ means all hyper-parameters are fixed to
$\alpha$=$\beta$=$\mu$=$\lambda$=1.0 without tuning hyper-parameters. We can
find that hyper-parameters have tiny impacts towards KCL, which demonstrates
the robustness of KCL.

Figure 9: Overall experimental results of hyper-parameter sensitivity
analysis.

##  Appendix 0.F Textual Prompts Provided by CuPL

The textual prompts used in this work is provided by CuPL [32], which is
generated by querying GPT-3 [7].

Table 8: CuPL [32] hand-written prompts.

Dataset | GPT-3 prompts
---|---
ImageNet | ‘Describe what a {} looks like’
‘How can you identify {}?’
‘What does {} look like?’
‘Describe an image from the internet of a {}’
‘A caption of an image of {}: ’
UCF | ‘What does a person doing {} look like’
‘Describe the process of {}’
‘How does a person {}’
DTD | ‘What does a {} material look like?’
‘What does a {} surface look like?’
‘What does a {} texture look like?’
‘What does a {} object look like?’
‘What does a {} thing look like?’
‘What does a {} pattern look like?’
FGVC | ‘Describe a {} aircraft’
Flowers | ‘What does a {} flower look like’
‘Describe the appearance of a {}’
‘A caption of an image of {}’
‘Visually describe a {}, a type of flower’
Cars | ‘How can you identify a {}’
‘Description of a {}, a type of car’
‘A caption of a photo of a {}:’
‘What are the primary characteristics of a {}?’
‘Description of the exterior of a {}’
‘What are the identifying characteristics of a {}, a type of car?’
‘Describe an image from the internet of a {}’
‘What does a {} look like?’
‘Describe what a {}, a type of car, looks like’
Eurosat | ‘Describe an aerial satellite view of {}’
‘How does a satellite photo of a {} look like’
‘Visually describe a centered satellite view of a {}’
SUN | ‘Describe what a {} looks like’
‘How can you identify a {}?’
‘Describe a photo of a {}’
Pets | ‘Describe what a {} pet looks like’
‘Visually describe a {}, a type of pet’
Caltech | ‘Describe what a {} looks like’
‘What does a {} look like’
‘Describe a photo of a {}’
Food | ‘Describe what a {} looks like’
‘Visually describe a {}’
‘How can you tell that the food in this photo is a {}?’

## References

  * [1] Bossard, L., Guillaumin, M., Van Gool, L.: Food-101–mining discriminative components with random forests. In: European Conference on Computer Vision. pp. 446–461 (2014)
  * [2] Chen, G., Yao, W., Song, X., Li, X., Rao, Y., Zhang, K.: Plot: Prompt learning with optimal transport for vision-language models. In: The Eleventh International Conference on Learning Representations (2022)
  * [3] Chen, H., Li, H., Li, Y., Chen, C.: Sparse spatial transformers for few-shot learning. Science China Information Sciences 66(11), 210102 (2023)
  * [4] Chen, H., Xu, Z., Gu, Z., Li, Y., Meng, C., Zhu, H., Wang, W., et al.: Diffute: Universal text editing diffusion model. Advances in Neural Information Processing Systems 36 (2024)
  * [5] Chen, L., Du, Q., Lou, Y., He, J., Bai, T., Deng, M.: Mutual nearest neighbor contrast and hybrid prototype self-training for universal domain adaptation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 6248–6257 (2022)
  * [6] Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3606–3613 (2014)
  * [7] Dale, R.: Gpt-3: What’s it good for? Natural Language Engineering 27(1), 113–118 (2021)
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255 (2009)
  * [9] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
  * [10] Fan, L., Krishnan, D., Isola, P., Katabi, D., Tian, Y.: Improving clip training with language rewrites. Advances in Neural Information Processing Systems 36 (2024)
  * [11] Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: 2004 conference on computer vision and pattern recognition workshop. pp. 178–178 (2004)
  * [12] Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., Qiao, Y.: Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision pp. 1–15 (2023)
  * [13] Geng, S., Yuan, J., Tian, Y., Chen, Y., Zhang, Y.: Hiclip: Contrastive language-image pretraining with hierarchy-aware attention. arXiv preprint arXiv:2303.02995 (2023)
  * [14] Gowda, K., Krishna, G.: The condensed nearest neighbor rule using the concept of mutual nearest neighborhood (corresp.). IEEE Transactions on Information Theory 25(4), 488–490 (1979)
  * [15] Guo, Z., Zhang, R., Qiu, L., Ma, X., Miao, X., He, X., Cui, B.: Calip: Zero-shot enhancement of clip with parameter-free attention. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 746–754 (2023)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [17] Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217–2226 (2019)
  * [18] Huang, R., Long, Y., Han, J., Xu, H., Liang, X., Xu, C., Liang, X.: Nlip: noise-robust language-image pre-training. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 926–934 (2023)
  * [19] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916 (2021)
  * [20] Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: Multi-modal prompt learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19113–19122 (2023)
  * [21] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision workshops. pp. 554–561 (2013)
  * [22] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900 (2022)
  * [23] Li, Y., Fan, H., Hu, R., Feichtenhofer, C., He, K.: Scaling language-image pre-training via masking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23390–23400 (2023)
  * [24] Li, Y., Yang, Y., Li, H., Chen, H., Xu, L., Li, L., Li, Y., Guo, Y.: Transductive aesthetic preference propagation for personalized image aesthetics assessment. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 896–904 (2022)
  * [25] Liu, J., Zhang, Y., Chen, J.N., Xiao, J., Lu, Y., A Landman, B., Yuan, Y., Yuille, A., Tang, Y., Zhou, Z.: Clip-driven universal model for organ segmentation and tumor detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21152–21164 (2023)
  * [26] Liu, Y., Lee, J., Park, M., Kim, S., Yang, E., Hwang, S.J., Yang, Y.: Learning to propagate labels: Transductive propagation network for few-shot learning. In: International Conference on Learning Representations (2019)
  * [27] Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)
  * [28] Menon, S., Vondrick, C.: Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 (2022)
  * [29] Miyai, A., Yu, Q., Irie, G., Aizawa, K.: Locoop: Few-shot out-of-distribution detection via prompt learning. Advances in Neural Information Processing Systems 36 (2024)
  * [30] Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian conference on computer vision, graphics & image processing. pp. 722–729 (2008)
  * [31] Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.: Cats and dogs. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 3498–3505 (2012)
  * [32] Pratt, S., Covert, I., Liu, R., Farhadi, A.: What does a platypus look like? generating customized prompts for zero-shot image classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15691–15701 (2023)
  * [33] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763 (2021)
  * [34] Ren, M., Triantafillou, E., Ravi, S., Snell, J., Swersky, K., Tenenbaum, J.B., Larochelle, H., Zemel, R.S.: Meta-learning for semi-supervised few-shot classification. In: International Conference on Learning Representations (2018)
  * [35] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [36] Roy, A., Shah, A., Shah, K., Roy, A., Chellappa, R.: Cap2aug: Caption guided image to image data augmentation. arXiv preprint arXiv:2212.05404v2 (2023)
  * [37] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems 35, 25278–25294 (2022)
  * [38] Seth, A., Hemani, M., Agarwal, C.: Dear: Debiasing vision-language models with additive residuals. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6820–6829 (2023)
  * [39] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
  * [40] Udandarao, V., Gupta, A., Albanie, S.: Sus-x: Training-free name-only transfer of vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2725–2736 (2023)
  * [41] Vapnik, V.: transductive inference and semi-supervised learning. Semi-Supervised Learning pp. 453–472 (2006)
  * [42] Vapnik, V.: The nature of statistical learning theory (2013)
  * [43] Wang, T.: Learning to detect and segment for open vocabulary object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7051–7060 (2023)
  * [44] Wang, Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., Liu, T.: Cris: Clip-driven referring image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11686–11695 (2022)
  * [45] Wang, Z., Liang, J., Sheng, L., He, R., Wang, Z., Tan, T.: A hard-to-beat baseline for training-free clip-based adaptation. arXiv preprint arXiv:2402.04087 (2024)
  * [46] Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition. pp. 3485–3492 (2010)
  * [47] Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., Huang, J.: Vision-language pre-training with triple contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15671–15680 (2022)
  * [48] Yao, H., Zhang, R., Xu, C.: Visual-language prompt tuning with knowledge-guided context optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6757–6767 (2023)
  * [49] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)
  * [50] Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., Li, H.: Tip-adapter: Training-free adaption of clip for few-shot classification. In: European Conference on Computer Vision. pp. 493–510 (2022)
  * [51] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16816–16825 (2022)
  * [52] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision 130(9), 2337–2348 (2022)
  * [53] Zhou, Z.H., Zhou, Z.H.: Semi-supervised learning. Machine Learning pp. 315–341 (2021)
  * [54] Zhou, Z., Lei, Y., Zhang, B., Liu, L., Liu, Y.: Zegclip: Towards adapting clip for zero-shot semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11175–11185 (2023)
  * [55] Zhu, X., Zhang, R., He, B., Zhou, A., Wang, D., Zhao, B., Gao, P.: Not all features matter: Enhancing few-shot clip with adaptive prior refinement. arXiv preprint arXiv:2304.01195 (2023)


# Supplemental materials: Ferroelastic twin angles at the surface of CaTiO3
quantified by PhotoEmission Electron Microscopy

##  I Photoelectron kinetic vector

Electrons with higher emission angles have lower kinetic energy as measured on
the scale of $E-E_{F}$ making the threshold appear at higher energy, as shown
in figure 1. Electron emission normal to the sample surface is favored when
the CA is centered and electrons emitted off-normal are favored when the CA is
off-centered. The extractor field is measured perpendicular to the average
sample surface so photoelectrons with higher emission angles have lower
kinetic energy ($E_{k}\approx v_{k}^{2}$) as measured on the scale of
$E-E_{F}$ and the threshold will appear at higher energy.

Figure 1: Schematic of the electron kinetic vector for a normal emission and
an emission with an angle $\alpha_{i}$.

##  II Model of the photoemission threshold evolution with CA lateral
displacement

Let an electron emitted at an angle $\alpha_{\mathrm{0}}$ make an angle
$\alpha_{\mathrm{i}}$ with respect to the PEEM electron optical axis and cross
the back focal plane at x from the axis. To extract an $x_{tilt}$ value at
each pixel of the analyzed area, we model the electron path in the PEEM. The
relevant parameters are the electron kinetic energy $E$, the PEEM column
energy $E_{i}$, the magnification in the first image plane $M$, the CA
position in the back focal plane $x$.

The phase conservation equation of the photoelectrons is written in equation
1. For simplicity, we assume an isotropic emission of electrons from 0° to
90°, i.e. $\alpha_{0}=90^{\circ}$ and the magnification of the image in the
first image plane is given by the ratio between the object size $r_{0}$ and
the first intermediate image size $r_{i}$. For a tilted surface, the optical
axis in the back focal plane is shifted by a distance $x_{tilt}$.

$\begin{split}\sqrt{E}r_{0}sin(\alpha_{0})&=\sqrt{E_{i}}r_{i}sin(\alpha_{i})\\\
sin(\alpha_{i})&=\frac{x}{l}\end{split}$ (1)

The shift in measured photoemission threshold $E_{Thr}$ with the CA position
for a surface with tilt angle $\alpha_{\mathrm{tilt}}$ is then given by
equation LABEL:suppeq:2.

$\begin{split}&\Delta{E_{Thr}}=\frac{E_{i}M^{2}}{l^{2}}(x+r_{ap}-x_{tilt})^{2},\>for\>x\leq-
r_{ap}+x_{tilt}\\\
&\Delta{E_{Thr}}=\frac{E_{i}M^{2}}{l^{2}}(x-r_{ap}-x_{tilt})^{2},\>for\>x\geq
r_{ap}+x_{tilt}\\\ &E_{Thr}=0,\>for-r_{ap}+x_{tilt}\leq x\leq
r_{ap}+x_{tilt}\end{split}$ (2)

It is a parabola with a central flat range fixed by $r_{ap}$. In our PEEM set-
up $l=165mm$, $M=32$, and $E_{i}=2000eV$. Fig. 2 shows the experimental data
for $x_{tilt}=0$ from a pixel in the domain $D_{1}$ together with, in red, eqn
LABEL:suppeq:2. Note that the best fit is obtained for $r_{ap}$ = 55 $\mu$m
rather than the nominal radius of 75 $\mu$m. We fit the increasing threshold
energy by two parabolic branches out with $\pm r_{ap}$ and calculate the off-
centering from the optical axis $x_{tilt}$ from the midpoint between the two
branches.

Figure 2: Experimental variation in photoemission threshold as a function of
the 150 $\mu$m CA position in the diffraction plane from -230 $\mu$m to 230
$\mu$m. In red plot of eqn.1 using $r_{ap}$ = 55 $\mu$m.

##  III Model of the surface tilt perturbation at the back focal plane

### Triangle Perturbation

Figure 3: Periodic triangular structure simulating ferroelastic topography and
representation of the triangular function Fourier series for $n_{max}$ = 1 to
3. The surface is tilted by the angle $\alpha_{tilt}$ to the z-axis.

To recover $\alpha_{\mathrm{tilt}}$ from $x_{\mathrm{tilt}}$ we model the roof
topography of the $CaTiO_{3}$ surface as a perturbation of the electric field
above the surface and solve the equations of motion for the photoemitted
electrons, We assume a periodic structure (Triangular Function) with a
homogeneous work function of period $2L$ and amplitude $0.5\mathrm{H}$. The
triangular structure is defined as

$z_{p}(x)=\left\\{\begin{array}[]{cc}H/L\left(x-2Ln+\frac{1}{2}L\right)&x\in[2Ln-L,2Ln],n\in\mathbb{Z}\\\
-H/L\left(x-2Ln-\frac{1}{2}L\right)&x\in[2Ln,2Ln+L],n\in\mathbb{Z}\end{array}\right.$

With $x$ and $y$ being the coordinates in the surface, $z$ normal to the
surface, and $z=0$ being in the surface. The $x$-coordinate is the direction
of the periodic structure (see Figure 3). The tilt angle to the surface normal
is

$\alpha_{tilt}=\left\\{\begin{array}[]{cc}\tan^{-1}\left(\frac{\mathrm{H}}{\mathrm{L}}\right)\approx\frac{\mathrm{H}}{\mathrm{L}}&x\in[2\operatorname{Ln}-L,2Ln],n\in\mathbb{Z}\\\
\tan^{-1}\left(-\frac{\mathrm{H}}{\mathrm{L}}\right)\approx-\frac{\mathrm{H}}{\mathrm{L}}&x\in[2Ln,2Ln+L],n\in\mathbb{Z}\end{array}\right.$

assuming the H<<L.

### Fourier Series Triangular Function

The Triangular Function can be expressed as a Fourier Series:

$z_{p}(x)=\frac{4H}{\pi^{2}}\sum_{n=1}^{\infty}\frac{1}{(2n-1)^{2}}\cos((2n-1)kx)$

With $k=2\pi/(2L)$. We will fix $n_{max}=1$ for the triangular Fourier series
as in the vicinity of ($z=0$,$\frac{x}{L}=\pm 0.5$), this is a good
approximation of the topography. Higher order terms would be necessary near to
the apices which correspond to the twin boundaries.

### Potential between Sample and Extractor

For a flat (unperturbed) sample, the potential can be written as

$\Phi_{0}(x,y,z)=-U_{0}\frac{z}{d}$

where $U_{0}$ is the voltage and $d$ the distance between sample and objective
lens. In the following discussion, we neglect the $y$-coordinate, considering
only a two-dimensional problem in x-z. With the factory-roof structure, we get
a perturbation potential at $z=0$ which can be written as [Bok1968] :

$\Phi_{p}(x,z=0)=\Phi_{p0}+\sum_{n=1}^{\infty}\Phi_{pn}\cos nkx$

This leads to the solution :

$\Phi(x,z)=-U_{0}\frac{z}{d}+\Phi_{p0}+\sum_{n=1}^{\infty}\left[\Phi_{pn}\cos
nkx\exp(-nkz)\right]$

### Boundary conditions

In order to determine the coefficients $\Phi_{pn}$ we substitute for $z_{p}$ :

$\displaystyle x\in[-L,0]$
$\displaystyle\Phi\left(x,z_{p}(x)\right)=0=-\frac{U_{0}}{d}\frac{\mathrm{H}}{\mathrm{L}}\left(x+\frac{1}{2}L\right)+\Phi_{p0}+\sum_{n=1}^{\infty}\left[\Phi_{pn}\cos
nkx\exp\left(-nk\alpha_{tilt}\left(x+\frac{1}{2}L\right)\right)\right];$
$\displaystyle x\in[0,L]$
$\displaystyle\Phi\left(x,z_{p}(x)\right)=0=\frac{U_{0}}{d}\frac{\mathrm{H}}{\mathrm{L}}\left(x-\frac{1}{2}L\right)+\Phi_{p0}+\sum_{n=1}^{\infty}\left[\Phi_{pn}\cos
nkx\exp\left(nk\alpha_{tilt}\left(x-\frac{1}{2}L\right)\right)\right];$

In the vicinity of $x\approx-1/2L$ we can neglect the influence of the term

$\exp\left(-nk\alpha_{tilt}\left(x+\frac{1}{2}L\right)\right)\approx 1$

And expressing the linear term of the potential as Fourier Series

$-\frac{U_{0}}{d}\alpha_{tilt}\left(x+\frac{1}{2}L\right)=-\frac{U_{0}}{d}\frac{4H}{\pi^{2}}\sum_{n=1}^{\infty}\frac{1}{(2n-1)^{2}}\cos((2n-1)kx)$

The comparison of coefficients yields

$\Phi_{pn}=\frac{U_{0}}{d}\frac{4H}{\pi^{2}}\frac{1}{(2n-1)^{2}}$

### Electric Field

From the electric potential, we yield the electric field:

$\begin{gathered}\frac{\partial}{\partial\mathrm{z}}\Phi(x,z)=\frac{\partial}{\partial\mathrm{z}}\Phi_{o}(z)+\frac{\partial}{\partial\mathrm{z}}\Phi_{p}(x,z)\\\
\frac{\partial}{\partial\mathrm{x}}\Phi(x,z)=\frac{\partial}{\partial\mathrm{x}}\Phi_{p}(x,z)\end{gathered}$

In the $z$ direction we can neglect the perturbation contribution:

$E_{z}(z)=\frac{\partial}{\partial\mathrm{z}}\Phi(x,z)\approx\frac{\partial}{\partial\mathrm{z}}\Phi_{o}(z)=-\frac{U_{0}}{d}$

For the $x$ direction we have:

$\begin{gathered}E_{x}(x,z)=\frac{\partial}{\partial\mathrm{x}}\Phi_{p}(x,z)\\\
=\frac{\partial}{\partial\mathrm{x}}\sum_{n=1}^{\infty}\left[\Phi_{pn}\cos
nkx\exp(-nkz)\right]\\\ =\sum_{n=1}^{\infty}\left[-nk\Phi_{pn}\sin
nkx\exp(-nkz)\right]\\\
=\sum_{n=1}^{\infty}\left[-\frac{U_{0}}{d}\frac{4H}{\pi^{2}}\frac{nk}{(2n-1)^{2}}\sin
nkx\exp(-nkz)\right]\\\
=-\frac{U_{0}}{d}\frac{4H}{\pi^{2}}k\sum_{n=1}^{\infty}\left[\frac{n}{(2n-1)^{2}}\sin
nkx\exp(-nkz)\right]\end{gathered}$

### Equations of motion

We use the classical approach since in the vicinity of the perturbated surface
the electrons do not have high kinetic energy:

$\begin{gathered}-eE_{x}(x,z)=m_{e}\frac{\partial^{2}x(z)}{\partial\mathrm{t}^{2}}\\\
-eE_{z}(z)=-e\left(-\frac{U_{0}}{d}\right)=m_{e}\frac{\partial^{2}z}{\partial\mathrm{t}^{2}}\end{gathered}$

### Initial Conditions

The momentum of an electron leaving the sample at $t=0$ at $z=0$ and $x=-1/2L$
with $E$ being the kinetic energy when leaving the sample surface, can be
written as

$\displaystyle p_{z0}=\sqrt{2m_{e}E}\cos\left(\alpha_{0}+\alpha_{tilt}\right)$
$\displaystyle p_{x0}=\sqrt{2m_{e}E}\sin\left(\alpha_{0}+\alpha_{tilt}\right)$

With the electron mass $m_{e}$, tilt angle $\alpha_{tilt}$ and take off angle
$\alpha_{0}\in[-\pi/2,\pi/2]$.

### Solution for the z-axis

In $z$ direction integration of the momentum over time yields

$\displaystyle p_{z}(t)=m_{e}\frac{\partial z}{\partial\mathrm{t}}$
$\displaystyle=\sqrt{2m_{e}E}\cos\left(\alpha_{0}+\alpha_{tilt}\right)-\int_{0}^{t}eE_{z}(z)dt^{\prime}$
$\displaystyle p_{z}(t)$
$\displaystyle=\sqrt{2m_{e}E}\cos\left(\alpha_{0}+\alpha_{tilt}\right)+e\frac{U_{0}}{d}t$

Second integration:

$z(t)=\int_{0}^{t}\frac{p_{z}(t)}{m_{e}}dt^{\prime}=\sqrt{2m_{e}E}\cos\left(\alpha_{0}+\alpha_{tilt}\right)t+2\frac{eU_{0}}{m_{e}d}t^{2}$

For threshold emission we neglect the contribution of the initial electron
momentum and work with the approximation:

$z(t)=2\frac{eU_{0}}{m_{e}d}t^{2}$

### Solution for the $x$-axis

Therefore the equation of motion in $\mathrm{x}$ direction becomes

$\displaystyle m_{e}\frac{\partial^{2}x(z(t))}{\partial\mathrm{t}^{2}}$
$\displaystyle=-eE_{x}(x,z)=-e\left(-\frac{U_{0}}{d}\frac{4H}{\pi^{2}}k\sum_{n=1}^{\infty}\left[\frac{n}{(2n-1)^{2}}\sin
nkx(t)\exp(-nkz(t))\right]\right)$
$\displaystyle=\frac{eU_{0}}{d}\frac{4H}{\pi^{2}}k\sum_{n=1}^{\infty}\left[\frac{n}{(2n-1)^{2}}\sin
nkx(t)\exp\left(-nk2\frac{eU_{0}}{m_{e}d}t^{2}\right)\right]$

Integration over $\mathrm{t}$ yields the momentum in $\mathrm{x}$ direction:

$\displaystyle p_{x}(t)=m_{e}\frac{\partial x(t)}{\partial\mathrm{t}}$
$\displaystyle=\sqrt{2m_{e}E}\sin\left(\alpha_{0}+\alpha_{tilt}\right)+\frac{eU_{0}}{d}\frac{4H}{\pi^{2}}k\sum_{n=1}^{\infty}\left[\frac{n}{(2n-1)^{2}}\int_{0}^{t}\sin\left(nkx\left(t^{\prime}\right)\right)\exp\left(-nk2\frac{eU_{0}}{m_{e}d}t^{\prime
2}\right)dt^{\prime}\right]$

### Approximation for the $x$-axis

We restrict the series of the electric field to $n=1$ :

$p_{x}(t)\approx\sqrt{2m_{e}E}\sin\left(\alpha_{0}+\alpha_{tilt}\right)+\frac{eU_{0}}{d}\frac{4H}{\pi^{2}}k\int_{0}^{t}\sin\left(kx\left(t^{\prime}\right)\right)\exp\left(-k2\frac{eU_{0}}{m_{e}d}t^{\prime
2}\right)dt^{\prime}$

In the proximity of $x(t)=-1/2L(x(t)=1/2L)$ we can approximate

$\begin{array}[]{ll}\sin(kx(t))\approx-1;&x(t)\approx-1/2L\\\
\sin(kx(t))\approx 1;&x(t)\approx 1/2L\end{array}$

Then, the momentum in $x$-direction simplifies to

$p_{x}(t)\approx\sqrt{2m_{e}E}\sin\left(\alpha_{0}+\alpha_{tilt}\right)+\alpha_{tilt}\frac{eU_{0}}{d}\frac{4}{\pi}\int_{0}^{t}\exp\left(-k2\frac{eU_{0}}{m_{e}d}t^{\prime
2}\right)dt^{\prime}$

Integration of the Gaussian can be approximated to $t\rightarrow\infty$ :

$\int_{0}^{\infty}\exp\left(-k2\frac{eU_{0}}{m_{e}d}t^{2}\right)dt=\frac{2}{\sqrt{\pi}}\sqrt{\frac{m_{e}d}{2eU_{0}k}}=\frac{2}{\pi}\sqrt{\frac{m_{e}dL}{2eU_{0}}}$

Hence, the momentum in $x$-direction can be written as

$\displaystyle
p_{x}\approx\sqrt{2m_{e}E}\sin\left(\alpha_{0}+\alpha_{tilt}\right)+\alpha_{tilt}\frac{eU_{0}}{d}\frac{4}{\pi}\frac{2}{\pi}\sqrt{\frac{m_{e}dL}{2eU_{0}}}$
$\displaystyle
p_{x}\approx\sqrt{2m_{e}E}\sin\left(\alpha_{0}+\alpha_{tilt}\right)+\alpha_{tilt}\frac{4}{\pi^{2}}\sqrt{2m_{e}eU_{0}}\sqrt{\frac{L}{d}}$

The wave vector $k_{x}=p_{x}/\hbar$ in $x$-direction follows as

$k_{x}\approx\frac{\sqrt{2m_{e}E}}{\hbar}\sin\left(\alpha_{0}+\alpha_{tilt}\right)+\alpha_{tilt}\frac{4}{\pi^{2}}\frac{\sqrt{2m_{e}eU_{0}}}{\hbar}\sqrt{\frac{L}{d}}$

The wave vector depends on two terms where the first term has a weak
dependence on the surface tilt ($\alpha_{tilt}$ is in the order of 1°) and the
second term depends on the surface tilt and period $2L$ of the triangular
topography. We can approximate the topography perturbation defined by the
surface tilt $\alpha_{tilt}$ on the wave vector by approximating the first
term as constant. The wave-vector difference from the surface topography,
$\Delta{k_{x}}$, is expressed as:

$\Delta{k_{x}}\approx\alpha_{tilt}\frac{4}{\pi^{2}}\frac{\sqrt{2m_{e}eU_{0}}}{\hbar}\sqrt{\frac{L}{d}}$

We use the factor $c_{k}=4.14\>\mathring{A}^{-1}.mm^{-1}$, which converts in
the PEEM an angular distance in the reciprocal space into a shift in the back
focal plane, defined as $\Delta{k_{x}}=c_{k}\cdot x_{tilt}$. We can then
express the surface domain tilt angle $\alpha_{tilt}$ from the CA shift in the
back focal plane $x_{tilt}$:

$\alpha_{tilt}=c_{k}\frac{\pi^{2}}{4}(\frac{\sqrt{2m_{e}eU_{o}}}{\hbar}\cdot\sqrt{\frac{L}{d}})^{-1}\cdot
x_{tilt}$

##  IV Standard deviation

The map of the standard deviation $\sigma$ of $\alpha_{\mathrm{tilt}}$ is
shown in figure 4. From the $\sigma$ map, the calculated $\alpha_{tilt}$ error
is approximately of $\pm$ 0.07° in the domains with an error extrema of $\pm$
0.1° around the ridge/valley domain walls. This can be understood since twin
walls mark the transition between two tilt angles, locally modifying the field
lines giving rise to a larger spread of angular values.

Figure 4: Map of the standard deviation $\sigma$ of $\alpha_{tilt}$ as
determined by PEEM.

##  V Possible strain configurations in CaTiO3

There are six possible strain states $S_{i}$ in orthorhombic $CaTiO_{3}$ which
are derived from the strain tensor compatibility across the symmetry allowed
twin walls [Sapriel1975, Yokota2014].

$\displaystyle\begin{split}S_{i}:\begin{pmatrix}+0.00057&+0.0055&0\\\
+0.0055&+0.00057&0\\\ 0&0&-0.00114\end{pmatrix}\\\
S_{ii}:\begin{pmatrix}+0.00057&-0.0055&0\\\ -0.0055&+0.00057&0\\\
0&0&-0.00114\end{pmatrix}\\\ S_{iii}:\begin{pmatrix}+0.00057&0&-0.0055\\\
0&-0.00114&0\\\ -0.0055&0&+0.00057\end{pmatrix}\\\
S_{iv}:\begin{pmatrix}-0.00114&0&0\\\ 0&+0.00057&+0.0055\\\
0&+0.0055&+0.00057\end{pmatrix}\\\ S_{v}:\begin{pmatrix}+0.00057&0&+0.0055\\\
0&-0.00114&0\\\ +0.0055&0&+0.00057\end{pmatrix}\\\
S_{vi}:\begin{pmatrix}-0.00114&0&0\\\ 0&+0.00057&-0.0055\\\
0&-0.0055&+0.00057\end{pmatrix}\end{split}$ (3)

From the possible spontaneous strain directions, we can deduce the possible
domain pairs ($S_{a},S_{b}$) and calculate the difference in their spontaneous
strain tensors $\Delta\epsilon=\epsilon_{a}-\epsilon_{b}$. For each pair, the
equations of the two possible compatible domain walls, one for each compatible
orientation, are calculated from the mechanical compatibility equation
$\Delta\epsilon_{ab}x_{a}x_{b}=0$ [Sapriel1975, Yokota2014]. The possible
strain pairs are regrouped in the following table LABEL:tab:1.

[ caption = Domain wall plane equations for all possible combinations of
strain tensors, wall angle with respect to (001), and twin angle., label =
tab:1, ] width = colspec = Q[171]Q[404]Q[108]Q[110]Q[142], row1 = Silver, row2
= Silver, row3 = c, row4 = c, row5 = c, row6 = c, row7 = c, row8 = c, row9 =
c, row10 = c, row11 = c, row12 = c, row13 = c, row14 = c, row15 = c, row16 =
c, row17 = c, row18 = c, row19 = c, row20 = c, row21 = c, row22 = c, row23 =
c, row24 = c, row25 = c, row26 = c, row27 = c, row28 = c, row29 = c, row30 =
c, row31 = c, row32 = c, cell11 = c, cell12 = c, cell14 = c, cell15 = c,
cell21 = c, cell22 = c, cell24 = c, cell25 = c, cell31 = r=2, cell51 = r=2,
cell71 = r=2, cell91 = r=2, cell111 = r=2, cell131 = r=2, cell151 = r=2,
cell171 = r=2, cell191 = r=2, cell211 = r=2, cell231 = r=2, cell251 = r=2,
cell271 = r=2, cell291 = r=2, cell311 = r=2, vlines,
hline1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33 = -,
hline4,6,8,10,12,14,16,18,20,22,24,26,28,30,32 = 2-5, & Azimuthal Inclinaison
Twin
Pairs Wall equation angle/[100] angle/(001) angle
$S_{i}/S_{ii}$ $x=0$ 90° 90° 180°
$y=0$ 180° 90° 180°
$S_{i}/S_{iii}$ $y=-z$ 180° 45° 180°
$3\epsilon_{1}1(y-z)+2\epsilon_{1}2x=0$ 98.8° 98.7° 179.4° (180.6°)
$S_{i}/S_{iv}$ $z=x$ 90° 135° 180°
$3\epsilon_{1}1(z+x)+2\epsilon_{1}2y=0$ 171.2° 81.3° 180.6° (179.4°)
$S_{i}/S_{v}$ $y=z$ 180° 135° 180°
$3\epsilon_{1}1(y+z)+2\epsilon_{1}2x=0$ 98.8° 81.3° 180.6° (179.4°)
$S_{i}/S_{vi}$ $z=-x$ 90° 45° 180°
$3\epsilon_{1}1(z-x)-2\epsilon_{1}2y=0$ 8.8° 81.3° 180.6° (179.4°)
$S_{ii}/S_{iii}$ $y=z$ 180° 135° 180°
$3\epsilon_{1}1(y+z)-2\epsilon_{1}2x=0$ 98.8° 81.3° 180.6° (179.4°)
$S_{ii}/S_{iv}$ $z=-x$ 90° 45° 180°
$3\epsilon_{1}1(z-x)+2\epsilon_{1}2y=0$ 171.2° 81.3° 180.6° (179.4°)
$S_{ii}/S_{v}$ $y=-z$ 180° 45° 180°
$3\epsilon_{1}1(y-z)-2\epsilon_{1}2x=0$ 98.8° 98.7° 179.4° (180.6°)
$S_{ii}/S_{vi}$ $z=x$ 90° 135° 180°
$3\epsilon_{1}1(z+x)-2\epsilon_{1}2y=0$ 8.8° 81.3° 180.6° (179.4°)
$S_{iii}/S_{iv}$ $x=-y$ 135° 90° 180.9° (179.1°)
$3\epsilon_{1}1(x-y)-2\epsilon_{1}2z=0$ 45° 167.6° 180°
$S_{iii}/S_{v}$ $x=0$ 90° 90° 181.3° (178.7°)
$z=0$ \- 0° -
$S_{iii}/S_{vi}$ $x=y$ 45° 90° 180.9° (179.1°)
$3\epsilon_{1}1(x+y)-2\epsilon_{1}2z=0$ 135° 167.6° 180°
$S_{iv}/S_{v}$ $x=y$ 45° 90° 180.9° (179.1°)
$3\epsilon_{1}1(x+y)+2\epsilon_{1}2z=0$ 135° 12.4° 180°
$S_{iv}/S_{vi}$ $y=0$ 180° 90° 178.7° (181.3°)
$z=0$ \- 0° -
$S_{v}/S_{vi}$ $x=-y$ 135° 90° 179.1° (180.9°)
$3\epsilon_{1}1(x-y)+2\epsilon_{1}2z=0$ 45° 12.4° 180°

Note that for each pairing, two angles are possible, the supplementary angle
is given in brackets in the table.

##  VI Tilt angle map and twin angles at another area on the sample with a
different ferroelastic ordering

We show in figure 5 a map of the domain tilts in another area of the same
sample. Three domains are identified, $D_{4}$ to $D_{6}$ with $D_{4}$ and
$D_{6}$ having tilt angles of opposite signs. The surface tilt values have a
standard deviation $\sigma$, reported in figure 5, of $\sim$ 0.05°. $D_{4}$
has the same tilt as $D_{3}$ in the area in the main manuscript. The angles
calculated from the angular map are reported in table VI. Within experimental
error, the twin angles are identical to those in the main manuscript. We
propose two spontaneous strain configurations corresponding to the measured
angles: $S_{v}/S_{iv}/S_{iii}$ or $S_{v}/S_{vi}/S_{iii}$.

Figure 5: (a) $\alpha_{\mathrm{tilt}}$ map determined by PEEM in another area
of the same sample, showing surface twin topography. (b) Map of $\sigma$ of
$\alpha_{tilt}$ and (c) Histogram of the $\alpha_{tilt}$ values. Table 1: Twin
angles calculated from the PEEM measurements in the second area. The
uncertainty comes from 2$\sigma$.

|  | Twin angle (°)
---|---|---
PEEM |  |
measured | $DW_{45}$ | 179.1 $\pm$ 0.16
| $DW_{56}$ | 179.2 $\pm$ 0.16
Possible |  |
theoretical |  |
W type |  |
walls | $S_{iii}/S_{iv}$ | 179.1
| $S_{iii}/S_{vi}$ | 179.1
| $S_{iv}/S_{v}$ | 179.1
| $S_{v}/S_{vi}$ | 179.1