# LOVM: Language-Only Vision Model Selection

Orr Zohar
Stanford Univeristy
<EMAIL_ADDRESS>&Shih-Cheng Huang
Stanford Univeristy
<EMAIL_ADDRESS>&Kuan-Chieh Wang
Stanford Univeristy
<EMAIL_ADDRESS>&Serena Yeung
Stanford Univeristy
<EMAIL_ADDRESS>
###### Abstract

Pre-trained multi-modal vision-language models (VLMs) are becoming
increasingly popular due to their exceptional performance on downstream vision
applications, particularly in the few- and zero-shot settings. However,
selecting the best-performing VLM for some downstream applications is non-
trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive
evaluation of all available VLMs on a novel application is not only time and
computationally demanding but also necessitates the collection of a labeled
dataset for evaluation. As the number of open-source VLM variants increases,
there is a need for an efficient model selection strategy that does not
require access to a curated evaluation dataset. This paper proposes a novel
task and benchmark for efficiently evaluating VLMs’ zero-shot performance on
downstream applications without access to the downstream task dataset.
Specifically, we introduce a new task LOVM: Language- Only Vision Model
Selection, where methods are expected to perform both model selection and
performance prediction based solely on a text description of the desired
downstream application. We then introduced an extensive LOVM benchmark
consisting of ground-truth evaluations of 35 pre-trained VLMs and 23 datasets,
where methods are expected to rank the pre-trained VLMs and predict their
zero-shot performance.

###  1 Introduction

Advancements in artificial intelligence (AI) have permeated diverse sectors,
but applications in areas such as medicine or those with long-tail
distributions often struggle to collect the sizable training datasets required
for the standard supervised learning framework. Pre-trained vision-language
models (VLMs) offer a promising solution, demonstrating robust performance on
diverse downstream vision tasks without the necessity of large-scale labeled
datasets (Radford et al., 2021; Jia et al., 2021). However, the performance of
VLMs can vary substantially across different tasks and domains, which
undermines the reliance solely on benchmark dataset performance for effective
VLM selection. Consequently, users aiming to _select a VLM_ for custom
downstream applications frequently face a predicament: the lack of established
performance rankings for these specific, non-conventional tasks.

As the number of pre-trained VLMs increases (see Fig. 1, (Ilharco et al.,
2021) and App. Tab. 4), the challenge of model selection escalates. Exhaustive
evaluation of all available VLMs on a novel application is not only time and
computationally demanding but also necessitates the collection of a labeled
dataset for evaluation. However, many users lack the resources or technical
proficiency to collect and label an evaluation dataset and subsequently
evaluate all available VLMs. Consequently, the development of methods that
efficiently select the most suitable model for a given task without relying on
access to the downstream task dataset has become critically important.

Figure 1: LOVM Motivation. Number of pre-trained VLMs released on open-clip
over time.

Recent studies have demonstrated that text embeddings from VLMs can be used as
a proxy for their corresponding image embeddings in various downstream tasks,
including classification and error slice discovery (Zhang et al., 2023;
Eyuboglu et al., 2022; Jain et al., 2022). Specifically, although Liang et al.
(2022) has shown that there exists a modality gap between text and image
embeddings generated from VLMs, the geometry of this modality gap permits
cross-modality transferability. This phenomenon allows text to serve as a
proxy to corresponding images and vice versa. Therefore we aim to explore the
utilization of cross-modality transferability to estimate VLM performance on a
novel vision task using text alone.

Figure 2: An overview of an application for LOVM methods. A user can type into
a search bar the details of the desired task, and then LOVM methods evaluate
and rank the available models.

Herein, we propose a novel problem setting - Language-Only VLM selection
(LOVM) as a novel model selection task. In the LOVM task, methods are expected
to select the optimal VLM and predict its expected performance given only a
text description of a downstream vision task/application, (see Fig. 2).
Importantly, LOVM eliminate the need to gather, organize, and annotate custom
datasets, thereby greatly simplifying the model selection process for
downstream users. To facilitate the development of LOVM methods in the future,
we collected a large dataset of ground-truth evaluations of 35 pre-trained
VLMs on 23 datasets. We then introduce the appropriate evaluation protocol and
method quality metrics to allow the evaluation and comparison of LOVM methods
in the future.

To show that such a challenging task is possible, we provide simple baselines
that utilize readily-available large language models to generate ‘text
datasets’ for a given vision task. By utilizing the cross-modality
transferability phenomenon, we show how simple baselines can be derived by
utilizing the cross-modality transferability phenomenon. Our results show that
text prompting may be an effective means of estimating zero-shot performance,
showing that such a challenging task is possible while providing a baseline
for future research.

The contributions of this study can be summarized as follows:

  * •

We propose a novel problem setting, LOVM: Language-Only VLM selection and
performance prediction. LOVM methods are expected to perform both model
selection and performance prediction using only a text description of the
desired zero-shot application.

  * •

We provide a benchmark consisting of 35 pre-trained VLMs and 23 datasets. We
evaluated all dataset-VLM combinations and reported their corresponding
performance, which is used as the ground truth when training and evaluating
LOVM methods. We also introduce the corresponding evaluation metrics and
protocols.

  * •

In developing the LOVM baselines, we introduce several novel methodological
contributions, such as using LLM models to generate text proxies for images.
Our text-based methods outperform simple baselines - such as ImageNet
benchmarking, showing the promise of the direction of LOVM.

  * •

By analyzing text-based score trends, we draw insights into VLM behavior and
shed light on why ResNet-based models perform better on datasets with low
visual diversity.

Our code and dataset are available at https://github.com/orrzohar/LOVM

Figure 3: Language-Only Vision Model Selection Overview. (i) Task. a LOVM
method is given a set of pre-trained VLMs, a text description of the desired
task, and the list of the classes of interest. Given these, LOVM methods are
expected to rank and predict the performance of all the available models on
the downstream task. (ii) Evaluation. Given the predicted (green) and  ground-
truth (blue) VLM ranking and performance, we evaluate the LOVM method’s
performance by accepted list ranking and accuracy metrics. (iii) Data
Collection. We exhaustively evaluated the selected 35 VLMs on the selected 23
datasets to produce the ground-truth (image-based) evaluations.

###  2 Language-Only Vision Model Selection

In order to train and evaluate LOVM methods, we need the ground-truth (GT)
zero-shot performance, i.e., image-based evaluation of many VLMs (differing by
architecture and pre-training) on many tasks and datasets. Once collected, we
can develop and evaluate LOVM methods. An ideal LOVM method should be able to
select the best performing VLM for a downstream vision task and estimate the
performance directly from text embeddings, eliminating the cost of image-based
model selection. The VLM, dataset selection criterion, and dataset collection
procedure are detailed in Sec. 2.1. Finally, the evaluation protocol of LOVM
methods is described in Sec. 2.2. For a discussion on why we only evaluate
zero-shot performace, see App. Sec. D.

##### Background.

We first recap how VLMs are used as in zero-shot vision tasks. Given a pre-
trained VLM $v$, along with an image $X\in\mathcal{X}$ or text
$Y\in\mathcal{Y}$ input, we can obtain their $L_{2}$-normalized embeddings
$\bm{x}$ or $\bm{y}$ from the image encoder $f_{x}:\mathcal{X}\mapsto\
\mathbb{R}^{n}$ or the text encoder $f_{y}:\mathcal{Y}\mapsto\
\mathbb{R}^{n}$, where $n$ is the dimension of the shared multi-modal
embedding space. To use a model $v$ on a particular task, one encodes the
class prompts, $Y^{c}$ for class $c$ using the model’s text encoder, producing
the class embeddings $\bm{y}^{c}=f_{y}(Y^{c})$. To produce the final class
prediction, one calculates the cosine similarity of an image embedding with
all the corresponding text embeddings to predict the class logits.

##### Task Definition

In the LOVM task, for any downstream application/dataset $d$, methods are
given a set of pre-trained VLMs, $\bm{V}=\\{v_{0},v_{1},..\\}\in\mathcal{V}$,
a text description of the downstream task $Y_{d}$ (e.g., classification) and a
list of the desired class names $Y^{c}_{d},\forall c\in C_{d}$ where $C_{d}$
is the number of classes in task $d$. Given this, LOVM methods are expected to
rank and predict the accuracy of the set of models (see Fig. 3, i):

$p_{v,d}=f_{\text{LOVM}}(v,\\{Y^{c}_{d}\\}_{c=1}^{C_{d}},Y_{d}),\;\;\forall\;v\in\bm{V},$
(1)

where $p_{v,d}\in\mathbb{R}$ is the relative/absolute performance of model $v$
on dataset $d$.

####  2.1 Data Collection

To train and evaluate LOVM methods, we need the zero-shot ground-truth
performance of many VLM models on many downstream datasets. We, therefore,
selected 35 VLMs and 23 Datasets and then performed image-based evaluations of
each model on all the datasets - a total of 805 evaluations using the same
prompting strategies discussed by Radford et al. (2021), See Fig. 3, iii.
These ground truth zero-shot image-based model rankings and accuracies
constitute the bulk of our benchmark. The proposed LOVM benchmark consists of
the aforementioned evaluation tables as well as the per-dataset prompting
templates, class names, and domain descriptions.

Table 1: Details on the different datasets used, including the number of
classes, tasks, and domain.

Dataset | Classes | Task | Domain
---|---|---|---
Imagenet | 1000 | classification | natural image
Stanford Cars | 196 | classification | natural image
Flowers102 | 102 | classification | natural image
CIFAR100 | 100 | classification | natural image
GTSRB | 43 | classification | natural image
VOC2007 | 20 | classification | natural image
Oxford Pets | 37 | classification | natural image
STL10 | 10 | classification | natural image
DTD | 46 | classification | textural image
RESISC4 | 45 | classification | satellite images
EuroSAT | 10 | classification | satellite images
MNIST | 10 | classification | hand-writing
Retinopathy | 5 | classification | retina scan
PCam | 2 | classification | histopathology
SUN397 | 397 | scene und. | natural image
Country211 | 211 | geolocation | natural image
SVHN | 10 | OCR | natural image
CLEVR-C | 8 | object counting | natural image
CLEVR-D | 8 | distance est. | natural image
DMLab | 6 | distance est. | synthetic
FER2013 | 7 | fac. exp. rec. | natural image
KITTI | 4 | distance est. | natural image
Rendered SST2 | 2 | OCR | text image

##### Selected Datasets.

The proposed LOVM benchmark utilizes a heterogeneous assortment of 23
datasets. These datasets exhibit variability in the number of classes, their
target tasks, and corresponding domains. The benchmark encompasses a
comprehensive range of tasks such as classification, scene understanding,
geolocalization, and object counting, rendering it extensively applicable
across many applications. Further, the datasets span diverse domains,
including natural, satellite, text, and medical images (See Tab. 1). To ensure
maximal compatibility, we have opted for tasks that permit the utilization of
the same VLM architecture, precluding any requisite alterations or additional
training. This approach necessitated the exclusion of tasks such as
segmentation and object detection, which mandate additional training modules,
introducing extraneous noise during the evaluation of VLM performance.

##### VLM Model Candidates.

We utilize the open-clip library (Ilharco et al., 2021), a diverse collection
of pre-trained VLMs spanning various architectures, including but not limited
to CLIP and CoCa models, and utilizing encoders such as ResNet, ConvNext, and
ViT. These models have undergone pre-training on various datasets, such as WIT
(Radford et al., 2021), LAION 400m, and LAION 2b (Schuhmann et al., 2022),
with different hyperparameters. From the 87 models currently available, we
have carefully selected 35 for our study. A comprehensive list of all models
used in this benchmark can be found in the App. Tab. 4. We avoided
incorporating additional multi-modal models, such as BEIT(Wang et al., 2023)
and VLMO (Bao et al., 2022), as these models utilize a shared text-image
encoder and, therefore, cannot be evaluated on the same datasets as CoCa and
CLIP. Utilizing models from the open-clip library ensures maximum
compatibility and reproducibility in our work. Currently, CLIP models comprise
a significant portion of VLMs employed in practice.

####  2.2 LOVM Evaluation Protocol

On our benchmark, methods are expected to rank 35 pre-trained multi-modal
models that differ in architecture and pre-training datasets on 23 target
datasets, and compare these rankings to the ground-truth rankings (see Fig. 3
(ii)) and report the performance for each of the 23 datasets as well as their
averaged values.

##### Model Ranking.

When evaluating model ranking on a particular dataset, one has access to the
performance of all the models on all the datasets besides the one being
evaluated. We use the following metrics:

  * •

Top-5 Recall ($R_{5}$) – We used $R_{5}$ to evaluate a LOVM method’s model
ranking capability. It is defined as the ratio of correctly identified models.

  * •

Kendall’s Rank Correlation ($\tau$) – We used $\tau$ to evaluate a LOVM
method’s model selection capability and give s fine-grained picture of how
well the method ranked the high-performing models and is defined as Kendall’s
rank over the top-5 selected models.

##### Performance Prediction.

When evaluating a model’s prediction on a dataset, the GT performance of that
model on all datasets and the performance of all models on that dataset are
held out.

  * •

Mean Absolute Error ($L_{1}$) – We used $L_{1}$ to evaluate a LOVM method’s
performance prediction capability. Specifically, we compute the $L_{1}$ loss
of all models’ predicted vs. actual mean per-class recall/top-1 accuracy.

###  3 LOVM Baselines

The assessment of model performance in traditional supervised methods often
relies on benchmark dataset performance. Given that most pre-trained vision-
language models (VLMs) are evaluated on ImageNet, it is convenient to utilize
it as a baseline for comparison (This is our ImageNet Benchmark baseline).
Alternatively, a large language model could generate many probable image
captions, which could be encoded using the different VLMs text encoder,
producing the corresponding text embeddings. Treating these embeddings as
image-proxies, one can calculate different widely-accepted scores (see Sec.
3.2) and fit a linear regression model to predict performance or rank VLMs.
Specifically, from every VLM-dataset combination, one extracts these scores
and then fits the model:

$\displaystyle p_{v,d}$ $\displaystyle=\bm{w}\cdot\bm{s}_{v,d}+b,$ (2)
$\displaystyle s^{i}_{v,d}$
$\displaystyle=f_{\text{feat}}^{i}(v,\texttt{TextGen}(\\{Y^{c}_{d}\\}_{c=1}^{C_{d}},Y_{d})),$
(3)

where $p_{v,d}\in\mathbb{R}$ is the relative/absolute performance of model $v$
on dataset $d$, $\bm{w}$, $b$ are the weights and bias of the linear model.
$s^{i}_{\text{v, d}}$ is the $i$-th element in the score vector,
$\bm{s}_{v,t}=[s^{1}_{\text{v, d}},s^{2}_{\text{v, d}},...]^{T}$, produced by
the corresponding feature/score function $f^{i}_{\text{feat}}$. The function
TextGen is a function that generates text given the class names,
$\\{Y^{c}_{d}\\}_{c=1}^{C_{d}}$ and task description $Y_{d}$ of the desired
task/dataset $d$. We discuss the different scores, $s^{i}_{\text{v, d}}$, in
Sec. 3.2 and the TextGen function in Sec. 3.1. To evaluate model rankings on a
dataset, we hold out the data for that particular dataset and fit a linear
model on all the other datasets. Meanwhile, to evaluate the performance
prediction of some model on a particular dataset, we hold out the data for
that dataset and model and fit a linear model on the remaining combinations.
We refer to the baselines by the combination of scores used in the linear
model.

####  3.1 Text Data Generation

The impressive progress in large language models (LLMs) (OpenAI, 2023; Touvron
et al., 2023) has rendered the generation of potential - and realistic -
‘image captions’ practically limitless, thus rendering text data generation
remarkably attainable. In our study, we employ GPT-3.5, tasked to produce two
distinct text-based datasets, each corresponding to a given vision task. These
generated datasets serve as the foundation for extracting essential features
for our task.

##### Captions Dataset.

To generate the captions dataset, $\bm{D^{\text{cap}}}$, we prompt an LLM to
generate realistic - but confusing - captions for images containing the user-
provided classes in the user-provided domain. We extracted the dataset
description and class names from each dataset and prompted the LLM:

> Generate long and confusing image captions for the {domain} domain, which
> will be used to evaluate a Vision-Language Model’s {task} performance.
>
> Generate 50 captions for {classname}:

Where we assume the user supplies the target domain and task description. For
examples of different dataset’s domain and task, see Tab. 3.

##### Synonyms Dataset.

Prior studies have already leveraged synonyms to evaluate LLMs (van der Lee et
al., 2022). For example, if an LVM has seen many instances of the class
‘chair’ referenced as a ‘chair’, ‘seat’, etc., we expect these embeddings to
be closely located in the shared embedding space. To evaluate this aspect of
the VLM using text, we prompt an LLM to generate a list of semantically
similar/synonyms for every object class. For example, for the class “chair”,
we get: “seat”, “bench”, “armchair”, and “furniture”. We prompt the LLM with
the following:

> Please list the superclasses/synonyms for {classname}. For example:
>
> chair: [furniture, seat, bench, armchair, sofa]
>
> {classname}:

We collect the results from this prompt to form the synonyms dataset,
$\bm{D^{\text{syn}}}$.

####  3.2 Text-Derived Scores

There are many widely reported metrics for model transferability, dataset
difficulty, and dataset granularity scores developed on image embeddings. We
extract different commonly used features/metrics from the text dataset
embeddings and calculate their text-only counterparts.

##### Text Classification Scores (C).

We use the generated captions dataset as image proxies and evaluate the
resulting model performance. Specifically, we replace the images with the
generated image captions and evaluate each model’s text top-1 accuracy, text-
acc1 and f1-score text-f1.

##### Dataset Granularity Scores (G).

Cui et al. (2019) introduced the use of two widely used dataset granularity
measures for image classification, Fisher criterion (Fisher, 1936),
$\phi_{\text{fisher}}$ and Silhouette score (Rousseeuw, 1987),
$\varphi_{\text{sil}}$, and their normalization constant, Class Dispersion
score, $\rho_{\text{disp}}$. The Fisher criterion measures the degree of
similarity of the classes or the extent of their separation. The Silhouette
score is a well-established metric used to quantify the tightness of the same-
class samples to the separation of different-class samples. The Class
Dispersion score quantifies the degree of same-class tightness or data cone
radius.

Recently, van der Lee et al. (2022) has shown that synonym consistency can be
used in large language models to correlate the degree of familiarity of the
model with a particular concept. Using the Synonym dataset, we compare the
cosine similarity between the text embedding of each class and its
corresponding synonyms. A high Synonym Consistency score,
$\gamma_{\text{syn}}$, between the class and its corresponding synonyms
indicates that the model is aware of the semantic meaning of the class.

For detailed definitions of these metrics, see App. Sec. B.

###  4 Experiments and Results

In Sec. 4.1, we evaluate the model selection capabilities of the proposed
baselins on the LOVM benchmark. In Sec. 4.2, we evaluate the proposed
baselines performance prediction cababilities. We then analyse score trends
and draw insights in Sec. 4.3.

Table 2: LOVM Benchmark. We evaluate our method’s performance over 23 datasets
and 35 pre-trained models, and when predicting the top-1 accuracy and mean
per-class recall (averaged over all datasets, for the per-dataset breakdown,
see App. Tab. 5 and 6 ). INB - ImageNet Baseline, C - Text Classification
scores, G - Granularity scores. As can be seen, mixed aproaches achieve the
best VLM ranking and performance prediction.

used mean per-class recall top-1 accuracy scores $R_{5}(\uparrow)$
$\tau(\uparrow)$ $L_{1}(\downarrow)$ $R_{5}(\uparrow)$ $\tau(\uparrow)$
$L_{1}(\downarrow)$ INB 0.504 0.186 0.228 0.452 0.177 0.220 C 0.252 0.058
0.182 0.226 0.058 0.176 G 0.270 -0.014 0.145 0.252 -0.014 0.143 G+C 0.261
0.101 0.145 0.252 -0.014 0.143 INB+C 0.522 0.214 0.182 0.461 0.223 0.176 INB+G
0.539 0.197 0.145 0.461 0.096 0.141 INB+G+C 0.539 0.197 0.145 0.461 0.223
0.141

####  4.1 Model Selection

A core aspect of this benchmark is model selection, as it allows the user to
quickly and easily select the optimal model for the desired downstream task.
From Tab. 2, we can see that, when predicting/ranking by the models mean per-
class recall, the (C+G)-baseline can achieve a top-5 recall of $0.261$,
indicating that, on average, more than one model is correctly ranked as a
top-5 performing model. Meanwhile, the INB-baseline had a $R_{5}$ of $0.505$.
Combining the text and ImageNet scores, the (INB+G)-baseline achieves the
highest accuracy of $0.539$, a $\sim 15\%$ improvement over the INB-baseline.
When studying Kendall’s rank correlation, the (G+C)-, INB-, and
(INB+G)-baselines achieve a $\tau$ of $0.101,0.186$, and $0.214$,
respectively. The (INB+G)-baseline had the highest rank correlation with $\sim
6\%$ improvement over the INB-baseline. Similar results can be seen when
predicting the top-1 accuracy. The consistent improvement of the baselines
over the INB-baseline indicates the utility of both text-based and benchmark
features. Interestingly, C-score (or the text-acc1) appears to be more
influential in predicting/ranking model’s the top-1 accuracy than the mean
per-class recall.

####  4.2 Performance Prediction

Figure 4: Predicted vs. Ground-Truth Accuracy. Predicted vs. actual top-1
accuracy on the proposed LOVM benchmark.

Based on Tab. 2, it is clear that the granularity scores (G) are instrumental
to predicting a model’s top-1 accuracy and mean per-class recall. The
G-baseline approach can achieve an average $L_{1}$ error of 0.145 and 0.141
for predicting the mean-per-class recall and top-1 accuracy, respectively.
Adding any other scores does not lead to an improvement in performance
prediction. The INB-baseline, which uses Imagenet performance as prediction,
leads to a much higher $L_{1}$ error of 0.228 and 0.220 compared to the text-
base baselines (text-based performance estimation outperformed INB-baselines
by $\sim 36\%$). Finally, adding the ImageNet benchmark score to the text
features in the Unified baseline did not improve the $L_{1}$ compared to the
text-only baseline. This is expected as the imagenet performance cannot be
used to predict the performance on a different dataset. Fig. 4 shows the
predicted vs. ground-truth accuracy. Our approach had a $R^{2}$ score (or
coefficient of determination) of $=0.55$, showing significant room for
improvement in accuracy prediction.

####  4.3 Insights into VLM Behavior

In this section, we visualize the dependence of the text-derived features on
the pre-training datasets and model architectures while averaging them across
the different datasets (see Fig. 5).

##### Model Size.

From studying Fig. 5, we can we can identify a clear trend of Fisher criterion
and Silhouette score improving with model size, while Class Dispersion score
and Synonym Consistency score degrade with model size. Silhouette score
quantifies the degree of inter-class overlap or the degree of overlap between
different classes in the embedding space. As the model size of the visual
encoder increases, the embeddings from different classes become more and more
orthogonal, decreasing the inter-class overlap. Fisher criterion quantifies
the degree of granularity a model perceives the target datasets to be. As
model size decreases, Fisher criterion decreases, or the degree of perceived
granularity increases. Class Dispersion score quantifies the degree of intra-
class dispersion, or how similar embeddings of the same class are.
Specifically, as we increase model size, Class Dispersion score decreases, and
therefore the class embeddings become more varied, effectively expanding the
class cone radius. Synonym Consistency score quantified the closeness of a
class to its synonyms and behaved similarly to Fisher criterion.

##### Pre-training Dataset.

When studying the effect of pre-training dataset size, it is clear that there
is a positive correlation between pre-training dataset size and all of the
metrics when comparing models of the same size. As the pre-training dataset
increases, the intra-class simularity increases more rapidly than the inter-
class simularity, hence effectively different classes are more seperated.
Specifically, Fisher criterion and Silhouette score increase, or the degree of
perceived granularity decreases, and embeddings from different classes become
less orthogonal, increasing the inter-class overlap. As the pre-training
dataset size increases, Class Dispersion score increases and the intra-class
dispersion is more condensed, leading to a smaller effective radius of a class
dataset cone. Interestingly, larger models are more affected by the increase
in dataset size (as seen by the large slope of ViT-L compared to ViT-B) -
which could explain previous works’ observation that larger models benefit
more when trained on larger datasets (Fang et al., 2022).

##### Model Architecture.

Pre-training datasets and model architectures significantly influence each
other. ResNets and ViTs, for instance, consistently demonstrated differing
behaviors and appeared to reside at distinct points on the class separation-
dispersion trade-off curve. In particular, ResNets displayed lower Class
Dispersion score and Silhouette score, indicating challenges in encoding
instances of the same class within the feature space compared to ViTs. This
may account for ResNets’ superior performance on datasets with low visual
variation, like MNIST; as the visual variation is relatively low, we would not
expect the Class Dispersion score to be the limiting factor in model
performance, making them less affected by this aspect of the dataset.
Intriguingly, ConvNEXT models exhibited characteristics more in line with ViT-
base models than ResNet-based ones. What leads to variation between WIT and
L400m remains unclear, necessitating further investigation.

Figure 5: Analyzing Score Trends. Average text scores dependence on pre-
training datasets and model architecture on our text-derived scores. (left)
scores quantifying inter-class similarity (right) scores quantifying intra-
class similarity. ResNet ( ) and ConvNext ($\times$) based models are grouped
separately to evaluate their effect on the score trends.

###  5 Related Work

##### Vision-Language Models.

The field of vision-language models (VLMs) has witnessed significant progress
in recent years, particularly with the introduction of contrastive pre-trained
VLMs such as CLIP (Radford et al., 2021). These models leverage large-scale
datasets of aligned image-caption pairs to obtain shared embedding spaces that
capture rich visual and textual features. The learned image and text encoders
from these VLMs have demonstrated impressive feature extraction capabilities
and even set state-of-the-art zero-shot performances. However, the performance
of VLMs can vary significantly across different datasets, especially when
there exists a domain, content, or distribution shift (Fang et al., 2022). As
the number of model architectures & pre-training datasets (Ilharco et al.,
2021; Schuhmann et al., 2022) increase, it is challenging to select a pre-
trained VLM, as good performance on existing benchmarks does not always
translate to the downstream task. Therefore, there is a need to develop
strategies that can estimate VLM performance on a new task without requiring
an exhaustive evaluation of these models using the target dataset.

##### Text as a Proxy For Images.

While these VLMs aim to project representations from different modalities into
a shared embedding space, Liang et al. (2022) found that corresponding image
and text pairs don’t completely overlap in the embedding space. Instead, a
“modality gap” exists between the image embeddings and text embeddings sub-
space. Subsequently, Zhang et al. (2023) has found that this gap can be
approximated as an orthogonal constant between true pairs of image and text
and is, therefore, parallel to the decision boundaries for a given modality.
This suggests that cross-modality transferability - using one modality as
input to the other’s classifier - is possible for these contrastively pre-
trained VLMs. Severak studies have demonstrated the utility of the cross-
modality transferability phenomenon in different tasks. For instance, Domino
leveraged the cross-modal embeddings to identify error slices and generate
natural language descriptions of the error slices (Eyuboglu et al., 2022).
Similarly, Jain et al. (2022) used these embeddings to discover model failure
directions in the multi-modal embedding space. Meanwhile, Zhang et al. (2023)
proposed the DrML, which diagnoses and rectifies vision classifiers using
natural language inputs. In this study, we also use text as a proxy for
images, but for the novel task of ranking and estimating VLM performance.

##### Unsupervised Model Selection.

Unsupervised model selection was recently introduced by Sun et al. (2021), to
select the best model for a new target domain without utilizing labeled data.
Their work only considered domain (and not content) shifts and proposed
constructing a proxy dataset that captures/closely approximates this shift.
This proxy dataset is constructed by minimizing different dataset statistics
using several labeled datasets. Evaluating models on this proxy set performs
well for model selection/ranking. However, such a strategy is limiting in the
setting of evaluating VLMs - the size of these models and their pre-training
datasets makes it too computationally expensive to achieve the desired goal of
evaluating model performance on any downstream task.

##### Unsupervised Accuracy Estimation.

Unsupervised or label-free accuracy estimation aims to estimate classifier
model performance with only access to the unlabeled test set of a new task.
Platanios et al. (2017, 2016) proposed strategies to apply probabilistic
modeling approaches, such as the probabilistic logic or Bayesian modeling, to
analyze and aggregate predictions from multiple classifiers. Other works
approach this task by fitting models on feature statistics of the target
dataset (Risser-Maroix and Chamand, 2023). Some studies evaluated model
agreement, where many classifiers are used on the target dataset, and the
degree of agreement was correlated with model performance (Chen et al., 2021;
Jiang et al., 2022) Other approaches for unsupervised accuracy estimation
include training a neural network on the weight distribution statistics
(Unterthiner et al., 2020) or composing a meta-dataset with available
datasets, such that the meta-dataset matched some target dataset statistics
(Deng and Zheng, 2021). Some have attempted to craft embedding-based scores,
trying to quantify the separability of clusters in the embeddings spaces
(Pándy et al., 2022; Ding et al., 2022). All these methods assume access to
the unlabeled dataset of the target task. Instead, our method only requires
text descriptions of the novel task to estimate the model’s performance.

###  6 Conclusion

In this work, we introduce a new problem setting and task LOVM, which aims to
select the best-performing VLMs for a downstream vision task by only using its
textual description. To demonstrate the feasibility of such a task, we show
how large language models, in combination with the cross-modal transferability
phenomenon, can be leveraged for such a task. We exhaustively test these
methods on the proposed LOVM benchmark, consisting of 35 VLMs and 23 benchmark
datasets. Our findings validate the viability of our proposed LOVM task, with
unified (both text scores and ImageNet benchmarking) baselines outperforming
the ImageNet benchmarking baseline. This suggests that text-based model
selection methods (i.e., LOVM methods) provide additional benefits to baseline
selection based on a model’s performance on ImageNet. Furthermore, we found
that the granularity-based scores influence performance prediction and modal
ranking more greatly. These findings bolster the research direction of
developing methods for VLM selections using only text.

Our proposed LOVM benchmark aims to foster this research direction. We see two
promising avenues for future research: (i) improving text-based classification
correlation with ground-truth accuracy by either text generation, evaluation
metrics, or cross-modal transferability, and (ii) introducing new granularity
and transferability scores to the text-only paradigm. Namely, we anticipate
the development of methods improving over our proposed baselines presented in
Tab. 2. Our work aims to facilitate future research in this area and provide a
more accurate and reliable means of comparing pre-trained VLMs, accelerating
their utilization in downstream applications.

For a discussion about broader and potential negative societal impacts please
see App. Sec E.

##### Acknowledgments.

We gratefully acknowledge the computational credits provided by Google Cloud
Platform through Stanford’s HAI Institute for Human-Centered Artificial
Intelligence. We also thank the Knight-Hennessy Scholars Foundation for
generously funding Orr Zohar.

### References

  * Agostinelli et al. [2022] Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari.  How stable are transferability metrics evaluations?  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _Computer Vision – ECCV 2022_ , pages 303–321, Cham, 2022. Springer Nature Switzerland.  ISBN 978-3-031-19830-4.
  * Bao et al. [2022] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei.  VLMo: Unified vision-language pre-training with mixture-of-modality-experts.  In _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=bydKs84JEyw.
  * Chen et al. [2021] Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha.  Detecting errors and estimating accuracy on unlabeled data with self-training ensembles.  _Advances in Neural Information Processing Systems_ , 34:14980–14992, 2021.
  * Cheng et al. [2017] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote sensing image scene classification: Benchmark and state of the art.  _Proceedings of the IEEE_ , 105(10):1865–1883, Oct 2017.  ISSN 1558-2256.  doi: 10.1109/jproc.2017.2675998.  URL http://dx.doi.org/10.1109/JPROC.2017.2675998.
  * Cimpoi et al. [2014] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi.  Describing textures in the wild.  In _Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2014.
  * Coates et al. [2011] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * Cui et al. [2019] Yin Cui, Zeqi Gu, Dhruv Mahajan, Laurens van der Maaten, Serge Belongie, and Ser-Nam Lim.  Measuring dataset granularity, 2019.  URL https://arxiv.org/abs/1912.10154.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Deng and Zheng [2021] Weijian Deng and Liang Zheng.  Are labels always necessary for classifier accuracy evaluation?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15069–15078, June 2021.
  * Ding et al. [2022] Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut.  Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks.  In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, _ECCV 2022_ , pages 252–268, Cham, 2022\. Springer Nature Switzerland.  ISBN 978-3-031-19830-4.
  * Dumitru Ian Goodfellow [2013] Yoshua Bengio Dumitru Ian Goodfellow, Will Cukierski.  Challenges in representation learning: Facial expression recognition challenge, 2013.
  * Ethayarajh et al. [2021] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.  Understanding dataset difficulty with $\mathcal{V}$-usable information, 2021.  URL https://arxiv.org/abs/2110.08420.
  * Everingham et al. [2007] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results.  http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html, 2007\.
  * Eyuboglu et al. [2022] Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Re.  Domino: Discovering systematic errors with cross-modal embeddings.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=FPCMqjI0jXN.
  * Fang et al. [2022] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip), 2022.  URL https://arxiv.org/abs/2205.01397.
  * Feng et al. [2022] Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir Abdi.  Towards better selective classification, 2022.  URL https://arxiv.org/abs/2206.09034.
  * Fisher [1936] Ronald A Fisher.  The use of multiple measurements in taxonomic problems.  _Annals of eugenics_ , 7(2):179–188, 1936.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _International Journal of Robotics Research (IJRR)_ , 2013.
  * Helber et al. [2019] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2019.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Jain et al. [2022] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry.  Distilling model failures as directions in latent space, 2022.  URL https://arxiv.org/abs/2206.14754.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  _CoRR_ , abs/2102.05918, 2021.  URL https://arxiv.org/abs/2102.05918.
  * Jiang et al. [2022] Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter.  Assessing generalization of SGD via disagreement.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=WvOGCEAQhxl.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910, 2017.
  * Kaggle and EyePacs [2015] Kaggle and EyePacs.  Kaggle diabetic retinopathy detection, 2015.  URL https://www.kaggle.com/c/diabetic-retinopathy-detection/data.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)_ , Sydney, Australia, 2013.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Liang et al. [2022] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  In _NeurIPS_ , 2022.  URL https://openreview.net/forum?id=S7Evzt9uit3.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In _NIPS Workshop on Deep Learning and Unsupervised Feature Learning_ , 2011.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _Indian Conference on Computer Vision, Graphics and Image Processing_ , Dec 2008.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Pándy et al. [2022] Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink.  Transferability estimation using bhattacharyya class separability.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9172–9182, June 2022.
  * Parkhi et al. [2012] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012\.
  * Platanios et al. [2017] Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric J Horvitz.  Estimating accuracy from unlabeled data: A probabilistic logic approach.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/95f8d9901ca8878e291552f001f67692-Paper.pdf.
  * Platanios et al. [2016] Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell.  Estimating accuracy from unlabeled data: A bayesian approach.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1416–1425, New York, New York, USA, 20–22 Jun 2016. PMLR.  URL https://proceedings.mlr.press/v48/platanios16.html.
  * Pándy et al. [2022] Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink.  Transferability estimation using bhattacharyya class separability.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9162–9172, 2022.  doi: 10.1109/CVPR52688.2022.00896.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _CoRR_ , abs/2103.00020, 2021.  URL https://arxiv.org/abs/2103.00020.
  * Risser-Maroix and Chamand [2023] Olivier Risser-Maroix and Benjamin Chamand.  What can we learn by predicting accuracy?  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 2390–2399, January 2023.
  * Rousseeuw [1987] Peter J. Rousseeuw.  Silhouettes: A graphical aid to the interpretation and validation of cluster analysis.  _Journal of Computational and Applied Mathematics_ , 20:53–65, 1987.  ISSN 0377-0427.  doi: https://doi.org/10.1016/0377-0427(87)90125-7.  URL https://www.sciencedirect.com/science/article/pii/0377042787901257.
  * Scheidegger et al. [2018] Florian Scheidegger, Roxana Istrate, Giovanni Mariani, Luca Benini, Costas Bekas, and Cristiano Malossi.  Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy, 2018.  URL https://arxiv.org/abs/1803.09588.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=M3Y74vmsMcY.
  * Stallkamp et al. [2011] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.  The german traffic sign recognition benchmark: a multi-class classification competition.  In _The 2011 international joint conference on neural networks_ , pages 1453–1460. IEEE, 2011.
  * Sun et al. [2021] Xiaoxiao Sun, Yunzhong Hou, Weijian Deng, Hongdong Li, and Liang Zheng.  Ranking models in unlabeled new environments.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11761–11771, 2021.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.  URL https://arxiv.org/abs/2302.13971.
  * Unterthiner et al. [2020] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin.  Predicting neural network accuracy from weights, 2020.  URL https://arxiv.org/abs/2002.11448.
  * van der Lee et al. [2022] Chris van der Lee, Thiago Castro Ferreira, Chris Emmery, Travis Wiltshire, and Emiel Krahmer.  Neural data-to-text generation based on small datasets: Comparing the added value of two semi-supervised learning approaches on top of a large language model, 2022.  URL https://arxiv.org/abs/2207.06839.
  * Veeling et al. [2018] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant CNNs for digital pathology, June 2018.
  * Wang et al. [2023] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei.  Image as a foreign language: BEiT pretraining for vision and vision-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Wong et al. [2022] Lauren J. Wong, Sean McPherson, and Alan J. Michaels.  Assessing the value of transfer learning metrics for rf domain adaptation, 2022.
  * Xiao et al. [2010] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 3485–3492, June 2010.  doi: 10.1109/CVPR.2010.5539970.
  * Zhai et al. [2020] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
  * Zhang et al. [2023] Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung.  Diagnosing and rectifying vision models using language.  _arXiv preprint arXiv:2302.04269_ , 2023.

LOVM: Language-Only Vision Model Selection
Appendix

##

###  Appendix A LOVM Benchmark Details

We evaluated 35 on 23, a total of 805 evaluations. This constituted the bulk
of our compute with a total of 4 days on an nvidia V100 instance. Evaluations
were carried out using the CLIP_benchmark repository
(https://github.com/LAION-AI/CLIP_benchmark).

####  A.1 LOVM Benchmark - Datasets

The proposed LOVM benchmark comprises of 23 datasets, which were selected to
maximize diversity. Specifically, these datasets vary in the number of classes
(2 to 1000), their target tasks, and domains. The benchmark encompasses a
comprehensive range of tasks such as classification, scene understanding,
geolocalization, object counting, and more, with the goal of rendering it
extensively applicable across many applications. Further, the datasets span
diverse domains, including natural, satellite, text, and medical images (See
Tab. 3 for a comprehensive account of the datasets and their source). To
ensure maximal compatibility, we have opted for tasks that permit the
utilization of the same VLM architecture, precluding any requisite alterations
or additional training. This approach necessitated the exclusion of tasks such
as segmentation and object detection, which mandate additional training
modules, introducing extraneous noise while evaluating VLM performance.
However, it is worth noting that previous transferability works have shown
that these approaches may generalize to more complex applications such as
semantic segmentation [Pándy et al., 2022, Agostinelli et al., 2022].

####  A.2 LOVM Benchmark - Vision-Language Models

Tab. 4 presents a list of models and their corresponding pre-training datasets
used in the LOVM benchmark. We utilize the open-clip library [Ilharco et al.,
2021], a diverse collection of pre-trained VLMs spanning various
architectures, including but not limited to CLIP and CoCa models, and
utilizing encoders such as ResNet, ConvNext, and ViT. These models have
undergone pre-training on various datasets, such as WIT [Radford et al.,
2021], LAION 400m, and LAION 2b [Schuhmann et al., 2022], with different
hyperparameters. From the 87 models currently available, we have carefully
selected 35 for our study. A comprehensive list of all models used in this
benchmark can be found in Tab. 4. We avoided incorporating additional multi-
modal models, such as BEIT[Wang et al., 2023] and VLMO [Bao et al., 2022], as
these models utilize a shared text-image encoder and, therefore, cannot be
evaluated on the same datasets as CoCa and CLIP. Using models from the open-
clip library ensures maximum compatibility and reproducibility in our work.
Currently, CLIP models comprise a significant portion of VLMs employed in
practice. Tab. 4 includes 35 entries, each identified by an ID number. The
first four columns indicate the ID number, model name, model abbreviation, and
pre-training dataset name. The fifth column shows the abbreviation of the pre-
training dataset name. The models listed in the table include ResNet (RN50,
RN101, etc.) and Vision Transformer (ViT) with different sizes (B/32, B/16,
L/14, etc.), and the pre-training datasets include OpenAI’s WIT dataset and
two variants of LAION (L400m and L2b) datasets.

####  A.3 LOVM Benchmark - Ground-Truth Model Ranking

To evaluate the validity and generalizability of the LOVM benchmark, we first
present the ground-truth model ranking over all datasets to show that the
model order is not constant across the datasets. We organized the benchmarks
from natural image classification (Fig. 6, left) to non-natural image / non-
classification benchmarks (Fig. 6, right). As depicted in Fig. 6, the
distribution exhibits a non-uniform pattern, indicating the utility of LOVM
methods and the importance of VLM selection methods in general. Interestingly,
ranking variations are more significant on the non-natural image / non-
classification benchmarks. This exemplifies the need for LOVM methods to
contend with content shift (i.e., changing what classes are in the target
domain) and domain/task shift.

Figure 6: Ground-Truth VLM Ranking. As can be seen, there is a lot of
variation in the ground-truth model ranking across both the natural image
(left) and other (right) benchmarks.  Table 3: Details on the different
benchmarks used in the study, including the number of classes, tasks, and
target domain.

Dataset | Classes | Task | Domain
---|---|---|---
Imagenet [Deng et al., 2009] | 1000 | classification | natural image
Stanford Cars [Krause et al., 2013] | 196 | classification | natural image
Flowers102 [Nilsback and Zisserman, 2008] | 102 | classification | natural image
CIFAR100 [Krizhevsky et al., 2009] | 100 | classification | natural image
GTSRB [Stallkamp et al., 2011] | 43 | classification | natural image
VOC2007 [Everingham et al., 2007] | 20 | classification | natural image
Oxford Pets [Parkhi et al., 2012] | 37 | classification | natural image
STL10 [Coates et al., 2011] | 10 | classification | natural image
DTD [Cimpoi et al., 2014] | 46 | classification | textural image
RESISC45 [Cheng et al., 2017] | 45 | classification | satellite images
EuroSAT [Helber et al., 2019] | 10 | classification | satellite images
MNIST [LeCun et al., 2010] | 10 | classification | hand-writing
Retinopathy [Kaggle and EyePacs, 2015] | 5 | classification | retina scan
PCam [Veeling et al., 2018] | 2 | classification | histopathology
SUN397 [Xiao et al., 2010] | 397 | scene und. | natural image
Country211 [Radford et al., 2021] | 211 | geolocation | natural image
SVHN [Netzer et al., 2011] | 10 | OCR | natural image
Rendered SST2 [Radford et al., 2021] | 2 | OCR | text image
FER2013 [Dumitru Ian Goodfellow, 2013] | 7 | fac. exp. rec. | natural image
CLEVR-C [Johnson et al., 2017] | 8 | object counting | natural image
CLEVR-D [Johnson et al., 2017] | 8 | distance est. | natural image
DMLab [Zhai et al., 2020] | 6 | distance est. | synthetic
KITTI [Geiger et al., 2013] | 4 | distance est. | natural image

Table 4: Translation of open clip to model/pre-training dataset names used in
paper. When renaming the datasets we tried to group models with similar
optimization schemes to minimize the number of pre-training datasets without
causing undo overlap.

ID | Model | Name | Dataset | Name
---|---|---|---|---
1 | RN50 | RN50 | openai | WIT
2 | RN101 | RN101 | openai | WIT
3 | RN50x4 | RN50x4 | openai | WIT
4 | RN50-16 | RN50x16 | openai | WIT
5 | RN50x64 | RN50x64 | openai | WIT
6 | ViT-B-32 | ViT-B/32 | laion400m_e31 | L400m
7 | ViT-B-32 | ViT-B/32 | laion400m_e32 | L400m
8 | ViT-B-32-quickgelu | ViT-B/32 | laion400m_e32 | L400m
9 | ViT-B-32 | ViT-B/32 | openai | WIT
10 | ViT-B-32 | ViT-B/32 | laion2b_s34b_b79k | L2b-b
11 | ViT-B-32 | ViT-B/32 | laion2b_e16 | L2b-c
12 | ViT-B-16 | ViT-B/16 | laion400m_e32 | L400m
13 | ViT-B-16 | ViT-B/16 | openai | WIT
14 | ViT-B-16-240 | ViT-B/16-240 | laion400m_e32 | L400m
15 | ViT-L-14 | ViT-L/14 | laion400m_e31 | L400m
16 | ViT-L-14 | ViT-L/14 | laion400m_e32 | L400m
17 | ViT-L-14 | ViT-L/14 | laion2b_s32b_b82k | L2b-b
18 | ViT-L-14 | ViT-L/14 | openai | WIT
19 | ViT-L-14-336 | ViT-L/14-336 | openai | WIT
20 | ViT-G-14 | ViT-G/14 | laion2b_s12b_b42k | L2b-a
21 | ViT-G-14 | ViT-G/14 | laion2b_s34b_b88k | L2b-a
22 | ViT-H-14 | ViT-H/14 | laion2b_s32b_b79k | L2b-b
23 | coca_ViT-B-32 | CoCa-ViT-B/32 | laion2b_s13b_b90k | L2b-c
24 | coca_ViT-B-32 | CoCa-ViT-B/32 | mscoco_finetuned_laion2b_s13b_b90k | L2b-c + coco
25 | coca_ViT-L-14 | CoCa-ViT-L/14 | laion2b_s13b_b90k | L2b-c
26 | coca_ViT-L-14 | CoCa-ViT-L/14 | mscoco_finetuned_laion2b_s13b_b90k | L2b-c + coco
27 | convnext_base | ConvNEXT-B | laion400m_s13b_b51k | L400m-c
28 | convnext_base_w | ConvNEXT-BW | laion2b_s13b_b82k | L2b-d
29 | convnext_base_w | ConvNEXT-BW | laion2b_s13b_b82k_augreg | L2b-e
30 | convnext_base_w | ConvNEXT-BW | laion_aesthetic_s13b_b82k | L2b-f
31 | convnext_base_w_320 | ConvNEXT-BW-320 | laion_aesthetic_s13b_b82k | L2b-f
32 | convnext_base_w_320 | ConvNEXT-BW-320 | laion_aesthetic_s13b_b82k_augreg | L2b-g
33 | convnext_large_d | ConvNEXT-LD | laion2b_s26b_b102k_augreg | L2b-h
34 | convnext_large_d_320 | ConvNEXT-LD-320 | laion2b_s29b_b131k_ft | L2b-i
35 | convnext_large_d_320 | ConvNEXT-LD-320 | laion2b_s29b_b131k_ft_soup | L2b-j

###  Appendix B Baseline Details

Figure 7: Overview of the baselines. (top left) Using a text description of a
new task, we use a large language model to generate the Image Caption and
Class-Synonym datasets. We feed these text datasets into a VLMs text encoder,
which generates the text-derived multi-modal embeddings. Using these
embeddings, as well as the user-defined prompting strategies, we extract
different scores. Finally, we fit a linear model on the extracted scores to
predict model ranking and accuracy. (bottom) Schematic drawings of our
different proposed scores.

Fig. 7 shows an overview of our baselines. We first describe the prompting
protocol used in Sec B.1. We then give an in-detail description of the scores
used in the study in Sec. B.2. In Sec. B.3, we give additional details for the
text dataset generation. Finally, in Sec. B.4, we describe how we use noise to
corrupt the text caption dataset when calculating text-acc1 and text-f1
scores.

####  B.1 Prompting Templates

We use the same prompting strategy introduced by Radford et al. [2021] to
generate the model zero-shot weights (see Fig. 10 for examples of templates
from different datasets). Specifically, for every class $c$, we used the
reported templates to produce the text prompts $\bm{Y^{c}}$ and encoded these
prompts using the VLM text encoder, $f_{y}$, to produce the text embeddings
for class $c$, $\hat{\bm{y}}^{c}$:

$\bm{\hat{y}}^{c}=f_{y}(\bm{Y}^{c}).$

We then normalized each separate prompt by its $L_{2}$ norm and averaged the
resulting vector to produce $\bm{\bar{y}^{c}}$, or the unnormalized zero-shot
weight of class $c$:

$\displaystyle\bar{\bm{y}}^{c}$
$\displaystyle=\frac{1}{N}\sum_{\text{j=1}}^{\text{N}}\frac{\bm{y}_{j}^{c}}{||\bm{y}_{j}^{c}||_{2}},$

where $\bar{\bm{y}}^{c}$ is then normalized again to produce the final zero-
shot classification weight of class $c$,

$\displaystyle\bm{y}^{c}$
$\displaystyle=\frac{\bar{\bm{y}}^{c}}{||\bar{\bm{y}}^{c}||_{2}}.$ (4)

####  B.2 Text-Derived Scores

We define the six scores we derived for model selection and performance
prediction. The Text top-1 accuracy score and Text f1-score is used to
estimate the VLMs’ performance on a vision task using text as a proxy, while
the Fisher criterion and Silhouette score are used to understand the VLM’s
capability to separate samples from different classes in the target task
(inter-class similarity. To estimate dataset granularity, we use Class
Dispersion score. Finally, the Synonym Consistency score allows us to evaluate
the degree of content shift between the VLMs’ pre-training and target dataset
(intra-class similarity).

##### Text Classification scores.

We use the generated captions dataset (see Sec. 3.1) as a proxy for images and
evaluate the resulting model performance. Specifically, we use the VLM text
encoder to generate text-derived multi-modal embeddings. We then corrupt these
embeddings with Gaussian noise to approximate image-instance variation (see
Sec. B.4)and calculate their cosine similarity with the class prompt
embeddings - derived using the same prompt ensembling strategies proposed by
Radford et al. [2021] (see Fig. 7). We then calculate the text top-1 accuracy
(text-acc1) and text f1-score (text-f1).

##### Fisher criterion,

$\phi_{\text{fisher}}$. The Fisher criterion [Fisher, 1936] has been widely
used as a dataset granularity measure and has recently been shown to be
effective for classification by Cui et al. [2019]. The Fisher score measures
the degree of similarity of the classes or the extent of their separation. In
VLMs, The quality of the class separation can be evaluated using text by
assessing how close the different (text-derived) class prompt embeddings are.
We introduce the concept of Fisher criterion, a score that quantifies how
close the class prompt embeddings are to each other (see Fig. 7):

$\phi_{\text{fisher}}=\frac{1}{C}\sum_{j=1}^{C}\text{max}_{c,c\neq
j}\left[\theta(\bm{y}^{j},\bm{y}^{c})\right],$ (5)

where $\bm{y}^{c}$ is the class prompt embedding derived using the prompt
ensembling strategies proposed in Radford et al. [2021] for class $c$ (see
Sec. B.1), $\theta(\cdot,\cdot)$ is a function that calculates the cosine
similarity between two vectors, and $C$ is the number of classes.

##### Silhouette score,

$\varphi_{\text{sil}}$. The silhouette score [Rousseeuw, 1987] is a well-
established score that has been used to quantify the tightness of the same-
class samples to the separation of different-class samples [Scheidegger et
al., 2018, Cui et al., 2019]. Inspired by this score, we introduce the text-
based Silhouette score, $\varphi_{\text{sil}}$, which measures the separation
of different-class samples in the caption dataset $\bm{D^{\text{cap}}}$. To do
so, we evaluate the average cosine similarity of captions to the nearest other
class by:

$\varphi_{\text{sil}}=\frac{1}{C}\sum_{j=1}^{C}\text{max}_{c,c\neq
j}\left[\frac{1}{N}\sum_{k=1}^{N}\theta(\bm{D^{\text{cap}}}[j]_{k},\bm{y}^{c})\right],$
(6)

where $\bm{y}^{c}$ is the class prompt embedding derived using the prompt
ensembling strategies proposed in Radford et al. [2021] for class $c$ (see
Sec. B.1), $\theta(\cdot,\cdot)$ is a function that calculates the cosine
similarity between two vectors, and $C$ is the number of classes.
$\bm{D^{\text{cap}}}[j]_{k}$ representing sample $k$ of class $j$ in the
caption dataset $\bm{D^{\text{cap}}}$, and there is a total of $N$ such
samples in for each class.

.

##### Class Dispersion score,

$\rho_{\text{disp}}$. The Class Dispersion score is used as the normalization
constant to generate the Fisher and Silhouette scores, and it quantifies the
degree of same-class tightness or data cone radius (see Fig. 7).

$\rho_{\text{disp}}=\frac{1}{CN}\sum_{c=1}^{C}\sum_{k=1}^{N}\theta(\bm{D^{\text{cap}}}[c]_{k},\bm{y}^{c}),$
(7)

where $\bm{y}^{c}$ is the class prompt embedding derived using the prompt
ensembling strategies proposed in Radford et al. [2021] for class $c$ (see
Sec. B.1), $\theta(\cdot,\cdot)$ is a function that calculates the cosine
similarity between two vectors, and $C$ is the number of classes.
$\bm{D^{\text{cap}}}[c]_{k}$ representing sample $k$ of class $c$ in the
caption dataset $\bm{D^{\text{cap}}}$, and there is a total of $N$ such
samples in for each class.

##### Synonym Consistency score,

$\gamma_{\text{syn}}$. Synonym consistency has been shown in large language
models to correlate with the degree of familiarity of the model with a
particular concept [van der Lee et al., 2022]. Using the Synonym dataset, we
compare the cosine similarity between the text embedding of each class and its
corresponding synonyms. A high cosine similarity between the class and its
corresponding synonyms/supercategories indicates that the model is aware of
the semantic meaning of the class and is defined as:

$\gamma_{\text{syn}}=\frac{1}{CN}\sum_{c=1}^{C}\sum_{k=1}^{N}\theta(\bm{D^{\text{syn}}}[c]_{k},\bm{y}^{c}),$
(8)

where $\bm{y}^{c}$ is the class prompt embedding derived using the prompt
ensembling strategies proposed in Radford et al. [2021] for class $c$ (see
Sec. B.1), $\theta(\cdot,\cdot)$ is a function that calculates the cosine
similarity between two vectors, and $C$ is the number of classes.
$\bm{D^{\text{syn}}}[c]_{k}$ representing sample $k$ of class $c$ in the
synonym dataset $\bm{D^{\text{syn}}}$, and there is a total of $N$ such
samples in for each class.

####  B.3 Text Dataset Generation

To generate the Captions dataset, we used a large language model to generate
realistic (but confusing) image captions. It was necessary to request
confusing image captions to get sufficient variation in the image captions. We
used OpenAI’s ‘gpt-3.5-turbo-0301’ model with a temperature of 1. For the
synonym dataset, we reduced the temperature to 0.1 and only requested the
synonyms themselves. We then used the prompting templates with the synonym in
place of the original class name to generate $\bm{D^{\text{syn}}}$.

####  B.4 Text Classification and Noise Corruption

In this work, we introduce the use of Gaussian noise to corrupt text-derived
multi-modal embeddings to approximate image-instance variation. The corrupted
embeddings are then used to calculate the text top-1 accuracy and text-f1
score, which serves as a proxy for evaluating the performance of a vision
model. The scores are derived from the Captions dataset, a collection of
complex but probable image captions generated using a large language model for
images containing the user-provided classes in the user-provided domain and
for the user-provided task. For more, see Sec. 3.1.

To evaluate the effectiveness of the text top-1 accuracy, we systematically
increase the level of noise corruption and plot the text top-1 accuracy
against the ground-truth top-1 accuracy (See Fig. 8). We quantify this
correlation via the $R^{2}$ score or the degree of explained variance. As we
do not fit a linear model to the predicted vs. ground-truth predictions,
$R^{2}$ ranges from 1 (perfect linear fit) to $-\infty$, where non of the
variance is explained. The results show that, without noise corruption, the
text top-1 accuracy is too high and frequently saturates without any
corruption. However, as the noise level increases to 0.1, the text top-1
accuracy progressively improves until a better linear correlation can be seen.
This indicates that increasing noise corruption can better approximate image-
instance variation and improve the correlation of the text top-1 accuracy.
Beyond 0.1, however, the correlation between the text top-1 accuracy and top-1
accuracy progressively worsens. This shows that while noise corruption helps
improve the correlation of the text top-1 accuracy, there is a limit beyond
which further noise corruption degrades its effectiveness.

Figure 8: Ablating Noise Injection Effect on Text Top-1 Accuracy. Without
noise (sigma=0), the text top-1 accuracy saturates on many datasets and
models, with extremely high top-1 accuracy, making the correlation between the
ground-truth top-1 accuracy and text top-1 accuracy quite poor. By corrupting
the text embeddings with noise, we notice an improvement in correlation up to
sigma=0.1, after which the correlation is steadily corrupted.

###  Appendix C Additional Results

####  C.1 LOVM Per-Dataset Breakdown

Here, we show the per-dataset breakdown of our main results. In Tab. 5, we
show our model ranking and performance prediction results for top-1 accuracy.
In Tab. 6, we show our model ranking and performance prediction results for
mean per-class recall.

Table 5: LOVM Benchmark (top-1 accuracy).We evaluate our method’s performance
over 23 datasets and 35 pre-trained models. (top) Model Ranking results.
(bottom) Performance Prediction results. INB - ImageNet Benchmark score, C -
Text Classification scores G - Granularity Scores.

Stanford Cars CIFAR100 CLEVR-DIST. CLEVR-COUNT Country211 Retinopathy DMLab
DTD EuroSAT FER2013 Flowers102 GTSRB ImageNet KITTI MNIST PCam Oxford Pets
Rendered SST2 RESISC45 STL10 SUN397 SVHN VOC2007 [HTML]FFFFED  Mean $R_{5}$
INB 0.80 0.80 0.00 0.60 0.40 0.00 0.00 1.00 0.60 0.20 0.40 0.60 1.00 0.00 0.20
0.00 0.80 0.00 1.00 0.20 0.60 0.60 0.60 [HTML]FFFFED 0.452 C 0.00 0.20 0.20
0.60 0.40 0.00 0.00 0.60 0.40 0.40 0.20 0.40 0.00 0.20 0.20 0.20 0.00 0.20
0.40 0.20 0.40 0.00 0.00 [HTML]FFFFED 0.226 G 0.40 0.40 0.20 0.20 0.40 0.00
0.00 0.40 0.20 0.20 0.80 0.20 0.40 0.20 0.00 0.20 0.40 0.00 0.40 0.20 0.40
0.00 0.20 [HTML]FFFFED 0.252 G+C 0.40 0.40 0.20 0.20 0.40 0.00 0.00 0.40 0.20
0.20 0.80 0.20 0.40 0.20 0.00 0.20 0.40 0.00 0.40 0.20 0.40 0.00 0.20
[HTML]FFFFED 0.252 INB+C 0.80 0.80 0.00 0.60 0.60 0.00 0.00 1.00 0.60 0.20
0.60 0.60 0.80 0.00 0.20 0.20 0.80 0.00 1.00 0.20 0.60 0.40 0.60 [HTML]FFFFED
0.461 INB+G 0.80 0.80 0.00 0.60 0.40 0.00 0.00 1.00 0.60 0.20 0.40 0.60 1.00
0.00 0.40 0.00 0.80 0.20 1.00 0.40 0.60 0.40 0.40 [HTML]FFFFED 0.461 INB+C+G
0.80 0.80 0.00 0.60 0.40 0.00 0.00 1.00 0.60 0.20 0.40 0.60 1.00 0.00 0.40
0.00 0.80 0.20 1.00 0.40 0.60 0.40 0.40 [HTML]FFFFED 0.461 $\tau$ INB 0.33
0.67 0.00 0.33 0.00 0.00 0.00 -0.20 1.00 0.00 0.00 1.00 1.00 0.00 0.00 0.00
0.00 0.00 -0.40 0.00 -1.00 0.33 1.00 [HTML]FFFFED 0.177 C 0.00 0.00 0.00 1.00
0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 [HTML]FFFFED 0.058 G 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 -0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 [HTML]FFFFED -0.014 C+G 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
-0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 [HTML]FFFFED
-0.014 INB+C 0.33 0.67 0.00 0.33 -0.33 0.00 0.00 0.00 1.00 0.00 1.00 1.00 1.00
0.00 0.00 0.00 0.33 0.00 -0.20 0.00 -1.00 0.00 1.00 [HTML]FFFFED 0.223 INB+G
0.33 0.33 0.00 0.33 0.00 0.00 0.00 0.00 1.00 0.00 0.00 1.00 0.80 0.00 0.00
0.00 0.00 0.00 -0.60 0.00 -1.00 0.00 0.00 [HTML]FFFFED 0.096 INB+C+G 0.33 0.67
0.00 0.33 -0.33 0.00 0.00 0.00 1.00 0.00 1.00 1.00 1.00 0.00 0.00 0.00 0.33
0.00 -0.20 0.00 -1.00 0.00 1.00 [HTML]FFFFED 0.223 $L_{1}$ INB 0.16 0.07 0.58
0.42 0.49 0.52 0.61 0.09 0.08 0.26 0.01 0.26 0.00 0.50 0.03 0.26 0.17 0.18
0.06 0.21 0.03 0.17 0.08 [HTML]FFFFED 0.228 C 0.08 0.13 0.19 0.15 0.25 0.25
0.32 0.17 0.05 0.13 0.15 0.03 0.19 0.37 0.28 0.34 0.10 0.14 0.05 0.23 0.21
0.16 0.07 [HTML]FFFFED 0.176 G 0.03 0.03 0.19 0.22 0.44 0.16 0.23 0.01 0.03
0.07 0.45 0.01 0.22 0.29 0.29 0.04 0.09 0.11 0.02 0.21 0.03 0.10 0.02
[HTML]FFFFED 0.143 C+G 0.03 0.03 0.19 0.22 0.44 0.16 0.23 0.01 0.03 0.07 0.45
0.01 0.22 0.29 0.29 0.04 0.09 0.11 0.02 0.21 0.03 0.10 0.02 [HTML]FFFFED 0.143
INB+C 0.08 0.13 0.19 0.15 0.25 0.25 0.32 0.17 0.05 0.13 0.15 0.03 0.19 0.37
0.28 0.34 0.10 0.14 0.05 0.23 0.21 0.16 0.07 [HTML]FFFFED 0.176 INB+G 0.03
0.02 0.20 0.20 0.44 0.02 0.28 0.01 0.02 0.10 0.43 0.02 0.21 0.33 0.25 0.03
0.06 0.12 0.00 0.23 0.03 0.22 0.00 [HTML]FFFFED 0.141 INB+C+G 0.03 0.02 0.20
0.20 0.44 0.02 0.28 0.01 0.02 0.10 0.43 0.02 0.21 0.33 0.25 0.03 0.06 0.12
0.00 0.23 0.03 0.22 0.00 [HTML]FFFFED 0.141

Table 6: LOVM Benchmark (mean per-class recall). We evaluate our method’s
performance over 23 datasets and 35 pre-trained models. (top) Model Ranking
results. (bottom) Performance Prediction results. INB - ImageNet Benchmark
score, C - Text Classification scores G - Granularity Scores.

Stanford Cars CIFAR100 CLEVR-DIST. CLEVR-COUNT Country211 Retinopathy DMLab
DTD EuroSAT FER2013 Flowers102 GTSRB ImageNet KITTI MNIST PCam Oxford Pets
Rendered SST2 RESISC45 STL10 SUN397 SVHN VOC2007 [HTML]FFFFED  Mean $R_{5}$
INB 0.80 0.80 0.40 0.60 0.40 0.40 0.00 1.00 0.60 0.40 0.60 0.60 1.00 0.20 0.20
0.00 0.60 0.00 1.00 0.20 0.80 0.60 0.40 [HTML]FFFFED 0.504 C 0.00 0.20 0.20
0.60 0.40 0.20 0.20 0.60 0.40 0.40 0.00 0.40 0.00 0.40 0.20 0.20 0.20 0.20
0.40 0.20 0.40 0.00 0.00 [HTML]FFFFED 0.252 G 0.40 0.40 0.20 0.20 0.40 0.00
0.40 0.40 0.20 0.20 0.60 0.20 0.40 0.40 0.00 0.20 0.40 0.00 0.40 0.20 0.40
0.20 0.00 [HTML]FFFFED 0.270 G+C 0.00 0.40 0.20 0.20 0.60 0.40 0.60 0.40 0.20
0.20 0.00 0.60 0.20 0.20 0.40 0.20 0.00 0.00 0.40 0.20 0.40 0.20 0.00
[HTML]FFFFED 0.261 INB+C 0.80 0.80 0.40 0.60 0.60 0.40 0.00 1.00 0.60 0.40
0.80 0.60 0.80 0.20 0.20 0.20 0.80 0.00 1.00 0.20 0.80 0.40 0.40 [HTML]FFFFED
0.522 INB+G 0.80 0.80 0.40 0.60 0.40 0.40 0.00 1.00 0.60 0.40 0.60 0.60 1.00
0.20 0.40 0.20 0.60 0.00 1.00 0.40 0.80 0.60 0.60 [HTML]FFFFED 0.539 INB+C+G
0.80 0.80 0.40 0.60 0.40 0.40 0.00 1.00 0.60 0.40 0.60 0.60 1.00 0.20 0.40
0.20 0.60 0.00 1.00 0.40 0.80 0.60 0.60 [HTML]FFFFED 0.539 $\tau$ INB 0.67
0.67 0.00 0.33 0.00 0.00 0.00 0.00 1.00 0.00 1.00 1.00 1.00 0.00 0.00 0.00
-0.33 0.00 -0.40 0.00 -0.33 -0.33 0.00 [HTML]FFFFED 0.186 C 0.00 0.00 0.00
1.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 [HTML]FFFFED 0.058 G 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 -0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 [HTML]FFFFED -0.014 C+G 0.00 0.00 0.00 0.00 1.00 0.00 1.00 0.00 0.00
0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[HTML]FFFFED 0.101 INB+C 0.67 0.67 0.00 0.33 -0.33 0.00 0.00 -0.20 1.00 0.00
1.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 -0.20 0.00 0.00 0.00 0.00 [HTML]FFFFED
0.214 INB+G 0.67 0.33 0.00 0.33 0.00 0.00 0.00 0.00 1.00 0.00 1.00 1.00 0.80
0.00 0.00 0.00 0.33 0.00 -0.60 0.00 -0.33 -0.33 0.33 [HTML]FFFFED 0.197
INB+C+G 0.67 0.33 0.00 0.33 0.00 0.00 0.00 0.00 1.00 0.00 1.00 1.00 0.80 0.00
0.00 0.00 0.33 0.00 -0.60 0.00 -0.33 -0.33 0.33 [HTML]FFFFED 0.197 $L_{1}$ INB
0.16 0.07 0.58 0.42 0.49 0.52 0.61 0.09 0.08 0.26 0.01 0.26 0.00 0.50 0.03
0.26 0.17 0.18 0.06 0.21 0.03 0.17 0.08 [HTML]FFFFED 0.228 C 0.07 0.09 0.22
0.14 0.26 0.27 0.31 0.16 0.03 0.15 0.17 0.08 0.22 0.33 0.30 0.34 0.09 0.16
0.05 0.22 0.20 0.21 0.12 [HTML]FFFFED 0.183 G 0.14 0.03 0.25 0.16 0.44 0.23
0.19 0.01 0.01 0.12 0.46 0.03 0.30 0.15 0.28 0.03 0.07 0.09 0.04 0.17 0.02
0.09 0.02 [HTML]FFFFED 0.145 C+G 0.14 0.03 0.25 0.16 0.44 0.23 0.19 0.01 0.01
0.12 0.46 0.03 0.30 0.15 0.28 0.03 0.07 0.09 0.04 0.17 0.02 0.09 0.02
[HTML]FFFFED 0.145 INB+C 0.07 0.09 0.22 0.14 0.26 0.27 0.31 0.16 0.03 0.15
0.17 0.08 0.22 0.33 0.30 0.34 0.09 0.16 0.05 0.22 0.20 0.21 0.12 [HTML]FFFFED
0.183 INB+G 0.14 0.03 0.25 0.16 0.44 0.23 0.19 0.01 0.01 0.12 0.46 0.03 0.30
0.15 0.28 0.03 0.07 0.09 0.04 0.17 0.02 0.09 0.02 [HTML]FFFFED 0.145 INB+C+G
0.14 0.03 0.25 0.16 0.44 0.23 0.19 0.01 0.01 0.12 0.46 0.03 0.30 0.15 0.28
0.03 0.07 0.09 0.04 0.17 0.02 0.09 0.02 [HTML]FFFFED 0.145

####  C.2 Ablation Experiments

To understand the utility of each of our extracted scores, we exhaustively
ablated their effect on top-1 accuracy model ranking and performance
prediction (Tab. 7 and Tab. 8), and mean per-class recall model ranking and
performance prediction (Tab. 9 and Tab. 10). Specifically, we ablated each
score’s impact on the resulting model’s performance. As can be seen, using
more than $\sim 3$ features at a time seldom improves performance. Future work
can investigate the use of more sophisticated models that may be able to
utilize more scores in predicting model ranking and performance. Specifically,
for ranking models, text classification and scores quantifying intra-class
similarity (Class Dispersion score & Synonym Consistency score) were the most
dominant, while for performance prediction, granularity scores quantifying
both inter- and inta- class similarity was the most important. This

We ablate the model ranking performance to understand each extracted score’s
effect on ranking models. The text classification scores and scores
quantifying intra-class similarity were the most consequential in predicting
model ranking. Specifically, in ranking models, the text-f1 score, Class
Dispersion score ($\rho_{\text{disp}}$), and Synonym Consistency score
($\gamma_{\text{syn}}$) where the most dominant (Tab. 7 rows 8 & 11, Tab. 9
row 38). Overall, it seems like the text classification excelled at fine-
grained ranking (as quantified by $\tau$), while the inter-class granularity
scores improved the coarse ranking prediction (as quantified by $R_{5}$.
Meanwhile, granularity scores quantifying inter- and intra- class similarity
were the most dominant for performance prediction. Specifically, Class
Dispersion score ($\rho_{\text{disp}}$), Synonym Consistency score
($\gamma_{\text{syn}}$), and Silhouette score ($\varphi_{\text{sil}}$) were
the most influential (Tab. 8 rows 26 & 41, Tab. 10 row 60). INB does not aid
performance prediction, indicating that getting a course estimation of dataset
difficulty dominates performance prediction.

Table 7: LOVM Model Selection Ablation. Here, we ablate all the different
scores used in our baselines for model ranking by top-1 accuracy. We separated
the text classification (C) base scores, and the granularity-based scores that
quantify inter- and intra-class similarity. $a_{\text{IN}}$ \- Imagenet
Accuracy, $\phi_{\text{fisher}}$ \- Fisher criterion, text-f1 - caption
dataset f1-score, text-acc1 - text top-1 accuracy, $\gamma_{\text{syn}}$ \-
Synonym Consistency score, $\varphi_{\text{sil}}$ \- Silhouette score,
$\rho_{\text{disp}}$ \- Class Dispersion score.

|  | Scores | Metrics
---|---|---|---
| Row ID | $a_{\text{IN}}$ | text-f1 | text-acc1 | $\gamma_{\text{syn}}$ | $\rho_{\text{disp}}$ | $\varphi_{\text{sil}}$ | $\phi_{\text{fisher}}$ |  $\tau$ ($\uparrow$) |  $R_{5}$ ($\uparrow$)
Model Ranking | 1 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.177 |  [HTML]FFFFED 0.452
2 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.058 |  [HTML]FFFFED 0.226
3 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.029 |  [HTML]FFFFED 0.191
4 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.183
5 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED -0.014 |  [HTML]FFFFED 0.243
6 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED -0.014 |  [HTML]FFFFED 0.252
7 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.165
8 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.223 |  [HTML]FFFFED 0.461
9 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.200 |  [HTML]FFFFED 0.452
10 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.188 |  [HTML]FFFFED 0.426
11 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.096 |  [HTML]FFFFED 0.461
12 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.078 |  [HTML]FFFFED 0.417
13 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.119 |  [HTML]FFFFED 0.426
14 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.029 |  [HTML]FFFFED 0.226
15 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.191
16 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
17 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
18 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.183
19 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.191
20 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
21 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.191
22 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.174
23 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.200
24 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.235
25 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.174
26 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
27 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.165
28 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
29 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.093 |  [HTML]FFFFED 0.443
30 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.174 |  [HTML]FFFFED 0.443
31 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.188 |  [HTML]FFFFED 0.435
32 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.145 |  [HTML]FFFFED 0.443
33 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.096 |  [HTML]FFFFED 0.417
34 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.159 |  [HTML]FFFFED 0.435
35 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.188 |  [HTML]FFFFED 0.443
36 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.145 |  [HTML]FFFFED 0.452
37 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.067 |  [HTML]FFFFED 0.417
38 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.110 |  [HTML]FFFFED 0.426
39 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.188 |  [HTML]FFFFED 0.426
40 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.119 |  [HTML]FFFFED 0.417
41 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.067 |  [HTML]FFFFED 0.435
42 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.119 |  [HTML]FFFFED 0.417
43 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.119 |  [HTML]FFFFED 0.426
44 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.209
45 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
46 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
47 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
48 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.217
49 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
50 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.191
51 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
52 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
53 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
54 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
55 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
56 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.191
57 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.191
58 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
59 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
60 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
61 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.165
62 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.191
63 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.191
64 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.151 |  [HTML]FFFFED 0.452
| Scores | Metrics
---|---|---
Row ID | $a_{\text{IN}}$ | text-f1 | text-acc1 | $\gamma_{\text{syn}}$ | $\rho_{\text{disp}}$ | $\varphi_{\text{sil}}$ | $\phi_{\text{fisher}}$ |  $\tau$ ($\uparrow$) |  $R_{5}$ ($\uparrow$)
65 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.107 |  [HTML]FFFFED 0.443
66 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.093 |  [HTML]FFFFED 0.435
67 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.133 |  [HTML]FFFFED 0.443
68 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.174 |  [HTML]FFFFED 0.443
69 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.157 |  [HTML]FFFFED 0.443
70 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.075 |  [HTML]FFFFED 0.417
71 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.113 |  [HTML]FFFFED 0.452
72 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.096 |  [HTML]FFFFED 0.417
73 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.096 |  [HTML]FFFFED 0.417
74 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.159 |  [HTML]FFFFED 0.435
75 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.443
76 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.075 |  [HTML]FFFFED 0.417
77 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.113 |  [HTML]FFFFED 0.452
78 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.096 |  [HTML]FFFFED 0.417
79 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.096 |  [HTML]FFFFED 0.417
80 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.417
81 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.119 |  [HTML]FFFFED 0.417
82 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.119 |  [HTML]FFFFED 0.417
83 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.110 |  [HTML]FFFFED 0.417
84 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED -0.029 |  [HTML]FFFFED 0.209
85 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED -0.029 |  [HTML]FFFFED 0.217
86 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED -0.014 |  [HTML]FFFFED 0.217
87 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
88 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.226
89 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
90 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
91 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.226
92 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
93 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
94 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
95 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
96 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
97 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.200
98 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.226
99 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.116 |  [HTML]FFFFED 0.435
100 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.058 |  [HTML]FFFFED 0.435
101 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.133 |  [HTML]FFFFED 0.443
102 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.049 |  [HTML]FFFFED 0.443
103 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.133 |  [HTML]FFFFED 0.443
104 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.133 |  [HTML]FFFFED 0.443
105 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.443
106 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.104 |  [HTML]FFFFED 0.417
107 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.104 |  [HTML]FFFFED 0.417
108 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.435
109 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.443
110 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.104 |  [HTML]FFFFED 0.417
111 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.104 |  [HTML]FFFFED 0.417
112 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.435
113 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.101 |  [HTML]FFFFED 0.400
114 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED -0.029 |  [HTML]FFFFED 0.217
115 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED -0.043 |  [HTML]FFFFED 0.209
116 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED -0.029 |  [HTML]FFFFED 0.217
117 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.217
118 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
119 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.209
120 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.055 |  [HTML]FFFFED 0.443
121 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.090 |  [HTML]FFFFED 0.435
122 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.090 |  [HTML]FFFFED 0.435
123 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.090 |  [HTML]FFFFED 0.443
124 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.435
125 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.128 |  [HTML]FFFFED 0.435
126 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED -0.029 |  [HTML]FFFFED 0.217
127 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.046 |  [HTML]FFFFED 0.435

Table 8: LOVM Model Prediction Ablation. Here, we ablate all the different
scores used in our baselines for predicting model top-1 accuracy. We separated
the text classification (C) base scores, and the granularity-based scores that
quantify inter- and intra-class similarity. $a_{\text{IN}}$ \- Imagenet
Accuracy, $\phi_{\text{fisher}}$ \- Fisher criterion, text-f1 - caption
dataset f1-score, text-acc1 - text top-1 accuracy, $\gamma_{\text{syn}}$ \-
Synonym Consistency score, $\varphi_{\text{sil}}$ \- Silhouette score,
$\rho_{\text{disp}}$ \- Class Dispersion score.

|  | Scores | Metrics
---|---|---|---
| Row ID | $a_{\text{IN}}$ | text-f1 | text-acc1 | $\gamma_{\text{syn}}$ | $\rho_{\text{disp}}$ | $\varphi_{\text{sil}}$ | $\phi_{\text{fisher}}$ |  $L_{1}$ ($\downarrow$) )
Model Prediction | 1 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.220
2 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.176
3 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.177
4 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.188
5 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.200
6 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.223
7 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.170
8 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.189
9 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.184
10 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.215
11 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.205
12 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.215
13 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.167
14 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.180
15 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.190
16 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.184
17 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.180
18 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.160
19 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.183
20 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.176
21 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.179
22 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.159
23 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.199
24 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.188
25 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.154
26 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.143
27 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.163
28 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.166
29 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.191
30 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.197
31 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.189
32 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.187
33 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.159
34 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.196
35 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.189
36 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.185
37 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.155
38 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.212
39 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.219
40 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.156
41 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.141
42 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.160
43 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.167
44 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.184
45 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.187
46 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.186
47 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.170
48 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.190
49 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.189
50 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.160
51 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.148
52 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.163
53 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.167
54 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.183
55 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.187
56 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.159
57 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.147
58 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.159
59 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.167
60 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.148
61 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.154
62 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.161
63 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.153
64 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.200
| Scores | Metrics
---|---|---
Row ID | $a_{\text{IN}}$ | text-f1 | text-acc1 | $\gamma_{\text{syn}}$ | $\rho_{\text{disp}}$ | $\varphi_{\text{sil}}$ | $\phi_{\text{fisher}}$ |  $L_{1}$ ($\downarrow$) )
65 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.191
66 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.191
67 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.167
68 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.198
69 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.200
70 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.160
71 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.151
72 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.158
73 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.163
74 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.198
75 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.198
76 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.158
77 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.156
78 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.154
79 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.162
80 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.144
81 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.158
82 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.164
83 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.149
84 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.186
85 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.194
86 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.168
87 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.152
88 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.171
89 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.177
90 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.156
91 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.168
92 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.173
93 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.155
94 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.156
95 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.166
96 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.168
97 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.158
98 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.150
99 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.202
100 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.198
101 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.164
102 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.154
103 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.164
104 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.165
105 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.161
106 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.165
107 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.171
108 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.165
109 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.159
110 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.165
111 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.168
112 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.166
113 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.153
114 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.162
115 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.175
116 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.178
117 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.159
118 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.163
119 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.161
120 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.161
121 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.171
122 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.173
123 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.162
124 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.167
125 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.164
126 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.165
127 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.171

Table 9: LOVM Model Selection Ablation. Here, we ablate all the different
scores used in our baselines for model ranking by mean per-class recall. We
separated the text classification (C) base scores, and the granularity-based
scores that quantify inter- and intra-class similarity. $a_{\text{IN}}$ \-
Imagenet Accuracy, $\phi_{\text{fisher}}$ \- Fisher criterion, text-f1 -
caption dataset f1-score, text-acc1 - text top-1 accuracy,
$\gamma_{\text{syn}}$ \- Synonym Consistency score, $\varphi_{\text{sil}}$ \-
Silhouette score, $\rho_{\text{disp}}$ \- Class Dispersion score.

|  | Scores | Metrics
---|---|---|---
| Row ID | $a_{\text{IN}}$ | text-f1 | text-acc1 | $\gamma_{\text{syn}}$ | $\rho_{\text{disp}}$ | $\varphi_{\text{sil}}$ | $\phi_{\text{fisher}}$ |  $\tau$ ($\uparrow$) |  $R_{5}$ ($\uparrow$)
Model Ranking | 1 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.186 |  [HTML]FFFFED 0.504
2 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.058 |  [HTML]FFFFED 0.252
3 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.029 |  [HTML]FFFFED 0.191
4 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED -0.014 |  [HTML]FFFFED 0.217
5 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.261
6 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED -0.014 |  [HTML]FFFFED 0.270
7 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.157
8 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.214 |  [HTML]FFFFED 0.522
9 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.209 |  [HTML]FFFFED 0.513
10 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.116 |  [HTML]FFFFED 0.487
11 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.177 |  [HTML]FFFFED 0.530
12 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.186 |  [HTML]FFFFED 0.504
13 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.072 |  [HTML]FFFFED 0.478
14 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.029 |  [HTML]FFFFED 0.235
15 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.072 |  [HTML]FFFFED 0.235
16 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.243
17 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.252
18 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.235
19 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.072 |  [HTML]FFFFED 0.252
20 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
21 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.226
22 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.226
23 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED -0.014 |  [HTML]FFFFED 0.217
24 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.252
25 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.174
26 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.226
27 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.165
28 | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.217
29 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.145 |  [HTML]FFFFED 0.487
30 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.214 |  [HTML]FFFFED 0.504
31 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.200 |  [HTML]FFFFED 0.504
32 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.180 |  [HTML]FFFFED 0.496
33 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.122 |  [HTML]FFFFED 0.496
34 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.194 |  [HTML]FFFFED 0.496
35 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.180 |  [HTML]FFFFED 0.504
36 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.180 |  [HTML]FFFFED 0.504
37 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.136 |  [HTML]FFFFED 0.504
38 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.197 |  [HTML]FFFFED 0.539
39 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.145 |  [HTML]FFFFED 0.496
40 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.072 |  [HTML]FFFFED 0.470
41 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.174 |  [HTML]FFFFED 0.487
42 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.058 |  [HTML]FFFFED 0.487
43 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.072 |  [HTML]FFFFED 0.487
44 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.243
45 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.235
46 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
47 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.226
48 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.029 |  [HTML]FFFFED 0.252
49 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
50 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.101 |  [HTML]FFFFED 0.252
51 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.252
52 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
53 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
54 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
55 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.252
56 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.101 |  [HTML]FFFFED 0.261
57 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.226
58 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.226
59 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.217
60 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.226
61 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.165
62 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.209
63 | $\times$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.226
| 64 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.478
| Scores | Metrics
---|---|---
Row ID | $a_{\text{IN}}$ | text-f1 | text-acc1 | $\gamma_{\text{syn}}$ | $\rho_{\text{disp}}$ | $\varphi_{\text{sil}}$ | $\phi_{\text{fisher}}$ |  $\tau$ ($\uparrow$) |  $R_{5}$ ($\uparrow$)
65 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.487
66 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.478
67 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.145 |  [HTML]FFFFED 0.478
68 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.194 |  [HTML]FFFFED 0.496
69 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.186 |  [HTML]FFFFED 0.504
70 | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.136 |  [HTML]FFFFED 0.496
71 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.171 |  [HTML]FFFFED 0.504
72 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.122 |  [HTML]FFFFED 0.496
73 | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.122 |  [HTML]FFFFED 0.496
74 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.194 |  [HTML]FFFFED 0.496
75 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.180 |  [HTML]FFFFED 0.496
76 | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.151 |  [HTML]FFFFED 0.504
77 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.171 |  [HTML]FFFFED 0.513
78 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.151 |  [HTML]FFFFED 0.504
79 | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.151 |  [HTML]FFFFED 0.504
80 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.487
81 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.058 |  [HTML]FFFFED 0.478
82 | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.058 |  [HTML]FFFFED 0.478
83 | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.087 |  [HTML]FFFFED 0.487
84 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.235
85 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.000 |  [HTML]FFFFED 0.252
86 | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.261
87 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.243
88 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.235
89 | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.235
90 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
91 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.029 |  [HTML]FFFFED 0.261
92 | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.243
93 | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.243
94 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.252
95 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.252
96 | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.252
97 | $\times$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.043 |  [HTML]FFFFED 0.226
98 | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  [HTML]FFFFED 0.014 |  [HTML]FFFFED 0.261
99 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.487
100 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.116 |  [HTML]FFFFED 0.487
101 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.145 |  [HTML]FFFFED 0.487
102 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.478
103 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ |  [HTML]FFFFED 0.130 |  [HTML]FFFFED 0.487