$\displaystyle-\frac{\varepsilon^{2}(x,y)}{8\textup{tn}_{\underline{\kappa}}(\tfrac{d(x,y)}{2})}\geq-\frac{|\nabla\kappa|_{\infty}^{2}d(x,y)}{\underline{\kappa}}\geq
2\mathcal{C}(x,y)\frac{|\nabla\kappa|^{2}_{\infty}}{\underline{\kappa}^{2}}$

as desired. ∎

In order to obtain Lemma 3.7, we estimate the remaining terms:

$\displaystyle\Bigr{(}-4\lambda+3\overline{\kappa}+\textup{tn}_{\overline{\kappa}}^{2}(\tfrac{d(x,y)}{2})+\textup{tn}_{\underline{\kappa}}(\tfrac{d(x,y)}{2})\mathcal{C}(x,y)\Bigl{)}\mathcal{C}(x,y)$
$\displaystyle\geq$
$\displaystyle\Bigr{(}-4\lambda+3\overline{\kappa}+\textup{tn}_{\overline{\kappa}}^{2}(\tfrac{d(x,y)}{2})+\textup{tn}^{2}_{\underline{\kappa}}(\tfrac{d(x,y)}{2})\Bigl{)}\mathcal{C}(x,y)$
$\displaystyle\geq$
$\displaystyle\Bigr{(}-4\lambda+3\overline{\kappa}+(\overline{\kappa}-\underline{\kappa})(1+\frac{\pi}{2})\Bigl{)}\mathcal{C}(x,y),$

where we used the fact that due to $\kappa>0,$ $\mathcal{C}<0$ and moreover,

(7.16)
$\displaystyle\textup{tn}_{\overline{\kappa}}^{2}-\textup{tn}_{\underline{\kappa}}^{2}$
$\displaystyle=\int_{\underline{\kappa}}^{\overline{\kappa}}\frac{d}{d\kappa}\textup{tn}^{2}_{\kappa}(\tfrac{d}{2})\,d\kappa=\int_{\underline{\kappa}}^{\overline{\kappa}}\frac{d}{d\kappa}\kappa\tan^{2}(\sqrt{\kappa}\tfrac{d}{2})\,d\kappa$
$\displaystyle=\int_{\underline{\kappa}}^{\overline{\kappa}}\left[\tan^{2}(\sqrt{\kappa}\tfrac{d}{2})+\tfrac{d\sqrt{\kappa}}{2}\tan(\sqrt{\kappa}\tfrac{d}{2})\,\textup{sec}^{2}(\sqrt{\kappa}\tfrac{d}{2})\right]\,d\kappa$
$\displaystyle\leq(\overline{\kappa}-\underline{\kappa})\left(1+d\sqrt{\overline{\kappa}}\right)$
$\displaystyle\leq(\overline{\kappa}-\underline{\kappa})(1+\frac{\pi}{2}),$

where we used $D\leq\tfrac{\pi}{2\sqrt{\overline{\kappa}}}$ in the last two
inequalities. Putting all the above estimates together, Lemma 3.7 is proved.

######  Remark 7.8.

It is possible to sharpen the estimate of Lemma 3.7 with more careful
analysis. Doing so will improve the constants, but not essentially change the
form of the estimates.

## References

  * [AB89] Mark S Ashbaugh and Rafael Benguria.  Optimal lower bound for the gap between the first two eigenvalues of one-dimensional Schrödinger operators with symmetric single-well potentials.  Proceedings of the American Mathematical Society, 105(2):419–424, 1989.
  * [AC11] Ben Andrews and Julie Clutterbuck.  Proof of the fundamental gap conjecture.  Journal of the American Mathematical Society, 24(3):899–916, 2011\.
  * [AC13] Ben Andrews and Julie Clutterbuck.  Sharp modulus of continuity for parabolic equations on manifolds and lower bounds for the first eigenvalue.  Analysis & PDE, 6(5):1013 – 1024, 2013.
  * [And14] Ben Andrews.  Moduli of continuity, isoperimetric profiles, and multi-point estimates in geometric heat equations.  Surveys in differential geometry, 19(1):1–47, 2014.
  * [BCN+22] Theodora Bourni, Julie Clutterbuck, Xuan Hien Nguyen, Alina Stancu, Guofang Wei, and Valentina-Mira Wheeler.  The vanishing of the fundamental gap of convex domains in $\mathbb{H}^{n}$.  Ann. Henri Poincaré, 23(2):595–614, 2022.
  * [Bre14] Simon Brendle.  Two-point functions and their applications in geometry.  Bulletin of the American Mathematical Society, 51(4):581–596, 2014\.
  * [CE08] Jeff Cheeger and David G. Ebin.  Comparison theorems in Riemannian geometry.  AMS Chelsea Publishing, Providence, RI, 2008.  Revised reprint of the 1975 original.
  * [dC92] Manfredo Perdigão do Carmo.  Riemannian geometry.  Mathematics: Theory & Applications. Birkhäuser Boston, Inc., Boston, MA, 1992.
  * [DSW19] Xianzhe Dai, Shoo Seto, and Guofang Wei.  Fundamental gap comparison.  Surveys in Geometric Analysis 2018, pages 1–16, 2019.
  * [DSW21] Xianzhe Dai, Shoo Seto, and Guofang Wei.  Fundamental gap estimate for convex domains on sphere—the case $n=2$.  Comm. Anal. Geom., 29(5):1095–1125, 2021.
  * [EH90] J.-H. Eschenburg and E. Heintze.  Comparison theory for Riccati equations.  Manuscripta Math., 68(2):209–214, 1990.
  * [FRV12] Alessio Figalli, Ludovic Rifford, and Cédric Villani.  Nearly round spheres look convex.  American Journal of Mathematics, 134(1):109–139, 2012.
  * [Gag90] Michael E. Gage.  Curve shortening on surfaces.  Ann. Sci. École Norm. Sup. (4), 23(2):229–256, 1990.
  * [GLL16] Fuzhou Gong, Huaiqian Li, and Dejun Luo.  A probabilistic proof of the fundamental gap conjecture via the coupling by reflection.  Potential Anal., 44(3):423–442, 2016.
  * [HWZ20] Chenxu He, Guofang Wei, and Qi S Zhang.  Fundamental gap of convex domains in the spheres.  American Journal of Mathematics, 142(4):1161–1191, 2020.
  * [KN22] Gabriel Khan and Xuan Hien Nguyen.  Negative curvature constricts the fundamental gap of convex domains.  arXiv:2211.06404, 2022.
  * [KNTW22] Gabriel Khan, Xuan Hien Nguyen, Malik Tuerkoen, and Guofang Wei.  Log-concavity and fundamental gaps on surfaces of positive curvature.  arXiv:2211.06403, 2022.
  * [LR13] Zhiqin Lu and Julie Rowlett.  The fundamental gap of simplices.  Comm. Math. Phys., 319(1):111–145, 2013.
  * [LW87] Yng-Ing Lee and Ai Nung Wang.  Estimate of $\lambda_{2}-\lambda_{1}$ on spheres.  Chinese Journal of Mathematics, pages 95–97, 1987.
  * [Ni13] Lei Ni.  Estimates on the modulus of expansion for vector fields solving nonlinear equations.  J. Math. Pures Appl. (9), 99(1):1–16, 2013.
  * [SWW19] Shoo Seto, Lili Wang, and Guofang Wei.  Sharp fundamental gap estimate on convex domains of sphere.  Journal of Differential Geometry, 112(2):347 – 389, 2019.
  * [SWYY85] I.M. Singer, Bun Wong, Shing-Tung Yau, and Stephen S.-T. Yau.  An estimate of the gap of the first two eigenvalues in the Schrödinger operator.  Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 12(2):319–333, 1985.
  * [SY94] Richard M Schoen and Shing-Tung Yau.  Lectures on differential geometry, volume 2.  International press Cambridge, MA, 1994.
  * [VdB83] M Van den Berg.  On condensation in the free-boson gas and the spectrum of the Laplacian.  Journal of Statistical Physics, 31:623–637, 1983.
  * [Wan00] F-Y Wang.  On estimation of the Dirichlet spectral gap.  Archiv der Mathematik, 75:450–455, 2000.
  * [Yau86] Shing-Tung Yau.  Nonlinear analysis in geometry, Monographies de L’Enseignement Mathématique, vol. 33.  L’Enseignement Mathématique, Geneva, page 54, 1986.
  * [YZ86] Qi Huang Yu and Jia Qing Zhong.  Lower bounds of the gap between the first and second eigenvalues of the Schrödinger operator.  Transactions of the American Mathematical Society, 294(1):341–349, 1986.

##  Appendix A Convex domains whose fundamental gap is less than
$\frac{3\pi^{2}+\epsilon}{D^{2}}$

In this appendix, we construct domains on a surface which satisfy the estimate

$\Gamma(\Omega)\leq\frac{3\pi^{2}+\epsilon}{D^{2}},$

which implies that the leading term of our estimate cannot be improved.

This proof follows from elliptic regularity, but it requires a bit of care
because if one fixes the diameter, sufficiently thin domains actually have
very large fundamental gap (See Theorem 1.1 of [KNTW22]).

######  Proposition A.1.

Let $(M^{2},g)$ be a Riemannian surface. Then there is a convex domain
$\Omega\subset M^{2}$ which satisfies

(A.1) $\Gamma(\Omega)\leq\frac{3\pi^{2}+\epsilon}{D^{2}}$

###### Proof.

We start by considering four points in $\mathbb{R}^{2}$ whose convex hull form
a rectangle $R$ satisfying

$\Gamma(R)\leq\frac{3\pi^{2}+\frac{\epsilon}{4}}{L^{2}},$

where $L$ is the diameter of $R$ in terms of Euclidean distance. We use the
convention that the long side of $R$ are the _horizontal sides_ and the
shorter sides of $R$ are the _vertical sides_. Furthermore, we rescale the
rectangle so that the horizontal sides have length $1$ in Euclidean distance
and center the rectangle at the origin.

We then deform $R$ slightly by replacing all the sides by circular arcs of
radius $r\gg 1$, which bow outward. We call this domain $\Omega$, which is now
strongly convex in the sense that curvature of its boundary has a positive
lower bound. We pick $r$ sufficiently large so that

$\Gamma(\Omega)\leq\frac{3\pi^{2}+\frac{\epsilon}{2}}{L^{2}}.$

Note that the diameter of $\Omega$ might not be exactly the same as the
diameter of $R$, but will be very close.

We now consider a geodesic $\gamma$ in $M$ and let $(x,y)$ be Fermi
coordinates defined along $\gamma$ (see [KN22, Section 3.1] for a definition
of these coordinates). We then let $\rho$ be a positive parameter and define
$\rho\Omega\subset M^{2}$ to be the image of $\rho\Omega$ in these Fermi
coordinates. In other words, we consider the set $\rho\Omega$ as a subset of
$\mathbb{R}^{2}$ and then take the corresponding domain in $M$ under the
coordinate chart. Our goal now is to show that for sufficiently small $\rho$,
the domain is convex and satisfies the gap estimate (A.1).

###  A.1. Convexity of $\rho\Omega$

The same section of [KN22] shows that by taking $\rho$ small enough (small
enough here depends on bounds on the curvature and its first two derivatives,
as well as the height of the original rectangle $R$), the metric is $C^{2}$
close to a Euclidean metric in the Fermi coordinates. Hence, by taking
$\rho\ll\frac{1}{r}$ (so potentially even smaller), all of the circular arcs
defining the boundary of $\rho\Omega$ have positive geodesic curvature with
respect to the metric $g$. As such, the domain $\Omega$ is convex.

###  A.2. Continuity of the spectrum

To show that the fundamental gap of $\rho\Omega$ is not too large, we perform
a rescaling argument. More precisely, we rescale the metric so that the
diameter of $\rho\Omega$ is $1$. The rescaling which achieves this is roughly
$\frac{g}{\rho}$, but there are some lower order corrections.

Doing so, the curvature and all of its derivatives can be made arbitrarily
small (by taking $\rho$ sufficiently small). By elliptic regularity, the
spectrum will depend continuously on the coefficients of the metric. As a
result, we can choose ${\rho}$ sufficiently small so that the fundamental gap
of ${\rho\Omega}$ in the rescaled metric is $\frac{\epsilon}{2}$ from the
fundamental gap of the original domain $\Omega$ in the Euclidean metric. As
such, in the rescaled metric we have that

$\Gamma(\rho\Omega)\leq\frac{3\pi^{2}+\epsilon}{1^{2}}.$

However, this estimate is scale-invariant, so returning the metric back to its
original scale, we recover the estimate

$\Gamma(\rho\Omega)\leq\frac{3\pi^{2}+\epsilon}{D^{2}}.$

∎

In practice, what this is doing is taking a rectangle whose fundamental gap is
close to $3\pi^{2}$ and making it small enough so that the effects of the
curvature are negligible. This is a little bit subtle, because if one
considers domains of a fixed diameter in a positively curved surface and then
shrinks the inradius to zero, the fundamental gap will go to infinity (See
Theorem 1.1 of [KNTW22]). In particular, this argument requires that we shrink
the diameter and inradius to zero simultaneously.

######  Remark A.2.

In higher dimensions, one can apply a similar construction to obtain convex
domains whose gap is $\leq\frac{3\pi^{2}+\epsilon}{D^{2}}.$

To do so, we consider a rectangular paralleliped whose sides are all very
short except for one. For such a domain in Euclidean space, the fundamental
gap can be made arbitrarily close to 1.2.Then, we consider a domain $\Omega$
which is an approximation to a long rectangular parallelipiped. However, it is
important that this approximation is _strongly convex_ , in that there is a
positive (albeit small) lower bound on the second-fundamental form of its
boundary. From this, the construction can be repeated mutatis mutandis.


11footnotetext: Project page: https://yuhui-zh15.github.io/VLMClassifier-
Website/.

# Why are Visually-Grounded Language Models
Bad at Image Classification?

Yuhui Zhang1† Alyssa Unell1 Xiaohan Wang1 Dhruba Ghosh2 Yuchang Su3
Ludwig Schmidt1,2† Serena Yeung-Levy1†
1Stanford University 2University of Washington 3Tsinghua University
<EMAIL_ADDRESS>

###### Abstract

Image classification is one of the most fundamental capabilities of machine
vision intelligence. In this work, we revisit the image classification task
using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We
find that existing proprietary and public VLMs, despite often using CLIP as a
vision encoder and having many more parameters, significantly underperform
CLIP on standard image classification benchmarks like ImageNet. To understand
the reason, we explore several hypotheses concerning the inference algorithms,
training objectives, and data processing in VLMs. Our analysis reveals that
the primary cause is data-related: critical information for image
classification is encoded in the VLM’s latent space but can only be
effectively decoded with enough training data. Specifically, there is a strong
correlation between the frequency of class exposure during VLM training and
instruction-tuning and the VLM’s performance in those classes; when trained
with sufficient data, VLMs can match the accuracy of state-of-the-art
classification models. Based on these findings, we enhance a VLM by
integrating classification-focused datasets into its training, and demonstrate
that the enhanced classification performance of the VLM transfers to its
general capabilities, resulting in an improvement of 11.8% on the newly
collected ImageWikiQA dataset.

##  1 Introduction

The ability to recognize objects within images is a fundamental capability of
machine vision. Over the past 15 years, the field has experienced significant
breakthroughs due to deep learning and large-scale datasets [10, 44]. For
instance, on the renowned ImageNet dataset, designed to classify images into
1,000 categories, the error rate has dramatically decreased from 47.1% in 2009
to 9.1% in 2024, representing a 5-fold reduction [27, 11]. Consequently, these
classification models have superseded most human labelers.

Nowadays, the community has focused on more sophisticated and nuanced
capabilities in the quest for visual intelligence. Visually-grounded language
models (VLMs), which integrate visual signals from vision encoders with large
language models, have recently emerged as a promising paradigm [2, 33, 29].
VLMs like GPT-4V [33], Gemini-1.5 [40], or Claude-3 [3] have demonstrated
advanced visual understanding abilities, such as answering math questions from
table images or generating HTML code from design sketches.

In this work, we revisit the fundamental task of image classification using
VLMs. Surprisingly, we find that various public and proprietary VLMs struggle
with image classification in both open-world settings, where the class list is
unknown, and closed-world settings, where class names are provided in the
context (§2). Despite having many more parameters, there is a significant gap
between the performance of VLMs and their commonly used vision encoder CLIP
[34]. Our evaluation protocol involves feeding each image and a list of class
names (in the closed-world setting) to the VLM as context and asking what is
in the image; success is defined by whether the generated output contains the
ground-truth class name.

To understand why VLMs underperform in classification settings, we investigate
several hypotheses regarding VLMs’ inference (such as prompt variations, label
set size, inference strategy; §3.1), training (such as information lost,
training objective; §3.2), and data (such as data-performance correlation;
§3.3). Our extensive analyses suggest that the primary reason for the observed
gap is data. We find that the information necessary for classification is
encoded in the VLM’s latent space but can only be decoded with proper training
data. Specifically, there is a strong correlation between class presence
during VLM training and performance in those classes. Furthermore, training
VLMs on classification datasets achieves the same performance level as state-
of-the-art classification models.

Motivated by our analysis, we propose a simple method to enhance VLMs’ general
capabilities by integrating traditional classification-focused datasets into
VLM training (§4). We believe that classification is the foundation for more
complex, advanced visual capabilities; for example, recognizing an object is a
prerequisite for answering complex questions about it. To verify this, we
created ImageWikiQA, which contains complex real-world questions about
ImageNet objects. On ImageWikiQA, we find that VLMs fine-tuned on the ImageNet
classification dataset achieve substantially higher accuracy in recognizing
these objects and provide more accurate answers to these non-classification
questions, outperforming pre-trained VLMs by 11.8%. This suggests that
classical classification data can be beneficially reused in the VLM training
process to enhance VLM performance.

Figure 1: Overview. (Left) Different visually-grounded language models (VLMs)
underperform CLIP in classification by a large margin, though they often use
CLIP as a vision encoder. (Middle) We investigate several hypotheses about why
VLMs are bad classifiers and find that the main reason is data. Critical
information for image classification is encoded in the VLM’s latent space but
can only be decoded with enough data during VLM training. (Right) Based on our
analysis, we improve a VLM by integrating classification data into its
training, and find that the improved classification capabilities serve as
foundations for more advanced capabilities such as visual question answering.

##  2 VLMs are Bad at Image Classification

We begin by evaluating state-of-the-art visually-grounded language models
(VLMs) using standard image classification benchmarks. Our findings reveal
that these VLMs significantly underperform compared to state-of-the-art
classification models, such as CLIP.

###  2.1 Models

#### VLMs.

We selected ten widely-used state-of-the-art VLMs, covering different
architectures, training methods, and data. These VLMs include three
proprietary ones, GPT-4-Turbo (shortened as GPT4, same below) [33], Gemini-
Pro-Vision (GeminiPro) [40], and Claude-3-Opus (Claude3) [3], and seven public
ones, LLaVA1.5-Vicuna7B/13B (LLaVA1.5-7/13B) [29], LLaVANeXT-
Mistral7B/Vicuna7B (LLaVANeXT-M7B/V7B) [28], BLIP2-OPT2.7B (BLIP2-2.7B) [25],
and InstructBLIP-Vicuna7B/13B (IBLIP-7/13B) [9]). Details of these models are
provided in Appendix §A.1.

#### CLIPs.

For comparison, we used two state-of-the-art image classifiers, CLIP-
ViT-L/14-336px (shortened as CLIP-L, same below) [34] and EVA-ViT-G/14 (EVA-G)
[39]. Notably, CLIP-L and EVA-G are utilized by the LLaVA series [29] and the
BLIP series as vision encoders [25], respectively. Therefore, the VLMs should
theoretically have the same classification capacity as these vision models.
Details are listed in Appendix §A.1.

###  2.2 Data

We evaluated the aforementioned models on four widely-used image
classification benchmarks: ImageNet (shortened as , same below) [10],
Flowers102 () [32], StanfordCars () [19], and Caltech101 () [12], which
contain 50,000, 6,149, 8,041, and 4,331 test images from 1,000, 102, 196, and
101 classes, respectively. ImageNet and Caltech cover more coarse-grained
objects, while Flowers and Cars cover more fine-grained objects. Further
details are provided in Appendix §A.2.

###  2.3 Evaluation Protocol

#### VLMs.

We performed image classification in two settings: an open-world setting where
the label set is not provided and a closed-world setting where classes are
concatenated in the prompt. We feed the image and the prompt to the VLM and
let the VLM complete the rest of the tokens. One closed-world example is:
“<image> What type of object is in this photo? Choose one from <class name A>,
<class name B>, …” We define success on a single example as whether the
ground-truth label is included in the VLM generation. We report the success
rate of all test examples.

#### CLIPs.

CLIP can only be used in a closed-world setting where the label set is known.
Following Radford et al. [34], we used the prompt “a photo of a <class>” to
generate the text feature. For each image, we selected the class with the
highest cosine similarity to the image feature. We did not include prompt
ensembling to fairly compare with VLM. We report the accuracy of all examples.

###  2.4 Results

Model |  | Open-World Setting |  | Closed-World Setting
---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
Public VLM
BLIP2-2.7B [25] |  | 25.3 | 27.0 | 0.0 | 46.9 |  | N/A | 14.2 | 2.7 | 22.3
IBLIP-7B [9] |  | 14.6 | 1.9 | 0.0 | 36.5 |  | N/A | 26.8 | N/A | 58.4
IBLIP-13B [9] |  | 14.7 | 2.4 | 0.0 | 36.4 |  | N/A | 20.0 | N/A | 59.5
LLaVA1.5-7B [29] |  | 22.8 | 5.9 | 0.0 | 47.1 |  | N/A | 10.2 | 0.0 | 62.1
LLaVANeXT-V7B [29] |  | 29.4 | 12.8 | 0.0 | 52.5 |  | N/A | 8.5 | 0.0 | 66.6
LLaVA1.5-13B [29] |  | 24.3 | 5.3 | 0.0 | 49.9 |  | N/A | 7.2 | 0.1 | 70.9
LLaVANeXT-M7B [29] |  | 32.3 | 17.7 | 0.0 | 54.2 |  | N/A | 16.1 | 3.6 | 77.3
Proprietary VLM
Claude3 [3] |  | 53.6 | 51.2 | 0.3 | 68.6 |  | 51.1 | 58.3 | 45.1 | 90.9
GeminiPro [40] |  | 39.2 | 10.5 | 0.1 | 60.1 |  | 56.0 | 62.0 | 66.6 | 91.6
GPT4 [33] |  | 48.5 | 51.0 | 0.1 | 61.0 |  | 60.6 | 79.9 | 58.2 | 94.2
CLIP
CLIP-L [34] |  | N/A | N/A | N/A | N/A |  | 74.8 | 76.0 | 77.5 | 95.8
EVA-G [39] |  | N/A | N/A | N/A | N/A |  | 79.2 | 81.0 | 90.2 | 97.9
Table 1: Evaluations of VLMs and CLIPs on standard image classification
benchmarks. VLMs exhibit poor performance in image classification,
significantly lagging behind CLIP models. =ImageNet [10], =Flowers102 [32],
=StanfordCars [19], =Caltech101 [12].

Table 1 reports the performance of different VLMs and CLIP models on these
classification datasets.

We find that VLMs exhibit poor performance in image classification,
significantly lagging behind CLIP models. For instance, on the ImageNet
dataset, the best proprietary VLM, GPT4, only achieves an accuracy of 60.6% in
the closed-world setting, whereas the best CLIP, EVA-G, attains an accuracy of
79.2%. In the open-world setting, the best public VLM, LLaVANeXT-M7B, achieves
just 32.3% accuracy. The performance disparity is even more pronounced in
fine-grained classification datasets like Flowers102 and StanfordCars.
Notably, all LLaVA models use CLIP-L as the vision encoder, and although the
total parameter count of these models is at least 20 times greater than that
of the vision encoder, they significantly underperform compared to it.

Moreover, we find that closed-world setting often outperforms open-world
setting. This is expected as the provided label set narrows the prediction
space. However, since closed-world settings require including all class names
in the context, they can result in an extremely long context that leads to
high costs or even exceeds the VLM’s context limit. For example, most public
VLMs only support 4K context length, which cannot feed 1K ImageNet classes. We
also find that larger and better LMs slightly improve VLM performance. For
instance, LLaVA1.5-13B outperforms LLaVA1.5-7B, and LLaVANeXT-M7B outperforms
LLaVANeXT-V7B.

##  3 Why are VLMs Bad Image Classifiers?

Given that visually-grounded language models (VLMs) underperform CLIPs at
classification by a large margin, as reported in §2, we seek to understand the
reasons behind that. We investigate several hypotheses concerning major
differences between VLMs and CLIPs, which can be generally categorized into
inference (§3.1), training (§3.2), and data (§3.3):

  1. 1.

We start with inference-related questions. For example, does prompt variation,
such as chain of thought, affect final performance? Does reducing the label
set size in context narrow the gap between VLMs and CLIPs? Does performing
probabilistic inference to force the generation into the label set help? We
find none of these factors can fully close the gap between VLMs and CLIPs.

  2. 2.

Therefore, we switch to training-related questions. For example, is the visual
information from the vision encoder still preserved in the VLM’s latent space?
Is the text generation objective as effective as cross-entropy loss for
learning classification? Surprisingly, the results show that the information
is preserved, and the text generation objective is adequate for learning
classification.

  3. 3.

Finally, we investigate data-related questions. For example, does the VLM
training data include enough classification data and cover enough classes? We
find a strong correlation between class exposure in training and model
performance. Moreover, VLMs can achieve the same level of performance as CLIPs
when trained with enough data. These results suggest that data is the primary
cause of the poor classification performance of VLMs.

###  3.1 Inference

Method |  | LLaVA1.5-7B |  | BLIP2-2.7B
---|---|---|---|---
|  |  |  |  |  |  |  |  |  |
Prompt Variation
Base Prompt |  | 22.8 | 5.9 | 0.0 | 47.1 |  | 25.3 | 27.0 | 0.0 | 46.9
w/ Prompt Alternative 1 |  | 19.7 | 3.5 | 0.0 | 45.4 |  | 27.6 | 38.8 | 0.2 | 47.8
w/ Prompt Alternative 2 |  | 21.6 | 6.6 | 0.0 | 48.1 |  | 24.3 | 30.6 | 0.1 | 47.9
w/ Label (Fixed Order) |  | N/A | 6.1 | 0.1 | 70.5 |  | N/A | 28.0 | 2.1 | 53.8
w/ Label (Random Order) |  | N/A | 10.2 | 0.0 | 62.1 |  | N/A | 14.2 | 2.7 | 22.3
w/ Label (Random) + CoT |  | N/A | 18.1 | 0.4 | 64.5 |  | N/A | 8.5 | 0.0 | 16.1
Inference Strategy
Direct Generation |  | 22.8 | 5.9 | 0.0 | 47.1 |  | 25.3 | 27.0 | 0.0 | 46.9
Prob Inference (Sum Tokens) |  | 34.8 | 14.5 | 26.7 | 77.8 |  | 21.0 | 34.8 | 48.8 | 36.8
Prob Inference (Avg Tokens) |  | 35.3 | 16.5 | 18.2 | 65.6 |  | 5.1 | 19.9 | 1.2 | 12.3
Prob Inference (Avg) w/ CFG |  | 47.6 | 26.8 | 48.8 | 85.6 |  | 38.7 | 54.1 | 69.2 | 70.3
Table 2: Analysis of VLMs from the inference perspective. (Top) We explore
prompt variation such as wording, label order, chain-of-thought and find it
has limited impact on the performance. (Bottom) We leverage the probabilistic
inference strategy, which improves the performance but still fails to close
the gap between VLMs and CLIPs.

In this section, we investigate three questions related to VLM’s inference,
including prompt variation, label set size, and inference strategy.

#### Prompt variation.

It is well known that language models (LMs) are sensitive to prompts [6, 15,
48]. To understand the effect of prompts on classification performance, we
tested three semantically similar but differently worded prompts, compared
feeding the label in dataset order or random order within the context, and
leveraged the zero-shot chain-of-thought (CoT) prompting technique by adding
“let’s think step by step” at the end of the prompt [43, 18].

We find that prompt variation has a limited impact on the performance. From
Table 2, we can see that changing the wording of prompts results in a
performance variation within 3% for both LLaVA1.5-7B and BLIP2-2.7B on the
ImageNet dataset. Different label orderings impact LLaVA1.5-7B less than
BLIP2-2.7B. Chain-of-thought prompting consistently improves performance for
instruction-tuned model LLaVA1.5-7B but not for BLIP2-2.7B.

#### Label set size.

The label set size can be very large in practice (e.g., 1000 for ImageNet, 196
for StanfordCars), which results in an extremely long context when we
concatenate all the class names. Since LMs often struggle with long contexts
[30], we explore reducing the number of classes. For each image, we randomly
select $K=2,5,20,100$ classes, always including the ground-truth label, and
re-evaluate the VLM and CLIP performance.

We find that the performance gap between VLMs and CLIPs narrows with reduced
label size, but the gap always exists. As shown in Figure 2, when evaluating
LLaVA1.5-7B and CLIP-L on ImageNet, the gap decreases from 46% with 100
classes to 6% with 2 classes. However, the relative gap becomes larger,
evidenced by a 23.9x error rate with 2 classes compared to a 7.3x error rate
with 100 classes. The performance gap between VLMs and CLIPs always exists in
all the settings.

#### Inference algorithm.

The default inference algorithm for classification with VLMs is directly
generating the class name given a prompt. As the generation is open-ended,
even when provided with a list of candidate choices, the generation may not
match one of the pre-defined classes. To mitigate this problem, we employ
probabilistic inference techniques for VLMs. Specifically, for each class
name, we compute $\text{prob}(\text{class name}|\text{image},\text{prompt})$
and select the class name with the highest probability as the prediction.
Since class names can consist of multiple tokens (e.g., “guinea pig” consists
of two tokens), we either average the probabilities of all tokens or sum up
all the tokens [6]. We also explore classifier-free guidance (CFG) techniques
by ranking $t*\text{prob}(\text{class
name}|\text{image},\text{prompt})+(1-t)*\text{prob}(\text{class
name}|\text{prompt})$ with varying guidance coefficients $t$ [35, 45].

We find that the probabilistic inference method improves the performance, but
the gap persists. As shown in Table 2, LLaVA1.5-7B achieves 35.3% accuracy on
ImageNet using probabilistic inference compared to 22.7% using the direct
generation method. Adding classifier-free guidance further improves the
performance, where LLaVA1.5-7B performance boosts to 47.6% on ImageNet.
However, it still leaves around a 30% performance gap between VLMs and CLIPs,
as its vision encoder CLIP-L achieves 74.8% on ImageNet. Moreover, the
probabilistic inference approach is computationally expensive in practice
because we need to compute the probability of each class.

Figure 2: Analysis of the label set size. For each image, we randomly sample
100, 20, 5, 2 candidate classes from all the classes. The performance gap
between VLMs and CLIPs becomes smaller when the number of classes is reduced.
X-axis: number of classes; Y-axis: accuracy (%).

###  3.2 Training

Since inference-based modifications fail to close the performance gap between
VLMs and CLIPs, here we investigate two questions regarding to the training of
VLMs. We study whether the visual information is lost in the VLM and whether
the text generation objective is suitable for learning the classification
task.

#### Visual information lost in the VLM.

VLMs process images using an image encoder, such as CLIP, which has strong
classification capabilities. We hypothesize that, during the propagation of
image features output from the vision encoder in language model layers, the
necessary information for classification is lost. To test this hypothesis, we
conduct feature probing experiments. Specifically, the features from the VLM’s
last layer have a shape corresponding to the length of the inputs (image
tokens plus text tokens using the open-world prompt). We take the average of
these token features or the last token feature and train a simple linear
classifier on top of these frozen feature representations. The linear
classifier is trained on the training set and evaluated on the validation set.
Training details are provided in Appendix B.4. A higher accuracy indicates
that less information is lost.

Surprisingly, we find that the information necessary for classification is
largely preserved in the VLM’s latent space; however, it cannot be effectively
decoded. In Table 3, we show that the probing accuracy of LLaVA1.5-7B on
ImageNet is 77.1%, which is close to the 85.2% probing accuracy of CLIP-L, the
vision encoder used by LLaVA1.5-7B. The same conclusion holds for BLIP2-2.7B
and its vision encoder EVA ViT-G/14, demonstrating that most information is
preserved during the VLM computational process. However, this information
cannot be effectively decoded, as §2 shows that the best accuracy LLaVA1.5-7B
can achieve on ImageNet is 47.6%.

Model Feature Probing LLaVA1.5-7B Last Tok 76.9 94.5 81.0 96.7 LLaVA1.5-7B Avg Tok 77.1 96.2 82.8 97.3 BLIP2-2.7B Last Tok 80.3 98.8 91.0 98.0 BLIP2-2.7B Avg Tok 81.4 98.9 92.6 98.0 |  | Model | Trainable |  |  |  |
---|---|---|---|---|---
Fine-tuning
CLIP-L | Linear | 85.2 | 98.6 | 91.5 | 97.6
LLaVA1.5-7B | Proj | 85.7 | 97.6 | 90.4 | 97.5
LLaVA1.5-7B | Proj+LM | NaN | 97.9 | 90.7 | 97.8
EVA-G | Linear | 86.5 | 99.2 | 94.3 | 98.5
BLIP2-2.7B | Proj | 88.0 | 99.0 | 93.9 | 98.8
Table 3: Analysis of VLMs from the training perspective. (Left) We conduct
feature probing experiments on the VLM’s last layer and find that the
information required for classification is mostly preserved in the VLM’s
latent space. (Right) We fine-tune VLMs on the classification datasets using
the text generation objective and find that the text generation training
objective is as effective as the traditional cross-entropy for learning
classification, which eliminates VLM-CLIP performance gap.

#### Training objective.

Since information is encoded in VLMs, we wonder whether we can train them to
decode the information. VLMs can only be trained to perform classification by
auto-regressively generating text-form labels. We hypothesize that this
generative objective may be more difficult and less effective in learning
classification compared to the traditional cross-entropy loss. To understand
the effectiveness of this training objective, we explore fine-tuning VLMs on
classification datasets. We convert classification datasets into the text
generation format using the template (e.g., “<image> What type of object is in
this photo? <class name>”). We fine-tune two model architectures (LLaVA1.5-7B
and BLIP2-2.7B) across different datasets and compare their accuracy to that
of fine-tuned CLIP models. Additionally, we investigate fine-tuning different
parts of the models, such as only fine-tuning the projector between vision
encoder and language models or fine-tuning the projector along with the
language models using LoRA [16]. Training details are provided in Appendix
B.5.

We find that the text generation training objective is as effective as the
traditional cross-entropy for learning classification tasks, which eliminates
the performance gap between VLMs and CLIPs. From Table 3, we find that
LLaVA1.5-7B achieves 85.7% accuracy on ImageNet, same as 85.2% accuracy when
fine-tuning its vision encoder CLIP-L. The same findings apply to all the VLMs
on all the datasets.

Moreover, we find that fine-tuning only the projector is sufficient and has
better numerical stability. From Table 3, we can see the same level of
accuracy achieved by fine-tuning the projector and fine-tuning projector and
LLM using LoRA on the three datasets. We also find that fine-tuning LLM with
LoRA often leads to numerical instabilities, such as spikes in loss. While the
loss sometimes returns to normal, other times it does not. For example,
despite trying various hyperparameters for ImageNet, instability persisted
(see Appendix §B.5). In contrast, fine-tuning only the projector always
results in a steadily decreasing loss. Notably, the projector is often much
smaller (e.g., a 2-layer MLP in LLaVA) compared to LLMs, suggesting the
potential for parameter-efficient prefix tuning for VLMs [26].

###  3.3 Data

As observed, the visual information is preserved in the VLM’s latent space,
and the VLM’s training objective is sufficient for learning classification
tasks. We hypothesize that the poor classification performance of VLMs is due
to their training data. For example, there might be insufficient
classification data or a lack of diverse classes. To investigate this, we
analyze the LLaVA1.5-7B training data [29], the only fully publicly available
VLM training dataset. We examined the relationship between the frequency of
class occurrence and the VLM’s classification performance on those
classes***LLaVA training contains two stages. Here we combine the data from
the two stages. More in Appendix B.6..

Our findings indicate that data determines VLM classification performance. As
shown in Figure 3, there is a strong correlation between the presence of class
labels in the VLM training data and the VLM’s classification accuracy for
those classes. The LLaVA1.5-7B model achieves 82.7% zero-shot accuracy on
ImageNet classes with more than 10,000 occurrences in its training data, but
only 3.0% accuracy on classes with fewer than 10 occurrences. The Spearman
correlation between class frequency and class performance is very high (0.76
on ImageNet; see Appendix §B.6). In contrast, there is no correlation between
its vision encoder CLIP-L’s performance on those classes with the same VLM
training data, and after fine-tuning LLaVA1.5-7B on the ImageNet
classification data, the strong correlation disappears. Combining all of our
results, we conclude that data plays a critical role in determining the VLM’s
classification performance. In Appendix §D, we further show that the data type
(e.g., classification or captioning data of a given class) is unimportant.

Figure 3: Analysis of VLMs from the data perspective. We study the relation
between the ImageNet class frequency in the VLM training data and the VLM’s
classification performance on those classes. A strong correlation is observed,
indicating that data determines VLM classification performance.

##  4 Improving VLM with Classification Data

Based on the analysis presented in §3, in this section, we discuss the
enhancement of visually-grounded language models (VLMs) by integrating
classification-focused data into their training. We demonstrate that this data
intervention not only boosts the VLM’s classification performance but also
enhances its general capabilities.

###  4.1 Motivation

Classification is fundamental to enabling more advanced capabilities of VLMs,
such as visual question answering and reasoning. For example, suppose a
virtual assistant is helping visually impaired individuals prepare mushrooms
as food. In that case, the model must correctly identify the mushroom species
to answer questions like “Is this mushroom poisonous?” The poor classification
performance of VLMs lays a weak foundation for their advanced capabilities.

From §3, we identify the primary cause of poor classification performance is
the lack of data. Therefore, we propose a straightforward solution:
integrating classification-focused data into the VLM training process. We hope
that incorporating classification data not only enhances classification
accuracy but also improves general capabilities.

| Model | Acc
---|---
Naive Baselines
Random | 25.0
Max Freq | 25.9
GPT4 w/ GT Class | 100.0
GPT4 w/o Image | 0.0
Human w/ GT Class+Wiki | 96.5
LLaVA1.5-7B w/ GT Class | 55.9
Proprietary VLM
GeminiPro [40] | 49.1
Claude3 [3] | 54.3
GPT4 [33] | 61.2
| Model | Acc
---|---
Public VLM
BLIP2-2.7B [25] | 21.7
IBLIP-7B [9] | 36.3
LLaVANeXT-V7B [28] | 37.0
IBLIP-13B [9] | 37.5
LLaVA1.5-13B [29] | 37.8
LLaVA1.5-7B [29] | 38.0
LLaVANeXT-M7B [28] | 41.9
Finetuned VLM
LLaVA1.5-7B Finetuned on ImageNet | 30.6
LLaVA1.5-7B Finetuned on ImageNet+LLaVA | 49.8
Table 4: Evaluations of VLMs on ImageWikiQA. ImageWikiQA is a multiple-choice
question-answering dataset collected by feeding the Wikipedia pages of
ImageNet classes to GPT-4. We find that current VLMs perform poorly in
answering these questions, suggesting that their poor classification
performance is a fundamental limitation for more advanced capabilities.
Integrating classification data into VLM training enhances both their
classification and overall capabilities.

###  4.2 ImageNetQA

To verify our hypothesis, we need a dataset that evaluates both the
classification and advanced capabilities of VLMs. However, current visual
question answering benchmarks, such as VQAv2 [14], MM-Vet [46], and MMMU [47],
primarily focus on advanced capabilities, such as reasoning and knowledge
grounding, rather than classification. The objects in these datasets are
relatively simple, and questions can often be answered by identifying only the
general category of an image, such as “mushroom” or “flower”, rather than
specific types of mushrooms or flowers. In contrast, classification datasets
have no questions related to more advanced capabilities.

To bridge the gap between classification and advanced capabilities, we
introduce ImageWikiQA, an object-centric, knowledge-intensive question
answering dataset that combines both worlds. Each question in ImageWikiQA is a
multiple-choice question with four options and one correct answer. Although
each question does not directly ask for the category of the object within the
image, the question can only be accurately answered if the class of the object
is correctly identified. Example questions from ImageWikiQA can be found in
Figure 1 and Appendix §C.1.

ImageWikiQA is created by generating questions based on Wikipedia pages of
ImageNet classes using GPT4. Specifically, for each class in ImageNet, we
parsed the Wikipedia content of the class following Bujwid et al. [7], and
then provided this content along with a prompt to GPT4 to generate five
questions per class. We instructed GPT4 to replace the class name with “this
object” in the questions so that the ground-truth class name is not provided.
We retained all questions that GPT4 could answer correctly with the ground-
truth class name and could not guess the answer without the class name. To
ensure the question quality generated by GPT4, four human annotators attempted
to answer the questions with the ground-truth class name and Wikipedia page
and achieved an accuracy of 96.5%. Afterward, we randomly sampled at most 3
ImageNet images for each question, rebalanced the choice distribution, and
composed the final ImageWikiQA dataset. In total, there are 2000 multiple-
choice questions, each with an image, question, four candidate choices, and a
reference to Wikipedia sentences, with a random guess accuracy of 25.0% and
max frequency accuracy of 25.9%.

###  4.3 Results

Table 4 presents the performance of various VLMs on the ImageWikiQA dataset.

We find that current state-of-the-art VLMs perform poorly on answering these
questions given images. For example, GPT4 achieves 100% accuracy when the
ground-truth class name is provided, but only achieves 61.2% accuracy with
images. Similarly, Claude3 and GeminiPro only achieve 54.3% and 49.1%
accuracy, respectively. These results indicate that the poor classification
performance of VLMs is a fundamental limitation for more advanced
capabilities.

Furthermore, we find that integrating classification data into VLM training
improves its classification and general capabilities. We fine-tuned
LLaVA1.5-7B on the ImageNet 1.28M classification data and original 665K LLaVA
instruction-tuning data (training detail in §C.2), which is able to achieve
84.4% accuracy on ImageNet classification compared to 22.8% for non-fine-tuned
models. This improvement in classification translates to an 11.8% accuracy
boost on ImageWikiQA, demonstrating classification is indeed a foundation for
VLM’s advanced capabilities. However, it is worth noting that fine-tuning
solely on the classification task harms general capabilities. When fine-tuning
LLaVA1.5-7B on ImageNet only, their accuracy on ImageWikiQA drops to 30.6%.
Therefore, fine-tuning should be performed on a combined dataset. Developing
fine-tuning methods that prevent such catastrophic forgetting is a promising
future research direction.

##  5 Related Work

#### Visually-grounded language models.

Visually-grounded language models (VLMs) refer to a large family of models
that integrate visual signals into language models by modeling
$p(y_{t}|y_{<t},x)$, where $y_{i}$ is a text token and $x$ is a visual input
such as an image or video. Recently, many powerful VLMs have been developed,
including proprietary ones like GPT-4V [33], Gemini [40], Claude [3], Flamingo
[2] and public ones like LLaVA [29, 28], BLIP [25, 9], OpenFlamingo [4], Otter
[24], Fuyu [1], QwenVL [5]. The typical architecture of VLMs comprises three
components: a visual encoder (often CLIP [34]), a language model, and a
projector that bridges the visual outputs and language model inputs. The
projector can be as simple as a linear layer or MLP (e.g., LLaVA [29]) or a
complex architecture such as Transformer with cross-modal attention (e.g.,
BLIP [25]). VLMs are usually trained on image-text captioning data [37, 36]
and carefully designed instruction-tuning data [29], with both the vision
encoder and language model initialized from pre-trained versions. In this
work, we evaluate widely used VLMs in classification settings and analyze two
prominent architectures, LLaVA [29] and BLIP [25].

#### Analysis of visually-grounded language models.

While VLMs have demonstrated impressive performance, many questions remain
unanswered. Recent works have explored their limitations from various
perspectives, including architectures [31], training recipes [17, 22], data
[13, 42], vision encoders [41], language models [2, 40], input resolution
[38], and proposed solutions for these limitations. For instance, Tong et al.
[41] discovered that CLIP vision encoders, employed by most VLMs, often fail
to distinguish between certain image pairs despite their apparent visual
differences. They suggested utilizing alternative vision encoders, such as
DINO, to address this problem. Our work aligns with these studies in
understanding VLM limitations and proposing solutions. We find that current
VLMs struggle with image classification, and we thoroughly investigate the
reasons behind this, proposing a simple solution to integrate classification-
focused data into VLM training.

#### Image classification.

Image classification is one of the most fundamental capabilities of machine
intelligence. Starting in the 1990s, researchers collected datasets like MNIST
[23] for digit classification and CIFAR [20] for object classification, using
various machine learning algorithms such as SVMs and MLPs to tackle the
problem. A significant milestone was the ImageNet [10], which elevated the
scale and quality of datasets to a new level, driving advancements in deep
learning. With the rise of deep supervised and self-supervised learning [21,
8], performance on these datasets soon began to saturate. As a result,
classification models have superseded most human labelers, leading researchers
to focus on more advanced visual intelligence tasks like visual reasoning. In
this work, we revisit this simple yet fundamental task, discovering that
current VLMs struggle with it. We demonstrate that classification remains
essential, as it forms the foundation for more advanced capabilities, and
enhancing VLMs’ classification capabilities improves their overall
performance.

##  6 Conclusion

In this work, we explored the use of visually-grounded language models (VLMs)
as image classifiers. We found that their performance is limited across
various datasets. We then analyzed the reasons behind these limitations and,
based on our findings, trained a VLM with enhanced general capabilities.

## References

  * [1] Adept.  Fuyu-8b: A multimodal architecture for ai agents, 2023.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In NeurIPS, 2022.
  * [3] Anthropic.  Introducing the next generation of claude, 2024.
  * [4] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  arXiv preprint arXiv:2308.01390, 2023.
  * [5] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  arXiv preprint arXiv:2308.12966, 2023.
  * [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In NeurIPS, 2020.
  * [7] Sebastian Bujwid and Josephine Sullivan.  Large-scale zero-shot image classification from rich and diverse textual descriptions.  In EACL Workshop, 2021.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  In NeurIPS, 2023.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [12] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In CVPR Workshop, 2004.
  * [13] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  In NeurIPS, 2023.
  * [14] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In CVPR, 2017.
  * [15] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In ICLR, 2020.
  * [16] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In ICLR, 2021.
  * [17] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh.  Prismatic vlms: Investigating the design space of visually-conditioned language models.  arXiv preprint arXiv:2402.07865, 2024.
  * [18] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  In NeurIPS, 2022.
  * [19] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In ICCV Workshop, 2013.
  * [20] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NIPS, 2012.
  * [22] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.  What matters when building vision-language models?  arXiv preprint arXiv:2405.02246, 2024.
  * [23] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [24] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  arXiv preprint arXiv:2305.03726, 2023.
  * [25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [26] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In ACL, 2021.
  * [27] Yuanqing Lin, Fengjun Lv, Shenghuo Zhu, Ming Yang, Timothee Cour, Kai Yu, Liangliang Cao, and Thomas Huang.  Large-scale image classification: Fast feature extraction and svm training.  In CVPR, 2011.
  * [28] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
  * [29] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In NeurIPS, 2023.
  * [30] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.  Lost in the middle: How language models use long contexts.  TACL, 2023.
  * [31] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al.  Mm1: Methods, analysis & insights from multimodal llm pre-training.  arXiv preprint arXiv:2403.09611, 2024.
  * [32] M-E. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
  * [33] OpenAI.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [35] Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman.  Stay on topic with classifier-free guidance.  arXiv preprint arXiv:2306.17806, 2023.
  * [36] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [37] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [38] Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell.  When do we not need larger vision models?  arXiv preprint arXiv:2403.13043, 2024.
  * [39] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  arXiv preprint arXiv:2303.15389, 2023.
  * [40] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [41] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  arXiv preprint arXiv:2401.06209, 2024.
  * [42] Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge.  No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance.  arXiv preprint arXiv:2404.04125, 2024.
  * [43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In NeurIPS, 2022.
  * [44] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In ICML, 2022.
  * [45] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  TMLR, 2022.
  * [46] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  arXiv preprint arXiv:2308.02490, 2023.
  * [47] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  In CVPR, 2024.
  * [48] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.  Calibrate before use: Improving few-shot performance of language models.  In ICML, 2021.

## Acknowledgements

We thank Irena Gao, Jeffrey Gu, Weixin Liang, Alejandro Lozano, Shiori Sagawa,
Ruocheng Wang, Yunzhi Zhang, Orr Zohar for providing valuable feedback to this
project. This work is in part supported by the NSF AI Institute for
Foundations of Machine Learning (IFML), Open Philanthropy, and the Allen
Institute for AI. Serena Yeung-Levy is a Chan Zuckerberg Biohub — San
Francisco Investigator.

## Broader Impacts and Ethics Statement

In this work, we identify a core limitation of visually-grounded language
models (VLMs) and propose a simple solution to address this issue. We find
that VLMs underperform in image classification. By integrating classification
data into VLM training, we have demonstrated that this intervention not only
enhances classification accuracy but also improves the general capabilities of
VLMs. This advancement holds promise for real-world applications, such as
virtual assistants for visually impaired individuals. However, there are
potential risks associated with incorporating race-based or gender-based
classification data, which could amplify discriminatory biases in VLMs. We
strongly emphasize the importance of ethical and responsible use of our
approach to ensure it contributes positively to our digital ecosystem.

## Reproducibility Statement

We provide an open-source implementation of our work and have released the
ImageWikiQA dataset at https://github.com/yuhui-zh15/VLMClassifier. This will
enable researchers to reproduce all experiments described in this paper and
conduct their own analyses on additional VLMs and datasets.

## Limitations

Due to computational constraints, we cannot evaluate all possible VLMs and
datasets. We select two representative ones, LLaVA and BLIP, and conduct
analyses on four datasets. Additionally, as we find that VLMs have information
in their latent space, we conduct large-scale training to decode them.
Developing a zero-shot method to decode this information without extensive
training could be a promising direction for future research.

## Compute Resources

We use four NVIDIA L40S GPUs for all experiments. Most inference experiments
take 5-20 hours on a single L40S GPU. The most computationally intensive step
is fine-tuning. Fine-tuning LLaVA1.5-7B on 1.28M ImageNet data takes 1.4 days
for one epoch using four GPUs. Fine-tuning LLaVA1.5-7B on 1.28M ImageNet data
combined with 665K original LLaVA instruction-tuning data takes 3.5 days for
one epoch using four GPUs. For the 13B model, the cost is approximately 40%
higher.

## Summary of Appendix

We provide additional details for each section in the main paper.

  * •

In §A, we provide details of models and data for §2.

  * •

In §B, we provide details of experimental settings and results for §3.

  * •

In §C, we provide details of our collected ImageNetQA dataset for §4.

  * •

In §D, we discuss how data types affect VLM classification performance.

##  Appendix A Supplementary Section 2

###  A.1 Model

We include model details in Table 5.

Model | Link | Property | Version
---|---|---|---
GPT4 [33] | https://platform.openai.com/docs/guides/vision | Proprietary | gpt-4-turbo-2024-04-09
GeminiPro [40] | https://ai.google.dev/ | Proprietary | gemini-1.0-pro-vision-001
Claude3 [3] | https://console.anthropic.com/ | Proprietary | claude-3-opus-20240229
LLaVA1.5-7B [29] | https://huggingface.co/liuhaotian/llava-v1.5-7b | Public | -
LLaVA1.5-13B [29] | https://huggingface.co/liuhaotian/llava-v1.5-13b | Public | -
LLaVANeXT-V7B [28] | https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b | Public | -
LLaVANeXT-M7B [28] | https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b | Public | -
BLIP2-2.7B [25] | https://huggingface.co/Salesforce/blip2-opt-2.7b | Public | -
IBLIP-7B [9] | https://huggingface.co/Salesforce/instructblip-vicuna-7b | Public | -
IBLIP-13B [9] | https://huggingface.co/Salesforce/instructblip-vicuna-13b | Public | -
CLIP-L [34] | https://github.com/openai/CLIP | Public | -
EVA-G [39] | https://github.com/baaivision/EVA | Public | -

Table 5: Model details.

###  A.2 Data

We include data details in Table 6.

Dataset | Link | Training | Validation | # Classes
---|---|---|---|---
ImageNet [10] | https://www.image-net.org/ | 1.28M | 50K | 1000
Flowers102 [32] | https://www.tensorflow.org/datasets/catalog/oxford_flowers102 | 2.0K | 6.1K | 102
StanfordCars [19] | https://www.tensorflow.org/datasets/catalog/cars196 | 8.1K | 8.0K | 196
Caltech101 [12] | https://www.tensorflow.org/datasets/catalog/caltech101 | 4.3K | 4.3K | 101

Table 6: Data details.

##  Appendix B Supplementary for Section 3

###  B.1 Prompt Variation Detail

#### Prompt details.

We use three different prompts with varying wordings. Base Prompt: “What type
of object is in this photo?” Prompt Alternative 1: “Identify the object in
this image.” Prompt Alternative 2: “What is the main object depicted in this
photograph?”

#### Label order details.

For the fixed order, we use the default dataset order for each image.

###  B.2 Label Set Size Details

#### Performance details.

The table version of Figure 2 is provided in Table 7.

#### Relative performance gaps.

We observe that while the absolute performance gap between VLMs and CLIP
decreases when reducing the number of classes, the relative performance gap
increases. This trend is indicated by the error rates between VLMs and CLIP
shown in Figure 4.

Figure 4: Analysis of the label set size. For each image, we randomly sample 100, 20, 5, and 2 candidate classes from the entire set of classes. While the absolute performance gap between VLMs and CLIPs decreases as the number of classes is reduced, the relative performance gap increases. The X-axis represents the number of classes, and the Y-axis represents the relative error rate between LLaVA1.5-7B and CLIP-L. Model | # Classes |  |  |  |
---|---|---|---|---|---
LLaVA1.5-7B | 2 | 94.3 | 67.1 | 78.5 | 96.4
| 5 | 92.3 | 50.5 | 67.9 | 96.6
| 20 | 82.0 | 29.6 | 22.0 | 91.8
| 100 | 46.6 | 10.0 | 1.0 | 62.1
BLIP2-2.7B | 2 | 14.3 | 17.4 | 0.9 | 29.6
| 5 | 11.9 | 16.9 | 1.3 | 26.7
| 20 | 8.0 | 12.2 | 1.5 | 20.9
| 100 | 5.5 | 11.2 | 3.9 | 22.4
CLIP-L | 2 | 99.8 | 96.9 | 99.8 | 99.9
| 5 | 99.2 | 92.8 | 99.1 | 99.8
| 20 | 97.5 | 85.7 | 96.1 | 98.9
| 100 | 92.7 | 76.3 | 85.4 | 95.7
Table 7: Analysis of the label set size. This is the table version of Figure
2.

###  B.3 Inference Strategy Details

#### Experimental details.

Probabilistic inference methods require separate computations for each class,
which introduces significant computational costs. For example, on ImageNet
with 1000 classes, each example takes 1000 times longer to process than with a
direct generation pipeline. Therefore, we randomly sampled 1000 examples from
each dataset’s validation set for evaluation.

###  B.4 Information Loss Details

#### CLIP global vs. local features.

The global feature of CLIP is commonly used in traditional classification
settings, which has 1 token for each image. However, VLMs utilize local patch
features for classification, which has 576 tokens for each image using CLIP-L.
One might hypothesize that local features contain less information than global
features. In practice, VLMs use the second-to-last layer of CLIP local
features rather than the last layer. Since the last layer’s global feature is
the weighted average of the second-to-last layer’s features through self-
attention computation, we can theoretically conclude that local features have
at least the same level of information as CLIP global features.

#### Feature extraction details.

We extract features from VLM’s last layer by feeding the LLaVA default prompt
“USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT:”
into the LLaVA, or BLIP default prompt “<Image Tokens> Question: What type of
object is in this photo? Answer:” into the BLIP. We use either the last token
feature (i.e., the token “:”) or the average feature over all the tokens.

#### Linear probing details.

We train the linear layer on the training set with a batch size of 512, a
learning rate of 1e-3 using the Adam optimizer, and for 500 epochs. After
training, we report the best performance on the validation set.

#### Probing position.

In practice, we find that probing the last token or the average token results
in much better performance than probing other token positions, as shown in
Table 8. We leave this as an interesting question for future research.

Position | 0 | 100 | 200 | 300 | 400 | 500 | 600 | -3 | -2 | -1 (Last) | Average
---|---|---|---|---|---|---|---|---|---|---|---
Accuracy | 0.7 | 17.1 | 18.1 | 25.0 | 25.8 | 23.7 | 52.9 | 84.0 | 85.8 | 94.5 | 96.2
Table 8: Probing the last token or the average token results in much better
performance than probing other token positions. Experiments are done using
LLaVA1.5-7B on the Flowers102 dataset.

###  B.5 Training Objective Detail

Figure 5: Fine-tuning only the projector improves numerical stability. (Top)
Fine-tuning LLMs with LoRA often results in numerical instabilities,
manifesting as spikes in loss (purple, green, brown, orange curves). In
contrast, fine-tuning only the projector leads to a consistently steady
decrease in loss (teal curve). Despite experimenting with various
hyperparameters for ImageNet, the instability remained. (Bottom) Occasionally,
the spikes normalize with continued training. Here, we present an example
using the StanfordCars dataset (pink curve).

#### LLaVA fine-tuning details.

We convert each image and class label into the text format using the LLaVA
default template “USER: <576 Image Tokens> What type of object is in this
photo? ASSISTANT: <Class Name>.” We conduct two settings for fine-tuning
LLaVA. In the first setting, we only fine-tune the MLP projector between CLIP
and the language model (LM). The projector is trained on the training set with
a batch size of 64 and a learning rate of 2e-5 using the AdamW optimizer for
50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03. In the second
setting, we fine-tune both the MLP projector and the LM using LoRA. The
projector and LM are trained on the training set with a batch size of 64, a
learning rate of 2e-5 for the projector and 2e-4 for the LM, using the AdamW
optimizer for 50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03, a
LoRA rank of 128, and a LoRA alpha of 256. For both settings, we report the
best performance on the validation set after training. Fine-tuning on ImageNet
for 1 epoch requires 130 L40 GPU hours.

#### BLIP fine-tuning details.

We convert each image and class label into the text format using the BLIP
default template “<Image Tokens> Question: What type of object is in this
photo? Answer: <Class Name>.” We train the Q-former projector between CLIP and
the LM on the training set with a batch size of 64, a learning rate of 2e-5,
and a weight decay of 0.05 using the AdamW optimizer for 100 epochs (10 epochs
for ImageNet), with 1000 warmup steps. After training, we report the best
performance on the validation set. Fine-tuning on ImageNet for 10 epochs
requires 120 L40 GPU hours.

#### CLIP fine-tuning details.

We initialize a linear layer on top of the CLIP image encoder. The CLIP model
is frozen, and only the linear layer is trained. The linear layer is trained
on the training set with a batch size of 512, a learning rate of 1e-3, and no
weight decay using the Adam optimizer for 300 epochs (40 epochs for ImageNet).
After training, we report the best performance on the validation set.

#### Numerical instability.

When fine-tuning both the projector and the LM with LoRA for LLaVA, we
experience significant numerical instability. On ImageNet, despite adjusting
three different hyperparameters such as the learning rate and batch size, the
loss consistently peaks during training and does not recover. On the
StanfordCars dataset, numerical instability is also observed, but the loss
returns to normal during training. This can be seen in Figure 5.

###  B.6 Data Detail

| Spearman Correlation | Pearson Correlation
---|---|---
|  |  |  |  |  |  |  |
CLIP-L | -0.13 | 0.30 | 0.04 | -0.03 | -0.08 | 0.08 | 0.06 | 0.00
LLaVA1.5-7B Fine-tuned on ImageNet | -0.02 | -0.03 | 0.04 | 0.30 | -0.05 | 0.08 | 0.04 | 0.18
LLaVA1.5-7B | 0.76 | 0.40 | N/A | 0.69 | 0.35 | 0.64 | N/A | 0.27
Table 9: Correlation between class frequency and model’s accuracy on the
class. This supplements Figure 3.

#### Data processing details.

The training of LLaVA comprises two stages: pre-training and instruction
tuning. The pre-training stage utilizes noisy image captioning data, whereas
the instruction tuning stage employs meticulously curated multi-turn
conversation data. We tokenize all sentences within the data, compute the
frequency for each class, and then calculate the accuracy for each class. This
results in 1000 (frequency, accuracy) pairs for ImageNet’s 1000 classes. This
process is similarly applied to all other datasets.

#### Correlation results.

We compute the Spearman and Pearson correlations between frequency and
accuracy of zero-shot LLaVA1.5-7B. The results are presented in Table 9. For
comparison, we conducted the same analysis for CLIP-L and fine-tuned
LLaVA1.5-7B.

#### Correlation for two stages.

Given that LLaVA training involves two stages, we examine whether pre-training
or instruction-tuning data has a greater impact on classification performance.
We compute the Spearman and Pearson correlations separately for pre-training
and instruction-tuning data. Our findings indicate that the correlation is
similar for both stages. For instance, on ImageNet, the Spearman and Pearson
correlations are 0.73 and 0.34 for pre-training, and 0.76 and 0.33 for
instruction tuning. These values are close to the combined data correlations,
which are 0.76 and 0.35 for Spearman and Pearson correlations, respectively,
as reported in Table 9.

##  Appendix C Supplementary for Section 4

###  C.1 ImageWikiQA Details

#### More examples.

We provide more examples of the ImageWikiQA dataset in Table 11.

#### Data generation prompt.

We provide the prompt used to generate ImageWikiQA in Figure 6 and the prompt
used to filter ImageWikiQA in Figure 7.

⬇

Come up with five multiple-choice questions for a {classname} to examine the
knowledge of an expert.

The questions should come from the following Wikipedia articles on
{classname}:

‘‘‘

{wikipedia page}

‘‘‘

**Instruction**:

Each question should have four choices, one of which is the correct answer.

Note that the Wikipedia articles will not be accessible to the test-takers, so
please do not reference specific details from the articles in the questions.

Use "this object" rather than "{classname}" in the questions. We want to give
test-takers an image of this object, not the word itself.

Each question should have four fields: "question" (str), "choices"
(list[str]), "answer" (int, starting from 0)), and "reference" (str, original
sentences from Wikipedia articles).

Output in JSON format.

Figure 6:  Prompt used to generate ImageWikiQA using GPT4.

⬇

Answer the multiple-choice question below.

{question with ground-truth class name}

A. {choice A}

B. {choice B}

C. {choice C}

D. {choice D}

Output only one character A/B/C/D.

\-----

Answer the multiple-choice question below.

{question without ground-truth class name}

A. {choice A}

B. {choice B}

C. {choice C}

D. {choice D}

The question mentions "this object". However, we don’t know what the object
is. Try your best to guess the answer without knowing the object.

Output only one character A/B/C/D.

Figure 7:  Prompt used to filter ImageWikiQA using GPT4.

###  C.2 Results Details

#### LLaVA fine-tuning details.

For ImageNet, we convert each image and class label into the text format using
the LLaVA default template “USER: <576 Image Tokens> What type of object is in
this photo? ASSISTANT: <Class Name>.” Then, we concatenate the original LLaVA
665K instruction tuning dataset with the 1.28M ImageNet classification
dataset. Due to numerical stability concerns, we train only the projector
layer of LLaVA. The projector is trained on the combined dataset with a batch
size of 64, a learning rate of 2e-5, using the AdamW optimizer, for 1 epoch,
with a warmup ratio of 0.03. The training process takes approximately 340
hours on a single L40 GPU.

##  Appendix D Ablation on Data Types

In the main paper, we have demonstrated that data is crucial for
classification performance for visually-grounded language models (VLMs) and
proposed a simple data integration method to enhance VLMs. However, we are
curious whether the data type also plays a significant role in VLM
classification performance. For instance, to train a VLM to classify “guinea
pig”, should the model be trained with classification-focused data (e.g.,
“<image> Question: What type of object is in this photo? Answer: Guinea
pig.”), or can it be trained with any type of data, such as caption-focused
data (e.g., “<image> Question: Generate a caption for the image. Answer: A
guinea pig playing in the grass.”) or VQA-focused data (e.g., “<image>
Question: What is the native region of this guinea pig? Answer: South
America.”)?

While our initial analysis in §3.3 suggests that any type of data could work
based on the strong correlation between VLM performance and class occurrence
in LLaVA-training data, which is generally VQA-type rather than
classification-focused, we further investigate how data types affect final
performance in this section.

In previous experiments detailed in §3.2, we fine-tuned the VLM using a
classification-focused template (e.g., “<image> Question: What type of object
is in this photo? Answer: <class name>.”). Here, we fine-tune the VLM on
captioning data. This captioning data is generated by providing an image and
the ground-truth class name to GPT4 and requesting it to generate a caption
that includes the ground-truth name. Specifically, the prompt for GPT4 is:
“Generate a short caption for the image. The caption must include the ground-
truth label of the image, which is <class name>.” The generated captions are
diverse for each class. For example, for the “guinea pig” class, captions
could be “A guinea pig playing in the grass.” or “A close-up view of a guinea
pig in an indoor environment.” We fine-tune the VLM on this caption-focused
data using the same experimental settings and compare the model performance
with that fine-tuned on classification-focused data. Success is defined as
whether the generated caption for a validation image contains the ground-truth
class name.

The performances of fine-tuned LLaVA1.5-7B on three datasets are reported in
Table 10. Our findings indicate that data is the main determining factor for
VLM performance, and the data type does not matter much. From Table 10, we can
see that fine-tuning on both classification-focused data and caption-focused
data achieves a similar level of accuracy on Flowers102, StanfordCars, and
Caltech101, at least 48.6% higher than the non-fine-tuned model. This
indicates that to enable a VLM to perform classification on a specific class,
it is unnecessary to use classification-specific data. Instead, any data
related to that class can be used, provided that the data includes the class
name.

Model | Training Data Type |  |  |
---|---|---|---|---
LLaVA1.5-7B Zero-shot | - | 5.9 | 0.0 | 47.1
CLIP-L Fine-tuned | Classification Data | 98.6 | 91.5 | 97.6
LLaVA1.5-7B Fine-tuned | Classification Data | 97.6 | 90.4 | 97.5
LLaVA1.5-7B Fine-tuned | Captioning Data | 92.0 | 85.4 | 95.7
Table 10: Analysis of data types. We fine-tune the VLM on the caption-focused data generated by GPT4 using the same experimental settings and find that data is the main determining factor for VLM performance, and the data type does not matter much.  Image |  Question |  Choices |  Reference Wikipedia
---|---|---|---
|  Which position is particularly important for this object when transporting a patient in shock? |  A. Fowler’s position
B. Upright position
C. Trendelenburg position
D. Flat position |  The feet can be raised to what is called the Trendelenburg position, indicated for patients in shock.
|  Which characteristic is NOT typical for the temperament of this object? |  A. Very intelligent and loyal
B. Easily bored and highly energetic
C. Quiet and suspicious of strangers
D. Highly aggressive and loud |  Giant Schnauzers are usually a quiet breed… It has the potential to be aggressive, but Giant Schnauzers are usually reserved
|  What is the traditional color combination of this object’s coat? |  A. Black with white markings
B. Solid red
C. Red with white markings
D. White with black markings |  The breed’s coat only comes in a single colour combination of white with red markings, usually in a piebald pattern.
|  What steering mechanism is traditionally associated with this object when it possesses four wheels? |  A. A rudder
B. Turntable or fifth wheel
C. Flap control
D. A steering wheel |  A four-wheeled vehicle is also steered by the shafts or pole, which are attached to the front axle; this swivels on a ’turntable’ or ’fifth wheel’ beneath the vehicle.
|  What is a notable behavior of this object during mating? |  A. Males compete in physical combat
B. Releases a sound to attract mates
C. The female releases a pheromone to reduce aggression in males
D. Both sexes change color to a brighter hue |  The female releases a pheromone which causes the males to become less aggressive and to begin courtship.
|  Which of the following is known as the world’s largest this object? |  A. The LEGO Store
B. Hamleys
C. Toys "R" Us
D. FAO Schwarz |  Notable Examples. - Hamleys, the world’s largest toy shop
|  In which habitats is this object predominantly found? |  A. Low-altitude woodlands and forest edges
B. Polar and subpolar zones
C. Temperate forests and grasslands
D. Desert and arid regions |  They are principally birds of low-altitude woodlands and forests, and particularly of forest edge and canopy.
|  In which type of climates does this object predominantly thrive? |  A. Mediterranean
B. Subarctic
C. Tropical rainforest
D. Desert |  The plant tolerates seasonal drought, and the Middle Eastern and Mediterranean climates are especially suitable to it.
|  Which of the following features was introduced by Philips in the development of this object? |  A. Eight-to-fourteen modulation (EFM)
B. Diagonal error correction
C. Anti-shock buffering
D. Cross-interleaved Reed-Solomon Coding (CIRC) |  Philips also contributed eight-to-fourteen modulation (EFM), which offers a certain resilience to defects such as scratches and fingerprints.
|  What is a common contemporary use for this object in urban settings? |  A. Display in museums
B. Storing chilled goods for retail
C. Transporting large amounts of heavy equipment
D. Delivering business mail across the city |  Messenger bag: one long strap worn across the body, inspired by bags worn by urban messengers to deliver business mail, a modern silhouette
Table 11: Examples of ImageWikiQA. Each example has an image, question, four
choices (correct choice highlighted in orange), and reference Wikipedia
sentences.