# What Makes for Good Visual Tokenizers for
Large Language Models?

Guangzhi Wang1,2 Yixiao Ge2,3 Xiaohan Ding3 Mohan Kankanhalli1 Ying Shan2,3
1National University of Singapore 2ARC Lab This work is done during Guangzhi’s
internship at ARC Lab, Tencent PCGProject lead. Tencent PCG 3Tencent AI Lab

###### Abstract

We empirically investigate proper pre-training methods to build good visual
tokenizers, making Large Language Models (LLMs) powerful Multimodal Large
Language Models (MLLMs). In our benchmark, which is curated to evaluate MLLM’s
visual semantic understanding and fine-grained perception capabilities, we
discussed different visual tokenizers pre-trained with dominant methods (i.e.,
DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models
capture more semantics than self-supervised models, but the gap is narrowed by
scaling up the pre-training dataset. ii) Self-supervised models are better at
fine-grained perception, where patch-level supervision is particularly
effective. iii) Tuning the visual tokenizer leads to the loss of semantics
obtained from large-scale pretraining, which is unfavorable with relatively
small-scale instruction-tuning dataset. Given the findings, we reviewed
methods that attempted to unify semantics and fine-grained visual
understanding, e.g., patch-level feature distillation with semantically-rich
targets. We obtain an intriguing insight: mask-based strategies that were once
all the rage may not be applicable for obtaining good visual tokenizers. Based
on this critical observation, we obtain a new MLLM equipped with a tailored
Good Visual Tokenizer – GVT, which exhibits strong visual comprehension
capability at multiple scales. In particular, without introducing extra
parameters and task-specific fine-tuning, GVT achieves superior performance on
visual question answering, image captioning, and other fine-grained visual
understanding tasks such as object counting and multi-class identification.
Project released at: https://github.com/TencentARC/GVT

##  1 Introduction

Large Language Models (LLMs) [1; 2; 3; 4] have demonstrated remarkable
performance for various downstream tasks without task-specific fine-tuning.
Recently, based on the powerful LLMs, there has been a surge of research [5;
6; 7; 8; 9; 10; 11; 12] that successfully adapt LLMs to vision-language tasks,
resulting in powerful Multimodal LLMs (MLLMs), e.g., BLIP-2 [5]. When properly
fed with visual data, they are shown to be capable of understanding the visual
world and responding to instructions accordingly. Such vision-language
understanding capability makes LLM a universal interface for multimodal tasks,
contributing towards a tentative yet promising direction towards Artificial
General Intelligence (AGI) [13; 14].

Figure 1:  Different tasks require visual understanding of different
perspectives. Mainstream vision-language tasks, e.g., (a) VQA and (b) Image
Captioning mainly focus on semantic understanding of the image. In this work,
we also study two fine-grained visual understanding tasks: (c) Object Counting
(OC) and (d) Multi-Class Identification (MCI).

Within this framework, images are projected to the linguistic space for the
LLMs to understand, where the common practice employs an image-text pre-
trained visual tokenizer111In this work, we study visual tokenizers which map
images into a continuous latent space., i.e., CLIP [15]. However, even though
CLIP has shown strong capacity for image representations, to the best of our
knowledge, it is yet to be explored whether CLIP is the optimal visual
tokenizer for MLLMs. The absence of such investigation calls for a
comprehensive comparison of existing visual tokenizers under MLLMs’ framework.
However, recent MLLMs have mostly investigated their performance in terms of
generation quality [7; 8] or on a small set of questions [9], leaving a
comprehensive quantitative evaluation untouched.

To this end, we curated a new benchmark to study what makes for a Good Visual
Tokenizer (GVTBench). It is especially designed to evaluate an MLLM’s visual
understanding capability from two important perspectives: semantic
understanding and fine-grained visual perception capabilities. As shown in
Figure 1, the former is evaluated on Visual Question Answering (VQA) and image
captioning. While the latter is tested on two new tasks: Object Counting (OC)
and Multi-Class Identification (MCI), which requires in-depth understanding of
fine-grained visual information. Based on this benchmark, we comprehensively
evaluated existing visual tokenizers with same architecture but different
pretraining methods, including fully supervised (DeiT [16]), text-guided
weakly supervised (CLIP [17]) and self-supervised (MAE [18], DINO [19], DINOv2
[20]) models (Section 2). Our main observations are i) fully supervised and
text-guided weakly supervised visual tokenizers demonstrate better semantic
representation capacity than their self-supervised counterparts, but the gap
is narrowed by scaling up the pretraining dataset (i.e., CLIP vs. DINOv2). ii)
Self-supervised visual tokenizers show better fine-grained visual perception
capacity, where patch-level supervision leads to superior region-level
understanding. iii) On instruction tuning datasets which are often smaller
than visual tokenizer pretraining dataset [8; 7], jointly tuning the visual
tokenizer leads to noticeable semantic loss (i.e., frozen CLIP performs much
better than tunable CLIP on semantic understanding tasks).

Given the fact that none of the previous visual tokenizers exhibit both good
semantic and fine-grained visual perceptual capabilities, we reviewed existing
methods that integrate semantic and region supervision and question whether
they bring the best of the two worlds into a visual tokenizer. Existing
methods can be mainly divided into two categories. Methods in the first group
[21; 22] enhance a pretrained CLIP with region-level supervision, which comes
from a pretrained Region Proposal Network (RPN) or bounding box annotations.
However, we found that this leads to the loss of original semantics, which can
not be justified by the limited improvements on fine-grained visual perception
capabilties. The other group of methods [23; 24] utilize patch features from a
pretrained CLIP as region supervision to train a new model, intending to
enhance its fine-grained visual perceptual capability while maintaining the
rich semantics. Specifically, [23; 25] uses CLIP features to supervise the
training of Masked-Image-Modeling (MIM), while Feature Distillation [24]
directly distills the CLIP feature into a new model without patch masking.
Nonetheless, the introduction of [MASK] token in MIM leads to train-test
mismatch, requiring the visual tokenizer to be jointly optimized in the
instruction-tuning process, which again leads to semantic loss with the small-
scale instruction tuning dataset. As such, we argue that the mask-based
strategies that were once all the rage may not be applicable for obtaining
good visual tokenizers under MLLM’s framework.

Based on these insights, we seek a new visual tokenizer with both strong
semantic understanding and fine-grained visual perception capabilities via
Feature Distillation [24]. Specifically, given a pretrained CLIP with rich
semantics, we distill it into a new model by using the patch features as
supervision, without patch masking. In this way, the rich semantics from
large-scale image-text contrastive pretraining is preserved, and the fine-
grained visual perceptual capability is greatly enhanced with patch
supervision. With our new visual tokenizer and the language model Vicuna [26],
we obtain a new MLLM with Good Visual Tokenizer (GVT). Benefiting from the
versatile visual tokenizer, GVT is able to perform well vision language tasks
that require visual understanding at multiple levels. Without introducing
extra parameters, we achieve superior performance on semantic understanding
tasks, i.e., VQA and image captioning, as well as fine-grained visual
understanding tasks: instance counting and multi-class identification.

To summarize, our contributions are as follows:

  * •

To effectively evaluate MLLM’s visual understanding capacity at different
levels, we curate a new benchmark (GVTBench) which includes both semantic
understanding tasks (VQA and image captioning) as well as fine-grained visual
understanding tasks (Object Counting and Multi-Class Identification). Based on
GVTBench, we perform extensive experiments to study what makes for a good
visual tokenizer for MLLMs and make three main observations.

  * •

We reviewed methods that combine CLIP with fine-grained supervision to see if
they can achieve the best of both worlds in terms of visual semantics and
fine-grained understanding. We found that the SOTA pre-trained models (i.e.,
EVA) are inapplicable due to the train-test mismatch caused by MIM. Such mask-
based visual tokenizers rely on further tuning with instructions, which leads
to the loss of pre-trained rich semantics.

  * •

Based on the insights, we tailor a new visual tokenizer by distilling the
patch-level semantics of a pre-trained CLIP without masking. With our visual
tokenizer and Vicuna [26], we arrive at a superior MLLM (GVT) with strong
visual understanding capability, achieving state-of-the-art performance on our
curated benchmark.

##  2 GVTBench for Empirical Study

To comprehensively study what makes for good visual tokenizers for MLLMs, we
conduct a series of experiments to study the property of various visual
tokenizers with same architecture but different pretraining methods. In this
work, we mainly investigate MLLMs’ visual understanding capability from two
important perspectives: semantic understanding and fine-grained visual
perception.

###  2.1 Experimental Setup

GVTBench. A comprehensive evaluation requires a benchmark that suitably
quantify MLLM’s visual understanding capability. Nonetheless, existing vision-
language tasks mainly focus on semantic understanding [27; 28], leaving a
special focus on fine-grained visual perception untouched. To this end, we
curated a new benchmark – GVTBench. It evaluates the semantic understanding
capability of an MLLM on VQA [28] and Image Captioning (IC) [29]. We report
accuracy for the former and CIDEr [30] and SPICE [31] for the latter. For
fine-grained visual perception capability evaluation, we specially design two
new tasks for MLLMs:

  * •

Object Counting (OC). We ask the model to count the number of a certain object
appearing in the image with the prompt “Question: How many {obj} are there in
the image? Answer:”. We regard it as a classification task and report a
model’s prediction accuracy.

  * •

Multi-Class Identification (MCI). We ask the model if a certain object exists
in the image with the prompt “Question: Does {obj} exist in the image?
Answer:”. The model is expected to answer “ Yes/No”, resulting in a binary
classification problem. We report accuracy for this task.

Notably, in the VQAv2 [28] benchmark, there are also questions related to
numbers. Nevertheless, these questions are often coupled with high-level
semantics, making it unsuitable to strictly evaluate fine-grained visual
understanding capabilities. In contrast, our OC and MCI tasks attend to
individual objects, which is decoupled from high-level semantics and thus a
more appropriate test bed for fine-grained visual understanding evaluation.

Experimental Setting. We use different visual tokenizers to encode an image
into a set of visual tokens. Then, we follow Flamingo [32] to use the
Perceiver Resampler [33] to reduce the number of visual tokens to a fixed
length, which are fed into LLM (i.e., Vicuna). The models are trained on a
instruction dataset which contains about 5M image-text pairs. In the training
process, the language model is always frozen, while the visual tokenizer can
be frozen or jointly optimized. For more implementation details, please refer
to the appendix.

Table 1: Comparison of visual tokenizers of ViT-B with different pretraining strategies. The best result is bold while the second best is underlined. Joint | Supervised | Visual | # Pretraining | VQA | Captioning | OC | MCI | Avg
---|---|---|---|---|---|---|---|---
Tuning | Tokenizer | Images | Acc | CIDEr | SPICE | Acc | Acc
$\times$ | Fully |  DeiT [16] | 1.28 M | 48.3 | 65.8 | 15.9 | 37.5 | 83.6 | 58.8
Self |  DINO [19] | 1.28 M | 50.1 | 45.0 | 13.5 | 46.5 | 80.8 | 55.6
MAE [18] | 1.28 M | 48.4 | 37.3 | 11.8 | 47.5 | 82.7 | 53.4
DINOv2 [20] | 142 M | 51.3 | 67.9 | 16.1 | 47.0 | 86.0 | 63.1
Weakly |  CLIP [17] | 400 M | 52.2 | 69.3 | 16.6 | 42.5 | 86.0 | 62.5
✓ | Fully |  DeiT [16] | 1.28 M | 50.7 | 38.4 | 10.0 | 41.0 | 86.9 | 54.3
Self |  DINO [19] | 1.28 M | 47.3 | 54.1 | 14.5 | 44.5 | 86.6 | 58.1
MAE [18] | 1.28 M | 48.9 | 48.0 | 14.2 | 47.5 | 88.7 | 58.2
DINOv2 [20] | 142 M | 50.5 | 49.6 | 13.0 | 43.5 | 84.1 | 56.9
Weakly |  CLIP [17] | 400 M | 47.7 | 64.2 | 15.4 | 45.5 | 88.0 | 61.4

###  2.2 Comparing Visual Tokenizers

On GVTBench, we evaluate visual tokenizers with the same architecture (ViT-B
[34]) but different pretraining strategies, including fully-supervised (DeiT
[16]), self-supervised (DINO [19], DINOv2 [20], MAE [18]) and text-guided
weakly supervised (CLIP [17]) pretraining. Based on the results in Table 1, we
arrive at the following conclusions.

Fully/weakly supervised models capture more semantics than self-supervised
ones, but the gap is narrowed by scaling up the pre-training dataset. With
tokenizers pretrained on relative small-scale dataset (i.e., ImageNet-1k [35]
with 1.28M images), DeiT demonstrates better image captioning performance
(65.8 CIDEr) than self-supervised models DINO (45.0) and MAE (37.3), without
jointly tuning the visual tokenizer. However, with 142M images for
pretraining, the self-supervised model – DINOv2 outperforms the supervised
DeiT on image captioning (67.9) and VQA (51.3), and is only inferior to CLIP
which is pretrained with weak supervision from a large-scale dataset with 400M
image-text pairs. This indicates that supervision is beneficial for semantic
representation capability, but this can also emerge from large-scale
pretraining with self-supervision.

Self-supervised models are better at fine-grained perception, where patch-
level supervision is particularly effective. On fine-grained visual
understanding tasks, i.e., OC and MCI, self-supervised models demonstrate
consistently better performance than those with supervision. When they are
jointly tuned on the instruction dataset, their OC and MCI performance are
mostly boosted, indicating their fine-grained visual perception capability
gets improved. Among all the self-supervised models, MAE achieves the best
performance, indicating the patch-based supervision is particularly effective
for improving fine-grained visual understanding.

Tuning semantic-rich visual tokenizer leads to semantic loss on small-scale
instruction tuning dataset. When the tokenizer is jointly optimized on the
instruction tuning dataset, the rich semantics obtained from large-scale
pretraining in CLIP and DINOv2 have noticeably dropped (e.g., CLIP VQA 52.2
$\rightarrow 47.7$ and DINOv2 captioning 67.9 $\rightarrow$ 49.6). We
conjecture this is due to the relatively small scale of our instruction
dataset ($\sim$5M $\ll$ 142M). As such, for modern MLLMs that are often tuned
on small-scale and high-quality instruction datasets [7; 8], jointly tuning
the visual tokenizer may not be a good option.

##  3 Unifying Semantic and Fine-grained Visual Understanding

###  3.1 CLIP with Region-based Training

The generalist MLLMs call for a versatile visual tokenizer that could properly
represent an image’s content at multiple levels. However, based on the results
in Table 1, none of existing pretraining methods leads to a good visual
tokenizer that excels at both semantic and fine-grained visual perception
capabilities. This motivates us to explore whether the best of the two worlds
can be achieved by any other method.

Table 2: Comparison of Visual region supervised methods and CLIP. Joint | Visual | VQA | Captioning | OC | MCI | Avg
---|---|---|---|---|---|---
Tuning | Tokenizer | Acc | CIDEr | SPICE | Acc | Acc
$\times$ |  CLIP [17] | 52.2 | 69.3 | 16.6 | 42.5 | 86.0 | 62.5
$\times$ |  RegionCLIP [21] | 48.7 | 28.5 | 10.3 | 41.0 | 86.0 | 51.5
$\times$ |  Owl-ViT [22] | 44.0 | 32.5 | 8.5 | 43.0 | 80.8 | 50.1
$\checkmark$ |  CLIP [17] | 47.7 | 64.2 | 15.4 | 45.5 | 88.0 | 61.4
$\checkmark$ |  RegionCLIP [21] | 49.7 | 65.5 | 14.1 | 47.5 | 86.4 | 62.3
$\checkmark$ |  Owl-ViT [22] | 50.8 | 61.2 | 14.0 | 38.5 | 87.1 | 59.4

Fine-tuning CLIP with region supervision. One stream of work [21; 22]
attempted to improve region representation capability of a pretrained CLIP by
fine-tuning it with region supervision, which has demonstrated improved
performance for open-vocabulary object detection. This motivates us to study
if this also enhances CLIP as a visual tokenizer. We mainly investigated
RegionCLIP [21] and Owl-ViT [22]. The former finetunes a CLIP with region-
level supervision from bounding boxes generated by a pretrained RPN, while the
latter utilizes the region annotation from an object detection dataset. We
compared these methods with CLIP, and show the results in Table 2. It can be
observed that, without joint tuning the visual tokenizer, both RegionCLIP and
Owl-ViT show severe performance drop on image captioning and VQA, indicating
the rich semantics in the original CLIP is lost during their region fine-
tuning process. On the other hand, when the visual tokenizers are jointly
tuned on the instruction-tuning dataset, their fine-grained representation
capability improves by a margin (on OC and MCI performance), but this can not
justify the loss of semantic representation capability, resulting in inferior
overall performance compared to the original CLIP.

Table 3: Comparison of different strategies of utilizing CLIP features with ViT-B architecture. Method | Joint | Patch | VQA | Captioning | OC | MCI | Avg
---|---|---|---|---|---|---|---
Tuning | Masking | Acc | CIDEr | SPICE | Acc | Acc
CLIP [17] | $\times$ | - | 52.2 | 69.3 | 16.6 | 42.5 | 86.0 | 62.5
FD [24] | $\times$ | $\times$ | 49.4 | 72.1 | 15.8 | 46.5 | 86.7 | 63.7
EVA [23] | $\times$ | $\checkmark$ | 42.9 | 27.0 | 10.0 | 46.9 | 70.5 | 46.8
CLIP [17] | $\checkmark$ | - | 47.7 | 64.2 | 15.4 | 45.5 | 88.0 | 61.4
FD [24] | $\checkmark$ | $\times$ | 49.3 | 53.3 | 12.7 | 40.5 | 85.8 | 57.2
EVA [23] | $\checkmark$ | $\checkmark$ | 51.4 | 61.6 | 12.3 | 45.9 | 87.1 | 61.5
Figure 2:  Framework of our GVT. We first distill the features of a pretrained
CLIP via smoothed $\mathcal{L}_{1}$ loss. Then, we use it to encode images
into a set of tokens, which are fed into the Perceiver Resampler [33] as soft
prompts. Together with language instructions, these prompts are fed into LLM
to generate responses. Only the Perceiver Resampler is optimized in this
process.

Semantic Feature as Region Supervision. Another stream of work utilized CLIP’s
patch feature as region-level supervision for pretraining, aiming to obtain a
model with both strong semantics and better region representations.
Specifically, EVA [23] and MVP [23] use CLIP’s patch feature as regression
target for Masked Image Modeling (MIM) pretraining, while FD [24] does not
employ the masking strategy and directly distills CLIP’s patch feature into a
new model. We compared these methods in Table 3. Without jointly tuning the
visual tokenizer, FD results in performance improvement on both semantic and
fine-grained visual understanding upon CLIP. However, when patch masking
strategy is adopted, the performance of EVA significantly drops. This can be
attributed to the introduction of the [MASK] token for MIM, which is only used
for pretraining the visual tokenizer but discarded afterwards. In this way,
the train-test mismatch arises without tuning the visual tokenizer, leading to
unsatisfactory performance for downstream tasks. On the other hand, when the
visual tokenizer is jointly optimized with the instruction data, they are
inferior to the original CLIP on VQA and image captioning, indicating semantic
loss occurs.

Given the fact that modern MLLMs are often trained on high-quality and small-
scale instruction datasets [7; 8], our observation suggests that visual
tokenizer should be frozen to maintain the powerful semantic representation
capability from large-scale pretraining. Nonetheless, for visual tokenizers
pretrained with MIM, the introduction of the [MASK] token inevitably leads to
train-test mismatch, necessitating it to be jointly tuned on the instruction
data. This contradiction indicates that mask-based pretraining may not lead to
a good visual tokenizer under MLLM’s framework.

###  3.2 MLLM with Good Visual Tokenizer

In this work, we tune a new visual tokenizer which unifies the advantage of
semantic representation, fine-grained visual perception and semantic
maintenance capabilities. Based on the insights above, we achieve this
objective by utilizing a visual tokenizer pretrained on large-scale datasets,
and properly integrate it with patch-level supervision. Furthermore, we does
not use any mask-based strategy, so as the rich semantics could be preserved
by freezing it in the instruction tuning process. Specifically, we take the
powerful EVA-CLIP [36] based on ViT-L as the teacher model, and randomly
initialize another model with identical architecture as student. The patch
features from the teacher model is normalized by a whitening operation, and is
taken as regression target for the student model. Afterwards, the visual
tokenizer can be used for MLLMs and kept frozen during instruction tuning.

Based on the tuned visual tokenizer, we construct a new MLLM with Good Visual
Tokenizer (GVT). The framework of GVT is shown in Figure 2. Following [32], we
also random initialize a Receiver Resampler [33] with 32 learnable queries to
attend to the features from the visual tokenizer. Then, the features output
from Perceiver Resampler are taken as soft prompts, and are fed into the LLM
together with the language prompts. In this work, we choose the instruction-
tuned Vicuna-7B [26] as the LLM. The whole model is trained by the language
modeling loss, and only the Perceiver Resampler is optimized in this process.

##  4 Experiments

###  4.1 Experimental Setup

We train our model on a joint dataset of image-text pairs, including CC3M
[37], SBU [38], Visual Genome [39] and MS-COCO [29]. We formulate these
datasets as image captioning task, and use "what does the image describe?" as
prompt during training. Besides, we also use two object detection datasets –
Object365 [40] and OpenImagesV6 [41] to design a set of object-centric tasks
following [42]. The LLaVA-150k [8] dataset is also utilized for joint
training. This results in a total of 15M image-text pairs. The images are
resized to 224 $\times$ 224, and we adopt random resized crop and horizontal
flipping for data augmentation during training. The model is trained for 50k
steps with 2k steps for linear warmup. We use AdamW [43] optimizer with a
learning rate of 1e-4 and batch size 1024. The training process takes about 2
days on 32 Tesla V100 GPUs. For more implementation details, please refer to
our appendix.

###  4.2 Comparison with State-of-the-art Methods

Without task-specific fine-tuning, we evaluate GVT on our GVTBench, which
includes VQA [28], Image Captioning [29], Object Counting (OC) and Multi-Class
Identification (MCI). Besides evaluating OC and MCI on MS-COCO validation set,
we also evaluate these two tasks based on the validation set of the VCR
dataset [44]. We compared our method with recent MLLMs, including Flamingo
[32], BLIP-2 [5], KosMos-1 [10], LLaVa [8], miniGPT4 [7]. We evaluate open-
sourced models under our GVTBench and use reported results for others. The
results are shown in Table 4.

On these tasks, our GVT achieves the best overall performance across
competitors. Specifically, on tasks requiring fine-grained visual perception,
i.e., OC and MCI on both COCO and VCR datasets, GVT surpasses models with
larger visual tokenizer and more curated data. This indicates our visual
tokenizer can better capture the fine-grained visual information, providing
representations with better details. For semantic understanding tasks, GVT
achieves the second best with an accuracy of 60.4 on VQA. This result is only
inferior to BLIP-2, which utilized a much larger instruction dataset with
high-quality image captions filtered by [45]. On image captioning task, our
GVT achieves the highest SPICE score and second best CIDEr, showing it also
has strong semantic understanding capability.

Table 4: Comparison with State-of-the-arts. The best results are bold and the
second best are underlined.

Model | #Vis. Tok. | VQA | COCO-Caption | COCO-OC | COCO-MCI | VCR-OC | VCR-MCI | Avg
---|---|---|---|---|---|---|---|---
Params | Acc | CIDEr | SPICE | Acc | Acc | Acc | Acc
Flamingo-9B [6] | 438 M | 51.8 | 79.4 | - | - | - | - | - | -
Kosmos-1 [10] | 307 M | 51.0 | 84.7 | 16.8 | - | - | - | - | -
LLaVa [8] | 307 M | 39.0 | 48.3 | 15.0 | 22.2 | 52.0 | 24.6 | 66.9 | 44.7
miniGPT4 [7] | 1.0 B | 58.2 | 80.6 | 19.5 | 21.5 | 76.8 | 25.1 | 70.1 | 55.4
BLIP-2 [5] | 1.0 B | 62.4 | 93.3 | 17.3 | 48.0 | 81.9 | 20.2 | 68.9 | 62.5
GVT (Ours) | 307 M | 60.4 | 89.9 | 19.6 | 56.2 | 89.3 | 40.3 | 78.9 | 69.2

###  4.3 Ablation Study

We adopt the training protocol in Section 2 to study the design of our GVT.

Choice of Distillation Target. According to the results in Table 1, we observe
that DINOv2, which is pretrained with self-supervision on a dataset with 142M
images also demonstrates good overall performance. To find the best target for
feature distillation, we compared it with the CLIP model from [36], both in
ViT-L architecture. The results are shown in Table 5. It can be seen that CLIP
has demonstrated better overall performance, which can be attributed to their
large-scale pretraining dataset and advanced training strategies [36].

Number of Latent Queries. We study the number of latent queries in the
Perceiver Resampler. The results are shown in Table 6. It can be observed that
the overall performance generally increases with the number of latent queries,
where 32 query results in a satisfactory performance. Besides, increasing the
number of query to 64 leads to limited improvements.

Table 5: Comparison of visual tokenizers under ViT-L architecture. Visual | VQA | Captioning | COCO-OC | COCO-MCI | Avg
---|---|---|---|---|---
Tokenizer | Acc | CIDEr | SPICE | Acc | Acc
DINO-v2-Large [20] | 53.9 | 69.9 | 15.0 | 45.5 | 83.6 | 63.2
CLIP-Large [36] | 55.5 | 71.9 | 16.5 | 45.2 | 83.5 | 64.0
Table 6: Comparison of the number of latent queries in the Perceiver Resampler. #Latent | VQA | Captioning | COCO-OC | COCO-MCI | Avg
---|---|---|---|---|---
Query | Acc | CIDEr | SPICE | Acc | Acc
8 | 53.4 | 60.0 | 15.4 | 50.0 | 78.0 | 60.3
16 | 55.0 | 61.7 | 15.8 | 51.1 | 83.5 | 62.8
32 | 55.5 | 71.9 | 16.5 | 45.2 | 83.5 | 64.0
64 | 54.0 | 71.1 | 16.4 | 47.0 | 84.2 | 64.1

###  4.4 Qualitative Results

Figure 3: Qualitative Comparison on OC and MCI. Our method shows better
performance on recognizing detailed clues in the image.

We show some qualitative comparison on OC and MCI between our GVT and BLIP-2
in Figure 3. It can be observed that our method demonstrate better fine-
grained visual understanding capabilities than the baseline method. Take the
first example in OC as an example, our method not only recognize the 3 people
in the foreground, but also takes the fourth person who is far away from the
camera into consideration. Besides, GVT also successfully recognize non-
salient or small-sized objects in the image, such as the mouse, bicycle and
broccoli in the three examples in MCI.

##  5 Related Work

###  5.1 Multimodal Large Language Models

LLMs have demonstrated strong capabilities for various downstream tasks
without task-specific fine-tuning. Based on this, recent work has utilized it
to accomplish vision and language tasks, enabling powerful Multimodal Large
Language Models. The common practice uses a visual tokenizer to encode the
image, followed by potential bridges such as MLP [8] or Perceiver Resampler
[33] to encode them into soft prompts. For example, Flamingo [6] adopts a
contrastive pre-trained visual tokenizer, followed by a Perceiver Resampler
[33] to aggregate the image tokens into fixed length. These tokens are fed
into a frozen language model with the help of gated attention attached to
transformer blocks. BLIP-2 [5] tokenizes the image with a pretrained CLIP [21;
36], which is later input into the language model with the bridge of an
attention-based Q-former. Instead of freezing the language model, Kosmos-1
[10] freezes the visual tokenizer while trains the language model from scratch
with large-scale text and image-text data.

Recently, with the open source of Large Language Models [2; 26; 3; 46], a lot
of large multimodal models are constructed based on them. Mini-GPT4 [7] is
built on the instruction-tuned Vicuna [26] and the visual encoder from BLIP-2
[5], with only a linear layer trained to bridge the two modules. This simple
design results in a powerful multi-modal chatbot, with noticeable vision-
language understanding capability. LLaVa [8] adopts CLIP as visual tokenizer,
and trains the projector with a curated dataset with balanced concepts. The
model then can be finetuned for downstream tasks, e.g., ScienceQA [47]. Apart
from using frozen visual tokenizer, mPLUG-OWL [9] tunes the Perceiver
Resampler with large-scale image-text data in the first stage, followed by the
finetuning of language model with LoRA [48] in the second stage. Although
these generalist models have demonstrated impressive capability on multimodal
tasks, we find that they mostly focus on the semantic understanding of the
image, ignoring more fine-grained visual perception. To tackle this
incapability, we tune a new visual tokenizer with better fine-grained visual
perception capabilities to further advance MLLMs as generalists.

###  5.2 Visual Tokenizer Pretraining

Visual encoders have been shown to benefit from large-scale pretraining for
downstream tasks. The most common approach first pretrains the model on a
large dataset with annotations, e.g., ImageNet [35], and finetunes it for
downstream tasks such as semantic segmentation [49] and object detection [29].
Recently, self-supervised pre-training have also shown to improve model’s
representation capability. The typical contrastive-based methods [19; 50; 51]
trains the model by aligning views from the same image. Inspired by the idea
of mask-language-modeling for pretraining language models [52], masked-image-
modeling has also evolved for visual encoder pretraining. These methods mask a
proportion of image patches before feeding them into the model, and ask the
model to recover the masked patches. Some methods [53] discretize the masked
patches via a pretrained tokenizer [54], and ask the model to find the ID of
the masked patch during pretraining. Besides, the momentum update of a model
itself can also be used as an effective tokenizer [55; 20]. Recently, auto-
encoder based [18] methods ask the model to directly generate the masked patch
in the continuous space. Another stream of visual encoders is pretrained on
massive image-text pairs via contrastive learning. The most typical model CLIP
[17] has been shown to be capable of various downstream tasks in zero-shot
manner. It has also evolved with more training data [15] and better
optimization strategy [36].

##  6 Discussions

###  6.1 Potential Societal Impacts

Potential Positive Impacts. In this work, we systematically investigated
various visual pretraining methods under MLLM’s framework. Our findings may
further motivate researchers in the community to design new visual pretraining
algorithms.

Potential Negative Impacts. The training process of large models often
requires huge computation resources, which consumes a lot of energy and can
exacerbate the emission of carbon dioxide. Furthermore, the training datasets
may contain harmful contents, leading to biased prediction or harmful
generation.

###  6.2 Limitations

In this work, our investigations are mostly based on released checkpoints,
aiming to provide a guideline for researchers to select visual tokenizer
accordingly. Given that these models can be pretrained with different dataset
and protocol, a more in-depth study could be performed by fully aligning their
training procedure.

##  7 Conclusion and Future Work

We comprehensively studied various visual tokenizers through the lens of MLLM.
Our investigation reveals that i) fully/weakly supervised models perform
better than self-supervised ones on semantic representation, but this gap can
be narrowed by scaling up pretraining dataset. ii) Self-supervised models are
better at fine-grained visual perception, where patch-level supervision is
particularly effective. iii) jointly tuning the visual tokenizer on the small-
scale instruction dataset leads to the loss of rich semantics from large-scale
pretraining. Based on these findings, we hope to find a visual tokenizer that
excels at both semantic understanding and fine-grained visual perception. We
reviewed existing methods and find that directly fine-tuning CLIP with region-
supervision does not lead to a versatile visual tokenizer. Besides, the mask
strategy for pretraining is not suitable due to the train-test mismatch. Based
on the insights above, we tune a new visual tokenizer, which distills CLIP
patch feature into a new model without masking. Equipped with our visual
tokenizer, Vicuna can better understand images at multiple levels, results in
superior performance on vision-language tasks including VQA, Image Captioning,
Object Counting and Multi-Class Identification. For future work, we would like
to explore more versatile visual tokenizer that is capable of more challenging
visual understanding tasks such as open-vocabulary object detection.

## References

  * [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, 2020.
  * [2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  preprint, 2021.
  * [4] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  NeurIPS, 2022.
  * [5] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  ICML, 2023.
  * [6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  NeurIPS, 2022.
  * [7] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.
  * [8] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [9] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178, 2023.
  * [10] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al.  Language is not all you need: Aligning perception with language models.  arXiv preprint arXiv:2302.14045, 2023.
  * [11] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  arXiv preprint arXiv:2303.11381, 2023.
  * [12] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * [13] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  arXiv preprint arXiv:2303.12712, 2023.
  * [14] OpenAI.  Gpt-4 technical report, 2023.
  * [15] mlfoundations.  Openclip.  https://github.com/mlfoundations/open_clip, 2023.
  * [16] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, 2021.
  * [17] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, 2022.
  * [19] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [20] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * [21] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al.  Regionclip: Region-based language-image pretraining.  In CVPR, 2022.
  * [22] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, and Xiaohua Zhai.  Simple open-vocabulary object detection with vision transformers.  In ECCV, 2022.
  * [23] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  CVPR, 2023.
  * [24] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo.  Contrastive learning rivals masked image modeling in fine-tuning via feature distillation.  Tech Report, 2022.
  * [25] Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian.  Mvp: Multimodality-guided visual pre-training.  In ECCV, 2022.
  * [26] FastChat.  Vicuna.  https://github.com/lm-sys/FastChat, 2023.
  * [27] Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth.  Every picture tells a story: Generating sentences from images.  In ECCV, 2010.
  * [28] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In CVPR, 2017.
  * [29] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [30] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In CVPR, 2015.
  * [31] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  SPICE: semantic propositional image caption evaluation.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016.
  * [32] ml_foundations.  Openflamingo.  https://github.com/mlfoundations/open_flamingo, 2023.
  * [33] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In ICML, 2021.
  * [34] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 2015.
  * [36] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  arXiv preprint arXiv:2303.15389, 2023.
  * [37] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [38] Tomás F Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras.  Large-scale training of shadow detectors with noisily-annotated shadow examples.  In ECCV, 2016.
  * [39] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 2017.
  * [40] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In ICCV, 2019.
  * [41] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  IJCV, 2020.
  * [42] AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova.  Pre-training image-language transformers for open-vocabulary tasks.  arXiv preprint arXiv:2209.04372, 2022.
  * [43] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [44] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In CVPR, 2019.
  * [45] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [46] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [47] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  NeurIPS, 2022.
  * [48] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In ICLR, 2022.
  * [49] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  IJCV, 2019.
  * [50] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [51] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In CVPR, 2021.
  * [52] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT, 2019.
  * [53] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  In ICLR, 2021.
  * [54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In ICML, 2021.
  * [55] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  Image bert pre-training with online tokenizer.  In ICLR, 2022.

## A. Implementation Detail

### A.1 Implementation Detail for Empirical Studies

For the experiments in empirical studies, we use a combination of 1) image
captioning datasets: MS-COCO [29], SBU [38], CC-3M [37] and Visual Genome [39]
and 2) two object detection datasets, including Object365 [40] and
OpenImagesV6 [41]. For image captioning data, we take the question "what does
the image describe?" as input prompt and ask the model to generate the
descriptions. For object detection datasets, we use a total of 6 tasks to
fully utilize the rich annotations. Please refer to Section D in this appendix
for more details. The training dataset is uniformly sampled during training.
We optimize the model with a learning rate of 1e-4 and a batch size 1024. The
whole model is optimized by the AdamW [43] optimizer and we set $\beta_{1}$ to
0.9 and $\beta_{2}$ to 0.98. We train the model for 10k steps, while the
learning rate is linearly warmuped from 0 in the first 1k steps, and is cosine
decayed to 0 afterwards. We optimize all models using float16.

### A.2 Implementation Detail for GVT

The implementation detail of our GVT is similar to that in the empirical
studies, except that we use more data and more training steps. Besides the
image captioning and object detection dataset, we also used LLaVa-150k dataset
[8], which is generated by external powerful LLM. We trained the model for 50k
steps, with 2k steps for linear warmup. Then, we use cosine decay to decrease
the learning rate to 0.

### A.3 Evaluation Details.

VQA. Modern Language Models mainly generate one or multiple sentences, making
it infeasible to directly evaluate the MLLMs in the standard evaluation
protocol which requires the prediction and ground truth to be exactly matched.
As such, we slightly relax the original evaluation protocol. We use the first
sentence generated my MLLM as prediction result, and treated it as correct if
contains the ground truth answer.

Image Captioning. When MLLMs generate multiple sentences, we use the first
sentence as the captioning result for evaluation. Since MLLMs tend to generate
multiple sentences, we use the prompt "Describe this image in a sentence: This
is an image of" as prompt to condense the prediction for effective evaluation.

Object Counting. We extract the number of word from the first generated
sentence, and compare it with ground truth number.

Object Existence. We extract "yes" or "no" from the first generated sentence,
and compare it with ground truth.

Table 7: Dataset set statistics of our dataset for evaluation. Task | Split | Dataset | # of Instance
---|---|---|---
Visual Question Answering | validation | VQAv2 [28] | 440k
Image Captioning | validation | MS-COCO [29] | 25k
Object Counting | validation | MS-COCO [29] | 10k
Object Counting | validation | VCR [44] | 10k
Multi-Class Identification | validation | MS-COCO [29] | 10k
Multi-Class Identification | valitdaion | VCR [44] | 10k

## B. Benchmarking Fine-Grained Visual Understanding Tasks

We provide the details of the dataset used for evaluation in each task in
Table 7. In this work, we constructed two fine-grained perception tasks:
object counting and object existence based on instance-level annotations from
existing datasets. Specifically, they are constructed on MS-COCO [29] and VCR
[44] validation datasets. We provide their details as follows.

### B.1 Object Counting

Besides the visual features, the prompt of this task – "Question: How many
{obj} are there in the image? Answer:¨ is fed into the MLLM for evaluation. We
select the object name {obj} from the object list of the dataset. Since there
are often a single object of a certain class in one image, we select a maximum
of 3 objects with highest occurrence in the image to make this benchmark
challenging. Similar to object counting benchmarks, we report Mean Absolute
Error (MAE) and Root Mean Square Error (RMSE). Furthermore, we also report
accuracy which treats the counting as a classification problem during
evaluation. Both COCO-OC and VCR-OC contain a total of 10k tasks.

### B.2 Multi-Class Identification

Multi-label classification can be used as task to evaluate the model’s multi-
instance understanding capability. However, given the open-ended nature of
language models, the evaluation process is not stable since the language model
may generate more fine-grained object names than the dataset categories,
making a stable and fair evaluation difficult. To this end, we change the
format of this task and make the evaluation process more stable. We design the
prompt as "Question: Does {obj} exist in the image?" Answer:¨, and the model
is expected to answer "Yes" or "No". We select the object name {obj} from the
object list of the dataset. For each image, we randomly select at most 3
objects that exist in the image, and the same number of objects that does not
appear in the image, so as to make the evaluation set balanced. Both COCO-MCI
and VCR-MCI contain a total of 10k tasks.

## C. More Fine-grained Visual Understanding Results

In this section, we provide more detailed results on our two new tasks: OC and
OCI.

Detailed Object Counting Results. We show the detailed results of Object
Counting task on MS-COCO in Table 8. It can be observed that, when the images
contains relatively small number of objects (1-3), all methods can understand
the number of objects to some extend, where ours is significantly better than
others. However, when the images become more complex, where the number of
occurrence increases (4-6, 7-9), the performance has significantly dropped.
Similar trend can also observed in Table 9. These results demonstrate that
current MLLMs still struggle at correctly counting the objects, indicating
future research are required to make them more capable of challenging visual
understanding tasks.

Table 8: Detailed results on the Object Counting on MS-COCO dataset.

GT range | 1 - 3 | 4 - 6 | 7 - 9 | Overall
---|---|---|---|---
Method | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$
MiniGPT4 | 23.0 | 0.96 | 1.60 | 11.0 | 1.68 | 2.19 | 0.0 | 4.09 | 4.24 | 21.1 | 1.36 | 2.1
LLaVa | 26.5 | 0.89 | 1.86 | 11.0 | 1.72 | 3.25 | 1.58 | 4.75 | 5.83 | 22.0 | 1.36 | 2.70
BLIP-2 | 61.1 | 0.47 | 0.82 | 12.1 | 2.10 | 2.50 | 0.47 | 4.97 | 2.57 | 48.0 | 1.15 | 2.05
GVT (Ours) | 74.7 | 0.25 | 0.51 | 4.7 | 2.26 | 2.49 | 0.02 | 2.29 | 5.25 | 56.0 | 1.01 | 1.93

Table 9: Detailed results on the Object Counting on VCR dataset.

GT range | 1 - 3 | 4 - 6 | 7 - 9 | Overall
---|---|---|---|---
Method | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ | Acc $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$
MiniGPT4 | 25.0 | 0.84 | 1.32 | 13.0 | 1.48 | 1.82 | 0.00 | 4.34 | 4.46 | 25.0 | 1.51 | 2.24
LLaVa | 24.0 | 0.91 | 2.24 | 13.3 | 1.53 | 1.99 | 1.16 | 4.46 | 4.75 | 24.0 | 1.58 | 2.77
GVT (Ours) | 63.9 | 0.36 | 0.61 | 5.94 | 2.22 | 2.46 | 0.00 | 4.96 | 5.18 | 40.0 | 1.49 | 2.41

Detailed Multi-Class Identification Results. We provide more detailed results
on MCI task for MS-COCO in Table 10. The performance of all methods decrease
when the image becomes more complex (with more objects in the image). However,
the results on the VCR dataset does not show a stable trend. We conjecture
this can be related to the difference on the instruction tuning datasets,
which leads the model to focus on different types of objects.

Table 10: Detailed results on the Multi-Class Identification on MS-COCO dataset. #Objects | 1 - 9 | 10 - 20 | > 20 | Overall
---|---|---|---|---
MiniGPT4 | 80.7 | 72.3 | 96.1 | 76.8
LLaVa | 52.1 | 52.0 | 51.7 | 52.0
BLIP-2 | 85.4 | 77.6 | 75.2 | 81.9
GVT (Ours) | 89.7 | 87.0 | 84.5 | 88.2
Table 11: Detailed results on the Multi-Class Identification on VCR dataset. GT range | 1 - 9 | 10 - 20 | > 20 | Overall
---|---|---|---|---
MiniGPT4 | 71.2 | 70.2 | 71.1 | 70.8
LLaVa | 67.1 | 66.6 | 66.8 | 66.9
BLIP-2 | 67.6 | 70.3 | 70. | 68.9
GVT (Ours) | 77.1 | 80.6 | 81.5 | 78.8

## D. Object-centric Tasks

The work of [42] has proposed 4 tasks to utilize object detection dataset for
vision-language pretraining, including:

1\. List Objects
Input: "List all objects"
Output: "{obj1}, {obj2}, …"

2\. Object Existence
Input: "Does {obj} exist in the image?"
Output: "Yes/No."

3\. Group Existence
Input: "Does all of {obj1}, {obj2} and {obj3} exists in the image?"
Output: "Yes/No."

4\. Existence Selection
Input: "Which of {obj1}, {obj2}, {obj3} exist in the image?"
Output: "{obj1/2/3}"

To further utilize the rich annotations in object detection datasets, we also
design two tasks which facilitate the model’s learning on fine-grained visual
information.

5\. Object Counting
Input: "How many {obj}s are there in the image?"
Output: 1-9.

6\. Spatial Relation
Input: “What is the spatial relation between {obj1} and {obj2}? Choose one
from Top/Top Left/Left/Bottom Left/Bottom/Bottom Right/Right/Top Right"
Output: "Top/Top Left/Left/Bottom Left/Bottom/Bottom Right/Right/Top Right"

Task 6 is only performed when the selected {obj1} and {obj2} are unique in the
image, so as to avoid the referring ambiguity problem. For all tasks, we use
the input text as the prompt and ask the model to generate the output text.
The loss is only computed on the output texts. For each image, the task is
uniformly sampled on the two object detection datasets [40; 41].