C[1]>p#1

# VCR: Visual Caption Restoration

Tianyu Zhang1,2,3*, Suyuchen Wang1,3*, Lu Li4, Ge Zhang5,
Perouz Taslakian2, Sai Rajeswar2, Jie Fu6, Bang Liu1,3, Yoshua Bengio1,3
1 Mila, Québec AI Institute 2 ServiceNow Research 3 Université de Montréal
4 University of Pennsylvania 5 University of Waterloo 6 HKUST
{tianyu.zhang<EMAIL_ADDRESS>
{suyuchen.wang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{perouz.taslakian<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal
contribution.

###### Abstract

We introduce Visual Caption Restoration (VCR), a novel vision-language task
that challenges models to accurately restore partially obscured texts using
pixel-level hints within images. This task stems from the observation that
text embedded in images is intrinsically different from common visual elements
and natural language due to the need to align the modalities of vision, text,
and text embedded in images. While numerous works have integrated text
embedded in images into visual question-answering tasks, approaches to these
tasks generally rely on optical character recognition or masked language
modeling, thus reducing the task to mainly text-based processing. However,
text-based processing becomes ineffective in VCR as accurate text restoration
depends on the combined information from provided images, context, and subtle
cues from the tiny exposed areas of masked texts. We develop a pipeline to
generate synthetic images for the VCR task using image-caption pairs, with
adjustable caption visibility to control the task difficulty. With this
pipeline, we construct a dataset for VCR called VCR-wiki using images with
captions from Wikipedia, comprising $2.11M$ English and $346K$ Chinese
entities in both _easy_ and _hard_ configurations. Our results reveal that
current vision language models significantly lag behind human performance in
the VCR task, and merely fine-tuning the models on our dataset does not lead
to notable improvements. We release VCR-wiki and the data construction code to
facilitate future research.

##  1 Introduction

Figure 1: An example of the VCR task.

Recent advances in large language models, such as ChatGPT [51, 50] and Llama
[62], have spurred significant interest and progress in the field of vision-
language models. With models like GPT-4V [50] and Llava [38, 39, 40] blending
textual and visual information, the intersection of computer vision and
natural language processing has become a vibrant research frontier. These
integrated models aim to leverage the potential of vision and language
modalities to understand and interpret multimedia content more effectively.

Amidst this evolving landscape, we introduce Visual Caption Restoration (VCR),
a novel vision-language task designed to challenge existing models uniquely.
VCR challenges these models to restore obscured texts within images, a task
that demands an intricate synthesis of text, vision, and text embedded in the
image. The VCR task is grounded in two key insights: (1) text embedded within
images, with its characteristics different from common visual elements,
represents a distinct modality that requires careful alignment of vision,
textual data, and the structure of written texts, and (2) neuroscience
findings that suggest that humans are proficient in recognizing partially
occluded objects through sophisticated visual and cognitive processes [61, 52,
63, 16, 34]. By leveraging these insights, VCR seeks to explore how well
vision-language models can handle texts embedded within images, aligning
visual elements and natural language to mimic human-like multimodal
understanding and recognition.

The Visual Question Answering (VQA) task [3, 65, 47, 56] has been a popular
benchmark in assessing how well models align and interpret visual and
linguistic information. Traditional VQA approaches, however, predominantly
focus on direct queries about visible elements in images and do not address
the nuanced relationship between textual content embedded within the image and
the overall image context. This gap underscores the limited capabilities of
current models in processing integrated visual-textual data, particularly when
the textual component, which plays a critical role, is partially obscured or
altered.

To address these limitations, our VCR task builds on the premise that
effective text restoration from images requires an integrated understanding
beyond the capabilities of current VQA benchmarks. For example, in extreme
cases, models rely on existing Optical Character Recognition (OCR) system to
extract text from documents [56, 7]. The extracted text is then used as
context for generating answers, without a true semantic alignment between the
text and the visual elements of the document. This approach, while effective
in simple scenarios, falls short in more complex settings where text is
intricately woven into the visual narrative of the image.

To develop the VCR task, in this work, we introduce a pipeline for generating
synthetic images that allows for manipulation of the visibility of the textual
components of the image. This not only enhances the challenge posed by the
task, but also provides a scalable way to adjust task difficulty. The
resulting dataset, VCR-wiki, comprises 2.11M English data and 346K Chinese
data sourced from Wikipedia, featuring captions in both languages across
‘easy’ and ‘hard’ difficulty levels. Our evaluations indicate that existing
vision-language models significantly underperform compared to human
benchmarks, underscoring the need for novel model architectures and training
paradigms specifically geared towards this complex intermodal alignment.

By releasing VCR-wiki and the accompanying dataset construction code, we aim
to stimulate further research in this area, encouraging the development of
models that can more adeptly navigate the nuanced landscape of the restoration
of text embedded in images. This effort aligns with the broader goal of
advancing vision-language models to achieve a deeper, more intuitive
understanding of multimedia content, bridging the gap between human and
machine perception. The code is available at https://github.com/tianyu-z/VCR.

#### Contributions

The main contributions of this paper are:

  * C1

Introduce the Visual Caption Restoration (VCR) task to challenge vision-
language models to restore occluded texts in images.

  * C2

Develop a pipeline for generating synthetic images with embedded text that
allows for adjusting visibility of such text, thus providing a rich testing
environment for VCR.

  * C3

Create and release VCR-wiki, a dataset with multilingual captions from
Wikipedia images, designed to benchmark vision-language models (VLMs) on text
restoration tasks.

  * C4

Conduct empirical evaluations that show significant gaps between current
models and human performance on the VCR task. This highlights the
effectiveness of VCR for assessing advancements in VLMs, and underscores the
necessity for innovative model architectures and training techniques.

##  2 VCR Task Description

In this section, we compare the VCR task with other existing tasks and aim to
answer the following questions:

  1. Q1

What is the difference between VCR and other visual reconstruction tasks?

  2. Q2

Why should we care about VCR?

For better clarity, we define text embedded in image ($TEI$) as text
incorporated within the image. The term visual image (VI) pertains to the
portion of the image that excludes the text embedded in the image. The string
text (ST) is not part of the image itself, but is associated with it as a
distinct textual element. It is usually the question prompt in the form of
natural language, for example, ‘What are the covered texts in the image?
Please only guess the covered texts without outputting an explanation.’.
Consequently, an element of a Visual Caption Restoration (VCR) task can be
expressed as $(ST,(VI,TEI))$, where $ST$ is represented as a string and both
$VI$ and $TEI$ are presented in image form. This notation does not imply that
$VI$ and $TEI$ can be physically separated into two distinct image components.
Instead, this definition is adopted merely to facilitate a clearer explanation
of the concepts involved. Please refer to Figure 2 for an illustration of
$VI$, $TEI$, and $ST$.

A1 Existing tasks that are similar to VCR are the tasks of VQA and OCR. VQA
takes as input images and a natural language question and generates a free-
form response. As the ground-truth response is not unique, evaluating VQA
poses a major challenge. In contrast to VQA, OCR is a task where the ground-
truth responses are unique: OCR takes as input complete characters in image
form and outputs a string representing the characters in the image, without
considering the image context. Models pretrained with OCR are able to retrieve
texts embedded in the input image, even if they are incomplete or vague.
However, as the vagueness or occlusion of the textual components of the image
increases, retrieving the original text without considering the remaining
nontextual image context becomes harder, and OCR is no longer a good approach.
VCR bridges the gap between OCR and VQA: it reconstructs the unique text found
in the image while also considering the visual context of the rest of the
image.

Figure 2 is an example VCR task in hard mode, and Figure 1 shows an example
VCR task in the easy mode. Although humans can still fill the blanks easily in
the hard mode, it is nearly impossible for models with only OCR capabilities
to recover the covered texts without using the context. This is because the
pixel-level hints of single characters no longer correspond to a unique
solution.

A2 The proposed VCR task is significant in two aspects.

The first aspect of importance stems from discoveries in neuroscience about
human cognitive abilities to recognize partially occluded objects [16, 34].
Although existing models can recognize objects and texts in images, they often
struggle with the complexity of occluded objects due to significant
information loss in the images. In contrast, humans excel at filling in
missing information using a combination of low-level visual processing and
high-level cognitive functions, such as those managed by the prefrontal
cortex. This cortical area is known to handle complex cognitive processes such
as decision making and memory retention, which are essential for integrating
fragmented visual input into coherent objects. We believe that the occlusion
restoration task serves as a probe that can effectively distinguish low-level
recognition and high-level cognition involving reasoning. In addition,
understanding these neural mechanisms can inspire new algorithms capable of
mimicking human-like perception and interpretation in dynamic, real-world
conditions where occlusion is common.

The second aspect underscores the unique challenge presented by the VCR task,
distinguishing it significantly from existing benchmarks, such as traditional
VQA or the occluded object restoration task. By occluding texts instead of
common visual objects, VCR targets the models’ text-image alignment
capability, which is one of the major challenges for vision-language models.
VCR mandates that models align textual and visual information in a manner that
replicates human-like understanding involving the utilization of both textual
and visual clues. This task requires a deep integration of visual ($VI$),
embedded textual ($TEI$), and contextual interpretation across modalities,
pushing beyond simple text extraction as performed in OCR tasks. In OCR, the
focus is primarily on recognizing visible characters, often without the need
to understand their contextual relevance within the image narrative. In
contrast, VCR introduces complexity by requiring the model to use available
partial texts and the visual context collaboratively to reconstruct the
obscured content accurately. This not only tests the model’s ability to
process $TEI$-$VI$ modalities effectively, but also challenges it to maintain
internal consistency, akin to human cognitive processes where context and
visual clues guide understanding and response. Besides, the difficulty of the
task can be adjusted by altering the extent of text occlusion, offering a
scalable and flexible framework for systematically enhancing model
capabilities in text-visual alignment and semantic comprehension. This
rigorous testing ground will help evolve vision-language models to better
grasp the nuanced interplay between text and imagery.

##  3 Dataset Creation

Figure 2: Illustration of the dataset creation pipeline for VCR-wiki. visual
image ($VI$), text embedded in image ($TEI$) and string text ($ST$) in an
example of the English Hard configuration of VCR-wiki. The solid line-enclosed
contents ($VI$ and $TEI$) are part of the image, whereas the dotted line-
enclosed content ($ST$) is given separately from the image.

The VCR task requires aligning visual images ($VI$) with text embedded in
images ($TEI$). Therefore, the dataset creation process relies on a set of
highly correlated image-text pairs. We utilize the primary images and their
corresponding captions from Wikipedia as the data source111Datasource:
https://huggingface.co/datasets/wikimedia/wit_base. to create VCR-wiki, a
Wikipedia-based VCR dataset. The pipeline for creating VCR-wiki is shown in
Figure 2. Before constructing the dataset, we first filter out instances with
sensitive content, including NSFW and crime-related terms, to mitigate AI risk
and biases.

The VCR-wiki dataset is formatted as a VQA task, where each instance includes
an image, a question, and a ground-truth answer. The images are synthesized
from text-image pairs by stacking the image ($VI$) with its corresponding text
description ($TEI$) vertically, mimicking the format of a captioned image.
This stacked image is referred to as a stacked $VI$ +$TEI$ image. Each $VI$
+$TEI$ image is resized to a width of 300 pixels. To avoid excessive image
height, we truncate $TEI$ to a maximum of five lines. We filter the dataset to
exclude instances with $VI$ +$TEI$ images exceeding 900 pixels in height to
avoid drastic resolution changes during data pre-processing.

Within $TEI$, we use spaCy to randomly select several 5-grams in the caption
for masking. To ensure the restoration process is doable by a human without
too much professional domain knowledge, the 5-grams do not contain numbers,
person names, religious or political groups, facilities, organizations,
locations, dates and time labeled by spaCy, and the total masked token does
not exceed 50% of the tokens in the caption. We exclude all instances that do
not have a single eligible 5-gram. The selected 5-grams are partially obscured
by a white rectangle that reveals only the upper and lower parts of the text,
with the proportion of coverage varying to adjust task difficulty.
Furthermore, to assess the impact of $VI$ on model performance, we create an
ablation for each image, maintaining the resolution of the $VI$ +$TEI$ image,
but retaining only the $TEI$ part in the center of the image.

The VCR task involves a predefined question that prompts the model to produce
the obscured text in the image. The ground truth answer corresponds to the
caption displayed in the uncovered portion of the stacked image. Due to the
extensive availability of vision-language models and a significant user base
in both English and Chinese, we have chosen to develop the dataset in these
two languages. For each language, we meticulously select the covering
proportion to create two task variants: (1) an easy version, where the task is
easy for native speakers but open-source OCR models almost always fail, and
(2) a hard version, where the revealed part consists of only one to two pixels
for the majority of letters or characters, yet the restoration task remains
feasible for native speakers of the language.

We release the dataset under the CC BY-SA 4.0 license. All configurations of
VCR-wiki are publicly available at https://huggingface.co/collections/vcr-
org/vcr-visual-caption-recognition-6661393b1761e2aff7b967b9.

###  3.1 Dataset Format and Statistics

Table 1: Basic statistics of the dataset. Note that the Easy and Hard
configurations for each language share the same statistics. We report the
mean, standard deviation, and the 5th and 95th percentile ($\eta_{.5}$ and
$\eta_{.95}$) for the stacked image height and the number of obscured text
spans. Unit is in pixels.

| # Train | # Val | # Test |  $VI$ +$TEI$ Image Height | # Obscured Text Spans
---|---|---|---|---|---
| Mean | SD | $\eta_{.5}$ | $\eta_{.95}$ | Mean | SD | $\eta_{.5}$ | $\eta_{.95}$
English | 2095733 | 5000 | 5000 | 375.52 | 106.01 | 253 | 564 | 1.62 | 0.63 | 1 | 3
Chinese | 336448 | 5000 | 5000 | 360.44 | 102.76 | 239 | 562 | 2.06 | 0.94 | 1 | 4

The VCR dataset comprises four configurations: English Easy, English Hard,
Chinese Easy and Chinese Hard. Each configuration can be further divided into
training, validation, and test splits. The validation and test splits contain
5,000 entities each. The training set for English configurations and Chinese
configurations contains $2,095,733$ and $336,448$ instances, respectively,
which can be used for model continuous pretraining. We include more detailed
statistics of the dataset in Table 1.

##  4 Experiments

In this section, we report the experiment results of existing state-of-the-art
vision-language models on our proposed VCR tasks. The fine-tuning and
evaluation of open-source models are conducted on a mix of NVIDIA A100 80G and
L40S 48G GPUs in an internal cluster.

###  4.1 Models

We report evaluation results of the following models:

#### Closed-source Models.

We evaluate several most advanced proprietary models with their provided APIs.
The evaluated models include GPT-4o (gpt-4o-2024-0513), GPT-4 Turbo
(gpt-4-turbo-2024-04-09), GPT-4V (gpt-4-1106-vision-preview) [51, 50], Claude
3 Opus (claude-3-opus-20240229), Claude 3.5 Sonnet
(claude-3-5-sonnet-20240620) [2], Gemini 1.5 pro (gemini-1.5-pro-001) [59],
Reka Core (reka-core-20240501) [60], and Qwen-VL-Max (tested on May 2024) [4].

#### Open-source Models.

We evaluate open-source models with the best performance on the OpenVLM
Leaderboard222We selected the highest-performing open-source models with fewer
than 40 billion parameters from the OpenVLM Leaderboard as of May 23, 2024.
Details are available at
https://huggingface.co/spaces/opencompass/open_vlm_leaderboard. and state-of-
the-art Chinese VLM models. The evaluated models include InternVL-
Chat-V1.5[11], MiniCPM-Llama3-V2.5 [25], InternLM-XComposer2-VL-7B [14],
CogVLM2-Llama3-19B-Chat [67], Idefics2-8B [32], Yi-VL-34B [1], Yi-VL-6B [1],
Qwen-VL-Chat [4], DeepSeek-VL-7B-Chat [43], DeepSeek-VL-1.3B-Chat [43], Monkey
[41, 35] and DocOwl-1.5 [22]. Out of these models, Idefics2-8B is an English-
only model, and CogVLM2-Llama3-19B-Chat has its Chinese variant,
CogVLM2-Llama3-19B-Chinese-Chat. Please refer to Table 2 for the model
specifications.

#### Finetuned Models.

To test whether VLMs can learn to conduct VCR via fine-tuning, we select two
models from the open-sourced models, CogVLM2-Llama3-19B-Chat and MiniCPM-
Llama3-V2.5, and fine-tune them on a subset of VCR’s training set.

More specifically, we fine-tune CogVLM2-Llama3-19B-Chat and MiniCPM-
Llama3-V2.5 in the English Hard configuration, and CogVLM2-Llama3-19B-Chinese-
Chat and MiniCPM-Llama3-V2.5 on the Chinese Hard configuration. The models are
finetuned using LoRA [23] with $r=8$ and $\alpha=32$. We adopt the schedule-
free AdamW optimizer [12] with a learning rate $2e-4$. The effective batch
size is 64. Each model is trained on the first 16,000 examples of the training
set for 1 epoch. All fine-tuning experiments are performed on a single node
with 4 NVIDIA L40S 48G GPUs.

###  4.2 Metrics

We measure the quality of the model’s restoration of each masked $n$-gram
(where $n=5$ in our setting, as specified in Section 3). Due to the
variability of different models’ outputs, for each masked $n$-gram
$m\in\mathbb{V}_{e}^{n}$, where $\mathbb{V}_{e}$ is the vocabulary of the
evaluation tokenizer333We utilize spaCy’s en_core_web_sm’s and
zh_core_web_sm’s tokenizer for English and Chinese evaluation, respectively.,
we extract the most similar $n$-gram $\hat{m}\in\mathbb{V}_{e}^{n}$ with the
least edit distance in the model’s generation.

We report the two metrics below in our experiment section to measure the
restoration quality: Exact Match ($EM$), which measures whether the restored
$n$-gram $\hat{m}$ totally matches the ground-truth $m$; and Jaccard Index
($J$), which measures the similarity of $\hat{m}$ and $m$ as bag-of-words.

  * •

Exact Match ($EM$), which measures whether the restored $n$-gram $\hat{m}$
totally matches the ground-truth $m$;

$EM(m,\hat{m})=\cases{1}&\text{if}m=\hat{m},\\\ 0\text{otherwise}.$

  * •

Jaccard Index ($J$), which is a more relaxed metric that measures the
similarity of $\hat{m}$ and $m$ as bag-of-words.

$J(m,\hat{m})=\frac{|S(m)\cap S(\hat{m})|}{|S(m)\cup S(\hat{m})|},$

where $S(m)$ represents the set of tokens in $m$.

###  4.3 Experimental Results

Please refer to the exact match score and the Jaccard-index of the evaluation
in Table 4.3.

Table 2: Model specifications

Model name | Model size | Open-sourced
---|---|---
Claude 3 Opus | - |  2613
GPT-4 Turbo | - |  2613
GPT-4o | - |  2613
GPT-4V | - |  2613
Qwen-VL-Max | - |  2613
CogVLM2 444https://huggingface.co/THUDM/CogVLM2-Llama3-chat-19B | 19B | $\checkmark$
CogVLM2-Chinese 555https://huggingface.co/THUDM/cogvlm2-llama3-Chinese-chat-19B | 19B | $\checkmark$
DeepSeek-VL 666https://huggingface.co/deepseek-ai/deepseek-vl-1.3b-chat | 1.3B | $\checkmark$
DeepSeek-VL 777https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat | 7B | $\checkmark$
Idefics2 888https://huggingface.co/HuggingFaceM4/Idefics2-8B | 8B | $\checkmark$
InternVL-V1.5 999https://huggingface.co/OpenGVLab/InternVL-Chat-V1-5 | 25.5B | $\checkmark$
InternLM-XComposer2-VL 101010https://huggingface.co/InternLM/InternLM-XComposer2-VL-7B | 7B | $\checkmark$
MiniCPM-V2.5 111111https://huggingface.co/OpenBMB/MiniCPM-Llama3-V-2_5 | 8B | $\checkmark$
Qwen-VL 121212https://huggingface.co/Qwen/Qwen-VL-Chat | 7B | $\checkmark$
Yi-VL131313https://huggingface.co/01-ai/Yi-VL-34B | 34B | $\checkmark$
Yi-VL141414https://huggingface.co/01-ai/Yi-VL-6B | 6B | $\checkmark$
Monkey151515https://huggingface.co/echo840/Monkey-Chat | 7B | $\checkmark$
DocOwl-1.5-Omni161616https://huggingface.co/mPLUG/DocOwl1.5-Omni | 8B | $\checkmark$

#### Open-Source Models.

We evaluate each open-source model based on the whole 5,000 examples in the
test set. Note that Idefics2-8B only supports the English task, hence it has
no evaluation score on the Chinese task.

Although achieving state-of-the-art performance on the Open VLM leaderboard,
almost all the tested models achieve a low exact match accuracy in the English
Easy configuration and fail on the other settings. The best open-source model
across the 4 configurations (English Easy, English Hard, Chinese Easy, and
Chinese Hard) is CogVLM2-Llama3-Chat. This might be attributed to its
pretraining process and the special architecture. We also notice that VI has a
negative impact for most models on the exact match scores ($\Delta<0$), which
means that the image information is not properly utilized. The best performed
open-source model, CogVLM2-Llama3-Chat, and its fine-tuned version have
positive $\Delta$, except for the Chinese Hard configuration. This indicates
that information from VI could help improve the model performance on VCR.

For different languages, we noticed a large performance drop when testing the
model in Chinese configurations, despite the fact that all models claim to
have basic English-Chinese duolingual capabilities. This is somehow
surprising, since Chinese characters, due to their logographic nature, may
exhibit a higher degree of recognizability compared to languages that use
alphabetic scripts in one order [69, 79].

Table 3: Results of various open-source and closed-source vision language
models on the VCR task, using both English (EN) and Chinese (ZH), in both easy
and hard modes. FT means the model is finetuned on 16,000 samples from the
VCR-wiki training set. The best results among the finetuned models are
underlined while the best results among the models without finetuning are
highlighted in bold. Subscripts provide the standard deviation obtained from
bootstrap.

@|c|c|p4cmC2cmC2.3cmC2.3cmC2cmC2.3cmC2.3cmC2cm|@

Language Mode Model name Model size Exact match (%) $\uparrow$ Jaccard index
(%) $\uparrow$
$VI$ \+ $TEI$ $TEI$ $\Delta$ $VI$ \+ $TEI$ $TEI$ $\Delta$

English Easy Closed-source models
Claude 3 Opus - $62.0_{0.13}$ $77.0_{0.5}$ -15 $77.67_{0.32}$ $88.41_{0.39}$
-10.74
Claude 3.5 Sonnet - $63.85_{1.71}$ $72.8_{1.56}$ -8.94 $74.65_{1.33}$
$83.48_{1.14}$ -8.83
Gemini 1.5 Pro - $62.73_{1.66}$ $82.98_{1.3}$ -20.25 $77.71_{1.21}$
$91.56_{0.76}$ -13.85
GPT-4 Turbo - $78.74_{0.13}$ $81.94_{0.25}$ -3.2 $88.54_{0.24}$ $92.18_{0.3}$
-3.65
GPT-4o - $\boldsymbol{91.55_{0.29}}$ $\boldsymbol{94.56_{0.13}}$ -3.01
$\boldsymbol{96.44_{0.11}}$ $\boldsymbol{97.76_{0.06}}$ -1.32
GPT-4V - $52.04_{0.24}$ $37.86_{0.22}$ 14.17 $65.36_{0.39}$ $54.13_{0.41}$
11.23
Qwen-VL-Max - $76.8_{0.5}$ $85.53_{0.19}$ -8.74 $85.71_{0.28}$ $91.45_{0.29}$
-5.74
Reka Core - $66.46_{1.64}$ $78.51_{1.42}$ -12.05 $84.23_{0.86}$ $90.45_{0.7}$
-6.22
Open-source models
CogVLM2 19B $\boldsymbol{83.25_{0.07}}$ $\boldsymbol{78.29_{0.04}}$ 4.96
$\boldsymbol{89.75_{0.1}}$ $\boldsymbol{88.07_{0.08}}$ 1.68
CogVLM2-FT 19B $\underline{93.27_{0.03}}$ $\underline{92.63_{0.07}}$ 0.64
$\underline{97.62_{0.02}}$ $\underline{97.4_{0.01}}$ 0.22
DeepSeek-VL 1.3B $23.04_{0.05}$ $31.09_{0.12}$ -8.04 $46.84_{0.07}$
$52.36_{0.06}$ -5.52
DeepSeek-VL 7B $38.01_{0.12}$ $45.94_{0.1}$ -7.93 $60.02_{0.15}$
$64.72_{0.04}$ -4.7
DocOwl-1.5-Omni 8B $0.84_{0.01}$ $1.55_{0.02}$ -0.71 $13.34_{0.03}$
$14.62_{0.04}$ -1.28
Monkey 7B $50.66_{0.1}$ $56.2_{0.08}$ -5.54 $67.6_{0.09}$ $72.82_{0.08}$ -5.22
Idefics2 8B $15.75_{0.11}$ $27.77_{0.11}$ -12.02 $31.97_{0.02}$ $51.0_{0.03}$
-19.03
InternLM-XComposer2-VL 7B $46.64_{0.1}$ $46.4_{0.11}$ 0.24 $70.99_{0.1}$
$72.14_{0.07}$ -1.14
InternVL-V1.5 25.5B $14.65_{0.13}$ $75.06_{0.1}$ -60.41 $51.42_{0.04}$
$87.1_{0.03}$ -35.68
MiniCPM-V2.5 8B $31.81_{0.08}$ $40.05_{0.09}$ -8.25 $53.24_{0.1}$ $63.2_{0.1}$
-9.96
MiniCPM-V2.5-FT 8B $40.96_{0.14}$ $44.62_{0.07}$ -3.67 $64.4_{0.05}$
$67.62_{0.1}$ -3.22
Qwen-VL 7B $49.71_{0.17}$ $52.15_{0.15}$ -2.44 $69.94_{0.07}$ $72.28_{0.08}$
-2.34
Yi-VL 34B $0.82_{0.03}$ $1.61_{0.04}$ -0.79 $5.59_{0.04}$ $7.72_{0.03}$ -2.13
Yi-VL 6B $0.75_{0.01}$ $1.65_{0.01}$ -0.9 $5.54_{0.02}$ $7.76_{0.03}$ -2.22
Hard Closed-source models
Claude 3 Opus - $37.8_{0.28}$ $50.0_{0.33}$ -12.2 $57.68_{0.8}$ $70.16_{0.64}$
-12.48
Claude 3.5 Sonnet - $41.74_{1.69}$ $44.72_{1.78}$ -2.98 $56.15_{1.46}$
$58.54_{1.6}$ -2.4
Gemini 1.5 Pro - $28.07_{1.58}$ $38.76_{1.68}$ -10.68 $51.9_{1.22}$
$59.62_{1.27}$ -7.72
GPT-4 Turbo - $45.15_{0.28}$ $48.64_{0.57}$ -3.5 $65.72_{0.25}$ $67.86_{0.2}$
-2.14
GPT-4o - $\boldsymbol{73.2_{0.16}}$ $\boldsymbol{82.43_{0.17}}$ -9.22
$\boldsymbol{86.17_{0.21}}$ $\boldsymbol{92.01_{0.2}}$ -5.84
GPT-4V - $25.83_{0.44}$ $14.95_{0.3}$ 10.87 $44.63_{0.48}$ $30.08_{0.67}$
14.56
Qwen-VL-Max - $41.65_{0.32}$ $52.72_{0.2}$ -11.07 $61.18_{0.35}$
$70.19_{0.37}$ -9.01
Reka Core - $6.71_{0.89}$ $11.18_{1.15}$ -4.47 $25.84_{0.95}$ $35.83_{1.05}$
-9.99
Open-source models
CogVLM2 19B $\boldsymbol{37.98_{0.18}}$ $\boldsymbol{17.68_{0.06}}$ 20.3
$\boldsymbol{59.99_{0.05}}$ $\boldsymbol{39.69_{0.03}}$ 20.3
CogVLM2-FT 19B $\underline{77.44_{0.05}}$ $\underline{66.07_{0.13}}$ 11.38
$\underline{90.17_{0.03}}$ $\underline{83.41_{0.07}}$ 6.76
DeepSeek-VL 1.3B $0.16_{0.01}$ $0.39_{0.02}$ -0.23 $11.89_{0.02}$
$11.47_{0.03}$ 0.42
DeepSeek-VL 7B $1.0_{0.02}$ $1.75_{0.03}$ -0.75 $15.9_{0.08}$ $17.2_{0.04}$
-1.3
DocOwl-1.5-Omni 8B $0.04_{0.0}$ $0.02_{0.0}$ 0.01 $7.76_{0.01}$ $7.74_{0.02}$
0.03
Monkey 7B $1.96_{0.04}$ $2.43_{0.03}$ -0.48 $14.02_{0.03}$ $14.11_{0.03}$
-0.09
Idefics2 8B $0.65_{0.01}$ $0.94_{0.02}$ -0.29 $9.93_{0.05}$ $12.57_{0.02}$
-2.64
InternLM-XComposer2-VL 7B $0.7_{0.01}$ $0.92_{0.01}$ -0.22 $12.51_{0.02}$
$13.23_{0.02}$ -0.72
InternVL-V1.5 25.5B $1.99_{0.02}$ $6.49_{0.04}$ -4.5 $16.73_{0.06}$
$26.4_{0.03}$ -9.67
MiniCPM-V2.5 8B $1.41_{0.03}$ $1.96_{0.02}$ -0.55 $11.94_{0.02}$
$13.37_{0.04}$ -1.43
MiniCPM-V2.5-FT 8B $13.86_{0.1}$ $13.73_{0.05}$ 0.12 $36.89_{0.06}$
$36.51_{0.06}$ 0.38
Qwen-VL 7B $2.0_{0.03}$ $2.32_{0.03}$ -0.32 $15.04_{0.05}$ $14.27_{0.05}$ 0.77
Yi-VL 34B $0.07_{0.0}$ $0.05_{0.0}$ 0.02 $4.31_{0.02}$ $5.89_{0.02}$ -1.58
Yi-VL 6B $0.06_{0.0}$ $0.04_{0.0}$ 0.02 $4.46_{0.02}$ $5.91_{0.01}$ -1.46
Chinese Easy Closed-source models
Claude 3 Opus - $0.9_{0.3}$ $1.0_{0.31}$ -0.1 $11.5_{0.48}$ $10.0_{0.49}$ 1.49
Claude 3.5 Sonnet - $1.0_{0.31}$ $0.8_{0.28}$ 0.2 $7.54_{0.54}$ $7.5_{0.51}$
0.03
Gemini 1.5 Pro - $1.1_{0.32}$ $0.5_{0.22}$ 0.6 $11.1_{0.56}$ $11.47_{0.48}$
-0.37
GPT-4o - $\boldsymbol{14.87_{1.14}}$ $\boldsymbol{22.46_{1.35}}$ -7.58
$\boldsymbol{39.05_{0.99}}$ $\boldsymbol{48.24_{1.09}}$ -9.19
GPT-4 Turbo - $0.2_{0.14}$ $0.1_{0.1}$ 0.1 $8.42_{0.36}$ $6.97_{0.29}$ 1.45
Qwen-VL-Max - ${6.34_{0.08}}$ ${9.92_{0.09}}$ -3.58 ${13.45_{0.41}}$
${22.86_{0.46}}$ -9.42
Reka Core - $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.43_{0.26}$ $3.15_{0.2}$ 0.28
Open-source models
CogVLM2-Chinese 19B $\boldsymbol{33.24_{0.04}}$ $\boldsymbol{30.7_{0.07}}$
2.54 $\boldsymbol{57.57_{0.06}}$ $\boldsymbol{53.66_{0.04}}$ 3.91
CogVLM2-Chinese-FT 19B $\underline{61.69_{0.05}}$ $\underline{59.85_{0.08}}$
1.84 $\underline{78.14_{0.05}}$ $\underline{77.12_{0.04}}$ 1.02
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.56_{0.01}$ $3.17_{0.02}$ 3.4
DeepSeek-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.08_{0.01}$ $6.84_{0.01}$ -2.76
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.14_{0.01}$ $3.38_{0.01}$ -2.23
Monkey 7B $0.62_{0.01}$ $1.44_{0.01}$ -0.82 $8.34_{0.06}$ $10.95_{0.03}$ -2.61
InternLM-XComposer2-VL 7B $0.27_{0.01}$ $0.23_{0.01}$ 0.04 $12.32_{0.02}$
$12.28_{0.03}$ 0.04
InternVL-V1.5 25.5B $4.78_{0.02}$ $5.32_{0.02}$ -0.55 $26.43_{0.03}$
$21.7_{0.04}$ 4.72
MiniCPM-V2.5 8B $4.1_{0.02}$ $5.05_{0.08}$ -0.95 $18.03_{0.07}$ $22.94_{0.04}$
-4.9
MiniCPM-V2.5-FT 8B $7.44_{0.03}$ $7.92_{0.03}$ -0.49 $29.87_{0.04}$
$31.32_{0.03}$ -1.45
Qwen-VL 7B $0.04_{0.01}$ $0.0_{0.0}$ 0.04 $1.5_{0.01}$ $0.34_{0.01}$ 1.15
Yi-VL 34B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.44_{0.01}$ $1.8_{0.01}$ 2.64
Yi-VL 6B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.37_{0.01}$ $1.76_{0.0}$ 2.6
Hard Closed-source models
Claude 3 Opus - $0.3_{0.18}$ $0.1_{0.1}$ 0.2 $9.22_{0.38}$ $8.09_{0.33}$ 1.13
Claude 3.5 Sonnet - $0.2_{0.15}$ $0.0_{0.0}$ 0.2 $4.0_{0.33}$ $2.37_{0.23}$
1.63
Gemini 1.5 Pro - $0.7_{0.26}$ $0.5_{0.23}$ 0.2 $11.82_{0.51}$ $11.75_{0.44}$
0.07
GPT-4o - $\boldsymbol{2.2_{0.47}}$ $\boldsymbol{1.8_{0.4}}$ 0.4
$\boldsymbol{22.72_{0.67}}$ $\boldsymbol{22.89_{0.65}}$ -0.17
GPT-4 Turbo - $0.0_{0.0}$ $0.2_{0.13}$ -0.2 $8.58_{0.3}$ $6.87_{0.28}$ 1.72
Qwen-VL-Max - ${0.89_{0.06}}$ ${1.38_{0.1}}$ -0.49 ${5.4_{0.19}}$
${12.29_{0.18}}$ -6.89
Reka Core - $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.35_{0.23}$ $2.97_{0.2}$ 0.38
Open-source models
CogVLM2-Chinese 19B $\boldsymbol{1.34_{0.03}}$ $\boldsymbol{2.67_{0.02}}$
-1.32 $\boldsymbol{17.35_{0.03}}$ $\boldsymbol{19.51_{0.03}}$ -2.16
CogVLM2-Chinese-FT 19B $\underline{42.11_{0.09}}$ $\underline{45.63_{0.06}}$
-3.51 $\underline{65.67_{0.15}}$ $\underline{69.28_{0.04}}$ -3.61
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.46_{0.01}$ $3.22_{0.02}$ 3.24
DeepSeek-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $5.11_{0.01}$ $7.21_{0.01}$ -2.1
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.37_{0.01}$ $4.07_{0.02}$ -2.7
Monkey 7B $0.12_{0.01}$ $0.07_{0.0}$ 0.05 $6.36_{0.01}$ $6.68_{0.03}$ -0.32
InternLM-XComposer2-VL 7B $0.07_{0.01}$ $0.09_{0.0}$ -0.02 $8.97_{0.02}$
$8.51_{0.01}$ 0.46
InternVL-V1.5 25.5B $0.03_{0.0}$ $0.1_{0.01}$ -0.07 $8.46_{0.01}$
$6.27_{0.04}$ 2.19
MiniCPM-V2.5 8B $0.09_{0.0}$ $0.08_{0.0}$ 0.01 $7.39_{0.02}$ $7.89_{0.01}$
-0.5
MiniCPM-V2.5-FT 8B $1.53_{0.01}$ $1.11_{0.02}$ 0.42 $18.0_{0.03}$
$15.35_{0.02}$ 2.65
Qwen-VL 7B $0.01_{0.0}$ $0.01_{0.0}$ 0 $1.17_{0.01}$ $0.12_{0.0}$ 1.06
Yi-VL 34B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.12_{0.0}$ $1.81_{0.01}$ 2.31
Yi-VL 6B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.0_{0.01}$ $1.88_{0.01}$ 2.12

Moreover, we found that models, such as internlm-xcomposer2, are good at OCR
and understanding image documents (as demonstrated by DocOwl 1.5 and Monkey)
still have the potential to be improved in the VCR task. This highlights the
unique and indispensable role of VCR in the current suite of benchmarks.
Excelling in other document-related benchmarks does not guarantee similar
performance in VCR tasks, emphasizing VCR’s distinct challenges and value.

#### Closed-Source Models.

We evaluate every closed-source model with the first 500 examples in the test
set. In English tasks, GPT-4o scores the best among the models that have not
been finetuned. Even though GPT-4 series support Chinese, we found that GPT-4V
(gpt-4-1106-vision-preview) is not able to recognize Chinese characters
embedded in the image even without any occlusion. Thus, we do not test GPT-4V
on Chinese tasks.

In English configurations, closed-source models outperform all open-source
models except CogVLM2, which indicates that model scaling might help improve
performance on the VCR task. However, compared with the human evaluation
results in Section 4.4, we notice a large performance gap, especially in the
English Hard configuration. This shows significant room for improvement in the
current state-of-the-art models.

Refer to Table A to compare open and closed source models using the same 500
test cases.

###  4.4 Human Evaluation

We recruited 7 volunteers to perform human evaluation on a subset of the
samples of our datasets. Two out of the seven evaluators are native English
speakers, while five are native Chinese speakers who are also fluent in
English171717The TOEFL scores of the non-native English-speaking participants
range from 102/120 to 112/120.. All volunteers have earned postgraduate
degrees majoring in one of the following fields: biology, statistics, computer
science, and economics. The evaluations were conducted on a voluntary basis
and participants received no rewards.

We gave the volunteer the following instructions: (1) We ask the volunteers to
focus on the puzzles. Each example in the hard collection may require 30
seconds to 2 minutes of focused attention; and (2) we ask the volunteers to
utilize the context rather than directly brute-force the puzzle.

Every sample is solved by at least 3 volunteers. In English, we release the
exact match score in 2 splits: all errors counted (All), and only count errors
not related to date and person names (Filtered).

Table 4: Human evaluation results on the VCR task for in terms of exact
matches. $N$ is the number of puzzles in each language.

| EN Easy (N = 169) | EN Hard (N = 169) | ZH Easy (N = 188) | ZH Hard (N = 188)
---|---|---|---|---
| Mean (%) | SD (%) | Mean (%) | SD (%) | Mean (%) | SD (%) | Mean (%) | SD (%)
All | 96.65 | 0.34 | 91.12 | 1.18 | 98.58 | 0.31 | 91.84 | 0.81
Filtered | 98.62 | 0.34 | 97.63 | 2.13 | 99.47 | 0.00 | 96.63 | 1.11

Refer to Table A to compare all models with human evaluation results using the
same test cases.

##  5 Related Work

#### Masked modeling.

Masked language modeling (MLM) introduced by BERT [13] and its autoregressive
counterparts by GPT [8] have been the foundations of pre-training modern
natural language processing (NLP) models. Both are trained to predict a
masked-out proportion based on the context. Similarly, inspired by BERT and
GPT, iGPT [9] uses the input sequences of the pixels to predict the next
unknown pixels. Masked Autoencoders (MAE) [19] introduce Masked Image Modeling
(MIM) by masking out random patches from input images and reconstructing the
missing parts in the images in pixel space. The goal is to encourage the model
to learn the representations of the image by understanding the context and
structure of the visible patches, which helps to reconstruct the masked images
accurately.

#### Visual Question Answering (VQA).

Several datasets have been proposed for visual question answering VQA [3, 77,
17, 47]. FVQA [65] and OK-VQA[44] are datasets about knowledge-based visual
question answering and contains questions that necessitate the usage of
external knowledge resources. CLEVR [30] is a synthetic VQA dataset that
mainly focuses on visual reasoning abilities. Recognizing the need to develop
VQA models that can understand text, Text-VQA [56, 6, 48, 68] aims to read and
reason about texts embedded within images in the context of image-question
answering. Several datasets [56, 6, 48] have been developed for the Text-VQA
task, such as the TextVQA dataset [56] and the ST-VQA dataset [6] on natural
images, the OCR-VQA dataset [47] on book or movie covers, the InfographicVQA
[45] dataset on infographics, and the DocVQA dataset [46] on document images.

#### Vision Language Model.

Vision-language models are designed for tasks that involve understanding and
generating content from images and text [58, 40, 31, 32]. For example, models
have been developed to combine Llama3 with advanced vision-language processing
capabilities to handle complex multimodal tasks [75, 71, 24, 74, 67, 14].
Qwen-VL [4] enhances visual-linguistic representations for more accurate
contextual interpretations, while OpenGVLab-InternVL-Chat [11, 10] merges the
InternVL framework with interactive chat capabilities. These studies typically
employ a multimodal encoder [53, 76, 70] to process multimodal data, which is
then mapped to the same input space of the language model. General-purpose
models such as the GPT-4 series models [51, 50], the Claude series models [2],
the Gemini series models [59] and the Reka series models [60] have also been
adapted for vision-language tasks, demonstrating strong performance in
multimodal tasks. Finally, DocLLM [64] specializes in document understanding
by integrating visual and textual data to enhance the interpretation and
generation of document-related content. These models collectively represent
significant advancements in vision-language integration, contributing unique
capabilities and enhancements to the understanding and generation of
multimodal information.

#### Optical Character Recognition (OCR).

OCR [49] and its subproblems [21, 57, 54, 15] have been well-studied in the
literature in the constrained setting. However, classical OCR methods often
cannot perform well on images captured in the wild in an unconstrained
setting. Many new methods have been developed for advancing scene-text
recognition on camera-captured images [5, 18, 26, 28, 66, 55, 81, 33]. In
addition to the detection and recognition of OCR tasks, visual question
answering has emerged as an important downstream task in the OCR literature.
With the development of Text-VQA, new methods for improving the reading
abilities in VQA utilizing OCR have been proposed. For example, LoRRA [56]
extends a VQA model Pythia [29] with an OCR module to better handle Text-VQA
tasks. TAP [72] incorporates scene texts that are generated from OCR engines
during pretraining to further improve Text-VQA capabilities.

#### Scene Text Detection (STD).

Identifying and interpreting text in natural images serves as a foundational
step towards developing a comprehensive VQA system. This capability allows for
the integration of textual information, enhancing overall scene comprehension.
Existing approaches mainly consist of two steps: text detection and
recognition. Several methods based on Fully Convolutional Neural Networks have
been proposed for scene text detection [36, 37, 81, 20, 80]. In terms of text
recognition, [27] proposed to directly recognize texts from entire input
images using a 90k-class convolutional neural network with each class
corresponding to an English word [37]. In addition, several methods [73, 78,
42] have been proposed to tackle the issue of irregular text in scene text
detection.

##  6 Conclusion

In this work, we introduced the Visual Caption Restoration (VCR) task, a novel
vision-language challenge aimed at promoting the integration of visual and
textual modalities, including text embedded in both natural language tokens
and image formats and highly obscured text embedded in the image. We developed
a specialized pipeline to create a dataset tailored to this task, utilizing
correlated image-text pairs. This task stands out from existing methods by
requiring a more profound integration of visual cues and partially obscured
text, highlighting its uniqueness and importance in the field.

We conducted extensive evaluations of state-of-the-art vision-language models
(VLMs) in both English and Chinese. The results demonstrated significant room
for improvement, suggesting that current models have not yet fully exploited
the capabilities necessary for VCR. We selected models representing both the
highest and average performance tiers for additional fine-tuning with our
dataset. Although fine-tuning exhibited potential for enhancing VCR
capabilities, it did not consistently result in significant improvements,
indicating the complexity and challenges of adapting models to this task.

By introducing the VCR task and its specialized dataset, we aim to advance
research in vision-language interaction. The unique challenges of VCR seek to
improve model development and training, extending the limits of multimodal AI.
We invite the community to utilize our dataset and develop innovative
strategies to boost the performance of vision-language models.

## Acknowledgments and Disclosure of Funding

We acknowledge the contribution of Mila for supporting the evaluation of
various models on our datasets through providing GPU access. None of the
authors received third-party funding during the last 36 months prior to this
submission.

## References

  * [1] 01.AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.  Yi: Open foundation models by 01.ai, 2024.
  * [2] Anthropic.  The claude 3 model family: Opus, sonnet, haiku.  2024\.
  * [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh.  VQA: Visual Question Answering.  In International Conference on Computer Vision (ICCV), 2015.
  * [4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  arXiv preprint arXiv: 2308.12966, 2023.
  * [5] Alessandro Bissacco, Mark Cummins, Yuval Netzer, and Hartmut Neven.  Photoocr: Reading text in uncontrolled conditions.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013.
  * [6] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marcal Rusinol, Ernest Valveny, C.V. Jawahar, and Dimosthenis Karatzas.  Scene text visual question answering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [7] Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar.  Rosetta: Large scale system for text detection and recognition in images.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 71–79, New York, NY, USA, 2018. Association for Computing Machinery.
  * [8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In International conference on machine learning, pages 1691–1703. PMLR, 2020.
  * [10] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al.  How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.  arXiv preprint arXiv:2404.16821, 2024.
  * [11] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai.  Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.  arXiv preprint arXiv:2312.14238, 2023.
  * [12] Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky.  The road less scheduled, 2024.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [14] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang.  Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.  arXiv preprint arXiv: 2401.16420, 2024.
  * [15] Volkmar Frinken, Andreas Fischer, R Manmatha, and Horst Bunke.  A novel word spotting method based on recurrent neural networks.  IEEE transactions on pattern analysis and machine intelligence, 34(2):211–224, 2011.
  * [16] Amber M Fyall, Yasmine El-Shamayleh, Hannah Choi, Eric Shea-Brown, and Anitha Pasupathy.  Dynamic representation of partially occluded objects in primate prefrontal and visual cortex.  eLife, 6:e25784, September 2017.
  * [17] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [18] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
  * [20] Wenhao He, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu.  Deep direct regression for multi-oriented scene text detection.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [21] Nicholas R Howe.  Document binarization with automatic parameter tuning.  International journal on document analysis and recognition (ijdar), 16:247–258, 2013.
  * [22] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou.  mplug-docowl 1.5: Unified structure learning for ocr-free document understanding.  arXiv preprint arXiv: 2403.12895, 2024.
  * [23] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [24] Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Large multilingual models pivot zero-shot multimodal learning across languages.  arXiv preprint arXiv:2308.12038, 2023.
  * [25] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.
  * [26] Weilin Huang, Yu Qiao, and Xiaoou Tang.  Robust scene text detection with convolution neural network induced mser trees.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 497–511, Cham, 2014. Springer International Publishing.
  * [27] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Reading text in the wild with convolutional neural networks.  International journal of computer vision, 116:1–20, 2016.
  * [28] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman.  Deep features for text spotting.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 512–528, Cham, 2014. Springer International Publishing.
  * [29] Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh.  Pythia v0.1: the winning entry to the vqa challenge 2018, 2018.
  * [30] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [31] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh.  Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
  * [32] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.  What matters when building vision-language models?, 2024.
  * [33] Chen-Yu Lee and Simon Osindero.  Recursive recurrent nets with attention modeling for ocr in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [34] Bao Li, Chi Zhang, Long Cao, Panpan Chen, Tianyuan Liu, Hui Gao, Linyuan Wang, Bin Yan, and Li Tong.  Brain Functional Representation of Highly Occluded Object Recognition.  Brain Sciences, 13(10):1387, October 2023.
  * [35] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai.  Monkey: Image resolution and text label are important things for large multi-modal models.  arXiv preprint arXiv: 2311.06607, 2023.
  * [36] Minghui Liao, Baoguang Shi, and Xiang Bai.  Textboxes++: A single-shot oriented scene text detector.  IEEE Transactions on Image Processing, 27(8):3676–3690, 2018.
  * [37] Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu.  Textboxes: A fast text detector with a single deep neural network.  In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  * [38] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023.
  * [39] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
  * [40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023.
  * [41] Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai.  Textmonkey: An ocr-free large multimodal model for understanding document.  arXiv preprint arXiv: 2403.04473, 2024.
  * [42] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60:91–110, 2004.
  * [43] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan.  Deepseek-vl: Towards real-world vision-language understanding, 2024.
  * [44] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [45] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar.  Infographicvqa.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1697–1706, January 2022.
  * [46] Minesh Mathew, Dimosthenis Karatzas, and C.V. Jawahar.  Docvqa: A dataset for vqa on document images.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, January 2021.
  * [47] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947–952, 2019.
  * [48] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947–952, 2019.
  * [49] G. Nagy.  Twenty years of document image analysis in pami.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(01):38–62, jan 2000.
  * [50] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, et al.  Gpt-4 technical report.  arXiv preprint arXiv: 2303.08774, 2023.
  * [51] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in neural information processing systems, 35:27730–27744, 2022.
  * [52] Luiz Pessoa, Evan Thompson, and Alva Noë.  Finding out about filling-in: A guide to perceptual completion for visual science and the philosophy of perception.  Behavioral and Brain Sciences, 21(6):723–748, 1998.
  * [53] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and I. Sutskever.  Learning transferable visual models from natural language supervision.  International Conference on Machine Learning, 2021.
  * [54] Faisal Shafait, Daniel Keysers, and Thomas Breuel.  Performance evaluation and benchmarking of six-page segmentation algorithms.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(6):941–954, 2008.
  * [55] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11):2298–2304, 2017.
  * [56] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019.
  * [57] Ray Smith.  A simple and efficient skew detection algorithm via text row accumulation.  In Proceedings of 3rd International Conference on Document Analysis and Recognition, volume 2, pages 1145–1148. IEEE, 1995.
  * [58] Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang.  Alpha-clip: A clip model focusing on wherever you want.  arXiv preprint arXiv: 2312.03818, 2023.
  * [59] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  arXiv preprint arXiv: 2403.05530, 2024.
  * [60] Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie.  Reka core, flash, and edge: A series of powerful multimodal language models.  arXiv preprint arXiv: 2404.12387, 2024.
  * [61] G. Thinés, A. Costall, and G. Butterworth.  Michotte’s Experimental Phenomenology of Perception.  Routledge Library Editions: Phenomenology. Taylor & Francis, 2013.
  * [62] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv: 2307.09288, 2023.
  * [63] Rob van Lier and Walter Gerbino.  Perceptual completions.  In The Oxford Handbook of Perceptual Organization. Oxford University Press, 08 2015.
  * [64] Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu.  Docllm: A layout-aware generative language model for multimodal document understanding.  arXiv preprint arXiv: 2401.00908, 2023.
  * [65] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel.  Fvqa: Fact-based visual question answering.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(10):2413–2427, 2018.
  * [66] Tao Wang, David J. Wu, Adam Coates, and Andrew Y. Ng.  End-to-end text recognition with convolutional neural networks.  In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pages 3304–3308, 2012.
  * [67] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.  Cogvlm: Visual expert for pretrained language models, 2023.
  * [68] Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang.  On the general value of evidence, and bilingual scene-text visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [69] Shilian Wu, Yongrui Li, and Zengfu Wang.  Chinese text recognition enhanced by glyph and character semantic information.  International Journal on Document Analysis and Recognition (IJDAR), 27(1):45–56, March 2024.
  * [70] Yusong Wu, K. Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and S. Dubnov.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  IEEE International Conference on Acoustics, Speech, and Signal Processing, 2022.
  * [71] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang.  LLaVA-UHD: an lmm perceiving any aspect ratio and high-resolution images.  arXiv preprint arXiv:2403.11703, 2024.
  * [72] Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo.  Tap: Text-aware pre-training for text-vqa and text-caption.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8751–8761, June 2021.
  * [73] Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu.  Detecting texts of arbitrary orientations in natural images.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1083–1090, 2012.
  * [74] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al.  Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.  arXiv preprint arXiv:2312.00849, 2023.
  * [75] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness.  arXiv preprint arXiv:2405.17220, 2024.
  * [76] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  IEEE International Conference on Computer Vision, 2023.
  * [77] Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Yin and Yang: Balancing and answering binary visual questions.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [78] Zhengdong Zhang, Arvind Ganesh, Xiao Liang, and Yi Ma.  Tilt: Transform invariant low-rank textures.  International journal of computer vision, 99:1–24, 2012.
  * [79] Yuliang Zhao, Xinyue Zhang, Boya Fu, Zhikun Zhan, Hui Sun, Lianjiang Li, and Guanglie Zhang.  Evaluation and recognition of handwritten chinese characters based on similarities.  Applied Sciences, 12(17), 2022.
  * [80] Zhuoyao Zhong, Lianwen Jin, and Shuangping Huang.  Deeptext: A new approach for text proposal generation and text detection in natural images.  In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1208–1212, 2017.
  * [81] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: An efficient and accurate scene text detector.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.

##  Appendix A Additional evaluation results on first 100 and 500 test cases

Table 5: Results of various open-source and closed-source vision language
models on the VCR task using the first 100 test cases. Each test case includes
one or more puzzles. FT means that the model is finetuned on 16,000 samples
from the VCR-wiki train dataset. The best results among the finetuned models
are underlined while the best results among the models without finetuning are
highlighted in bold. Subscripts provide the standard deviation obtained from
bootstrap.

@|c|c|p4cmC3cmC2.5cmC2.5cmC2.2cmC2.5cmC2.5cmC2.2cm|@

Language Mode Model name Model size Exact match (%) $\uparrow$ Jaccard index
(%) $\uparrow$
$VI$ \+ $TEI$ $TEI$ $\Delta$ $VI$ \+ $TEI$ $TEI$ $\Delta$

English Easy Closed-source models
Claude 3 Opus - $62.0_{0.76}$ $82.0_{0.63}$ -20 $78.06_{0.24}$ $91.12_{0.13}$
-13.06
Claude 3.5 Sonnet - $70.41_{3.46}$ $75.15_{3.36}$ -4.73 $78.1_{2.85}$
$86.5_{2.18}$ -8.4
Gemini 1.5 Pro - $71.01_{3.4}$ $86.98_{2.67}$ -15.98 $82.89_{2.27}$
$94.21_{1.32}$ -11.32
GPT-4 Turbo - $78.47_{0.22}$ $86.6_{0.79}$ -8.13 $88.08_{0.25}$ $94.15_{0.2}$
-6.07
GPT-4o - $\boldsymbol{90.91_{0.36}}$ $\boldsymbol{95.69_{0.23}}$ -4.78
$\boldsymbol{96.77_{0.16}}$ $\boldsymbol{98.45_{0.06}}$ -1.68
GPT-4V - $25.36_{0.5}$ $18.18_{0.54}$ 7.18 $35.64_{0.22}$ $28.49_{0.23}$ 7.15
Qwen-VL-Max - $82.3_{0.19}$ $88.04_{0.43}$ -5.74 $89.73_{0.32}$ $92.55_{0.17}$
-2.82
Reka Core - $65.68_{3.78}$ $78.11_{3.19}$ -12.43 $83.14_{2.04}$ $90.43_{1.49}$
-7.29
Open-source models
CogVLM2 19B $\boldsymbol{86.39_{0.66}}$ $\boldsymbol{84.62_{0.92}}$ 1.78
$\boldsymbol{91.39_{0.11}}$ $\boldsymbol{91.63_{0.11}}$ -0.24
CogVLM2-FT 19B $\underline{94.08_{0.2}}$ $\underline{94.67_{0.26}}$ -0.59
$\underline{98.03_{0.07}}$ $\underline{98.22_{0.03}}$ -0.2
DeepSeek-VL 1.3B $19.53_{0.69}$ $26.04_{1.47}$ -6.51 $43.73_{0.18}$
$48.03_{0.16}$ -4.3
DeepSeek-VL 7B $36.09_{1.36}$ $44.97_{0.79}$ -8.88 $57.81_{0.18}$
$61.83_{0.33}$ -4.01
DocOwl-1.5-Omni 8B $0.59_{0.14}$ $1.18_{0.14}$ -0.59 $12.69_{0.04}$
$13.3_{0.06}$ -0.61
Monkey 7B $46.75_{0.44}$ $48.52_{0.41}$ -1.78 $67.82_{0.22}$ $68.59_{0.13}$
-0.76
Idefics2 8B $14.79_{0.72}$ $26.63_{0.37}$ -11.83 $34.2_{0.37}$ $51.96_{0.1}$
-17.76
InternLM-XComposer2-VL 7B $47.93_{0.69}$ $47.34_{0.57}$ 0.59 $73.88_{0.22}$
$74.58_{0.16}$ -0.7
InternVL-V1.5 25.5B $15.38_{0.29}$ $75.15_{0.7}$ -59.76 $52.21_{0.16}$
$85.87_{0.29}$ -33.66
MiniCPM-V2.5 8B $30.18_{0.66}$ $36.09_{0.34}$ -5.92 $53.1_{0.18}$
$59.06_{0.14}$ -5.96
MiniCPM-V2.5-FT 8B $39.05_{0.69}$ $46.75_{0.59}$ -7.69 $63.05_{0.28}$
$69.89_{0.33}$ -6.84
Qwen-VL 7B $47.34_{0.44}$ $46.75_{0.57}$ 0.59 $69.02_{0.35}$ $69.19_{0.37}$
-0.17
Yi-VL 34B $1.78_{0.16}$ $1.18_{0.11}$ 0.59 $6.21_{0.06}$ $7.5_{0.08}$ -1.3
Yi-VL 6B $2.37_{0.13}$ $1.78_{0.22}$ 0.59 $6.24_{0.07}$ $8.05_{0.11}$ -1.81
Hard Closed-source models
Claude 3 Opus - $34.0_{1.12}$ $51.0_{0.5}$ -17 $57.02_{0.24}$ $70.32_{0.15}$
-13.31
Claude 3.5 Sonnet - $46.75_{3.58}$ $43.2_{3.83}$ 3.55 $57.74_{3.33}$
$54.13_{3.51}$ 3.61
Gemini 1.5 Pro - $33.73_{3.69}$ $43.79_{3.74}$ -10.06 $57.09_{2.67}$
$62.34_{2.76}$ -5.25
GPT-4 Turbo - $53.11_{0.46}$ $57.42_{0.5}$ -4.31 $71.75_{0.19}$ $73.82_{0.24}$
-2.07
GPT-4o - $\boldsymbol{74.16_{0.31}}$ $\boldsymbol{84.69_{0.31}}$ -10.53
$\boldsymbol{86.99_{0.09}}$ $\boldsymbol{93.19_{0.07}}$ -6.21
GPT-4V - $28.71_{0.49}$ $16.27_{0.73}$ 12.44 $49.89_{0.15}$ $33.64_{0.16}$
16.25
Qwen-VL-Max - $40.67_{0.38}$ $55.02_{0.46}$ -14.35 $61.8_{0.19}$
$72.46_{0.15}$ -10.66
Reka Core - $7.1_{2.01}$ $10.65_{2.38}$ -3.55 $25.49_{1.99}$ $36.78_{2.19}$
-11.29
Open-source models
CogVLM2 19B $\boldsymbol{44.97_{0.83}}$ $\boldsymbol{21.3_{0.47}}$ 23.67
$\boldsymbol{65.39_{0.2}}$ $\boldsymbol{43.86_{0.27}}$ 21.53
CogVLM2-FT 19B $\underline{75.74_{0.72}}$ $\underline{67.46_{0.64}}$ 8.28
$\underline{90.6_{0.13}}$ $\underline{84.26_{0.08}}$ 6.34
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $11.17_{0.03}$ $10.88_{0.06}$ 0.29
DeepSeek-VL 7B $0.59_{0.09}$ $1.78_{0.17}$ -1.18 $16.71_{0.11}$ $18.09_{0.13}$
-1.38
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $7.89_{0.05}$ $8.28_{0.05}$ -0.4
Monkey 7B $1.18_{0.22}$ $3.55_{0.18}$ -2.37 $12.66_{0.21}$ $15.97_{0.08}$
-3.31
Idefics2 8B $1.18_{0.2}$ $0.59_{0.1}$ 0.59 $10.81_{0.08}$ $11.34_{0.12}$ -0.53
InternLM-XComposer2-VL 7B $0.0_{0.0}$ $0.59_{0.09}$ -0.59 $12.69_{0.08}$
$14.05_{0.11}$ -1.35
InternVL-V1.5 25.5B $1.78_{0.21}$ $7.1_{0.22}$ -5.33 $16.28_{0.09}$
$26.6_{0.14}$ -10.32
MiniCPM-V2.5 8B $1.18_{0.12}$ $1.78_{0.12}$ -0.59 $12.02_{0.12}$
$12.41_{0.07}$ -0.39
MiniCPM-V2.5-FT 8B $10.06_{0.43}$ $13.02_{0.54}$ -2.96 $34.67_{0.2}$
$36.43_{0.19}$ -1.76
Qwen-VL 7B $1.78_{0.21}$ $2.96_{0.12}$ -1.18 $15.7_{0.14}$ $15.06_{0.19}$ 0.63
Yi-VL 34B $0.59_{0.09}$ $0.0_{0.0}$ 0.59 $4.39_{0.07}$ $5.49_{0.08}$ -1.1
Yi-VL 6B $0.59_{0.13}$ $0.0_{0.0}$ 0.59 $5.12_{0.03}$ $5.5_{0.06}$ -0.38
Chinese Easy Closed-source models

Claude 3 Opus $0.53_{0.51}$ $0.53_{0.55}$ 0 $11.34_{1.07}$ $9.14_{0.93}$ 2.2
Claude 3.5 Sonnet - $1.6_{0.91}$ $2.13_{1.05}$ -0.53 $8.07_{1.29}$
$9.9_{1.48}$ -1.84
Gemini 1.5 Pro - $0.53_{0.56}$ $0.0_{0.0}$ 0.53 $12.94_{1.26}$ $12.77_{1.17}$
0.16
GPT-4o - $\boldsymbol{14.89_{2.51}}$ $\boldsymbol{21.81_{2.98}}$ -6.91
$\boldsymbol{38.57_{2.46}}$ $\boldsymbol{48.29_{2.43}}$ -9.72
GPT-4 Turbo - $0.53_{0.55}$ $0.0_{0.0}$ 0.53 $11.09_{1.05}$ $7.51_{0.65}$ 3.58
Qwen-VL-Max - ${5.93_{0.19}}$ ${8.7_{0.37}}$ -2.77 ${13.53_{0.11}}$
${18.5_{0.1}}$ -4.97
Reka Core - $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.04_{0.53}$ $2.42_{0.45}$ 0.61
Open-source models
CogVLM2-Chinese 19B $\boldsymbol{34.57_{0.66}}$ $\boldsymbol{34.04_{1.01}}$
0.53 $\boldsymbol{58.78_{0.13}}$ $\boldsymbol{57.26_{0.12}}$ 1.52
CogVLM2-Chinese-FT 19B $\underline{66.49_{0.74}}$ $\underline{67.55_{0.73}}$
-1.06 $\underline{79.48_{0.17}}$ $\underline{81.78_{0.09}}$ -2.3
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.69_{0.07}$ $2.92_{0.02}$ 3.78
DeepSeek-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.99_{0.07}$ $6.71_{0.02}$ -2.72
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.23_{0.04}$ $2.97_{0.02}$ -1.75
Monkey 7B $1.06_{0.12}$ $0.53_{0.06}$ 0.53 $9.23_{0.08}$ $12.29_{0.13}$ -3.06
InternLM-XComposer2-VL 7B $1.06_{0.09}$ $0.53_{0.07}$ 0.53 $13.1_{0.03}$
$13.26_{0.03}$ -0.16
InternVL-V1.5 25.5B $4.26_{0.28}$ $3.19_{0.38}$ 1.06 $26.9_{0.23}$
$16.31_{0.14}$ 10.59
MiniCPM-V2.5 8B $4.79_{0.16}$ $7.45_{0.35}$ -2.66 $20.58_{0.11}$
$25.38_{0.13}$ -4.81
MiniCPM-V2.5-FT 8B $6.91_{0.33}$ $7.98_{0.4}$ -1.06 $30.8_{0.07}$
$31.46_{0.52}$ -0.66
Qwen-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.41_{0.02}$ $0.66_{0.03}$ 0.76
Yi-VL 34B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.53_{0.03}$ $1.84_{0.05}$ 2.69
Yi-VL 6B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.73_{0.02}$ $1.55_{0.02}$ 3.18
Hard Closed-source models
Claude 3 Opus - $1.06_{0.77}$ $0.53_{0.54}$ 0.53 $9.23_{1.04}$ $7.77_{0.83}$
1.45
Claude 3.5 Sonnet - $0.53_{0.51}$ $0.0_{0.0}$ 0.53 $4.11_{0.84}$ $3.32_{0.71}$
0.79
Gemini 1.5 Pro - $1.06_{0.71}$ $1.06_{0.77}$ 0 $11.58_{1.14}$ $13.34_{1.2}$
-1.76
GPT-4o - $\boldsymbol{2.66_{1.16}}$ $\boldsymbol{1.6_{0.92}}$ 1.06
$\boldsymbol{23.69_{1.65}}$ $\boldsymbol{23.69_{1.48}}$ 0
GPT-4 Turbo - $0.0_{0.0}$ $0.53_{0.53}$ -0.53 $8.51_{0.7}$ $8.02_{0.78}$ 0.49
Qwen-VL-Max - ${1.19_{0.12}}$ ${1.98_{0.09}}$ -0.79 ${6.19_{0.1}}$
${11.09_{0.11}}$ -4.9
Reka Core - $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.22_{0.51}$ $3.62_{0.57}$ -0.4
Open-source models
CogVLM2-Chinese 19B $\boldsymbol{3.19_{0.19}}$ $\boldsymbol{3.19_{0.32}}$ 0
$\boldsymbol{18.33_{0.14}}$ $\boldsymbol{21.38_{0.09}}$ -3.05
CogVLM2-Chinese-FT 19B $\underline{46.81_{0.32}}$ $\underline{46.28_{0.49}}$
0.53 $\underline{66.85_{0.39}}$ $\underline{69.79_{0.12}}$ -2.95
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.5_{0.03}$ $4.16_{0.03}$ 2.34
DeepSeek-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $5.22_{0.04}$ $7.45_{0.06}$ -2.23
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.35_{0.02}$ $3.57_{0.04}$ -2.23
Monkey 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.15_{0.11}$ $6.62_{0.11}$ -0.47
InternLM-XComposer2-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $8.17_{0.03}$
$7.99_{0.03}$ 0.18
InternVL-V1.5 25.5B $0.0_{0.0}$ $0.0_{0.0}$ 0 $7.7_{0.08}$ $4.67_{0.04}$ 3.03
MiniCPM-V2.5 8B $0.53_{0.07}$ $0.53_{0.07}$ 0 $7.28_{0.06}$ $7.71_{0.06}$
-0.43
MiniCPM-V2.5-FT 8B $1.06_{0.08}$ $2.13_{0.19}$ -1.06 $18.46_{0.1}$
$16.42_{0.22}$ 2.03
Qwen-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.1_{0.04}$ $0.06_{0.01}$ 1.04
Yi-VL 34B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.17_{0.04}$ $2.02_{0.04}$ 2.15
Yi-VL 6B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.15_{0.06}$ $2.38_{0.04}$ 1.77

Table 6: Results of various open-source and closed-source vision language
models on the VCR task using the first 500 test cases. Each test case includes
one or more puzzles. FT means the model is finetuned on 16,000 samples from
the VCR-wiki train dataset. The best results among the finetuned models are
underlined while the best results among the models without finetuning are
highlighted in bold. Subscripts provide the standard deviation obtained from
bootstrap.

@|c|c|p4cmC3.3cmC2.5cmC2.5cmC2.3cmC2.5cmC2.5cmC2.3cm|@

Language Mode Model name Model size Exact match (%) $\uparrow$ Jaccard index
(%) $\uparrow$
$VI$ \+ $TEI$ $TEI$ $\Delta$ $VI$ \+ $TEI$ $TEI$ $\Delta$

English Easy Closed-source models
Claude 3 Opus - $62.0_{0.13}$ $77.0_{0.5}$ -15 $77.67_{0.32}$ $88.41_{0.39}$
-10.74
Claude 3.5 Sonnet - $63.85_{1.71}$ $72.8_{1.56}$ -8.94 $74.65_{1.33}$
$83.48_{1.14}$ -8.83
Gemini 1.5 Pro - $62.73_{1.66}$ $82.98_{1.3}$ -20.25 $77.71_{1.21}$
$91.56_{0.76}$ -13.85
GPT-4 Turbo - $78.74_{0.13}$ $81.94_{0.25}$ -3.2 $88.54_{0.24}$ $92.18_{0.3}$
-3.65
GPT-4o - $\boldsymbol{91.55_{0.29}}$ $\boldsymbol{94.56_{0.13}}$ -3.01
$\boldsymbol{96.44_{0.11}}$ $\boldsymbol{97.76_{0.06}}$ -1.32
GPT-4V - $52.04_{0.24}$ $37.86_{0.22}$ 14.17 $65.36_{0.39}$ $54.13_{0.41}$
11.23
Qwen-VL-Max - $76.8_{0.5}$ $85.53_{0.19}$ -8.74 $85.71_{0.28}$ $91.45_{0.29}$
-5.74
Reka Core - $66.46_{1.64}$ $78.51_{1.42}$ -12.05 $84.23_{0.86}$ $90.45_{0.7}$
-6.22
Open-source models
CogVLM2 19B $\boldsymbol{83.11_{0.28}}$ $\boldsymbol{79.63_{0.33}}$ 3.48
$\boldsymbol{89.43_{0.27}}$ $\boldsymbol{88.65_{0.26}}$ 0.79
CogVLM2-FT 19B $\underline{92.8_{0.06}}$ $\underline{92.67_{0.13}}$ 0.12
$\underline{97.51_{0.24}}$ $\underline{97.45_{0.07}}$ 0.06
DeepSeek-VL 1.3B $21.86_{0.17}$ $30.68_{0.3}$ -8.82 $45.4_{0.33}$
$52.02_{0.73}$ -6.62
DeepSeek-VL 7B $37.76_{0.42}$ $45.47_{0.21}$ -7.7 $59.07_{0.43}$
$64.26_{0.57}$ -5.2
DocOwl-1.5-Omni 8B $0.62_{0.06}$ $1.86_{0.06}$ -1.24 $12.65_{0.3}$
$14.09_{0.12}$ -1.44
Monkey 7B $47.2_{0.2}$ $54.16_{0.41}$ -6.96 $65.7_{0.4}$ $71.17_{0.72}$ -5.47
Idefics2 8B $14.91_{0.14}$ $29.07_{0.2}$ -14.16 $31.63_{0.3}$ $51.5_{0.21}$
-19.87
InternLM-XComposer2-VL 7B $46.09_{0.35}$ $46.34_{0.25}$ -0.25 $71.11_{0.2}$
$71.76_{0.67}$ -0.65
InternVL-V1.5 25.5B $15.78_{0.23}$ $74.91_{0.27}$ -59.13 $52.0_{0.31}$
$86.82_{0.47}$ -34.82
MiniCPM-V2.5 8B $32.8_{0.16}$ $36.77_{0.25}$ -3.98 $52.56_{0.25}$
$60.89_{0.19}$ -8.32
MiniCPM-V2.5-FT 8B $42.36_{0.3}$ $45.34_{0.35}$ -2.98 $65.39_{0.6}$
$67.85_{0.43}$ -2.46
Qwen-VL 7B $45.47_{0.35}$ $52.17_{0.33}$ -6.71 $66.81_{0.74}$ $71.73_{0.59}$
-4.93
Yi-VL 34B $0.87_{0.06}$ $1.24_{0.04}$ -0.37 $5.61_{0.28}$ $7.63_{0.42}$ -2.02
Yi-VL 6B $1.12_{0.03}$ $1.37_{0.14}$ -0.25 $5.93_{0.16}$ $7.33_{0.23}$ -1.39
Hard Closed-source models
Claude 3 Opus - $37.8_{0.28}$ $50.0_{0.33}$ -12.2 $57.68_{0.8}$ $70.16_{0.64}$
-12.48
Claude 3.5 Sonnet - $41.74_{1.69}$ $44.72_{1.78}$ -2.98 $56.15_{1.46}$
$58.54_{1.6}$ -2.4
Gemini 1.5 Pro - $28.07_{1.58}$ $38.76_{1.68}$ -10.68 $51.9_{1.22}$
$59.62_{1.27}$ -7.72
GPT-4 Turbo - $45.15_{0.28}$ $48.64_{0.57}$ -3.5 $65.72_{0.25}$ $67.86_{0.2}$
-2.14
GPT-4o - $\boldsymbol{73.2_{0.16}}$ $\boldsymbol{82.43_{0.17}}$ -9.22
$\boldsymbol{86.17_{0.21}}$ $\boldsymbol{92.01_{0.2}}$ -5.84
GPT-4V - $25.83_{0.44}$ $14.95_{0.3}$ 10.87 $44.63_{0.48}$ $30.08_{0.67}$
14.56
Qwen-VL-Max - $41.65_{0.32}$ $52.72_{0.2}$ -11.07 $61.18_{0.35}$
$70.19_{0.37}$ -9.01
Reka Core - $6.71_{0.89}$ $11.18_{1.15}$ -4.47 $25.84_{0.95}$ $35.83_{1.05}$
-9.99
Open-source models
CogVLM2 19B $\boldsymbol{41.74_{0.25}}$ $\boldsymbol{16.77_{0.22}}$ 24.97
$\boldsymbol{62.56_{0.33}}$ $\boldsymbol{38.41_{0.44}}$ 24.15
CogVLM2-FT 19B $\underline{75.9_{0.13}}$ $\underline{65.22_{0.18}}$ 10.68
$\underline{89.75_{0.14}}$ $\underline{82.71_{0.27}}$ 7.04
DeepSeek-VL 1.3B $0.37_{0.02}$ $0.12_{0.01}$ 0.25 $11.42_{0.09}$
$11.41_{0.22}$ 0.01
DeepSeek-VL 7B $0.75_{0.02}$ $1.61_{0.1}$ -0.87 $15.8_{0.29}$ $17.18_{0.41}$
-1.38
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $7.34_{0.06}$ $7.61_{0.16}$ -0.27
Monkey 7B $1.37_{0.05}$ $2.24_{0.15}$ -0.87 $13.16_{0.18}$ $14.45_{0.24}$
-1.29
Idefics2 8B $0.62_{0.02}$ $0.62_{0.06}$ 0 $9.24_{0.11}$ $11.0_{0.16}$ -1.75
InternLM-XComposer2-VL 7B $0.5_{0.04}$ $0.37_{0.05}$ 0.12 $12.38_{0.13}$
$13.22_{0.11}$ -0.83
InternVL-V1.5 25.5B $1.74_{0.13}$ $6.34_{0.13}$ -4.6 $16.85_{0.17}$
$26.11_{0.24}$ -9.26
MiniCPM-V2.5 8B $1.74_{0.08}$ $1.61_{0.08}$ 0.12 $11.55_{0.24}$ $11.69_{0.38}$
-0.15
MiniCPM-V2.5-FT 8B $11.43_{0.11}$ $14.29_{0.16}$ -2.86 $35.13_{0.19}$
$36.65_{0.68}$ -1.52
Qwen-VL 7B $1.61_{0.03}$ $1.74_{0.03}$ -0.12 $15.28_{0.13}$ $14.43_{0.54}$
0.85
Yi-VL 34B $0.12_{0.01}$ $0.0_{0.0}$ 0.12 $4.31_{0.08}$ $5.45_{0.13}$ -1.14
Yi-VL 6B $0.12_{0.02}$ $0.0_{0.0}$ 0.12 $4.49_{0.05}$ $5.7_{0.12}$ -1.21
Chinese Easy Closed-source models
Claude 3 Opus - $0.9_{0.3}$ $1.0_{0.31}$ -0.1 $11.5_{0.48}$ $10.0_{0.49}$ 1.49
Claude 3.5 Sonnet - $1.0_{0.31}$ $0.8_{0.28}$ 0.2 $7.54_{0.54}$ $7.5_{0.51}$
0.03
Gemini 1.5 Pro - $1.1_{0.32}$ $0.5_{0.22}$ 0.6 $11.1_{0.56}$ $11.47_{0.48}$
-0.37
GPT-4o - $\boldsymbol{14.87_{1.14}}$ $\boldsymbol{22.46_{1.35}}$ -7.58
$\boldsymbol{39.05_{0.99}}$ $\boldsymbol{48.24_{1.09}}$ -9.19
GPT-4 Turbo - $0.2_{0.14}$ $0.1_{0.1}$ 0.1 $8.42_{0.36}$ $6.97_{0.29}$ 1.45
Qwen-VL-Max - ${6.34_{0.08}}$ ${9.92_{0.09}}$ -3.58 ${13.45_{0.41}}$
${22.86_{0.46}}$ -9.42
Reka Core - $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.43_{0.26}$ $3.15_{0.2}$ 0.28
Open-source models
CogVLM2-Chinese 19B $\boldsymbol{33.63_{0.15}}$ $\boldsymbol{31.44_{0.19}}$
2.2 $\boldsymbol{57.97_{0.56}}$ $\boldsymbol{54.05_{0.54}}$ 3.92
CogVLM2-Chinese-FT 19B $\underline{63.97_{0.55}}$ $\underline{62.67_{0.17}}$
1.3 $\underline{79.71_{0.41}}$ $\underline{79.22_{0.47}}$ 0.49
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.1_{0.1}$ $3.25_{0.05}$ 2.85
DeepSeek-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.28_{0.07}$ $7.3_{0.05}$ -3.02
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.19_{0.05}$ $3.83_{0.06}$ -2.63
Monkey 7B $0.2_{0.01}$ $1.4_{0.05}$ -1.2 $7.89_{0.3}$ $10.26_{0.24}$ -2.37
InternLM-XComposer2-VL 7B $0.6_{0.05}$ $0.2_{0.04}$ 0.4 $12.34_{0.25}$
$12.52_{0.14}$ -0.18
InternVL-V1.5 25.5B $3.99_{0.09}$ $4.69_{0.18}$ -0.7 $25.88_{0.45}$
$20.73_{0.53}$ 5.15
MiniCPM-V2.5 8B $4.59_{0.11}$ $4.89_{0.09}$ -0.3 $18.12_{0.33}$ $22.28_{0.18}$
-4.17
MiniCPM-V2.5-FT 8B $7.29_{0.14}$ $7.09_{0.12}$ 0.2 $29.36_{0.39}$
$30.67_{0.38}$ -1.31
Qwen-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.25_{0.03}$ $0.43_{0.06}$ 0.82
Yi-VL 34B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.69_{0.09}$ $1.71_{0.06}$ 2.98
Yi-VL 6B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.28_{0.06}$ $1.66_{0.04}$ 2.62
Hard Closed-source models
Claude 3 Opus - $0.3_{0.18}$ $0.1_{0.1}$ 0.2 $9.22_{0.38}$ $8.09_{0.33}$ 1.13
Claude 3.5 Sonnet - $0.2_{0.15}$ $0.0_{0.0}$ 0.2 $4.0_{0.33}$ $2.37_{0.23}$
1.63
Gemini 1.5 Pro - $0.7_{0.26}$ $0.5_{0.23}$ 0.2 $11.82_{0.51}$ $11.75_{0.44}$
0.07
GPT-4o - $\boldsymbol{2.2_{0.47}}$ $\boldsymbol{1.8_{0.4}}$ 0.4
$\boldsymbol{22.72_{0.67}}$ $\boldsymbol{22.89_{0.65}}$ -0.17
GPT-4 Turbo - $0.0_{0.0}$ $0.2_{0.13}$ -0.2 $8.58_{0.3}$ $6.87_{0.28}$ 1.72
Qwen-VL-Max - ${0.89_{0.06}}$ ${1.38_{0.1}}$ -0.49 ${5.4_{0.19}}$
${12.29_{0.18}}$ -6.89
Reka Core - $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.35_{0.23}$ $2.97_{0.2}$ 0.38
Open-source models
CogVLM2-Chinese 19B $\boldsymbol{1.2_{0.07}}$ $\boldsymbol{2.3_{0.09}}$ -1.1
$\boldsymbol{16.83_{0.22}}$ $\boldsymbol{19.86_{0.23}}$ -3.04
CogVLM2-Chinese-FT 19B $\underline{42.51_{0.32}}$ $\underline{45.91_{0.23}}$
-3.39 $\underline{65.79_{0.24}}$ $\underline{69.46_{0.46}}$ -3.68
DeepSeek-VL 1.3B $0.0_{0.0}$ $0.0_{0.0}$ 0 $6.87_{0.09}$ $3.53_{0.07}$ 3.33
DeepSeek-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $5.49_{0.07}$ $7.57_{0.05}$ -2.08
DocOwl-1.5-Omni 8B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.68_{0.04}$ $4.42_{0.07}$ -2.73
Monkey 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $5.69_{0.15}$ $6.3_{0.13}$ -0.61
InternLM-XComposer2-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $8.36_{0.09}$
$7.92_{0.09}$ 0.44
InternVL-V1.5 25.5B $0.0_{0.0}$ $0.0_{0.0}$ 0 $7.9_{0.12}$ $6.11_{0.26}$ 1.79
MiniCPM-V2.5 8B $0.2_{0.03}$ $0.2_{0.01}$ 0 $7.23_{0.18}$ $7.6_{0.13}$ -0.37
MiniCPM-V2.5-FT 8B $1.2_{0.03}$ $1.4_{0.06}$ -0.2 $18.01_{0.35}$
$15.25_{0.25}$ 2.76
Qwen-VL 7B $0.0_{0.0}$ $0.0_{0.0}$ 0 $1.1_{0.07}$ $0.15_{0.01}$ 0.94
Yi-VL 34B $0.0_{0.0}$ $0.0_{0.0}$ 0 $4.49_{0.09}$ $1.73_{0.1}$ 2.76
Yi-VL 6B $0.0_{0.0}$ $0.0_{0.0}$ 0 $3.95_{0.05}$ $2.08_{0.09}$ 1.87

We show the table of evaluation results on first 100 and 500 test cases for
better comparison with human evaluation results and closed-source models
correspondingly.