# Fully Authentic Visual Question Answering Dataset from Online Communities

Chongyan Chen1, Mengchen Liu2, Noel Codella 2, Yunsheng Li2 , Lu Yuan 2 ,
Danna Gurari 1,3
$~{}^{1}$ University of Texas at Austin $~{}^{2}$ Microsoft$~{}^{3}$
University of Colorado Boulder

###### Abstract

Visual Question Answering (VQA) entails answering questions about images. We
introduce the first VQA dataset in which all contents originate from an
authentic use case. Sourced from online question answering community forums,
we call it VQAonline. We then characterize our dataset and how it relates to
eight other VQA datasets. Observing that answers in our dataset tend to be
much longer (e.g., with a mean of 173 words) and thus incompatible with
standard VQA evaluation metrics, we next analyze which of the six popular
metrics for longer text evaluation align best with human judgments. We then
use the best-suited metrics to evaluate six state-of-the-art vision and
language foundation models on VQAonline and reveal where they struggle most.
To facilitate future extensions, we publicly share the dataset at:
https://vqaonline.github.io/.

##  1 Introduction

We are witnessing a renewed excitement in visual question answering (VQA), the
task of answering questions about images, with recent successes of large
foundation models in _zero-shot settings_. Moreover, many papers are reporting
that such models can achieve or even exceed human performance [54, 37, 44].

While modern VQA datasets enabled this exciting progress in VQA model
development, they have two key limitations. First, most publicly-available VQA
datasets [17, 49, 46] are artificially-constructed to probe model performance
rather stemming from authentic use cases. Yet, repeatedly, prior work [19, 57,
20] has revealed that models developed for artificially-constructed settings
generalize poorly to authentic use cases due to significant domain shifts. The
second key limitation of existing datasets lies in a limited diversity of
content representing authentic use cases. The only VQA dataset reflecting an
authentic use case is VizWiz-VQA [18]. However, VizWiz-VQA is constrained to
one population, specifically blind individuals seeking assistance in learning
about their visual surroundings. A further limitation of the dataset is that
the answers were collected years later to support model benchmarking, rather
than having been validated as acceptable by those who asked the questions.

Figure 1:  Shown is an example from our VQAonline dataset alongside an example
from each of three related VQA datasets [17, 49, 46] to illustrate key
distinctions. VQAonline is the first to originate from an _authentic_ VQA use
case end to end, including with authentic context and answers, as well as the
first VQA dataset sourced from an online question answering community. These
distinctions result in many unique properties, including lengthy answers.

We address these limitations by introducing a new dataset called _VQAonline._
Sourced from online question answering communities, it is the the first VQA
dataset with all contents originating from an authentic use case. We conduct
analysis to highlight how this dataset differs from prior datasets; examples
include that it contains: (1) authentic context that clarifies the question,
(2) an answer the individual asking the question validated as acceptable from
all community-provided answers, (3) answers that are considerably longer
(e.g., a mean of 173 words versus typically 11 words or fewer in prior work),
and (4) user-chosen topics for each visual question from 105 diverse topics
revealing the dataset’s inherent diversity. Such differences are exemplified
in Figure 1. Next, observing that previous VQA evaluation metrics are ill-
suited for lengthy textual answers, we assess which related evaluation metrics
for long-form text best align with human judgement for our dataset. Using
these metrics, we then benchmark the performance of six modern foundation
models in the zero-shot setting. Our findings reveal that modern models
perform poorly overall and our fine-grained analysis reveals opportunities for
how to improve such models.

We expect improved models designed for VQAonline can also benefit many other
real-world applications including other community question answering platforms
(e.g., Yahoo! Answers, Wiki Answers, Quora), learning assistance VQA tools for
children [22], virtual assistants, visual dialog and query, as well as visual
search and browsing. To encourage community-wide progress, we publicly-share
the dataset at: https://vqaonline.github.io/.

##  2 Related Work

##### Visual Question Answering Datasets.

A commonality of all VQA datasets (e.g., [1, 17, 19, 49, 27, 25, 11]) is the
inclusion of artificially-generated content. The closest dataset to reflect an
authentic use case is VizWiz-VQA [19], as the visual questions were all
submitted by individuals seeking assistance to learn about their visual
surroundings. However, the answers were crowdsourced and never shared with or
validated by those asking the questions. To our knowledge, our work is the
first to provide an authentic dataset end-to-end, with each example including
a natural language question, image, context, and answer validated by the
individual asking the question. We conduct extensive analysis to reveal unique
aspects of our dataset (Section 3.2).

Our work also aligns with VQA datasets providing context that supplements each
visual question: Context-VQA [42] and ScienceQA [46]. However, _context_ in
Context-VQA [42] is defined as the kind of website the image is sourced from
rather than our definition of supplementary data that supports answering the
visual question. More similar to our work is ScienceQA [46], however the
context is both artificially-generated and has distinct compositions and
content, as discussed in Section 3.2.

##### Community Question Answering (CQA) Datasets.

Numerous datasets have been developed around CQA. However, all focus on text-
based question answering [50] or natural language questions with multimedia
answers [43]. To our knowledge, our work is the first to introduce a community
multimodal question answering dataset specifically for visual question
answering.

##### VQA Models.

Traditional VQA models generate short answers [52] and do not consider extra
context to help answer the questions. Yet, our dataset features answers that
are lengthy, typically containing multiple sentences, and provides additional
context that can help answer visual questions. Accordingly, we benchmark large
Vision and Language Models (VLMs) [12, 32, 44, 55, 59, 35] which, by their
generative design, are better-suited to generate long-form answers using their
strong natural language understanding capabilities. Overall, we find these
models perform poorly and provide fine-grained analysis to reveal
opportunities for improving such models.

##  3 VQAonline Dataset

We now introduce our VQAonline dataset originating from an authentic use case
where people used a web-based platform to post and answer visual questions.

###  3.1 Dataset Creation

##### Source.

We created our dataset by scraping data from the online question-answering
platform Stack Exchange. We used the publicly-available data that spanned from
August 2010 to November 2022, and adhered to all data license restrictions
(cc-by-sa 4.0) set forth by Stack Exchange.

Figure 2: Number of visual questions per topic (in log scale). The colors, as
defined in the legend, indicate the super-category for each topic.

From the Stack Exchange archive, we collected all examples that contained each
the following four components: a natural language question, a natural language
context, an image, and at least one answer. Originally, this information was
collected from Stack Exchange users via four user entry interactions. The
natural language question was collected as the post "title", using a text
entry field paired with instructions "be specific and imagine you’re asking a
question to another person" when writing the question. The image was collected
as a URL link. The natural language context was provided via a "body" text
entry field that supported inline images (shared via a URL link) with users
instructed to supply "all the information someone would need to answer your
question". These three pieces of information were then shared publicly, after
which users on the site could publicly provide answers.111Stack Exchange users
could either propose edits to existing answers, which had to then be approved
by high-reputation users, or could propose new answers. Further community
deliberation to arrive at high-quality answers could occur when users posted
comments under the question or answers. The mean number of answers per visual
question is 1.45.  Finally, the individuals who posted the visual questions
could mark which answer was the “accepted answer"; we only included examples
for which the individuals did this.

Of note, the Stack Exchange platform has two key mechanisms that contribute to
high quality VQAs. First, the platform enforces a stringent reputation system
with moderation policies, as documented at
https://meta.stackexchange.com/help. Second, it features topic-specific sites
for users to engage with a relevant community; e.g., music.stackexchange.com
and gardening.stackexchange.com. We used the data from 105 topical sites that
belong to five super-categories, with 46 about culture & recreation (e.g.,
travel, sports, Christianity, French language), 25 about life & arts (e.g.,
physical fitness, parenting, pets), 24 about science (e.g., physics,
chemistry, biology), 7 about being professional, and 3 about business.222We
excluded the 75 technology topics (e.g., Stack Overflow, LaTeX), because they
typically are language problems disguised as VQA problems. In particular, the
questions typically ask about code shown in screenshots. All topics are shown
in Figure 2.

##### Filtering.

We started with 331,000 candidate VQAs from the source and removed visual
questions without accepted answers. Then, we conducted two filtering steps to
obtain our final dataset with 64,696 visual questions. First, we removed low-
quality VQAs based on Stack Exchange’s feature that enables users to up-vote
or down-vote each post (either question or answer). Each up-vote added 1 to
the score and each down-vote subtracted 1 from the score. We removed visual
questions with scores 0 or less for either the question or answer. Second, we
excluded visual questions with multiple images (e.g., GIF) or with visual
answers.

##### Dataset Splits.

While our work is inspired by the zero-shot setting, we designed our dataset
to support few-shot settings given the recent exciting developments around in-
context few-shot learning with foundation models. To do so, we randomly
selected from each of the 105 topics covered in our dataset 7 examples for
training and 3 examples for validation for all topics with at least 20
examples. The result is 665 examples in the training set, 285 in the
validation set, and 63,746 examples in the test set.

###  3.2 Dataset Analysis

VQA Dataset | $\overline{Q}$ | $\overline{A}$ | $N_{\text{img}}$ | # Auth. Topics | Context | Auth. VQ | Auth. Context | Vld. A
---|---|---|---|---|---|---|---|---
VQAonline (Ours) | 9.3 | 173.2 | 64.7K | 105 | ✓ | ✓ | ✓ | ✓
Context-VQA [42] | 6.5 | 11.0 | 42 | ✗ | ✗ | ✗ | ✗ | ✗
VQAv2 [17] | 6.1 | 1.2 | 204.7K | ✗ | ✗ | ✗ | ✗ | ✗
VizWiz-VQA [19] | 6.7 | 1.7 | 20.5K | ✗ | ✗ | ✓ | ✗ | ✗
OK-VQA [38] | 8.1 | 1.3 | 14K | ✗ | ✗ | ✗ | ✗ | ✗
DocVQA [39] | 9.5 | 2.4 | 12.8K | ✗ | ✗ | ✗ | ✗ | ✗
ScienceQA [46] | 12.11 | 4.4 | 6.5K | 26 | ✓ | ✗ | ✗ | ✗
InfographicVQA [40] | 11.5 | 1.6 | 5.5K | ✗ | ✗ | ✗ | ✗ | ✗
INFOSEEK [11] | 8.9 | 1.5 | 1.36M | ✗ | ✗ | ✗ | ✗ | ✗

Table 1: Characterization of existing VQA datasets and our new VQAonline in
terms of mean question length (i.e., $\overline{Q}$), mean answer length
(i.e., $\overline{A}$), number of images (i.e., $N_{\text{img}}$), number of
authentic topics (i.e., # Auth. Topics), inclusion of context, inclusion of
authentic visual questions (i.e., Auth. VQ), inclusion of authentic context,
and inclusion of answers validated by those asking the questions (i.e., Vld.
A)).

####  3.2.1 Comparison to Existing Datasets

##### Setup.

We start by contextualizing how our dataset compares to the current focus of
the research community. To do so, we compare VQAs in our dataset to those in
eight existing popular VQA datasets. We selected VQAv2 [17] because it is the
most popular (e.g., highly cited) VQA dataset, Context-VQA [42] and ScienceQA
[46] 333We focus on the subset of ScienceQA with both image and context.
because they similarly claim to provide context, VizWiz-VQA [19] because it
similarly originates from an authentic use case, INFOSEEK [11] because it
similarly centers on information seeking rather than probing an AI model, OK-
VQA [38] because it similarly requires domain-specific knowledge, as well as
DocVQA [39] and Infographic VQA [40] because they similarly contain VQAs with
documents and infographics.

For each dataset, we report the mean number of words in each question, mean
number of words in each answer, number of images, number of topics which were
authentically used to label clusters of VQAs by theme, whether context is
included, whether the visual questions are authentic, whether the context is
authentic, and whether the answers are validated by those asking the
questions. Results are reported in Table 1.

Our findings reveal commonalities of our VQAonline dataset with existing
datasets. For example, we observe that the visual questions themselves are
similar in terms of the typical question length and number of images.
Specifically, questions have a mean of 9 words in our dataset compared to 6 to
11 words in other datasets and the nearly 65,000 images in our dataset are
comparable to existing datasets that typically contain tens of thousands of
examples.

Our analysis also demonstrates significant differences from existing datasets
across two key dimensions. The first dimension is the _dataset authenticity_.
Our VQAonline dataset is the only fully authentic dataset for the entire VQA
pipeline, meaning all contents originate from an authentic use case. The
second key difference of our dataset is the typical _answer length_. Answers
in our VQAonline dataset are orders of magnitude longer than in existing
datasets, with a mean length of 173 words in VQAonline versus 11 or fewer
words in existing datasets. This larger length is not anomalous, as the median
answer length in VQAonline is 120 words. This longer length in part reflects
responses are provided in paragraph form, with most commonly 3 sentences per
answer. This distinction renders existing evaluation metrics for VQA models
largely ineffective, as existing metrics assume brief responses [1]. This
issue will be addressed in Section 4.1.

####  3.2.2 Fine-Grained Analysis

We next provide a richer analysis of the questions, images, context, and
answers in our VQAonline dataset.

##### Topic.

The authentic user-based posting of each VQA within a topic community reveals
the diversity of content in our dataset. We report the number of VQAs in each
of the 105 topics in Figure 2. As shown, there can be as few as 10 examples in
a topic to over 20,000.

##### Questions.

Our VQAonline dataset is the first with authentic visual questions asked by
users of an online question-answering platform. This results in a domain shift
from the only other VQA dataset with authentically asked visual questions,
VizWiz-VQA [19], which were asked by blind users of a mobile phone
application. For instance, blind people typically asked about their physical
surroundings (e.g., "What is this?", "What does this say?") while Stack
Exchange users often asked for more fine-grained or specialized information
about the images (e.g., "What’s a better term to describe these ‘stripes’ in
blouse (Screen shot attached)", "What is the third character on this sign seen
in Taipei?"). Another distinction stems from Stack Exchange’s feature for
avoiding redundant questions by letting users remove duplicates of other
already submitted questions. The consequence is that only 0.2% of natural
language questions in our VQAonline dataset are duplicated compared to 37.9%
of questions in VizWiz-VQA.

##### Context.

Our VQAonline dataset is the first to include authentic context. The only
other VQA dataset containing context, as defined by supplementary data that
supports answering the visual question, is ScienceQA [46]. However, the
context was contrived with heuristic rules for extracting data from elementary
and high school science curricula. Consequently, context in ScienceQA is
typically narrated in third-person, while context in our VQAonline often are
narrated in first-person. Another difference is ScienceQA has a shorter
average length of the context compared to that in our dataset; i.e., a mean of
41 words, median of 28 words, and most commonly 3 sentences versus a mean of
127 words, median of 94 words, and most commonly 4 sentences. A further
difference is that the context in ScienceQA only relates to 26 science topics
while the context in our dataset spans 105 topics that extend beyond science.

##### Image Contents.

We next quantify the tendency for images to include screenshots and
infographics to reflect the tendency for visual questions to be related to the
digital realm and data analysis. We identified an image as a "screenshot" when
contents were copied digitally from a computing device and an "infographic"
when the image shows a chart, table, graph, attention map, map, or diagram. We
manually labeled 105 images sampled _from each of the 105 topics_ as well as
an independent random selection of 100 images from the _entire dataset_. We
found for the topic-sampled set that 79% are screenshots and 17% show
infographics and for the samples from the entire dataset that 84% are
screenshots and 33% show infographics. The similarly high prevalence of
screenshots across both samples underscores Stack Exchange users’ tendency to
ask questions about the digital realm. We hypothesized the different
prevalence of infographics across both samples was due to topical differences,
as 52% of all VQAs (i.e., 33,384 out of 64,696) belong to only 23% of the
topics (i.e., 24 science topics). Supporting this hypothesis, we found from
inspecting 100 random VQAs from each subset of the science topics and non-
science topics that infographics were in 58% and 6% of images respectively.

##### Importance of Images.

We next examine the value of images for answering the questions. To do so, we
flagged each image as _question-aligned_ if elements described in the question
are found in the image and _context-aligned_ if elements described in the
context are found in the image. We additionally flagged whether an image is
_necessary_ , based on whether the answer can be obtained without the image.
We manually labeled 105 examples from each of the 105 topics. We found that
91.4% of the images are _question-aligned_ and 99% of the images are _context-
aligned_. This finding contrasts that of the only other dataset with authentic
visual questions, VizWiz-VQA [19], where only 71% are deemed answerable
because blind photographers struggled to capture contents of interest in their
images. We also found that 65.7% images are _necessary_ for answering the
question in our dataset. Often, the purpose of unnecessary images was to
motivate or show scenarios in which the asker encounters the question.
Altogether, these findings underscore the value of VQA models analyzing the
images to arrive at the answers.

Figure 3: Examples of three common user intentions for visual questions in our
dataset: instruction, evidence, and verification.

##### User Intention.

We finally characterized intents behind the visual questions to inform how AI
models can better meet users’ information wants. Our work complements two
prior works that identified user intentions for three community question
answering platforms (Answerbag, Metafilter, and Yahoo! Answer) [21] and the
music StackExchange community [15]. Our work is the first to investigate user
intentions for _VQAs_. Through a multi-round analysis of the data, described
in the Supplementary Materials, we defined eight potential intents:
_instruction_ , _evidence_ , _verification_ , _advice_ , _identification_ ,
_opinion_ , _reason_ , and _other_. Examples for three of these categories are
shown in Figure 3. We then recruited vetted crowdworkers from Amazon
Mechanical Turk to label one primary intent per visual question when shown the
question, image, context, and answer. They labeled 105 examples, with one
randomly selected from each of the 105 topics. We used the majority vote label
per VQA. The resulting distribution of assigned categories is 19% are
instruction, 18% evidence, 17% verification, 16% advice, 10% identification,
10% opinion, 8% reason, and 1% other. Examples from the three most common
categories are shown in Figure 3.

##  4 Experiments

###  4.1 Evaluation Methodology

As noted in Section 3.2, the uniquely lengthy answers in our VQAonline dataset
render the mainstream VQA evaluation metric unsuitable since it assumes short
answers [1, 17]. Consequently, we first establish an appropriate evaluation
methodology. To do so, we assess which widely-used metrics for longer text
evaluation align best with human judgments of the quality of model-generated
results.

##### Candidate Evaluation Metrics.

We identified six popular evaluation metrics for longer text from related
fields, including long-form question answering and image captioning. We
primarily focused on reference-based metrics, which assess the similarity
between model-generated answers and human-generated answers, because of their
widespread use. Specifically, we chose ROUGE-L [33], METEOR [13], RefCLIP-S
[23], BERTscore [58], and the GPT-4 metric [56, 16] 444For GPT-4 metric, we
prompted GPT-4 to ”compare the ground truth and prediction from AI models and
give a correctness score for the prediction. The correctness score is 0
(totally wrong), 1 (partially correct), or 2 (totally correct).” We then
rescale GPT-4 metric to range from 0 to 1. . For completeness, we also include
the reference-free metric CLIP-S [23], which assesses compatibility between
the image and model-generated answer. All metrics are set to output values in
the range of 0 to 1.

##### Models.

We chose to evaluate the following six modern zero-shot Vision and Language
Models (VLMs) that achieved strong performance in early 2023 for various VQA
datasets and subtasks.

  * •

BLIP-2 [32]: state-of-the-art (SOTA) for zero-shot setting on four VQA
datasets, including ScienceQA and VizWiz-VQA according to LVLM-eHub, a
comprehensive evaluation benchmark for VLMs [53].

  * •

MiniGPT-4 [59]: ranked first overall in open-world VQA scenario according to
LLVM-eHub [53].

  * •

LLaVA [35]: top-two performance in the zero-shot setting for three VQA
datasets, including for DocVQA [39, 2] according to LLVM-eHub [53].

  * •

mPLUG-Owl [55]: SOTA for VQAv2 [17] in early 2023 and ranked second overall in
open-world VQA scenario according to LLVM-eHub [53].

  * •

InstructBLIP [12]: SOTA for zero-shot setting on four VQA datasets, including
OK-VQA [41, 38, 25, 3] according to LLVM-eHub [53].

  * •

GPT-4V [44]: SOTA for numerous VQA datasets, such as Infographic VQA [39] and
TVQA [30].

##### Human Judgement Collection.

We hired ten human annotators for an IRB-approved human evaluation study. Each
individual was a domain expert in one of ten topics covered in our dataset
(i.e., Physical Fitness, Botany, Music, Mechanics, Politics, Law, Chinese
Language, Economics, Statistics, and Artificial Intelligence), with at least a
Master’s degree in a field related to their assigned VQA topic. Each domain
expert was assigned 20 VQAs on their topic, and was required to rate the
performance of six model-generated answers per VQA. As a result, we collected
for 200 VQAs a total of 1200 human judgments (i.e., $10*20*6$).

For the annotation task, each domain expert was shown for each VQA the
original visual question, context, and answer alongside the six model-
generated answers in a home-grown web-based annotation tool. The task was to
rate each model-generated answer as either incorrect (i.e., score of 0),
partially correct (i.e, score of 0.5), or correct (i.e., score of 1). The
definitions for these ratings are: Correct: At least one of these two items
needs to be true for the answer to be correct: (1) Aligns with all the key
points in the reference answer. (2) The answer also makes sense based on your
best judgment, even if not included in the reference answer. Partially
Correct: The answer matches part of the key points in the reference answer
that answers the question but has errors in the content. Incorrect: The
response doesn’t include any key points from the reference answer and doesn’t
make sense.

Metric | Correlation | p-value
---|---|---
METEOR | 0.971 | 0.001
GPT-4 metric | 0.944 | 0.005
BERTscore | 0.900 | 0.015
RefCLIP-S | 0.892 | 0.017
ROUGEL | 0.874 | 0.023
CLIP-S | 0.662 | 0.152
Table 2: Correlation between human judgment and popular automated evaluation
metric scores for 220 VQAs.

##### Analysis Design and Results.

We calculated the mean score for each model across the 200 VQAs for
humans555The resulting scores are GPT-4V: 0.76, mPLUG-Owl: 0.57, LLaVA:0.5,
MiniGPT-4: 0.37, InstructBLIP:0.12, and BLIP2: 0.1. as well as using each
evaluation metric. Then we calculated the Pearson correlation between mean
scores between humans and each evaluation metric. Results are shown in Table
2.

Overall, we observe a high positive correlation between all the reference-
based metrics and human judgments with statistical significance (i.e., p-value
less than 0.05). Metrics with the strongest correlation to human judgments are
METEOR metric followed by GPT-4 metric and BERTscore. Consequently, we will
use these three top-performing metrics for our algorithm benchmarking.

For the reference-free metric CLIP-S, although the results are not
statistically significant, we found it performs the worst of all metrics. This
contrasts the finding from [23] that CLIP-S can outperform other reference-
based metrics for image captioning.

###  4.2 Model Benchmarking

We next analyze the performance of modern models on our VQAonline dataset to
reveal what makes it challenging for the research community.

##### Models and Evaluation Metrics.

We evaluate the same six models [32, 12, 55, 44, 35, 59] analyzed in Section
4.1, with the three most human-aligned evaluation metrics 666We provided the
overall results for the six evaluation metrics in supplementary materials..
However, due to GPT-based models requiring a fee for their usage [14], we only
evaluate GPT-4V on a subset of data and only use the GPT-4 metric for that
subset of data. We created this subset by randomly selecting 20 VQAs from each
of the 105 topics, except for the 17 topics containing less than 20 examples
where we used all available VQAs. The resulting subset contains 1903 VQAs. For
a fair comparison, we also report the performance of all other models on this
subset.

##### Overall Results.

Results are shown in Table 9. We observe that all models perform poorly,
especially when considering the results from the METEOR and GPT-4 metrics. In
other words, considerable progress still remains to achieve the status of some
modern models [54, 55, 44] which are purported to match or even surpass human
performance on some VQA datasets [17, 37].

When comparing across models on the whole test set, we found that mPLUG-Owl is
the best VLM model, and if on the subset, GPT-4V is the top-performing model,
followed closely by mPLUG-Owl.

| All | Subset
---|---|---
| M. | B. | M. | B. | G.
GPT-4V [44] | - | - | 0.12 | 0.76 | 0.55
mPLUG-Owl [55] | 0.12 | 0.76 | 0.10 | 0.75 | 0.40
LLaVA [35] | 0.08 | 0.75 | 0.08 | 0.75 | 0.38
MiniGPT-4 [59] | 0.09 | 0.75 | 0.08 | 0.74 | 0.32
InstructBLIP [12] | 0.05 | 0.67 | 0.06 | 0.69 | 0.30
BLIP2 [32] | 0.04 | 0.69 | 0.04 | 0.69 | 0.27
Table 3: Performance of six modern VQA models on our dataset with respect to
three evaluation metrics. (M. = METEOR, B. = BERTscore, G.= GPT-4 Metric)

##### Analysis With Respect to Input Types.

We next analyze the predictive power of each input by analyzing the
performance of ablated model variants where one input type is removed from
each: i.e., Q+C, Q+I, and C+I. Results for the two top-performing models are
shown in Table 4. Unsurprisingly, we find that the model receiving the maximum
amount of information (i.e., Q+C+I) performs the best.

We observe that the worst-performing variant across both models and all
evaluation metrics are the models lacking _context_ as input. This highlights
that current models find the context (which is typically language-based) to be
the most valuable source for arriving at an answer.

When comparing the models that lack questions as input (i.e., Q+C) with models
receiving all inputs (Q+C+I), the questions’ predictive power appears to be
nearly negligible. We suspect this is because often the information provided
by the question is included in the context. For example, a question asking
"What test am I taking, anyways?" while the part of the context includes "can
you tell me what field of math the test was on?", which overlaps with the
question.

When comparing the models lacking images as input (i.e., Q+C) with models
receiving all inputs (Q+C+I), we similarly observe that the predictive power
of the image seems very small. We suspect this result does not imply the
irrelevance of images for the task, as further validated through a small-scale
study in Section 3.2. Instead, we believe this argues for improving models so
they can better utilize images when predicting answers.

Models | METEOR | BERT. | ~GPT-4 Metric
---|---|---|---
mPLUG(Q+C+I) | 0.12 | 0.76 | 0.40
mPLUG(C+I) | 0.12 | 0.76 | 0.37
mPLUG(Q+C) | 0.11 | 0.76 | 0.36
mPLUG(Q+I) | 0.07 | 0.74 | 0.25
~GPT-4V (Q+C+I) | 0.12 | 0.76 | 0.55
~GPT-4V (C+I) | 0.11 | 0.76 | 0.55
~GPT-4V (Q+C) | 0.12 | 0.75 | 0.55
~GPT-4V (Q+I) | 0.07 | 0.74 | 0.40
Table 4: Performance of the top-two VQA models with different input types (Q+C+I, C+I, Q+C, Q+I). Models or scores labeled with "~" are evaluated on the dataset subset.  | All | Subset
---|---|---
Metric | METEOR | BERTscore | METEOR | BERTscore | GPT-4 Metric
Model | mPLUG-Owl | mPLUG-Owl | mPLUG-Owl | GPT-4V | mPLUG-Owl | GPT-4V | mPLUG-Owl |  GPT-4V
Science | 0.14 | 0.77 | 0.13 | 0.13 | 0.75 | 0.77 | 0.38 | 0.56
Life&Arts | 0.13 | 0.76 | 0.12 | 0.12 | 0.75 | 0.77 | 0.42 | 0.54
Culture&Rec. | 0.11 | 0.75 | 0.11 | 0.11 | 0.75 | 0.77 | 0.39 | 0.56
Business | 0.13 | 0.76 | 0.14 | 0.14 | 0.77 | 0.77 | 0.42 | 0.58
Professional | 0.13 | 0.76 | 0.13 | 0.13 | 0.76 | 0.77 | 0.48 | 0.57
Table 5: Results for mPLUG-Owl and GPT-4V on five super-categories. (Culture&
Rec. = Culture&Recreation.) Figure 4: Performance of the two top-performing
VQA models, mPLUG-Owl and GPT-4V, for each of 105 topics with their five
super-categories represented in 5 different colors. Results are shown with
respect to the top two human-aligned evaluation metrics: (a) METEOR and (b)
GPT-4 Metric. For visualization simplicity, we show text labels only for
topics with interesting identified trends (We omitted "language" for each
language topic; e.g., "Esperanto" instead of "Esperanto Language").  Figure 5:
Results from the GPT-4V model on three examples in our dataset from three
topics, puzzling and two language topics.

##### Analysis With Respect to VQA Topics.

We next analyzed each model’s performance per topic. First, we visualize
performance for the two top-performing models with respect to the top two
human-aligned evaluation metrics, METEOR and GPT-4 metric, in Figure 4. Scores
associated with GPT-4V model or GPT-4 metric are reported on the subset and
otherwise are on the entire test set. We also report performance with respect
to the five super-categories in Table 5.

One common outlier for the per-topic visualization (i.e., Figure 4) is
consistently very low performance for the _puzzling_ topic across both
evaluation metrics. We attribute this to such topics requiring jointly strong
vision understanding and reasoning skills. An example of the _puzzling_ topic
with GPT-4V is shown in Figure 5, Example 1.

Another trend we observe for the per-topic visualization (i.e., Figure 4) is
different performance for topics related to different languages. The models
work best on visual questions related to Germanic Languages; e.g., English as
exemplified in Figure 5, Example 2. The next best languages are Romance
Languages (e.g., Spanish, French, Italian, Portuguese), then East Asian
Languages (e.g., Chinese, Korean), and finally Russian. We suspect causes of
such consistent performance discrepancies across languages are: (1) Data
availability: languages which have fewer datasets and less diversity (e.g.,
one exemplified in Figure 5, Example 3) perform worse than those with greater
diversity and abundance of datasets (e.g., English), (2) Language Complexity:
Languages with simpler grammar (e.g., Esperanto) or familiar grammatical
structures (e.g., Romance languages for an English-trained model) are easier
for models to learn, and (3) Text recognition can be more challenging for
languages with unique scripts, such as Chinese or Korean while these language-
related VQAs often pertain to text recognition. We believe these findings shed
light on future research directions for multilingual VQA and text VQA
research.

With respect to the five super-categories (i.e., Table 5), overall across all
evaluation metrics, the models perform worst on Culture & Recreation. We
suspect this is because such visual questions require a higher reliance on
vision understanding, an area where prevailing VLMs fail.

##  5 Conclusions

We introduce the first fully authentic end-to-end VQA dataset. Our analysis
reveals numerous distinct features of our new dataset, most importantly
atypically long answers. As a result, we conducted human-based studies to
establish a new VQA evaluation methodology for our dataset and found that
well-suited metrics are GPT-4 metric, METEOR, and BERTscore. Subsequent
benchmarking of six modern VQA models in zero-shot settings reveals they all
perform poorly. Our fine-grained analysis across the 105 topics covered in the
dataset reveals some topics are easier (e.g., English-based and Computer
Science-based topics) while other topics are very challenging (e.g.,
puzzling). To facilitate extensions of this work, we publicly share our
dataset at https://vqaonline.github.io/.

## References

  * Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433, 2015.
  * Biten et al. [2022] Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and R Manmatha.  Latr: Layout-aware transformer for scene-text vqa.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16548–16558, 2022.
  * Bitton-Guetta et al. [2023] Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz.  Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2616–2627, 2023.
  * Bolotova et al. [2022] Valeriia Bolotova, Vladislav Blinov, Falk Scholer, W Bruce Croft, and Mark Sanderson.  A.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1196–1207, 2022.
  * Brady et al. [2013] Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P Bigham.  Visual challenges in the everyday lives of blind people.  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 2117–2126, 2013.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset, 2022.
  * Cambazoglu et al. [2021] B Barla Cambazoglu, Leila Tavakoli, Falk Scholer, Mark Sanderson, and Bruce Croft.  An intent taxonomy for questions asked in web search.  In _Proceedings of the 2021 Conference on Human Information Interaction and Retrieval_ , pages 85–94, 2021.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568, 2021.
  * Chen et al. [2012] Long Chen, Dell Zhang, and Levene Mark.  Understanding user intent in community question answering.  In _Proceedings of the 21st international conference on world wide web_ , pages 823–828, 2012.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chen et al. [2023] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang.  Can pre-trained vision and language models answer visual information-seeking questions?  _arXiv preprint arXiv:2302.11713_ , 2023.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Elliott and Keller [2013] Desmond Elliott and Frank Keller.  Image description using visual dependency representations.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1292–1302, 2013.
  * Exchange [2023] Stack Exchange.  Gpt-4v pricing, 2023.  Accessed: 2023-11-16.
  * Fu and Fan [2016] Hengyi Fu and Yun Fan.  Music information seeking via social q&a: An analysis of questions in music stackexchange community.  In _Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries_ , pages 139–142, 2016.
  * Gilardi et al. [2023] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _arXiv preprint arXiv:2303.15056_ , 2023.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Gurari and Grauman [2017] Danna Gurari and Kristen Grauman.  Crowdverge: Predicting if people will agree on the answer to a visual question.  In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ , pages 3511–3522, 2017.
  * Gurari et al. [2018] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3608–3617, 2018.
  * Gurari et al. [2020] Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya.  Captioning images taken by people who are blind.  _arXiv preprint arXiv:2002.08565_ , 2020.
  * Harper et al. [2010] F Maxwell Harper, Joseph Weinberg, John Logie, and Joseph A Konstan.  Question types in social q&a sites.  _First Monday_ , 2010.
  * He et al. [2017] Bin He, Meng Xia, Xinguo Yu, Pengpeng Jian, Hao Meng, and Zhanwen Chen.  An educational robot system of visual question answering for preschoolers.  In _2017 2nd International Conference on Robotics and Automation Engineering (ICRAE)_ , pages 441–445. IEEE, 2017.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In _EMNLP_ , 2021.
  * Hu et al. [2023] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang.  Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities.  _arXiv preprint arXiv:2302.11154_ , 2023.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6700–6709, 2019.
  * Ignatova et al. [2009] Kateryna Ignatova, Cigdem Toprak, Delphine Bernhard, and Iryna Gurevych.  Annotating question types in social q&a sites.  In _Tagungsband des GSCL Symposiums ‘Sprachtechnologie und eHumanities_ , pages 44–49, 2009.
  * Johnson et al. [2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2901–2910, 2017.
  * Kofler et al. [2016] Christoph Kofler, Martha Larson, and Alan Hanjalic.  User intent in multimedia search: a survey of the state of the art and future challenges.  _ACM Computing Surveys (CSUR)_ , 49(2):1–37, 2016.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123(1):32–73, 2017.
  * Lei et al. [2018] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg.  Tvqa: Localized, compositional video question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1369–1379, 2018.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_ , 2023a.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023b.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Liu et al. [2023a] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  _arXiv preprint arXiv:2306.14565_ , 2023a.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023b.
  * Liu [2015] Zhe Liu.  Understanding and modeling user behavior in social question and answering.  2015\.
  * Lu et al. [2023] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  _arXiv preprint arXiv:2310.02255_ , 2023.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Mathew et al. [2021] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.  Docvqa: A dataset for vqa on document images.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 2200–2209, 2021.
  * Mathew et al. [2022] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar.  Infographicvqa.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 1697–1706, 2022.
  * Mishra et al. [2019] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In _2019 international conference on document analysis and recognition (ICDAR)_ , pages 947–952. IEEE, 2019.
  * Naik et al. [2023] Nandita Naik, Christopher Potts, and Elisa Kreiss.  Context-vqa: Towards context-aware and purposeful visual question answering.  _arXiv preprint arXiv:2307.15745_ , 2023.
  * Nie et al. [2011] Liqiang Nie, Meng Wang, Zhengjun Zha, Guangda Li, and Tat-Seng Chua.  Multimedia answering: enriching text qa with media information.  In _Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval_ , pages 695–704, 2011.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023.
  * Ordonez et al. [2011] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  _Advances in neural information processing systems_ , 24, 2011.
  * Saikh et al. [2022] Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya.  Scienceqa: a novel resource for question answering on scholarly articles.  _International Journal on Digital Libraries_ , 23(3):289–301, 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, 2018.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8317–8326, 2019.
  * Srba and Bielikova [2016] Ivan Srba and Maria Bielikova.  A comprehensive survey and classification of approaches for community question answering.  _ACM Transactions on the Web (TWEB)_ , 10(3):1–63, 2016.
  * Toba et al. [2014] Hapnes Toba, Zhao-Yan Ming, Mirna Adriani, and Tat-Seng Chua.  Discovering high quality answers in community question answering archives using a hierarchy of classifiers.  _Information Sciences_ , 261:101–115, 2014.
  * Wu et al. [2017] Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel.  Visual question answering: A survey of methods and datasets.  _Computer Vision and Image Understanding_ , 163:21–40, 2017.
  * Xu et al. [2023] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo.  Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.  _arXiv preprint arXiv:2306.09265_ , 2023.
  * Yang et al. [2023] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  _arXiv preprint arXiv:2309.17421_ , 9, 2023.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality, 2023.
  * Yu et al. [2023] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _arXiv preprint arXiv:2308.02490_ , 2023.
  * Zeng et al. [2020] Xiaoyu Zeng, Yanan Wang, Tai-Yin Chiu, Nilavra Bhattacharya, and Danna Gurari.  Vision skills needed to answer visual questions.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(CSCW2):1–31, 2020.
  * Zhang et al. [2019] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

##  6 Supplementary Material

VQA Dataset | Which Images? | Who Asked? | From User?
---|---|---|---
Our dataset | StackExchange Users | StackExchange Users | ✓
Context-VQA [42] | Six types of websites | Annotators | ✗
VQAv2 [17] | MSCOCO | Crowd workers (AMT) | ✗
VizWiz-VQA [19] | Captured by Blind people | Blind people | ✓
OKVQA [38] | MSCOCO | Crowd workers (AMT) | ✗
DocVQA [39] | UCSF Industry Documents Library | Remote workers | ✗
ScienceQA [46] | Online K-12 learning platform | Online K-12 learning platform | ✗
InfographicVQA [40] | Bing and Google Image Search | Annotators | ✗
INFOSEEK-Wikidata [11] | 9 datasets following OVEN | Template-based auto-generation | ✗
INFOSEEK-Human [11] | 9 datasets following OVEN | Annotators | ✗

Table 6: Comparison of visual questions from eight existing VQA datasets and
our new VQAonline dataset regarding image and question sources. Like OVEN
[24], INFOSEEK sources images from nine image classification and retrieval
datasets.

This document supplements the main paper with more information about:

  1. 1.

Dataset Collection (Supplements Section 3.1)

  2. 2.

Dataset Analysis (Supplements Section 3.2)

  3. 3.

Algorithm Benchmarking (Supplements Section 4.2)

##  I Dataset Collection

###  I.1 Dataset Source and Filtration

The Stack Exchange data is hosted at https://archive.org/details/stackexchange
in XML format. We started with 331,000 candidate visual questions. After
removing visual questions without an accepted answer, it resulted in 165,766
visual questions. As mentioned in the main paper, we then conducted two
filtering steps. After removing visual questions with scores of 0 or less for
either the question or answer, we had 119,177 visual questions. Next, after
removing visual questions with multiple images, we had 85,573 visual
questions. Subsequently, removing visual questions with visual answers, left
65,849 visual questions. Examples of filtered visual questions with visual
answers and multiple images are shown in Figures 6 and 7 respectively.
Finally, after removing examples for which the image could not be downloaded
from the provided link, we got to our final dataset of 64,696 visual
questions. For data format consistency, we converted all images to png format.

Figure 6: An example of a visual question with a "visual answer". Figure 7: An
example of a visual question with multiple images. The answer is omitted to
save space.

###  I.2 User Intention Taxonomy and Definitions.

User Intention Taxonomy. We first brainstormed 11 free-form user intention
categories. To do so, we solicited help from GPT-3.5 by having it indicate via
a zero-shot prompt the user intentions when given a question and context. We
identified an initial 11 free-form user intention categories, which are shown
in the second column of Table 7. We then refined these categories based on the
frequency of their occurrence in the literature on user intention (shown in
the third to the last column in Table 7) for related fields such as question
answering and search queries [4, 7, 26, 21, 51, 9, 28, 36] to the following 7
categories: advice, evidence, identification, instruction, opinion, reason,
and verification. We then drafted the definitions for each taxonomy, mainly by
adapting definitions from [7, 4, 21].

We then finalized the taxonomy with definitions via four rounds of annotator
analysis on a total of 105 visual questions. Specifically, two annotators
(authors) tagged the category for each visual question. Any disagreements that
arose were discussed and resolved, and the definitions associated with each
taxonomy category were adjusted accordingly. We observed slight agreement
accuracy 777The agreement accuracy is calculated by dividing the number of
agreements between two annotators by the total number of examples annotated.
improvement and Cohen’s kappa agreement improvement after each round.
Specifically, from round 1 to round 4, agreement accuracy improved from 53.3%
to 87.5% and Cohen’s kappa agreement improved from 45.24% to 84.69%. This
process culminated in our final taxonomy and their definition, which includes
the following eight categories mentioned in the main paper: _advice, evidence,
identification, instruction, opinion, reason, verification_ , and _other_.

As shown in Table 7, our final taxonomy is different from [4, 7, 26, 21, 51,
9, 28, 36] as they focus solely on text-based question or search queries
whereas our work centers on visual questions. [5] also explores user
intentions for authentic visual questions for blind individuals. However,
blind people’s user intents of visual questions significantly differ from that
of online users’ visual questions. In fact, only one of the user intents
identified for blind individuals (i.e., identification) overlaps with the user
intents we have identified for our dataset.

##### Final user intention taxonomy and definitions.

We show the final user intention taxonomy and definitions as follows:

  * •

Verification: These fact-checking questions aim to confirm or refute a
hypothesis through affirmative answers (yes/no or multiple choice options).
Since this question is based on fact, it cannot include subjective questions
which should be classified as Opinion.

  * •

Identification: The expected answer for these questions is a named entity or
object identification. The answer is objective.

  * •

Reason: These questions require answers that explain causes underlying a
particular action or event. Most "why" questions fall into this category.

  * •

Evidence-based: This category primarily includes questions that ask for the
definition, features, description, or process of a concept, idea, object,
event, or one of its attributes. It also covers translation questions of
seeking the meaning of a sentence or word. If a question can be classified as
Verification, it should not be classified as Evidence-based.

  * •

Instruction: These questions typically involve "how to do" inquiries, seeking
instructions, guidelines, or procedures for a specific action in real life.

  * •

Advice: This category includes questions where the user seeks personalized
guidance on a specific topic. Advice questions differ from Instruction
questions in that they expect subjective recommendations, while Instruction
questions seek objective, step-by-step processes for performing a specific
action. Advice questions may also involve finding resources or seeking better
translations of sentences/words. Additionally, this category can include
questions where users are looking for ideas or comments on how to improve an
existing solution. If a question can be classified as Instruction, it should
not be categorized as Advice.

  * •

Opinion: These questions aim to elicit subjective opinions on a topic of
interest (e.g., "what do you think about" or "is X good/bad"). It might
include religious questions. This category excludes Advice questions, where
the focus is on the user asking the question.

  * •

Other: Other.

###  I.3 User Intention Annotations.

##### Hiring Crowdworkers.

We hired three crowdworkers from Amazon Mechanical Turk to perform our task.
For quality control, we only accepted workers located in the United States who
had completed more than 500 Human Intelligence Tasks (HITs) with over a 95%
approval rating.

Ours | free-form intent | Ignatova (2009) [26] | Harper (2010)[21] | Toba (2014) [51] | Cambazoglu(2021) [7] | Bolotova (2022) [4] | Fu (2016) [15]
---|---|---|---|---|---|---|---
Instruction | How | Procedural | Prescriptive | Proceure | Process | Instruction | Instruction
Evidence-based | Comprehend fact | General Info need | Factual | Factoid | Description | Evidence-based | Factual
Reason | Why | Causal | - | Reason | Reason | Reason | -
Verification | Validate | Verification | - | Yes/No | Verification | Debate | -
Opinion | - | Disjunctive | Disapproval/Quality | Opinion | Opinion | - | Opinion
Identification | Recognize | Concept completion | Identification | - | Entity | - | Identifying resources
Advice | Advise | - | Advice | - | Advice | Experience | Recommend/Solution
- | Prove | Quantification | - | - | Quantity | - |
- | Compare | Comparison | - | - | - | Comparison |
- | - | Definition | - | Definition | - | - |
- | Find | - | - | - | Resource | - |
- | Language | - | - | - | Language | - |
- | - | - | - | - | Temporal | - |
- | - | - | - | - | Calculation | - |
- |  | - | - | - | Attribute | - |
- | - | - | - | - | List | - |
- | - | - | - | - | Weather | - |
- |  | - | - | - | Location | - |
| Explain | - | - | - | - | - |
- | - | - | - | - | - | - | Request research
Other | - | - | - | - | - | - |

Table 7: User intent taxonomy for our dataset, first round free-form intent
(Bootstrapping from GPT-3.5 and Manual Labeling) and existing research. Except
that ours is user intention taxonomy for visual question answering and [7] for
Bing query search, others [21, 4, 51, 26, 15] are for community question
answering (text-only).

##### Annotation Task Design and Collection.

We provided instructions, taxonomy definitions, and two examples per user
intention category. The annotation task interface showed for each visual
question the question, context, and answer.

To facilitate collecting high-quality results, we provided each crowdworker
with one-on-one Zoom training. We also provided a qualifying annotation test
that all workers passed to verify they understood the instructions.

To enable the assessment of high-quality results, we collected two user
intention annotations from two independent workers (worker A and B) per VQA
instance (i.e. image-question-context-answer) for 105 randomly selected VQAs,
one from each of 105 topics. We then detected if their annotations matched. We
found that 25 out of 105 didn’t match. Thus, we hired a third worker (worker
C) to break the tie by instructing that individual to select one of the two
provided user intentions. We paid $40 Amazon Gift Card to workers A and B and
$5 Amazon Gift Card to worker C.

##### Instructions for User Intention Annotation.

The following are the instructions provided for workers A and B:

  * •

Read definitions and two examples for each category.

  * •

We will present to you 105 questions. Each question contains a question,
context, and a reference answer. Please read them and categorize the primary
user intention according to step 1.

  * •

If there are any non-English questions, or if the questions contain many
proper nouns that are difficult to interpret, you can use a translation tool
to translate them into your native language.

  * •

Fill in the user intent category for each question in the provided
spreadsheet.

##  II Dataset Analysis

###  II.1 Comparison of VQAonline and Other Datasets

We supplement the main paper by comparing the sources of the visual questions
for eight datasets in Table 6. Only our dataset and VizWiz-VQA dataset are
sourced from authentic use cases with both the images and visual questions
coming from real users.

Model | Model Configuration | Img-text Data | Visual Instruction Data
---|---|---|---
Models | Img Encoder | Lang Encoder | Adapter | Source | Size | Source | Size
BLIP2 [32] | ViT-g /14 | FlanT5XL | Q-Former |  CC∗, VG, SBU, L400 | 129M | - | -
MiniGPT-4 [59] | BLIP2-VE | Vicuna 13B | FC layer | CC, SBU, L400 | 5M | CC+ChatGPT | 3.5K
LLaVA [35] | ViT-L/14 | LLaMA 13B | FC layer | CC3M | 595K | LLaVA-I | 158K
mPLUG-Owl [55] | ViT-L/14 | LLaMA 7B | LoRA |  CC∗, CY, L400 | 204M | LLaVA-I | 158K
InstructBLIP [12] | ViT-g/14 | FlanT5XL | Q-Former | - | - |  QA∗ | 16M
GPT-4V [44] | - | - | - | - | - | - | -
Table 8: Details about the six benchmarked models’ model configuration and
training data. CC∗ comprises datasets from COCO [10], CC3M [48], and CC12M
[8]. CC stands for Conceptual Caption [48, 8]; VG stands for Visual Genome
[29]; CY stands for COYO-700M [6]; L400 stands for LAION 400M [47]; SBU [45]
contains 1 million images with captions. LLaVA-I stands for 158K multimodal
instruction-following data in LLaVA [35]. QA∗ stands for 13 question-answering
datasets in InstructBLIP [12].

##  III Algorithm Benchmarking

###  III.1 Algorithm Architecture

Details about each of the six benchmarked models are provided in Table 8.
Specifically, we report each model’s image encoder, language encoder, adapter,
and their training data [53, 44, 59, 12, 35, 55, 32]. Due to computational
constraints, as we only have V100 GPU, we only evaluated FlanT5XL as the
language encoder for BLIP2 and InstructBLIP, though other larger language
encoders are available (i.e., FlanT5XXL for BLIP2 and Vicuna-13B-1.1 for
InstructBlip).

###  III.2 Experiments with Domain Experts

##### Domain Expert Hiring.

We first manually selected 20 candidate topics for human evaluation. Then
based on expert availability, we hired ten domain experts to represent each of
ten fields (topics). To guarantee English proficiency, as most of our visual
questions are written in English, we only accepted experts located at the
United States.

##### Annotation Task Interface.

We show a screenshot of the annotation instructions in Figure 8 and task
interface in Figure 9. The link to the code for our web-based annotation tool
is available at https://github.com/VQAonline/VQAonlineVisualization.git.

##### Data Collection Quality Control Mechanism.

We provided a one-on-one training session with each domain expert to introduce
the task and instructions. Afterwards, we gave each expert our contact
information so that they could contact us with any questions about their tasks
and receive feedback quickly. We also inspected the time each expert spent on
each visual question as a proxy for assessing whether the hired domain experts
were finishing the task by quickly selecting random options. The median and
mean times are 1.4 minutes and 1.29 minutes respectively for assessing each
model answer per visual question. We compensated each domain expert with a $75
Amazon Gift Card, resulting in an average hourly wage of $26/hour.

##### Correct, Partially Correct, and Incorrect Examples From Expert
Evaluation.

We show examples of expert annotations for the six benchmarked models in
Tables 10, 11, 12, 13, 14, 15, spanning those that are correct (10), partially
correct (Table 11 and Table 12), and incorrect (13, 14, 15).

One mechanics’s example shows where most of the models can answer correctly
for a closed-ended visual question related to recognition (Table 10).

Two economics examples show where the models partially fail for analyzing an
_infographic_ and requiring _specific domain knowledge_ (Table 11 and Table
12). These highlight that models labeled "partially correct" can occur because
of correctly answered closed-ended questions with incorrect explanations or
insufficient explanations matching the reference answer. For example, GPT-4V,
mPLUG-OWl, MiniGPT-4, and LLaVA answer "yes" but don’t match any key points of
the reason in the reference answer and have factual errors in their given
reasons 888Of note, the errors in all models’ answers to the economic VQ
examples are highlighted in red by an economic Ph.D. student. . InstructBLIP
answered "yes" without providing any reason, but simply rephrased the context.
For each example, key points in the reference answer were either
unsatisfactorily conveyed in the model’s answer or were altogether absent.

We also show examples where models gave incorrect answers from a failure to
handle the conflict in the question, context, and image. These are shown in
Tables 13, 14, and 15. In Table 13 and 14, while the context and question
provided incorrect information by indicating the plant has mold and asking how
to remove the mold, the ground-truth answer corrects the questioner saying
it’s not mold but moss/algae so there is no need to remove it. All the models
trusted the natural language context and question more than the image and then
thus were misled to answer the question (partially) incorrectly999Among them,
GPT-4V and LLaVA are partially correct as they each matched one key point with
the reference answers: The method 2 mentioned by GPT-4V can also remove
moss/algae; LLaVA mentioned the moss (though LLaVA identified as mold) might
not be harmful.  by offering methods to remove mold101010Of note, models’
answers to the Gardening & Landscaping VQ examples are verified with a Botany
Ph.D. student.. The failure to handle conflict in question, context, and image
is also common in other topics (e.g., Chinese Language), as shown in Table 15.
While the context provided incorrect information by indicating the characters
are 心兴, the ground-truth answer corrects the questioner saying it’s indeed 心头.
All the models trusted the context more than the image and then thus were
misled to answer the question incorrectly. This type of failure, also known as
the hallucination issue, is also discussed in [31, 34], where VLMs may
generate descriptions or answers conflicting with the given image to align
with the natural language instructions or questions, influenced by the strong
language priors of large language models. We will release our dataset to
facilitate further exploration of the hallucination issue in the authentic use
cases.

###  III.3 Automated Model Benchmarking

Supplementing the main paper, where we reported the performance of the six
models with respect to the three most human-aligned metrics (i.e., METEOR,
GPT-4 metric, and BERTscore), we report here performance with respect to all
six metrics. Overall results are shown in Table 9. We observed that all models
perform poorly on our dataset, especially when considering ROUGE-L, METEOR,
and GPT-4 metrics. When comparing across metrics, we find that all metrics
except CLIP-S indicate GPT-4 and mPLUG-Owl are the best two models.

Models | ROUGE-L | METEOR | BERTscore | CLIP-S | RefCLIP-S | ~GPT-4 Metric
---|---|---|---|---|---|---
~GPT-4V[44] | 0.16 | 0.12 | 0.76 | 0.70 | 0.75 | 0.55
mPLUG-Owl [55] | 0.16 | 0.12 | 0.76 | 0.72 | 0.75 | 0.4
LLaVA [35] | 0.15 | 0.08 | 0.75 | 0.73 | 0.75 | 0.38
MiniGPT-4 [59] | 0.15 | 0.09 | 0.75 | 0.70 | 0.74 | 0.34
InstructBLIP [12] | 0.09 | 0.05 | 0.67 | 0.68 | 0.71 | 0.29
BLIP2 [32] | 0.09 | 0.04 | 0.69 | 0.66 | 0.71 | 0.27
Table 9: Results for the state-of-the-art vision and language models on our
dataset, evaluated with reference-based metrics ( ROUGE-L, METEOR, BERTscore,
RefCLIP-S, and GPT-4) and reference-less metric (CLIP-S). Models or scores
labeled with "~" are evaluated on 1,903 random samples. We only reported
GPT-4V model and GPT-4 metric on samples due to the high cost of calling
GPT-4V and GPT-4’s API.  Figure 8: Instructions for our domain expert
annotation task.

###  III.4 Qualitative Examples from Different Models

We additionally provide a qualitative example from different models in Table
16, where the visual questions require significant reasoning. We think the
visual question in Table 16 is interesting as algorithms like AlphaGo or
MuZero can surpass humans in chess games, but the vision and language models
fail in this domain. For example, MiniGPT4, InstructBlip, and BLIP2 didn’t
even understand the visual question. LLaVA directly copied the example
provided in the context. mPLUG-Owl also directly copied the example provided
in the context except omitted two moves, leading to an illegal move at round 8
(half move #15 Rh4). In contrast, GPT-4V gave an answer that on the surface-
level seemed to make sense. Yet, GPT-4V violated the requirement of not
capturing any pawn in the third round (half move #5 Bxf7+) as well as made an
illegal move in round 11th (halfmove #21 gxf6). Of note, we verified all
models’ answers with https://www.apronus.com/chess/pgnviewer/.

Figure 9: Screen shot of the annotation task interface. Experts recorded their
results in spreadsheets.

A visual question where most of the models can answer correctly:
---
Category |  Mechanics
|
Question |  Tire sidewall damage, replace immediately?
Context |  Somehow I was lucky enough to hit something when going downhill. And now my tire looks like in the picture. So should I still drive with this tire or get it replace immediately? And to replace it, is it safe to drive to the tire shop, which is a few miles away?
Reference Answer |  Sidewall damage is a serious issue. Sidewall is the structural part of the tire, and damage to it can result in dangerous consequences. Replace your tire immediately. If you know how to put the spare tire on (and there is a spare tire in your car), please do it now, and drive only to the tire shop. Spare tires are not designed for extended drives, so don’t start driving around in the spare tire. If you have driven more than a few miles already with the tire before looking at the damage, you may have enough data to indicate that it may survive to the tire shop. I would however prefer the spare tire approach in your case. If there’s no spare tire, then that would be a difficult decision: to have your car towed with the broken tire off the ground, or to drive the car. You know more about the details in your case than I do, because you didn’t indicate how many miles exactly you drove with the tire after the incident.
GPT-4V (Correct) |  This looks like a significant sidewall damage and driving on this tire can be dangerous. It is recommended to replace the tire immediately. If you have a spare tire, it would be safer to use the spare and drive to the tire shop to get a new tire. If you don’t have a spare, drive slowly and cautiously to the tire shop, and avoid highways and high speeds.
mPLUG-Owl (Partially correct) |  Yes, it is advisable to replace the tire immediately if it has suffered significant damage. The sidewall of the tire is crucial for maintaining its structural integrity while driving. Damage to the sidewall can lead to air loss, which can increase the risk of a blowout, especially on high speeds or when making sharp turns. In such cases, driving with a damaged tire can be extremely dangerous and could result in a catastrophic accident. It is safer to replace the tire as soon as possible to ensure your safety on the road. To drive to the tire shop, it is usually safe to do so as long as the tire is not leaking air. However, it’s important to check for any visible signs of air leakage while driving and to ensure that the tire pressure is maintained at the recommended level. If you notice any signs of air leakage, pull over to a safe location and stop as soon as possible to check for any further issues.
LLaVA (Correct) |  Given the severe sidewall damage to the tire, it is recommended to replace it immediately. While it is technically possible to drive with the damaged tire, there are risks associated with doing so. The damage could worsen, leading to a potential blowout, which could result in loss of control, accidents, or further tire damage. Additionally, driving with a damaged tire can impact the vehicle’s handling and braking capabilities. It is safer to replace the damaged tire with a new one, ensuring optimal road safety and vehicle performance. As for driving to the tire shop, it is advisable to do so if the tire shop is nearby and if there are no other more severe damages to the vehicle that require immediate attention. If in doubt, consult an expert or seek assistance from roadside services if available.
MiniGPT-4 (Correct) |  It is not recommended to drive with a damaged tire, as it could result in a blowout or loss of control of your vehicle. If the tire is severely damaged, it is important to replace it immediately, as it is not safe to continue driving on it. If the tire is only slightly damaged, it may still be safe to drive on it for a short distance, but it is still recommended to have it inspected and potentially replaced. It is safe to drive to the tire shop a few miles away, but it would be wise to make sure that the tire shop is open and able to serve you before making the trip. If you cannot replace the tire immediately, you should drive as little as possible and have the tire inspected as soon as possible.
InstructBLIP (Correct) |  replace immediately
BLIP2 (Incorrect) |  yes, it is safe to drive to the tire shop

Table 10: An example where most of the models can answer correctly. Models’
answers were verified and the errors were highlighted in red by a Mechanics
Ph.D.

A visual question needing specific domain knowledge:
---
Category |  Economics
|
Question |  Measuring income effect before knowing substitution effect
Context |  Suppose on the x axis we have shelter and on the y axis we have composite goods. Now, if the price of shelter increases, the optimal bundle changes from point A to point D. Standard textbook tells me to draw a line parallel to the new budget line B1, which is tangent to the indifference curve I0. In this way we can get the substitution effect. The income effect follows. My question is, can we reverse the two procedures and measure the income effect first by drawing a line parallel to the original budget line and tangent to the new indifference line? Thanks!
Reference Answer |  You can definitely do so! There are two versions of the income effect. The one that you are referring to is called the Equivalent variation. The two ways of looking at the problem were introduced by Sir John Richard Hicks. To see the impact of a price change in monetary terms, we ask how much money should have been taken before the price change to leave a consumer at the same utility level he attains after the price change. This is the same thing as you had asked: can we reverse the two procedures and measure the income effect first by drawing a line parallel to the original budget line and tangent to the new indifference line? I hope this has helped but if you wish to explore further, see the ’Consumer Surplus’ chapter in Hal Varian’s ’Intermediate Microeconomics’.
GPT-4V (Partially correct) |  Yes, you can reverse the two procedures and measure the income effect first, but it is less intuitive and less accurate compared to measuring the substitution effect first. The reason for this is because the income effect is the difference between the old and new optimal bundles (A and D) after removing the substitution effect. Since the substitution effect is a result of changes in relative prices, in order to accurately measure the income effect, you would first need to find the new optimal bundle that would have been chosen if the relative prices had remained the same. In practice, this is difficult to do because you would have to make assumptions about how the individual’s preferences would have changed, and you would need to find the new optimal bundle on the new indifference curve. It would be much easier and more accurate to simply measure the substitution effect first, and then subtract that from the total effect to find the income effect.

Table 11: A visual question that requires specific domain knowledge in
Economics and infographic understanding. Models’ answers are from GPT-4V.
Errors are marked in red by an economics PhD student for reader convenience.

A visual question needing specific domain knowledge:
---
mPLUG-Owl (Partially correct) |  Yes, you can measure the income effect before knowing the substitution effect by drawing a line parallel to the original budget line and tangent to a new indifference curve. The income effect is the effect on consumption of a given good when the income of a consumer increases or decreases. To measure the income effect, we need to find the new indifference curve that represents consumption choices of consumers after their income has changed. The original budget line represents consumption choices of consumers before the change in income, while the new indifference curve represents consumption choices after the change in income. To find the new indifference curve, we can use the budget line equation: y = C(x, y) = w(y - t(x, y)) where x and y represent consumption of shelter and composite goods, respectively, and t(x, y) represents the marginal utility of composite goods. C(x, y) is the budget line, and w represents the consumer’s budget. The budget line is the line of points where the total utility is equal to a given consumer’s budget. To measure the income effect, we need to find the point of intersection between the original budget line and the new indifference curve, which represents consumption choices after the change in income. The new indifference curve can be found by solving the budget equation using the new income level and the new marginal utility of composite goods. To find the point of intersection between the original budget line and the new indifference curve, we need to solve the budget equation using the new income level and the new marginal utility of composite goods: w(y – t(x, y)) = y – t(x, y) Substituting the new marginal utility of composite goods into the budget equation, we get: w(y – t(x, y)) = y – t(x, y) = y – t((x1, y1), (x2, y2)) where (x1, y1) and (x2, y2) are the consumption points on the new indifference curve. We can then solve for the point of intersection between the original budget line and the new indifference curve by substituting x1 and y1 into the budget equation: w(y – t(x1, y1)) = y – t(x1, y1) = y.
LLaVA (Partially correct) |  Yes, it is possible to measure the income effect first by drawing a line parallel to the original budget line and tangent to the new indifference curve. This can be done by following the steps you described in your question. The income effect represents the change in consumption due to a change in income, and it can be calculated by comparing the consumer’s initial position (point A) to the new optimal consumption point (point D) after the change in the price of shelter. The substitution effect, on the other hand, is the effect of changing the relative price of shelter and composite goods on the consumer’s choice of consumption bundle. It can be derived from the difference between the tangent line parallel to the new budget line and the indifference curve, and the income effect.
MiniGPT-4(Partially correct) |  Yes, you can certainly measure the income effect first by drawing a line parallel to the original budget line and tangent to the new indifference curve. This approach is commonly used in economics to measure the income effect of a change in the price of a good or service. The income effect is the change in the quantity demanded of a good or service due to a change in income. When income increases, the income effect causes the consumer to shift the demand curve to the right, leading to an increase in the quantity demanded of the good or service. Similarly, when income decreases, the income effect causes the demand curve to shift left, resulting in a decrease in the quantity demanded. To measure the income effect using a substitution effect approach, you would draw a line parallel to the original budget line (B1) that is tangent to the new indifference curve (I0’) that represents the consumer’s maximum utility point after the change in the price of the good or service. The point of tangency is denoted by the letter ’P’ on the new budget constraint. The ratio of the areas of the triangles PBC and PC defines the magnitude of the income effect, which can be measured in terms of the quantity demanded of the good or service. Note that this approach is not the most intuitive way to measure income effect. It’s more common to draw a new budget constraint to measure the income effect. The new budget constraint after
InstructBLIP (Partially correct) |  Yes, it is possible to reverse the two procedures and measure the income effect first by drawing a line parallel to the original budget line and tangent to the new indifference line.
BLIP2 (Incorrect) |  No, you can’t measure the income effect first by drawing a line parallel to the original budget line and tangent to the new indifference line.

Table 12: A visual question that requires specific domain knowledge in
Economics and infographic understanding. Models’ answers are from mPLUG-owl,
LLaVA, MiniGPT-4, InstructBLIP, and BLIP2. Errors are marked in red by an
economics Ph.D. student for reader convenience.

A visual question when the question and the context are misleading while the
image contains true information: :
---
Category |  Gardening & Landscaping
|
Question |  How do I safely eradicate mold without hurting cactus seedlings?
Context |  Here are some of my seedlings, sown a little bit less than a month ago. As you can see, there’s mold starting to form. I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect from contamination. What would be an easy way to get rid of the mold without hurting the seedlings? Someone suggested to use hydrogen peroxide, but I’m not sure it’s safe for the seedlings. Ideas?
Reference Answer |  That isn’t mold, it’s moss/algae. You can tell (by the green) that it’s growing from energy obtained by photosynthesis, so it’s not a big threat competition-wise. It’s a sign that the mix is constantly moist on the surface, and also grows faster in high humidity. It will not damage the cactus seedlings, but here’s a tip. When cacti germinate, they grow very slowly/remain the same size for a long time. That’s because they’re developing a strong root system. A sign they are established is that they will begin to grow at a faster rate. This means the root system is growing deep, and you can allow the top 1/2 of mix to dry between waterings. This will keep moss from establishing. For now, you can use a toothpick and stir up the top layer if you want, to slow the moss, as described here: Is there anything wrong with allowing moss to grow around herbs? Or you can use a pair of tweezers and carefully remove the moss. In the long run, it really doesn’t matter. The moss won’t be a problem after the seedlings mature a little, and you don’t have to water so frequently. And by the way, those seedlings look great! I hope you have success with them.
GPT-4V (Partially correct) |  To get rid of mold without hurting your cactus seedlings, try the following methods: 1. Cinnamon: Cinnamon has antifungal properties and can be used as a natural fungicide. Simply sprinkle some ground cinnamon over the affected areas, and it should help control the mold growth. 2. Baking soda: Baking soda is also effective at controlling mold growth. Mix one tablespoon of baking soda with a quart of water, and spray the solution over the affected areas. 3. Vinegar: Vinegar also has antifungal properties and can be used as a natural fungicide. Mix one part vinegar with three parts water, and spray over affected areas. Some other tips to help prevent mold growth in the future include: - Limiting the amount of moisture in the soil - Increasing air circulation around the seedlings - Avoiding overwatering - Keeping soil temperature between 70-80 degrees Fahrenheit - Keeping relative humidity between 40-60%. Don’t use hydrogen peroxide as it can be harmful to the seedlings.
mPLUG-Owl (Incorrect) |  It is essential to take steps to safely eradicate any mold growth on your cactus seedlings without harming them. One option is to use hydrogen peroxide. However, it is important to be cautious when using it on seedlings. Hydrogen peroxide is a strong oxidizing agent that can cause burns or damage to the cactus seedlings if used improperly. To safely use hydrogen peroxide to treat mold on the seedlings, follow these steps: 1. Use a dilute solution of hydrogen peroxide. The best dilution ratio is around 1 tablespoon of 3% hydrogen peroxide per gallon of water. 2. Pour the hydrogen peroxide solution into a shallow dish or container. 3. Soak the affected cactus seedlings in the solution for a few minutes. 4. Remove the seedlings from the solution and place them back in the pot with the Ziploc bag. 5. Seal the Ziploc bag, making sure to leave a small opening at the top to allow ventilation. By following these steps, you can ensure that the hydrogen peroxide solution will not come in direct contact with the cactus seedlings and will effectively eradicate any mold without harming the plants. It is essential to monitor the seedlings carefully and remove them from the hydrogen peroxide solution if any signs of injury or damage appear. If you are not confident in using hydrogen peroxide or have any concerns about its effectiveness, you may consider using a more gentle method, such as washing the seedlings with a gentle soap and water solution and allowing it to dry thoroughly.

Table 13: A visual question when the question and context are misleading while
the image contains true information. Model answers are from GPT-4V and mPLUG-
Owl, all misled by the question and context. Models’ answers verified by a
Ph.D. in Botany major.

A visual question when the question and context are misleading while the image
contains true information:
---
LLaVA (Partially correct) |  To safely eradicate mold without hurting the cactus seedlings, you can follow these steps: 1. Identify the mold: Look closely at the mold’s appearance and determine if it’s harmful to cacti. Some molds can be beneficial, such as those that create the beautiful Cordyceps fungus, while others can be detrimental to the plant’s health. 2. Remove the mold: If the mold is detrimental, gently pick it off using a clean, dry cotton swab or a soft brush. Avoid using water or wetting the cotton swab, as this could spread the mold spores and affect the seedlings. 3. Improve air circulation and light: Ensure that your seedlings are receiving adequate light and air circulation. Mold thrives in dark, damp environments, so providing proper light and ventilation can help prevent mold growth. 4. Keep the seedlings well-spaced: Overcrowding can lead to mold, as it allows for insufficient air circulation. Keep the seedlings well-spaced and avoid overwatering to maintain a suitable environment for healthy growth. 5. Use hydrogen peroxide cautiously: If you must use hydrogen peroxide, dilute it to a low concentration (1-3%), and apply it only to the affected area, avoiding the seedlings. Remember that hydrogen peroxide can be harmful to some plants, so use it with caution and dilute it properly to minimize risk. Always prioritize the health and safety of your seedlings, and avoid using harsh chemicals or treatments that could harm them. Focus on maintaining a clean and well-lit environment to encourage healthy growth and prevent mold.
MiniGPT-4 (Incorrect) |  It’s important to take mold in your cactus seedlings seriously and take measures to prevent its spread. While hydrogen peroxide (H2O2) is known for its antimicrobial properties, it can be harmful to plants, especially at high concentrations. I would not recommend using H2O2 to get rid of mold in your seedlings. Instead, you can use the following methods: 1. Water and sunlight: Water the affected area thoroughly and place it in direct sunlight. The sun’s UV rays can help to kill the mold spores. 2. Vinegar: Mix a solution of equal parts water and white vinegar in a spray bottle, and spray the mold off the seedlings. Vinegar is a natural fungicide. 3. Bleach: Mix a solution of 1 part bleach and 9 parts water and spray it onto the seedlings. Be careful not to overdose the bleach, as it can be harmful to plants and the environment. Remember, after treating the mold, to thoroughly clean and disinfect the affected area and surrounding areas to prevent the mold from re-appearing. Also keep the humidity and temperature of your grow area in check, to prevent mold from appearing again. Please let me know if you
InstructBLIP (Incorrect) |  The cactus seedlings in the image were sown a little bit less than a month ago. As you can see, there’s mold starting to form. I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination. As you can see, there’s mold starting to form. I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination. What would be an easy way to get rid of the mold without hurting the seedlings? Someone suggested to use hydrogen peroxide, but I’m not sure it’s safe for the seedlings. Ideas?
BLIP2 (Incorrect) |  I’m using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination I am using the baggie method whereby the pot with the seeds is enclosed in a Ziploc bag, to maintain high humidity and protect against contamination.

Table 14: A visual question when the question and context are misleading while
the image contains true information. Model answers are from LLaVA, MiniGPT-4,
InstructBLIP, and BLIP2, all misled by question and context. Models’ answers
verified by a Ph.D. in Botany major.

A visual question requires text recognition skills:
---
Category |  Chinese Language
|
Question |  What is the meaning of 心兴?
Context |  What is the meaning of 心兴 in the picture below (towards bottom left in image)? I am guessing it means to think, but doesn’t 想 mean to think?
Reference Answer |  That character is not「兴」(to rise > excited, happy), it is「头」(head). 「心头」(literally heart + head) means mind - the picture is saying that you use your mind to think/reason.
GPT-4V (Incorrect) |  心兴 is a colloquial expression in Chinese. It roughly means "to have a happy mood or personally interested in something". It’s often used in the context of "心兴来了" (to suddenly become interested or happy) or "心兴去了" (to suddenly lose interest or become unhappy). In the picture, the character is saying "我心兴来了" which means "I am suddenly interested" or "I am suddenly in a good mood". As for your question about 想 , 想 does mean "to think", but it can also mean "to want" or "to wish". Chinese is a contextual language, so the meaning of a word or phrase can change depending on the context it is used in.
mPLUG-Owl (Incorrect) |  Yes, you are correct! The meaning of "心兴" is to be enthusiastic or have enthusiasm. It is often used to describe someone who is excited or passionate about something, such as a new project or idea. The meaning of the phrase "心兴" in the picture is to have enthusiasm for learning. It is a part of a Chinese character that is often used in children’s books and learning materials, such as the colorful picture you have provided. The image shows a young girl holding a book with this character on it, indicating that she is interested in learning and has an enthusiasm for reading and education.
LLaVA (Incorrect) |  In this context, "心兴" likely means "thinking" or "intelligence" in Chinese. The image, which includes pictures of children with various expressions and a book with Chinese characters, suggests that the scene involves a learning activity or a discussion about the concepts covered in the book. The presence of "心兴" in the picture implies that it is a theme or an important idea related to the content being presented.
MiniGPT-4 (Incorrect) |  "心兴" (xīn xìng) in the picture is the Chinese word for "heartfelt" or "heartfelt" (a feeling of contentment or happiness). The character for "心" (xīn) means "heart" or "mind," while the character for "兴" (xìng) means "to feel happy or content." Therefore, "心兴" (xīn xìng) can be translated as "feeling contented or happy." The character 想 (cāng) in Chinese means to think or to ponder, so it’s not the same as "心兴" (xīn xìng) .
InstructBLIP (Incorrect) |  yes
BLIP2 (Incorrect) |  ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Table 15: A visual question related to the Chinese Language and requires text
recognition skills. Models’ answers verified by a Chinese native speaker.

A visual question requires multi-hop reasoning process:
---
Category |  Chess
|
Question |  Devouring Nobility
Context |  Provide, with the minimum number of moves, a legal chess game where all pieces are captured except the sixteen pawns and the two kings. Example of final configuration: Which is done in 21 moves: 1. e4 Nc6 2. Bb5 e5 3. Bxc6 dxc6 4. Nf3 Bg4 5. h4 Bxf3 6. Rh3 Bxd1 7. Kxd1 f5 8. Rg3 Nf6 9. Rg4 fxg4 10. d4 Ba3 11. Bg5 Qd5 12. Bxf6 Qb3 13. bxa3 Rd8 14. Bxd8 Kf7 15. axb3 Rxd8 16. Nc3 Rd5 17. exd5 Kf6 18. Nb5 cxb5 19. Kd2 Kf5 20. Re1 Kf4 21. Re4+ Kxe4 Can you beat 21 moves?
Reference Answer |  13 moves: 1. e3 e6 2. Na3 Bxa3 3. bxa3 Na6 4. Bxa6 bxa6 5. Rb1 Qf6 6. Rb8 Qa1 7. Rxc8+ Ke7 8. Ne2 Qxc1 9. O-O Qxd1 10. Rxg8 Qxe2 11. Rxh8 Qxf1+ 12. Kxf1 Re8 13. Rxe8+ Kxe8 12½: 1. e3 e6 2. Na3 Bxa3 3. bxa3 Na6 4. Bxa6 bxa6 5. Qf3 Qf6 6. Qxa8 Qxa1 7. Qxc8+ Ke7 8. Qxg8 Qxc1+ 9. Ke2 Qxg1 10. Qxh8 Qxh1 11. Qe8+ Kxe8 12. a4 Qe1+ 13. Kxe1 Try it online!
GPT-4V (Incorrect) |  Yes, it can be done in 19 moves: 1. e4 Nc6 2. Bc4 e5 3. Bxf7+ Kxf7 4. Qh5+ g6 5. Qxe5 Nxe5 6. d4 Ng4 7. h4 Nh2 8. Rxh2 d5 9. Bg5 Qxg5 10. hxg5 Bf5 11. gxf6 Bc5 12. exf5 Bxd4 13. Rxh7+ Rxh7 14. fxg6+ Kxg6 15. Nf3 Rf8 16. Nxd4 Rxf6 17. Nd2 Rh1+ 18. Nxh1 Rf4 19. Ng3 Kg5 (Note: There may be even more efficient solutions, but this is the best I was able to come up with.
mPLUG-Owl (Incorrect) |  Yes, it is possible to beat the given game of chess with 19 moves. The game is in a drawish position, and it’s possible to force a win by sacrificing a piece and exploiting some of the tactics present in the game. Here are the moves to beat the game: 1\. e4 Nc6 2. Bb5 e5 3. Bxc6 dxc6 4. Nf3 Bg4 5. h4 Bxf3 6. Rh3 Bxd1 7. Kxd1 f5 8. Rh4 fxg4 9. d4 Ba3 10. Bg5 Qd5 11. Bxf6 Qb3 12. bxa3 Rd8 13. Bxd8 Kf7 14. axb3 Rxd8 15. Nc3 Rd5 16. exd5 Kf6 17. Nb5 cxb5 18. Kd2 Kf5 19. Re1 Kf4 (19. Re4+ Kxe4 would lead to a stalemate as white can capture the king with checkmate on the next turn. The move 19. Re1 Kf4 is therefore a better option to avoid stalemate).
LLaVA (Incorrect) |  I cannot play a game of chess directly with you, but I can provide you with a solution to the problem you have presented in the image. The image shows a chess board with a total of 16 pawns and two kings, with a total of 18 pieces on the board. In order to capture all the pieces except for the two kings and the 16 pawns, the following sequence of moves can be followed: 1\. e4 Nc6 2. Bb5 e5 3. Bxc6 dxc6 4. Nf3 Bg4 5. h4 Bxf3 6. Rh3 Bxd1 7. Kxd1 f5 8. Rg3 Nf6 9. Rg4 fxg4 10. d4 Ba3 11. Bg5 Qd5 12. Bxf6 Qb3 13. bxa3 Rd8 14. Bxd8 Kf7 15. axb3 Rxd8 16. Nc3 Rd5 17. exd5 Kf6 18. Nb5 cxb5 19. Kd2 Kf5 20. Re1 Kf4 21. Re4+ Kxe4 This sequence of moves will capture all the pieces except for the two kings and the 16 pawns. The solution has been found using the minimax algorithm, which is an optimal algorithm for solving two-player games such as chess.
MiniGPT-4 (Incorrect) |  As an AI language model, I can certainly analyze the position you provided and try to find a continuation that may lead to a win for White, but it’s important to note that I am not a chess engine and my analysis will not be as deep or as accurate as a specialized chess engine like Stockfish. Given the position you provided, I can see that Black’s king is exposed and vulnerable to a quick checkmate. One move that may be try to force a win for White is 21. Rh8+. This move aims to put the king in check and force black king to move, and then exploit the exposed king with a series of moves like Rh6, Rh5, and Rh4, putting pressure on the king and threatening a checkmate. It is also possible to try to exploit the open position with a pawn storm with moves like e4 and e5, to put pressure on the black king and try to win the game. It’s important to note that this is just one possible move and that a skilled chess player player could find other other other moves to try to to try to win the the game. Also, Stockfish or other chess engines will likely provide a more detailed analysis of the position and a better best continuation.
InstructBLIP (Incorrect) |  Chess
BLIP2 (Incorrect) |  bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 bxa3 b .

Table 16: A visual question requires a significant reasoning process.