# Grounding Answers for Visual Questions Asked by Visually Impaired People

Chongyan Chen1, Samreen Anjum2, Danna Gurari1,2
$~{}^{1}$ University of Texas at Austin $~{}^{2}$ University of Colorado
Boulder

###### Abstract

Visual question answering is the task of answering questions about images. We
introduce the VizWiz-VQA-Grounding dataset, the first dataset that visually
grounds answers to visual questions asked by people with visual impairments.
We analyze our dataset and compare it with five VQA-Grounding datasets to
demonstrate what makes it similar and different. We then evaluate the SOTA VQA
and VQA-Grounding models and demonstrate that current SOTA algorithms often
fail to identify the correct visual evidence where the answer is located.
These models regularly struggle when the visual evidence occupies a small
fraction of the image, for images that are higher quality, as well as for
visual questions that require skills in text recognition. The dataset,
evaluation server, and leaderboard all can be found at the following link:
https://vizwiz.org/tasks-and-datasets/answer-grounding-for-vqa/.

##  1 Introduction

Visual question answering (VQA) is the task of providing a natural language
answer to a question about an image. While most VQA services only return a
natural language answer, our work is motivated by the belief that it is also
valuable for a VQA service to return the region in the image used to arrive at
the answer. We call this task of locating the relevant visual evidence _answer
grounding_.

Numerous applications would be possible if answer groundings were provided in
response to visual questions. _First_ , they enable assessment of whether a
VQA model reasons based on the correct visual evidence. This is valuable as an
explanation as well as to support developers in debugging models. _Second_ ,
answer groundings enable segmenting the relevant content from the background.
This is a valuable precursor for obfuscating the background to preserve
privacy, given that photographers can inadvertently capture private
information in the background of their images [14] (exemplified in Figure 1b).
_Third_ , users with low vision could more quickly find the desired
information if a service instead magnified the relevant visual evidence. This
is valuable in part because answers from VQA services can be insufficient,
including because humans suffer from “reporting bias” meaning they describe
what they find interesting without understanding what a person/population is
seeking. This is exemplified in Figure 1c, where the most popular response
from 10 answers is the generic answer ‘pasta’ rather than the specific flavor,
‘Creamy Tomato Basil Penne’.

Figure 1: (a) We introduce a new dataset challenge that supports the task of
grounding the visual evidence needed to answer visual questions asked by
people with vision impairments. This enables valuable use cases including (b)
background obfuscation to limit inadvertent privacy leaks and (c) automatic
magnification to expedite low vision users’ abilities to answer their
questions.

While datasets have been introduced to encourage progress on the answer
grounding problem, all proposed dataset challenges originate from contrived
visual questions [43, 9, 11, 22, 18, 17, 6, 38, 26]. This includes scraping
images from photo-sharing websites (e.g., Flickr) and then generating
questions automatically [6], by using image annotations paired with question
templates to create questions about the images or (2) manually [9, 11, 43,
18], by asking crowdworkers to make up questions about an image that would
stump a robot. Yet, prior work has shown that such contrived settings can
manifest different characteristics from authentic VQA use cases [15, 40]. This
can cause algorithms trained and evaluated on contrived datasets to perform
poorly when deployed for authentic use cases [15, 16]. Moreover, this can
limit the designs of algorithms since developers are oblivious to the
additional challenges their algorithms must overcome.

We introduce the first answer grounding dataset that originates from an
authentic use case. We focus on visual questions originating from blind people
who both took the pictures and asked the questions about them in order to
overcome real visual challenges [5]. This use case has been shown to manifest
different challenges than contrived settings, including that images are lower
quality [8], questions are more conversational [15], and different vision
skills are needed to arrive at answers [40]. For approximately 10,000 image-
question pairs submitted by this population, we collected answer groundings.
Then, we analyzed the answer groundings to reveal their characteristics and
show how they relate/differ to five existing answer grounding datasets.
Finally, we benchmarked state-of-the-art VQA and answer grounding models on
our dataset and demonstrate what makes this dataset difficult for them,
including smaller answer groundings, images that are of higher quality, and
visual questions that require skills in text recognition.

We offer this work as a foundation for designing models that are robust to a
larger range of potential challenges that can arise in real-world VQA
settings. Challenges observed in our dataset can generalize to other
scenarios, such as robotics and lifelogging, which similarly encounter varying
image quality and textual information (e.g., grocery stores). To encourage
community-wide progress on such challenges, we have organized a dataset
challenge with public evaluation server and leaderboard. Details can be found
at the following link: https://vizwiz.org/tasks-and-datasets/answer-grounding-
for-vqa/.

##  2 Related Work

#### VQA Datasets.

Many large-scale VQA datasets have been proposed over the past six years [39,
19, 33, 34]. A key challenge the community has faced in developing such
datasets is the language bias problem [12, 27, 23, 30]. In particular, models
can learn to exploit superficial correlations observed in datasets by
identifying common pairings of answers and questions and never looking at the
images. For example, when given an image of a green banana with an associated
question asking “what is the color of the banana?”, the VQA model might answer
“yellow” instead of green because the answer “yellow” appears more frequently
in the answers for questions related to bananas. This problem has been
possible, in part, because common evaluation metrics [4] only assess model
performance based on textual answers. To combat such biases, new VQA datasets
have been created to more equally capture the range of possible answers for
each question [13] and many new models have been developed using these
balanced datasets. Our work contributes to this body of work by enabling
algorithm developers to go beyond only evaluating the textual answers and also
directly evaluate whether algorithms rely on the correct visual evidence.

#### Answer Grounding Datasets.

Towards disentangling the vision problem for VQA, several answer grounding
datasets already have been introduced that locate the visual content needed to
arrive at each language-based answer [9, 11, 43, 18, 6]. While some were
created by tracking where humans look when presented with a visual question
[9, 6] or collecting bounding boxes around the relevant visual evidence [43,
22, 17], our work more closely aligns with those where humans who annotated
the relevant visual evidence using a segmentation [11, 18, 26]. That is
because we also collect segmentations. We propose the first answer grounding
dataset that reflects an authentic VQA use case and conduct extensive analysis
to demonstrate how it relates/differs to five existing answer grounding
datasets.

#### VQA and Answering Grounding Algorithms.

Modern VQA algorithms often rely on attention maps to determine where to look
to find the answers to visual questions, with this having been the dominant
approach for a number of years [9, 29, 42]. Recently, some researchers have
shifted their focus towards delivering the best VQA models possible under the
constraint that the model should outperform all other VQA models in attending
to the correct visual evidence [38]. We benchmark the state-of-the-art VQA and
answer grounding models on our dataset to examine to what extent they succeed
in correctly grounding answers. Experiments reveal that our new dataset is
challenging for modern algorithms, and show what aspects make it challenging.

#### Assistive Technology for People with Vision Impairments.

Many people with visual impairments rely on visual assistance devices to learn
about their surroundings. For instance, people with low vision often rely on
magnification tools to better observe content of interest [20, 28, 31], given
that they have limited sight that can’t be regained fully with corrective
measures such as glasses. In addition, people with low vision and no vision
rely on on-demand technologies [5, 1, 2] that deliver answers to submitted
visual questions. A challenge for this latter use case is that blind people
have no way to check if they inadvertently capture private information in
their images. Yet, roughly 12% of pictures taken for a VQA use case contained
privacy information [14] and visually impaired people have expressed their
discomfort with leaking private information [3, 36, 35]. Our work can
contribute to a wide range of interests for people with vision impairments
since answer groundings can serve as a valuable precursor to smartly magnify
visual answers to visual questions, mitigate biases in VQA services, and
support increased privacy measures with obfuscation.

##  3 VizWiz-VQA-Grounding Dataset

We now introduce our dataset for grounding answers to visual questions asked
in an authentic use case where people who are blind were trying to learn about
their visual surroundings. We call this dataset “VizWiz-VQA-Grounding”.

###  3.1 Dataset Creation

#### Dataset Source.

Our work builds upon the VizWiz-VQA dataset [15], which consists of 32,842
image question pairs where each comes with 10 crowdsourced answers. In
addition to the VQA triplets for the publicly-available train and validation
splits, we also consider those from the test split since the authors of [15]
provided us with answer annotations for this purpose. The images and questions
come from visually impaired people who shared them to solicit visual
assistance in their daily lives.

#### Dataset Filtering.

We designed our dataset to focus on grounding answers for visual questions
that could unambiguously be grounded to a single region. Towards achieving
this, we filtered the initial dataset using a combination of automated and
manual techniques that are described in the Supplementary Materials. In
summary, we removed all questions that were non-answerable, embedded multiple
sub-questions, referred to multiple regions in an image due to ambiguity,
could not be grounded, or for which the majority of crowd did not agree on a
single answer. This process left a total of 9,998 VQAs, which we use for our
new dataset. We focus on grounding only the single most popular answer for
each visual question.

#### Grounding Task Design.

Through iterative pilot studies, we designed a user interface for grounding
answers to visual questions. A person is shown the question-answer pair just
above the image and must then demarcate the answer grounding by clicking a
series of points on the image to create a connected polygon. We provide
extensive instructions to cover many annotation scenarios that may be tricky.
For instance, when multiple regions containing the same thing need to be
annotated (e.g., a bunch of flowers), we instruct the annotator to demarcate
all relevant content with a single polygon when the regions are connected.
When the question asks about a property of a visual entity (e.g., color of
clothes), we instruct the annotator to comprehensively annotate all regions
that would lead to the answer rather than a minimum viable region.

#### Annotation Collection.

We implemented a number of techniques to support collecting high quality
results. First, we developed a rigorous three-step filtering process to
recruit expert crowdworkers from Amazon Mechanical Turk to complete our
annotation tasks, which we describe in the Supplementary Materials. In
summary, we limit which users can complete our tasks, provide a qualification
test, and then select several expert workers who we identified as excelling at
our task for a larger number of tasks. Next, for all work submitted by these
expert workers, we conduct automated and manual quality control to verify that
we can continue to trust their work. Finally, we collected two answer
groundings per VQA instance (i.e. image-question-answer triplet). Based on our
subsequent analysis for when and why answer groundings differed, we decided to
select the larger of the two annotations as our ground truth annotation
because we found that annotation differences typically occurred because one
was a sub-region of the other.

###  3.2 Dataset Analysis

We now analyze the VizWiz-Visual Grounding dataset, which consists of 9,998
groundings for the 9,998 VQA triplets. To do so, we compute for each grounding
its:

  * •

Location: position of its center of mass relative to the entire image; i.e., a
(x,y) coordinate. Each coordinate can range from 0 to 1.

  * •

Boundary complexity: entropy of the histogram of the normalized centroid
contour distance [7], where centroid contour distance is the distance of every
point on the segmentation boundary to the segmentation’s center of mass.
Values can range from 0 to 1.

  * •

Image coverage: fraction of pixels it occupies from all pixels in the image.
Values can range from 0 to 1.

To compare our dataset to the current focus of the research community, we also
evaluate answer groundings from existing datasets that were similarly
generated through a manual segmentation annotation process: VQS [11], VQA-X
[18], and TextVQA-X [26]. For completeness, we also include CLEVR-Answers [38]
and GQA [17], as these groundings were used to validate the state-of-the-art
answer grounding method [38]. Unlike our dataset, the images for all these
datasets are either scraped from the Internet—including VQS, VQA-X, and GQA
which leverage images from COCO [25] and TextVQA-X which leverages images from
Open Images v3 [21]—or computer generated, as is the case for CLEVR-Answers.
Another difference from our dataset is that the questions for these datasets
were either generated by crowd workers —for VQS, VQA-X, TextVQA-X—or computer
generated, as is the case for GQA and CLEVR-Answers. To support fair
comparison, we only consider visual questions in these datasets for which
there is exactly one answer grounding region.

#### Overall Results.

For all datasets, statistics about the answer groundings’ _location_ are shown
in Table 1 and _boundary complexity_ and _image coverage_ are shown in Figure
2.

Overall, we observe that all datasets have answer groundings that typically
lie close to the center of the image (Table 1). This is evident from the
combination of mean centroids around the coordinates (0.5, 0.5) and relatively
small standard deviations from those coordinates. We found this result
surprising for our dataset since visually impaired photographers cannot verify
they center the content of interest when taking the pictures.

Regarding the complexity of the boundaries for answer groundings, our dataset
lies in the middle of all datasets. On one extreme lies both datasets for
which bounding boxes are collected: CLEVR-Ans and GQA. That is because the
value computed for boundary complexity is 0 when the boundary is a rectangle.
Interestingly, we observe for VQS that over half of the answer groundings also
are rectangles, as shown by its median score of 0. At the other extreme, lies
the following datasets: VQA-X and TextVQA-X. We suspect the higher boundary
complexity for TextVQA-X is due to its annotation collection approach.
Specifically, it is the only dataset which collected groundings using a
paintbrush rather than a series of points clicked around the boundary of an
object. Implicitly, this annotation approach leads to a seemingly more complex
boundary when a simple polygon might suffice. Our dataset is most similar to
VQA-X, which has the highest median complexity, but our dataset also exhibits
a smaller range of complexity values.

| Relative location of groundings
---|---
Ours | (0.48$\pm$0.14, 0.51$\pm$0.15)
VQA-X | (0.50$\pm$0.16, 0.53$\pm$0.19 )
VQS | (0.50$\pm$0.20, 0.52$\pm$0.21)
TextVQA-X | (0.49$\pm$0.21 , 0.48$\pm$0.25)
CLEVR-Ans | (0.50$\pm$0.19, 0.45$\pm$0.13)
GQA | (0.50$\pm$0.20, 0.54$\pm$0.19)
Table 1: Shown is the mean and standard deviation of the location of all
answer groundings with respect to the images for each dataset. Across all
datasets, answer groundings tend to be located near the center of the image.
Figure 2: The box plot shows, for each dataset, the range of values for the
answer groundings’ (a) boundary complexity and (b) image coverage. For each
box, the central mark denotes the median score, the box edges denote the 25th
and 75th percentiles scores, the whiskers denote the most extreme data points
not considered outliers, and the outliers are plotted individually. Overall,
our dataset shows a moderate range for boundary complexity and the greatest
range for answer grounding size.

The key criteria that makes our dataset different from the other datasets,
overall, is that its answer groundings occupy a larger range of image coverage
values than all other datasets. This is evident when comparing the box sizes
for all datasets in Figure 2(b). TextVQA-X occupies the smallest region. We
suspect that VQA-X, VQS, and GQA share similar sizes because they all were
generated from the same image source: COCO images. We attribute our dataset’s
inclusion of many considerably larger answer groundings than the other
datasets to the possibility that zooming into the content of interest when
taking the picture may be a more realistic approach photographers would take
in real-world scenarios as they try to only photograph the pertinent content
for answering the question. This distinction is exemplified in Figure 3, where
the visually impaired photographer took a photo by positioning a keyboard
close to the hand which held the camera. Altogether, this finding underscores
a unique benefit of our dataset in that it motivates the design of algorithms
that can simultaneously locate very large and very small regions.

| What is this | What color is this | What color is this shirt | What is in this box | What does this say
---|---|---|---|---|---
Ours | (0.48$\pm$0.11,0.51$\pm$0.11) | (0.50$\pm$0.07,0.51$\pm$0.08) | (0.50$\pm$0.06,0.50$\pm$0.07) | (0.55$\pm$0.12,0.64$\pm$0.15) | (0.48$\pm$0.14,0.47$\pm$0.16)
VQS | (0.69$\pm$0.09,0.45$\pm$0.09) | (0.8,0.46) | - | - | -
TextVQA-X | (0.47$\pm$0.37,0.56$\pm$0.13) | - | - | - | (0.62,0.21)

Table 2: Shown is the mean and standard deviation of the location of all
answer groundings with respect to the images for each dataset for the five
most common questions observed in our dataset. “-” denotes that no statistics
were computed because the dataset lacks the question. The results highlight
that different questions often have different typical locations, especially
across different datasets. Figure 3: Answer grounding for a “keyboard” in our
dataset and in the VQS dataset. This exemplifies that answer groundings in our
dataset can pertain to large regions in the images since photographers in an
authentic use case zoom into the content of interest to try and only
photograph the pertinent content. Figure 4: For each dataset, the box plot
shows the range of values for the answer groundings for the most common
_questions_ in our dataset with respect to: (a) boundary complexity and (b)
image coverage. (Figure 2 describes the box plot visualization)

#### Most Common Questions.

We next evaluate how our dataset compares with existing answer grounding
datasets for the five most common visual questions from our VizWiz-VQA-
Grounding dataset. The most common questions are: “What is this”, “What color
is this”, “What color is this shirt”, “What is in this box”, and “What does
this say”.111We group the following questions together: “what is this”, “what
is it”, “what is this item”, “what is that”, “what’s that”, “what is this
please” into the “What is this” group. We also group the following questions
together: “what color is this” and “what color is it”.  Note that none of
these questions are observed in the VQA-X, CLEVR-Answers, and GQA datasets.
Consequently, we exclude these three datasets from comparison. For each
dataset, analysis of the groundings with respect to these questions are shown
in Table 2 for _location_ and Figure 4 for _boundary complexity_ and _image
coverage_.

Across all datasets, we observe different types of questions manifest
different typical locations, boundary complexities, and sizes from each other.
For instance, in Figure 4(b), we see that the average boundary complexity for
“What does this say” is uniquely near zero for our dataset and so usually can
be grounded with a rectangle. We suspect algorithms will be able to take
advantage of these differences to learn predictive cues for grounding answers.

We also observe that the characteristics of answer groundings for the same
question are considerably different across the different datasets. This is
evident when examining the results for “What color is this” and “What does
this say”; i.e., there are considerable differences for the typical location
of the answer grounding (Table 2), the typical range of boundary complexity
values (Figure 4a), and the typical range of image coverage values (Figure
4b). Consequently, if models trained on other datasets are learning biases
between specific questions and answer grounding locations without truly
understanding the question, they would generalize poorly to our new dataset
(and vice versa).

Figure 5: For each dataset, the box plot shows the range of values for the
answer groundings for the most common _answers_ in our dataset with respect
to: (a) boundary complexity and (b) image coverage. (Figure 2 describes the
box plot visualization)

#### Most Common Answers.

We perform parallel analysis to what we conducted for the most common
questions but now with respect to five of the most common answers from the
VizWiz-VQA-Grounding dataset: “yes”, “white”, “no”, “keyboard”, and
“dog”.222We restricted our analysis to one color-related answer to support
greater diversity in our analysis. For each dataset, we analyze groundings
with respect to these specific answers. We observe that the relative location
is similar for all answers, with answers typically grounded at the center of
the images. Due to space constraints, we include these results in the
Supplementary Materials. In contrast, we observe that different answers
manifest different statistics for the boundary complexity (Figure 5a) and
relative sizes (Figure 5b), both within each dataset and across different
datasets. This reinforces our findings from the most common questions. Answer
differences can yield valuable predictive cues for grounding answers and
cross-dataset evaluation could be valuable for preventing models from learning
superficial correlations between answers and answer grounding characteristics
in a specific dataset.

#### Vision Skills Needed to Answer Visual Questions.

We also conduct fine-grained analysis of our dataset with respect to the types
of skills needed to answer visual questions, using the labels provided in the
VizWiz-VQA-Skills dataset [40]. Overall, we again observe that answer
groundings typically are positioned at the center of the image with respect to
vision skills. Due to space constraints, we include these results in the
Supplementary Materials. In contrast, we observe that different vision skills
are correlated with different statistics for image coverage (Figure 6b).
Consequently, vision skills also could provide valuable predictive cues that
algorithms could latch onto when trying to ground answers. For instance,
visual questions about trying to read text tend to have a relatively small
visual grounding area in the image and, often, can be grounded with a simple
bounding region (e.g., with four points). In contrast, questions related to
recognizing color tend to have a larger visual grounding area and more complex
boundary. We attribute this distinction to text being a consistently well-
structured entity with clear printing boundaries whereas color can be used to
describe less structured entities such as articles of clothing which can
manifest a variety of shapes.

Figure 6: The box plot shows the range of values for (a) boundary complexity
and (b) image coverage for all answer groundings in our dataset with respect
to visual questions that require different vision skills. (Figure 2 describes
the box plot visualization)

##  4 Automatically Grounding Answers for VQA

Using our grounding dataset, we now quantify to what extent state-of-the-art
VQA models look at the correct regions in images to answer the questions.

#### Dataset Splits.

Our VizWiz-VQA-Grounding dataset’s train/val/test/ splits match the
train/val/test splits of the original VizWiz-VQA dataset [15]. This resulted
in 6,494, 1,131 and 2,373 visual questions in the training, validation, and
test sets respectively.

#### Baseline Models.

We benchmark a total of six models.

First, we chose the top-performing VQA algorithms with publicly-available code
for the 2021 VizWiz-VQA dataset challenge [15] and the mainstream 2021 VQA
dataset challenge [13], that is _LXMERT_ [37] and _OSCAR_ [24] _with VinVL
image features_ [41] respectively. Both models are pretrained on the train
splits of their respective challenge datasets. In order to generate attention
masks, we follow the process described in [37] to analyze attention maps
extracted from each model. Using the default parameters, attention weights
across the multiple attention heads are extracted and averaged to obtain the
final attention map. A threshold of 0.5 is then applied to generate the final
binary segmentation mask.

We also chose the state-of-the-art model for answer grounding: MAC-Caps [38].
Given an image and question, MAC-Caps predicts an answer and attention weights
on the image. As done in [38], we obtain the final binary segmentation mask by
applying a threshold of 0.5 to the attention weights extracted from the last
reasoning step. We benchmark four variants of MAC-Caps. We use the two models
that were pretrained on GQA and CLEVR respectively, as described in the
original paper. Next, we train the MAC-Caps algorithm from scratch using the
train split of the VizWiz-VQA [15] dataset. Finally, we also train the MAC-
Caps algorithm from scratch using the train split of the VQA-v2 dataset [13],
since this dataset was by design intended to prevent models from learning
language biases and to instead encourage models to look at the images.

#### Evaluation Metric.

We employ Intersection over Union (IoU) to measure the similarity of each
binary segmentation mask to the ground truth segmentation. Values range from 0
to 1, with higher values indicating better performance. We compute the mean
IoU score across all test examples and report results are percentages (i.e.,
IoU value x 100).

We also evaluate with the common metric for detection and localization tasks:
mAP@IoU. Following the COCO evaluation protocol, we use different IoU
thresholds, from 0.25 to 0.75, and average AP values with IoU thresholds in
the range of 0.5 to 0.95 with a step size of 0.05. Due to space constraints,
results are provided in the Supplementary Materials. In summary, these results
reinforce our findings with respect to the IoU metric, as described below.

#### Overall Results.

The performance for each model on the VizWiz-VQA-Grounding test split is shown
in Table 3. Performance is reported for all visual questions (column 3) as
well as only the subset of visual questions for which each model correctly
predicted the answers (column 4).

(a) Image Coverage

(b) Image Quality

(c) Vision Skills

Figure 7: Comparison of MAC-Caps (pretrained on VizWiz), LXMERT, and OSCAR’s performance on (a) visual questions containing different sizes of attention areas for the answers, (b) visual questions for images with different severity of quality issues, and (c) visual questions requiring different vision skills to answer correctly. Overall, we observe (a) smaller answer groundings, (b) images with no quality issues, and (c) visual questions requiring counting skills are most challenging for the models.  Model | Pretrained | Avg IoU | Avg IoU (correct)
---|---|---|---
LXMERT | VizWiz-VQA | 22.09 |  26.96 (906)
OSCAR | VQA-v2 | 15.48 | 19.79 (693)
MAC-Caps | GQA | 12.56 | 17.77 (270)
MAC-Caps | CLEVR | 15.31 | 10.98 (60)
MAC-Caps | VQA-v2 | 17.42 | 19.58 (374)
MAC-Caps | VizWiz-VQA | 27.43 |  32.8 (352)
Table 3: Performance of six models when evaluated on the VizWiz-VQA-Grounding
test set: two state-of-art VQA models (LXMERT [37] and OSCAR [24]) and four
variants of the state-of-art VQA model for answer grounding (MAC-Caps [38]).
IoU scores (averaged over all 2,373 samples for each model) are reported in
percentages. The number in parentheses is the total number of answers
correctly predicted by each model.

We observe that all models performed poorly overall. For example, the top-
performing MAC-Caps model that was trained on the VizWiz-VQA dataset achieves
an IoU score of 27.43%. The story improves only modestly when considering just
those visual questions for which the model predicted correct answers; i.e.,
the IoU jumps $\sim$5 percentage points to 32.8%. These findings indicate that
existing mechanisms intended to guide models to look at the correct visual
evidence are insufficient. This includes both the state-of-the-art algorithm
for answer grounding (i.e., MAC-Caps) and the mainstream VQA dataset [13]
which was designed to encourage models to look at the images.

Our results reveal that the best indicator of better answer groundings is that
models were pre-trained on the VizWiz-VQA dataset. In particular, the top two
approaches are LXMERT and MAC-Caps trained on VizWiz-VQA. Interestingly,
neither of these models were trained with answer groundings and so neither
could benefit from direct supervision of what answer groundings look like.
Altogether, this finding highlights a considerable domain shift between the
real-world use case for people with visual impairments and the contrived
settings for generating the other datasets.

We also observe that, for the VQA task, the state-of-the-art VQA algorithms
considerably outperform the state-of-the-art answer grounding models that are
pretrained on the same datasets. Specifically, compared to MAC-Caps, LXMERT
predicts over 2.5 times as many correct answers while OSCAR predicts almost 2
times as many correct answers. This finding suggests that a large part of the
success for state-of-art VQA models still stems from learned biases that are
unrelated to the relevant visual evidence.

In what follows, we conduct fine-grained analysis for the top-performing
visual grounding model (MAC-Caps pre-trained on VizWiz-VQA) and the two state-
of-the-art VQA models (LXMERT and OSCAR).

#### Analysis With Respect to Image Coverage.

We next assess each model’s ability to accurately locate the answer groundings
based on the answer groundings’ relative size. To do so, we divide the test
VQA instances into three buckets of “Small”, “Medium”, and “Large” based on
their image coverage, specifically whether they occupy up to 1/3 of the image,
between 1/3 and 2/3 of the image, and more than 2/3 of the image respectively.
In total, there are 1456 small examples, 458 medium examples, and 459 large
examples in the test set. Results are shown in Figure 7(a).

Overall, we observe that all models struggle most to predict answer groundings
for the small set. While this finding is not necessarily surprising, we
believe it is still worthwhile to be shown experimentally.

Interestingly, the advantage the top-performing answer grounding model (MAC-
Caps) has over the two state-of-the-art VQA models (LXMERT and OSCAR) in
grounding answers stems from its ability to better ground small and medium
sized regions. This is evident when examining the median IoU scores, which are
roughly double for MAC-Caps what is observed for the other two models. A
valuable area for future work will be to decipher what enables MAC-Caps to
better attend to these smaller answer regions compared to the mainstream VQA
models.

#### Analysis With Respect to Image Quality.

We next assess each model’s ability to accurately locate the answer groundings
based on the image quality issues defined in [8]. First, we follow the process
described in [16] to divide the VQAs into three buckets of “None”, “Medium”,
and “Severe” quality issues based on quality ratings from five crowdworkers,
resulting in 1,930, 351, and 92 examples respectively. Results are shown in
Figure 7(b). We also assess performance with respect to specific quality
issues (i.e., poor framing, blurry, too dark, too bright, obfuscations, and
improper rotations) and, due to space constraints, provide results in the
Supplementary Materials.

Surprisingly, we observe that images without any quality issues (“None”) are
the most challenging for models, particularly the VQA models (LXMERT and
OSCAR). Upon further analysis, we found that this is because these images have
smaller answer regions, and hence show similar performance as observed for
images with “Small” attention areas for image coverage. Specifically, the
average attention areas for “None”, “Medium” and “Severe” images are 0.28,
0.47 and 0.83 respectively. This also explains why models performed better on
“Severe” images as they tend to have larger answer regions. Further analysis
also shows 82% “Severe” images contain questions asking about color, which we
will see in the next section, are also examples where the models typically
perform better.

#### Analysis With Respect to Vision Skills.

Next, we assess each model’s ability to accurately locate the answer
groundings based on the vision skills needed to answer the questions, as
introduced in Section 3.2. Specifically, these skills are object recognition,
color recognition, text recognition, and counting. Results are reported in
Figure 7(c).

We consistently observe across all models that they perform worse for
questions involving text recognition and counting while they perform better
for questions involving object recognition and color recognition. We suspect
we observe improved performance for the latter two skills because color
recognition has a relatively simple image analysis component and object
recognition models have become quite advanced compared to many other vision
tasks, given the large focus on the problem which was spurred by the ImageNet
dataset challenge [10]. This finding also could be due in part to our prior
observation that answer groundings for text recognition tend to have smaller
answer groundings while color recognition tends to have larger visual
grounding areas (Section 3.2) and models perform worse for smaller answer
groundings (Section 4).

Figure 8: Qualitative results exemplifying answer groundings from the best
model, MAC-Caps pretrained on VizWiz-VQA.

#### Analysis of the Presence of Text.

We also analyze how often answer groundings contain text. We apply the
Microsoft Read API to detect and recognize text in grounded areas. Of the
9,998 answer groundings, about 52% (i.e., 5,207 images) were detected as
containing text. For this subset, we compared the extracted text to the ground
truth answers. We found that the ground truth answer was present in the
detected text for only 7% (i.e., 372 images) of visual questions. This
suggests that most language answers cannot be found directly from detected
text in answer groundings.

#### Qualitative Results.

We finally show qualitative results from the top-performing answer grounding
model, MAC-Caps pretrained on VizWiz-VQA. Results are shown in Figure 8. These
examples illustrate answer groundings for a large range of sizes (e.g., row 3
vs row 4) as well as visual questions that require different vision skills,
such as text recognition for rows 1 and 3, object recognition for row 2, and
color recognition for row 4. When observing the model’s predictions, we
observe that the results reinforce our quantitative findings that the model
often fails. These answer grounding failures include for both when the model
predicts correct natural language answers (rows 1 and 4) and predicts
incorrect answers (rows 2 and 3).

##  5 Conclusions

Our VizWiz-VQA-Grounding dataset offers a strong foundation for supporting the
community to design less biased VQA models and more accurate answer grounding
models which can serve as a valuable precursor for a range of practical
applications. We will publicly-release the dataset alongside a public
evaluation server and leaderboard to spur community progress on this important
answer grounding problem. Our benchmarking of state-of-the-art models reveal
current limitations for future models to overcome. Future work will need to
establish how to ensure such algorithms truly learn to understand the visual
questions rather than learning superficial correlations between properties of
visual questions and their answer groundings.

#### Acknowledgments.

We are grateful for funding support from Microsoft AI4A and Amazon Mechanical
Turk.

## References

  * [1] Be my eyes - see the world together.  https://www.bemyeyes.com/.  Accessed: 2021-11-15.
  * [2] bespecular.  https://www.bespecular.com/.  Accessed: 2021-11-15.
  * [3] Taslima Akter, Bryan Dosono, Tousif Ahmed, Apu Kapadia, and Bryan Semaan.  ” i am uncomfortable sharing what i can’t see”: Privacy concerns of the visually impaired with camera based assistive applications.  In 29th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 20), pages 1929–1948, 2020.
  * [4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [5] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al.  Vizwiz: nearly real-time answers to visual questions.  In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342, 2010.
  * [6] Shi Chen, Ming Jiang, Jinhui Yang, and Qi Zhao.  Air: Attention with reasoning capability.  arXiv preprint arXiv:2007.14419, 2020.
  * [7] Yinpeng Chen and Hari Sundaram.  Estimating complexity of 2d shapes.  In 2005 IEEE 7th Workshop on Multimedia Signal Processing, pages 1–4. IEEE, 2005.
  * [8] Tai-Yin Chiu, Yinan Zhao, and Danna Gurari.  Assessing image quality issues for real-world problems.  arXiv preprint arXiv:2003.12511, 2020.
  * [9] Abhishek Das, Harsh Agrawal, Larry Zitnick, Devi Parikh, and Dhruv Batra.  Human attention in visual question answering: Do humans and deep networks look at the same regions?  Computer Vision and Image Understanding, 163:90–100, 2017.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [11] Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong.  Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmentation.  In Proceedings of the IEEE international conference on computer vision, pages 1811–1820, 2017.
  * [12] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
  * [13] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [14] Danna Gurari, Qing Li, Chi Lin, Yinan Zhao, Anhong Guo, Abigale Stangl, and Jeffrey P Bigham.  Vizwiz-priv: A dataset for recognizing the presence and purpose of private visual information in images taken by blind people.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 939–948, 2019.
  * [15] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608–3617, 2018.
  * [16] Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya.  Captioning images taken by people who are blind.  arXiv preprint arXiv:2002.08565, 2020.
  * [17] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700–6709, 2019.
  * [18] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach.  Multimodal explanations: Justifying decisions and pointing to the evidence.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8779–8788, 2018.
  * [19] Kushal Kafle and Christopher Kanan.  An analysis of visual question answering algorithms.  In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [20] Richard L Kline and Ephraim P Glinert.  Improving gui accessibility for people with low vision.  In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 114–121, 1995.
  * [21] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://github.com/openimages, 2017.
  * [22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123(1):32–73, 2017.
  * [23] Gouthaman KV and Anurag Mittal.  Reducing language biases in visual question answering with visually-grounded question encoder.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16, pages 18–34. Springer, 2020.
  * [24] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [26] Varun Nagaraj Rao, Xingjian Zhen, Karen Hovsepian, and Mingwei Shen.  A first look: Towards explainable TextVQA models via visual and textual explanations.  In Proceedings of the Third Workshop on Multimodal Artificial Intelligence, pages 19–29, Mexico City, Mexico, June 2021. Association for Computational Linguistics.
  * [27] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen.  Counterfactual vqa: A cause-effect look at language bias.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12700–12710, 2021.
  * [28] Stephan Pölzer, Elias Gander, and Klaus Miesenberger.  Gaze based magnification to assist visually impaired persons.  In International Conference on Computers Helping People with Special Needs, pages 333–337. Springer, 2018.
  * [29] Tingting Qiao, Jianfeng Dong, and Duanqing Xu.  Exploring human-like attention supervision in visual question answering.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [30] Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee.  Overcoming language priors in visual question answering with adversarial regularization.  arXiv preprint arXiv:1810.03649, 2018.
  * [31] Thorsten Schwarz, Arsalan Akbarioroumieh, Giuseppe Melfi, and Rainer Stiefelhagen.  Developing a magnification prototype based on head and eye-tracking for persons with low vision.  In International Conference on Computers Helping People with Special Needs, pages 354–363. Springer, 2020.
  * [32] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019.
  * [33] Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee.  Visual question answering using deep learning: A survey and performance analysis.  arXiv preprint arXiv:1909.01860, 2019.
  * [34] Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee.  Visual question answering using deep learning: A survey and performance analysis.  In International Conference on Computer Vision and Image Processing, pages 75–86. Springer, 2020.
  * [35] Abigale Stangl, Kristina Shiroma, Nathan Davis, Bo Xie, Kenneth R Fleischmann, Leah Findlater, and Danna Gurari.  Privacy concerns for visual assistance technologies.  ACM Transactions on Accessible Computing (TACCESS), 2022.
  * [36] Abigale Stangl, Kristina Shiroma, Bo Xie, Kenneth R Fleischmann, and Danna Gurari.  Visual content considered private by people who are blind.  In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility, pages 1–12, 2020.
  * [37] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [38] Aisha Urooj, Hilde Kuehne, Kevin Duarte, Chuang Gan, Niels Lobo, and Mubarak Shah.  Found a reason for me? weakly-supervised grounded visual question answering using capsules.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8465–8474, 2021.
  * [39] Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton van den Hengel.  Visual question answering: A survey of methods and datasets.  Computer Vision and Image Understanding, 163:21–40, 2017.
  * [40] Xiaoyu Zeng, Yanan Wang, Tai-Yin Chiu, Nilavra Bhattacharya, and Danna Gurari.  Vision skills needed to answer visual questions.  Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2):1–31, 2020.
  * [41] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579–5588, 2021.
  * [42] Yundong Zhang, Juan Carlos Niebles, and Alvaro Soto.  Interpretable visual question answering by visual grounding from attention supervision mining.  In 2019 ieee winter conference on applications of computer vision (wacv), pages 349–357. IEEE, 2019.
  * [43] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei.  Visual7w: Grounded question answering in images.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.

## Supplementary Material

This document supplements the main paper with more information about:

  1. 1.

Method for filtering visual questions (supplements Section 3.1)

  2. 2.

Dataset collection (supplements Section 3.1)

     * •

Method for hiring expert crowdworkers (supplements Section 3.1)

     * •

Screenshot of our annotation task interface (supplements Section 3.1)

     * •

Method for reviewing work from crowdworkers (supplements Section 3.1)

  3. 3.

Dataset analysis (supplements Section 3.2)

  4. 4.

Qualitative results for model benchmarking (supplements Section 4)

##  I Method for filtering visual questions

Our aim was to ensure our dataset focused on visual questions for which
answers could be unambiguously grounded to a single region. To do so, we
performed five rounds of filtering visual questions from the initial VizWiz-
VQA dataset, that we decribe below.

First, we applied a filter to remove all questions which were unanswerable.
This occurs regularly in the VizWiz-VQA dataset because the photographers
could not verify the content in their images due to being blind. We removed
all visual questions which were labelled as “unanswerable” in the provided
“answer_type” metadata.

We also removed visual questions for which at least half of the crowd did not
agree on the same answer. We used a stringent string matching approach to
detect if at least 5 out of the 10 answers per visual question are identical.

Another filter we applied is to remove all so-called “questions” that actually
embed multiple questions. An example is “What is this and is it to be put in
the microwave? Or does it even say?” We did not simply filter visual questions
with more than two question marks because we observed that users often ask
refinements to their questions or ask the same question several times in
different ways. Examples include “What color are these jeans? Pink or gold?”
and “Can I wear these two pieces together? Do they match?”. From pilot
testing, we observed an effective automated mechanism is to remove visual
questions that have more than five words while containing the word “and”
(i.e., 901 visual questions). We also process those visual questions that
contained repetition of a single question, e.g., “what is this? what is
this?”. We trim these down to a single question to remove redundancy. We then
had two expert crowdworkers review each candidate visual question to identify
whether each question actually asked more than one question. We filtered a
visual question if at least one person flagged it as containing more than one
question. Of the 11,085 visual questions that were reviewed, 0.6% (66) were
tagged as containing more than one question.

We also filtered visual questions for which there is ambiguity where the
answer is located in an image, meaning answers referred to more than one image
region. We conducted preliminary analysis to understand the prevalence of this
case. From 200 random visual questions from the VizWiz training dataset, we
found only two contained this issue. From further analysis, we found that this
issue often appears when the questions contains plural nouns333We used the
NLTK package to detect parts of speech, including plural nouns. or when they
contain phrases such as “how many”. While the occurrences were rare, we still
had two expert crowdworkers review each VQA to identify whether more than one
polygon is needed to locate the region to which the answer is referring. A VQA
was removed if at least one person flagged it as needing more than one
polygon. Of 11,019 visual questions that were reviewed, 1.66% (183) were
marked as requiring more than one polygon to locate the answer grounding
region. From inspection of some of these visual questions, possible reasons
for why an answer refers to more than one region are that multiple regions are
suitable for arriving at the same answer (Figure 9a) and that the answer
actually embeds multiple answers that align with distinct visual content
(Figure 9b).

Figure 9: Answer groundings that refer to multiple regions.

Finally, we filtered visual questions that were answerable but could not be
grounded. For example, often this occurs for questions that lead to the answer
“No”, such as “Is there a key?”. To do so, we had two expert crowdworkers
review each VQA triplet to identify whether the answer is not shown in the
image. A VQA was removed if at least one person flagged it as not present.

##  II Dataset Collection

### Method for hiring expert crowdworkers

We only accepted candidates who previously had completed at least 500 Human
Intelligence Tasks (HITs) with over a 95% acceptance rate and were from the
United States. The latter requirement gave us some confidence that the workers
had English proficiency.

Next, we required the crowdworkers to pass a qualification test which included
challenging grounding tasks covered in our instructions. In this test, each
worker is asked to annotate 10 QA pairs for which the ground truth (GT)
annotations were manually drawn by us. The grounding annotation by the worker
is deemed correct if the region has more than a 70% IoU score with the GT
region. The workers had to annotate all of the 10 QA pairs correctly to pass
the qualification test. The user interface blocked a worker from moving to the
next of the 10 tasks until the generated annotation sufficiently matched our
pre-annotated ground truth. Completing this qualification task ensured a
worker understood the task and how to handle challenging annotation scenarios.

All workers who passed the qualification test were eligible to complete 20
grounding tasks. In total, 27 workers completed these tasks. We reviewed all
annotations from them and hired nine workers who consistently generated high
quality results. We limited our number of workers because we prioritized high
quality annotations more than the efficiency from having more workers; i.e.,
it is easier to track the performance of fewer workers.

### Screenshot of our annotation task interface

We show the crowdsourcing task that we created to collect grounding
annotations, including a screen shot of the instructions in Figure 10 and the
annotation task in Figure 11. The link to this code is available at
https://github.com/CCYChongyanChen/VizWizVQAGroundingCrowdSourcing.

### Method for reviewing work from crowdworkers

The nine workers hired to create all answer groundings were given our contact
information so they could contact us with any questions and we gave them the
link to a live document where we frequently added our feedback to their
questions about tricky examples. As they submitted their work, we leveraged
automated quality control checks and manually inspected random samples of
their results to ensure the annotation quality remained high. For the
automated checks, we used the following rules to help us identify quality
issues. For each HIT, we recorded the total number of times a worker answered
‘Yes’ in Step 1 (contains multiple questions), ‘Yes’ in Step 2 (contains
multiple regions), and ‘cannot draw’ in Step 3 (rather than drawing a
segmentation). If the total number for any of these steps was more than 3, we
reviewed all results in the HIT. This is because, our pre-processing steps
(described in Section I) meant that most of the QA pairs should contain one
question and our preliminary analysis indicated that we seldom observed either
that there are multiple regions or that the answer is not present in the
image. We also monitored the time the worker spent on each HIT. If it was less
than 30 seconds, we inspected the results.444In pilot studies, we found that
initially crowdworkers took an average of 4.38 mins to finish a HIT but this
time dropped to 2.8 mins as workers became familiar with the task. Finally, we
also calculated the number of points used to draw the polygon. If a worker
drew less than 5 points for an answer grounding for more than two visual
questions in a single HIT, we manually inspected all results for the HIT.
Examples of high quality answer grounding results are shown in Figure 12.

Figure 10: Instructions for our annotation task interface. Figure 11: A
screenshot of our annotation task interface. Figure 12: Examples of answer
groundings for a variety of question types.

##  III Dataset Analysis

#### Reasons why crowdworkers selected “cannot draw”.

The crowdworkers explained why they indicated the answer cannot be localized
in the image (i.e., by selecting “cannot draw” in the task user interface) in
a free-text box. The top 10 reasons are ‘nothing to draw a polygon’ (106
times), ‘incomplete text’ (28 times), ‘wrong answer’ (25 times), ‘the image is
too blurry/blurry’ (26 times), ‘no clue(s)’ (20 times), ‘subjective’ (9
times), ‘the answer is no so therefore can’t draw something not there’ (7
times), ‘nothing on screen’ (4 times), ‘answer is not shown in the image and
the answer can be answered without the image’ (4 times), and ‘nothing found’
(4 times). Examples of these flagged visual questions are shown in Figure 13.

Figure 13: Examples of visual questions for which workers indicated the answer
cannot be grounded in the image paired with a more general reason why (shown
in blue text).

#### Annotation agreement.

Recall that two answer grounding annotations were collected per visual
question from two crowdworkers. A histogram showing the IoU scores between
each pair of annotations per visual question across the 9,998 visual questions
is shown in Figure 14. The majority (around 60%) of the IoU scores are between
0.8 and 1.0, around 20% lie between 0.6 and 0.8, and slightly more than 10%
lie between 0 and 0.2. This shows that, typically, there is high annotation
agreement. We attribute low scores in part to the IoU being a poor metric when
accounting for smaller regions. An example is shown in Figure 15, where the
IoU score is $\sim$58% despite that both annotations are visually similar and
correct. We selected as ground truth the larger region from the two groundings
since often the smaller one is contained in the larger one.

Figure 14: Histogram of IoU scores indicating similarity between each pair of
answer groundings per visual question. The majority have high agreement, in
the range between 0.8 and 1.0.  Figure 15: Example of two answer groundings
for a small area. The IoU score between these two annotations is 58%. This
exemplifies the tendency for IoU scores to be low for small regions when
groundings appear similar.

#### Whole Image

The visual answer is labelled as referring the whole image for 0.9% (903) of
visual questions. Often, workers selected “Whole Image” when the question
related to color, the camera is set too close to the object, or the questioner
asks about the general description of the scene. Examples are shown in the
last column of Figure 12.

#### Location of answer grounding.

Expanding on Table 1 in the main paper, we visualize the center of mass for
all answer groundings in the different datasets. Results are shown in Figure
16. As shown, our new dataset clusters as a circle and has a smaller range of
values than the other datasets.

Figure 16: Center of the mass of answer groundings in the VizWiz-VQA-
Grounding, VQS, VQA-X and the TextVQA-X datasets.

#### Grounding of most common answers.

Expanding on the analysis in the main paper, we report statistics about the
most common answers in Table 4. Intuitively, we observe that objects with
rectangular shapes need fewer points, e.g., keyboard, laptop, and shampoo. In
contrast, objects with complex boundary require more points, e.g., dog, hand,
chair, and cat. Less intuitively, the mean of the grounding size for the
answer ‘yes’ is smaller than those for which the answer is ‘no’. We suspect
groundings for ‘no’ answers more often refer to the whole image while ‘yes’
primarily focus on specific regions, as exemplified in Figure 17.

Answer | Size (%) | Points | Images
---|---|---|---
yes | 89,264 (45%) | 13 | 525
no | 112,648 (56.3%) | 10 | 240
keyboard | 125,786 (62.9%) | 7 | 118
dog | 63,598 (31.8%) | 40 | 85
laptop | 122,471 (61.2%) | 9 | 66
pepsi | 55,055 (27.5%) | 12 | 48
coca cola | 47,644 (23.8%) | 11 | 46
orange | 82,201 (41.4%) | 14 | 42
corn | 57,947 (29.0%) | 11 | 37
green beans | 56,246 (28.1%) | 10 | 34
pen | 16,731 (8.4%) | 17 | 33
lotion | 42,835 (21.4%) | 16 | 32
cat | 54,714 (27.4%) | 34 | 30
water bottle | 61,062 (30.5%) | 24 | 28
phone | 98,399 (49.2%) | 15 | 28
soup | 44,875 (22.4%) | 11 | 28
Shampoo | 35,190 (17.6%) | 8 | 26
hand sanitizer | 68,540 (34.3%) | 23 | 26
hand | 86,867 (43.4%) | 39 | 26
chair | 81,435 (40.7%) | 37 | 26
remote | 63,090 (31.5%) | 17 | 26
Table 4: Mean value of properties describing the regions for most common
answers (excluded color-related answer, as it is reported in Table 2. Figure
17: Example illustrating the trend that the ‘yes’ answer groundings are
typically smaller than the ‘no’ answer groundings.

We show for the most common answers examples of the grounded area as well as
the average images in Figure 18. For the same language answer, visual
groundings can be diverse. For example, the dog has different breeds, colors,
postures, and locations. The dog can be partially visible and under different
illumination. Also, the answer “dog” can refer to an animal or a picture of a
dog. A more blurry/grey average image is indicative of a greater diversity of
images for the answer.

Figure 18: Examples of answer groundings in our VizWiz-VQA-Grounding dataset.
Shown are answering groundings for common answers (i.e., yes, no, keyboard,
dog, laptop, pepsi, orange) as well as the average image across all groundings
for each answer.

#### Dataset comparison.

We summarize how the characteristics of the eight related answer grounding
datasets, discussed in the related work section of the main paper, relate and
differ to our dataset in Table 5. Our summary indicates the visual annotation
type, number of images with visual annotation, and source VQA dataset.

Dataset | Visual Annotation Type |  | # Images
---
($\times$ Annotations per Image)
VQA dataset
Visual7W (2016)[43] | Bounding box | 47,300 | COCO [25]
VQA-HAT (2017) [9] | Human Att. (deblur image) | 59,849 | VQA v1
VQS (2017) [11] | Segmentation+bounding box | 37,868 | COCO
VQA-X (2018) [18] | Segmentation | 6,000 ($\times$ 1) | COCO, VQAv2
GQA (2019) | Bounding box | 355,530 ($\times$1) | GQA (2019) [17]
AiR (2020) [6] | Human Att. (eye-tracking) | 987 ($\times$ 20) | GQA [17]
Text VQA-X (2021) [26] | Segmentation (brush) | 11,681($\times$ 1) | TextVQA[32]
CLEVR-Ans (2021) [38] | Bounding box | 445,268 ($\times$1) | CLEVR
Ours | Segmentation | 9,998 ($\times$ 2) | VizWiz-VQA [15]
Table 5: Comparison between existing VQA answer grounding datasets and our
dataset.

##  IV Algorithm

#### mAP@IoU results.

The performance for each model on the VizWiz-VQA-Grounding test split with
respect to mAP@IoU score is shown in Table 6. Overall, the low mAP scores
reinforce our findings in the main paper that the models perform poorly on our
new dataset and that the best indicator of better answer groundings is that
models were pre-trained on the VizWiz-VQA dataset (Sec 4).

Model (Pretrained) | mAP25 | mAP50 | mAP75 | mAP
---|---|---|---|---
LXMERT (VizWiz) | 14.21% | 1.99% | 0.02% | 0.49%
OSCAR (VQA-v2) | 7.10% | 0.15% | 0.00% | 0.03%
MAC-Caps (GQA) | 3.94% | 0.09% | 0.00% | 0.02%
MAC-Caps (CLEVR) | 6.38% | 0.12% | 0.00% | 0.01%
MAC-Caps (VQA-v2) | 9.32% | 0.45% | 0.01% | 0.08%
MAC-Caps (VizWiz) | 22.32% | 4.09% | 0.17% | 0.96%
Table 6: Performance of six models when evaluated on the VizWiz-VQA-Grounding
test set: two state-of-art VQA models (LXMERT [37] and OSCAR [24]) and four
variants of the state-of-art VQA model for answer grounding (MAC-Caps [38])
with respect to mAP@IoU. Results are provided based on the COCO evaluation
protocol of using different IoU thresholds, from 0.25 to 0.75, and averaging
AP values with IoU threshold ranges from 0.5 to 0.95 with a step size of 0.05

#### Extraction of attention maps: baseline models.

As discussed in the main paper, we extract attention maps for the two VQA
models: LXMERT and OSCAR. For LXMERT, the output consists of four different
attentions: self-vision attention, self-language attention, image-guided
question attention, and question-guided image attention. Following the
authors’ recommendation, we picked the question-based image attention. For
each visual question, the model predicts 12 attention heads, where each head
was of size 20x36. We first average the attention maps for each head by
length, and then over all the heads to obtain the final attention map. For
OSCAR, we extract attention weights from the last layer. It has 114x114 for 16
heads; For the 114 sequence length, the first 64 dimensions are the language
attention, while the later 50 dimensions are image attentions. We uses self-
vision attention.

#### Naive baseline.

As a naive baseline, we treated predicting the whole image as the grounded
area. This baseline receives an average IoU score of 33%.

#### Analysis With Respect to Image Quality.

As mentioned in the main paper, we report here our fine-grained analysis to
assess each model’s ability to accurately locate the answer groundings based
on the image quality issues defined in [8]: poor framing, blurry, too dark,
too bright, obfuscations, and improper rotations. Results are shown in Figure
19. We observe that images with obscured image quality issue are the most
challenging to ground and images with the rotation issue are the second most
challenging to ground.

Figure 19: Comparison of MAC-Caps (pretrained on VizWiz), LXMERT, and OSCAR’s
performance on visual questions for images with different quality issues.

#### Qualitative results.

Figure 20: Qualitative results exemplifying answer groundings from 6 models.

We show examples for the answer groundings predicted by the six benchmarked
models. Results are shown in Figure 20. The third column exemplifies the top-
performing MAC-Caps model pre-trained on VizWiz. We observe examples for which
the MAC-Caps pretrained on VizWiz predicts incorrectly (rows 1-3) and
correctly (rows 4-6). Examples highlight that the MAC-Caps model struggles
with images containing text (row 2, 3), can correctly predict the answer
without grounding the correct region (row 6), and can fail to predict both the
correct answer and answer grounding (row 2, 3). We observe for LXMERT that it
can similarly predict the answer and answering grounding incorrectly (column
6; row 1, 3) as well as predict the answer correctly while failing to
accurately locate the answer grounding (column 7; row 2, 4, 5, 6). We observe
for the Oscar model pretrained on VQAv2 that it can detect the foreground
object better compared to other models, but fails when the visual evidence is
a small region (column 8; row 2, 3).

We found that models (MAC-CAPs and LXMERT) pretrained on VizWiz-VQA failed to
answer the question and locate the visual evidence for visual questions
requiring color recognition (row 2, column 3,7), while models (MAC-CAPs and
OSCAR) pretrained on VQAv2 can answer questions related to color and ground
the correct region (row 1, column 4, 8). We found this surprising since the
VQAv2 has less visual questions related to color than the VizWiz dataset and
also represents a distinct domain [40].