11institutetext: University of Central Florida, Orlando, FL, USA
22institutetext: Goethe University Frankfurt, Frankfurt, Hesse, Germany
33institutetext: MIT-IBM Watson AI Lab, Cambridge, MA, USA

# Weakly Supervised Grounding for VQA in Vision-Language Transformers

Aisha Urooj Khan 11 0000-0001-6521-2512 Hilde Kuehne 2233 0000-0003-1079-4441
Chuang Gan 33 0000-0003-4031-5886 Niels Da Vitoria Lobo 11 0000-0001-5354-2805
Mubarak Shah 11 0000-0001-6172-5572

###### Abstract

Transformers for visual-language representation learning have been getting a
lot of interest and shown tremendous performance on visual question answering
(VQA) and grounding. But most systems that show good performance of those
tasks still rely on pre-trained object detectors during training, which limits
their applicability to the object classes available for those detectors. To
mitigate this limitation, the following paper focuses on the problem of weakly
supervised grounding in context of visual question answering in transformers.
The approach leverages capsules by grouping each visual token in the visual
encoder and uses activations from language self-attention layers as a text-
guided selection module to mask those capsules before they are forwarded to
the next layer. We evaluate our approach on the challenging GQA as well as
VQA-HAT dataset for VQA grounding. Our experiments show that: while removing
the information of masked objects from standard transformer architectures
leads to a significant drop in performance, the integration of capsules
significantly improves the grounding ability of such systems and provides new
state-of-the-art results compared to other approaches in the field111Code will
be available at https://github.com/aurooj/WSG-VQA-VLTransformers.

###### Keywords:

visual grounding, visual question answering, vision and language

Figure 1:  (a) Proposed architecture: Given the question-image pair, grid
features are used to obtain visual capsules using a Capsule encoding layer.
Output embedding for [CLS] token from text transformer layer is then used to
do capsule features selection. Selected capsules encodings with position
information is then input to the visual encoder. We use sentence embedding
from each textual transformer layer to select capsules for each visual
transformer layer. The selected capsules are then input to the next visual
transformer layer along with the output of the previous layer in the visual
encoder. Finally, a cross-attentional block allows for a fine-grained
interaction between both modalities to predict the answer.  (b) Attention from
the proposed vision-language transformer with VQA supervision. We look at the
self-attention of the [IMG] token on the heads of the last layer. These maps
show that the model automatically learns to ground relevant objects leading to
weakly-supervised grounded VQA (GVQA). blue box is ground truth, orange is
predicted box.

##  1 Introduction

Enabling VQA systems to be explainable can be important for variety of
applications such as assisting visually-impaired people to navigate [17, 67]
or helping radiologists in early diagnosis of fatal diseases [1, 68]. A system
that only produces a good answering accuracy will not be sufficient in these
applications. Instead, VQA systems for such uses should ideally also provide
an answer verification mechanism and grounding is a convincing way to obtain
this direct verification.

Following upon success in natural language processing and multi-modal
understanding, a variety of transformer-based methods have been introduced for
joint representation learning of vision and language and respective downstream
tasks, including VQA. Such approaches, e.g., [60, 43, 37] are usually trained
based on region masks of detected objects generated by a pre-trained object
detector [29]. The assumption that object masks are provided at input time
limits detection to pretrained objects only and comes at the risk of image
context information being lost.

Detector-free methods avoid this bias toward pre-trained object classes while
being simpler and faster because they do not need to extract region-based
features from a pre-trained object detector. Other works [31, 24, 10, 52] have
therefore focused on removing the dependency on object detectors while
achieving comparable if not better performance (e.g., on retrieval and VQA
tasks). Among those, [10] and [52] also show good visual representations by
qualitative examples, but do not provide an evaluation of their answer (or
question) grounding ability.

In this work, we want to address this issue and focus on the problem of weakly
supervised grounding for the VQA task in visual-language transformer based
systems. Compared to detector-free referential expression grounding [64, 7,
41], VQA breaks the assumption that the region description is always part of
the input phrase as the answer word may not be present in the input question.
It is therefore not enough to only learn a direct mapping between text and
image features, but also requires processing multiple image-text mapping steps
with the correct relation between them.
To address the task of VQA grounding in transformer-based architectures with
the question-answering supervision alone, we propose a generic extension of
the visual encoder part of a visual-language transformer based on visual
capsule layers together with a text-guided selection module. Capsule networks
learn to group neurons into visual entities. They thus try to capture entities
(objectness) and relationships among them with promising results on various
visual detection and segmentation tasks [13, 34, 14]. To make use of this
ability in context of transformers, we transfer inputs as well as intermediate
layers’ feature tokens to capsule encodings, from which the most relevant ones
will be selected by the textual tokens of a parallel language encoder. We
propose to interleave transformer layers with such masked residual capsule
encodings. This extension provides a combination of visual input routing and
text-based masking which significantly improves the visual grounding ability
of such systems.

We evaluate existing methods as well as the proposed approach on the
challenging GQA and VQA-HAT datasets. To this end, we consider the attention
output obtained from these methods and evaluate it on various metrics, namely
overlap, intersection over union, and pointing game accuracy. Our results on
the original architectures show a significant gap between task accuracy and
grounding of existing methods indicating that existing vision-language systems
are far from learning an actually grounded representation. The proposed method
bridges the gap and outperforms the best contenders in terms of overlap,
intersection over union, and pointing game accuracy achieving SOTA performance
on the GQA dataset. It also achieves best performance (in terms of mean-rank
correlation score) on VQA-HAT[8] dataset among methods which do not use
attention supervision.

We summarize contributions of our architecture as follows: a) we propose a
capsule encoding layer to generate capsule-based visual tokens for
transformers; b) we propose a text-guided capsule selection with residual
connections to guide the visual features at each encoding step; and c) we
propose generic interleaved processing that can be integrated in various
vision language architectures.

##  2 Related Work

Visual-language Representation Learning. Learning a robust visual-language
representation is currently an active area of research [30] with impressive
progress on downstream tasks including VQA [42, 43, 37, 60, 6, 46, 38, 35,
59]. A majority of these methods rely on object detections making the
downstream task simpler. Some works have attempted to avoid this dependency on
object detections and show comparable performance using spatial features or
image patches [36, 31, 23, 24]. Our work also falls into the later category
and uses grid features as input.

Weakly-supervised Grounding and VQA. Weakly-supervised visual grounding is
well studied for phrase-grounding in images [64, 7, 41, 9, 3, 5, 61]. Few
works also focused on phrase grounding in videos [58, 65, 22]. However, less
attention has been paid to the VQA grounding despite having significance for
many critical applications. There are many works on making questions visually
grounded [53, 48, 57, 70, 62, 71], but only a handful of works are focused to
evaluate their grounding abilities [57, 26, 28, 55, 8, 51]. GQA leverages
scene graphs from Visual Genome dataset providing visual grounding labels for
question and answer making it feasible to evaluate VQA logic grounding.
Recently, xGQA [49] has been introduced as a multilingual version of GQA
benchmark. GQA [26] and [28] discuss the evaluation of VQA systems for
grounding ability. VQA-HAT [8] on the other hand, provides human attention
maps used for answering the question in a game-inspired attention-annotation
setup. A handful of methods [51, 44, 66] evaluate their systems for
correlation between machine-generated attention and human attention maps on
VQA-HAT. However, with the emergence of transformers as the current SOTA, the
focus moves towards grounding abilities of those systems for VQA task.
Unfortunately, none of these transformer-based methods have yet focused on the
evaluation of weakly supervised grounding. However, the fact that only few
real-world dataset provide grounding labels makes this task challenging. We
therefore finetune three existing detector-free transformer methods on GQA and
evaluate them for the weak grounding task.

Transformers with Capsules. There are few works who focused on the idea of
combining transformers and capsules [47, 16, 50, 63, 40, 45, 12]. For
instance, [63] studies text-summarization, image-processing, and 3D vision
tasks; [50, 47] uses capsule-transformer architecture for image-
classification, and [40] uses capsules-transformers for stock movements
prediction. To the best of our knowledge, no one has studied the combination
of capsules with transformers for VQA grounding.

##  3 Proposed Approach

Given an input image-question pair with image $I$ and question $Q$, we want to
localize the relevant question and answer objects with only VQA supervision.
We start from a two stream visual-language model where the language encoder
$L_{e}$ guides input and intermediate respresentations of the visual encoder
$V_{e}$. The input text to the language encoder $L_{e}$ is a sequence of word
tokens from a vocabulary $V$ appended with special tokens $[CLS]$ and $[SEP]$
at the start and end of the word tokens. As input to the visual encoder, our
model takes convolutional features as image embeddings. The convolutional
features $X\in\mathbb{R}^{h\times w\times d1}$ are extracted from a pre-
trained ResNet model, $h,w$ are the feature height and width, and $d1$ is the
extracted features dimension. A 2D convolutional layer then yields an
embedding $X^{\prime}$ of size $\mathbb{R}^{h\times w\times d}$, where $d$ is
the model dimension size. These input embeddings are used to produce capsule
encodings $X_{c}$ as explained in section 3.2.

In the following, we first explain the motivation to use capsules in Sec. 3.1
followed by details about the capsule encoding in Sec. 3.2, the text-guided
selection of the capsules in Sec. 3.3, as well as the text-based residual
connection in Sec. 3.4. We close the section with an overview of the
pretraining procedure in Sec. 3.5 and describe the details of the VQA
downstream task in Sec. 4.1.

|
---|---

(a) Capsule Encoding layer (b) Capsule layer

Figure 2: (a) Capsule encoding layer: grid features
$X^{\prime}\in\mathbb{R}^{h\times w\times d}$ are transformed into capsules
$X_{c}$ for each spatial position. Output embedding $h^{1}_{cls}$ for $[CLS]$
token from the first text encoder layer generates a mask $m^{1}$ for capsule
selection. The selected capsules $X^{1}_{mc}$ are flattened along capsule
dimension to get a set of visual tokens (of length $h*w$) where each token is
denoted by $x^{1}_{j_{mc}},j=\\{1,2,...,hw\\}$;
$x^{1}_{j_{mc}}\in\mathbb{R}^{d_{c}}$ where $d_{c}$=capsule dimension, is then
upsampled to model dimension $d$ using a fully connected layer. A position
embedding is added to visual tokens with the special token [IMG] at position
0. The output capsule encodings are then input to the visual transformer for
future steps. (b) Capsule layer: Similar to the Capsule encoding, input tokens
$X^{\prime}$ are first transformed into capsules $X_{c}$. The output feature
$h^{i}_{cls}$ corresponding to $[CLS]$ token from the text encoder layer $i$
learns the presence probability of a certain capsule at the layer $i$ in
visual encoder. This mask $m^{i}$ selects the capsules respective to the
attended words at text encoder layer $i$. The resulting capsules are then
flattened and upsampled (denoted by $\hat{x}^{i}_{j_{mc}}$) and added to the
output $h^{i-1}_{cls}$ of the previous visual transformer layer $i-1$ to
obtain input features $\hat{h}^{i-1}_{v_{j}}$ for the next visual transformer
layer $i$.

###  3.1 Capsule Networks

Convincing amount of evidence exist in psychology that humans parse visual
scenes into part-whole hierarchies by modelling the viewpoint-invariant
spatial relationship between a part and a whole [18, 27] Neural Networks can
profit from understanding images in the same way humans do to become more
transparent and interpretable. However, standard NNs lack this ability to
dynamically represent a distinct part-whole hierarchy tree structure for each
image[19].

This inability motivated the introduction of a type of model called Capsule
Networks [20] which was later formalized in [56]. A Capsule Network is a
neural network that is designed to model part-whole hierarchical relationships
more explicitly than Convolutional Neural Networks (CNNs), by using groups of
neurons to encode entities and learning the relationships between these
entities. The promising performance of capsules can be attributed to their
ability to learn part-whole relationships for object entities via routing-by-
agreement [56] between different capsule layers. A capsule is represented by a
group of neurons; each capsule layer is comprised of multiple capsules and
multiple capsules layers can be stacked together. Capsule routing is a non-
linear, iterative and clustering-like process that occurs between adjacent
capsule layers, dynamically assigning part capsules $i$ in layer $\ell$ to
object capsules $j$ in layer $\ell+1$ , by iteratively calibrating the routing
coefficients $\boldsymbol{\gamma}$ [54]. Unlike most previous works which use
a loss over object classes to learn a set of capsule classes, we do not have
any object level supervision available for capsules, but instead combine the
power of transformers and capsules by interleaving capsules as intermediate
layers within the transformer and use VQA supervision to model visual entities
as capsules.

###  3.2 Capsule Encodings

We use matrix capsules [21] as follows: given an image embedding
$X^{\prime}\in\mathbb{R}^{h\times w\times d}$, matrix capsules
$X_{c}\in\mathbb{R}^{h\times w\times d_{c}}$, as shown in Figure 2(a), are
obtained as follows: The image embedding $X^{\prime}$ is input to a
convolutional layer producing primary capsules $X_{p}$ where each capsule has
a pose matrix of size $K\times K$ and an activation weight. The primary
capsule layer outputs $C_{p}$ number of capsules for each spatial location.
The output dimensions for poses is $\mathbb{R}^{h\times w\times C_{p}\times
K\times K}$ and for activation is $\mathbb{R}^{h\times w\times C_{p}\times
1}$. To treat each capsule as a separate entity, the pose matrix and
activation is grouped together for each capsule. Hence, the primary capsules
$X_{p}$ have the dimensions $\mathbb{R}^{h\times w\times d_{p}}$ where
$d_{p}=C_{p}\times(K\times K+1)$. The primary capsules are then passed through
an EM-Routing layer to vote for capsules in the next layer. Assuming we have
$C_{v}$ number of capsules in the next layer, the routing yields capsule
encodings $X_{c}$ where $X_{c}\in\mathbb{R}^{h\times w\times d_{c}}$,
$d_{c}=C_{v}\times(K\times K+1)$. We use an equal number of capsules in both
layers i.e., $C=C_{p}=C_{v}$. Our system employs the capsule representation
$X_{c}$ as visual embeddings.

Since transformers take a sequence of tokens as input, we flatten the capsule
embeddings across spatial dimension to get a sequence of visual tokens of
length $h*w$, where each visual token is denoted by
$x_{j}\in\mathbb{R}^{d_{c}}$ for $j\in{1,2,...,hw}$. A special trainable token
$[IMG]$ is then concatenated to these tokens to form the final set of visual
tokens $\\{[IMG],x_{1},x_{2},...,x_{hw}\\}$. A learnable position embedding is
added to these visual tokens to keep the notion of spatial position in the
sequence. Each of the visual tokens except $[IMG]$ is represented by $C$
capsules.

###  3.3 Text-guided Capsule Selection

As the language encoder is attending different words at each layer, we select
visual capsules based on the text representation at each visual encoder layer.
Let $h^{i}_{cls}$ be the feature output corresponding to $[CLS]$ token from
the $i^{th}$ text encoder layer; we take the feature output $h^{1}_{cls}$
against $[CLS]$ token from first text encoder layer and input to a fully
connected layer $\phi$. The output is $C$ logits followed by a softmax
function to learn presence probability $m^{1}\in\mathbb{R}^{C}$ of attended
words at layer 1. This mask is applied to $X_{c}$ to select the respective
capsules and mask out the rest resulting in the masked capsule representation
$X^{1}_{mc}$.

$m^{1}=softmax(\phi(h^{1}_{cls})).$ (1)

$X^{1}_{mc}=m^{1}\odot X_{c}$ (2)

The masking is only applied to the visual tokens $x_{j}$ without affecting
$[IMG]$ token.

###  3.4 Text-based Residual Connection

To keep the capsule representation between intermediate layers, we add
capsules via a residual connection to the inputs of each intermediate visual
encoder layer. The input capsules to the intermediate layer are also selected
based on the intermediate features output from text encoder. Let $m^{i}$ be
the probability mask for attended words in the text feature output
$h^{i}_{cls}$ from the $i^{th}$ layer:

$m^{i}=softmax(\phi(h^{i}_{cls})),\forall i\in\\{1,2,...,L\\},$ (3)

and $x^{i}_{j_{mc}}$ denotes the $j^{th}$ token with visual capsules selected
using mask $m^{i}$.

$x^{i}_{j_{mc}}=m^{i}\odot X_{c},$ (4)

The $i^{th}$ visual encoder layer takes features from $(i-1)^{th}$ layer to
produce features $h^{i}_{v_{j}}$ for $j^{th}$ position. Let $f^{i}_{v}$ be the
$i^{th}$ layer in the visual encoder. The output and input follow the notation
below:

$h^{i}_{v_{j}}=f^{i}_{v}(h^{i-1}_{v_{j}}).$ (5)

To keep reasoning flowing from text to image, we propose to add the residual
connection from visual capsules for $j^{th}$ token by adding $x^{i}_{j_{mc}}$
to the input of $i^{th}$ encoder layer. However, there is a dimension mismatch
between $x^{i}_{j_{mc}}\in\mathbb{R}^{d_{c}}$ and
$h^{i-1}_{v_{j}}\in\mathbb{R}^{d}$. We upsample $x^{i}_{j_{mc}}$ to dimension
size $d$ using a fully connected layer $\sigma$ and get the upsampled capsule-
based features $\hat{x}^{i}_{j_{mc}}\in\mathbb{R}^{d}$. The input to the
visual encoder layer will be as follows:

$\hat{h}^{i-1}_{v_{j}}=f^{i}_{v}(h^{i-1}_{v_{j}}+\hat{x}^{i}_{j_{mc}}).$ (6)

The output feature sequences from both encoders are then input to our cross
attentional module which allows token-level attention between the two
modalities. The aggregated feature outputs corresponding to $[CLS]$ and
$[IMG]$ tokens after cross attention are input to a feature pooling layer
followed by respective classifiers for pretraining and downstream tasks. We
discuss the implementation about modality-specific encoders, feature pooling,
and cross attention in detail in the supplementary.

###  3.5 Training

To perform well, transformers require pretraining on large-scale datasets
before finetuning for the downstream tasks, i.e., GQA and VQA-HAT in our case.
Therefore, we first pretrain our capsules-transformer backbone on three
pretraining tasks: image-text matching (ITM), masked language modeling (MLM),
and visual question answering (VQA). The system is pre-trained in two stages:
first, we do joint training of modality-specific encoders only to learn text-
guided capsules representation; the representation learned in encoders is kept
fixed during the second stage of pre-training where we add cross-attentional
blocks on top of the modality encoders allowing token-level interaction
between text features and visual features. While the first stage of
pretraining uses pooled features from text and from visual encoders, the
second stage pools features after cross attention: therefore, the second stage
pre-training tasks uses cross-modal inputs as language and image features. For
details about pretraining tasks in context of our method, refer to section 1.2
in supplementary. We finally finetune the pretrained capsules-transformer
backbone to solve VQA as our downstream task.

##  4 Experiments and Results

###  4.1 Datasets

Pre-training. We use MSCOCO [39] and Visual Genome [33] for pretraining our
backbone. We use the same data as [60] which also include MSCOCO-based VQA
datasets: Visual7W, VQAv2.0, and GQA. However, we exclude the GQA validation
set from pretraining and finetuning as we evaluate grounding on this set
(scene graphs for GQA test and testdev are not publicly available). We use
train sets of MSCOCO and VG with $\sim$7.5M sentence-image pairs for
pretraining. MSCOCO val set is used for validating pretraining tasks.

Downstream. We consider two datasets for the downstream task, GQA[26] and VQA-
HAT[8].

GQA poses visual reasoning in the form of compositional question answering. It
requires multihop reasoning to answer the question, so GQA is a special case
of VQA. GQA is more diverse than VQA2.0 [15] in terms of coverage of
relational, spatial, and multihop reasoning questions. It has 22M QA pairs
with 113K images. GQA provides ground truth boxes for question and answer
objects making it a suitable test bed for our task.

VQA-HAT dataset provides human attention maps for VQA task. This dataset is
based on VQA v1.0[2] dataset and provides 1374 QA pairs with 488 images in
validation set. To evaluate on this dataset, we train our system on VQA v1.0.
The answer vocabulary of VQA train set has a long tail distribution. We follow
previous works [2, 8] and use 1000 most frequent answers. We first combine
training (248,349 QA pairs) and validation data (121,512 QA pairs) to get a
total of 368487 QA pairs. We then filter out the questions with out-of-
vocabulary answers resulting in 318827 QA pairs.

###  4.2 Evaluation Metrics

For GQA, VQA accuracy is reported for task accuracy. For grounding task on
transformers, we take attention scores from [IMG] token to image from the last
cross-attentional layer for all heads. Answer grounding performance is
evaluated in terms of the following: Overlap– overlap between the ground truth
bounding box for answer object and the detected attention region is reported
in terms of precision (P), recall (R), and F1-score (F1); IOU– intersection
over union (IOU) between the ground truth object and detected region is
reported in terms of P, R, and F1-score. Pointing Game– proposed by [69] is a
metric for weakly-supervised visual grounding methods. For pointing game, we
consider the point detected from each head as a part of distribution, and
perform k-means clustering (k=1) on those points. The cluster center is
considered as the detected point from the system and used for evaluating
accuracy. For VQA-HAT, we report mean rank correlation between system
generated attention and human attention maps to compare with previous methods.

###  4.3 Implementation details.

We use $L=5$ layers in both text and image encoders, and $2$ layers in cross-
attention module. The transformer encoder layers have the same configuration
as BERT with 12 heads and feature dimension $d=768$. A batch size of 1024 with
learning rate $lr=1e-4$ is used for pretraining. First stage pretraining is
done for 20 epochs and further trained for 10-15 epochs during second stage.
We use Imagenet pre-trained ResNet model to extract features of dimensions
$7\times 7\times 2048$. For finetuning on GQA, we use batch size=32 with
$lr=1e-5$ and 5-10 training epochs. For VQA-HAT, we use batch size=64 with
$lr=4e-5$ trained for 20 epochs.

To evaluate the grounding results, we follow the best practice of DINO [4] and
take the attention map from the last cross-attentional layer. To compute
overlap and IOU for GQA, we threshold over the attention map with an attention
threshold of 0.5 to get high attention regions. Each connected region is
considered a detection. For pointing game, we find the single cluster center
over maximum attention points from all heads and use it for evaluation. We
ignore the test samples with empty ground truth maps for pointing game since
there is no ground truth bounding box to check for a hit or a miss. For the
VQA-HAT evaluation, we follow [8] and use mean rank correlation between the
generated attention maps and the ground truth.

Method | Layer | Pointing Game Acc.
---|---|---
Random | - | 18.80
Center | - | 33.30
MAC [25] | mean | 8.90
MAC-Caps [28] | mean | 28.46
LXMERT [60] | last | 29.00
ALBEF [36]-GC | last | 32.13
ALBEF [36]-ATN | last | 32.11
ViLT [31] | last | 11.99
Ours-no-init (C=16) | last | 34.59
Ours-no-init (C=32) | last | 34.43
Ours-nogqa (C=32) | last | 37.04
Table 1: Pointing game accuracy for GQA. For MAC and MAC-Caps, mean attention maps over reasoning steps are used. For transformer-based methods, maximum attention points from all heads are used for clustering. The cluster center is then used for the pointing game evaluation. For ALBEF, GC=GradCAM output, ATN=attention output. Ours-no-init is the full model with residual connections and trained from scratch, Ours-nogqa uses no GQA samples at pretraining stage. Numbers are reported in percentages. |  |  |  |  |  | Overlap |  | IOU
---|---|---|---|---|---|---|---|---
Method | Obj. | Backbone | Pre-training | Layer | Acc. | P | R | F1 |  | P | R | F1
MAC [25] | A | ResNet | - | last | 57.09 | 5.05 | 24.44 | 8.37 |  | 0.76 | 3.70 | 1.27
MAC-Caps [28] | A | ResNet | - | last | 55.13 | 5.46 | 27.9 | 9.13 |  | 0.97 | 4.94 | 1.62
LXMERT-patches | A | Faster RCNN | MSCO,VG | last | 48.65 | 7.13 | 64.21 | 12.83 |  | 0.95 | 8.66 | 1.71
ALBEF [36]-GC | A | ViT+BERT |  | last | 64.16 | 6.94 | 99.92 | 12.98 |  | 0.89 | 13.43 | 1.67
ALBEF [36]-ATN | A | ViT+BERT | MSCO,VG, | last | 64.20 | 5.13 | 99.92 | 9.75 |  | 0.64 | 12.98 | 1.21
ALBEF [36]-GC | A | ViT+BERT | SBU,GCC | 8 | 64.20 | 4.41 | 99.92 | 8.44 |  | 0.54 | 12.85 | 1.04
ALBEF [36]-ATN | A | ViT+BERT |  | 8 | 64.20 | 4.82 | 99.92 | 9.19 |  | 0.60 | 12.88 | 1.14
ViLT [32] | A | ViT | MSCO,VG, | last-cos | 66.33 | 0.34 | 6.13 | 0.65 |  | 0.04 | 0.63 | 0.07
ViLT [32] | A | ViT | SBU,GCC | last-ATN | 66.33 | 0.28 | 4.10 | 0.53 |  | 0.08 | 1.20 | 0.15
Ours (C=16) | A | ResNet | MSCO,VG | last | 56.65 | 14.53 | 85.47 | 24.84 |  | 2.30 | 13.61 | 3.94
MAC [25] | Q | ResNet | - | last | 57.09 | 10.79 | 16.38 | 13.01 |  | 1.39 | 2.09 | 1.67
MAC-Caps [28] | Q | ResNet | - | last | 55.13 | 17.39 | 28.10 | 21.49 |  | 1.87 | 2.96 | 2.29
LXMERT-patches | Q | Faster RCNN | MSCO,VG | last | 48.65 | 32.46 | 64.02 | 43.08 |  | 3.48 | 6.87 | 4.62
ALBEF [36]-GC | Q | ViT+BERT |  | last | 64.20 | 22.15 | 99.90 | 36.26 |  | 1.96 | 9.22 | 3.24
ALBEF [36]-ATN | Q | ViT+BERT | MSCO,VG, | last | 64.20 | 16.50 | 99.90 | 28.33 |  | 1.40 | 8.90 | 2.43
ALBEF [36]-GC | Q | ViT+BERT | SBU,GCC | 8 | 64.20 | 14.21 | 99.90 | 24.88 |  | 1.19 | 8.71 | 2.09
ALBEF [36]-ATN | Q | ViT+BERT |  | 8 | 64.20 | 15.51 | 99.90 | 26.85 |  | 1.31 | 8.77 | 2.27
ViLT [32] | Q | ViT | MSCO,VG, | last-cos | 66.33 | 1.02 | 5.64 | 1.73 |  | 0.10 | 0.54 | 0.17
ViLT [32] | Q | ViT | SBU,GCC | last-ATN | 66.33 | 0.34 | 1.56 | 0.56 |  | 0.08 | 0.38 | 0.14
Ours (C=16) | Q | ResNet | MSCO,VG | last | 56.65 | 47.03 | 81.67 | 59.69 |  | 4.72 | 8.08 | 5.96
Table 2: Results on GQA validation set (for last layer). All methods are
evaluated for weak VQA grounding task. For transformer-based models, attention
was averaged over all heads. Results are based on grounding of objects
referenced in the answer (A) and the question (Q). C=no.of capsules, we report
results from our best model with C=16. Refer to table 4 for more variants. For
ViLT, we obtain results using cosine similarity (cos.) between text and image
features as proposed by the authors as well as from raw attention scores
(ATN). For ALBEF, GC is the gradcam output used for evaluation, ATN is the
attention output. ALBEF uses layer 8 as grounding layer, we also report
grounding performance on this layer. Our method outperforms all baselines for
overlap F1-score and IOU F1-score. See section 4.4 for more details. Numbers
are reported in percentages.

###  4.4 Comparison to State-of-the-Art

We compare the performance of our method to other best-performing methods in
the field of weakly supervised VQA grounding and VQA in general, namely MAC
[25] and MAC-Caps [28] as representation of visual reasoning architectures,
and LXMERT [60], ViLT [31], and ALBEF [36] as state-of-the-art transformer
architectures without object features.

For LXMERT, we take the provided backbone pre-trained on object features and
finetune it using image patches of size $32\times 32\times 3$ on GQA. In case
of ViLT, we use the provided pre-trained backbone and finetune it on GQA.
Following ViLT, we generate a heat map using the cosine similarity between
image and word tokens evaluating the similarity scores as well as raw
attention for grounding performance for all three metrics. For ALBEF we report
results on last layer as well as on layer 8 which is specialized for grounding
[36] using the visualizations from gradcam as well as raw attention maps.

GQA We first look at the results of our evaluation on GQA, considering
pointing game accuracy in Table 1 and for overlap and IOU in Table 2. Our
method outperforms both MAC and MAC-Caps for answer grounding on the last
attention map. We achieve an absolute gain of 16.47% (overlap F1-score) and
2.67% increase in IOU F1-score, and an improvement of 25.69% in pointing game
accuracy for MAC. When compared with MAC-Caps, our best method (C=16, no-init)
improves overlap F1-score by 15.71% $\uparrow$, IOU F1-score by 2.32%
$\uparrow$, and pointing game accuracy by 6.13% $\uparrow$. Similar
performance gain is observed for question grounding with an improvement of
38.2% $\uparrow$ for overlap F1-score and 3.67% $\uparrow$ gain for IOU
F1-score.

To evaluate LXMERT finetuned on image patches (LXMERT-patches), we take the
attention score maps from the last cross modality layer. We improve over
LXMERT by 12.01% $\uparrow$ absolute points w.r.t overlap F1-score, 2.23%
$\uparrow$ w.r.t. IOU F1-score and 5.43% $\uparrow$ gain in pointing game
accuracy. For question grounding, LXMERT achieves an overlap F1-score: 43.08%
(vs. ours 59.69%) and IOU F1-score of 4.62% (vs. ours 5.96%).

ViLT outperforms all methods in terms of VQA accuracy of 66.33%. However, on
the grounding task, it demonstrates the lowest performance for all metrics
(table 1: row 7, table 2: rows 8-9). Similar behavior is observed for the
question grounding task.

ALBEF produces visualization using GradCAM. We compare with ALBEF using both
GradCAM output and attention maps. ALBEF has a very high recall (R) both in
terms of overlap and IOU. However, it lacks in precision (P) leading to lower
F1-scores for both metrics. Our best model outperforms ALBEF-VQA by a
significant margin on both answer grounding and question grounding.

Method | Mean Rank-Correlation
---|---
Random |  0.000 $\pm$ 0.001
Human |  0.623 $\pm$ 0.003
Unsupervised |
SAN [66] |  0.249 $\pm$ 0.004
HieCoAtt [44] |  0.264 $\pm$ 0.004
Ours (C=16) | 0.479 $\pm$ 0.0001
Supervised |
HAN [51] |  0.668 $\pm$ 0.001
Table 3:  Results on VQA-HAT val dataset. Unsupervised: no attention
supervision, Supervised: use attention refinement.  Figure 3: Success cases
for VQA-HAT dataset. VQA-HAT provides 3 human attention maps for each image.
Here, we show the best matched ground truth map (GT HAT).

VQA-HAT We further evaluate our system on the VQA-HAT dataset. To this end, we
follow the protocol of VQA-HAT and resize the human attention maps and the
output attention maps from our system to the common resolution of 14x14. We
then rank both of them. VQA-HAT val set provides three human attention maps
for each question. We compute the rank correlation of generated attention map
with each human attention map and take the average score. Mean rank
correlation score over all QA pairs is reported.

We compare our approach on VQA-HAT with three different baselines: SAN [66]
and HieCoAtt [44] as unsupervised bounding box free systems, and HAN [51]
which uses attention supervision during training. The evaluation is shown in
table 3. It shows that the proposed system is able to significantly outperform
both methods using VQA-only supervision.

Without any attention supervision during training, we are able to narrow the
gap between unsupervised methods and methods such as HAN, which uses human
ground truth attention maps during training.

Figure 3 shows success cases on VQA-HAT, comparing our generated attention
result to the closest human attention map.

|  | Overlap |  | IOU | Pointing
---|---|---|---|---|---
Method | Acc. | P | R | F1 |  | P | R | F1 | Game
(1) no skip(C=32) | 56.83 | | 11.06 | 77.60 | 19.37 | | | 1.39 | 9.85 | 2.43 | 29.81
(2) w/skip (C=32) | 55.41 | | 10.09 | 71.95 | 17.70 | | | 1.41 | 10.09 | 2.47 | 34.43
(3) w/skip (C=16) | 56.65 | | 14.53 | 85.47 | 24.84 | | | 2.30 | 13.61 | 3.94 | 34.59
(4) w/skip (C=24) | 56.26 | | 10.90 | 74.03 | 19.00 | | | 1.54 | 10.56 | 2.69 | 31.08
(5) w/skip (C=32) | 55.41 | | 10.09 | 71.95 | 17.70 | | | 1.41 | 10.09 | 2.47 | 34.43
(6) w/skip (C=48) | 53.65 | | 10.28 | 68.94 | 17.89 | | | 1.59 | 10.73 | 2.78 | 29.70
(7) no-init (C=32) | 55.41 | | 10.09 | 71.95 | 17.70 | | | 1.41 | 10.09 | 2.47 | 34.43
(8) vit-bert init (C=32) | 58.86 | | 11.11 | 74.67 | 19.34 | | | 1.55 | 10.44 | 2.69 | 27.06
Table 4: Ablations over the design choices for the proposed architecture on
GQA val set. Average attention over all heads in the last transformer layer is
used to evaluate the grounding performance. We perform ablation study with
C=32 caspules except rows 3-6 where we train the proposed architecture with
varying number of capsules. Ablation (1) no skip is our system without
residual connections, (2) w/skip is the full model. Rows 7 and 8 is the
ablation over weight initialization from pre-trained transformers for vision
(ViT) and text (BERT).

###  4.5 Ablations and Analyses

Impact of Residual Connections. We compare our full system with an ablated
variant without residual connections. We observe a drop in performance in
terms of overlap, but a slight increase in terms of IOU. Without residual
connections, pointing game accuracy is lower than with residual connections
(4.62% $\downarrow$). We conclude that using residual connections is
beneficial for pointing game.

Number of capsules. We ablate our system with varying number of capsules. We
train the proposed system with C=16, 24, 32, and 48 capsules. We observe that
increase in number of capsules not only decreases VQA accuracy, but also hurts
the overlap and IOU in terms of precision, recall and F1-score. Our best
method therefore uses 16 capsules with residual connections and pre-trained
from scratch.

ViT + BERT + Ours. ViLT and ALBEF initialize their image and text encoders
from ViT and/or BERT weights. Although our model is shallow than both models
(5 layers in modality specific encoders compared to 12 layers in ViLT and
ALBEF), we experimented to initialize our text encoder with BERT weights and
image encoder with ViT weights from last 5 layers. We find a gain in VQA
accuracy (58.86% vs. 56.65%) but it hurts the grounding performance.

Figure 4: Qualitative comparison: each row shows the input example, and the
last layer’s attention visualizations (averaged over all heads) with the
predicted answer from all methods. Column 1 shows the question and ground
truth answer, column 2 is the input image, column 3 shows the attention output
from our method, column 4-7 are results from the baselines. Blue box is the
ground truth bounding box for the answer object, orange boxes are the detected
regions from each system. We can see that ours is attending relevant answer
object with the plausible predicted answer even when the prediction mismatches
with the ground truth answer (row 3-5). In row 4, the question is vague;
therefore we can say, except LXMERT, all methods choose the correct answer.
ALBEF has attention spread over all image which explains the high recall it
achieves for overlap and IOU. Refer to section 4.6 for more details and
discussion. Best viewed in color.

###  4.6 Qualitative Analysis

In Figure 4, we show some qualitative comparison of our method with the
baseline methods. For all examples including the ones where our system
predicted the wrong answer, the grounding attention was correct (row 1,4 and
5). Also, the answers are plausible. For instance, in row 3, the correct
answer is ’aircraft’, and our method predicted it as ’airplane’ with the
correct localization. Overall, we notice that compared to our method, the
baselines were either attending most of the image (ALBEF in rows 1,3, and 5
which explains the high recall in table 2), or generate small attention maps
(MAC-Caps, ViLT) or looking at the wrong part of the image (LXMERT). More
examples and analysis are in the supplementary document.

##  5 Conclusion

In this work, we show the trade-off between VQA accuracy and the grounding
abilities of the existing SOTA transformer-based methods. We propose to use
text-guided capsule representation in combination with transformer encoder
layers. Our results demonstrate significant improvement over all baselines for
all grounding metrics. Extensive experiments demonstrate the effectiveness of
the proposed system over the baselines.

## References

  * [1] Abacha, A.B., Hasan, S.A., Datla, V.V., Liu, J., Demner-Fushman, D., Müller, H.: Vqa-med: Overview of the medical visual question answering task at imageclef 2019. CLEF (Working Notes) 2 (2019)
  * [2] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE international conference on computer vision. pp. 2425–2433 (2015)
  * [3] Arbelle, A., Doveh, S., Alfassy, A., Shtok, J., Lev, G., Schwartz, E., Kuehne, H., Levi, H.B., Sattigeri, P., Panda, R., et al.: Detector-free weakly supervised grounding by separation. arXiv preprint arXiv:2104.09829 (2021)
  * [4] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers (2021)
  * [5] Chen, K., Gao, J., Nevatia, R.: Knowledge aided consistency for weakly supervised phrase grounding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4042–4050 (2018)
  * [6] Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations (2019)
  * [7] Chen, Z., Ma, L., Luo, W., Wong, K.Y.K.: Weakly-supervised spatio-temporally grounding natural sentence in video. arXiv preprint arXiv:1906.02549 (2019)
  * [8] Das, A., Agrawal, H., Zitnick, C.L., Parikh, D., Batra, D.: Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2016)
  * [9] Datta, S., Sikka, K., Roy, A., Ahuja, K., Parikh, D., Divakaran, A.: Align2ground: Weakly supervised phrase grounding guided by image-caption alignment. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2601–2610 (2019)
  * [10] Desai, K., Johnson, J.: VirTex: Learning Visual Representations from Textual Annotations. In: CVPR (2021)
  * [11] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [12] Duan, S., Cao, J., Zhao, H.: Capsule-transformer for neural machine translation. arXiv preprint arXiv:2004.14649 (2020)
  * [13] Duarte, K., Rawat, Y., Shah, M.: Videocapsulenet: A simplified network for action detection. In: Advances in Neural Information Processing Systems. pp. 7610–7619 (2018)
  * [14] Duarte, K., Rawat, Y.S., Shah, M.: Capsulevos: Semi-supervised video object segmentation using capsule routing. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 8480–8489 (2019)
  * [15] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6904–6913 (2017)
  * [16] Gu, S., Feng, Y.: Improving multi-head attention with capsule networks. In: CCF International Conference on Natural Language Processing and Chinese Computing. pp. 314–326. Springer (2019)
  * [17] Gurari, D., Li, Q., Stangl, A.J., Guo, A., Lin, C., Grauman, K., Luo, J., Bigham, J.P.: Vizwiz grand challenge: Answering visual questions from blind people. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3608–3617 (2018)
  * [18] Hinton, G.: Some demonstrations of the effects of structural descriptions in mental imagery. Cognitive Science 3(3), 231–250 (1979)
  * [19] Hinton, G.: How to represent part-whole hierarchies in a neural network. arXiv preprint arXiv:2102.12627 (2021)
  * [20] Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: International conference on artificial neural networks. pp. 44–51. Springer (2011)
  * [21] Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with em routing. In: International conference on learning representations (2018)
  * [22] Huang, D.A., Buch, S., Dery, L., Garg, A., Fei-Fei, L., Niebles, J.C.: Finding" it": Weakly-supervised reference-aware visual grounding in instructional videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5948–5957 (2018)
  * [23] Huang, Z., Zeng, Z., Huang, Y., Liu, B., Fu, D., Fu, J.: Seeing out of the box: End-to-end pre-training for vision-language representation learning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [24] Huang, Z., Zeng, Z., Liu, B., Fu, D., Fu, J.: Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. CoRR abs/2004.00849 (2020), https://arxiv.org/abs/2004.00849
  * [25] Hudson, D.A., Manning, C.D.: Compositional attention networks for machine reasoning. International Conference on Learning Representations (ICLR) (2018)
  * [26] Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [27] Kahneman, D., Treisman, A., Gibbs, B.J.: The reviewing of object files: Object-specific integration of information. Cognitive psychology 24(2), 175–219 (1992)
  * [28] Khan, A.U., Kuehne, H., Duarte, K., Gan, C., Lobo, N., Shah, M.: Found a reason for me? weakly-supervised grounded visual question answering using capsules (2021)
  * [29] Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M.: Transformers in vision: A survey. arXiv preprint arXiv:2101.01169 (2021)
  * [30] Khan, S., Naseer, M., Hayat, M., Zamir, S.W., Khan, F.S., Shah, M.: Transformers in vision: A survey. arXiv preprint arXiv:2101.01169 (2021)
  * [31] Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 5583–5594. PMLR (18–24 Jul 2021), http://proceedings.mlr.press/v139/kim21k.html
  * [32] Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 5583–5594. PMLR (18–24 Jul 2021), http://proceedings.mlr.press/v139/kim21k.html
  * [33] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123(1), 32–73 (2017)
  * [34] LaLonde, R., Bagci, U.: Capsules for object segmentation. arXiv preprint arXiv:1804.04241 (2018)
  * [35] Li, G., Duan, N., Fang, Y., Gong, M., Jiang, D.: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 11336–11344 (2020)
  * [36] Li, J., Selvaraju, R.R., Gotmare, A.D., Joty, S., Xiong, C., Hoi, S.: Align before fuse: Vision and language representation learning with momentum distillation. In: NeurIPS (2021)
  * [37] Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019)
  * [38] Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al.: Oscar: Object-semantics aligned pre-training for vision-language tasks. In: European Conference on Computer Vision. pp. 121–137. Springer (2020)
  * [39] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [40] Liu, J., Lin, H., Liu, X., Xu, B., Ren, Y., Diao, Y., Yang, L.: Transformer-based capsule network for stock movement prediction. In: Proceedings of the First Workshop on Financial Technology and Natural Language Processing. pp. 66–73 (2019)
  * [41] Liu, Y., Wan, B., Ma, L., He, X.: Relation-aware instance refinement for weakly supervised visual grounding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5612–5621 (2021)
  * [42] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265 (2019)
  * [43] Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-task vision and language representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10437–10446 (2020)
  * [44] Lu, J., Yang, J., Batra, D., Parikh, D.: Hierarchical question-image co-attention for visual question answering. Advances in neural information processing systems 29 (2016)
  * [45] Mazzia, V., Salvetti, F., Chiaberge, M.: Efficient-capsnet: Capsule network with self-attention routing. arXiv preprint arXiv:2101.12491 (2021)
  * [46] Miech, A., Alayrac, J.B., Laptev, I., Sivic, J., Zisserman, A.: Thinking fast and slow: Efficient text-to-visual retrieval with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9826–9836 (2021)
  * [47] Mobiny, A., Cicalese, P.A., Nguyen, H.V.: Trans-caps: Transformer capsule networks with self-attention routing (2021), https://openreview.net/forum?id=BUPIRa1D2J
  * [48] Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.S., Wen, J.R.: Counterfactual vqa: A cause-effect look at language bias. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12700–12710 (2021)
  * [49] Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.M.O., Roth, S., Vulić, I., Gurevych, I.: xgqa: Cross-lingual visual question answering. arXiv preprint arXiv:2109.06082 (2021)
  * [50] Pucci, R., Micheloni, C., Martinel, N.: Self-attention agreement among capsules. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 272–280 (2021)
  * [51] Qiao, T., Dong, J., Xu, D.: Exploring human-like attention supervision in visual question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
  * [52] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision (2021)
  * [53] Ramakrishnan, S., Agrawal, A., Lee, S.: Overcoming language priors in visual question answering with adversarial regularization. arXiv preprint arXiv:1810.03649 (2018)
  * [54] Ribeiro, F.D.S., Duarte, K., Everett, M., Leontidis, G., Shah, M.: Learning with capsules: A survey. arXiv preprint arXiv:2206.02664 (2022)
  * [55] Riquelme, F., De Goyeneche, A., Zhang, Y., Niebles, J.C., Soto, A.: Explaining vqa predictions using visual grounding and a knowledge base. Image and Vision Computing 101, 103968 (2020)
  * [56] Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: NIPS (2017)
  * [57] Selvaraju, R.R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., Parikh, D.: Taking a hint: Leveraging explanations to make vision and language models more grounded. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2591–2600 (2019)
  * [58] Shi, J., Xu, J., Gong, B., Xu, C.: Not all frames are equal: Weakly-supervised video grounding with contextual similarity and visual clustering losses. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10444–10452 (2019)
  * [59] Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019)
  * [60] Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transformers. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (2019)
  * [61] Wang, L., Huang, J., Li, Y., Xu, K., Yang, Z., Yu, D.: Improving weakly supervised visual grounding by contrastive knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14090–14100 (2021)
  * [62] Whitehead, S., Wu, H., Ji, H., Feris, R., Saenko, K.: Separating skills and concepts for novel visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5632–5641 (June 2021)
  * [63] Wu, L., Liu, X., Liu, Q.: Centroid transformers: Learning to abstract with attention. arXiv preprint arXiv:2102.08606 (2021)
  * [64] Xiao, F., Sigal, L., Jae Lee, Y.: Weakly-supervised visual grounding of phrases with linguistic structures. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5945–5954 (2017)
  * [65] Yang, X., Liu, X., Jian, M., Gao, X., Wang, M.: Weakly-supervised video object grounding by exploring spatio-temporal contexts. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 1939–1947 (2020)
  * [66] Yang, Z., He, X., Gao, J., Deng, L., Smola, A.: Stacked attention networks for image question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 21–29 (2016)
  * [67] Zeng, X., Wang, Y., Chiu, T.Y., Bhattacharya, N., Gurari, D.: Vision skills needed to answer visual questions. Proceedings of the ACM on Human-Computer Interaction 4(CSCW2), 1–31 (2020)
  * [68] Zhan, L.M., Liu, B., Fan, L., Chen, J., Wu, X.M.: Medical visual question answering via conditional reasoning. In: Proceedings of the 28th ACM International Conference on Multimedia. pp. 2345–2354 (2020)
  * [69] Zhang, J., Bargal, S.A., Lin, Z., Brandt, J., Shen, X., Sclaroff, S.: Top-down neural attention by excitation backprop. International Journal of Computer Vision 126(10), 1084–1102 (2018)
  * [70] Zhang, S., Qu, L., You, S., Yang, Z., Zhang, J.: Automatic generation of grounded visual questions. arXiv preprint arXiv:1612.06530 (2016)
  * [71] Zhang, Y., Niebles, J.C., Soto, A.: Interpretable visual question answering by visual grounding from attention supervision mining. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 349–357 (2019). https://doi.org/10.1109/WACV.2019.00043
  * [72] Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L.: Visual7w: Grounded question answering in images. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4995–5004 (2016)

## Supplementary: Weakly Supervised Grounding for VQA in Vision-Language
Transformers

In this supplementary document, we further discuss about the proposed work as
follows:

  * •

Additional implementation details (section 0.A)

  * •

Additional architecture details (section 0.B)

  * •

Training objectives’ details (section 0.C)

  * •

Grounding Performance Evaluation (section 0.D)

  * •

Performance analysis w.r.t. varying detection threshold (section 0.E)

  * •

Grounding accuracy w.r.t. each head (section 0.F)

  * •

Results w.r.t. question type (section 0.G)

  * •

Entities represented by visual capsules (section 0.H)

  * •

Training parameters in our method (section 0.I)

  * •

VQA accuracy comparison (section 0.J)

  * •

Additional details about evaluation on VQA-HAT (section 0.K)

  * •

Qualitative Results (section 0.L)

##  Appendix 0.A Additional Implementation details

Here, we discuss the additional design choices used in our model. The capsules
use routing to agree or disagree about the presence of certain entities in the
input image. This decision is independent of the question. For instance, if an
image has a bus and an elephant in it, the question cannot affect what is in
the image. To this end, the capsule routing is performed only once in our
method. However, depending on the question being asked, we may require
selecting different entities from the image. We achieve this by doing text-
based capsule selection at each layer. The capsule selection layer $\phi$
(equation 3 in the main paper) has shared weights for all encoder layers. We
use 8 16GB AMD GPUs for pretraining; finetuning for GQA is performed on a
single 16GB GPU. To pretrain with 48 capsules, the batch size of 640 is used.

##  Appendix 0.B Additional Architectural details

Language Encoder: The language encoder $L_{e}$ is composed of $L$ transformer
encoder layers. Its input is a tokenized sentence $S_{l}$ of length $l$. The
language encoder $L_{e}$ takes the set of words tokens
$\\{[CLS],w_{1},w_{2},...,w_{l},[SEP]\\}$ as input, and outputs feature
representations
$\\{h^{i}_{cls},h^{i}_{T_{1}},h^{i}_{T_{2}},...,h^{i}_{T_{l}},h^{i}_{sep}\\}$
at every $i^{th}$ layer, where $h^{i}_{T_{k}}$ denotes the text feature for
the $k^{th}$ input word token ($k\in{1,2,...,l}$) from layer $i$. Intermediate
encoder layer $i$ takes output of previous layer $(i-1)$ as input. $[CLS]$
token is used as the sentence embedding in transformers [11]. Additionally, we
use it for capsule selection in visual encoder (section 3.3 in main).

Visual Encoder: The visual encoder $V_{e}$ has the same architecture as the
language encoder with the same dimension size and number of layers. The image
embeddings $X^{\prime}$ are transformed to visual capsules encodings $X_{c}$
and input to the visual encoder. Intermediate layers of visual encoder takes
selected visual capsules as residual connection to keep the capsule
representation intact while training the system. The final features output
$h^{L}_{v_{j}}$ of the visual encoder is used for token-level cross-modality
interactions in future steps. Where, $h^{L}_{v_{j}}$ is the feature output for
$j^{th}$ visual token ($j\in{1,2,...,hw}$) from the last layer $L$.

Feature Pooling The feature pooling layer takes text-based features and image-
based features as input and outputs a $d$ dimensional feature. This output
feature can be used as a pooled output for image-text matching and VQA tasks.
The feature pooling layer is a fully connected layer followed by a $tanh$
activation layer. We pretrain our system in two stages before finetuning for
VQA task. To be specific, during first stage pretraining , the input is the
concatenated features $[h^{L}_{cls},h^{L}_{img}]$ for special tokens from text
and image encoders; where $h^{L}_{cls}$, and $h^{L}_{img}$ are used as
aggregated features over text input and image input respectively. Let $f_{P}$
be the feature pooling layer, the pooled feature output $h_{{1}_{pooled}}$
will be as follows:

$h_{{1}_{pooled}}=f_{P}([h^{L}_{cls},h^{L}_{img}])$ (7)

During second stage pretraining, the concatenated features pair after cross
attention is indicated as $[h^{\hat{L}}_{cls},h^{\hat{L}}_{img}]$ and the
pooled feature output is denoted by $h_{{2}_{pooled}}$. The equation is as
follows:

$h_{{2}_{pooled}}=f_{P}(h^{\hat{L}}_{cls},h^{\hat{L}}_{img}])$ (8)

####  0.B.0.1 Cross-Attention Module

Given two input feature sequences (output $h^{L}_{T_{k}}$ from text encoder
and $h^{L}_{v_{j}}$ from image encoder), cross-attention module is a co-
attentional transformer which applies attention from one feature sequence to
the other by taking queries from first sequence and keys and values from the
second sequence, and vice versa. Multiple layers of these cross-attention
blocks can be stacked. The final text output feature $h^{\hat{L}}_{cls}$
corresponding to the $[CLS]$ token and final visual output feature
$h^{\hat{L}}_{img}$ corresponding to the $[IMG]$ token are used for
pretraining and finetuning the model. Where, $N_{c}$ is the number of layers
in cross-attention module and $\hat{L}=L+N_{c}$ denotes the depth of the model
in terms of number of layers.

##  Appendix 0.C Training objectives

Masked Language Modeling Masked Language Modeling is a self-supervised
language modeling task where a small percentage of words are masked before
giving the sentence as input to the language encoder. The task is to predict
the masked words using the context from other words in the sentence. This
self-supervised approach is very effective to learn strong text
representations [11]. The features output $h^{L}_{T_{k}}$ from language
encoder $L_{e}$ is used for training on this task. In the second stage,
instead of predicting missing words from solely text features, the masked word
is predicted from the visual-guided language features i.e., we take features
outputs $h^{\hat{L}}_{k}$ from the last text-based cross-attention layer.

Image-Text Matching (ITM) To predict whether the input pair of image-text
features is a matching pair or not, we take the output $h_{{1}_{pooled}}$ (eq.
7) from feature pooling layer and input to a fully connected layer which
outputs logits for each class: ’matching’ or ’non-matching’. At the second
pretraining stage, the output features corresponding to $[IMG]$ and $[CLS]$
token after cross-attentional module (each of dimension $d$) are used for
prediction. The pretraining head now uses $h_{{2}_{pooled}}$ (eq. 8) as input
for image-text matching task.

Visual Question Answering Inspired by [60], we also use VQA as one of our
pretraining tasks. We use Visual7W [72], GQA [26] and VQA [15] in our
pretraining. Like ITM, we take the pooled features from text and visual
encoders and input to a classifier. The classifier is comprised of two fully
connected layers. An activation function and layer norm is used between the
two layers. The final output is probability scores for each answer. In the
first stage of pretraining, $h_{{1}_{pooled}}$ is used as the pooled feature.
For second stage pretraining, pooled cross-modal feature output
$h_{{2}_{pooled}}$ is used for answer prediction. A separate softmax cross-
entropy loss function is used to optimize each of the above heads. We give
equal weights to each loss term during pretraining.

Finetuning parameters for the baselines. To finetune LXMERT, we use the same
training parameters as our method. ViLT is finetuned with batch size of 256
with lr=$1e-5$. ALBEF is finetuned with batch size of 16. Learning rate is
increased to $2e-5$ to speed up training for ALBEF. ALBEF and ViLT use the
maximum batch size which could fit in the GPU memory. All models are trained
on GQA for upto 10 epochs.

##  Appendix 0.D Grounding Performance Evaluation

####  0.D.0.1 Choice to use last layer’s attention for grounding:

It is common in the vision-language community to employ the last layer’s
analysis e.g., DINO[4] uses the last layer’s attention for the object
segmentation task without any specialized training objective or architecture.
We follow the protocol of previous works in the field for SOTA comparison to
allow for a fair evaluation, namely following MAC [25], MAC-Caps [28] with
mean (last) attention scores, ALBEF [36] by using the 8th and last layer with
Grad-CAM (GC) and attention scores (ATN) and ViLT [31] for the last layer
cosine (cos) and attention scores (see Tab. 2 in the main paper) and achieve
SOTA grounding performance. However, there is a possibility that some
intermediate layer does the better job at grounding such as ALBEF finds that
layer 8 in their model is good at grounding. Nevertheless, searching for the
best layer is expensive in terms of time and computational cost. Our approach
outperforms on grounding even when evaluated for the last layer only. This
choice also eliminates the need to search for the best grounding layer within
each model and well suited to test the systems for unseen data.

| Overlap |  | IOU
---|---|---|---
Obj. label | P | R | F1 |  | P | R | F1
Answer (A) | 17.64 | 89.81 | 29.49 |  | 2.05 | 10.45 | 3.42
Question (Q) | 49.57 | 81.86 | 61.75 |  | 4.01 | 6.48 | 4.95
Table 5:  GradCAM results for our model. Compare to Tab. 2 in the paper.

Ours + Grad-CAM: To compare with ALBEF, we also evaluated our system with
Grad-CAM output of the last layer. We observe $\approx 2-4\%\uparrow$ increase
in overlap F1-score for both Q & A and a ($\approx 0.5-1.01\%\downarrow$)
decrease in IOU F1-score still achieving better grounding results than the
baselines (see Tab. 5 and Tab. 2 (main)).

####  0.D.0.2 Generating heatmaps using ViLT demo:

The demo code visualizes word-to-patch attention for the matching image-
caption pair. For VQA grounding, we consider question-to-image attention
(attention from [CLS] token to visual tokens). The provided code for optimal
transport algorithm leads to numerical instability for the question token
([CLS]) generating NaN. Hence, we used the cosine scores (computed before
optimal transport) as well as raw attention to generate heatmaps. Heatmaps are
generated with the same post processing as provided in the demo. To verify
this, ($<question\\_id>$, $<image\\_id>$) pairs for Fig.4 (main) are:
{(‘00798998’, ‘2356417’), (‘02451905’, ‘2386586’), (‘00653991’, ‘2324955’),
(‘00511505’, ‘2410567’), (‘01782610’, ‘2409395’) }.

|  |
---|---|---
|  |

(a) (b)

(c) (d)

(e) (f)

Figure 5: Comparison with baselines for varying overlap and IOU detection
threshold from 0.05 to 0.95. Plots (a), (b), and (c) show the results for
varying detection threshold for overlap in terms of precision, recall, and
F1-score respectively. Plots (d), (e), and (f) are the results for comparison
when varying IOU threshold in terms of precision, recall, and F1-score
respectively. Our method is significantly outperforming the baselines for all
values of overlap thresholds in terms of precision, and subsequently
F1-scores. For IOU, the proposed method is doing well for threshold values as
high as 0.8 in terms of precision and F1-score, whereas, IOU-Recall is
comparable to ALBEF.

|  |
---|---|---

(a) (b)

(c)

Figure 6: Grounding performance from the proposed model (C=16) for each head in the last cross-attention layer. (a) reports overlap accuracies in terms of precision, recall, and F1-score; (b) shows IOU in terms of precision, recall, and F1-score; and (c) shows pointing game accuracy for each head. Overall, head 7 and head 10 show best grounding performance among all heads. For pointing game, head 7 achieves the highest accuracy of 23.08%. Using the proposed way to evaluate the pointing game performance, i.e., clustering over maximum points from all heads, improves pointing game accuracy significantly (34.59%).  |  |  |  | Overlap |  | IOU
---|---|---|---|---|---|---
Q Type | Example | Method | Acc. | P | R | F1 |  | P | R | F1
Open | How is the weather in the image? | no-caps | 62.43 | 23.03 | 46.02 | 30.70 |  | 4.83 | 9.66 | 6.44
ours | 56.65 | 43.06 | 85.57 | 57.29 |  | 6.62 | 13.24 | 8.83
Binary | Is it cloudy today? | no-caps | 62.43 | 33.19 | 66.10 | 44.20 |  | 7.13 | 14.25 | 9.50
ours | 56.65 | 42.54 | 84.67 | 56.63 |  | 12.12 | 24.23 | 16.15
Query | What kind of fruit is on the table? | no-caps | 62.43 | 33.19 | 66.10 | 44.20 |  | 4.83 | 9.66 | 6.44
ours | 56.65 | 43.06 | 85.57 | 57.29 |  | 6.62 | 13.24 | 8.83
Compare | Who is taller, the boy or the girl? | no-caps | 62.43 | 9.96 | 19.91 | 13.27 |  | 1.11 | 2.21 | 1.47
ours | 56.65 | 37.83 | 75.00 | 50.29 |  | 2.43 | 4.87 | 3.24
Choose | Is it sunny or cloudy? | no-caps | 62.43 | 24.62 | 49.19 | 32.82 |  | 7.86 | 15.72 | 10.48
ours | 56.65 | 43.11 | 85.85 | 57.40 |  | 13.29 | 26.59 | 17.73
Category | What kind of fruit is it, an apple or a banana? | no-caps | 62.43 | 30.14 | 60.10 | 40.14 |  | 8.66 | 17.32 | 11.54
ours | 56.65 | 42.90 | 85.49 | 57.13 |  | 11.73 | 23.46 | 15.64
Relation | Is there an apple on the black table? | no-caps | 62.43 | 33.45 | 66.60 | 44.53 |  | 4.16 | 8.31 | 5.54
ours | 56.65 | 43.10 | 85.60 | 57.33 |  | 5.84 | 11.68 | 7.79
Attribute | what color is the apple? | no-caps | 62.43 | 9.96 | 19.91 | 13.27 |  | 1.11 | 2.21 | 1.47
ours | 56.65 | 37.83 | 75.00 | 50.29 |  | 2.43 | 4.87 | 3.24
Table 6: Comparison of our backbone model with no capsules (no-caps) and the
proposed model with 16 capsules (Ours (C=16)). Results are shown w.r.t. each
question type. Adding capsules to the backbone model significantly improves
the grounding performance for all question types.

##  Appendix 0.E Performance analysis w.r.t. varying detection threshold

We use detection threshold=0.5 for all our results in the submitted paper. For
overlap, a detection is considered to be a true positive when the overlap
between the ground truth box and the predicted region is greater than 0.5.
Similarly, a detected region with an IOU of greater than 0.5 over a ground
truth bounding box is considered a true positive for IOU. In figure 5, we
study the impact of having a very low detection threshold vs. employing high
thresholds by varying the threshold from 0.05 upto 0.95. We observe that the
proposed method is robust to detection threshold for the overlap metric in
terms of precision and F1-score even for the very high threshold of 0.95. For
IOU, we also perform well for precision and F1-score. For IOU in terms of
recall, our method and ALBEF show comparable results.

##  Appendix 0.F Grounding accuracy w.r.t. each head

Grounding accuracy for individual heads in the last cross-attentional layer
are shown in figure 6. The results are reported in terms of precision, recall,
and F1-score for overlap and IOU. For pointing game, the maximum point over
the attention map produced from each head is used to evaluate the per-head
pointing game accuracy. Using clustering over the points obtained from each
head outperforms the best performing head by $\uparrow 11.51\%$ (best head:
23.08% vs. clustering: 34.59%).

##  Appendix 0.G Results w.r.t. question type

Table 6 shows grounding results of our best model for different question
types. GQA has questions classified with respect to structural type and
semantic type. We compare our model with our backbone model which uses no
capsules. Our system outperforms over all question types for both overlap and
IOU particularly for question type “compare”, “choose”, and “attribute”.
Examples for each question type are provided in table 6.Refer to GQA [26] for
more details about the question types present in this dataset.

|  |
---|---|---
|  |
|  |

(a) capsule 1: surfing, outdoor sports (b) capsule2: food

(c) capsule 4: toys, zebra, horse (d) capsule 5: elephants, wild animals

(e) capsule 6: sports, tennis (f) capsule 9: pizza

(g) capsule 11: humans (h) capsule 13: buildings

(i) capsule 15: bathroom, trains

Figure 7: Images represented by individual capsules. Here, we show the group
of images where a certain capsule has the highest activation, e.g., capsule 9
has the highest activation when there is pizza in the image.

##  Appendix 0.H Entities represented by capsules

Since, we do not use class labels to train the capsules, and use VQA
supervision instead for training the whole system, it is hard to guess which
entity is being represented by each capsule. To examine what individual
capsules are learning, we take the average over capsule activations for each
spatial location resulting in a vector of dimension C (C=number of capsules).
Each feature in that C-dimensional vector shows the average activation
(presence probability) of an individual capsule for that image. The highest
activated capsule is used to sort the images into C groups. Figure 7 shows the
images where a given capsule had the highest activation. In the figure, we can
see different capsules are focused on different types of images, e.g., capsule
1 is mostly focused on outdoor sports like surfing. Since these capsule
representations are learned in a weakly-supervised manner, they show
overlapping behavior over certain image classes. Some of them exhibit an
interesting behavior. For instance, while capsule 2 is focused on food items,
capsule 9 is fond of pizza; capsule 5 has learned what an elephant looks like,
but also good at identifying giraffes and cows in the wild; capsule 13 is
focused on buildings. We used our best model with C=16 capsules for these
visualizations.

Method | depth (#transformer layers) | #Params (M)
---|---|---
LXMERT [60] | 12 | 239.8
ALBEF [36] | 12 | 209.5
ViLT [31] | 12 | 87.4
Ours | 7 | 141.0
Table 7: Number of parameters in all transformer-based methods.

##  Appendix 0.I No. of training parameters

We compare the proposed model with other transformer-based methods in table 7.
Our proposed system is shallower than the baseline methods using 5 layers in
each modality-specific encoders followed by 2 cross-attentional layers. We
denote the length of a vertical stack of transformer layers as the model’s
depth. We follow [31, 60] and consider one single modality layer as $1/2$ of a
multimodal layer. Hence, the proposed model has the depth=7 compared to the
baselines with depth=12. In comparison to other transformer-methods, the
proposed system uses less parameters ($\approx$ 141M) than LXMERT [60]
(239.8M) and ALBEF [36] (209.5M). ViLT has the least number of parameters
(87.4M). However, it is a single stream model compared to all other two-stream
methods considered for this work including the proposed architecture. We
excluded the text embedding layer when computing the number of training
parameters since it is shared among all vision-language transformers which are
used in this study [31].

Answer Plausibility | ALBEF | ViLT | Ours
---|---|---|---
for all | 92.12 | 92.28 | 92.30
for mispredicted | 85.14 | 86.35 | 87.15
Table 8: Plausibility comparison on GQA-val set with ALBEF and ViLT for all
question-answer pairs and the mispredicted question-answer pairs. We perform
on par (even slightly better) than the baselines in terms of the predicted
answer’s plausibility. This verifies the system is predicting reasonable
answers in the real-world context.

##  Appendix 0.J VQA accuracy of ours vs. baselines:

Our proposed system while achieving better VQA accuracy than previous
grounding SOTA on GQA dataset (MAC-Caps [28]]) and LXMERT (a transformer-based
model with object-detection), performs lower than ViLT and ALBEF. We attribute
this to two reasons: 1) Less training data – ViLT and ALBEF are using SBU and
GCC additionally with strong data augmentations, so we assume that using
additional data and comparable resources for training would improve our
accuracy as well. 2) Considering failure cases in more detail, we find that
the lower accuracy is mainly driven by semantically correct, but literally
wrong answers such as girl vs women or herd vs cow (see examples in Fig. 8,
11, and 10). The answer prediction despite being reasonable is incorrect in
terms of language mismatch with the ground truth. It could be possible that
capsules help to prevent dataset biases, as they regularize and constrain the
training and therefore suppress "shortcuts" based on dataset noise. To
validate this further, we compute the plausibility metric for all questions as
well as incorrect predictions. Plausiblity measures that an answer is
reasonable in the real-world context e.g., it is unlikely to see a ‘blue’
apple in real-world. We perform on par with ViLT and ALBEF for all predicted
answers. When compared on the mispredicted questions for all three methods,
our system predicts 2% more plausible answers than ALBEF and 0.8% better than
ViLT (see table 8). This study maintains the observation about the predicted
answer while being reasonable in the real-world is considered incorrect in
terms of exact match with the ground truth consequently leading to decreased
VQA accuracy.

##  Appendix 0.K Additional details about evaluation on VQA-HAT dataset

VQA-HAT dataset provides human attention maps for VQA task. This dataset is
based on VQA v1.0 dataset and provides 1473 QA pairs with 488 images in
validation set. To evaluate on this dataset, we train our system on VQA v1.0
and evaluate on VQA-HAT validation set. The answer vocabulary of VQA train set
has a long tail distribution. We follow previous works [2, 8] and use 1000
most frequent answers. We first combine training (248,349 QA pairs) and
validation data (121,512 QA pairs) to get a total of 368487 QA pairs. We then
filter out the questions with out-of-vocabulary answers (answer vocab size is
kept 1K) resulting in 318827 QA pairs. We separate out 10K QA pairs from the
training set (after above mentioned question filtering) and use it as a
validation set to pick our best model. We therefore use 308K QA pairs from VQA
v1.0 train and val set for finetuning our pretrained backbone with 16
capsules. The learning parameters used for this training are lr=4e-5, batch
size=64, with bert optimizer and trained for 20 epochs. The best model on
validation set is used for evaluation.

##  Appendix 0.L Qualitative Results

In figure 8 and 9, we show more examples for qualitative comparison with
baselines. Our system consistently produces correct grounding attention when
compared to the baselines. In figure 10, we show some failure cases for our
system in terms of grounding output as well as answer prediction. For
grounding failure (in terms of IOU with the groundtruth box), we observe that
the system’s attention is cogent. For instance, in the top left example, for
the question where is the giraffe?, the system is looking at the surroundings
of giraffe and predicting the answer zoo. In the right two examples, the
system is grounding correctly even generating reasonable answers. However,
these answers are considered incorrect in terms of language mismatch with the
groundtruth answer (herd vs. cow and beach vs. sand). Finally, in figure 11,
we present more qualitative examples from our system.

Figure 8: More qualitative examples where the model predicted the answer
correctly with attention (with detected orange boxes) on the correct image
regions (blue boxes). Best viewed in color.  Figure 9: Qualitative comparison
for examples where our model predicted the wrong answer. The attention is over
the correct image region.  Figure 10: Some failure cases for our system. Left
two examples show the failure of grounding, right two examples are failure
cases in terms of answer prediction. However, both the grounding output and
the answers are plausible.

Figure 11: More qualitative examples from our system.