# Predicting Eye Gaze Location on Websites

Ciheng Zhang1, Decky Aspandi2, Steffen Staab23
1Institute of Industrial Automation and Software Engineering, University of
Stuttgart, Stuttgart, Germany
2Institute for Parallel and Distributed Systems, University of Stuttgart,
Stuttgart, Germany
3Web and Internet Science, University of Southampton, Southampton, United
Kingdom
zhang.ciheng@dotted-eighth.com,{decky.aspandi-latif<EMAIL_ADDRESS>stuttgart.de

###### Abstract

World-wide-web, with website and webpage as a main interface, facilitates
dissemination of important information. Hence it is crucial to optimize
webpage design for better user interaction, which is primarily done by
analyzing users’ behavior, especially users’ eye-gaze locations on the
webpage. However, gathering these data is still considered to be labor and
time intensive. In this work, we enable the development of automatic eye-gaze
estimations given webpage screenshots as input by curating of a unified
dataset that consists of webpage screenshots, eye-gaze heatmap and website’s
layout information in the form of image and text masks. Our curated dataset
allows us to propose a deep learning-based model that leverages on both
webpage screenshot and content information (image and text spatial location),
which are then combined through attention mechanism for effective eye-gaze
prediction. In our experiment, we show benefits of careful fine-tuning using
our unified dataset to improve accuracy of eye-gaze predictions. We further
observe the capability of our model to focus on targeted areas (images and
text) to achieve accurate eye-gaze area predictions. Finally, comparison with
other alternatives shows state-of-the-art result of our approach, establishing
a benchmark for webpage based eye-gaze prediction task.

##  1 INTRODUCTION

Analysis of user behavior during their interaction with the website (and
contained webpages) is important to evaluate the website overall quality. This
information can be used to create more optimized webpages for better
interactions, as such, there are needs to characterize these behaviors. Some
of the characteristics include users’ tendency to focus on certain areas (e.g.
upper left corner) during browsing [Shen and Zhao, 2014]. Other important
characteristics can be defined from users’ eye-gaze data, which normally
represents visual attention of the users during website interaction. This data
is usually acquired from human pupil-retinal location and movement relative to
the object of interest that allows one to pinpoint exact webpage area, where
users focus during their interactions and its relations with overall webpage
structure. Although this information is crucial to create a more optimized
website for better interactions, However, acquiring these eye-gaze data from
every user during their browsing duration is difficult, given the complexity
of the acquisition setting. Thus there is currently a need for an automatic
approach to predict these eye-gaze locations given an observation of the
webpages.

In computer vision field, saliency prediction task has been extensively
investigated that allows estimation of people’s attention given an image. The
main task is to estimate saliency map, which highlights first location (or
region) where observers’ eyes focus, with photographs of natural scenes [Zhang
et al., 2018, Kroner et al., 2020] commonly used as input. Several methods
have been developed so far to solve this task, including machine learning [Hou
and Zhang, 2008, Li et al., 2012] and recently deep learning based approaches
[Kroner et al., 2020] with quite high accuracy achieved. With this progress,
there is an opportunity to adopt these automatic, visual based saliency
predictors for eye-gaze predictions task, by representing input observation in
a form of webpage screenshot. This enables adaptation of task objective for
eye-gaze heatmap prediction, in lieu of saliency map, with both predicted maps
representing area where most people (or in our case users) attend.

The challenges however are differences between saliency and eye-gaze
prediction tasks in terms of content, structure and layout of input images.
For instance, there is not any concept of depth on webpage screenshot as
opposed to natural images, an aspect that is highly utilized for general
visual saliency estimation. In addition, high contrast and varied colored
areas on natural image are commonly regarded as saliency area, which may not
be the case for webpage screenshots, given that user eye-gaze locations are
highly dependent on the type of interactions during website browsing. This
problem can be rectified by the use of a data-driven approach [Pan and Yang,
2010], which can be made feasible through fine-tuning using a specialized
dataset on a particular task. However, this attempt is currently hampered by
the lack of such dataset.

In this work, we curate a generalized dataset of eye-gaze prediction from
webpage screenshots to enable effective training for machine and deep learning
approaches. Using this dataset, we propose a deep learning based method that
incorporates image and textual locations of webpage through mask modalities
and combines them with attention fusion mechanism. We then evaluate the impact
of transfer learning, including comparison with other alternatives, to
establish a benchmark for this task. Specifically, the contributions of this
work are:

  1. 1.

The establishment of a unified benchmark dataset for eye-gaze detection from
webpage screenshots, derived from website and user interactions data.

  2. 2.

A novel deep learning-based and multi-modal eye-gaze detector with internal
attention that leverages characteristics of input contents and importance of
each stream of modality for accurate eye-gaze predictions.

  3. 3.

Benchmark results of automatic eye-gaze location estimators and our state-of-
the-art results for eye-gaze detection task given webpage screenshot inputs.

##  2 RELATED WORK

An early example of work analyzing website content is done by Shen and Zhao
et.al. [Shen and Zhao, 2014] where three types of webpages are analyzed: Text-
based, Pictorial-based, and Mixed (combination of Text and Pictorial)
websites. The authors show that some attention characteristics of users during
their interactions with webpage exist, with main finding that users usually
pay more attention to some relevant parts of websites (such as left-top corner
of a website) and their tendency to focus on areas where large images are
present. Furthermore, on the websites from ‘Text’ category, their preference
to focus on certain parts of websites (middle left and bottom left regions) is
perceived. Lastly, they propose multi-kernel machine learning inferences for
eye-gaze heatmap predictions (which represent visual attention of users) for
their developed website eye-gaze dataset of Fixations in Webpage Images
dataset (FiWI).

In computer vision field, the task of locating (or predicting) users’
attention to natural image is commonly called as saliency prediction. These
tasks are commonly solved by utilising machine learning techniques, given
their automation capability. An earliest example of this method is Incremental
Coding Length (ICL) [Hou and Zhang, 2008] which aims to predict the activation
of saliency location by measuring perspective entropy gain of each input
feature (several image patches) as a linear combination of sparse coding basis
function. Other algorithm of Context-Aware Saliency Detection (CASD)
capitalises on the concept of dominant objects as additional context to
improve their prediction [Goferman et al., 2011]. Furthermore, D Houx et.al.
[Houx et al., 2012] propose an approach that aims to solve figure-ground
separation problem for prediction. They use a binary and holistic image
descriptor of Image Signature, which is defined as sign function of Discrete
Cosine Transform (DCT) of an image as additional input. Subsequently,
Hypercomplex Fourier Transform (HFT) [Li et al., 2012] is used to transform
input image to frequency domain to estimate saliency map. One relevant work
utilizes deep learning based method, given their accurate estimations and
ability to leverage huge datasets size (that are increasingly present). This
approach is based on Encoder and Decoder structure with a pre-trained VGG
network to predict saliency maps [Kroner et al., 2020]. Finally in recent
years, several researchers extend the saliency approach to work with image and
depth input. One example is the work from Zhou et. al. [Zhou et al., 2021]
that uses Hierarchical Multimodal Fusion Network to process both RGB (Red,
Gren and Blue) colours image and depth maps (one channel) as additional
modality for the input, which results in more accurate gaze maps prediction.

Even though all of the described methods work for general visual saliency
predictions, however, their capability to predict users’ eye-gaze location on
webpage is not yet investigated, given lack of dataset available. Thus in this
work, we propose to create a specialized webpage based eye-gaze prediction
datasets to allow for the development of automatic eye-gaze predictions of
webpage screenshot inputs, then utilize it to develop our deep learning based
eye-gaze location predictor.

Figure 1: The flow of dataset generation.

##  3 METHODOLOGY

###  3.1 Dataset Gathering and Processing

In this work, we search, process and curate a generalized eye-gaze dataset
from available datasets in literature. Here, we focus on three publicly
available user-website interaction datasets (where eye-gaze and webpage
screenshots are present): GazeMining [Menges, 2020], Contrastive Website
[Aspandi et al., 2022a], and Fixations in Webpage Images (FiWI) dataset [Shen
and Zhao, 2014]. In this section, we provide a short description of datasets,
and proceed with description of pre-processing algorithm that we conduct to
produce our unified dataset. The processed dataset is available at
111https://doi.org/10.18419/darus-3251 and available after request.

  * •

GazeMining is a dataset of video and interaction recordings on dynamic
webpages [Menges, 2020]. In this work, the authors focused on websites which
contain constantly changing scenes (thus dynamic). The data was collected on
March 2019 from four participants who interacted with 12 different websites.

  * •

Contrastive Website dataset is a recent website interaction-based dataset,
where participants were asked to visit two sets of websites (each of four) and
performed respective tasks (flight planning, route search, new searching and
online shopping) resulting on more than 160 sessions [Aspandi et al., 2022a].

  * •

FiWI dataset [Shen and Zhao, 2014] is a dataset that focuses on website-based
saliency prediction. However, it is small in term of webpage screenshot
availability (only 149 images) compared to two previous datasets that prevents
its use for larger scale evaluations (this is especially true for a deep
learning model). Therefore FiWI dataset is mostly (and commonly) utilized as
comparative evaluation with other models, but not for model training.

Figure 2: Structure of Multi Mask Input Attentional Network.

Figure 1 shows the flow of our dataset curation, data pre-processing and the
results of the generalized dataset. The first step of pre-processing is
elimination of empty screenshots (which exist in small quantities on
GazeMining dataset), i.e. no screenshots are present, which can come due to
rapid sampling during acquisition. Secondly, due to dynamic nature of the
observed websites, webpages’ static parts (that consist of all unchanged,
rendered webpages’ elements during interaction time) are removed (or
blackened) on original datasets, which necessitates us to find and collect
only recorded screenshot - thus removing irrelevant observations (black
parts). Thirdly, we remove the duplicate webpage screenshots of both datasets
for efficiency and generate respective locations of Image and Text as
independent layers (called Image and Text Mask, which we will detail in
Section 3.2.1). Lastly, eye-gaze heatmap layers are generated (as ground
truth) with respect to duration of each observed eye-gaze. This pipeline is
applied to all datasets, with the exception of Contrastive Website, where
first and second steps are skipped, and for FiWI where only last step is
necessary. In the end, our unified dataset includes four sub-folders (data,
label, Imagemask, TextMask) with a total of 3119 screenshot examples (1546
samples or 49.6% from GazeMining, 1424 instances or 45.7% from Contrastive
Website dataset and 149 examples or 4.7% from FiWI) along with the associated
ground truths.

###  3.2 Multi Mask Input Attentional Network (MMIAN) for Eye Gaze Prediction

Our pre-processed datasets allows us to propose a deep learning based model to
benefit from sizable number of observations [Rodriguez-Diaz et al., 2021].
Here we propose a Multi Mask Input Attentional Network (MMIAN) that aims to
predict eye gaze location given a webpage screenshot as input. Specifically,
given input webpage screenshot images of $X_{i..n}$, with $X$ as 2D matrix of
webpage screenshot image, and n as number of batch size, MMIAN estimates eye-
gaze locations of $\hat{Y}_{i..n}$, where each $\hat{Y}$ is a 2D matrix of
eye-gaze heatmap, with each cell value represents probability of eye-gaze
locations. The structure of MMIAN follows encoder-decoder scheme that is
inspired by Multi-scale Information Network (MSI-Model) [Kroner et al., 2020]
consisting of several modules: input and Mask Encoder, Mask Generator,
Attentional Feature Fusion, Atrous Spatial Pyramid Pooling (ASPP) Module and
Decoder. In this work, we further propose several important modifications:

  1. 1.

Incorporation of several input masks to include webpage content information of
textual and images spatial location.

  2. 2.

Addition of attention mechanisms for effective fusion of input modalities.

Overall architecture of MMIAN can be seen in Figure 2. Specifically, an input
webpage screenshot image is passed into mask generator to produce an image
mask and text mask. Then both masks are concatenated and fed into Mask Encoder
to extract relevant features, while input image is simultaneously fed into
input Encoder. Feature Fusion module then fuses extracted mask features and
input features with several attentional modules and combines them in a
conjoint block. This feature block is then fed into ASPP part to enlarge field
of views, capturing different resolution views from the input, thus producing
richer feature representations. These features are then passed through series
of up-sampling layers (Decoder) creating an eye-gaze saliency map as a final
result. The implementation code of our model is available on our
repository222https://github.com/ZackCHZhang/WebToGaze.

####  3.2.1 Input Masks Generator

(a)

(b)

(c)

Figure 3: One case of input and text mask generation. Figure (a) is an example
of webpage screenshot, Figure (b) and (c) are the generated text mask and
image mask respectively.

It has been observed that webpage layout (which in principle is arrangement of
images and texts) is an important aspect of webpage [Shen and Zhao, 2014]. To
benefit from this information, we propose to incorporate spatial locations of
both image and text on websites with mask representations: image and text
masks. These masks are represented in form of a matrix, where each cell is
activated in the presence of both image and text in webpage screenshot.

  * •

Image Mask Generation:  For image location recognition, we use method from M.
Xie et al. [Xie et al., 2020] which produces bounding-box locations of an
input image in a binary map (i.e. map value is set to 1 where image is
present, otherwise 0).

  * •

Text Mask Generation: we use one of available optical character recognition
(OCR) methods of Efficient and Accuracy Scene Text detection (EAST) [Zhou et
al., 2017] to locate location of text of webpage screenshot input. Similar to
Image Mask, this process generates corresponding Text Mask in Binary map
format.

One example of generated masks from our curated dataset can be seen in Figure
3. We can see that both of masks contain quite accurate locations of both
Image and Text on an input webpage screenshot. Even though some mild
imperfections occur (i.e. some buttons can be recognized as images, it does
not fully detect locations of certain images due to small size, and some text
within images are included on text masks), however in general the locations of
both image and text on input webpage screenshots are properly recognized.
These masks are then concatenated to be fed into Mask Encoder part.

####  3.2.2 Input and Mask Encoder

Encoder part is a series of Convolutional Neural Network that aims to extract
visual features from input matrix, which is original web screenshot image and
concatenated generated Masks, for Input and Mask Encoder respectively. Both
Encoders are based on VGG16 [Simonyan and Zisserman, 2014] architectures with
last fully connected layer removed, and further reductions of convolution
layers (five convolutional layers with Relu and two max-polling layers)
applied for Mask Encoder. Input and Mask encoder then produce MaskFeatures and
Screenshot Features to be combined through Multi-Modal Attentional Fusion
module.

####  3.2.3 Multi-modal Attentional Fusion

We employ multi-modal inputs (that consists of web-page screenshot and
generated masks) to benefit from additional stream of information [Zhou et
al., 2021, Aspandi et al., 2020]. Moreover, we introduce attentional fusion
mechanism for more effective fusion of these modalities, as opposed to simple
fusion operations of summation or concatenation [Aspandi et al., 2022b], and
further solving the problems of inconsistent input semantics and scales [Dai
et al., 2021]. An example of attentional feature fusion block (AFF) can be
seen in Figure 4 where it receives two streams of inputs (denoted as F1 and
F2) and initially fuses these input features through matrix addition (denoted
as $\bigoplus$). The result from addition of internal Multi-scale Channel
Attention module (that aggregates channel context by a series of point-wise
convolutions) and a sigmoid function generates a fusion weight. This fusion
weight and its another counterpart (where it is negated by one) are multiplied
(denoted as $\bigotimes$) by each of original inputs respectively, and then
added together ($\bigoplus$ symbol) to produce their weighted average as fused
feature of Z. This fusion process is applied to the received MaskFeatures and
Screenshot Features three times, to allow for observation of different scales
of respective features (these operations are marked as circled A in Figure 2).
The resultants of the fused features are then concatenated to a conjoint block
for subsequent pipelines.

Figure 4: Left is Attentional Feature Fusion block, right is Multi-scale
Channel Attention module [Dai et al., 2021].

####  3.2.4 Atrous Spatial Pyramid Pooling Module (ASPP)

ASPP module is made by superimposing different atrous convolutions [Chen et
al., 2017] filters to obtain features from larger receptive fields. This is
beneficial given that it is common for webpage layouts to be modularized into
a variety of different sub-layouts. This thus increases receptive fields (or
field-of-view) of internal convolutional layers (where the result from several
filters are combined together) enables us to capture such layout
configurations. In our ASPP module, we use dilation rates of 4, 8, and 12,
with number of $1\times 1$ filter to be 256 to ensure kernel size
compatibility. This module is applied to conjoint block input, and subsequent
features are obtained for decoding operation.

####  3.2.5 Output Decoder

Decoder part consists of a series of convolution layers with up-sampling to
decode input features, generating final prediction of eye-gaze heatmap
location (in the form of a 2D tensor and in binary mode). Specifically, number
of up-sampling blocks consisting of bilinear scaling operations (each
sequentially doubles the number of input tensor) and a subsequent
convolutional layer with kernel size $3\times 3$ are used. With this module,
we provide the features generated from ASPP as input features and final eye-
gaze heatmap predictions are obtained. Finally, we convert binary heatmap to
gray-scale format, to conform to the original scale of eye-gaze heatmap from
ground-truth map.

###  3.3 Training Procedure

To evaluate the impact of training of models utilizing webpage screenshot
based eye-gaze prediction datasets, first we perform training process using an
original saliency prediction model of Kroner’s Model (MSI-Model) [Kroner et
al., 2020]. This model has been pre-trained on general visual saliency dataset
(SALICON) [Jiang et al., 2015], which we call as P-MSI, that serves as
baseline. Given this pre-trained model, we perform fine-tuning using our pre-
processed datasets (GazeMining and Contractive Website dataset) producing FT-
MSI-GM and FT-MSI-CW respectively, and evaluate observed accuracy gains. We
did the model fine-tuning instead of performing training from scratch to
maximize the potential accuracy obtained - we empirically found that the
overall obtained accuracy of the latter to be substantially lower.

Additionally, we propose to evaluate the impact of training using a combined
dataset (i.e. we combine examples of both of GazeMining and Contrastive
Website) by further training MSI-Model with this merged dataset, that we name
as FT-MSI-CMB. The training is conducted until convergence and no further
accuracy improvement is perceived. Afterward, we proceed to training stage of
our proposed model of MMIAN by first transferring Encoder and Decoder’s weight
from best performing models of the previous step, and initialized weights of
both Mask Encoder and ASPP by zero-mean uniform distribution.

All of the training is conducted by minimizing Kullback-Leibler divergence
(KLD) loss function as shown in Equation 1, with $Y$ indicating ground truth
and $\epsilon$ is a regularization constant to guarantee that denominator is
not zero. In this loss, estimation of saliency maps can be regarded as a
probability distribution prediction task, as formulated by Jetley et. al.
[Jetley et al., 2016]. The output of estimator is normalized to a non-negative
number, with KLD value used to measure level of differences between
predictions and ground truth. Finally, Adam optimizer [Kingma and Ba, 2014]
with a learning rate of $10^{-4}$ is used for overall optimization.

$D_{KL}(\hat{Y}||Y)=\sum_{i}~{}Y_{i}~{}ln(\epsilon+\frac{Y_{i}}{\epsilon+\hat{Y}_{i}})$
(1)

###  3.4 Experiment Setting

####  3.4.1 Dataset Experiment Setting

In this experiment, all of three pre-processed datasets (cf. Section 3.1) are
utilized, with training, validation and test instance numbers are shown in
Table 1. Specifically, both GazeMining and Contrastive Website datasets are
divided with 60%, 20% and 20% of samples for training, validation set, and
test set respectively. Whereas for FiWI dataset [Shen and Zhao, 2014], all
samples (of 149 images) are used for testing.

Splits | GazeMining | Contrastive Website | FiWI
---|---|---|---
Training | 928 | 868 | -
Validation | 309 | 278 | -
Test | 309 | 278 | 149

Table 1: Training, validation and test set for each dataset.

####  3.4.2 Quantitative Metrics

We use three quantitative metrics of Area under Receiver Operating
Characteristic curve (AUC), Normalized Scanpath Saliency (NSS), and Person’s
Correlation Coefficient (CC) to judge the quality of models’ prediction.

(a)

(b)

(c)

Figure 5: Comparison between pre-trained results from P-MSI (baseline) and FT-
MSI-CW (Fine-tuned). Figure (a) shows webpage screenshot with ground truth
(eye-gaze heatmap). Figure (b) and (c) show predictions from P-MSI and FT-MSI-
CW.

  * •

NSS is commonly used for general saliency prediction tasks as a direct
correspondence measure between predicted saliency maps and ground truth, which
is computed as average normalized saliency at fixated locations [Peters et
al., 2005]. Furthermore, NSS is sensitive to false positives, relative
differences in saliency across evaluated image, and general monotonic
transformations [Bylinskii et al., 2018]. With $Y^{B}$ as a binary map of true
fixation location and $N$ to indicate total pixel number and $i$ indicates
each pixel instance, NSS value can be calculated using an equation as shown
below:

$NSS(\hat{Y},Y^{B})=\frac{1}{N}\sum_{i}\hat{Y}_{i}\times Y_{i}^{B}$ (2)

  * •

AUC-J evaluates predicted eye-gaze heatmaps as a classification task, where
each prediction pixel is evaluated through a binary classification setting.
Here, a certain threshold value is used to decide whether it is deemed to be
correctly predicted as eye-gaze locations, thus emphasizing frequency of true
positive. We use method described by Judd et. al. [Judd et al., 2009] to
select all required thresholds.

  * •

CC metric aims to evaluate level of linear relationship between two input
distributions. In eye-gaze location prediction task, both generated eye-gaze
location map and ground truth are treated as random variables [Le Meur et al.,
2007], and level value is calculated with both of these map inputs following
equation 3. Thus, with operator $\sigma$ as covariance matrix, CC value can be
calculated as:

$CC(\hat{Y},Y)=\frac{\sigma(\hat{Y},Y)}{\sigma(Y)\times\sigma(\hat{Y})}$ (3)

##  4 EXPERIMENT RESULT

In this section, we first present results of both baseline and our proposed
approach using part of our pre-processed dataset (GazeMining and Contrastive
Website), as outlined in Section 3. Then, we provide a comparison of best
results of our approach with alternative saliency prediction models using full
set of our pre-processed dataset, establishing a benchmark for eye-gaze
prediction task given webpage screenshot inputs.

###  4.1 Impact of Fine-tuning using independent and combined dataset.

No. | Models | GazeMining | Contrastive Website
---|---|---|---
NSS | AUC-J | CC | NSS | AUC-J | CC
1. | P-MSI | 1.037 | 0.699 | 0.130 | 0.839 | 0.683 | 0.109
2. | FT-MSI-GM | 1.398 | 0.753 | 0.170 | 0.717 | 0.651 | 0.093
3. | FT-MSI-CW | 0.695 | 0.655 | 0.081 | 2.364 | 0.777 | 0.277
4. | FT-MSI-CMB | 1.447 | 0.752 | 0.170 | 2.269 | 0.789 | 0.269

Table 2: Results of fine-tuned models on both preprocessed GazeMining and
Contrastive Website datasets.

Table 2 shows results of four alternative models that are trained using
different datasets. Here we can see that pre-trained MSI (P-MSI) model
performs worse in comparison to other models, which suggests its inability to
generalize to eye-gaze saliency prediction task. We can further observe that
when fine-tuning is applied, improvement from original P-MSI is noticeable,
especially when it is tested on similar dataset used for fine-tuning. This can
be seen with higher values achieved on all metrics of FT-MSI-GM and FT-MSI-CW
compared to P-MSI on GazeMining dataset and Contrastive Website dataset
respectively.

When model is trained using combined dataset, however, accuracy improvement is
instead lower. This is indicated by lower quantitative values achieved by FT-
MSI-CMB as opposed to their counterparts (FT-MSI-GM and FT-MSI-CW), that are
trained separately. This phenomenon may come from difficulty of estimator to
learn from these two datasets, which are quite distinct, and also from the
nature of task executed by the users on each dataset.

Figure 5 presents visual prediction examples for baseline model (P-MSI) and
best performing model (FT-MSI-CW) for Contrastive Website Dataset (where the
example is originated). The screenshot example is a part of route search task,
where user has to enter vehicle information to estimate fuel consumption. Thus
it is natural for users to focus on dialog box (displayed in the center of the
webpage screenshot) resulting in users’ fixations being located within this
area.

By evaluating the predictions of eye-gaze heatmaps of baseline model of P-MSI,
we can see that even though it manages to predict parts of ground-truth (eye-
gaze) locations, however, it also falsely predicts other less relevant
locations as eye-gaze locations (i.e. advertisement part). These results come
from its tendency to detect high color contrast as area of interest, which is
common in natural image datasets (SALICON [Jiang et al., 2015]). However, this
leads to higher occurrences of false positive, thus reducing its accuracy. Our
fine-tuned model however, manages to reduce existing inaccuracies by absorbing
the characteristics of dataset during fine-tuning, adapting model to this
specific eye-gaze prediction task. In this example, we see that the
predictions of fine-tuned models are more precise, especially on dialog box
and they exclude some of high contrast area that are normally recognized as
saliency area (e.g. advertisement locations).

From the experimental result, it can be concluded that saliency detection
model trained on natural image dataset does not work well for direct
application to web screenshot scenario. This can be mitigated by performing
careful fine-tuning to only use a dedicated dataset. Given this result, then
we use FT-MSI-GM and FT-MSI-CW as base for MMIAN optimization, and for
comparison in the next section.

###  4.2 Impact of Multi-modal Masks and Attention-based Fusion

Figure 6: Example of prediction results of MMIAN and Fine-tuned MSI Model (FT-
MSI). First column of (a) and (d) shows webpage screenshot with outlined text
(light green box) and image (purple box) area, along with eye-gaze heatmap
ground truth. Second column (b) and (e) shows predicted result from FT-MSI and
MMIAN model, with third column of (c) and (f) shows respective Grad-CAM
heatmap.

Table 3 presents results of our proposed MMIAN model and best results of fine-
tuned MSI-Model from previous section. From this result, it can be seen that
MMIAN produces higher value across quantitative metrics in overall,
outperforming results from fine-tuned MSI-Model. The gains on all of these
metrics altogether, suggest that estimation results of MMIAN are more accurate
compared to the best of FT-MSI results (from FT-MSI-GM and FT-MSI-CW), with
improvements on both true positive (as judged by AUC-Judd) and false positive
Instances (as evaluated by NSS and CC score).

In order to provide a more comprehensive analysis of the impact of the use of
image and text Masks and attention mechanisms, we first show locations of
detected images and texts on webpage screenshot input to provide semantic
explanations of their relevances for our MMIAN model to produce accurate
predictions. Then, we implemented a Gradient-weighted Class Activation Mapping
(Grad-CAM) [Selvaraju et al., 2017] by propagating multiplied error value and
gradient to each convolutional layer, enabling us to investigate the relevant
activation of convolutional kernels - with respect to image input - indicating
most prominent part of webpage screenshot for prediction. The example of
original webpage screenshot input that includes image and text locations
(drawn as bounding box), associated eye-gaze location ground-truth, generated
Grad-CAM and each prediction of fine-tuned MSI and MMIAN is shown in Figure 6.

On Figure 6, first we can see that prediction of MMIAN is more accurate than
FT-MSI, with larger and correctly identified eye-gaze areas are present. This
is especially noticeable in areas where both image and text are present. For
examples there are areas where button resides (that is recognized as image)
and product description, which in this case, are indeed the locations where
user looks at. This perceived higher accuracy can be attributed to the use of
image and text masks during learning, which is combined effectively through
attention mechanism to properly ’guide’ learning process to put priority in
this area. This impact is also apparent from observation of grad-cam heatmap
of FT-MSI and MMIAN, where grad-cam heatmap of MMIAN are seen to be more
concentrated on both of images and text area simultaneously (indicating where
models’ attention is) as opposed to ones produced by FT-MSI. Lastly, we also
notice that there are more frequent activations of grad-cam heatmaps of MMIAN
compared to ones from FT-MSI, which may suggest that larger perceptual field
of view from MMIAN is indicative to be able to produce more accurate eye-gaze
estimations in overall.

###  4.3 State-of-the-Arts Comparison

We compare best predictions of our approach against other available, seven
generalized visual saliency models:

  1. 1.

Context-Aware Saliency Detection (CASD) [Goferman et al., 2011].

  2. 2.

Discrete Cosine Transform (DCTS) [Houx et al., 2012].

  3. 3.

Hypercomplex Fourier Transform (HFT) [Li et al., 2012].

  4. 4.

Incremental Coding Length (ICL) [Hou and Zhang, 2008].

  5. 5.

RARE [Riche et al., 2012].

  6. 6.

SeoMilanfar [Seo and Milanfar, 2009].

  7. 7.

Spectral Residual (SR) [Hou and Zhang, 2007].

including one specialized eye-gaze location estimator of Multiple Kernel
Learning (MKL) [Shen and Zhao, 2014].

Table 4 shows results from all evaluated approaches. Here we see state-of-the-
art results of MMIAN that outperforms other alternatives, including eye-gaze
predictor of MKL with a large margin on FiWI dataset (note that there is no
training involved for FiWI dataset evaluation). This result is mainly due to
large differences in task characteristics between general visual saliency
prediction (where first seven models are trained) and webpage screenshot based
eye-gaze estimations tasks (where MKL and MMIAN are specialized). This
highlights the inability of visual saliency based models to generalize on this
specific task. Furthermore, our approach performs better than MKL on FiWI,
demonstrating the effectiveness of our overall approach for eye-gaze
estimation task.

No. | Models | GazeMining | Contrastive Website
---|---|---|---
NSS | AUC-J | CC | NSS | AUC-J | CC
1. | FT-MSI | 1.398 | 0.753 | 0.170 | 2.364 | 0.777 | 0.277
2. | MMIAN | 1.579 | 0.764 | 0.184 | 2.487 | 0.787 | 0.292

Table 3: Quantitative results of both fine-tuned MSI models and MMIAN model.

No. | Models | GazeMining | Contrastive Website | FiWI
---|---|---|---|---
NSS | AUC | CC | NSS | AUC | CC | NSS | AUC | CC
1. | CASD [Goferman et al., 2011] | 0.567 | 0.653 | 0.064 | 0.419 | 0.614 | 0.053 | 0.680 | 0.732 | 0.233
2. | DCTS [Houx et al., 2012] | 0.479 | 0.618 | 0.053 | 0.256 | 0.552 | 0.035 | 0.541 | 0.671 | 0.195
3. | HFT [Li et al., 2012] | 0.707 | 0.644 | 0.088 | 0.534 | 0.593 | 0.067 | 0.740 | 0.737 | 0.251
4. | ICL [Hou and Zhang, 2008] | 0.485 | 0.518 | 0.057 | 0.192 | 0.490 | 0.030 | 0.444 | 0.618 | 0.162
5. | RARE [Riche et al., 2012] | 0.632 | 0.653 | 0.072 | 0.382 | 0.589 | 0.052 | 0.850 | 0.758 | 0.280
6. | SeoMilanfar [Seo and Milanfar, 2009] | 0.393 | 0.584 | 0.038 | 0.350 | 0.571 | 0.044 | 0.445 | 0.651 | 0.163
7. | SR [Hou and Zhang, 2007] | 0.566 | 0.639 | 0.062 | 0.510 | 0.612 | 0.062 | 0.635 | 0.714 | 0.216
8. | MKL [Shen and Zhao, 2014] | - | - | - | - | - | - | 1.200 | 0.702 | 0.382
9. | MMIAN (proposed) | 1.579 | 0.764 | 0.184 | 2.487 | 0.787 | 0.292 | 1.385 | 0.786 | 0.397

Table 4: Comparison of existing saliency predictors evaluated on test sets of
our full pre-processed dataset. The boldface indicates best results, red color
implies second-best results, and third-best results are marked by blue
coloured fonts. Figure 7: Five examples of screenshot inputs from FiWI dataset
(with respective eye-gaze heatmaps ground-truth overlaid) and predictions from
other five saliency predictors (including ours).

Figure 7 shows predictions example of our MMIAN and four other best performing
model on FiWI for evaluation (excluding MKL, due to lack of available
implementation of the model). Based on this figure, we can observe that most
of alternative models produce a large area of webpage screenshot as potential
eye-gaze locations. However, this leads to largely false positive prediction,
given the mismatch between predictions against actual eye-gaze locations from
user of these webpage screenshots (i.e. most area are falsely identified as
eye-gaze locations). In contrast, our approach produces more precise
estimates, as observed by more refined (and accurate) predicted area of eye-
gaze locations (that explains large margin in terms of NSS metrics value from
our models compared to others, as shown in Table 4). This is mainly due to the
tendency of visual saliency models to focus on pure appearance of webpage
screenshot (i.e. high contrast images), as opposed to MMIAN that has been
conditioned to the specific characteristics of our pre-processed eye-gaze
dataset (which inherently contains eye-gaze characteristics of users, given
webpage screenshot). Remarkably, our models is also capable of correctly
predicting eye-gaze locations on relevant areas, such as text (third row),
image (fourth row) and their combinations (first, second and fifth row), in
comparison with other approaches. Here we see that our models’ predictions are
consistently more accurate than alternatives, demonstrating the effectiveness
of our approach.

##  5 CONCLUSION

In this work, we enable the development of automatic eye-gaze estimations
given webpage screenshot inputs to enable the improvement of the webpage
layout, and hence respective user interaction. We do this by developing a
unified eye-gaze dataset from three available website and user interaction-
based datasets: GazeMining, Contrastive Website and FiWI. We then pre-process
each dataset to produce necessary data for eye-gaze prediction task, such as
webpage screenshot and corresponding eye-gaze heatmaps as ground truth. In
addition, we generate image and textual locations from webpage (in the form of
masks) which can be used for training and modeling. Given our unified eye-gaze
dataset, then we propose a novel deep learning based and multi-modal
attentional network eye-gaze predictor to benefit from the characteristics of
the dataset. Our proposed approach leverages spatial locations of text and
image (in form of masks), which is further fused with attentional mechanisms
to enhance the prediction results.

During analysis of the impact of fine-tuning using our pre-processed dataset,
and the effects of training when the combined dataset is used, we found that
the prediction results are indeed improved when careful fine-tuning is
conducted. Then, we evaluate the prediction of our full approach (MMIAN) with
respect to the ground truth, existing text and image locations from webpage
screenshot, and part of Grad-CAM activations. Here we notice accurate
predictions of our approach, especially on textual and image area where users
are looking at, with large and wide activation of Grad-CAM that are
concentrated on these locations. This observation demonstrates the benefit of
the use of both image and text masks as input in combination with attention
mechanism.

To measure the competitive results of our proposed approach, we compare them
with other saliency prediction alternatives, establishing a benchmark for eye-
gaze prediction task. In our comparison, we found state-of-the-art results of
our model with high scores across quantitative metrics, and lower false
positive rates than other approaches. Visual analysis further confirms our
findings, that our approach produces more accurate prediction of eye-gaze
locations on relevant website locations, including where text and image are
present. The result suggests the superiority of our approach in capturing user
behavior. Future work will be to incorporate other user behavior
characteristics (e.g. mouse trajectory), including users’ identity (if
accessible) such as age, and locations as additional modalities to further
benefit from this information to improve prediction accuracy.

## ACKNOWLEDGEMENT

This work is funded by UDeco project by Germany BMBF-KMU Innovativ -
01IS20030B.

## REFERENCES

  * Aspandi et al., 2022a Aspandi, D., Doosdal, S., Ülger, V., Gillich, L., and Staab, S. (2022a).  User interaction analysis through contrasting websites experience.  arXiv preprint arXiv:2201.03638.
  * Aspandi et al., 2020 Aspandi, D., Mallol-Ragolta, A., Schuller, B., and Binefa, X. (2020).  Latent-based adversarial neural networks for facial affect estimations.  In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 606–610. IEEE.
  * Aspandi et al., 2022b Aspandi, D., Sukno, F., Schuller, B. W., and Binefa, X. (2022b).  Audio-visual gated-sequenced neural networks for affect recognition.  IEEE Transactions on Affective Computing.
  * Bylinskii et al., 2018 Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., and Durand, F. (2018).  What do different evaluation metrics tell us about saliency models?  IEEE transactions on pattern analysis and machine intelligence, 41(3):740–757.
  * Chen et al., 2017 Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. (2017).  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848.
  * Dai et al., 2021 Dai, Y., Gieseke, F., Oehmcke, S., Wu, Y., and Barnard, K. (2021).  Attentional feature fusion.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3560–3569.
  * Goferman et al., 2011 Goferman, S., Zelnik-Manor, L., and Tal, A. (2011).  Context-aware saliency detection.  IEEE transactions on pattern analysis and machine intelligence, 34(10):1915–1926.
  * Hou and Zhang, 2007 Hou, X. and Zhang, L. (2007).  Saliency detection: A spectral residual approach.  In 2007 IEEE Conference on computer vision and pattern recognition, pages 1–8. Ieee.
  * Hou and Zhang, 2008 Hou, X. and Zhang, L. (2008).  Dynamic visual attention: Searching for coding length increments.  Advances in neural information processing systems, 21.
  * Houx et al., 2012 Houx, D., HAREL, J., and KOCH, C. (2012).  Image signature: highlighting sparse salient regions.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(1):194–201.
  * Jetley et al., 2016 Jetley, S., Murray, N., and Vig, E. (2016).  End-to-end saliency mapping via probability distribution prediction.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5753–5761.
  * Jiang et al., 2015 Jiang, M., Huang, S., Duan, J., and Zhao, Q. (2015).  Salicon: Saliency in context.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1072–1080.
  * Judd et al., 2009 Judd, T., Ehinger, K., Durand, F., and Torralba, A. (2009).  Learning to predict where humans look.  In 2009 IEEE 12th international conference on computer vision, pages 2106–2113. IEEE.
  * Kingma and Ba, 2014 Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Kroner et al., 2020 Kroner, A., Senden, M., Driessens, K., and Goebel, R. (2020).  Contextual encoder–decoder network for visual saliency prediction.  Neural Networks, 129:261–270.
  * Le Meur et al., 2007 Le Meur, O., Le Callet, P., and Barba, D. (2007).  Predicting visual fixations on video based on low-level visual features.  Vision research, 47(19):2483–2498.
  * Li et al., 2012 Li, J., Levine, M. D., An, X., Xu, X., and He, H. (2012).  Visual saliency based on scale-space analysis in the frequency domain.  IEEE transactions on pattern analysis and machine intelligence, 35(4):996–1010.
  * Menges, 2020 Menges, R. (2020).  Gazemining: A dataset of video and interaction recordings on dynamic web pages. labels of visual change, segmentation of videos into stimulus shots, and discovery of visual stimuli.
  * Pan and Yang, 2010 Pan, S. J. and Yang, Q. (2010).  A survey on transfer learning.  IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359.
  * Peters et al., 2005 Peters, R. J., Iyer, A., Itti, L., and Koch, C. (2005).  Components of bottom-up gaze allocation in natural images.  Vision Research, 45(18):2397–2416.
  * Riche et al., 2012 Riche, N., Mancas, M., Gosselin, B., and Dutoit, T. (2012).  Rare: A new bottom-up saliency model.  In 2012 19th IEEE International Conference on Image Processing, pages 641–644. IEEE.
  * Rodriguez-Diaz et al., 2021 Rodriguez-Diaz, N., Aspandi, D., Sukno, F. M., and Binefa, X. (2021).  Machine learning-based lie detector applied to a novel annotated game dataset.  Future Internet, 14(1):2.
  * Selvaraju et al., 2017 Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017).  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626.
  * Seo and Milanfar, 2009 Seo, H. J. and Milanfar, P. (2009).  Static and space-time visual saliency detection by self-resemblance.  Journal of vision, 9(12):15–15.
  * Shen and Zhao, 2014 Shen, C. and Zhao, Q. (2014).  Webpage saliency.  In European conference on computer vision, pages 33–46. Springer.
  * Simonyan and Zisserman, 2014 Simonyan, K. and Zisserman, A. (2014).  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556.
  * Xie et al., 2020 Xie, M., Feng, S., Xing, Z., Chen, J., and Chen, C. (2020).  UIED: A Hybrid Tool for GUI Element Detection, page 1655–1659.  Association for Computing Machinery, New York, NY, USA.
  * Zhang et al., 2018 Zhang, D., Fu, H., Han, J., Borji, A., and Li, X. (2018).  A review of co-saliency detection algorithms: Fundamentals, applications, and challenges.  ACM Transactions on Intelligent Systems and Technology (TIST), 9(4):1–31.
  * Zhou et al., 2021 Zhou, W., Liu, W., Lei, J., Luo, T., and Yu, L. (2021).  Deep binocular fixation prediction using a hierarchical multimodal fusion network.  IEEE Transactions on Cognitive and Developmental Systems, pages 1–1.
  * Zhou et al., 2017 Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., and Liang, J. (2017).  East: An efficient and accurate scene text detector.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).


# Language Guided Visual Question Answering: Elevate Your Multimodal Language
Model Using Knowledge-Enriched Prompts

Deepanway Ghosal†, Navonil Majumder†, Roy Ka-Wei Lee†,
Rada Mihalcea△, Soujanya Poria†
† Singapore University of Technology and Design, Singapore
△ University of Michigan, USA
<EMAIL_ADDRESS>
{navonil_majumder, roy_lee<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Visual question answering (VQA) is the task of answering questions about an
image. The task assumes an understanding of both the image and the question to
provide a natural language answer. VQA has gained popularity in recent years
due to its potential applications in a wide range of fields, including
robotics, education, and healthcare. In this paper, we focus on knowledge-
augmented VQA, where answering the question requires commonsense knowledge,
world knowledge, and reasoning about ideas and concepts not present in the
image. We propose a multimodal framework that uses language guidance (LG) in
the form of rationales, image captions, scene graphs, etc to answer questions
more accurately. We benchmark our method on the multi-choice question-
answering task of the A-OKVQA, Science-QA, VSR, and IconQA datasets using CLIP
and BLIP models. We show that the use of language guidance is a simple but
powerful and effective strategy for visual question answering. Our language
guidance improves the performance of CLIP by 7.6% and BLIP-2 by 4.8% in the
challenging A-OKVQA dataset. We also observe consistent improvement in
performance on the Science-QA, VSR, and IconQA datasets when using the
proposed language guidances. The implementation of LG-VQA is publicly
available at https://github.com/declare-lab/LG-VQA.

##  1 Introduction

Visual understanding is one of the most complex tasks in artificial
intelligence. Among the many challenges associated with it, image question
answering has been formulated as a task that tests the ability of a system to
understand the elements of an image in a way similar to how humans interact
with images. This task involves creating models that can accurately answer
questions based on the content of an image. While significant progress has
been made in image question answering Wang et al. (2022a); Chen et al. (2022),
most of the existing approaches focus solely on analyzing the visual features
associated with the image or the linguistic content or representation of the
question (and possibly candidate answers), without utilizing additional
sources of guidance. However, incorporating external guidance into these
models has the potential to improve their performance and enhance their
understanding of visual content.

Figure 1: The above image and the question At what time of day are the
skateboarders probably skating on the beach? are given. In our LG-VQA
framework, we generate the caption: a group of skate boarders in front of palm
trees at sunset and the question constrained rationale: the sun is just
beginning to set behind the sand to provide the answer: sunset.

In this paper, we propose a multimodal framework, LG-VQA that leverages
language guidance to improve the accuracy of image question-answering systems.
Language guidance is sourced from elements such as image captions, scene
graphs, and rationales created in response to questions. These sources serve
to enrich textual instructions with valuable knowledge. Our method is
evaluated on the multi-choice A-OKVQA dataset Schwenk et al. (2022). We choose
A-OKVQA specifically because the questions in this dataset require a broad
base of commonsense and world knowledge to answer. We also evaluate our method
on the challenging ScienceQA Lu et al. (2022), Visual Semantic Reasoning (VSR)
Liu et al. (2023), and IconQA Lu et al. (2021) datasets. With the advent of
large-scale multimodal pre-training Wang et al. (2022a, b), the performance in
the commonly used VQA and VQA v2 datasets Antol et al. (2015); Goyal et al.
(2017) has saturated.

The datasets considered in this paper provide a more challenging test bed for
VQA where factual, commonsense, physical, scientific, and visual knowledge are
required, going beyond the usual image recognition, attribute detection tasks
that are generally found in the VQA, VQA v2 datasets. Previously Shah et al.
(2019); Wu et al. (2017); Zheng et al. (2021) have used concrete knowledge
bases such DBPedia, Freebase, or named entity knowledge graphs in VQA. We
instead show that simple language guidance without using knowledge graphs is
also a powerful technique for improving VQA performance in various challenging
datasets.

We benchmark our approach with the CLIP Radford et al. (2021) and BLIP-2 Li et
al. (2023) models and show significant improvements over those, demonstrating
that incorporating language guidance can be an effective strategy for visual
question answering.

##  2 Background

We use the following two strong multi-modal models in our framework:

#### CLIP

Radford et al. (2021) is a model for learning transferable visual
representations from natural language supervision. CLIP is trained on 400
million (image, text) pairs to learn a joint embedding space where image
representations and their corresponding text representations are close to each
other. We use the CLIP model with the ViT-L/14 image encoder Dosovitskiy et
al. (2020) and the GPT-2 text encoder Radford et al. (2019). CLIP encodes the
image and the text separately through the corresponding encoders. Then, a
normalized dot product between the encoded vectors provides the image-text
matching score. We denote the image and text encoder as $I_{CLIP}$ and
$T_{CLIP}$. Given an image $img$ and text instance $txt$, the matching score
is computed as follows:

$\begin{split}I&=\text{{l2\\_normalize}}(I_{CLIP}(img))\\\
T&=\text{{l2\\_normalize}}(T_{CLIP}(txt))\\\ score&=e^{t}*(I\cdot
T)=e^{t}*\sum_{i=1}^{m}I_{m}T_{m}\end{split}$ (1)

where $t$ is a learned temperature parameter for scaling the dot product
similarity. We use this formulation of the (image, text) matching methodology
for CLIP models in our VQA framework.

#### BLIP-2

Li et al. (2023) bootstraps joint pre-training over image and language data
with off-the-shelf frozen image encoders and frozen large language models
(LLMs). A small transformer network called Query Transformer or Q-Former is
trained to model the interaction between the frozen image and language models.

In the first stage of BLIP-2 pre-training, the Q-Former learns to extract
visual representations from the image that is most relevant to the text using
image-text contrastive learning, and image-grounded text generation tasks. In
the second stage, the trained Q-Former is used to extract a set of query
embeddings about the image. The query embeddings are converted into the input
embedding space of the LLMs through a linear projection. The projected vector
functions as the soft visual prompts that condition the LLM to generate the
desired text. BLIP-2 uses paired (image, text) data for both stages of pre-
training.

We denote the image encoder and Q-Former of BLIP2 as $I_{BLIP}$ and
$Q_{BLIP}$. The image-text matching score is computed as follows:

$\begin{split}I&=I_{BLIP}(img)\\\ features&=Q_{BLIP}(I,txt,q)\\\
score&=Proj(features)\end{split}$ (2)

where $q$ is the leaned embeddings during pre-training and $Proj$ is the
projection head. Additionally, we use the FlanT5 Chung et al. (2022) model as
the LLM for image-constrained text generation. Here, the Q-Former does not
take any text as input, but the LLM encoder may use a prefix text $prefix$ on
which the $output$ text is conditioned:

$\begin{split}v_{prompt}&=Q_{BLIP}(I,q)\\\ encoded&=\text{{FlanT5
Encoder}}(v_{prompt},prefix)\\\ output&=\text{{FlanT5
Decoder}}(encoded)\end{split}$ (3)

We use the image-text matching methodology and the (visual prompt, prefix)
conditioned text generation in various stages of our framework.

##  3 LG-VQA Framework

###  3.1 Zero-Shot Visual Question Answering

The CLIP and BLIP-2 models show impressive zero-shot performance in various
computer vision tasks. For ImageNet type classification tasks, the method
generally works by creating sentence instances from the labels: A photo of a
dog, A photo of a cat, A photo of a bird, etc. Then, similarity scores are
computed between the image and all the sentence instances created from the
labels. The label (e.g. bird) is then predicted from the sentence (e.g. A
photo of a bird) which provided the maximum similarity score.

We apply a similar method for the zero-shot multi-choice visual question-
answering task. Given image $img$, question $question$, multiple answer
choices $a_{1},a_{2},\dots,a_{n}$, we create sentences by concatenating the
question and the answer $\\{question,a_{i}\\}$. The concatenated sentence
providing the highest image-text matching score corresponds to the predicted
answer:

$\begin{split}txt_{i}&=\\{question,a_{i}\\}\quad\forall i=1,\dots,n\\\
s_{i}&=match(img,txt_{i})\\\ best&=argmax([s_{1},\dots,s_{n}])\\\
answer&=a_{best}\end{split}$ (4)

###  3.2 Unguided Visual Question Answering

We use the term unguided to denote the conventional visual question-answering
task with full fine-tuning without using any additional guidance. Here, we
train a model with the image $img$, question $question$, and multiple answer
choices $a_{1},\dots,a_{n}$ to predict the correct answer choice $a_{k}$.

We obtain the image-text matching score $s_{i}$ for each (image, question,
answer) triplet from CLIP or BLIP-2. We then normalize the scores with a
softmax layer across the $n$ choices:

$\begin{split}txt_{i}&=\\{q,a_{i}\\}\quad\forall i=1,\dots,n\\\
s_{i}&=match(img,txt_{i})\\\
\bar{s_{i}}&=\frac{e^{s_{i}}}{\sum_{j=1}^{n}e^{s_{j}}}\end{split}$ (5)

Finally, we use the cross-entropy loss to train the underlying CLIP or BLIP-2
model. Assuming that $a_{k}$ is the correct answer, the loss is as follows:

$\begin{split}\mathcal{L}&=-\sum_{i=1}^{n}p_{i}log(\bar{s_{i}})=-log(\bar{s_{k}})\\\
\end{split}$ (6)

where we denote $p_{i}$ to be the class label of the answer choices. The value
of $p_{k}$ is 1 as $a_{k}$ is the correct answer, whereas the values of the
other $p_{i}$ are zero as they are the incorrect answers. The loss thus
simplifies to the $-log(\bar{s_{k}})$ as shown in Eq. 6. The loss is
equivalent to the cross-entropy loss used in multi-class classification
problems.

###  3.3 Constained Inference Generation

We use the BLIP-2 model with FLAN-T5 to generate inferences (such as
rationales, explanations and captions) constrained on the image $img$ and
question $question$ that could potentially be helpful for the VQA task. We
follow the setting specified earlier in Eq. 3. We do not use any $prefix$, and
thus the FLAN-T5 model is constrained entirely upon the visual prompt
extracted from the question and the image with the Q-Former.

We use the rationales in the A-OKVQA dataset and the explanations for the VQA
and VQA v2 provided in Li et al. (2018); Park et al. (2018) to fine-tune the
BLIP-2 with FLAN-T5 model. We also use COCO Captions Chen et al. (2015) for
image captioning. We keep the visual encoder and the FLAN-T5 frozen and only
tune the Q-Former network for the tasks of rationales or explanation or
caption generation. We use the generated rationales, explanations, and
captions as language guidance in our main model (Section 3.5).

###  3.4 Scene Graphs and Object Detection

We use the Relation Transformer (RelTR) network network Cong et al. (2023) for
scene graph generation from the images. RelTR uses a transformer model based
on DETR Carion et al. (2020) pre-trained on the Visual Genome dataset Krishna
et al. (2017) to predict the scene graph triplets about an image. We use the
RelTR model to predict scene graphs for the images. We concatenate the
triplets with [SEP] token to create the scene graph guidance. We also use the
UniDet Zhou et al. (2022) and DETR models for object detection in the images.
After detection, we aggregate the objects using their counts: two dogs, one
girl, three toys. The objects detected are potentially helpful for questions
that require counting or numerical reasoning.

###  3.5 Language Guided Visual Question Answering

We use the notation $guide$ to denote the various guidance strategies or their
combinations. Concretely it could be either the rationales, explanations,
captions, scene graphs, objects, or any of their combinations. The combination
is achieved by simply concatenating the various guidances. We use the notation
$txt\\_guide_{i}$ to denote the concatenation of $\\{question,a_{i},guide\\}$.

We follow two strategies for using the $guide$ in CLIP or BLIP-2. For CLIP,
the strategy for computing the matching score is similar to Eq. 5 with
$text\\_guide_{i}$:

$\begin{split}s_{i}&=match(img,txt\\_guide_{i})\end{split}$ (7)

Then, we use the softmax normalization with cross-entropy loss specified in
Eq. 6 for training. CLIP was originally trained on a maximum length of 77
input tokens. We extend the maximum length to 512 tokens to incorporate
$guide$ as part of the input and update these positional embeddings during
training.

For BLIP-2, we found that passing $txt_{i}$ and $txt\\_guide_{i}$ through two
forward passes followed by feature merging to be the optimal strategy:

$\begin{split}x_{1}&=Q_{BLIP}(I,txt_{i},q)\\\
x_{2}&=Q_{BLIP}(I,txt\\_guide_{i},q)\\\
features&=|x_{1},x_{2},x_{1}-x_{2},x_{1}*x_{2}|\\\ s_{i}&=Proj(features)\\\
\end{split}$ (8)

The model is trained with the cross-entropy loss with softmax normalized
scores over the choices. We train the Q-Former of BLIP-2, the projection layer
$Proj$ and keep the visual encoder frozen. We found that fine-tuning the
visual encoder is computationally expensive, with marginal improvement in
performance.

##  4 Experiments

#### Dataset Used

We use the multi-choice question answering setup of the A-OKVQA Schwenk et al.
(2022) dataset for evaluating our VQA framework. Each instance has four
possible answer choices, among which only one is correct. We report the scores
in the validation dataset as the test set labels are not available. We also
report results for the easy and hard subset as specified in the dataset. The
dataset has a total of 25K instances.

We also experiment with the ScienceQA Lu et al. (2022), Visual Semantic
Reasoning (VSR) Liu et al. (2023), and IconQA Lu et al. (2021) datasets.
ScienceQA is collected from elementary and high school science curricula
containing 10k questions with image context. The number of answer choices
varies between 2 and 5, among which only one is correct. Visual Spatial
Reasoning (VSR) is a corpus of caption-image pairs with true/false labels.
Each caption describes the spatial relation of two individual objects in the
image. The task is to determine if the caption correctly describes the image.
We use the true/false labels as the two answer choices for multi-choice
question answering. IconQA is a dataset for diagram understanding and
cognitive reasoning in real-world diagram word problems. We use the multi-
text-choice subtask which has a total of 31k instances.

Mode | CLIP | BLIP-2
---|---|---
Overall | Easy | Hard | Overall | Easy | Hard
Zero-Shot | 58.52 | 58.98 | 51.43 | 64.98 | 65.95 | 50.00
No Guidance | 68.30 | 68.74 | 61.43 | 75.02 | 76.09 | 58.57
LG-VQA |  |  |  |  |  |
Rationale | 74.11 | 74.72 | 64.71 | 76.77 | 77.77 | 61.43
Explanation | 68.21 | 68.93 | 57.14 | 76.16 | 77.02 | 62.86
Captions | 69.08 | 69.67 | 60.00 | 76.68 | 77.77 | 60.00
Scene Graph | 68.03 | 66.23 | 65.71 | 76.61 | 77.51 | 62.86
Objects | 67.77 | 68.56 | 55.71 | 75.94 | 77.07 | 58.57
All | 75.98 | 76.65 | 65.71 | 79.83 | 80.47 | 70.00

Table 1: Results on the A-OKVQA validation set. We report VQA accuracy scores
for zero-shot, unguided, and various kinds of guidance for the overall, easy,
and hard sets. Scores are average of three runs.

#### Results for A-OKVQA

We report the results for the overall set and the easy, hard subsets for
A-OKVQA in Table 1. In the first two rows of the Table 1, we report the
results in the baseline settings: i) zero-shot performance using the image-
text matching scores without any training (Section 3.1), and ii) training the
models in the conventional VQA setting without any guidance (Section 3.2). We
obtain a zero-shot overall accuracy of 58.52% which improves to 68.30% when
trained without guidance for the CLIP model. In comparison the BLIP-2 model
shows a zero-shot accuracy of 64.98% and without guidance accuracy of 75.02%.

The lower part of Table 1 shows the results with various kind of guidances.
Most of the guidances improve performance when used individually on their own.
For individual guidance, we obtain the highest performance with rationales for
both CLIP and BLIP-2. In particular, the improvement over no guidance is close
to 6% for CLIP, which is also somewhat expected as the rationale generator
BLIP-2 model is trained on the A-OKVQA training set itself. The performance
with all the guidances is significantly better compared to no guidance for
both models, with 7.6% improvement in CLIP and 4.8% improvement in BLIP-2 in
the overall set. We also report the scores for the easy and hard subsets in
A-OKVQA. We note that the improvement in all guidance setting comes from
improvement in both the easy and hard subsets. Notably, BLIP-2 achieves a hard
set accuracy of 70% with all guidance compared to 58.57% with no guidance.

#### Results for other datasets

We report results for the other three datasets in Table 2. For ScienceQA
without guidance the CLIP model achieves an accuracy of 85.32%, which is a
percent higher than the BLIP-2 accuracy of 84.28%. We observe 1-2 % increase
in accuracy with the guidances for both models. The objects guidance in CLIP
provides the maximum accuracy of 87.22%.

In VSR, the BLIP-2 models heavily outperform the CLIP models. We observe a
minor improvement in performance for CLIP with the various guidances. For
BLIP-2, the improvement is close to 1.5%, when we use the captions, scene
graphs, and objects (CSO) guidance together.

We observe a similar trend in performance for IconQA. The use of various
guidances helps in improving the accuracy of both models. However, the margin
of increase is not as high in BLIP-2 as it is in CLIP. We observed 2% increase
in accuracy for CLIP and 0.5% in increase in accuracy for BLIP-2.

Mode | ScienceQA | VSR | IconQA
---|---|---|---
CLIP | BLIP-2 | CLIP | BLIP-2 | CLIP | BLIP-2
No Guidance | 85.32 | 84.28 | 63.99 | 76.76 | 82.36 | 86.21
LG-VQA |  |  |  |  |  |
Captions | 86.22 | 84.88 | 64.10 | 76.68 | 83.98 | 86.47
Scene Graph | 86.91 | 86.32 | 63.75 | 77.00 | 82.54 | 86.18
Objects | 87.22 | 85.28 | 63.95 | 77.40 | 83.25 | 86.52
CSO | 86.02 | 86.27 | 64.24 | 78.15 | 84.44 | 86.72
Lecture | 86.92 | 86.27 | - | - | - | -
CSOL | 86.37 | 86.56 | - | - | - | -

Table 2: Results on the ScienceQA, VSR, and IconQA test sets. CSO denotes
captions, scene graphs, and objects guidance. The L in CSOL denotes additional
lecture guidance. The lecture guidance is available only for the ScienceQA
dataset. Scores are average of three runs.

#### Analysis

For A-OKVQA, we analyze the performance of our models across various wh-
question types in Table 3. In A-OKVQA, `What' type questions are the most
prevalent type. The incorporation of all guidance helps both CLIP and BLIP-2
to improve on the `What' question types in addition to all the other question
types. For CLIP, the rationale-based guidance outperforms the no guidance
across all the question types. Notably, the `Why' accuracy jumps to 90% from
77%.

#### Error Analysis

We found some common error patterns from our analysis: (i) questions about
very small or tiny objects present in the image are erroneously predicted as
the visual encoders of CLIP or BLIP-2 are not sensitive enough to detect those
objects; (ii) questions about optical characters embedded in the image are
often wrongly predicted (which number birthday is being celebrated? for a cake
with the number 10 drawn with cream); (iii) questions about abduction (Given
the position of the bat and ball the batter most likely did what?) or visual
occlusions (What is likely in front of the rug?) are also incorrectly
predicted because of insufficient world knowledge.

##  5 Conclusion

In this work, we presented a simple method for visual question answering with
language guidance. Our language guidance consists of rationale and
explanations constrained upon the image and the question, image captions,
scene graphs, and description of the objects present in the image. We propose
an effective strategy of fusing the language guidance with the pre-trained
multi-modal CLIP and BLIP-2 models. Our language guidance improves the
performance of CLIP by 7.6% and BLIP-2 by 4.8% in the challenging A-OKVQA
dataset. We also observe consistent improvement in performance with guidance
on the ScienceQA, VSR, and IconQA datasets.

Model | Mode | What | Which | Why | How | Where
---|---|---|---|---|---|---
CLIP | Zero-Shot | 60.14 | 53.33 | 55.74 | 45.00 | 66.0
No Guidance | 68.96 | 56.00 | 77.05 | 58.33 | 86.0
Rationale | 75.60 | 64.00 | 90.16 | 63.33 | 86.0
All | 76.40 | 68.00 | 88.52 | 66.67 | 86.0
BLIP-2 | Zero-Shot | 66.67 | 49.33 | 72.13 | 50.00 | 74.0
No Guidance | 75.49 | 60.00 | 88.52 | 65.00 | 90.0
Rationale | 77.32 | 65.33 | 86.89 | 63.33 | 90.0
All | 80.53 | 66.67 | 90.16 | 70.00 | 94.0

Table 3: Results of CLIP and BLIP-2 with different guidance across question
types. Scores are average of three runs.

##  6 Limitations

As mentioned before, our proposed model under-performs for specific categories
of questions where tiny object recognition or optical character recognition on
the images is necessary for answering the questions. We also found that the
baseline models CLIP and BLIP-2, and our subsequently developed models with
guidance cannot adequately model complex world knowledge and commonsense
knowledge for answering difficult questions about the image. In the future, we
would like to incorporate explicit world and commonsense knowledge as part of
the guidance to improve the VQA models.

##  7 Acknowledgments

We thank the anonymous reviewers for their constructive feedback. We also
thank Anubhav Jangra for providing feedback on the paper. This project is
supported by the AcRF MoE Tier-2 grant (Project no. T2MOE2008, and Grantor
reference no. MOE-T2EP20220-0017) titled: “CSK-NLP: Leveraging Commonsense
Knowledge for NLP”, and the SRG grant id: T1SRIS19149 titled “An Affective
Multimodal Dialogue System”. This project was partially funded by a grant from
the Michigan Automotive Research Center (``ARC''). Any opinions, findings, and
conclusions or recommendations expressed in this material are those of the
authors and do not necessarily reflect the views of the Michigan Automotive
Research Center.

## References

  * (1)
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 2425–2433.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-end object detection with transformers. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_. Springer, 213–229.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ (2015).
  * Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022\.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_ (2022).
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022\.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ (2022).
  * Cong et al. (2023) Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. 2023\.  Reltr: Relation transformer for scene graph generation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2023).
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _International Conference on Learning Representations_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6904–6913.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ 123 (2017), 32–73.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023\.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Li et al. (2018) Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. 2018.  Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 552–567.
  * Liu et al. (2023) Fangyu Liu, Guy Emerson, and Nigel Collier. 2023.  Visual spatial reasoning.  _Transactions of the Association for Computational Linguistics_ 11 (2023), 635–651.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.  Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_.
  * Lu et al. (2021) Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021.  IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning. In _The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks_.
  * Park et al. (2018) Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018.  Multimodal explanations: Justifying decisions and pointing to the evidence. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 8779–8788.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  (2019).
  * Schwenk et al. (2022) Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022\.  A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII_. 146–162.
  * Shah et al. (2019) Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019.  Kvqa: Knowledge-aware visual question answering. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 8876–8884.
  * Wang et al. (2022b) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In _International Conference on Machine Learning_. PMLR, 23318–23340.
  * Wang et al. (2022a) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022a.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  _arXiv preprint arXiv:2208.10442_ (2022).
  * Wu et al. (2017) Qi Wu, Chunhua Shen, Peng Wang, Anthony Dick, and Anton Van Den Hengel. 2017.  Image captioning and visual question answering based on attributes and external knowledge.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 6 (2017), 1367–1381.
  * Zheng et al. (2021) Wenfeng Zheng, Lirong Yin, Xiaobing Chen, Zhiyang Ma, Shan Liu, and Bo Yang. 2021\.  Knowledge base graph embedding module design for Visual question answering model.  _Pattern recognition_ 120 (2021), 108153.
  * Zhou et al. (2022) Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. 2022\.  Simple multi-dataset detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7571–7580.

##  Appendix A Experimental Details and Resources

We train our models with the AdamW optimizer Loshchilov and Hutter (2017) with
a batch size of 8 and 8 epochs. We used a learning rate between {1e-6, 3e-6,
5e-6}. We train our models on a single RTX A6000 GPU. The main task training
on A-OKVQA takes around 4 hours for CLIP and 8 hours for BLIP-2 with 8 epochs.