# Prophet: Prompting Large Language Models with Complementary Answer
Heuristics for Knowledge-based Visual Question Answering

Zhou Yu , Xuecheng Ouyang, Zhenwei Shao,
Meng Wang , Jun Yu  This work was supported in part by the Zhejiang Provincial
Natural Science Foundation of China under Grant LR22F020001, in part by the
National Natural Science Foundation of China under Grants 62125201, 62072147,
62020106007 and 61836002, and in part by the Zhejiang Provincial Natural
Science Foundation of China under Grant LDT23F02025F02. (Corresponding author:
Jun Yu.) Z. Yu, Z. Shao, J. Yu are with the Key Laboratory of Complex Systems
Modeling and Simulation, the School of Computer Science, Hangzhou Dianzi
University, China. (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>X. Ouyang is with the HDU-ITMO Joint Institute, Hangzhou
Dianzi University, China. (e-mail<EMAIL_ADDRESS>M. Wang is with the
School of Computer Science and Information Engineering, Hefei University of
Technology, China. (e-mail<EMAIL_ADDRESS>
###### Abstract

Knowledge-based visual question answering (VQA) requires external knowledge
beyond the image to answer the question. Early studies retrieve required
knowledge from explicit knowledge bases (KBs), which often introduces
irrelevant information to the question, hence restricting the performance of
their models. Recent works have resorted to using a powerful large language
model (LLM) as an implicit knowledge engine to acquire the necessary knowledge
for answering. Despite the encouraging results achieved by these methods, we
argue that they have not fully activated the capacity of the _blind_ LLM as
the provided textual input is insufficient to depict the required visual
information to answer the question. In this paper, we present Prophet—a
conceptually simple, flexible, and general framework designed to prompt LLM
with answer heuristics for knowledge-based VQA. Specifically, we first train a
vanilla VQA model on a specific knowledge-based VQA dataset without external
knowledge. After that, we extract two types of complementary answer heuristics
from the VQA model: answer candidates and answer-aware examples. Finally, the
two types of answer heuristics are jointly encoded into a formatted prompt to
facilitate the LLM’s understanding of both the image and question, thus
generating a more accurate answer. By incorporating the state-of-the-art LLM
GPT-3 [1], Prophet significantly outperforms existing state-of-the-art methods
on four challenging knowledge-based VQA datasets. To demonstrate the
generality of our approach, we instantiate Prophet with the combinations of
different VQA models (_i.e._ , both discriminative and generative ones) and
different LLMs (_i.e._ , both commercial and open-source ones).

###### Index Terms:

Visual Question Answering (VQA), large language models (LLMs), knowledge-based
VQA, multimodal learning.

##  1 Introduction

Recent advances in deep learning have enabled substantial progress in visual
question answering (VQA) which requires a machine to answer free-form
questions by reasoning about given images. Benefiting from large-scale vision-
language pretraining, the state-of-the-art methods have even surpassed human
level on several representative benchmarks [2, 3, 4]. Despite the success of
these methods, their reasoning abilities are far from satisfactory, especially
when _external knowledge_ is required to answer the questions. In this
situation, the task of knowledge-based VQA is introduced to validate models’
abilities to leverage external knowledge. Early knowledge-based VQA benchmarks
additionally provide structured knowledge bases (KBs) and annotate required
knowledge facts for all the questions [5, 6]. More recently, benchmarks
emphasizing on _open-domain_ knowledge have been established [7, 8], which
means KBs are no longer provided and any external knowledge resource can be
used for answering. We focus on the task with open-domain knowledge in this
paper.

A straightforward solution for knowledge-based VQA is to retrieve knowledge
entries from explicit KBs, _e.g._ , Wikipedia and ConceptNet [9]. Then, a KB-
augmented VQA model performs joint reasoning over the retrieved knowledge,
image, and question to predict the answer [10, 11, 12, 13, 14]. However, the
performance of these retrieval-based approaches is limited for two reasons:
(i) the required knowledge may not be successfully retrieved from the KBs; and
(ii) even if the required knowledge is retrieved, plenty of irrelevant
knowledge is inevitably introduced, which hampers the learning of VQA models.

\begin{overpic}[width=433.62pt]{fig1.pdf} \put(57.0,46.5){ \footnotesize{PICa
\cite[cite]{[\@@bibref{}{pica}{}{}]}}} \put(47.0,28.0){ \footnotesize{KAT
\cite[cite]{[\@@bibref{}{kat}{}{}]} / REVIVE
\cite[cite]{[\@@bibref{}{revive}{}{}]}} } \put(52.0,9.5){ \small{{Prophet
(ours)}}} \end{overpic} Figure 1: Conceptual comparisons of three knowledge-
based VQA frameworks using a frozen LLM model, _e.g._ , GPT-3 [1]. While PICa
[15], KAT [16], and REVIVE [17] directly feed the caption (C) and question (Q)
into the LLM as the prompt, we argue that the information they provide for the
LLM is insufficient thus cannot fully activate the LLM’s potential. In
contrast, our Prophet learns a vanilla VQA model without external knowledge to
produce _answer heuristics_ , which endows the LLM with richer and more task-
specific information for answer prediction. In contrast to the counterparts
that resort to specific VQA models and LLMs, our Prophet is more general that
can be instantiated with the combinations of different VQA models (_i.e._ ,
discriminative [18] and generative ones [19]) and different LLMs (_i.e._ ,
commercial [1] and open-source ones [20, 21]).

Apart from those studies using explicit KBs, another line of research resorts
to pretrained large language models (LLMs), _e.g._ , GPT-3 [1], as implicit
knowledge engines for knowledge acquisition. A pioneering work by PICa employs
the frozen GPT-3 model to answer the question with a formatted prompt as its
input [15]. Given a testing image-question pair, PICa first translates the
image into a caption using an off-the-shelf captioning model. The question,
caption, and a few in-context examples are then integrated into a textual
prompt that can induce GPT-3 to predict the answer directly. Thanks to the
powerful knowledge reasoning ability of GPT-3, PICa achieves significant
performance improvements compared to those retrieval-based methods using
explicit KBs. Inspired by PICa, KAT [16] and REVIVE [17] learn KB-augmented
VQA models to exploit both the implicit knowledge from LLMs and explicit
knowledge from KBs for answer prediction. The synergy of the two knowledge
resources brings further improvements to their models. Despite the promising
results achieved by these methods, they have not fully activated the
capability of the LLMs due to the following limitations:

  1. (i)

The generated captions cannot cover all the necessary information in the
image. Consider the example in Fig. 1, the caption “a group of people walk in
a city square” contributes nothing to answering the question “what fruit comes
from these trees”. In this situation, the LLM has to make an aimless and
biased guess to answer the question.

  2. (ii)

LLMs like GPT-3 employ a few-shot learning paradigm that requires a few in-
context examples to adapt to new tasks. Therefore, the choice of these
examples is critical to model performance. As reported in [15], all its
example selection strategies achieve far inferior performance to the oracle
strategy that uses the similarity of ground-truth answers.

We ask: _Is it possible to endow the LLM with some heuristics to enhance its
capacity for knowledge-based VQA?_

In this paper, we present Prophet—a conceptually simple yet effective
framework designed to prompt LLMs with answer heuristics for knowledge-based
VQA. By answer heuristics, we mean some promising answers that are presented
in a proper manner in the prompt. Specifically, we introduce two types of
complementary answer heuristics, namely _answer candidates_ and _answer-aware
examples_ , to overcome the limitations in (i) and (ii), respectively. Given a
testing input consisting of an image and a question, the answer candidates
refer to a list of promising answers to the testing input, where each answer
is associated with a confidence score. The answer-aware examples refer to a
list of in-context examples, where each example has a similar answer to the
testing input. Interestingly, these two types of answer heuristics can be
simultaneously obtained from any vanilla VQA model trained on a specific
knowledge-based VQA dataset. A schematic of Prophet is illustrated at the
bottom of Fig. 1.

Without bells and whistles, Prophet surpasses previous state-of-the-art
single-model results on the challenging OK-VQA and A-OKVQA datasets [7, 8],
including the heavily-engineered Flamingo-80B model trained on 1.8B image-text
pairs [2]. Moreover, Prophet is friendly to most researchers, as our results
can be reproduced using a single GPU and a number of GPT-3 invocations.

A preliminary version of this manuscript was published in [22]. Based on that
version, we have made the following contributions to further improve the
performance and validate the generality of Prophet: (i) we investigate diverse
types of VQA models, including the classical discriminative models trained
from scratch and the latest generative VQA models pretrained on large-scale
corpus; (ii) we expand the used LLM from the commercial GPT-3 model to a wide
range of open-source models; (iii) apart from OK-VQA and A-OKVQA, we conduct
more experiments on two other knowledge-based VQA datasets, namely ScienceQA
[23] and TextVQA [24]. The source code is made available
here111https://github.com/MILVLG/prophet. We hope these studies may serve as a
new baseline to inspire future research on knowledge-based VQA and universal
vision-language learning.

Figure 2: Our Prophet framework has two stages: answer heuristics generation
and heuristics-enhanced prompting. In the answer heuristics generation stage,
a vanilla VQA model trained on specific knowledge-based VQA dataset is
employed to generate two types of complementary answer heuristics, _i.e._ ,
answer candidates and answer-aware examples. In the heuristics-enhanced
prompting stage, the answer heuristics, question, and caption are integrated
into a formatted prompt to instruct a frozen LLM (_e.g._ , GPT-3) to predict
an answer. As shown in the example, both answer heuristics contribute to the
answer of “helium”.

##  2 Related Work

Visual Question Answering (VQA). VQA has been of growing interest over the
last few years. Recent studies in VQA research can be roughly divided into the
following categories: better visual features [25, 26, 27], more powerful model
architectures [28, 29, 30, 18], and more effective learning paradigms [31, 32,
33, 34, 35]. Most current state-of-the-art VQA methods employ the Transformer
architecture [36]. By incorporating vision-language pretraining on large-scale
datasets, they have approached or even surpassed human-level performance on
several representative benchmarks [2, 3, 4, 37, 38]. Besides these studies on
general-purpose VQA, there is also a growing trend towards exploring more
granular VQA tasks with specific reasoning skills, _e.g._ , neural-symbolic
reasoning [39, 40] and knowledge utilization [5, 7].
Knowledge-based VQA. The core of this task lies in knowledge acquisition and
integration. Early explorations parse the inputs into structured queries and
retrieve supporting knowledge from fixed knowledge bases (KBs) to obtain the
answers [5, 6]. As the provided knowledge resources are not sufficient to
represent general knowledge, subsequent research mainly focuses on acquiring
explicit knowledge from multiple open-domain knowledge resources, _e.g._ ,
ConceptNet [9], Wikipedia [41], and Google Images [12]. This retrieved
knowledge is integrated with the image-question pair for answer prediction
[12, 42, 13]. Motivated by the powerful capacities of LLMs (_e.g._ , GPT-3
[1]) in knowledge reasoning, recent state-of-the-art approaches regard an LLM
as an implicit knowledge engine. They either utilize it to predict answers
from given questions and extract visual captions [15] or to extract answer
candidates with evidence to improve answer prediction [16, 17]. Nevertheless,
they have not fully activated the reasoning capability of LLMs, as the
necessary visual information to answer the question is not represented
exactly. This motivates us to explore the strategies for prompting LLMs with
question-aware information (_i.e._ , answer heuristics). Similar to Prophet, a
concurrent work PromptCap also aims to enhance the input information for LLMs
by learning a question-aware captioning model [43]. However, PromptCap needs
to use LLM in both the training and testing phases, which incurs tremendous
computational costs as the training set is usually large. In contrast, Prophet
is more economical as it only utilizes LLM in the testing phase.
In-context learning. Unlike the _pretrain-then-finetune_ paradigm for language
models like BERT [44], GPT-3 innovatively introduces a _few-shot in-context
learning_ paradigm and has become the de facto standard for subsequent LLMs.
To adapt to a new task, GPT-3 only needs to concatenate a few examples of the
task with the input as the _prompt_ at inference time and requires no
parameter updates. This appealing property has inspired research on training
multimodal few-shot learners [2]. Empirical studies show that a huge model
(_e.g._ , 80B parameters in Flamingo [2]) is required for effective few-shot
learning, which is unaffordable for most people to reproduce their results.

##  3 The Prophet Framework

Our Prophet is a conceptually simple two-stage framework. In the answer
heuristics generation stage, a vanilla VQA model is learned to generate two
types of answer heuristics, _i.e._ , answer candidates and answer-aware
examples (detailed in $\S$3.2). In the heuristics-enhanced prompting stage,
the answer heuristics, question, and caption are integrated into a formatted
prompt to instruct a frozen LLM to predict an answer (detailed in $\S$3.3). An
overview of the Prophet framework is depicted in Fig. 2.

###  3.1 Preliminaries

Before presenting the Prophet, we briefly introduce the in-context learning
paradigm developed by GPT-3 and its adaptation to knowledge-based VQA by PICa
[15].

GPT-3 is an autoregressive language model pretrained on a tremendous dataset.
During inference, in-context few-shot learning formulates a new downstream
task as a text sequence generation task on the frozen model. Given a testing
input $\boldsymbol{x}$, its target $\boldsymbol{y}$ is predicted conditioned
on a formatted prompt
$\boldsymbol{p}(\boldsymbol{h},\mathcal{E},\boldsymbol{x})$, where
$\boldsymbol{h}$ refers to a prompt head, _aka_ instruction, that describes
the task,
$\mathcal{E}=\\{\boldsymbol{e}_{1},\boldsymbol{e}_{2},...,\boldsymbol{e}_{n}\\}$
corresponds to $n$ in-context examples. Let the target
$\boldsymbol{y}=(y^{1},y^{2},...,y^{L})$ be a text sequence of $L$ tokens. For
notational convenience, we denote $[l]$ as a set of natural numbers from 1 to
$l$ and use $\boldsymbol{y}^{[l]}=(y^{1},...,y^{l})$ to represent a sub-
sequence containing the first $l$ words of $\boldsymbol{y}$. At each decoding
step $l$, we have:

$y^{l}=\mathop{\text{argmax}}\limits_{\hat{y}^{l}}p_{\textrm{GPT-3}}(\hat{y}^{l}|\boldsymbol{p},\boldsymbol{y}^{[l-1]})$
(1)

where each in-context example
$\boldsymbol{e}_{i}=(\boldsymbol{x}_{i},\boldsymbol{y}_{i})$ contains an
input-target pair of the task, which is constructed manually or sampled from
the training set.

To adapt LLMs like GPT-3 to address the knowledge-based VQA task, the key is
to design proper prompts. Given a question $q$ and an image $v$ as inputs, the
VQA task aims to predict a target answer $a$. Since LLMs do not understand
images intrinsically, the image needs to be translated into a caption $c$
using an off-the-shelf captioning model. PICa formulates the testing input
$\boldsymbol{x}$ as the following template:

Context: $c$ \n Question: $q$ \n Answer:
---

where the variables marked in blue will be substituted by specific testing
inputs. \n stands for a line break in the template. Accordingly, each in-
context example $\boldsymbol{e_{i}}$ is formulated into a similar template as
follows:

Context: $c_{i}$ \n Question: $q_{i}$ \n Answer: $a_{i}$
---

where $c_{i}$, $q_{i}$, and $a_{i}$ refer to an image-question-answer triplet
collected from the training set. The complete prompt of PICa consists of a
fixed prompt head, a few in-context examples, and a testing input. This prompt
is fed into a frozen LLM for answer prediction.

Our Prophet inherits the pipeline of PICa. In addition, we introduce answer
heuristics into the prompt structure to better activate the reasoning
capability of the LLM, which leads to more accurate answers.

###  3.2 Stage-1: Answer Heuristics Generation

We introduce two types of answer heuristics: answer candidates and answer-
aware examples. Given a testing input consisting of an image and a question,
the answer candidates refer to a list of promising answers to the testing
input, where each answer is associated with a confidence score. The answer-
aware examples refer to a list of in-context examples, where each example has
similar answers to the testing input. Interestingly, these two types of answer
heuristics can be obtained simultaneously from any vanilla VQA model trained
on specific knowledge-based VQA task.

As shown in Fig. 3, existing VQA methods can be categorized into
_discriminative_ and _generative_ ones based on the ways they obtain answers.
This discrepancy leads to different strategies for answer heuristics
generation. We elaborate the strategy for each of the two classes of VQA
models below.

Figure 3: Discriminative _vs._ generative VQA models. Taking an image (V) and
a question (Q) as inputs, a typical discriminative VQA model like MCAN [18]
performs multi-class classification to predict the most relevant answer (may
contain multiple words) from a predefined answer vocabulary, while a typical
generative VQA model like mPLUG [19] iteratively predicts one answer word at a
time to constitute the final answer.

####  3.2.1 Discriminative VQA models

Denote a VQA training dataset as
$\mathcal{D}=\\{(v_{i},q_{i},a_{i})\\}_{i=1}^{M}$, where $v_{i},q_{i},a_{i}$
refer to the image, question, and answer, respectively. The most frequent
answers in the training set form an answer vocabulary
$\mathcal{V}=\\{\boldsymbol{w}_{j}\\}_{j=1}^{S}$, where $S$ is the answer
vocabulary size. A discriminative VQA model $\mathcal{M}_{\mathrm{disc}}$ is
learned from $\mathcal{D}$ to perform an $S$-way classification over the
answers. Generally, the model $\mathcal{M}_{\mathrm{disc}}$ can be separated
into two submodels, _i.e._ , a backbone $\mathcal{M}^{B}_{\mathrm{disc}}$ and
a prediction head $\mathcal{M}^{H}_{\mathrm{disc}}$. The backbone
$\mathcal{M}^{B}_{\mathrm{disc}}$ acts as an encoder to fuse multimodal inputs
$v$ and $q$ and obtain a fused feature $z$:

${z}=\mathcal{M}^{B}_{\mathrm{disc}}(v,q)$ (2)

The prediction head $\mathcal{M}_{H}$ simply adopts a linear layer followed by
a sigmoid function to project the fused feature $z$ into a score vector
$y\in\mathbb{R}^{S}$ over the answer vocabulary:

${y}=\mathcal{M}^{H}_{\mathrm{disc}}({z})$ (3)

where the $i$-th element of $y$ represents the confidence score for answer
$\boldsymbol{w}_{i}$. Based on the above definitions, we explain how to
generate the two types of answer heuristics below. Note that although the
learned VQA model $\mathcal{M}_{\mathrm{disc}}$ does not incorporate any
external knowledge, it can be used for knowledge-based VQA when trained
properly. We regard it as a reference model and compare its performance to
Prophet in the experiments to show the effectiveness of LLM for knowledge-
based VQA.
Answer candidates. Given a testing input $(v,q)$, we obtain its score vector
${y}$ for all answers using Eq.(3). Denoting $s_{i}\in\mathbb{R}^{+}$ as the
$i$-th element of $y$, we obtain the top-$K$ answers with the highest scores
as follows:

$\mathcal{I}_{\textrm{AC}}=\mathop{\text{argTopK}}\limits_{j\in\\{1,2,...,S\\}}s_{j}$
(4)

where $\mathcal{I}_{\textrm{AC}}$ denotes an index set of the top-$K$ answer
candidates. The answer candidates $\mathcal{C}$ are defined as follows:

$\mathcal{C}=\\{(\boldsymbol{w}_{j},s_{j})\ |\
j\in\mathcal{I}_{\textrm{AC}}\\}$ (5)

where $\boldsymbol{w}_{j}$ and $s_{j}$ are an answer candidate and its
confidence score, respectively. To make the formats of the in-context examples
and testing input consistent, for each example $\boldsymbol{e_{i}}$ we also
calculate and provide a set of answer candidates $\mathcal{C}_{i}$.
Answer-aware examples. Several previous studies have shown that the choice of
in-context examples is crucial for GPT-3’s few-shot learning performance [15].
Their results motivate us to devise an _answer-aware_ example selection
strategy.

Given a testing input $(v,q)$ and any training input $(v_{i},q_{i})$, we can
obtain their corresponding fused features $z$ and $z_{i}$ from Eq.(2) using
the trained model. Since the fused features are linearly projected for answer
prediction, we conjecture that these fused features lie in a _latent answer
space_ that contains rich semantics of the answers to the given image-question
pairs. If $z$ and $z_{i}$ are close in the latent space, they are more likely
to share similar answers and image-question inputs.

We calculate the cosine similarity of the fused feature between the testing
input and each training input, then select top-$N$ nearest neighbors in the
latent space as the answer-aware examples:

$\mathcal{I}_{\textrm{AE}}=\mathop{\text{argTopN}}\limits_{i\in\\{1,2,...,M\\}}\frac{{z}^{T}{z}_{i}}{\|{z}\|_{2}\|{z}_{i}\|_{2}}$
(6)

where $\mathcal{I}_{\textrm{AE}}$ is an index set of the top-$N$ similar
samples in $\mathcal{D}$. The answer-aware examples $\mathcal{E}$ are defined
as follows:

$\mathcal{E}=\\{(v_{i},q_{i},a_{i})\ |\ i\in\mathcal{I}_{\textrm{AE}}\\}$ (7)

Note that the fused features of the training inputs can be computed and stored
beforehand, allowing efficient answer-aware example selection.

####  3.2.2 Generative VQA models

Recent state-of-the-art VQA models tend to use generative model architectures
due to their remarkable scalability and generalizability [37, 19, 31].

Given the same VQA training dataset
$\mathcal{D}=\\{(v_{i},q_{i},a_{i})\\}_{i=1}^{M}$ as above, a generative VQA
model $\mathcal{M}_{\mathrm{gen}}$ is learned from $\mathcal{D}$ to generate
answers word-by-word from a pre-defined word vocabulary
$\mathcal{V}=\\{w_{j}\\}_{j=1}^{S}$, where $S$ is the word vocabulary size.
Each answer can be represented as a text sequence with a dynamic length of $L$
words:

$\boldsymbol{w}=(w^{1},w^{2},...,w^{L})$ (8)

where $w^{1}=[\mathtt{BOS}]$ refers to a special start-of-sentence token and
$w^{L}=[\mathtt{EOS}]$ refers to an end-of-sentence token.

Similar to the discriminative model, $\mathcal{M}_{\mathrm{gen}}$ can also be
separated into a backbone $\mathcal{M}^{B}_{\mathrm{gen}}$ and a prediction
head $\mathcal{M}^{H}_{\mathrm{gen}}$. The backbone
$\mathcal{M}^{B}_{\mathrm{gen}}$ corresponds to an encoder-decoder or a pure
decoder architecture that fuses multimodal inputs $v$ and $q$, and then
generates latent feature of each answer word using an autoregressive manner:

$z^{l}=\mathcal{M}^{B}_{\mathrm{gen}}(v,q,\boldsymbol{w}^{[l-1]})$ (9)

where $z^{l}$ denotes the latent feature of $l$-th answer word. On top of the
latent feature $z^{l}$, the prediction head $\mathcal{M}^{H}_{\mathrm{gen}}$
applies a linear projection (or a MLP) followed by a softmax function to
decode it into a score distribution $y^{l}\in\mathbb{R}^{S}$ over the whole
word vocabulary:

$y^{l}=\mathcal{M}^{H}_{\mathrm{gen}}(z^{l})$ (10)

where the $l$-th answer word $w^{l}$ is obtained from $y^{l}$ by greedily
choosing the word with the highest score. Until an $[\mathtt{EOS}]$ token is
generated, $w^{l}$ is appended to $\boldsymbol{w}^{[l-1]}$ to obtain
$\boldsymbol{w}^{[l]}$, which is iteratively fed into the model
$\mathcal{M}_{\mathrm{gen}}$ to predict the next word.
Answer candidates. Given a testing input $(v,q)$, we can obtain its most
relevant answer using the greedy decoding strategy above. However, how to
obtain the answer candidates consisting of the top-$K$ answers and their
confidence scores is not straightforward. We resort to the _beam search_
algorithm, which is widely used in neural machine translation [45] and visual
captioning [46], to address the issue.

Similar to Eq. (5), we denote the top-$K$ answer candidates as a set of tuples
as follows:

$\mathcal{C}=\\{(\boldsymbol{w}_{1},s_{1}),(\boldsymbol{w}_{2},s_{2}),...,(\boldsymbol{w}_{K},s_{K})\\}$
(11)

where each $\boldsymbol{w}_{j}$ represents an answer consisting of a sequence
of answer words and $s_{j}\in\mathbb{R}^{+}$ denotes its corresponding
confidence score calculated over all the answer words. The answer candidate
set $\mathcal{C}$ is obtained from the generative model
$\mathcal{M}_{\mathrm{gen}}$ equipped with the beam search strategy.
Specifically, we initialize each answer $\boldsymbol{w}_{j}$ with the same
$[\mathtt{BOS}]$ token. At each decoding step $l$, each $\boldsymbol{w}_{j}$
of length $l$ is first passed through $\mathcal{M}_{\mathrm{gen}}$ to obtain
its top-$K$ candidate words with the highest scores. After that, an _expand-
then-reduce_ strategy is performed to update the $K$ answers: (i) expand step:
each $\boldsymbol{w}_{j}$ is expanded $K$ times to combine with the $K$
candidate words, resulting in $K*K$ new candidates answers of length $l+1$;
(ii) reduce step: among the $K*K$ candidate answers, only the top-$K$ ones
with the highest accumulated scores $s=\sum_{i=1}^{l}\mathrm{log}~{}y^{i}$ are
retained, which are then regarded as the inputs to the next decoding step.
Answer-aware examples. Similar to the example selection strategy for
discriminative models, the answer-aware examples for generative models are
also obtained by performing kNN search in a latent answer space. It is worth
noting that the granularity of the latent features is different for the two
types of VQA models: each latent feature obtained from a discriminative VQA
model refers to an answer entry in the answer vocabulary, while each latent
feature obtained from a generative VQA model refers to an answer word.

Given a testing input $(v,q)$ and $i$-th training input $(v_{i},q_{i})$, the
latent features for their multi-word answers can be respectively represented
as feature groups $Z=[z^{1},z^{2},...,z^{L}]\in\mathbb{R}^{L\times d}$ and
$Z_{i}=[z_{i}^{1},z_{i}^{2},...,z_{i}^{L_{i}}]\in\mathbb{R}^{L_{i}\times d}$,
where $d$ is the common dimensionality of the latent answer space, $L$ and
$L_{i}$ refer to the answer lengths of $Z$ and $Z_{i}$, respectively. We
define a simple score function as follows to average the dot-product
similarity of each paired features $z_{j}\in Z$ and $z_{i}^{k}\in Z_{i}$ :

$\pi_{i}=\frac{1}{L*L_{i}}\sum\limits_{j=1}^{L}\sum\limits_{k=1}^{L_{i}}\frac{z^{j}z_{i}^{k}}{\|z^{j}\|_{2}\|z_{i}^{k}\|_{2}}$
(12)

Using the score function above, we obtain the top-$N$ nearest neighbors of the
query input in the training set and then format them as the answer-aware
examples $\mathcal{E}$ as follows:

$\displaystyle\mathcal{I}_{\textrm{AE}}$
$\displaystyle=\mathop{\text{argTopN}}\limits_{i\in\\{1,2,...,M\\}}\pi_{i}$
(13) $\displaystyle\mathcal{E}$ $\displaystyle=\\{(v_{i},q_{i},a_{i})\ |\
i\in\mathcal{I}_{\textrm{AE}}\\}$

where $\mathcal{I}_{\textrm{AE}}$ is an index set of the top-$N$ nearest
neighbors in the training set $\mathcal{D}$.

###  3.3 Stage-2: Heuristics-enhanced Prompting

After obtaining the answer heuristics (_i.e._ , answer candidates
$\mathcal{C}$ and answer-aware examples $\mathcal{E}$) from the stage-1, we
encode them into a heuristics-enhanced prompt to facilitate the few-shot
learning capacity of the LLM for knowledge-based VQA.

A prompt consists of a prompt head, a set of in-context examples, and a
testing input. The prompt head describes the VQA task in natural language. We
refer to the prompt head designed in PICa and supplement it with a new
description of the answer candidates. Although we encourage LLM to generate
answers according to the answer candidates, we also allow it to take broad
explorations and generate answers beyond the candidates. The complete format
of our prompt head is shown in the yellow box of Fig. 2.

Our in-context examples are derived from the obtained $N$ answer-aware
examples
$\mathcal{E}=\\{\boldsymbol{e}_{1},\boldsymbol{e}_{2},...,\boldsymbol{e}_{N}\\}$.
Based on PICa’s template in $\S$3.1, for example $\boldsymbol{e}_{i}$, we
introduce its answer candidates $\mathcal{C}_{i}$ by adding _one_ line of code
as follows:

Context: $c_{i}$ \n Question: $q_{i}$ \n
---
Candidates: $\boldsymbol{w}_{j_{1}}$($s_{j_{1}}$),
$\boldsymbol{w}_{j_{2}}$($s_{j_{2}}$),…,$\boldsymbol{w}_{j_{K}}$($s_{j_{K}}$)
\n
Answer: $a_{i}$

where $j_{1},j_{2},\cdots,j_{K}$ correspond to the actual indices of the
elements in $\mathcal{C}_{i}$. Each answer candidate $w_{j_{k}}$ is paired
with its confidence score $s_{j_{k}}$ within a bracket. The confidence scores
additionally offer the reliability of the corresponding answer candidates,
which helps the LLM focus more on the promising candidates and be more
tolerant of the less relevant candidates. For the testing input, its template
is similar to that for the in-context examples, except that the answer slot is
left blank for the LLM to fill with.

To better exploit available examples, we use the multi-query ensemble strategy
[15]. Specifically, we increase the number of answer-aware examples to $N$*$T$
to obtain $T$ paralleled prompts, where each prompt still contains $N$
examples. By prompting the LLM for $T$ times, we obtain $T$ answer
predictions. The majority voting is performed over the $T$ predictions to
determine the final answer. The effects of different $N$ and $T$ will be
verified in the experiments.

##  4 Experiments

We mainly evaluate the performance of Prophet on two prevalent knowledge-based
VQA datasets: OK-VQA [7] and A-OKVQA [8]. We conduct comprehensive ablation
experiments to explore the effectiveness of Prophet. By taking the ablation
results into account, we perform thorough comparisons of Prophet and state-of-
the-art methods. Moreover, we showcase the generalization ability of Prophet
on two diverse knowledge-based VQA datasets ScienceQA [23] and Text-VQA [24],
which require external science and OCR knowledge, respectively.

###  4.1 Datasets

OK-VQA is a commonly used knowledge-based VQA dataset [7]. The dataset
contains 9K and 5K image-question pairs for training and testing,
respectively. All questions are manually filtered to ensure that outside
knowledge is required to answer the questions. Each data sample is annotated
with ten open-ended answers. The accuracy computed by the soft scores is used
as the evaluation metric [47]. We use the 1.1 version of OK-VQA in the
experiments.
A-OKVQA is currently the largest knowledge-based VQA dataset [8]. The dataset
is split into three subsets: 17K training, 1K validation, and 7K testing. Each
question is annotated with ten open-ended answers for direct answer (DA)
evaluation. Besides, it provides a multiple choice (MC) evaluation to choose
the correct answer from four choices.
ScienceQA is a dataset that consists of about 21K questions over a diverse set
of science topics [23]. Out of the 21K questions, only the ‘IMG’ subset of
10.3K (48.7%) samples have image content, which is used in our experiments.
Consequently, the retained dataset consists of 6.2K training, 2.1K validation,
and 2.0K testing samples. The questions require high school-level science
knowledge to arrive at the correct answer chosen from multiple choices.
TextVQA contains 28K images and 45K questions, where each question requires
models to read and reason about the text in the image to give a correct answer
[24]. The dataset is split into three subsets of 34.6K training, 5K
validation, and 5.7K testing questions. Similar to OK-VQA, each question is
annotated with ten open-ended answers by humans and soft-voting accuracy is
used as the evaluation metric. Following the strategy in [48, 49], we
supplement the training set with the augmented VQA samples from ST-VQA [50].

###  4.2 Implementation Details

Default settings on OK-VQA. We use the MCAN-large [18] as our default VQA
model to generate answer heuristics. To improve the model capability, we
modify the original MCAN model by: (i) replacing the original bottom-up-
attention region-based features with the grid-based features extracted from
CLIP’s visual encoder with a RN50$\times$64 backbone [51]; and (ii) replacing
the original LSTM network with a pretrained BERT-large model [44].

Similar to [11], we apply the transfer learning paradigm to further enhance
the model capability. The model is first pretrained on the VQAv2 dataset [47]
and Visual Genome dataset [52]. To prevent data contamination, we remove those
samples from the pretraining dataset, whose images are used in the testing
split of OK-VQA. After that, the pretrained model is further finetuned on the
training split of OK-VQA to obtain our final VQA model. Note that the answer
vocabulary of the pretrained model (with 3,129 answers) is quite different
from the vocabulary of OK-VQA. To bridge this gap, we merge the answer
vocabulary of OK-VQA222Similar to [25], we collect answers that appear more
than eight times in the training set of OK-VQA, resulting in 2,794 answers.
with the existing vocabulary, resulting in an expanded answer vocabulary with
4,477 answers for model finetuning. This model is trained on a _single_ Nvidia
RTX 3090 GPU, which is affordable for most people.

During the prompting stage using LLMs, we follow PICa to use OSCAR+ as the
captioning model [26]. Unless otherwise noted, we set the number of answer
candidates $K$=10, the number of in-context examples $N$=16, and the number of
queries $T$=5 as our default settings. The default version of GPT-3 used in
our experiments is text-davinci-002 and the sampling temperature is set to 0.

VQA model, paradigm | stage-1 acc. | accuracy
---|---|---
ViLBERT, retrieval [12] | 35.20 | 40.28 (+5.08)
ViLBERT, prompt† | 35.28 | 44.97 (+9.69)
(a) Prompting _vs_. retrieval. Our prompting-based paradigm is more effective
than the retrieval-based one in MAVEx [12]. †: our re-implementation.

visual features | stage-1 acc. | accuracy
---|---|---
Bottom-Up [25] | 46.83 | 55.34 (+8.51)
VinVL [26] | 47.88 | 56.23 (+8.35)
CLIP-ViT-L/14 [51] | 52.03 | 60.12 (+8.09)
CLIP-RN50$\times$64 [51] | 53.04 | 60.84 (+7.80)
(b) Capability of VQA models. More powerful VQA models lead to higher
accuracies, but obtain slightly less relative improvements from stage-2.

#candidates ($K$) | hit rate | accuracy
---|---|---
0 | - | 49.63
1 | 53.04 | 56.04
5 | 75.20 | 60.17
10 | 79.83 | 60.84
(c) Answer candidates. They are crucial to Prophet and increasing $K$ leads to
better performance.

example selection |  hit rate | accuracy
---|---|---
(a) rand | 5.31 | 58.66
(b) ques + img [15] | 59.58 | 59.82
(c) fused | 83.63 | 60.84
(d) fused + ques + img | 82.45 | 60.38
(e) answer logits | 79.25 | 60.40
(d) Example selection strategy. Our answer-aware example selection based on
fused features is more effective than the others.

#examples ($N$) |  accuracy ($T$=1) |  accuracy ($T$=5)
---|---|---
0 | 49.97 | 49.97
1 | 54.89 | 56.75
8 | 57.49 | 59.91
16 | 57.52 | 60.84
20 | 57.91 | 61.10
(e) Numbers of examples and queries. Increasing $N$ and $T$ improves model
performance at the expense of linearly increasing overheads.

variants | accuracy
---|---
(a) default | 60.84
(b) w/o prompt head | 60.54
(c) w/o confidence scores | 55.46
(d) w/o image captions | 58.27
(e) default+tags [15] | 60.51
(f) Prompt contents. The default settings contain the exact necessary
information for prompting.

TABLE I: Ablation experiments for Prophet. All the reported results are
evaluated on the testing set of OK-VQA v1.1. The best result in each table is
bolded and the result with the default settings is marked in gray.

Settings on other datasets. The settings and strategies for OK-VQA can be
directly transferred to A-OKVQA to address its DA task. For the MC task, we
follow the strategy in [8] to project the predicted answer to the nearest
answer choice. Moreover, we design a Prophet variant for the MC task. It uses
a slightly different prompt by adding the multiple choices to in-context
examples and testing input, and instructs the LLM to _choose_ the correct one
from four choices.

For ScienceQA, we reuse all the default settings for OK-VQA. If a training
sample provides extra textual hint, we simply append the text to the generated
caption as the new context of the corresponding image. For TextVQA, we use the
commercial system from Amazon to extract OCR from images
333https://aws.amazon.com/textract/, whose effectiveness has been verified in
previous work [49]. The extracted OCR texts are provided in both the in-
context examples and testing input to instruct the LLM.

Settings of other VQA models. In addition to MCAN, we also experiment with one
generative VQA model mPLUG [19], which is first pretrained on task-agnostic
image-text corpus and then finetuned on specific VQA dataset. Following the
aforementioned two-stage transfer learning paradigm for MCAN, the pretrained
mPLUG model is first finetuned on the VQAv2 dataset and then further finetuned
on specific knowledge-based VQA dataset.

###  4.3 Ablation Studies

We conduct ablation experiments for Prophet on OK-VQA using the default
settings above. Results shown in Table I and Fig. 4 are discussed in detail
below.
Prompting _vs_. retrieval. Prophet uses a prompting-based paradigm to predict
the answer based on a set of promising answer candidates. In contrast, a
previous work MAVEx [12] exploits answer candidates but adopts a retrieval-
based paradigm to search knowledge from external KBs to determine the answer.
As both Prophet and MAVEx train a VQA model to generate answer candidates
(stage-1), we can compare the superiority of the two paradigms (stage-2). In
Table I(a), we show the performance of the two paradigms in terms of stage-1
accuracy and final accuracy, respectively.

For a fair comparison, we re-implement the VQA model used in MAVEx, _i.e._ ,
ViLBERT [32], to generate answer heuristics for our Prophet. From the results,
we can see that based on the same VQA model, our Prophet outperforms MAVEx by
a large margin (44.97% _vs_. 40.28%), showing the superiority of our
prompting-based paradigm over MAVEx’s retrieval-based paradigm in external
knowledge acquisition and integration.
Capability of VQA models. In Table I(b), we study how the VQA models of
different capabilities impact the performance of Prophet. To better control
the model capability, we use the same MCAN model trained with four visual
features: region-based Bottom-Up [25] and VinVL [26] features and grid-based
CLIP features from two backbones (ViT-L/14 and RN50$\times$64) [51]. Results
show that more powerful VQA models (reflected in the stage-1 accuracies) lead
to better performance of Prophet, as they provide answer heuristics of higher
quality. Combining the results in Table I(a), we also observe that more
powerful VQA models achieve less relative improvements from GPT-3, which can
be explained by the intrinsic diminishing return property. As a by-product, we
verify that the visual features are important to the performance of knowledge-
based VQA, which is consistent with the observations in [17]. The models with
CLIP-based visual features significantly outperform those with region-based
features, indicating that the CLIP’s visual features contain richer visual
knowledge due to large-scale pretraining.

In addition to using different visual features for MCAN, we can also replace
the whole MCAN model with any generative models pretrained on large-scale
multimodal datasets as mentioned in $\S$3.1. These results will be reported in
the main results.
Answer candidates. Table I(c) varies the number of answer candidates $K$ from
0 to 10 to explore its effect on Prophet. For each testing sample, if the
ground-truth answer is hit by one of the $K$ answer candidates, we accumulate
the soft score of that ground-truth answer444In practice, multiple ground-
truth answers are provided. If multiple answers are hit simultaneously, we
choose the answer with the largest soft score for accumulation.. The hit rate
is calculated over the testing set by dividing the accumulated score by the
number of samples.

From the results, we can see that: (i) without any answer candidates,
Prophet’s accuracy drops by 6.4 points ($K$=0 _vs_. $K$=1), showing the
importance of answer candidates in Prophet; (ii) with the increase of answer
candidates, the hit rate and final accuracy grow accordingly but they exhibit
a tendency to saturate. This is because the quality of answer candidates
eventually meets saturation as $K$ increases; (iii) when $K$=1, the final
accuracy is even higher than the hit rate (56.04% _vs_. 53.04%), which implies
that GPT-3 has a strong capability to correct the wrong answer candidates
while keeping the correct ones.
Example selection strategy. To show the effectiveness of our answer-aware
example selection strategy, we compare it to other example selection
strategies in Table I(d). The compared strategies include: (a) _rand_ :
examples that are randomly selected; (b) _ques + img_ : examples that are
selected based on the joint similarity of question and image features, which
is used in PICa; (c) _fused_ : our default strategy that selects examples
based on the similarity of fused features; (d) _fused + ques + img_ : a
combination of our default strategy and PICa’s strategy; and (e) _answer
logits_ : examples that are selected based on the similarity of answer logits
obtained in Eq.(3). Besides the final accuracy, we also report the hit rate of
answers within the selected examples for each strategy.

The results show that the accuracy is positively correlated with the hit rate
of answers, which verifies our hypothesis that answer-aware examples
contribute significantly to the performance of Prophet. Compared with other
strategies, our default strategy (c) achieves the best performance with the
highest hit rate. The strategy (d) that integrates other information (ques +
img) into the (c) leads to worse performance due to the introduction of
irrelevant and noisy information. Finally, strategy (e) reports slightly worse
performance than (c). We conjecture that this is because the answer logits
have lost too much information of the input question and image, which is also
useful for GPT-3 to perform knowledge reasoning.
Numbers of examples and queries. Table I(d) contains the ablation studies for
the numbers of examples and queries. We choose different numbers of examples
$N\in\\{0,1,8,16,20\\}$ for each query and different numbers of queries
$T\in\\{1,5\\}$, respectively. The results show that the performance of
Prophet improves with the increase of $N$ and $T$, which is consistent with
the results in PICa. By increasing $T$ from 1 to 5, the entries with larger
$N$ enjoy greater performance improvements at the expense of linearly
increasing overheads.

Interestingly, the Prophet variant with $N$=0 delivers worse performance than
the VQA model in stage-1 (49.97% _vs_. 53.04%), even though answer candidates
are provided. Meanwhile, when given one example ($N$=1), the Prophet variant
distinctly surpasses the VQA model (56.75% _vs_. 53.04%). This suggests the
necessity of few-shot in-context examples for GPT-3 to activate its capability
to adapt to the knowledge-based VQA task.
Prompt contents. In Table I(f), we ablate the prompt contents in the default
settings by: (b) removing the prompt head; (c) removing the confidence scores
for answer candidates; (d) removing image captions; and (e) adding predicted
tags from external models [15].

The results lead to the following observations: First, the confidence scores
are of critical importance to the performance of our Prophet. This is because
they carry the necessary information for GPT-3 to understand the answer
candidates. Second, without image captions, Prophet still works steadily. This
reflects the fact that our answer heuristics in prompts already provide
sufficient information for Prophet to solve the task. Third, the prompt head
is of less importance, indicating that GPT-3 is capable of understanding the
task directly from the in-context examples. Finally, introducing extra
information like object tags leads to a slight performance drop, which is
contrary to the results in PICa. We conjecture this information has already
been encoded in answer heuristics implicitly.

(a) behavior distribution

(b) per-type accuracy

Figure 4: Prophet’s prediction behaviors in terms of (a) distribution and (b) per-type accuracy. As Prophet takes $K$ answer candidates as inputs, we define three prediction behaviors for Prophet as follows: “keep top-1”, “in top 2-$K$”, and “beyond top $K$”. All the testing samples can be categorized into one of the three classes.  Stage 1 pred. Stage 2 pred. |  correct |  wrong
---|---|---
correct | 54.4% | 4.2%
wrong | 12.0% | 29.4%
TABLE II: Prophet’s combinatorial prediction behaviors in two stages. Prophet
maintains the majority of correct predictions at stage-1, and the accuracy
improvement by stage-2 is mainly because the number of _wrong-to-correct_
samples is larger than that of the _correct-to-wrong_ samples.

Prediction behaviors in different stages. In Table I(b), we can observe a
significant performance improvement of Prophet (stage-2) over its
corresponding MCAN model (stage-1). To better understand this improvement, we
conduct a statistical analysis of Prophet’s prediction behaviors. As Prophet
takes $K$ answer candidates from MCAN as inputs, we define three prediction
behaviors for Prophet: “keep top-1”, “in top 2-$K$”, and “beyond top-$K$”. All
the testing samples can be categorized into one of the three classes. The
statistical results in Figure 4 show that: (i) for 68.1% of the testing
samples (green slice), Prophet keeps the top-1 predictions of MCAN. These
samples achieve a 69% accuracy and are mostly easy samples; (ii) for 21.8% of
the testing samples (blue slice), Prophet selects answers from the top 2-$K$
answer candidates. These samples are relatively hard, so that MCAN delivers a
24% accuracy while Prophet has a much higher 40% accuracy; (iii) for the
remaining 10.1% of the testing samples (yellow slice), Prophet predicts
answers beyond the answer candidates555The probability that Prophet’s
prediction is constituted of the combination of candidates is rare that can be
neglected.. For these most difficult samples, MCAN only delivers a 12%
accuracy while Prophet magnificently achieves a 42% accuracy.

As a supplement to the above results, we calculate the distribution of four
situations of the predictions from stage-1 and stage-2 in Table II. From the
results, we can see that: (i) Prophet maintains the majority of correct
predictions by MCAN and only 4.2% samples are overturned; (ii) the improvement
of Prophet is mainly due to the fact that the proportion of _wrong-to-correct_
samples (12.4%) is larger than that of the _correct-to-wrong_ samples (4.2%);
(iii) there are still a considerable amount of samples (29.4%) that both MCAN
and Prophet fail to give the correct answer, which leaves sufficient room for
future improvement.

LLM (version or size) |  per-sample average cost | accuracy
---|---|---
commercial models |  |
GPT-3 (text-davinci-002) | $0.2 | 60.8
GPT-3 (3.5-turbo-instruct) | $0.015 | 58.9
open-source models |  |
†LLaMA-1 (7B) [20] | 2.6s | 51.8
†LLaMA-1 (13B) [20] | 4.6s | 56.1
†LLaMA-1 (30B) | 8.7s | 57.1
†LLaMA-1 (65B) [20] | 16.6s | 58.8
†Falcon (7B) [21] | 2.7s | 50.5
†Falcon (40B) [21] | 14.1s | 57.1
LLaMA-2 (7B) [53] | 2.7s | 56.6
LLaMA-2-Chat (7B) [53] | 2.7s | 54.0
LLaMA-2 (13B) [53] | 4.8s | 57.9
LLaMA-2-Chat (13B) [53] | 4.8 | 56.5
LLaMA-2 (70B) [53] | 18.3s | 59.6
Mistral (7B) [54] | 3.0s | 59.7
TABLE III: Ablation study of different LLMs. All variants use the default
settings and are evaluated on the testing set of OK-VQA. The per-sample
average costs of the open-source models are measured by the GPU running time
on a server with A100 GPUs while the costs of the commercial models are
measured by money. † indicates the LLM’s max token length is insufficient for
$N$=16 examples. For these LLMs, we reduce $N$ to fit their maximum capacity.

Different LLMs. In Table III, we investigate the effects of different LLMs by
replacing the default GPT-3 (text-davinci-002) with the latest commercial and
open-source models. From the results, we have the following observations: (i)
the capability of the default GPT-3 model significantly outperforms all the
compared LLMs, including its accelerated variant (3.5-turbo-instruct) with
0.075$\times$ running cost; (ii) for the LLMs of the same class but different
sizes (_e.g._ , 7B and 13B LLaMA-1 models [20]), the large-size ones show
better performance than the small-size ones at the expense of near-linearly
increasing running time; (iii) the chat-oriented variants like LLaMA-2-Chat
[53], which are additionally trained by instruction tuning and human feedback
[55], deliver inferior performance to their non-chatty counterparts. This can
be explained by the introduced _alignment tax_ when aligning the model with
human behaviors; (iv) with only 7B model parameters, the latest LLM Mistral
[54] reports near GPT-3 level performance, revealing the potential of open-
source LLMs in the near future.

###  4.4 Main Results

For the comparisons below, we use all the default settings except the number
of examples $N$. We set $N$=20 for OK-VQA and A-OKVQA and respectively set
$N$=7 and $N$=16 for ScienceQA and TextVQA as they need extra hint and OCR
tokens. By instantiating Prophet with two VQA models, we obtain Prophet (MCAN)
and Prophet (mPLUG).
Comparative results on OK-VQA. Table IV contains the comparisons of our
Prophet and existing state-of-the-art methods on OK-VQA. The table is split
into three sections. The first section lists the retrieval-based methods
leveraging external KBs [10, 56, 11, 42, 12, 13]. The second section contains
the methods that are directly pretrained on a large-scale multimodal corpus
[57, 2, 58]. The last section shows the methods that incorporate the large
language model GPT-3, which is publicly available via an online API [15, 16,
17, 43].

method | accuracy
---|---
methods with external knowledge bases |
Mucko [10] | 29.2∗
ConceptBERT [56] | 33.7∗
KRISP [11] | 38.9
Visual Retriever-Reader [42] | 39.2
MAVEx [12] | 40.3
TRiG [13] | 49.4
UnifER [59] | 42.1
methods with multimodal pretraining |
Unified-IO (2.8B) [57] | 54.0
Flamingo (80B) [2] | 57.8
PALI (17B) [58] | 64.5
methods with GPT-3 API |
PICa [15] | 48.0
KAT† [16] | 53.1
REVIVE† [17] | 56.6
PromptCap (OFA)† [43] | 60.4
Prophet (MCAN) | 61.1
Prophet (mPLUG) | 62.5
TABLE IV: Comparisons to the state-of-the-art methods on OK-VQA testing set.
The compared methods are split into three groups based on their knowledge
resources and usages. ∗: accuracy is evaluated on OK-VQA v1.0. †: method needs
to query GPT-3 during training.

Our Prophet belongs to the last section. It outperforms all the compared
methods by a distinct margin. Prophet is 13.1 points higher than PICa [15]
when both methods use GPT-3 as the only knowledge resource. This confirms our
hypothesis that the capacity of GPT-3 has not been fully activated in previous
studies. Compared to KAT [16] and REVIVE [17], which utilize GPT-3 and other
external KBs together in sophisticated systems, our Prophet is much simpler
and more effective. Moreover, KAT, REVIVE, and PromptCap need to use GPT-3 to
process all the training samples for their model training, which significantly
increases the costs. In contrast, our Prophet only uses GPT-3 at inference
time, which is more economical. Compared to the Flamingo-80B equipped with 32
in-context examples [2], Prophet (MCAN) delivers a significant performance
improvement. Despite the fact that Prophet (MCAN) has a clear performance gap
compared to PALI-17B [58], Prophet is more resource-efficient from the
perspective of reproducibility666Flamingo-80B is trained on 1,536 TPUv4 for 15
days and PALI is trained on 1,024 TPUv4 for 7 days, which are unaffordable for
most researchers. In contrast, Prophet (MCAN) uses one RTX-3090 to train a VQA
model for 4 days and a certain number of GPT-3 invocations.. Finally, by
replacing MCAN with the pretrained generative model mPLUG, our method exhibits
a 1.4-point further improvement, showing the substantial contribution of a
powerful VQA model for Prophet.

method | DA | MC
---|---|---
val | test | val | test
ClipCap [8] | 30.9 | 25.9 | 56.9 | 51.4
ViLBERT [8] | 30.6 | 25.9 | 49.1 | 41.5
LXMERT [8] | 30.7 | 25.9 | 51.4 | 41.6
KRISP [8] | 33.7 | 27.1 | 51.9 | 42.2
GPV-2 [8] | 48.6 | 40.7 | 60.3 | 53.7
Unified-IO [57] | - | 45.2 | - | -
PromptCap (OFA) [43] | 56.3 | 59.6 | 73.2 | 73.1
Prophet (MCAN) | 58.2 | 55.7 | 76.4 | 73.6
Prophet (mPLUG) | 64.7 | 58.5 | 76.6 | 75.1
TABLE V: Comparisons to the state-of-the-art methods on A-OKVQA. DA and MC
refer to the direct-answer and multiple-choice tasks, respectively. For the MC
task, we devise a Prophet variant with a slightly different prompt.

Comparative results on A-OKVQA. Table V contains the comparative results on
the challenging A-OKVQA dataset. The results on the DA task show that Prophet
(MCAN) model significantly outperform most existing approaches, reflecting the
effectiveness and generalization of our method. Compared to the current state-
of-the-art method PromptCap [43] which also involves a pretrained VQA model
OFA [37] and GPT-3, Prophet (MCAN) exhibits similar performance when using a
weaker VQA model.

For the MC task, we introduce a Prophet variant with slightly modifying the
prompt used in the original Prophet. In particular, we add the multiple-choice
information into both the in-context examples and testing input to instruct
GPT-3 to _choose_ the correct one from four choices. Compared with all the
methods, Prophet (MCAN) surpasses all the counterparts on the MC task, showing
the flexibility and scalability of Prophet. Moreover, Prophet (mPLUG) steadily
outperforms Prophet (MCAN), emphasizing the significance of a powerful VQA
model to Prophet.
Results on ScienceQA and TextVQA To verify the generalization ability of
Prophet, we conduct experiments on two additional knowledge-based VQA datasets
ScienceQA (IMG) and TextVQA, which require different types of knowledge
(_i.e._ , scientific knowledge and OCR knowledge) than that for OK-VQA and
A-OKVQA. A Table VI shows that comparative results of Prophet and existing
state-of-the-art methods on respective datasets. As we have witnessed the
steady improvements of mPLUG over MCAN, we only report the results for Prophet
(mPLUG) on these two datasets. Specifically, Prophet surpasses all the
counterparts on ScienceQA (IMG), including the average human performance [23]
and the latest LLaVa model trained with visual instruction tuning [60]. On
TextVQA, Prophet outperforms the published state-of-the-art methods, including
those methods with text-aware or layout-aware pretraining on large-scale
scene-text image datasets [49, 48].

method | accuracy
---|---
MCAN [18] | 51.2
GPT-3 [1] | 65.7
Chameleon [61] | 77.6
InstructBLIP [62] | 79.5
LLaMA-Adapter [63] | 80.3
MM-CoT [64] | 82.9
Human Average [23] | 87.5
LLaVa [60] | 88.0
Prophet (mPLUG) | 88.2
(a) ScienceQA (IMG)

method | accuracy
---|---
LoRRA [24] | 27.6
M4C [65] | 40.5
PromptCap [43] | 51.9
TAG [66] | 53.7
TAP [48] | 54.0
Flamingo-80B [2] | 54.1
mPLUG-doc [67] | 57.6
LaTr [49] | 59.6
Prophet (mPLUG) | 61.3
(b) TextVQA

TABLE VI: Comparisons to the state-of-the-art methods on the testing set of
ScienceQA and TextVQA, respectively.

###  4.5 Qualitative Analysis

In Fig. 5, we illustrate two typical samples consisting of the testing inputs
and their in-context examples to explain how the answer heuristics work. The
results show that the synergy of answer candidates and the answer-aware
examples facilitates the generation of high-quality answers. In the first
sample, the candidate answer ‘ _lace_ ’ with a low confidence score is finally
selected by the LLM as it frequently appears in the in-context examples. In
the second sample, we see that Prophet can make a correct prediction beyond
the answer candidates when the proper answer heuristic (the word ‘ _leash_ ’)
is provided in the in-context examples.

Figure 5: We show two typical samples consisting of the testing inputs (left)
and their in-context examples (right). The predicted answers of Prophet have a
high probability to appear in the answer candidates and answer-aware examples,
showing the effectiveness of answer heuristics in enhancing LLM’s ability to
predict the correct answer.

##  5 Broader Impact

From a multimodal LLM (MLLM) point of view, Prophet is a loosely-coupled MLLM
consisting of a vision-language (VL) model and a frozen LLM, aiming to endow
the VL model with knowledge reasoning ability. Compared with the tightly-
coupled MLLMs (_e.g._ , Flamingo [2] and LLaVa [60]) which jointly optimize
the VL model and LLM in an end-to-end manner, Prophet is more flexible that
can support any open-source or commercial LLM.

Moreover, Prophet can also be regarded as a _learning-to-prompt_ paradigm that
learns an external model to generate prompts to better comprehend the target
task, thus facilitating the capability of the pretrained LLM (or MLLM). From
this point of view, recent studies like VoxPoser [68] and SoM-Prompting [69]
share a similar idea with our work. We believe this paradigm can be widely
used in a variety of LLM-related tasks.

##  6 Conclusion

In this paper, we present Prophet—a conceptually simple framework which uses
LLMs as the knowledge engine for knowledge-based VQA. To better activate the
few-shot learning capacity of LLMs, we introduce a novel paradigm to prompt
LLMs with two types of complementary answer heuristics. Extensive ablations,
comparative experiments, and comprehensive analyses on four diverse knowledge-
based VQA datasets show the superiority of Prophet over all existing state-of-
the-art methods. Notably, Prophet can be instantiated with varied combinations
of a wide range of VQA models and LLMs, showing its flexibility, scalability,
and generalizability. We hope that our work can inspire future research on
knowledge-based VQA and universal multimodal learning in the era of LLMs.

##  Appendix A More Implementation Details

###  A.1 The Default VQA Model

Our default VQA model is carefully designed in terms of model architecture and
training strategy. In the following table, we show the improvements of our
default MCAN model over the counterparts trained from scratch. More details
are provided next.

from scratch, original model [18] |  from scratch, improved model |  transfer learning, improved model
---|---|---
31.5 | 35.6 | 53.0

Improved model architecture. We introduce an improved variant of MCAN [18]
based on its open-sourced MCAN-large implementation. Our modifications to the
model architecture include: (i) we replace the original bottom-up-attention
features with the grid-based features extracted from the CLIP’s visual encoder
with RN50$\times$64 backbone [51]; (ii) we introduce the RoPE mechanism [70]
to each image self-attention layer of MCAN to supplement the grid-based
features with positional information; and (iii) we replace the original LSTM
network with a pre-trained BERT-large model [44] as the text encoder before
MCAN. Table VII shows the accuracies of different model variants on the
testing set of OK-VQA. By progressively adding the modifications to the
original MCAN model, our improved MCAN model reports a 53.0% accuracy, which
is on par with current state-of-the-art methods like KAT [16].

case | OK-VQA accuracy
---|---
original MCAN | 43.6
\+ CLIP visual feats | 49.6
\+ RoPE mechanism | 50.3
\+ BERT as the text encoder | 53.0
TABLE VII: Ablations for model architectures. ‘+’ denotes each modification is applied to the previous variant. training strategy | OK-VQA accuracy
---|---
(a) train from scratch | 35.6
(b) pretrain, w/o finetune | 41.1
(c) w/ finetune, replace last layer | 47.7
(d) w/ finetune, append new answers | 53.0
TABLE VIII: Ablations for training strategies. All variants use the improved
model architecture in the last row in Table VII.  Please answer the question
according to the context and the answer candidates. Each answer candidate is
associated with a
---
confidence score within a bracket. The true answer may not be included in the
candidates.
===
Context: The motorcycle racers are getting ready for a race.
===
Question: What sport are these guys doing?
===
Candidates: motorcross(0.94), motocross(0.79), bike(0.35), dirt bike(0.28),
motorcycle(0.03),
bmx(0.03), cycling(0.02), motorbike(0.02), race(0.02), bicycle(0.02)
===
Answer: motorcross
===
Context: a black motorcycle parked in a parking lot.
===
Question: What sport can you use this for?
===
Candidates: race(0.53), motorcycle(0.41), motocross(0.19), bike(0.17),
motorcross(0.15),
cycling(0.11), dirt bike(0.10), ride(0.08), bicycling(0.01), bicycle(0.01)
===
Answer:
TABLE IX: An exemplar prompt for the standard Prophet. We show _one_ in-
context example here due to space limitations. Following the implementations
in PICa [15] and KAT [16], we use a special symbol ‘===’ to separate each two
lines. Please choose the correct answer in the choices according to the
context, the question and the answer candidates. Each answer
---
candidate is associated with a confidence score within a bracket. The true
answer may not be included in the candidates.
===
Context: A young man riding a skateboard on a sidewalk.
===
Question: What part of his body will be most harmed by the item in his mouth?
===
Candidates: skateboard(0.02), nothing(0.02), table(0.01), leg(0.01),
helmet(0.00), knees(0.00),
skateboarding(0.00), head(0.00), teeth(0.00), falling(0.00)
===
Choices: (A) back, (B) lungs, (C) feet, (D) eyes
===
Answer: (B)
===
Context: a young boy kneeling on a skateboard on the street.
===
Question: What did this lad likely injure here?
===
Candidates: skateboard(0.18), shoes(0.02), shoe(0.02), skateboarding(0.01),
street(0.01),
flowers(0.01), skating(0.01), boy(0.01), head(0.00), skateboarder(0.00)
===
Choices: (A) knee, (B) elbow, (C) rear, (D) board
===
Answer:
TABLE X: An exemplar prompt for the Prophet variant on the MC task of A-OKVQA.
Compared to the standard prompt in Table IX, we add one extra line of choices
for the example and testing input, and change the output format to adapt to
the multiple-choice task. All the differences are marked in red.  Please
choose the correct answer in the choices according to the context, the
question and the answer candidates. Each answer
---
candidate is associated with a confidence score within a bracket. The true
answer may not be included in the candidates.
===
Context: A picture of a black and white model of a molecule. The model below
represents graphite. Graphite is used to make
pencil lead.
===
Question: Complete the statement. Graphite is ().
===
Candidates: an elementary substance(1.00), a compound(0.02), an adult
substance(0.01), an an elementary substance(0.01)
===
Choices: (A) a compound, (B) an elementary substance
===
Answer: (B)
===
Context: A pair of eye glasses with the word h on them. The model below
represents a molecule of hydrogen. Hydrogen gas was
once used to make large airships, such as blimps, float. It is no longer used
in airships because it catches fire easily.
===
Question:Complete the statement. Hydrogen is ().
===
Candidates: a compound(0.68), an elementary substance(0.32), the same
substance(0.00), the same amount(0.00)
===
Choices: (A) an elementary substance, (B) a compound
===
Answer:
TABLE XI: An exemplar prompt for the Prophet variant on ScienceQA (IMG). The
sentences marked in red are the optional text hints provided by the dataset.
Please answer the question according to the context and the answer candidates.
Each answer candidate is associated with a
---
confidence score within a bracket. The true answer may not be included in the
candidates.
===
Context: A close up of a cell phone with a keyboard.
===
OCR: Market, 3, Facebook, Browser, 5, 4, 6, 1, 8, 30.
===
Question: How many apps are on this page excluding market?
===
Candidates: 6(0.20), 5(0.19), 8(0.18), 9(0.12), 7(0.08), answering
does(0.05),10(0.05),13(0.05),12(0.04),4(0.04)
===
Answer: 7
===
Context: A screenshot of a yahoo mail page.
===
OCR: Free, Page, Nake WT My Page, ADVERTISEMENT, YAHOO!, FREE Camera Phone,
Notepad, MAIL, Yaboo! Mail.
===
Question: What is free on this page?
===
Candidates: amera(0.40), video camera(0.29), video(0.13), photos(0.04), video
call(0.04), webcam(0.03), videos(0.03),
photography(0.01), photoshop(0.01), internet explorer(0.01)
===
Answer:
TABLE XII: An exemplar prompt for the Prophet variant on TextVQA. Compared to
the standard prompt, we additionally introduce the OCR tokens (marked in red)
extracted from an off-the-shelf OCR system.

Training recipe. We first pretrain the model on the augmented _train+val+vg_
dataset from VQAv2 [47] and Visual Genome[52], with excluding the samples
whose images are used in the testing split of OK-VQA to avoid data
contamination. The settings for the pretraining stage are identical to the
original implementation of MCAN. After that, the model is finetuned on the
downstream OK-VQA and A-OKVQA datasets, respectively. For finetuning, the
commonly used strategy is to replace the last linear layer (_i.e._ , the
classification layer) with a new layer to adapt to the answer vocabulary of
the downstream dataset. However, the answer vocabularies of the pretraining
and finetuning datasets are _partially_ overlapped. To maximally utilize the
pretrained model parameters in the last layer, we inherit the parameters of
existing answers and append new parameters for the new answers. After that, we
freeze all the pretrained parameters and only update the new parameters for
one epoch as a warm-up, and then train all model parameters for the rest
training epochs.

Table VIII shows the effects of different training strategies. Even without
finetuning, the pretrained model (b) is superior to the model trained from
scratch (a), implying the importance of pretraining. Moreover, our new
finetuning strategy (d) leads to significantly better performance than the
commonly used strategy (c), showing the effectiveness of inheriting model
parameters for existing answers.

###  A.2 Prompt Formats

We show an exemplar prompt for the standard Prophet in Table IX and an
exemplar prompt for the variant designed for the MC task of A-OKVQA in Table
X. The exemplar prompts for ScienceQA and TextVQA are illustrated in Table XI
and XII, respectively.

##  Appendix B More Qualitative and Quantitative Analyses

We provide more in-depth analyses of Prophet’s performance on the testing set
of OKVQA. All results are carried out using the default settings.

category | MCAN | Prophet
---|---|---
Plants and Animals | 52.58 | 63.67
Science and Technology | 48.10 | 48.81
Sports and Recreation | 59.08 | 66.00
Geography, History, Language and Culture | 52.48 | 62.98
Brands, Companies and Products | 51.98 | 54.77
Vehicles and Transportation | 50.82 | 58.01
Cooking and Food | 55.53 | 62.09
Weather and Climate | 65.12 | 68.37
People and Everyday life | 49.44 | 54.67
Objects, Material and Clothing | 50.05 | 57.20
TABLE XIII: Per-category accuracies  of MCAN (stage-1) and Prophet (stage-2).
This performance improvements of using GPT-3 are observed on all categories.

We show the per-type accuracies of MCAN (stage-1) and Prophet (stage-2) in
Table XIII. Prophet outperforms MCAN on all categories, indicating that
generality of the knowledge in GPT-3. The improvement on the “Science and
Technology” category is not as large as the rest categories. which can be
explained that the required knowledge for this category is more specialized
and professional. These questions are also challenging for humans.

failure cause | proportion
---|---
(a) insufficient visual understanding | 27.3%
(b) incorrect knowledge reasoning |  44.1%
(c) correct but differently expressed answer | 22.8%
(d) others | 5.8%
TABLE XIV: The distribution of failure causes by human studies.  Figure 6:
Different categories and prediction behaviors. Each row contains four testing
samples from a specific knowledge category. The first to the third columns
correspond to the correctly answered samples of different prediction behaviors
(_i.e._ , keep top-1, in top 2-$K$, and beyond top-$K$). The last column
contains failure samples.

We perform human studies to analyze the causes of wrong predictions in Table
XIV. For each category, we randomly sample 10% testing samples that Prophet
fails to get the correct answer. This results in 172 samples. We ask three
annotators to categorize each sample into one of the following four failure
causes: (a) insufficient visual understanding; (b) incorrect knowledge
reasoning; (c) correct but differently expressed answer; (d) others (_e.g._ ,
the failure is caused by the ambiguity of the question). From the results, we
can see that the cause of “(b) incorrect knowledge reasoning” accounts for the
highest proportion, which suggests that the bottleneck of Prophet still lies
in the knowledge acquisition and reasoning. The cause of “(a) insufficient
visual understanding” has the second highest proportion, showing the potential
of devising more powerful VQA models. The cause of “(c) correct but
differently expressed answer” also accounts for a considerable proportion.
This reflects the limitation of the annotations and evaluation metric of OK-
VQA.

Figure 6 demonstrates some testing samples from different knowledge
categories. In the 1st-3rd columns, we show the correctly answered samples
with different prediction behaviors (_i.e._ , keep top-1, in top 2-$K$, and
beyond top-$K$). The visualized results indicate that Prophet can adaptively
choose suitable answers from candidates. In the last column, we show some
failure samples, implying that there is still room for future improvement.

## References

  * [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” in _NeurIPS_ , 2020, pp. 1877–1901.
  * [2] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” in _NeurIPS_ , 2022.
  * [3] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li _et al._ , “Florence: A new foundation model for computer vision,” _arXiv preprint arXiv:2111.11432_ , 2021.
  * [4] W. Wang, H. Bao, L. Dong, and F. Wei, “Vlmo: Unified vision-language pre-training with mixture-of-modality-experts,” in _NeurIPS_ , 2021.
  * [5] P. Wang, Q. Wu, C. Shen, A. Dick, and A. Van Den Hengel, “Fvqa: Fact-based visual question answering,” _IEEE TPAMI_ , vol. 40, no. 10, pp. 2413–2427, 2017.
  * [6] P. Wang, Q. Wu, C. Shen, A. R. Dick, and A. van den Hengel, “Explicit knowledge-based reasoning for visual question answering,” in _IJCAI_ , 2017\.
  * [7] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-vqa: A visual question answering benchmark requiring external knowledge,” in _CVPR_ , 2019, pp. 3195–3204.
  * [8] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi, “A-okvqa: A benchmark for visual question answering using world knowledge,” in _ECCV_. Springer, 2022, pp. 146–162.
  * [9] H. Liu and P. Singh, “Conceptnet: a practical commonsense reasoning tool-kit,” _BT technology journal_ , vol. 22, no. 4, pp. 211–226, 2004.
  * [10] Z. Zhu, J. Yu, Y. Wang, Y. Sun, Y. Hu, and Q. Wu, “Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering,” in _IJCAI_ , 2020, pp. 1097–1103.
  * [11] K. Marino, X. Chen, D. Parikh, A. Gupta, and M. Rohrbach, “Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa,” in _CVPR_ , 2021, pp. 14 111–14 121.
  * [12] J. Wu, J. Lu, A. Sabharwal, and R. Mottaghi, “Multi-modal answer validation for knowledge-based vqa,” in _AAAI_ , 2022, pp. 2712–2721.
  * [13] F. Gao, Q. Ping, G. Thattai, A. Reganti, Y. N. Wu, and P. Natarajan, “Transform-retrieve-generate: Natural language-centric outside-knowledge visual question answering,” in _CVPR_ , 2022, pp. 5067–5077.
  * [14] Y. Ding, J. Yu, B. Liu, Y. Hu, M. Cui, and Q. Wu, “Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering,” in _CVPR_ , 2022, pp. 5089–5098.
  * [15] Z. Yang, Z. Gan, J. Wang, X. Hu, Y. Lu, Z. Liu, and L. Wang, “An empirical study of gpt-3 for few-shot knowledge-based vqa,” in _AAAI_ , 2022, pp. 3081–3089.
  * [16] L. Gui, B. Wang, Q. Huang, A. Hauptmann, Y. Bisk, and J. Gao, “Kat: A knowledge augmented transformer for vision-and-language,” _NAACL_ , 2021\.
  * [17] Y. Lin, Y. Xie, D. Chen, Y. Xu, C. Zhu, and L. Yuan, “REVIVE: Regional visual representation matters in knowledge-based visual question answering,” in _NeurIPS_ , 2022.
  * [18] Z. Yu, J. Yu, Y. Cui, D. Tao, and Q. Tian, “Deep modular co-attention networks for visual question answering,” in _CVPR_ , 2019, pp. 6281–6290.
  * [19] C. Li, H. Xu, J. Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. Cao _et al._ , “mplug: Effective and efficient vision-language learning by cross-modal skip-connections,” in _EMNLP_ , 2022, pp. 7241–7259.
  * [20] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv preprint arXiv:2302.13971_ , 2023.
  * [21] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, “The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only,” _arXiv preprint arXiv:2306.01116_ , 2023.
  * [22] Z. Shao, Z. Yu, M. Wang, and J. Yu, “Prompting large language models with answer heuristics for knowledge-based visual question answering,” in _CVPR_ , 2023, pp. 14 974–14 983.
  * [23] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” in _NeurIPS_ , 2022, pp. 2507–2521.
  * [24] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” in _CVPR_ , 2019, pp. 8317–8326.
  * [25] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in _CVPR_ , 2018, pp. 6077–6086.
  * [26] P. Zhang, X. Li, X. Hu, J. Yang, L. Zhang, L. Wang, Y. Choi, and J. Gao, “Vinvl: Revisiting visual representations in vision-language models,” in _CVPR_ , 2021, pp. 5579–5588.
  * [27] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.-W. Chang, Z. Yao, and K. Keutzer, “How much can clip benefit vision-and-language tasks?” _ICLR_ , 2022.
  * [28] R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko, “Learning to reason: End-to-end module networks for visual question answering,” in _ICCV_ , 2017, pp. 804–813.
  * [29] J.-H. Kim, J. Jun, and B.-T. Zhang, “Bilinear attention networks,” _NeurIPS_ , vol. 31, 2018.
  * [30] L. Li, Z. Gan, Y. Cheng, and J. Liu, “Relation-aware graph attention network for visual question answering,” in _ICCV_ , 2019, pp. 10 313–10 322.
  * [31] J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” _ICML_ , 2022.
  * [32] J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in _NeurIPS_ , 2019.
  * [33] H. Tan and M. Bansal, “Lxmert: Learning cross-modality encoder representations from transformers,” _EMNLP_ , 2019.
  * [34] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in _ECCV_ , 2020, pp. 104–120.
  * [35] Y. Cui, Z. Yu, C. Wang, Z. Zhao, J. Zhang, M. Wang, and J. Yu, “Rosita: Enhancing vision-and-language semantic alignments via cross-and intra-modal knowledge integration,” in _ACM MM_ , 2021, pp. 797–806.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _NeurIPS_ , vol. 30, 2017.
  * [37] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang, “OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,” in _ICML_ , 2022, pp. 21 218–23 340.
  * [38] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, “Coca: Contrastive captioners are image-text foundation models,” _TMLR_ , 2022.
  * [39] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick, “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning,” in _CVPR_ , 2017, pp. 2901–2910.
  * [40] D. A. Hudson and C. D. Manning, “Gqa: A new dataset for real-world visual reasoning and compositional question answering,” in _CVPR_ , 2019, pp. 6700–6709.
  * [41] D. Vrandečić and M. Krötzsch, “Wikidata: A free collaborative knowledgebase,” _Communications of the ACM_ , vol. 57, no. 10, pp. 78–85, 2014.
  * [42] M. Luo, Y. Zeng, P. Banerjee, and C. Baral, “Weakly-supervised visual-retriever-reader for knowledge-based question answering,” _EMNLP_ , pp. 6417–6431, 2021.
  * [43] Y. Hu, H. Hua, Z. Yang, W. Shi, N. A. Smith, and J. Luo, “Promptcap: Prompt-guided image captioning for vqa with gpt-3,” in _ICCV_ , 2023, pp. 2963–2975.
  * [44] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _NAACL_ , 2019, pp. 4171–4186.
  * [45] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _NeurIPS_ , 2014.
  * [46] J. Yu, J. Li, Z. Yu, and Q. Huang, “Multimodal transformer with multi-view visual representation for image captioning,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 30, no. 12, pp. 4467–4480, 2020.
  * [47] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering,” in _CVPR_ , 2017.
  * [48] Z. Yang, Y. Lu, J. Wang, X. Yin, D. Florencio, L. Wang, C. Zhang, L. Zhang, and J. Luo, “Tap: Text-aware pre-training for text-vqa and text-caption,” in _CVPR_ , 2021, pp. 8751–8761.
  * [49] A. F. Biten, R. Litman, Y. Xie, S. Appalaraju, and R. Manmatha, “Latr: Layout-aware transformer for scene-text vqa,” in _CVPR_ , 2022, pp. 16 548–16 558.
  * [50] A. F. Biten, R. Tito, A. Mafla, L. Gomez, M. Rusinol, E. Valveny, C. Jawahar, and D. Karatzas, “Scene text visual question answering,” in _ICCV_ , 2019, pp. 4291–4301.
  * [51] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _ICML_ , 2021, pp. 8748–8763.
  * [52] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma _et al._ , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” _IJCV_ , vol. 123, no. 1, pp. 32–73, 2017.
  * [53] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [54] M. AI, “Mistral 7b,” _arXiv preprint arXiv:2310.06825_ , 2023.
  * [55] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” in _NeurIPS_ , 2022, pp. 27 730–27 744.
  * [56] F. Gardères, M. Ziaeefard, B. Abeloos, and F. Lecue, “Conceptbert: Concept-aware representation for visual question answering,” in _EMNLP_ , 2020, pp. 489–498.
  * [57] J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi, “Unified-io: A unified model for vision, language, and multi-modal tasks,” _arXiv preprint arXiv:2206.08916_ , 2022.
  * [58] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer _et al._ , “Pali: A jointly-scaled multilingual language-image model,” in _ICLR_ , 2023.
  * [59] Y. Guo, L. Nie, Y. Wong, Y. Liu, Z. Cheng, and M. Kankanhalli, “A unified end-to-end retriever-reader framework for knowledge-based vqa,” in _ACM MM_ , 2022, pp. 2061–2069.
  * [60] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * [61] P. Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.-C. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” _arXiv preprint arXiv:2304.09842_ , 2023.
  * [62] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” 2023.
  * [63] R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y. Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” _arXiv preprint arXiv:2303.16199_ , 2023.
  * [64] Z. Zhang, A. Zhang, M. Li, H. Zhao, G. Karypis, and A. Smola, “Multimodal chain-of-thought reasoning in language models,” _arXiv preprint arXiv:2302.00923_ , 2023.
  * [65] R. Hu, A. Singh, T. Darrell, and M. Rohrbach, “Iterative answer prediction with pointer-augmented multimodal transformers for textvqa,” in _CVPR_ , 2020\.
  * [66] J. Wang, M. Gao, Y. Hu, R. R. Selvaraju, C. Ramaiah, R. Xu, J. F. JaJa, and L. S. Davis, “Tag: Boosting text-vqa via text-aware visual question-answer generation,” 2022.
  * [67] J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, Y. Dan, C. Zhao, G. Xu, C. Li, J. Tian _et al._ , “mplug-docowl: Modularized multimodal large language model for document understanding,” _arXiv preprint arXiv:2307.02499_ , 2023.
  * [68] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3d value maps for robotic manipulation with language models,” _arXiv preprint arXiv:2307.05973_ , 2023.
  * [69] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” _arXiv preprint arXiv:2310.11441_ , 2023.
  * [70] J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” _arXiv preprint arXiv:2104.09864_ , 2021.