# K-VQG: Knowledge-aware Visual Question Generation for Common-sense
Acquisition

Kohei Uehara
The University of Tokyo
<EMAIL_ADDRESS>
Tatsuya Harada
The University of Tokyo / RIKEN
<EMAIL_ADDRESS>

###### Abstract

Visual Question Generation (VQG) is a task to generate questions from images.
When humans ask questions about an image, their goal is often to acquire some
new knowledge. However, existing studies on VQG have mainly addressed question
generation from answers or question categories, overlooking the objectives of
knowledge acquisition. To introduce a knowledge acquisition perspective into
VQG, we constructed a novel knowledge-aware VQG dataset called K-VQG. This is
the first large, humanly annotated dataset in which questions regarding images
are tied to structured knowledge. We also developed a new VQG model that can
encode and use knowledge as the target for a question. The experiment results
show that our model outperforms existing models on the K-VQG dataset.

_Keywords_ Visual Question Generation, Knowledge Acquisition, Common-sense
knowledge

##  1 Introduction

Asking questions is an important ability for humans in acquiring new
knowledge. Humans ask questions regarding what they see to acquire new
knowledge and become more intelligent. Therefore, to develop machine
intelligence that can actively learn about the world, it is essential to study
systems that can ask questions about what they see and acquire new knowledge.

Visual Question Generation (VQG) is a research field that aims to give
machines such ability to ask questions about an image. VQG was initially
studied as a task that simply uses an image as input and generates a question
related to the image [19]. However, it is impossible to control the questions
to be generated using only images as input because the targets and contents of
the questions are extremely diverse.

Figure 1: We proposed a new dataset and task in which the model is required to
generate a target-knowledge aware question for a given image. In this task,
the model is given a knowledge triplet with a missing part and is expected to
generate a question that can complement the missing part.

Recent research on VQG has focused on the way to providing information about
the target of a question to the VQG model. Existing studies have used possible
answers [14, 15], answer types [11, 25], answer categories [24], and question-
types [6] as target information. However, when using the answer as a
condition, the answer to the question must be known before generating the
question. Since questions are usually asked without knowing the answer, such a
problem setting is unnatural. Other target information used in existing
studies controls only the rough target of the question and cannot be used to
generate questions that ask for specific knowledge.

To solve these problems and establish a more natural and practical setting for
VQG, we introduce K-VQG, which is a task that utilizes target knowledge, i.e.,
knowledge to be obtained by the question, as target information. Following
previous studies on structured knowledge [23, 9], we represent knowledge as a
triplet of three words or phrases, i.e., $<$head, relation, tail$>$.
Specifically, the model takes an image and masked target knowledge, which is a
knowledge triplet in which a part of the triplet is masked out as input and
generates a question such that the answer will be helpful in complementing the
missing part.

For example, in the top example of Figure 1, the target knowledge is $<$lion,
is a, feline$>$, and the masked target knowledge is $<$something, is a,
feline$>$. The expected output would then be a question related to the
knowledge and whose answer would be “lion”, e.g., “What tan feline animal that
is on the grass called?” On the other hand, the question like “What animal in
the image is the top of the food chain?” is indeed a question whose answer
will be “lion”, but it is not related to the target knowledge.

Since there is no dataset with the necessary annotations (e.g., images,
questions, and associated knowledge triplets) for this task, we constructed a
new dataset called the K-VQG dataset. Our K-VQG dataset is the first VQG
dataset that is common-sense aware, human-annotated, and large-scale.

To solve this task, it is necessary to develop a model that can understand the
visual information of an image and masked target knowledge information
simultaneously. Existing methods for VQG consider only simple target
information, such as answers and categories, and thus cannot handle complex
auxiliary information, such as a knowledge triplet. Thus, we developed a novel
model for K-VQG, which can encode the image and masked target knowledge using
a multi-modal transformer based encoder to generate questions.

Our contributions are summarized as follows:

  * •

We introduce a novel VQG dataset with knowledge annotations called K-VQG.

  * •

We propose a knowledge-aware VQG model that uses a masked knowledge triplet as
input.

  * •

We evaluate the performance of the proposed model on the constructed dataset.

##  2 Related Work

###  2.1 Knowledge-aware VQA/VQG Dataset

In this section, we introduce Visual Question Answering (VQA) datasets in
addition to VQG datasets because even datasets built for VQA can be used as
VQG datasets by replacing the inputs and outputs. We summarize the main
features of various datasets in Table 1.

The largest and best-known VQA dataset is the VQA v1/v2 dataset [2, 8], which
is also used in the VQA challenge competition. The VQA v1/v2 dataset is the
most commonly used dataset in VQG studies [15, 14, 10, 11]. However, these
datasets do not contain any knowledge annotations.

There are several knowledge-aware VQA datasets such as the FVQA [28], OK-VQA
[17], K-VQA [22], and CRIC datasets [7]. The FVQA dataset [28], in which the
questions are annotated with common-sense triplets, is similar to our own.
However, the FVQA dataset is relatively small ($\sim$5K questions), and many
of the questions tend to refer primarily to the target knowledge and less to
the content of the images. The questions in the FVQA dataset often refer to
the image with only phrases like “…in the image”. Such questions can be easily
generated without understanding the content of the image and are therefore
unsuitable for use in VQG. The OK-VQA dataset [17] is intended to be a VQA
dataset that requires knowledge and is larger than the FVQA dataset ($\sim$10K
questions); however, it lacks annotations on “which knowledge is relevant to
the question.” The K-VQA dataset [22] is specialized for knowledge of named
entities (e.g., “Who is to the left of Barack Obama?”), and its question
annotations are template-based, making it less generalizable. CRIC [7] is a
more recently proposed dataset. This dataset is similar to that proposed in
our study in that it is a VQA dataset with common-sense triplet annotations.
However, this dataset is not annotated by humans, but is a rule-based dataset
that automatically generates sentences from scene graph information.

Compared to the existing datasets mentioned above, our dataset is the first
dataset that has all the features: the questions are associated with common-
sense knowledge triplet, annotated by humans, bounding box annotations of the
question target, and large scale.

Table 1:  Comparison of key features of the major VQG/knowledge-aware VQA datasets. Our dataset is the first manually-annotated VQG dataset that contains knowledge annotations and bounding box annotations.  | Num. of Q |  | knowledge
---
type?
| structured
---
knowledge?
| target
---
bounding box?
| manually
---
annotated?
VQAv2 [8] | 1.1M | N/A | ✗ | ✗ | ✓
FVQA [28] | 5,826 | common-sense | ✓ | ✗ | ✓
OK-VQA [17] | 14,055 | open knowledge | ✗ | ✗ | ✓
K-VQA [22] | 183,007 | named entities | ✓ | ✗ | ✗
CRIC [7] | 1.3M | common-sense | ✓ | ✓ | ✗
K-VQG | 16,098 | common-sense | ✓ | ✓ | ✓

###  2.2 VQG Model

VQG is the task of generating questions associated with images. The earliest
VQG model [19] used an RNN model to generate questions using only an image as
the input. However, such a model conditioned only on images cannot control the
target of a question. Therefore, researchers have been studying ways to
control the target of a question by providing additional information. In
addition to images, iQAN [14] and iVQA [15] use answers as inputs to generate
questions that can produce the desired answers. With these methods, the answer
to the question must be known in advance. Since questions are usually asked
without knowing the answers, such problem setting is unnatural.

Other methods use categories of answers as conditions for VQG [11, 25]. With
these methods, it is not necessary to know the answers to the questions;
therefore, the aforementioned problem can be overcome. However, there is a
problem that the granularity of the answer categories greatly affect the
quality of the control of the question content. Although existing studies [11,
25] use 15 categories, the classification is rather coarse because all answers
related to the name of the object are gathered in the “object” category. This
means that, when there are multiple objects in an image, it is impossible to
control which object should be the target of the generated question.

With our method, the input is a partially masked common-sense triplet. Thus,
our method has the advantage of being able to control the target in more
detail than the existing VQG models, and it is also easy to apply the acquired
information to a knowledge database.

##  3 K-VQG Task and Dataset

First, we provide an overview of the K-VQG task, which is a VQG task for
knowledge acquisition. In the K-VQG task, the model is given a masked target
knowledge triplet and an image, and the model is expected to generate a
question that can acquire the target knowledge. The masked target triplet is a
knowledge triplet in which a part of the question to be answered is masked,
e.g., $<$[MASK], IsA, feline$>$. By contrast, the target knowledge is a
complete triplet in which the masked parts are filled, e.g., $<$lion, IsA,
feline$>$. For example, the goal of this task is to generate questions from a
masked target triplet, such as $<$[MASK], IsA, feline$>$, such that “lion” can
be obtained as an answer, and knowledge $<$lion, IsA, feline$>$ can be
acquired.

Next, we describe the construction of the K-VQG dataset. Each sample in the
dataset contains the following information: the (1) image, (2) question, (3)
answer, (4) target knowledge triplet, (5) bounding box of the question target.

We asked crowd workers of Amazon Mechanical Turk
(AMT)111https://www.mturk.com/ to annotate the data. We sampled the images
from the Visual Genome dataset [12] and selected the target object and
candidates for the target knowledge (Subsection 3.1.1 (a)). We then asked the
workers to select one target knowledge and write questions about the image
that required the target knowledge to answer (Subsection 3.1.2 (b)). We
further conduct the question validation process to ensure the quality of the
dataset (Subsection 3.1.3 (c)).

###  3.1 Dataset Construction

####  3.1.1 (a) Knowledge triplet collection.

We utilized ConceptNet [23] and Atomic${}^{20}_{20}$ [9] as the sources of the
common-sense triplets.

ConceptNet is a large-scale knowledge base that contains knowledge collected
from several resources. Knowledge in ConceptNet is represented as a triplet of
the form $<$head, relation, tail$>$, such as $<$cat, AtLocation, sofa$>$.
ConceptNet contains approximately 34 million triplets and 37 types of
relations. Some relations seem to be unnatural targets for questions regarding
images, such as DistinctFrom or MotivatedByGoal. Thus, we selected 15 types of
relations that were considered suitable as targets for the questions.

Figure 2:  Screenshot of the AMT task (excluding instruction due to space
limitation). The information provided to the worker was displayed at the top
of the screen, including the image, target object, and candidate knowledge
triplets. Below that, there are sections for the selection of the answer
phrases and writing knowledge-aware questions corresponding to the selected
answers.

The second source of knowledge is Atomic${}^{20}_{20}$. The
Atomic${}^{20}_{20}$ consists of more then 1M knowledge triplets about
physical-entity relations (e.g., $<$bread, ObjectUse, make french toast$>$),
event-centered relations (e.g., $<$PersonX eats spinach, isAfter, PersonX
makes dinner$>$), and social-interactions (e.g., $<$PersonX calls a friend,
xIntent, to socialize with their friend$>$). We used only physical-entity
relations for our dataset construction because the other relation types were
less relevant to the images in the Visual Genome.

After the above pre-processing, we merged these two knowledge datasets. Then,
to remove knowledge that is unrelated to any objects in the images, we queried
the entity appearing as the head of the knowledge in the Visual Genome object
list and removed the knowledge if there was no matching object.

Finally, we obtained a total of $\sim$150K knowledge triplets as candidate
knowledge.

####  3.1.2 (b) Question collection.

We show the screenshot of the AMT interface in Figure 2.

In order to maintain quality, we selected workers who resided in the U.S. or
Canada and had an approval rate greater than 97%. The workers were given the
following information: the target image, a bounding box representing the area
of the target object (i.e., the head entity of the candidate knowledge), the
name of the target object, and a list of candidate knowledge triplets (up to
15). They were then asked to write knowledge-aware questions with the
following steps:

  1. 1.

From the list of candidate knowledge, select one knowledge that is appropriate
for the image and the target object.

  2. 2.

Select a phrase from the selected knowledge (i.e., head entity or tail entity)
to be the answer.

  3. 3.

Write a question whose answer will be the selected phrase and requires the
knowledge the worker have chosen to answer.

Table 2: Dataset Statistics. We compare the K-VQG dataset with FVQA and VQA v2 dataset. Num. of head/tail answers indicate the number of answers which is the head or tail entity of the knowledge triplet. Note that the FVQA dataset does not provide such information, and we automatically counted the number. However, because of spelling inconsistencies, we could not obtain an exact count, and thus we used an approximate number here.  | K-VQG | FVQA [28] | VQAv2 [8]
---|---|---|---
Num. of questions | 16,098 | 5,826 | 443,757
– Num. of head answers | 11,588 |  $\sim$4,430 | N/A
– Num. of tail answers | 4,510 |  $\sim$1,240 | N/A
Num. of images | 13,648 | 2,190 | 82,783
Num. of unique answers | 2,819 | 1,427 | 22,531
Num. of unique knowledge | 6,084 | 4,180 | N/A
Num. of unique head | 527 | 847 | N/A
Num. of unique tail | 4,922 | 2,871 | N/A
Average answer length | 1.46 | 1.23 | 1.10
Average question length | 13.88 | 9.55 | 6.20
Num. of non-knowledge words in questions | 3.35 | 0.99 | N/A

To make the VQG model capable of properly understanding the content of an
image, it is desirable for the questions to describe the relationships between
objects in the image. Thus, we instructed the workers to write questions that
included a description of the position of the object in relation to other
objects in the image, more than simple phrases such as “…in the image”. In
addition, we instructed them to assume that the bounding box of the target
object is not visible, i.e., phrases such as “surrounded by a red frame” or
“with a bounding box” are prohibited.

####  3.1.3 (c) Question validation.

To ensure the quality of the collected questions, we further conducted
validation of the collected annotations by AMT. We asked workers to evaluate
questions with the following criteria: (1) whether the question refers to the
visual content of the image, (2) whether the target knowledge is related to
the question, (3) whether the target knowledge is related to the image and the
target object, (4) whether the question contains typos or grammatical errors,
(5) whether the answer is proper for the question. We asked three workers per
question for evaluation, and excluded the questions in which all workers
unanimously gave negative ratings for any of the evaluation criteria. Note
that we evaluated some of the data ourselves in advance, and rejected
submissions from workers whose agreement rate with our evaluation was less
than 60%, in order to maintain the quality of the evaluation.

###  3.2 Dataset Statistics

Figure 3: The distribution of question lengths in K-VQG dataset, FVQA dataset
and VQA v2 dataset. The K-VQG dataset tends to have longer questions than the
other datasets.  Figure 4:  Word cloud for questions, answers, and head
entities in the K-VQG dataset. Note that basic stopwords are excluded for the
word cloud for questions.  Figure 5: Example questions and the corresponding
images, answers, target knowledge from the K-VQG dataset.

The basic statistics of our dataset and two existing datasets, FVQA and VQAv2,
are shown in Table 2.

We collected 16,098 questions, corresponding to 13,648 images, and 6,084
knowledge triplets. The K-VQG dataset has 2,819 unique answers. Among the
5,220 knowledge triplets, there are 527 unique heads, 15 unique relations, and
4,922 unique tails. Our K-VQG dataset is significantly larger than FVQA, which
is an existing knowledge-aware dataset.

From these statistics, we can conclude that our dataset is more challenging
for the VQG model. First, our dataset has a longer sentence length for
questions and answers than other datasets. This can also be observed in Figure
3, which shows the distribution of the length of the questions in each
dataset. In addition, compared with FVQA, our dataset has more non-knowledge
words in the questions (3.35 vs. 0.99). This means that most of the questions
in FVQA consist of words derived from the knowledge triplet, whereas the
questions in our dataset contain many words that are not derived from the
knowledge triplet (e.g., words regarding the content of the image). In other
words, for the FVQA dataset, the VQG model could generate questions without
understanding the content of the images. However, because the questions in the
K-VQG dataset require references to the image content, generating proper
questions is much more difficult.

In Figure 4, we show the word clouds of the most frequent words in the
questions, answers, and head entities in the annotations of the K-VQG dataset.
This indicates that the questions in our dataset pertain to diverse fields
(e.g., food, animals, and location).

We show some examples from the K-VQG dataset in Figure 5.

##  4 Model

Figure 6: The overview of the model. Our model takes an image and a target
knowledge triplet as input, and convert them to fused features by multi-modal
Transformer encoder. Then, a Transformer decoder takes the fused features as
input and generates a question in an auto-regressive manner.

We developed a baseline question generation model, which is an encoder-decoder
type model using a transformer [26]. The overview of the model is shown in
Figure 6. Our model consists of an encoder that encodes the image information
and target knowledge information and a decoder that uses the output of the
encoder to generate the question.

###  4.1 Encoder

Our encoder uses a pre-trained UNITER [4], which is a multi-modal transformer
model that can encode image and text information. In this model, the encoder
takes the visual embeddings $\bm{v}$ of the image and the target knowledge
embeddings $\bm{k}$ as the input and outputs the encoded representation
$\bm{h}$, that is, $\bm{h}=\mathrm{Enc}(\bm{v},\;\bm{k})$.

####  4.1.1 Visual Embeddings.

To obtain visual embeddings $\bm{v}$, we use a pre-trained Faster R-CNN model
[21] and extract region features [1] of the image. Following [4], to provide
the positional information of each image region, a seven-dimensional vector
representing the coordinates and area of the region was encoded by a linear
layer and added to the region image features.

####  4.1.2 Target Knowledge Embeddings.

As described in Section 3, we used partially masked knowledge triplets as
input to the model. We treat the masked target knowledge triplet as a sequence
of words. Input masked target knowledge is tokenized as asequence of tokens
$\bm{k}=\\{\bm{w}_{h},\;w_{\mathtt{[SEP]}},\;\bm{w}_{r},\;w_{\mathtt{[SEP]}},\;\bm{w}_{t}\\}$.
Here, $w_{\mathtt{[SEP]}}$ is a special token that indicates the separation of
each part, and $\bm{w}_{h},\;\bm{w}_{r},\;\bm{w}_{t}$ denote the tokens of the
head, relation, and tail phrases, respectively, e.g.,
$\bm{w}_{h}=\\{w_{h1},\;w_{h2},\;\ldots,\;w_{hn}\\}$. If the head or tail is
the masked part, token $\bm{w}$ is replaced by a special token
$w_{\mathtt{[TGT]}}$.

###  4.2 Decoder

The decoder is a module that receives the encoded input image and target
knowledge, and outputs the question, that is, $\bm{q}=\mathrm{Dec}(\bm{h})$.
Following the recent success of transformers in language generation, we
developed a transformer-based model for the decoder. Our decoder is an
autoregressive transformer model, adapted from BART [13], and consists of
several transformer blocks, each of which has a multi-head cross-attention and
self-attention mechanism.

Our model was trained in a teacher-forcing manner by minimizing the negative
conditional log-likelihood loss. The loss function can be expressed through
the following equation:

$L_{LM}=-\sum^{|\bm{q}|}_{n=1}\log
P_{\theta}(\bm{q}_{n}\,|\,\bm{q}_{<n},\,\bm{h}).$ (1)

###  4.3 Implementation Details

Following UNITER, we set the number of Transformer blocks in the encoder and
decoder to 12, and the number of hidden units in each block to 768. We
initialized the weights of the encoder from the pre-trained UNITER
model222downloaded using the script at https://github.com/ChenRocks/UNITER. We
used the AdamW optimizer [16] with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. As
the learning rate scheduling, we adapted the cosine annealing scheduling,
where warm-up steps were set to 10% of the total training steps. The maximum
learning rate was set to be $1.0\times 10^{-5}$. We trained the model for 2K
steps. The training took two hours on 8 $\times$ Tesla A100 GPU.

##  5 Experiments

We tested our model and several existing methods on the K-VQG dataset. We
split the dataset into training and validation sets, and we used the
validation set to evaluate the performance of the model. Out of total of
16,098 questions, 12,891 questions were used for training, and 3,207 questions
were used for validation. Note that we made sure to split the dataset so that
the images used in the UNITER pre-training did not contaminate the validation
split of the dataset.

###  5.1 Baselines.

We used several existing methods as the baselines. We did not use any answer-
aware VQG models because we did not assume a situation in which the model
already knew the expected answer. Thus, we pick VQG models that take images
and/or answer categories as input. We automatically annotated the answer
categories. If the answer is a word that is the head of the knowledge triplet,
we use hypernym dictionary in WordNet [3] to determine the answer category. If
the answer is the tail of the triplet, we use this relation as the answer
category.

I2Q [19]: The I2Q model is a baseline model based on the approach in [19] that
uses only the image as the input and generates a question.

IC2Q: The IC2Q model uses the image and the answer category as the inputs.

V-IC2Q [10, 11]: The V-IC2Q model is a variational auto-encoder (VAE) based
method, which encodes the answer category and question into a latent space,
and decodes the latent vector to generate a question.

IM-VQG [11]: IM-VQG model is another VAE based method. The model is trained to
maximize the mutual information between the image, question, and expected
answer. Simultaneously, another latent space is learned to encode the answer
category, which enables the model to generate questions from only the image
and category inputs, without any expected answers.

###  5.2 Input ablation.

To demonstrate the importance of input to the model, we performed an input
ablation study in which either the image or the target knowledge is excluded
from the input to the model (Ours w/o image, Ours w/o knowledge).

###  5.3 Evaluation metrics.

Following previous VQG research, we used BLEU [20], METEOR [5], and CIDEr [27]
as evaluation metrics.

In the K-VQG task, it is also important to evaluate whether the generated
questions correctly yield the target knowledge. To this end, we used the
Target Knowledge Parser to predict the masked target knowledge triplet from
the generated questions and checked the consistency with the expected
knowledge triplet. The Target Knowledge Parser has a similar structure as the
K-VQG model. It has a UNITER-based encoder to encode images and questions and
a BART-based decoder to generate/recover masked target knowledge. We used
Triplet-BLEU to evaluate the overall agreement between the generated triplets
and the ground truth by calculating the BLEU-4 score. In addition, we used
Head-Acc, Relation-Acc, and Tail-Acc to evaluate whether each part of the
triplet is correct.

Table 3: Qualitative results on the K-VQG dataset. The left-side of the table shows the metrics used to evaluate the quality of the questions. Here, B-4, M, and C represent BLEU-4, METEOR, and CIDEr, respectively. The right-side of the table shows the metrics for the knowledge consistency. Tri-BLEU, H-Acc, R-Acc, and T-Acc denote Triplet-BLEU, Head-Acc, Relation-Acc, and Tail-Acc, respectively. For all metrics, higher values are better. | Question Quality | Knowledge Consistency
---|---|---
| B-4 | M | C | Tri-BLEU | H-Acc | R-Acc | T-Acc
I2Q [19] | 11.74 | 17.05 | 27.30 | 4.50 | 69.69 | 55.35 | 1.15
IC2Q | 12.37 | 16.69 | 31.01 | 7.97 | 75.34 | 58.62 | 27.91
V-IC2Q [10, 11] | 11.78 | 17.18 | 28.72 | 4.70 | 68.66 | 55.60 | 1.53
IM-VQG [11] | 11.44 | 17.07 | 26.19 | 4.10 | 68.07 | 55.32 | 1.71
Ours w/o image | 17.28 | 21.06 | 113.1 | 61.99 | 81.95 | 83.13 | 58.59
Ours w/o knowledge | 10.65 | 16.45 | 33.92 | 6.99 | 65.73 | 51.01 | 4.37
Ours | 18.84 | 22.79 | 131.04 | 64.33 | 84.72 | 82.44 | 66.20

###  5.4 Results

We show the experimental results in Table 3. The left side of the table shows
the results in terms of the quality of the generated questions, and the right
side shows the metric of whether the generated questions yield the desired
knowledge.

####  5.4.1 Question Quality (vs. baselines)

For all metrics used to evaluate the quality of the question, our method
outperformed the baselines (Ours vs. others). The baseline method uses only
image (I2Q) or image and category (IC2Q, V-IC2Q, IM-VQG) information as input
for inference, which suggests that the model has not achieved the ability to
sufficiently control the content of the questions to be generated. By
contrast, our method directly encodes the target knowledge information and
thus succeeds in generating questions with content closer to the ground truth.

Figure 7:  Output examples of our method on the K-VQG dataset. We show the
input images, target knowledge, ground-truth questions, and generated
questions.

####  5.4.2 Knowledge Consistency (vs. baselines)

The right side of Table 3 shows the metrics for knowledge consistency. In
terms of Tri-BLEU, which evaluates the overall quality of the generated
triplet, our method significantly improves the score compared with other
methods. In addition, for part-level accuracy (Head-Accuracy, Relation-
Accuracy, and Tail-Accuracy), our method outperformed the other methods. For
Head-Accuracy and Relation-Accuracy, our method outperformed the other
methods, but the difference was smaller than with the Tail-Accuracy. This is
likely due to the fact that the head and relation are often shorter and less
diverse than the tail, making it relatively easy to answer correctly even with
a conventional method. It should be noted that although tails consist of
multiple words, which makes it difficult to generate them correctly, our
method can achieve a fairly high accuracy.

####  5.4.3 Input Ablation.

From the input ablation study, it can be seen that when only one of the inputs
(image or target knowledge) is used, the performance is worse than when both
are used. The performance degradation is particularly noticeable when no
target knowledge is input. This may be because target knowledge contains more
information about question content control than images. That is, when target
knowledge is input, information about what the answer should be is available
to the model, whereas when only images are input, such information critical to
question content control is not available.

These results highlights our claim that the use of desired knowledge as input
is important for controlling the content of VQG.

####  5.4.4 Output Examples.

We show several examples of generated questions in Figure 7. In general, our
method successfully generates questions that capture the input target
knowledge and the content of the images. The bottom three are examples where
our model failed to output. From these failed examples, we can see that our
model sometimes fails to generate questions when the target object is hidden
or too small. In the case of the bottom-right example, the generated question
is indeed related to the target knowledge, but the question is about the board
itself, not the board material. We believe that further research in methods of
encoding image content and knowledge targets will lead to more precise control
of question generation.

##  6 Conclusion

In this study, we introduce a novel VQG task that uses knowledge as the target
of the question. To this end, we constructed a novel knowledge-aware VQG
dataset called the K-VQG dataset. The K-VQG dataset is the first large-scale
and manually annotated knowledge-aware VQG dataset.

We also developed a benchmark model for the K-VQG task. Our experiments
demonstrated the effectiveness of our method, while showing some room for
improvement.

For future research, our proposed task and dataset have a variety of potential
applications. Given the nature of the task, in which the model acquires new
knowledge by asking questions, we believe that this task can contribute to the
development of learning frameworks, such as human-in-the-loop and learning-by-
asking [18]. We expect that this research will lead to the development of a
proactive learning system that acquires information about the external world
as images and actively learns new knowledge from humans by asking them
questions about the images.

## Acknowledgements

This work was partially supported by JST AIP Acceleration Research JPMJCR20U3,
Moonshot R&D Grant Number JPMJPS2011, JSPS KAKENHI Grant Number JP19H01115,
and JP20H05556 and Basic Research Grant (Super AI) of Institute for AI and
Beyond of the University of Tokyo. We would like to thank Naoyuki Gunji, Qier
Meng for the helpful discussions.

## References

  * [1] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 6077–6086 (2018)
  * [2] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: International Conference on Computer Vision (ICCV) (2015)
  * [3] Bond, F., Foster, R.: Linking and extending an open multilingual Wordnet. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1352–1362. Association for Computational Linguistics, Sofia, Bulgaria (Aug 2013), https://aclanthology.org/P13-1133
  * [4] Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: UNITER: Universal image-text representation learning. In: European conference on computer vision (ECCV). pp. 104–120. Springer (2020)
  * [5] Denkowski, M., Lavie, A.: Meteor universal: Language specific translation evaluation for any target language. In: Proceedings of the Ninth Workshop on Statistical Machine Translation. pp. 376–380. Association for Computational Linguistics, Baltimore, Maryland, USA (Jun 2014). https://doi.org/10.3115/v1/W14-3348, https://aclanthology.org/W14-3348
  * [6] Fan, Z., Wei, Z., Li, P., Lan, Y., Huang, X.: A question type driven framework to diversify visual question generation. In: IJCAI (2018)
  * [7] Gao, D., Wang, R., Shan, S., Chen, X.: Cric: A vqa dataset for compositional reasoning on vision and commonsense. arXiv preprint arXiv:1908.02962 (2019)
  * [8] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * [9] Hwang, J.D., Bhagavatula, C., Bras, R.L., Da, J., Sakaguchi, K., Bosselut, A., Choi, Y.: Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. In: AAAI (2021)
  * [10] Jain, U., Zhang, Z., Schwing, A.G.: Creativity: Generating diverse questions using variational autoencoders. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [11] Krishna, R., Bernstein, M., Fei-Fei, L.: Information maximizing visual question generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [12] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123(1), 32–73 (2017)
  * [13] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7871–7880. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.703, https://aclanthology.org/2020.acl-main.703
  * [14] Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X.: Visual question generation as dual task of visual question answering (June 2018)
  * [15] Liu, F., Xiang, T., Hospedales, T.M., Yang, W., Sun, C.: Ivqa: Inverse visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [16] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
  * [17] Marino, K., Rastegari, M., Farhadi, A., Mottaghi, R.: Ok-vqa: A visual question answering benchmark requiring external knowledge. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [18] Misra, I., Girshick, R., Fergus, R., Hebert, M., Gupta, A., van der Maaten, L.: Learning by asking questions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [19] Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., Vanderwende, L.: Generating natural questions about an image. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1802–1813. Association for Computational Linguistics, Berlin, Germany (Aug 2016). https://doi.org/10.18653/v1/P16-1170, https://aclanthology.org/P16-1170
  * [20] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311–318. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA (Jul 2002). https://doi.org/10.3115/1073083.1073135, https://aclanthology.org/P02-1040
  * [21] Ren, S., He, K., Girshick, R.B., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 1137–1149 (2015)
  * [22] Shah, S., Mishra, A., Yadati, N., Talukdar, P.P.: Kvqa: Knowledge-aware visual question answering. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 8876–8884 (Jul 2019). https://doi.org/10.1609/aaai.v33i01.33018876, https://ojs.aaai.org/index.php/AAAI/article/view/4915
  * [23] Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: An open multilingual graph of general knowledge. In: AAAI (2017)
  * [24] Uehara, K., Tejero-De-Pablos, A., Ushiku, Y., Harada, T.: Visual question generation for class acquisition of unknown objects. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
  * [25] Uppal, S., Madan, A., Bhagat, S., Yu, Y., Shah, R.R.: C3vqg: Category consistent cyclic visual question generation. In: Proceedings of the 2nd ACM International Conference on Multimedia in Asia. MMAsia ’20, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3444685.3446302, https://doi.org/10.1145/3444685.3446302
  * [26] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [27] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2015)
  * [28] Wang, P., Wu, Q., Shen, C., Dick, A., Van Den Hengel, A.: Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence 40(10), 2413–2427 (2017)
  * [29] Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)

##  Appendix A Appendix

###  A.1 Details of Target Knowledge Parser

Following K-VQG model, we used a model that consists of a UNITER-based encoder
[4] and BART-based decoder [13] as our Target Knowledge Parser model. The
encoder takes the visual embeddings $\bm{v}$ and the tokenized question
$\bm{q}$. We used region features obtained from Faster R-CNN [1] as visual
embeddings, as in our VQG model. The question is tokenized into input
sequences using WordPiece tokenizer [29].

Our model is trained to minimizing the negative conditional log-likelihood
loss function can be expressed through the following equation:

$L=-\sum^{|\bm{k}|}_{n=1}\log
P_{\theta}(\bm{k}_{n}\,|\,\bm{k}_{<n},\,\bm{h}_{t})$ (2)

where $\bm{h}_{t}=\textrm{Enc}(\bm{v},\;\bm{q})$, and
$\bm{k}=\\{\bm{w}_{h},\;w_{\mathtt{[SEP]}},\;\bm{w}_{r},\;w_{\mathtt{[SEP]}},\;\bm{w}_{t}\\}$.
is a special token that indicates the separation of each part, and
$\bm{w}_{h},\;\bm{w}_{r},\;\bm{w}_{t},\;w_{\mathtt{[SEP]}}$ denote the tokens
of the head, relation, tail phrases and special token, respectively.

###  A.2 Additional Examples of the K-VQG Dataset

We show additional examples of the K-VQG dataset below.

Figure 8:  Additional examples of the K-VQG dataset (1)  Figure 9:  Additional
examples of the K-VQG dataset (2)  Figure 10:  Additional examples of the
K-VQG dataset (3)