# ChatGPT Asks, BLIP-2 Answers:
Automatic Questioning Towards Enriched Visual Descriptions

Deyao Zhu Jun Chen∗ Kilichbek Haydarov∗
Xiaoqian Shen Wenxuan Zhang Mohamed Elhoseiny
King Abdullah University of Science and Technology
{deyao.zhu, jun.chen, kilichbek.haydarov,
xiaoqian.shen, wenxuan.zhang<EMAIL_ADDRESS>

###### Abstract

Asking insightful questions is crucial for acquiring knowledge and expanding
our understanding of the world. However, the importance of questioning has
been largely overlooked in AI research, where models have been primarily
developed to answer questions. With the recent advancements of large language
models (LLMs) like ChatGPT, we discover their capability to ask high-quality
questions when provided with a suitable prompt. This discovery presents a new
opportunity to develop an automatic questioning system. In this paper, we
introduce ChatCaptioner, a novel automatic-questioning method deployed in
image captioning. Here, ChatGPT is prompted to ask a series of informative
questions about images to BLIP-2, a strong vision question-answering model. By
keeping acquiring new visual information from BLIP-2’s answers, ChatCaptioner
is able to generate more enriched image descriptions. We conduct human-subject
evaluations on common image caption datasets such as COCO, Conceptual Caption,
and WikiArt, and compare ChatCaptioner with BLIP-2 as well as ground truth.
Our results demonstrate that ChatCaptioner’s captions are significantly more
informative, receiving three times as many votes from human evaluators for
providing the most image information. Besides, ChatCaptioner identifies 53%
more objects within the image than BLIP-2 alone measured by WordNet synset
matching. Code is available at https://github.com/Vision-CAIR/ChatCaptioner

††footnotetext: Preprint. ∗Equal contribution.

##  1 Introduction

“The important thing is not to stop questioning.”

Albert Einstein

Figure 1: Example of the dialog between ChatGPT and BLIP-2: BLIP-2 fails to
provide a detailed description in the first message exchange. More details
about the image highlighted in red are obtained through multiple
conversational interactions between the two models. Eventually, the questioner
is able to produce a more detailed caption about the image by focusing on
multiple aspects of the image.

Asking good questions is not only an essential component of effectively
acquiring knowledge, but also plays a pivotal role in enhancing our
intelligence and expanding our understanding of the world. Taking medical
diagnoses as an example, doctors must ask patients targeted questions about
their symptoms to gather relevant information and make accurate diagnoses.
Likewise, in scientific research, asking insightful questions is paramount to
advancing knowledge and discovering new findings that may have far-reaching
implications.

However, the primary focus in recent AI research has been on developing models
that can better answer questions, like InstructGPT [32] in Open-Domain
Question Answering [50, 35, 20] and BLIP-2 [22] in Visual Question Answering
[4, 15, 17]. Despite the significant progress in the question-answering
models, their effectiveness in providing useful information is heavily reliant
on the quality of the questions they receive. In essence, these models depend
on humans to ask insightful questions that can direct their generation of
informative answers. If we have an automatic questioning machine that keeps
asking informative questions, the human questioners can be replaced and the
question-answering models can be guided to provide more valuable knowledge
automatically.

Recent studies [46, 32, 48, 21] have highlighted the impressive zero-shot
learning abilities of Large Language Models (LLMs) that are fine-tuned to
follow instructions. These LLMs can perform new tasks in a zero-shot manner
when presented with well-crafted instruction prompts. We discover that such
LLMs like ChatGPT [31] have the ability to keep asking new and contextually
relevant questions when properly designed prompts are given. With this
capability in place, building an effective automatic questioning machine is
now a feasible task.

Based on our findings, we design an automatic questioning system on ChatGPT
and integrate it into image captioning, where strong vision-language models
like BLIP-2 [22] are available to answer image-related questions. Our method,
named ChatCationer, generates more informative and enriched image captions by
asking relevant questions to incrementally gain more information. In detail,
we design a prompting system that encourages ChatGPT to ask a series of
informative questions that maximize its knowledge of the image, building on
previous questions and answers. Note that ChatGPT is a pure language model and
cannot “see” any visual information. We present the inquired image to BLIP-2
and set it as the question answerer. At the end of the conversation, ChatGPT
is prompted to summarize the discussion into a few sentences as the final
enriched image description. An example of the conversation between ChatGPT and
BLIP-2 and the final caption is shown in Fig.1.

We evaluate ChatCaptioner’s captions on sampled images from COCO [26], WikiArt
[37], and CC [38] datasets based on the human subject evaluation experiment.
Compared to BLIP-2’s state-of-the-art direct captioning performance,
ChatCaptioner receives three times as many votes from human evaluators for
providing richer image information. Besides, ChatCaptioner identifies 53% more
objects than BLIP-2 alone within the image. Results verify the benefit of good
questions to acquire more knowledge from existing AI models and the
effectiveness of modern LLMs to serve as zero-shot automatic questioners.

Figure 2: ChatCaptioner contains a questioner ChatGPT and an answerer BLIP-2.
We start with a task instruction to set the context and outline the problem
for ChatGPT. Once the answer from BLIP-2 is received, we prompt the chat log
and question instruction to ChatGPT to continue asking information-acquiring
questions. Finally, we provide a summarization instruction to ChatGPT to
conclude the conversation as the image captions. Enriched details are
highlighted in red.

##  2 Related Works

#### Learning to Ask Questions

Question generation [30, 16] is the task of generating a question from a given
passage and an answer. Recent methods like [19, 49, 27, 14, 44] have explored
different neural network architectures and training strategies for better
performance. However, in cases when we do not have the answer and need to ask
questions for the answers, such methods are not applicable. Visual Question
Generation [29, 51] is a task aimed at generating natural and engaging
questions for a given image. Several works like [34, 33, 24, 18, 43, 39] have
been proposed to solve this task. They focus on generating independent
questions only and do not have the ability to keep asking new questions based
on the previous questions. Our work differs from previous studies
significantly. First, we focus on acquiring more knowledge via the generated
questions, instead of just generating them. Secondly, our method can keep
asking new and relevant questions based on previously questions. Third, our
approach leverages modern large language models and requires zero training for
questioning.

#### Large Language Model and Prompting

Recent research [7, 21, 48, 47, 46, 11, 32] has revealed the abilities of
Large Language Models (LLMs) like GPT-3 [7] or PaLM [9] to solve versatile
tasks specified by prompting. For example, GPT-3 [7] shows the capability to
learn new tasks by providing a few task examples provided in the prompt, named
in-context learning. Moreover, Chain-of-Thought methods [21, 48] demonstrate
that explicitly asking LLM to solve tasks step-by-step in the prompt improves
the performance significantly. Additionally, FLAN [46, 11] demonstrates that
LLMs with instruction tuning can accomplish new tasks in a zero-shot manner.
Further studies, including InstructGPT [32] and ChatGPT [31], show that the
performance of LLMs can be enhanced even further by using reinforcement
learning from human feedback [10, 40]. In our work, we leverage the
instruction-following ability of LLMs and design prompts that enable ChatGPT
to keep asking new questions about images.

#### Image Captioning and Visual Question Answering

Recent research in vision and language pertaining [8, 41, 3, 45, 23, 22] has
advanced the performance for image captioning and visual question answering
(VQA) by a large margin. For example, VisualGPT [8] shows the benefits of
initialization with pretrained language models for more data-efficient
training. Frozen [41] extend it by finetuning a vision encoder and aligning it
with a frozen LLM. BEiT-3 [45] and BLIP [23] pretrain models using unified
transformer architecture. Flamingo [3] proposes a cross-attention design to
align vision and language. BLIP-2 [22] introduces a lightweight Q-Former that
converts visual features into tokens that can be directly understood by a
frozen LLM, and achieves impressive results on both image captioning and VQA
tasks. In our work, our automatic questioning mechanism leverages the VQA
capability of BLIP-2 to extract additional image information and enrich the
image captions beyond the original BLIP-2 captions.

##  3 ChatCaptioner

In ChatCaptioner, we design an automatic questioning mechanism based on
ChatGPT’s zero-shot instruction-following ability to keep asking informative
questions about images. BLIP-2, the vision-language model, then provides new
image information according to the asked questions. Finally, ChatGPT is
prompted to summarize the chat history and generate the final image captions
with rich details. An overview of our method is demonstrated in Fig.2.

###  3.1 Automatic Questioning

To activate the questioning ability of ChatGPT, we design a prompting system
that enables ChatGPT to generate questions based on previous chat logs. Our
prompting system for ChatGPT contains three components: a task instruction for
explaining the task $\rho_{taskQ}$, a chat log to store previous questions and
answers $\rho_{chat}$, a question instruction for generating high-quality
questions $\rho_{q}$. Each question is generated given the context
$\rho_{taskQ}+\rho_{chat}+\rho_{q}$. In addition, we design a question-
trimming mechanism for the automatic post-processing of the generated
questions.

#### ChatGPT Task Instruction $\rho_{taskQ}$

The task instruction $\rho_{taskQ}$ sets the context and outlines the task
that ChatGPT is required to perform. $\rho_{taskQ}$ directs ChatGPT to
generate questions that aim to extract as much information as possible about
an image. $\rho_{taskQ}$ is designed as follows:

I have an image. Ask me questions about the content of this image. Carefully
asking me informative questions to maximize your information about this image
content. Each time ask one question only without giving an answer. Avoid
asking yes/no questions. I’ll put my answer beginning with “Answer:”.

The prompt “Each time ask one question only without giving an answer” in
$\rho_{taskQ}$ is designed to instruct ChatGPT to generate only one question
per round. Without this prompt, ChatGPT may generate a set of questions at
once, rather than generating them one by one in response to each answer.

Figure 3: Qualitative examples of the chat log and the captions from
ChatCaptioner in various images from COCO [26], WikiArt [37], and CC [38].
Compared to the initial description from BLIP-2, questions from ChatGPT
extract additional image information and lead to enriched final captions.

#### Chat Log $\rho_{chat}$

After the task instruction $\rho_{taskQ}$, we include a chat log $\rho_{chat}$
containing all the previous questions and answers. $\rho_{chat}$ follows a
template for each Question-Answer round, which is as follows:

Question: $\langle question\rangle$ Answer: $\langle answer\rangle$

Note that we hard-code the first question as “Describe the image in detail” to
start the conversation. Based on the first response of BLIP-2, which provides
a brief initial description of the image, ChatGPT is prompted to ask follow-up
questions to extract more information about the image.

#### Question Instruction $\rho_{q}$

To guide ChatGPT in generating new questions, we provide a question
instruction $\rho_{q}$ before each question generation. $\rho_{q}$, located
after the chat log, cues ChatGPT to generate a new question and aims to ensure
that the questions are of high quality. It’s designed as follows:

Next Question. Avoid asking yes/no questions. Question:

The prompt “Next Question” in $\rho_{q}$ is critical to ensure that ChatGPT
continues to ask questions. Without it, ChatGPT may produce undesired
sentences after a few Question-Answer rounds. Additionally, we notice that
ChatGPT prefers asking yes/no questions which are usually not as informative
as other questions. We therefore add the prompt “Avoid asking yes/no
questions” to reduce the generation of yes/no questions.

#### Question Trimming

Despite our explicit instruction to not answer the question itself, we observe
that sometimes ChatGPT fabricates an answer after asking the question.
Fortunately, we find that these fabricated answers always begin with the text
“Answer:”, following the template specified in the prompt. Therefore, we
automatically remove these fabricated answers by discarding the generated text
starting from “Answer:”.

###  3.2 Question Answering

Similar to ChatGPT, our BLIP-2 prompting mechanism consists of three
components: a task instruction $\rho_{taskA}$, the chat log $\rho_{chat}$ same
as the ChatGPT one, and an answer instruction $\rho_{a}$. Each answer
generation is prompted by $\rho_{taskA}+\rho_{chat}+\rho_{a}$. Also, we have
an answer-trimming mechanism for post-processing.

#### BLIP-2 Task Instruction $\rho_{taskA}$

We design the BLIP-2 task instruction $\rho_{taskA}$ to alleviate the issue of
hallucinating non-existent information in the image. $\rho_{taskA}$ includes
an uncertainty prompt “If you are not sure about the answer, say you don’t
know honestly” that encourages BLIP-2’s honest admission of lack of knowledge.
The instruction is as follows:

Answer given questions. If you are not sure about the answer, say you don’t
know honestly. Don’t imagine any contents that are not in the image.

#### Answer Instruction $\rho_{a}$

After the chat log $\rho_{chat}$, we provide a straightforward answer
instruction to guide BLIP-2’s answering process. The instruction is structured
as follows:

Question: $\langle question\rangle$ Answer:

#### Answer Trimming

Similar to ChatGPT, BLIP-2 occasionally generates a question after providing
an answer. As the LLM backend of BLIP-2, the FLAN-T5 model [11], has a much
weaker questioning ability than ChatGPT shown later in Sec.4.3, we
automatically filter out these questions by discarding any texts starting with
“Question:”.

###  3.3 Context Summarizing

To obtain a concise summary of the conversation between ChatGPT and BLIP-2 as
the final image caption, we use a summarization instruction after the
conversation. This instruction, located after the chat log, prompts ChatGPT to
generate a summary using the following structure:

Now summarize the information you get in a few sentences. Ignore the questions
with answers no or not sure. Don’t add information. Don’t miss information.
Summary:

Figure 4: An example of the limitation of traditional image caption metrics.
Although ChatCaptioner extracts more image details highlighted in red compared
to BLIP-2, it receives much lower scores under traditional metrics.

##  4 Experiments

In this section, we explore ChatCaptioner from various perspectives through a
series of experiments, including an assessment of the informativeness and
accuracy of its captions, as well as an analysis of the quality of its
generated questions. Qualitative results including chat logs and final
captions on various images from different datasets are shown in Fig.3.

Details of Model Deployment. For our experiments, we use the ChatGPT model
“gpt-3.5-turbo” available on the OpenAI API [1]. This model is the most
powerful GPT-3.5 model accessible through the API during our project. For
BLIP-2, we use the biggest version containing a FLAN-T5 [11] language model
with 11 billion parameters and a ViT-G/14 model from EVA-CLIP [13]. In all
experiments, BLIP-2 answers 10 questions per image, with the first question
being hard-coded as “Describe the image in detail.”. The remaining 9 questions
are from ChatGPT, unless otherwise specified.

Limitation of Traditional Metrics. The conventional image captioning metrics
such as[42], ROUGE [25], and METEOR [5] are frequently employed to measure the
quality of captions. However, the usage of these metrics to evaluate
ChatCaptioner can be limiting, because these metrics are designed to measure
the similarity between the tested caption and reference captions, assuming
that the reference captions are perfect image descriptions. Since
ChatCaptioner generates captions that contain a greater level of detail than
the reference captions, the metrics may yield low similarity scores, leading
to inferior overall performance scores compared to other image captioning
approaches like BLIP-2. This issue is depicted in Fig. 4. Thus, in our
experiments, we primarily rely on human assessments to comprehensively analyze
the performance of ChatCaptioner from various perspectives. The design of all
the human evaluation interfaces is presented in Appendix.

###  4.1 Information Analysis

#### Does ChatCaptioner extract more information from the image?

We design an experiment to evaluate whether ChatCaptioner is able to generate
captions with more information about the image than BLIP-2 alone and ground
truth captions. We randomly selected 100 photos from the COCO [26] validation
set, 100 artworks from WikiArt [37] dataset with ground truth captions from
ArtEmis [2], and 100 internet images from the CC [38] validation dataset.
Human evaluators on Amazon Mechanical Turk are presented with an image and
four captions - one from our method, one from BLIP-2, one ground truth
caption, and one fake caption for quality control. Evaluators are asked to
pick the caption that offers the richest information about the image. Results
are demonstrated in Tab.1. On average, ChatCaptioner receives three to four
times as many votes as pure BLIP-2’s captions and ground truth captions,
demonstrating the effectiveness of using an Automatic Question-Asking machine
to enhance image information extraction from a VQA model.

Table 1: Human votes on the captions containing the most image information.

Methods | COCO | WikiArt | CC | Avg.
---|---|---|---|---
GT | 26% | 14% | 8.5% | 16.2%
BLIP-2 | 21% | 12.5% | 23% | 18.8%
Ours | 53% | 73.5% | 68.5% | 65%

How many objects in images can ChatCaptioner discover? Here, we randomly
sampled 200 images from the Pascal VOC [12] dataset and considered all class
labels included in the segmentation masks as the ground truth objects. We then
assessed how many of these objects are included in the captions. We utilize
WordNet from the Natural Language Toolkit (NLTK) [6] to find words with
similar semantic meanings. Specifically, we match two words if the Wu-Palmer
Similarity of their synsets is greater than 0.9 or if one word’s synset is
included in the other’s closure. Tab.2 presents the experimental results,
where 1154 objects are identified in the 200 sampled images. BLIP-2 covers
only 383 of them, while with the help of automatic questioning, ChatCaptioner
increases the coverage by 53% to 586, suggesting that the automatically
generated questions help BLIP-2 find more objects in the images.

Table 2: Numbers of objects discovered by captions.

Methods | Covered/All | Ratio | Improved
---|---|---|---
BLIP-2 | 383/1154 | 33.2% | -
Ours |  586/1154 | 50.8% | 53.0%

###  4.2 Correctness Analysis

#### How accurate are the captions from ChatCaptioner?

To evaluate the correctness of ChatCaptioner’s captions, we conducted a human
evaluation where evaluators were presented with an image and a generated
caption, as well as all questions and answers between ChatGPT and BLIP-2. The
evaluators were asked to verify the correctness of the caption with respect to
the image, select any incorrect answers from BLIP-2, and judge whether the
incorrectness of the caption can be attributed to the wrong answers. The
experiments were performed on samples from COCO [26], WikiArt [37], and CC
[38] datasets, similar to previous experiments, and each image was evaluated
by 4 different evaluators. Results are presented in Tab.3. Our findings reveal
that approximately 80% of the generated captions are deemed correct. Moreover,
BLIP-2 is able to provide correct answers to around 67% of the questions asked
by ChatGPT. Among the incorrect captions, 94% of them are caused by BLIP-2’s
wrong answers, suggesting that BLIP-2 is the primary source of incorrectness.
This implies that using a more powerful visual question-answering (VQA) model
may help to enhance the overall performance of the system in the future.

Table 3: Correctness Analysis of ChatCaptioner. Overall, BLIP-2 can correctly
answer about 66.7% of ChatGPT’s questions. 81% of the final captions are
deemed correct by human evaluators. Besides, 94% of the wrong captions are
caused by BLIP-2’s wrong answers.

| COCO | WikiArt | CC | Avg.
---|---|---|---|---
Answer Correct Rate | 64% | 73% | 63% | 66.7%
Caption Correct Rate | 77% | 78% | 88% | 81%
Issues From BLIP-2 | 100% | 82% | 100% | 94%

Does BLIP-2 know it doesn’t know? BLIP-2 usually makes up answers if the
question cannot be answered based on the given image. In other words, BLIP-2
doesn’t know that it doesn’t know this information. To mitigate this issue, we
incorporate an uncertainty prompt “If you are not sure about the answer, say
you don’t know honestly.” in our BLIP-2 task instruction $\rho_{taskA}$. This
prompt encourages the model to say it doesn’t know when it is not confident in
its response. Two examples are demonstrated in Fig.5. In the first example,
BLIP-2 is presented with an image that only shows two hands and is asked to
determine the gender and age of the person. With the uncertainty prompt,
BLIP-2 changes its answer from guessing a young male to honestly saying “Don’t
know”. In the second example, BLIP-2 initially thought that the unrecognizable
store in the photo was a gas station, but with the addition of the uncertainty
prompt, it changes the answer to “Not sure”. More examples can be found in
Fig.3 and Appendix.

Figure 5: Examples of BLIP-2’s answers with and without the uncertainty
prompt. The uncertainty prompt helps BLIP-2 avoid making up an answer when it
encounters questions that cannot be answered based on the image.  Table 4:
Analysis on questions that BLIP-2 is unsure about. 60% deemed unanswerable by
humans. 20% cannot be correctly answered by BLIP-2. Overall, BLIP-2 makes
mistakes on 80% of these questions.

| Total Uncertain | Unanswerable | Answerable | Avoided Bad
---|---|---|---|---
| Questions | Questions | But Wrong | Answers
Num. | 147 | 88 | 30 | 118
Ratio | - | 60% | 20% | 80%

How effective is the uncertainty prompt? To investigate whether the questions
that BLIP-2 is unsure about can be answered by humans, we randomly selected
200 images from the CC [38] dataset and collected 1,800 questions based on
these images. We then identify 147 questions that BLIP-2 is uncertain about,
present these questions to human evaluators, and ask them to answer based on
the image content. Results presented in Tab.4 demonstrate that approximately
60% of these questions are deemed unanswerable based on the image content. For
the remaining answerable questions, BLIP-2 cannot correctly answer 30 of them.
In total, without the uncertainty prompt, BLIP-2 will generate 118 incorrect
answers out of 147 uncertain questions, resulting in an error rate of
approximately 80%. In addition, out of the original 1800 questions, BLIP-2 has
674 wrong answers. Taking the 147 potential wrong answers avoided by the
uncertainty prompt into account, the uncertainty prompt reduces about 15% of
the wrong answers.

###  4.3 Question Analysis

Figure 6: Visualization of question embeddings and question examples printed
at the corresponding positions. Our method can ask diverse questions focusing
on various perspectives of the image.

How diverse are the automatic generated questions? To investigate whether
ChatGPT-based Automatic Question Asking mechanism can produce diverse
questions, we analyze 1800 generated questions on 200 random image samples
from the CC [38] dataset. After filtering out the repeated ones, we have in
total of 1419 unique questions.

To visualize the diversity of the generated questions, we compute the sentence
embedding of each question using Sentence-BERT [36] and reduce their
dimensions to two for visualization using UMAP [28]. The resulting
visualization is presented in Fig.6 with question examples plotted at
corresponding positions.

Our analysis shows that the questions generated by ChatCaptioner are diverse
and cover various perspectives, including high-level information like “What is
the overall mood of this image?”, object properties like “What material are
the clothes in the closet made of?”, numbers of objects like “How many
chandeliers are in the lobby?”, facial expression like “What is the expression
on the girl’s face?”, environmental information like “What is the sky in the
image like?”, and so on.

Overall, our findings suggest that our ChatGPT-based automatic questioning
mechanism is able to generate diverse questions, explore various aspects of
the image, and acquire more informative responses.

Figure 7: Examples of questions from various language models based on initial
image descriptions. ChatGPT and InstructGPT demonstrate their ability to ask
new and informative questions automatically. In contrast, FLAN-T5 11B and OPT
6.7B tend to repeat old questions, indicating weaker questioning abilities.
The original GPT-3 struggles the most and fails to ask any related questions.

Can we use other LLMs as questioners? In addition to ChatGPT, we evaluate the
automatic questioning capabilities of several other large language models
(LLMs), including InstructGPT [32], the original GPT-3 [7] without instruction
finetuning, FLAN-T5 11B [11] (11B parameters), and OPT 6.7B [52] (6.7B
parameters). Fig.7 showcases the different question-asking abilities given the
initial description from BLIP-2. We observe that InstructGPT shows a similar
question-asking ability as ChatGPT and is able to generate meaningful and
diverse questions. While FLAN-T5 11B and OPT 6.7B also exhibits the ability to
ask questions, it repeats a single question several times. Interestingly, the
original GPT-3 model shows the worst question generation ability and fails to
generate any meaningful questions related to the image.

We further analyze their 1800 generated questions on 200 CC [38] samples, at a
rate of 9 questions per image. We skip the original GPT-3 model here as it is
not able to generate any meaningful questions in our early exploration. The
tested LLMs’ questioning abilities are evaluated in two ways. Firstly, we
measure the average number of unique questions per image, which indicates
whether the model can keep asking new questions in a single dialogue.
Secondly, we count the unique questions over all the 1800 asked questions to
see if the questioner could customize the questions according to the given
contexts or just ask fixed predefined questions.

Our findings, as shown in Tab.5, reveal that ChatGPT and InstructGPT almost
never repeat their question in a single nine-question dialogue and generate a
total of around 1400 unique questions out of 1800, suggesting that they are
able to ask new questions according to the contexts. In contrast, FLAN-T5 11B
and OPT 6.7B have a tendency to repeat old questions, averaging about only 2
unique questions per image and generating less than 170 unique questions in
total. Our study suggests that to develop a questioning machine that can
automatically generate novel and customized questions, it may be necessary to
utilize LLMs with at least dozens of billions of parameters that are
specifically fine-tuned for improved instruction following ability.

Table 5: Number of unique questions per dialogue and in total. InstructGPT and
ChatGPT excel at generating diverse questions, rarely repeating questions
within a dialogue, and outperforming OPT 6.7B and FLAN-T5 11B.

Unique Q/Total Q | OPT 6.7B | FLAN-T5 | InstructGPT | ChatGPT
---|---|---|---|---
Per Dialogue | 1.75/9 | 2.03/9 | 9/9 | 8.98/9
All Questions | 166/1800 | 169/1800 | 1400/1800 | 1419/1800

###  4.4 Limitation

The caption correctness of ChatCaptioner relies on the answers from BLIP-2.
Although we design the uncertainty prompt to reduce the number of wrong
answers from BLIP-2, a small portion of the answers are still incorrect.
Combining automatic questioning with better vision-language models in the
future may enhance its visual description ability. As ChatCaptioner is based
on LLMs, it also inherits the risks of LLM and might sometime generate
offensive or socially biased conversations and captions. Finetuning the system
with a filtered dataset or human feedback may alleviate this issue.

##  5 Conclusion

In this work, we discover that advanced large language models possess the
ability to pose insightful and diverse questions when provided with well-
crafted prompts. Based on our findings, we develop an automatic questioning
system named ChatCaptioner for the task of image captioning. By prompting
ChatGPT to keep asking questions that expand its understanding of an image,
ChatCaptioner guides BLIP-2 to provide comprehensive image information,
resulting in image captions that are significantly more detailed and enriched.
ChatCaptioner demonstrates the power of automatic questioning systems to
effectively extract desired information. Through our work, we aim to draw
attention to the potential of automatic questioning systems in AI and inspire
further research in various domains.

## References

  * [1] Openai api: Gpt-3.5.  https://platform.openai.com/docs/models/gpt-3-5, 2023.  Accessed: March 7, 2023.
  * [2] Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas.  Artemis: Affective language for visual art.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11569–11579, 2021.
  * [3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In Advances in Neural Information Processing Systems, 2022.
  * [4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [5] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * [6] Steven Bird, Ewan Klein, and Edward Loper.  Natural language processing with Python: analyzing text with the natural language toolkit.  ” O’Reilly Media, Inc.”, 2009.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [8] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny.  Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040, 2022.
  * [9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  Advances in neural information processing systems, 30, 2017.
  * [11] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [12] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88(2):303–338, June 2010\.
  * [13] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  arXiv preprint arXiv:2211.07636, 2022.
  * [14] Bilal Ghanem, Lauren Lutz Coleman, Julia Rivard Dexter, Spencer McIntosh von der Ohe, and Alona Fyshe.  Question generation for reading comprehension assessment by modeling how and what to ask.  arXiv preprint arXiv:2204.02908, 2022.
  * [15] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [16] Michael Heilman and Noah A Smith.  Good question! statistical ranking for question generation.  In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617, 2010.
  * [17] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * [18] Unnat Jain, Ziyu Zhang, and Alexander G Schwing.  Creativity: Generating diverse questions using variational autoencoders.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6485–6494, 2017.
  * [19] Xin Jia, Wenjie Zhou, Xu Sun, and Yunfang Wu.  How to ask good questions? try to leverage paraphrases.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6130–6140, 2020.
  * [20] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  arXiv preprint arXiv:1705.03551, 2017.
  * [21] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
  * [22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [24] Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou.  Visual question generation as dual task of visual question answering.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [25] Chin-Yew Lin and Eduard Hovy.  Manual and automatic evaluation of summaries.  In Proceedings of the ACL-02 workshop on automatic summarization, pages 45–51, 2002.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [27] Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu.  Learning to generate questions by learning what not to generate.  In The world wide web conference, pages 1106–1118, 2019.
  * [28] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger.  Umap: Uniform manifold approximation and projection.  Journal of Open Source Software, 3(29), 2018.
  * [29] Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende.  Generating natural questions about an image.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1802–1813, 2016.
  * [30] Jack Mostow and Wei Chen.  Generating instruction automatically for the reading strategy of self-questioning.  In AIED, pages 465–472, 2009.
  * [31] OpenAI.  Introducing chatgpt.  https://openai.com/blog/chatgpt, 2022.
  * [32] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al.  Training language models to follow instructions with human feedback.  In Advances in Neural Information Processing Systems, 2022.
  * [33] Badri Patro, Vinod Kurmi, Sandeep Kumar, and Vinay Namboodiri.  Deep bayesian network for visual question generation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
  * [34] Badri Narayana Patro, Sandeep Kumar, Vinod Kumar Kurmi, and Vinay Namboodiri.  Multimodal differential network for visual question generation.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4002–4012, Brussels, Belgium, Oct.-Nov. 2018\. Association for Computational Linguistics.
  * [35] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  arXiv preprint arXiv:1606.05250, 2016.
  * [36] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [37] Babak Saleh and Ahmed Elgammal.  Large-scale classification of fine-art paintings: Learning the right metric on the right feature.  arXiv preprint arXiv:1505.00855, 2015.
  * [38] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [39] Tingke Shen, Amlan Kar, and Sanja Fidler.  Learning to caption images through a lifetime by asking questions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10393–10402, 2019.
  * [40] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  * [41] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  Advances in Neural Information Processing Systems, 34:200–212, 2021\.
  * [42] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [43] Nihir Vedd, Zixu Wang, Marek Rei, Yishu Miao, and Lucia Specia.  Guiding visual question generation.  arXiv preprint arXiv:2110.08226, 2021.
  * [44] Qifan Wang, Li Yang, Xiaojun Quan, Fuli Feng, Dongfang Liu, Zenglin Xu, Sinong Wang, and Hao Ma.  Learning to generate question by asking question: A primal-dual approach with uncommon word generation.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 46–61, 2022.
  * [45] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442, 2022.
  * [46] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In International Conference on Learning Representations, 2022.
  * [47] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  Transactions on Machine Learning Research, 2022.
  * [48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In Advances in Neural Information Processing Systems, 2020.
  * [49] Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-gen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation.  arXiv preprint arXiv:2001.11314, 2020.
  * [50] Yi Yang, Wen-tau Yih, and Christopher Meek.  Wikiqa: A challenge dataset for open-domain question answering.  In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013–2018, 2015.
  * [51] Shijie Zhang, Lizhen Qu, Shaodi You, Zhenglu Yang, and Jiawan Zhang.  Automatic generation of grounded visual questions.  arXiv preprint arXiv:1612.06530, 2016.
  * [52] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.

##  6 Appendix

###  6.1 Cost

Our method is based on the ChatGPT model, specifically the gpt-3.5-turbo
version which we access through OpenAI’s API. At the time of our project, the
cost for using 1000 tokens in gpt-3.5-turbo was 0.002 US Dollars. On average,
we spent approximately 2500 tokens for each image for ten Question-Answer
rounds, which translates to a cost of approximately 0.005 US Dollars per
image.

###  6.2 Yes/No Question Ablation

Usually, yes/no questions contain relatively less information. To reduce the
generation of yes/no questions from ChatGPT, we explicitly add a prompt “Avoid
asking yes/no questions” in the task instruction $\rho_{taskQ}$ and the
question instruction $\rho_{q}$. Our ablation study in Tab.6 shows that this
prompt reduces the generation of yes/no questions from 33% of the cases to 2%
in 1800 questions on 200 random CC [38] samples, verifying its effectiveness.

Table 6: Effectiveness of the yes/no prompt.

| Total Question | Yes/No Question w/o Prompt | Yes/No Question with Prompt
---|---|---|---
Num. | 1800 | 595 | 38
Ratio | - | 33% | 2%

###  6.3 Human Evaluation Interface

(a) Human evaluation interface of the information experiments in Tab.1.

(b) Human evaluation interface of the correctness experiments in Tab.3.

(c) Human evaluation interface of the question-answerable experiments in
Tab.4.

Figure 8: Human evaluation interfaces

###  6.4 Random Examples of Generated Questions in Conceptual Captioning
Dataset

What is the material of the pier in the image?
What is the position of the sign that says ”No Swimming Allowed” on the dock?
What is the material of the valve?
What is the color of the plate on which the cake is placed?
What is the expression on the man’s face?
What is the boy’s expression while playing with the sailboat?
What is the angle of the camera in the image?
What type of flower is it?
What is the color of the woman’s glasses?
What is written on the quote on the mountain?
What is the design on the cake?
What is the woman’s hair color in the image?
Are the man and woman standing or sitting in the image?
What is the location of the scene depicted in the image?
What is the boy’s expression?
What is the material of the pink running shoes?
What is the expression on the man’s face?
What type of vegetation surrounds the pond in the image?
What is the size of the fountain in the image?
What is the name of the mountain range seen in the background of the image?
What is the name of the park?
What is the design of the woman’s dress?
What is the color of the chainsaw?
What is the ethnicity of the two men in the image?
What is the woman’s pose in the photo?
What modifications, if any, have been made to the car in the image?
What kind of donuts are in the box?
What is the woman’s age range in the image?
What is the weather like in the image?
What is the man’s posture like in the image?
What kind of lighting is in the room?
What is the woman’s hair color in the image?
What is the woman wearing in the image?
What is the woman’s pose in the image?
What is the type of the lightning bolt? (e.g. cloud-to-ground, cloud-to-cloud)
What is the context or setting of the image?
What type of event is taking place where the man is performing?
What is the shape of the base or stand of the ice sculpture?
What is the size of the tent?
What is the boy wearing?
What is the man doing?
What is the material of the sink in the image?
What is the age range of the people in the image?
What is the style of the hat?
What is the background of the image?
What is the surrounding environment like in the image?
What is the woman wearing?
What is the shape of the backyard?
What is the material of the backpack the man is carrying?
What is the texture of the granny square?
What is the style of the bridge?
What is the size of the sun in the image?

###  6.5 More Qualitative Results

Figure 9: Random samples from CC [38]. Figure 10: Random samples from WikiArt
[37]. Figure 11: Random samples from COCO [26].