# M3IT: A Large-Scale Dataset towards
Multi-Modal Multilingual Instruction Tuning

Lei Li$\dagger$, Yuwei Yin$\dagger$, Shicheng Li§, Liang Chen§, Peiyi Wang§,
Shuhuai Ren§, Mukai Li‡
Yazheng Yang$\dagger$, Jingjing Xu‡, Xu Sun§, Lingpeng Kong$\dagger$, Qi
Liu$\dagger$
$\dagger$ The University of Hong Kong
§ National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
‡ Shanghai AI Lab
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>{lpk<EMAIL_ADDRESS>

###### Abstract

Instruction tuning has significantly advanced large language models (LLMs)
such as ChatGPT, enabling them to align with human instructions across diverse
tasks. However, progress in open vision-language models (VLMs) has been
limited due to the scarcity of high-quality instruction datasets. To tackle
this challenge and promote research in the vision-language field, we introduce
the Multi-Modal, Multilingual Instruction Tuning (M3IT) dataset, designed to
optimize VLM alignment with human instructions. Our M3IT dataset comprises 40
carefully curated datasets, including 2.4 million instances and 400 manually
written task instructions, reformatted into a vision-to-text structure. Key
tasks are translated into 80 languages with an advanced translation system,
ensuring broader accessibility. M3IT surpasses previous datasets regarding
task coverage, instruction number and instance scale. Moreover, we develop
Ying-VLM, a VLM model trained on our M3IT dataset, showcasing its potential to
answer complex questions requiring world knowledge, generalize to unseen video
tasks, and comprehend unseen instructions in Chinese. We have open-sourced the
dataset to encourage further research.111Our dataset is available at
https://huggingface.co/datasets/MMInstruction/M3IT

##  1 Introduction

There has been a continuously increasing trend to develop intelligent
assistants that can follow human instructions [3, 36, 37]. In the natural
language processing (NLP) field, instruction tuning [35, 53] is a success
paradigm that leverages large-scale well-formatted instances to align large
language models (LLMs) to human instructions. By finetuning on instances with
specific task descriptions, LLMs learn to follow the instruction to perform
various tasks, and demonstrate strong generalization ability on unseen tasks
[29]. Expanding beyond NLP, a general-purpose intelligent agent must encompass
various modalities, such as vision, prompting recent efforts to investigate
instruction tuning in vision-language domains [63, 28, 7]. To develop powerful
vision-language models (VLMs), it is essential to have a well-constructed
dataset that encompasses diverse vision-language tasks and aligns with human
instructions. However, the instructional data supporting existing VLMs is
either not publicly available (e.g., GPT-4) or offers limited task and
language coverage (e.g., only tasks in English are considered). This scarcity
of comprehensive datasets has impeded the progress of open vision-language
models, highlighting the importance of multi-modal instruction tuning and the
need for high-quality datasets.

In this paper, we aim to advance instruction tuning research in the multi-
modal domain by introducing an open dataset M3IT, a Multi-Modal Multilingual
Instruction Tuning dataset, as an essential step towards building a versatile
general-purpose assistant. We build this dataset by converting existing
datasets into a unified vision-to-text schema with four stages: (1) manual
instruction writing, (2) dataset pre-processing, (3) careful quality check and
(4) dataset translation for key tasks. Our dataset encompasses a wide range of
tasks, including classic image-text tasks such as image classification, visual
question answering, and image captioning. Video-related tasks, such as video
question-answering, are also incorporated to ensure comprehensive coverage
across multiple modalities. We further integrate Chinese vision-language
datasets with corresponding Chinese instructions. The resulting dataset
compiles 40 diverse tasks and 400 instructions. Finally, key vision-language
tasks are translated into 80 languages with a strong translation system, to
support multilingual studies.

To evaluate the effectiveness of the proposed dataset, we develop a vision-
language model, Ying-VLM, by integrating a strong vision encoder, BLIP-2 [23]
with a large language model, Ziya-13B [61], derived from LLaMA [49]. Building
on the successful approach of incorporating visual tokens as textual prompts
in LLMs [7, 63, 28], we employ a two-stage training process: (1) the initial
stage aligns vision features with text embeddings through image captioning on
LAION400M [41], and (2) the second stage enhances the model by conducting
instruction tuning on selected tasks of our dataset. Experimental results
reveal that Ying-VLM surpasses strong baseline models in knowledgeable VQA
tasks and exhibits improved generalization performance to unseen video and
cross-lingual tasks. Further analysis indicates that the improved performance
corresponds to increased tasks for instruction tuning, while the diversity of
instructions also affects outcomes.

This paper presents two key contributions: (1) We introduce the open-source,
large-scale Multi-modal, multilingual Instruction Tuning (M3IT) dataset,
designed to enable the development of general-purpose multi-modal agents. (2)
We develop Ying-VLM, a visual assistant that excels in knowledgeable VQA
tasks, demonstrates strong generalization to unseen video QA and Chinese
multi-modal tasks, and offers valuable insights for future research.

##  2 Related Work

Table 1: Summary of multi-modal instruction tuning datasets.

Dataset # Tasks Multi-Lingual # of Instances Avg. # of Manual Instructions /
Task Open-Sourced MiniGPT4 N / A ✗ 5K N / A ✓ LLaVA 3 ✗ 1.15M N / A ✓
MultiModalGPT 3 ✗ 6K 5 ✗ MultiInstruct 26 ✗ $\sim$ 235K 5 ✗ InstructBLIP 28 ✗
$\sim$ 1.6M 9.7 ✗ M3IT (Ours) 40 ✓ 2.4M 10 ✓

Our work draws inspiration from recent language instruction tuning benchmarks
[53, 35], which have been proven effective for improving language models to
obtain cross-task generalization ability [29, 52]. In this paper, we focus on
exploring the instruction tuning paradigm from LLMs to multi-modal agents.
Unlike text-only tasks, vision-language tasks generally have more diverse
formats, which poses new challenges toward vision-language instruction tuning
benchmarks.

To develop a general-purpose vision-language model, it is crucial to create
high-quality multi-modal instruction tuning datasets encompassing diverse
tasks, languages, and instructions. Several studies have investigated multi-
modal instruction tuning for VLMs. LLaVA [28] and MiniGPT-4 [63] generate
visual content-related dialog by incorporating image caption data into
GPT-4/ChatGPT models. MultiInstruct [56] reformats a series of visual
classification tasks into an instruction-tuning format, while InstructBLIP [7]
adapts 28 existing image-to-text tasks. However, these datasets do not provide
an ideal multi-modal instruction tuning dataset due to their limited (1)
coverage of various task types in multi-modal fields, (2) diversity and
quality of instances, and (3) inclusion of multiple languages for wide
linguistic diversity. In this paper, we construct an improved multi-modal
instruction tuning dataset by expanding task coverage to 40 datasets,
supplementing instances with 10 manually written task instructions, and
including tasks in different languages. Table 1 compares the characteristics
of existing multi-modal instruction tuning datasets and M3IT.

##  3 M3IT: A Multi-Modal Multilingual Instruction Tuning Dataset

In this section, we introduce our proposed M3IT dataset by first elaborating
the dataset coverage (§ 3.1), followed by the details of the annotation
process(§ 3.2). Finally, we present the dataset format and provide the
statistics of the crafted datasets instructions(§ 3.3).

###  3.1 Task Coverage

Our dataset compiles diverse tasks of classical vision-language tasks,
including captioning, visual question answering (VQA), visual conditioned
generation, reasoning and classification.

Captioning This task aims to produce descriptions of the given images
according to different needs. We include MS COCO [27] (the Karpathy split) for
generic image descriptions. TextCaps [44] requires models to capture the text
presented in the image and generate captions accordingly. Image-Paragraph-
Captioning [21] focuses on generating detailed descriptions for images.

Reasoning This task evaluates specific reasoning capabilities. We incorporate
CLEVR [19] and NLVR [46] for spatial reasoning, Visual Commonsense Reasoning
(VCR) [60] for commonsense reasoning, Visual MRC [47] for reading
comprehensive over images, and Winoground [48] for fine-grained semantics
reasoning over text descriptions and image contents.

Visual Question Answering (VQA) This is the most widely studied multi-modal
task, which requires the model to answer a given question based on the image
correctly. Tasks include VQA v2 [15], Shapes VQA [1], DocVQA [33], OCR-VQA
[34], ST-VQA [2], Text-VQA [45], and GQA [18].

Knowledgeable Visual Question Answering Unlike traditional VQA tasks focusing
on the question relevant to the content image, knowledgeable visual question
answer (KVQA) requires the model to draw upon outside knowledge to answer
questions. We incorporate two outside knowledge VQA datasets: OK-VQA [32] and
A-OK-VQA [42], ScienceQA [31] which contains multi-modal science questions,
and ViQuAE [22] focusing on knowledge facts of named entities in images.

Classification This task involves classifying an image based on a given set of
candidate labels. ImageNet [40], Grounded Object Identification (COCO-GOI)
[27], COCO-Text [50], Image Text Matching (COCO-ITM) [27], e-SNLI-VE [20],
Multi-modal Fact Checking (Mocheg) [58], and IQA [9] are included. Due to
language model input length constraints, we reduce the number of options in
some datasets with extensive candidate labels, such as ImageNet.

Generation Visual conditional general requires models to understand the visual
content and make a composition meeting the task demand. We have Visual
Storytelling (VIST) [17], Visual Dialog (VisDial) [8], and multi-modal machine
translation Multi30k [10] in this category.

Chinese and multilingual Vision-Language Tasks To examine the effect of
instruction tuning on different languages, we incorporate several Chinese
vision-language tasks including FM-IQA [11] for VQA, COCO-CN [25] and
Flickr8k-CN [24] for captioning, Chinese Food Net [4] for classification, and
MMChat [62] for generation.

Video-Language Tasks Beyond the static images, we are interested in whether
instruction tuning can also be applied to video-text tasks. We include the
classic MSR-VTT datasets [55] for video captioning, MSRVTT-QA [54],
ActivityNet-QA [59], iVQA [57] and MSVD-QA [54] for video question answering,
Something-Something [14] for video action classification.

As shown in Figure 1, our dataset makes a wide coverage of the current
existing visual-language and video-language benchmarks, enabling different
skill sets for the language models, from simple image captioning to
complicated reasoning based on the image even beyond the visual content.

Figure 1: Tasks in our proposed multi-modal multilingual instruction tuning
dataset. The tasks in dashed white boxes are held-out evaluation sets that are
not adopted during training. Tasks with bold names are translated into $80$
languages.

###  3.2 Annotation Process

Table 2: The statistics of our instructions.

Number of different instructions 400 \- Image Captioning 52 \- Classification
113 \- Visual Question Answering 95 \- Knowledgeable Visual QA 40 \- Reasoning
60 \- Generation 40 Tokens per instruction $24.4\pm 9.6$ Instruction edit
distance among the same task $76.6\pm 37.2$ Instruction edit distance across
tasks $106.6\pm 39.5$

To build high-quality multi-modal instruction datasets, we rewrite various
datasets into a vision-to-text format. The annotation process includes four
steps: (1) writing instructions for each task, (2) structuring images and
texts into a unified schema, (3) checking the overall dataset quality, and (4)
building multilingual sets. Eight authors of this work are employed as human
annotators, each of whom is a graduate student familiar with relevant
literature.

Stage I: Instruction Writing To build high-quality instructions, we first ask
annotators to carefully read the dataset paper and check the original dataset
with some instances to get a clear understanding of the task. After that, they
are required to write $10$ diverse task instructions manually, covering the
key characteristics of the task. Table 2 shows the statistics of the written
instructions for each task. In total, we annotate $400$ instructions for all
tasks. The average length per instruction is $24.4$. To evaluate the diversity
of annotated instructions, we employ the average edit distance to measure the
similarity between two strings. The average edit distance within the same task
is $76.6$, indicating a good range of instruction diversity.

Stage II: Data Format Unification After the instruction has been written
according to the task characteristics, we further process the images and
corresponding text for a unified instance schema. For most datasets, we keep
the original images and text, where images are converted into corresponding
base64 encoded strings for easy data loading. We perform two modifications on
potential examples: (1) Adding Bounding Box to Images. For tasks designed for
specific regions in the image, a straightforward solution is to provide the
bounding box information in natural language for informing the language models
of the regions in interest. However, the image preprocessing techniques
adopted by different vision encoders may resize the original image, and the
original bounding box annotation thus needs further adjustments. Inspired by
the recent observation that common vision encoders such as CLIP [39] are
sensitive to the visual prompt [43], we directly tag the bounding box as a red
rectangle to the image, serving as a hint for VLMs to focus on the target
region. (2) Short Answer Paraphrasing. As recent studies have shown that the
original short and brief answers in the common VQA dataset could negatively
influence the model generation performance [7], we propose to utilize the
ChatGPT [36] model for paraphrasing the original answers, by providing origin
question and answer with potential extra contextual information. Contextual
information includes the caption of the original images and OCR tokens for the
scene-related question. The prompt used for answer paraphrasing can be found
in Appendix. Figure 2 illustrates the data modifications we performed on our
dataset.

Figure 2: (Left) On region-based tasks, bounding boxes are added to original
images to inform the model of the area in interest. (Right) Short answer
paraphrasing to improve the response quality.

Stage III: Quality Check In this stage, we assign a different annotator to
each task to review 10 examples from each split. During this stage, we
identify minor format inconsistencies between tasks and address them by
standardizing the task formats. We also observe that a few answers (less than
3% of examined instances) were not effectively paraphrased by ChatGPT due to
insufficient image information. We employ simple heuristics to filter these
paraphrased answers and use a basic template to convert the original answer
into a sentence. We find that this small portion of unsuccessful paraphrased
answers has negligible impact. Finally, the task dataset is deemed complete
once the annotator can successfully load it and re-examine the accuracy of the
instructions, inputs, and outputs for each instance examined.

Stage IV: Key Datasets Translation To boost the language diversity and support
the evaluation across different languages, we select a subset of datasets (OK-
VQA, ImageNet, Winoground, VQAv2, VIST, MSRVTT and MSRVTT-QA) that covers
different tasks and translate their evaluation data into $100$ languages
following FLORES-101 [13]. We translate $500$ samples for each split of each
task in our first version. More multilingual samples will be supported in the
future. We adopt the distillation version NLLB-1.3B [6] for translation, one
of the state-of-the-art open multilingual translation models. As there are no
native speakers for different languages, we adopt an automatic filtering
mechanism to ensure the translation quality, where languages with translation
BLEU scores from English larger than $20$ based on FLORES-101 results are
kept. After this step, only $80$ languages are kept (see Appendix for detailed
language names).

###  3.3 Dataset Format

The instance in our dataset consists of five fields: (1) Images: we represent
the images with the potentially added bounding box by a base64 string. (2)
Instruction: we randomly select an instruction from the task instruction pool
for each instance. (3) Inputs: we allocate this field for providing task-
specific inputs to the model, e.g., the question in the VQA tasks. For tasks
such as captioning, there is no extra input so the corresponding field is left
as an empty string. (4) Outputs: the required output to the specific tasks,
such as the description of the image for captioning tasks and the answer to
the image-related question. (5) Meta Data: we provide this field to preserve
important information such as image id for referencing the original dataset.
Figure 3 illustrates an instance in the unified format. With the clear
distinction of these fields, the user of our benchmark can flexibly construct
the training instances needed and evaluate the models conveniently. Table 3
gives the statistics aggregated by tasks, and we refer readers to Appendix for
detailed statistics and the license of each dataset.

Figure 3: A ViQuAE instance represented in the unified data instance schema
used in our dataset. Table 3: M3IT task descriptions and statistics,
encompassing image captioning (CAP), classification (CLS), visual question
answering (VQA), knowledgeable visual question answering (KVQA), reasoning
(REA), generation (GEN), Chinese vision-language, and video-language tasks. We
aggregate instance counts for training, validation, and test sets across all
tasks, totaling 2,429,264 instances.

Task | Description | Total #samples
---|---|---
Train | Val | Test
CAP | Given an image, write a description for the image. | 679,087 | 41,462 | 27,499
CLS | Given an image, classify the image into pre-defined categories. | 238,303 | 100,069 | 21,206
VQA | Given an image, answer a question relevant to the image. | 177,633 | 46,314 | 10,828
KVQA | Given an image, answer the question requires outside knowledge. | 39,981 | 11,682 | 5,477
REA | Given an image, conduct reasoning over the images. | 99,372 | 11,500 | 10,000
GEN | Given an image, make compositions with certain requirements. | 145,000 | 11,315 | 17,350
Chinese | CAP, CLS, VQA, and GEN tasks in Chinese. | 192,076 | 77,306 | 4,100
Video | CAP, CLS, and VQA tasks on video-language datasets. | 20,868 | 7,542 | 9,294
Multi-lingual | Translated tasks in $80$ languages | 0 | 240,000 | 184,000

##  4 Experiments

In this section, we build a VLM to validate the effectiveness of the proposed
M3IT dataset for multi-modal agents. We first introduce the experimental
setups (§ 4.1), then report and discuss the results (§ 4.2). Lastly, we
analyze the influence of task number and instruction diversity, and provide a
qualitative result (§ 4.3).

###  4.1 Experimental Settings

Implementation Details Inspired by the recent success of BLIP [23], we adopt
the vision encoder and the Q-former architecture in the BLIP2-OPT-2.7B [23]
model to extract relevant visual features from images. For the large language
models, we utilize Ziya-13B [61] derived from LLaMA [49] with bilingual
(English and Chinese) ability. We employ a two-staged training. Stage I
Visual-Text Alignment: To align the visual and textual feature space, we
utilize the instructions in the coco captioning and perform an initial
alignment training on LAION 400M [41]. We train the Q-former and the language
projection, resulting in a total $130$M parameters to optimize with AdamW
[30]. The batch size is set to $256$ to maximize the utilization of GPU and
the model is trained with $300$k steps. The learning rate linearly increases
to a peak value of 5$e$-5 in the first $2000$ steps and follows a cosine decay
scheduler. The weight decay is set to $0.05$. Stage II Multi-modal Instruction
Tuning: We further perform a multi-modal instruction tuning in our benchmark
to activate the great potential of LLMs. We train the model after alignment
training for $3$ epochs and with a lower learning rate of 1$e$-5 and a warmup
stage of $1000$ steps. Inspired by LoRa tuning [16], the weights for mapping
query and value vectors in the attention layer of LLMs are learnable in this
stage to better adapt to the instruction tuning dataset. Other training
parameters are consistent with Stage I. All experiments are conducted with 8
NVIDIA 80GB A100 GPUs. It took about 10 days for Stage I and Stage II can be
finished in a day.

Evaluation Setup To examine the generalization of instruction tuning, some
tasks are held-out for evaluation (see Figure 1 for held-in/out tasks). We are
interested in the following research questions: (RQ1) Can multi-modal
instruction tuning elicit world knowledge from LLMs? (RQ2) Can English-only
instruction tuning generalize to other languages such as Chinese? and (RQ3)
Can image-only multi-modal instruction tuning generalize to video-language
tasks? For RQ1, we evaluate our models on three KVQA tasks in our datasets,
i.e., OK-VQA [32], A-OKVQA [42] and ViQuAE. For RQ2 and RQ3, we perform zero-
shot transfer evaluation on Chinese vision-language and video-language
datasets, respectively. We use greedy decoding in inference if not otherwise
specified.

Metrics We adopt ROUGE-L [26] as an automatic metric to assess the consistency
between predictions and ground-truth answers, focusing on evaluating the
model’s conversational abilities. As the automatic metric may not fully
capture the nuances of conversational quality, we further introduce GPT-4 as a
proxy of human evaluators (§ 4).

Baselines We compare our models to recently proposed powerful multi-modal
agents, including (1) BLIP-2-Flan-T5-XXL [23] where an instruction-tuned
Flan-T5 [53] is connected with a powerful vision encoder to perform a series
of multi-modal tasks; (2) MiniGPT-4 which aligns a CLIP visual encoder with a
frozen Vicuna [5] with artificially collected dialog dataset; and (3)
InstructBLIP, a recently proposed instruction tuning enhanced multi-modal
agents with Vicuna-13B with converted multi-model datasets and the LLaVA [28]
dataset generated by GPT-4.

###  4.2 Main Results

Table 4:  ROUGE-L evaluation results of KVQA tasks. Our Ying-VLM outperforms
all the baselines consistently.

Model OK-VQA A-OKVQA ViQuAE BLIP2-Flan-T5-XXL 9.1 15.6 9.7 MiniGPT4 23.3 21.8
24.4 InstructBLIP 7.1 5.9 7.3 Ying-VLM (Ours) 27.5 24.5 29.6

Table 5: Zero-shot transfer to Chinese vision-language tasks. Our model
generalizes well on unseen Chinese captioning, VQA and classification tasks,
with the highest ROUGE-L.

Model Flickr-8k-CN FM-IQA Chinese-FoodNet MiniGPT4 9.6 20.1 5.0 InstructBLIP
5.2 2.3 1.0 Ying-VLM (Ours) 20.5 33.3 49.8

RQ1: Knowledgeable Visual Question Answer Evaluation The results on the KVQA
benchmarks are shown in Table 5. In comparison to the strongest baseline, our
model achieves an improvement of 3.2 and 2.7 ROUGE-L points for OK-VQA and
A-OKVQA, respectively. Additionally, Ying-VLM delivers the best performance on
the held-out ViQuAE dataset. These findings indicate that instruction tuning
on M3IT effectively harnesses knowledge from LLMs and elevates response
quality.

RQ2: Zero-Shot Transfer to Chinese Vision-Language Tasks We assess models on
three unseen Chinese vision-language tasks to investigate the cross-language
generalization capabilities of instruction tuning. BLIP-2 is not considered,
as Flan-T5 does not support Chinese.222For all models, we introduce a prompt
to promote Chinese outputs. See Appendix D for details. As illustrated in
Table 5, our model outperforms MiniGPT4 and InstructBLIP on all evaluated
tasks, demonstrating notable improvements. These findings indicate that
instruction tuning with English datasets can effectively generalize to
different languages, showcasing the promising potential that can be further
explored.

Table 6: Zero-shot transfer to video-language tasks. We report ROUGE-L score
for all tasks.

Model Video Captioning Video Question Answer MSRVTT iVQA ActivityNet-QA
MSRVTT-QA MSVD-QA BLIP-2-Flan-T5-XXL 8.8 11.1 8.9 10.3 13.2 InstructBLIP 14.3
6.3 9.3 4.0 7.0 Ying-VLM (Ours) 14.2 23.5 21.9 18.3 21.4

RQ3: Zero-Shot Transfer to Video-Language Tasks To evaluate performance on
video-language tasks, we uniformly sample $8$ frames from each video. A
comparison with MiniGPT4 is excluded, as it does not support video inputs.
Following the approach of InstructBLIP [7], we concatenate the visual
embedding extracted from the Q-former of each frame as a prefix embedding to
the language model. As demonstrated in Table 6, our model excels in these
challenging settings, significantly surpassing the BLIP-series baselines. It
is worth noting that the training dataset does not include any visual inputs
such as videos, implying that our instruction tuning effectively aids the
model in generalizing to video inputs with a temporal dimension.

Figure 4: Evaluation results using GPT-4 as an evaluator. Our model
outperforms MiniGPT-4 and InstructBLIP with a winning rate at 55.6% and 65.5%,
respectively.

GPT-4 Evaluation Results To further validate the quality of the generated
response, we propose to utilize the powerful GPT-4 model as a proxy of human
evaluators [38, 12]. Specifically, following Vicuna [5], we use GPT-4 to rate
the performance of different models against our Ying-VLM. Considering the API
cost of GPT-4, $300$ examples are randomly sampled from OK-VQA, A-OKVQA and
ViQuAE datasets as a subset for evaluation. For each sample, we construct a
prompt consisting of the original question, its corresponding reference
answer, the response generated by our Ying-VLM, and a baseline system output.
GPT-4 is queried with the prompt to rate both responses on a scale of ten
based on the given question and its reference answer. The ratings are
primarily based on the accuracy, relevance, and naturalness of the response to
meet the requirements when humans are interacting with multi-modal agents (see
Appendix for the detailed evaluation template). We employ the strategy
proposed by Wang et al. [51] to mitigate potential evaluation biases regarding
the response order.333https://github.com/i-Eval/FairEval Figure 4 shows that
our Ying-VLM outperforms all baseline models in most samples. Notably, Ying-
VLM beat the strongest baseline MiniGPT4 on $167$ over $300$ tested samples.
Consistent with the previous ROUGE-L evaluation, this result indicates that
the model fine-tuned on our instruction dataset can produce more accurate and
engaging responses on the challenging KVQA tasks.

###  4.3 Analysis

We investigate the effect of task number and instruction diversity on the
performance of learned models, providing insights for future studies to
utilize our benchmark better.

Figure 5: Performance increases with more instruction tuning datasets.

Figure 6: Performance changes with the varied number of instructions used for
training.

Effect of Task Number We investigate the influence of task numbers by randomly
shuffling our tasks and then selecting a subset to train the model during the
instruction tuning stage. Due to the computational resource limitation, we set
up a maximum of $5$k examples for each task and train all the models for $5$k
steps with a batch size of $64$. We select $0$, $4$, $8$, $16$ and all $27$
tasks for training, and report the individual ROUGE-L score and the average
score. As illustrated in Figure 6, increasing the number of tasks greatly
improves the results of the generalization performance. Besides, the
performance gain is not diminished as the task number increases. This is
promising as it indicates that we can continually improve performance by
introducing more tasks into the training. It would be interesting to
investigate the influence of different task clusters, which we leave for
future studies.

Effect Instruction Diversity To investigate the influence of instruction
diversity, we limit the number of instructions used in each dataset to $1$,
$2$, $4$, and $8$, resulting in varying levels of diversity for each task. The
other training parameters are consistent with those used in previous
experiments on task number investigation. Figure 6 shows that the performance
varies with the level of diversity. Specifically, our results suggest that
using four instructions per task is sufficient for achieving decent
performance. We leave a more in-depth analysis of the instruction diversity
for future work.

Qualitative Results

Figure 7: Case study of the model outputs. Correct answers are bolded with
green, wrong answers are in red and irrelevant answers are in grey. The model
trained with our datasets can provide natural and informative responses to
entity-centric questions, and generalize to the food classification task in
Chinese (English translation for visualization only).

We conduct a case study to provide a more straightforward understanding of
instruction-tuned models. The cases are chosen from the held-out ViQuAE and
ChineseFoodNet datasets. As shown in Figure 7, our model generates accurate
responses to all questions. In contrast, MiniGPT4 produces an incorrect answer
for the stadium question on the left and fails to follow instructions in the
subsequent cases, providing generic image descriptions instead. Additionally,
compared to InstructBLIP, which provides concise but less engaging answers for
the two questions requiring external knowledge, our model responds more
naturally and engagingly, underlining the value of our dataset. Our model also
successfully generalizes to Chinese inputs, accurately classifying the food
image based on the instruction. These cases emphasize the importance of
instruction tuning and demonstrate that our dataset can effectively enhance
the capabilities of VLMs.

##  5 Conclusion

In this paper, we present M3IT, a multi-modal multilingual instruction tuning
dataset for aiding the development of multi-modal large language models. The
dataset comprises 2.4 million carefully curated instances and 400 manually
written task instructions across 40 tasks. We build Ying-VLM to validate the
effectiveness of our dataset. Quantitative and qualitative results demonstrate
that the models trained with our datasets successfully follow human
instructions, provide more engaging responses, and achieve strong
generalization performance on unseen video and Chinese tasks. Further analysis
shows that the increased task number can continually boost performance, and
instruction diversity can influence results. We hope our proposed benchmark,
trained models, and experimental findings can facilitate future studies toward
building powerful multi-modal intelligent agents.

##  Appendix A Dataset Statistics

Table 7: Detailed task descriptions and statistics of our instruction tuning
tasks, including all datasets in all types of tasks. The column “Used”
indicates whether we use this dataset in the instruction tuning stage.

Task Dataset Used #samples License Train Val Test Captioning MS COCO [27] Yes
566,747 25,010 25,010 Custom TextCaps [44] Yes 97,765 13,965 0 Unknown Image-
Paragraph-Captioning [21] Yes 14,575 2,487 2,489 Custom Classification COCO-
GOI [27] Yes 30,000 2,000 0 Custom COCO-Text [50] Yes 118,312 27,550 0 Custom
ImageNet [40] Yes 30,000 50,000 0 Non-commercial COCO-ITM [27] Yes 30,000
5,000 5,000 Custom e-SNLI-VE [20] Yes 20,000 14,339 14,740 Unknown Mocheg [58]
Yes 4,991 180 466 CC BY 4.0 IQA [9] Yes 5,000 1,000 1,000 Custom VQA VQA v2
[15] Yes 30,000 30,000 0 CC-BY 4.0 Shapes VQA [1] Yes 13,568 1,024 1,024
Unknown DocVQA [33] Yes 39,463 5,349 0 Unknown OCR-VQA [34] Yes 11,414 4,940 0
Unknown ST-VQA [2] Yes 26,074 0 4,070 Unknown Text-VQA [45] Yes 27,113 0 5,734
CC BY 4.0 GQA [18] Yes 30,001 5,001 0 CC BY 4.0 KVQA OK-VQA [32] Yes 9,009
5,046 0 Unknown A-OK-VQA [42] Yes 17,056 1,145 0 Unknown ScienceQA [31] Yes
12,726 4,241 4,241 CC BY-NC-SA ViQuAE [22] No 1,190 1,250 1,236 CC By 4.0
Reasoning CLEVR [19] Yes 30,000 2,000 0 CC BY 4.0 NLVR [46] Yes 29,372 2,000 0
Unknown VCR [60] Yes 25,000 5,000 5,000 Custom VisualMRC [47] Yes 15,000 2,500
5,000 Unknown Winoground [48] No 0 0 800 Unknown Generation Visual
Storytelling [17] Yes 5,000 4,315 4,350 Unknown Visual Dialog [8] Yes 50,000
1,000 1,000 CC By 4.0 Multi30k [10] Yes 90,000 6,000 12,000 Non-commercial
Chinese FM-IQA [11] No 164,735 75,206 0 Unknown COCO-Caption CN [25] No 18,341
1,000 1,000 Non-commercial Flickr-8k-Caption CN [24] No 6,000 1,000 1,000 CC
By 3.0 Chinese Food Classification [4] No 0 0 1,100 Unknown Multimodal Chat
[62] No 3,000 1,000 1,000 Unknown Video Action-Classification [14] No 2,000
2,000 2,000 Custom iVQA [57] No 5,994 2,000 2,000 Unknown MSVD QA [54] No
1,161 245 504 Unknown ActivityNet QA [59] No 3,200 1,800 800 Unknown MSRVTT QA
[54] No 6,513 497 2,990 Unknown MSRVTT Captioning [55] No 2,000 1,000 1,000
Unknown

Table 7 lists the detailed statistics in our benchmark. We collect the dataset
license from PaperWithCode.444https://paperswithcode.com/ For datasets under
Unknown and Custom licenses, we suggest the users check the project page or
contact the dataset owner before usage.

##  Appendix B Template for Answer Paraphrase

We provide the paraphrase template in Table 8 for querying the ChatGPT to re-
write the original short answers, where {Q} and {A} is filled with the
question and the answer need to be paraphrased, respectively. We incorporate
an example to better inform the model of the paraphrasing tasks. For VQAv2
tasks, we add an extra {Caption} field in the template filled with
corresponding captions from the COCO dataset to provide extra context
information to help to paraphrase.

Table 8: Template used to query ChatGPT for answer paraphrasing. You are an AI
visual assistant. Now you are given a question related to an image and a short
ground-truth answer. Your task is to transform the ground-truth answer into a
natural and convincing response. Make sure the response is accurate, highly
relevant to the question, and consistent with the original answer. Question:
Which NASA space probe was launched to this planet in 1989? Answer: Magellan
Transformed Answer: NASA sent the Magellan spacecraft to Venus in 1989, which
was the first planetary spacecraft launched from a space shuttle. Question:
{Q} Answer: {A} Transformed Answer:

##  Appendix C Dataset Translation

Table 9: List of Language Codes, Scripts, and Languages Names for translated datasets. Language Code | Script | Language Name
---|---|---
af | afr_Latn | Afrikaans
am | amh_Ethi | Amharic
ar | arb_Arab | Modern Standard Arabic
as | asm_Beng | Assamese
ast | ast_Latn | Asturian
be | bel_Cyrl | Belarusian
bg | bul_Cyrl | Bulgarian
bn | ben_Beng | Bengali
bs | bos_Latn | Bosnian
ca | cat_Latn | Catalan
ceb | ceb_Latn | Cebuano
cs | ces_Latn | Czech
cy | cym_Latn | Welsh
da | dan_Latn | Danish
de | deu_Latn | German
el | ell_Grek | Greek
es | spa_Latn | Spanish
et | est_Latn | Estonian
fi | fin_Latn | Finnish
fr | fra_Latn | French
fuv | fuv_Latn | Nigerian Fulfulde
gl | glg_Latn | Galician
gu | guj_Gujr | Gujarati
ha | hau_Latn | Hausa
he | heb_Hebr | Hebrew
hi | hin_Deva | Hindi
hr | hrv_Latn | Croatian
hu | hun_Latn | Hungarian
hy | hye_Armn | Armenian
id | ind_Latn | Indonesian
ig | ibo_Latn | Igbo
is | isl_Latn | Icelandic
it | ita_Latn | Italian
ja | jpn_Jpan | Japanese
jv | jav_Latn | Javanese
ka | kat_Geor | Georgian
kk | kaz_Cyrl | Kazakh
km | khm_Khmr | Khmer
kn | kan_Knda | Kannada
ko | kor_Hang | Korean
ky | kir_Cyrl | Kyrgyz
lb | ltz_Latn | Luxembourgish
lg | lug_Latn | Ganda
lij | lij_Latn | Ligurian
li | lim_Latn | Limburgish
ln | lin_Latn | Lingala
lo | lao_Laoo | Lao
lt | lit_Latn | Lithuanian
lv | lvs_Latn | Standard Latvian
mi | mri_Latn | Maori
mk | mkd_Cyrl | Macedonian
ml | mal_Mlym | Malayalam
mr | mar_Deva | Marathi
mt | mlt_Latn | Maltese
my | mya_Mymr | Burmese
nl | nld_Latn | Dutch
ny | nya_Latn | Nyanja
oc | oci_Latn | Occitan
pa | pan_Guru | Eastern Panjabi
pl | pol_Latn | Polish
pt | por_Latn | Portuguese
ro | ron_Latn | Romanian
ru | rus_Cyrl | Russian
sd | snd_Arab | Sindhi
sk | slk_Latn | Slovak
sn | sna_Latn | Shona
so | som_Latn | Somali
sr | srp_Cyrl | Serbian
sv | swe_Latn | Swedish
ta | tam_Taml | Tamil
te | tel_Telu | Telugu
tg | tgk_Cyrl | Tajik
th | tha_Thai | Thai
tl | tgl_Latn | Tagalog
tr | tur_Latn | Turkish
uk | ukr_Cyrl | Ukrainian
ur | urd_Arab | Urdu
vi | vie_Latn | Vietnamese
wo | wol_Latn | Wolof
zh | zho_Hans | Chinese (Simplified)

We translate all the task instructions and evaluation sets of ImageNet,
Winoground, VQAv2, OK-VQA, VIST, MSRVTT and MSRVTT-QA into $80$ languages, as
shown in Table 9. Due to the computational resource constraint, we translate
the whole test of Winoground ( $800$ examples) and set a maximum instance
number of $500$ for each split in other tasks.

##  Appendix D Prompt for Zero-Shot Chinese Vision-Language Tasks

In our experiments, all Vision-Language models are fine-tuned exclusively
using English data. In our preliminary study, we observe that these models
tend to generate English responses, even when the input and instructions are
written in Chinese. We introduce a simple Chinese dialogue context during the
zero-shot Chinese Vision-Language Task evaluation for all models, as
illustrated in Table 10, Interestingly, this minor adjustment can encourage
models to produce reasonable Chinese output. We leave the analysis of
instruction-tuned VLM models’ multilingual capabilities for future research.

Table 10: Prompt for promoting Chinese outputs. <human>:
请根据我的指示，以及所给的图片，做出相应的回答。 <bot>: 好的。 <human>: {Instruction} {Input} <bot>: 好的。

##  Appendix E Template for GPT-4 Evaluation

We adopt the template in Table 11 to query GPT-4 and obtain the evaluation
results with FairEval 555https://github.com/i-Eval/FairEval to obtain more
stable results. Specifically, each tested instance is a quaternion: (question,
reference, response1, response2), where response1 and response2 are two
responses from our Ying-VLM and the baseline model, respectively. For each
instance, we query GPT-4 to judge which response is of better quality
regarding accuracy, relevance and naturalness. We populate the quaternion into
the evaluation template to form two query prompts: T(Q=question, R=reference,
R1=response1, R2=response2) and T(Q=question, R=reference, R1=response2,
R2=response1). We set the temperature of GPT-4 to $1$ and sample three
completions for each query prompt. Therefore, each response will receive $6$
scores, and we use the average score as the final score for each response. The
response with the higher final score is considered the better response. The
GPT-4 evaluation incurred a cost of $$20.45$ for InstructBlip and $$20.90$ for
MiniGPT-4.

Table 11: Template used to query GPT-4 for evaluating the response quality of
different models. [Question] {Q} [The Start of Reference Answer] {R} [The End
of Reference Answer] [The Start of Assistant 1’s Answer] {R1} [The End of
Assistant 1’s Answer] [The Start of Assistant 2’s Answer] {R2} [The End of
Assistant 2’s Answer] [System] We would like to request your feedback on the
performance of two AI assistants in response to the user’s multimodal question
displayed above. We provided no multimodal inputs other than question text,
but we provided a reference answer for this question. You need to evaluate the
quality of the two responses based on the question and the reference answer.
Please rate the on the follow aspects: 1\. Accuracy: whether the candidate’s
response is consistent with the original answer, this is important as we do
not want a misleading result; 2\. Relevance: whether the candidate’s response
is highly relevant to the question and image content; 3\. Naturalness: whether
the candidate’s response is engaging, providing a great communication
experience for the user when interacting with the AI visual assistant. of the
two Assistants’ responses. Each assistant receives an overall score on a scale
of 1 to 10, where a higher score indicates better overall performance. Please
first provide a comprehensive explanation of your evaluation, avoiding any
potential bias and ensuring that the order in which the responses were
presented does not affect your judgment. Then, output two lines indicating the
scores for Assistant 1 and 2, respectively. Output with the following format:
Evaluation evidence: <evaluation explanation here> The score of Assistant 1:
<score> The score of Assistant 2: <score>

## References

  * Andreas et al. [2016] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein.  Neural module networks.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 39–48, 2016.
  * Biten et al. [2019] A. F. Biten, R. Tito, A. Mafla, L. G. i Bigorda, M. Rusiñol, C. V. Jawahar, E. Valveny, and D. Karatzas.  Scene text visual question answering.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pages 4290–4300, 2019.
  * Brown et al. [2020] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Chen et al. [2017] X. Chen, Y. Zhu, H. Zhou, L. Diao, and D. Wang.  Chinesefoodnet: A large-scale image dataset for chinese food recognition.  _ArXiv preprint_ , abs/1705.02743, 2017.
  * Chiang et al. [2023] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
  * Costa-jussà et al. [2022] M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, et al.  No language left behind: Scaling human-centered machine translation.  _ArXiv preprint_ , abs/2207.04672, 2022.
  * Dai et al. [2023] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _ArXiv preprint_ , abs/2305.06500, 2023.
  * Das et al. [2017] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra.  Visual dialog.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 1080–1089, 2017.
  * Duanmu et al. [2021] Z. Duanmu, W. Liu, Z. Wang, and Z. Wang.  Quantifying visual image quality: A bayesian view.  _Annual Review of Vision Science_ , 7:437–464, 2021.
  * Elliott et al. [2016] D. Elliott, S. Frank, K. Sima’an, and L. Specia.  Multi30K: Multilingual English-German image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language_ , pages 70–74, 2016.
  * Gao et al. [2015] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu.  Are you talking to a machine? dataset and methods for multilingual image question.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 2296–2304, 2015.
  * Gilardi et al. [2023] F. Gilardi, M. Alizadeh, and M. Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _ArXiv preprint_ , abs/2303.15056, 2023.
  * Goyal et al. [2022] N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan.  The Flores-101 evaluation benchmark for low-resource and multilingual machine translation.  _Transactions of the Association for Computational Linguistics_ , 10:522–538, 2022.
  * Goyal et al. [2017a] R. Goyal, S. E. Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fründ, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic.  The "something something" video database for learning and evaluating visual common sense.  In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , pages 5843–5851, 2017a.
  * Goyal et al. [2017b] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 6325–6334, 2017b.
  * Hu et al. [2022] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_ , 2022.
  * Huang et al. [2016] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra, C. L. Zitnick, D. Parikh, L. Vanderwende, M. Galley, and M. Mitchell.  Visual storytelling.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1233–1239, 2016.
  * Hudson and Manning [2019] D. A. Hudson and C. D. Manning.  GQA: A new dataset for real-world visual reasoning and compositional question answering.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 6700–6709, 2019.
  * Johnson et al. [2017] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. B. Girshick.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 1988–1997, 2017.
  * Kayser et al. [2021] M. Kayser, O. Camburu, L. Salewski, C. Emde, V. Do, Z. Akata, and T. Lukasiewicz.  e-vil: A dataset and benchmark for natural language explanations in vision-language tasks.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 1224–1234, 2021\.
  * Krause et al. [2017] J. Krause, J. Johnson, R. Krishna, and L. Fei-Fei.  A hierarchical approach for generating descriptive image paragraphs.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 3337–3345, 2017.
  * Lerner et al. [2022] P. Lerner, O. Ferret, C. Guinaudeau, H. Le Borgne, R. Besançon, J. G. Moreno, and J. Lovón Melgarejo.  Viquae, a dataset for knowledge-based visual question answering about named entities.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 3108–3120, 2022\.
  * Li et al. [2023] J. Li, D. Li, S. Savarese, and S. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ArXiv preprint_ , abs/2301.12597, 2023.
  * Li et al. [2016] X. Li, W. Lan, J. Dong, and H. Liu.  Adding chinese captions to images.  In _Proceedings of the 2016 ACM on international conference on multimedia retrieval_ , pages 271–275, 2016.
  * Li et al. [2019] X. Li, C. Xu, X. Wang, W. Lan, Z. Jia, G. Yang, and J. Xu.  Coco-cn for cross-lingual image tagging, captioning, and retrieval.  _IEEE Transactions on Multimedia_ , 21(9):2347–2360, 2019.
  * Lin [2004] C.-Y. Lin.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, 2004.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2023] H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual instruction tuning.  _ArXiv preprint_ , abs/2304.08485, 2023.
  * Longpre et al. [2023] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _ArXiv preprint_ , abs/2301.13688, 2023.
  * Loshchilov and Hutter [2019] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_ , 2019.
  * Lu et al. [2022] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Marino et al. [2019] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi.  OK-VQA: A visual question answering benchmark requiring external knowledge.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 3195–3204, 2019.
  * Mathew et al. [2021] M. Mathew, D. Karatzas, and C. Jawahar.  Docvqa: A dataset for vqa on document images.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 2200–2209, 2021.
  * Mishra et al. [2019] A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In _2019 international conference on document analysis and recognition (ICDAR)_ , pages 947–952. IEEE, 2019.
  * Mishra et al. [2022] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, 2022\.
  * OpenAI [2022] OpenAI.  Introducing chatgpt.  2022\.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Peng et al. [2023] B. Peng, C. Li, P. He, M. Galley, and J. Gao.  Instruction tuning with gpt-4.  _ArXiv preprint_ , abs/2304.03277, 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever.  Learning transferable visual models from natural language supervision.  In M. Meila and T. Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763, 2021.
  * Russakovsky et al. [2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * Schuhmann et al. [2021] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _ArXiv preprint_ , abs/2111.02114, 2021.
  * Schwenk et al. [2022] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII_ , pages 146–162. Springer, 2022.
  * Shtedritski et al. [2023] A. Shtedritski, C. Rupprecht, and A. Vedaldi.  What does clip know about a red circle? visual prompt engineering for vlms.  _ArXiv preprint_ , abs/2304.06712, 2023.
  * Sidorov et al. [2020] O. Sidorov, R. Hu, M. Rohrbach, and A. Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pages 742–758. Springer, 2020.
  * Singh et al. [2019] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach.  Towards VQA models that can read.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 8317–8326, 2019.
  * Suhr et al. [2017] A. Suhr, M. Lewis, J. Yeh, and Y. Artzi.  A corpus of natural language for visual reasoning.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 217–223, 2017\.
  * Tanaka et al. [2021] R. Tanaka, K. Nishida, and S. Yoshida.  Visualmrc: Machine reading comprehension on document images.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 13878–13888, 2021.
  * Thrush et al. [2022] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5238–5248, 2022.
  * Touvron et al. [2023] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _ArXiv preprint_ , abs/2302.13971, 2023.
  * Veit et al. [2016] A. Veit, T. Matera, L. Neumann, J. Matas, and S. Belongie.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  _ArXiv preprint_ , abs/1601.07140, 2016.
  * Wang et al. [2023] P. Wang, L. Li, L. Chen, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui.  Large language models are not fair evaluators.  _ArXiv preprint_ , abs/2305.17926, 2023.
  * Wang et al. [2022] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat, S. Mishra, S. Reddy A, S. Patro, T. Dixit, and X. Shen.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, 2022.
  * Wei et al. [2022] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.  Finetuned language models are zero-shot learners.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_ , 2022.
  * Xu et al. [2017] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang.  Video question answering via gradually refined attention over appearance and motion.  In _Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017_ , pages 1645–1653, 2017.
  * Xu et al. [2016] J. Xu, T. Mei, T. Yao, and Y. Rui.  MSR-VTT: A large video description dataset for bridging video and language.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 5288–5296, 2016.
  * Xu et al. [2022] Z. Xu, Y. Shen, and L. Huang.  Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning.  _ArXiv preprint_ , abs/2212.10773, 2022.
  * Yang et al. [2021] A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid.  Just ask: Learning to answer questions from millions of narrated videos.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 1666–1677, 2021\.
  * Yao et al. [2022] B. M. Yao, A. Shah, L. Sun, J.-H. Cho, and L. Huang.  End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models.  _ArXiv preprint_ , abs/2205.12487, 2022.
  * Yu et al. [2019] Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao.  Activitynet-qa: A dataset for understanding complex web videos via question answering.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 9127–9134, 2019.
  * Zellers et al. [2019] R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi.  From recognition to cognition: Visual commonsense reasoning.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 6720–6731, 2019.
  * Zhang et al. [2022] J. Zhang, R. Gan, J. Wang, Y. Zhang, L. Zhang, P. Yang, X. Gao, Z. Wu, X. Dong, J. He, J. Zhuo, Q. Yang, Y. Huang, X. Li, Y. Wu, J. Lu, X. Zhu, W. Chen, T. Han, K. Pan, R. Wang, H. Wang, X. Wu, Z. Zeng, and C. Chen.  Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence.  _ArXiv preprint_ , abs/2209.02970, 2022.
  * Zheng et al. [2022] Y. Zheng, G. Chen, X. Liu, and J. Sun.  MMChat: Multi-modal chat dataset on social media.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 5778–5786, 2022.
  * Zhu et al. [2023] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _ArXiv preprint_ , abs/2304.10592, 2023.