#  All in a Single Image: Large Multimodal Models are In-Image Learners

Lei Wang1♠ Wanyu Xu2♠ Zhiqiang Hu3♠ Yihuai Lan4♠ Shan Dong1♠
Hao Wang4 Roy Ka-Wei Lee3 Ee-Peng Lim1
1Singapore Management University
2Southwest Jiaotong University
3Singapore University of Technology and Design
4The Hong Kong University of Science and Technology (Guangzhou)

###### Abstract

This paper introduces a new in-context learning (ICL) mechanism called In-
Image Learning (I2L) that combines demonstration examples, visual cues, and
instructions into a single image to enhance the capabilities of GPT-4V. Unlike
previous approaches that rely on converting images to text or incorporating
visual input into language models, I2L consolidates all information into one
image and primarily leverages image processing, understanding, and reasoning
abilities. This has several advantages: it avoids inaccurate textual
descriptions of complex images, provides flexibility in positioning
demonstration examples, reduces the input burden, and avoids exceeding input
limits by eliminating the need for multiple images and lengthy text. To
further combine the strengths of different ICL methods, we introduce an
automatic strategy to select the appropriate ICL method for a data example in
a given task. We conducted experiments on MathVista and Hallusionbench to test
the effectiveness of I2L in complex multimodal reasoning tasks and mitigating
language hallucination and visual illusion. Additionally, we explored the
impact of image resolution, the number of demonstration examples, and their
positions on the effectiveness of I2L. Our code is publicly available at
https://github.com/AGI-Edgerunners/IIL.

All in a Single Image: Large Multimodal Models are In-Image Learners

Lei Wang1♠ Wanyu Xu2♠ Zhiqiang Hu3♠ Yihuai Lan4♠ Shan Dong1♠ Hao Wang4 Roy Ka-
Wei Lee3 Ee-Peng Lim1 1Singapore Management University 2Southwest Jiaotong
University 3Singapore University of Technology and Design 4The Hong Kong
University of Science and Technology (Guangzhou)

##  1 Introduction

Figure 1: (a) Text-only in-context learning (T-ICL). (b) T-ICL with additional
image-to-text models (T-ICL-Img). (c) Visual-text interleaved in-context
learning (VT-ICL). (d) In-image learning (I2L).

Recently, there has been significant progress in large language models (LLMs)
Brown et al. (2020); Chowdhery et al. (2022); OpenAI (2022); Touvron et al.
(2023a); OpenAI (2023a). The popularity of LLMs like ChatGPT OpenAI (2022) has
led to the development of numerous open-source LLMs (Zhang et al., 2022a;
Touvron et al., 2023a, b) and novel prompting methods Wei et al. (2022); Zhou
et al. (2022); Kojima et al. (2022); Zhang et al. (2022b); Wang et al. (2023).
Following ChatGPT, GPT-4 with vision capabilities (GPT-4V OpenAI (2023b)), a
powerful and versatile large multimodal model (LMM), has been developed and
has gained growing interest in the research community due to its strong
abilities to process and interpret both text and image inputs OpenAI (2023a).

Despite the success of GPT-4V, there are still many failure cases in multi-
modal tasks Yang et al. (2023b). For example, when reading a complex plot, it
may not be able to fully understand complex information presented in the
image. The underperformance of GPT-4V in these tasks may be attributed to its
failure to fully utilize its capabilities Yang et al. (2023b, a); Zhang et al.
(2023b). Yang et al. (2023b) discover that GPT-4V has the ability to
understand visual cues depicted directly on images. To effectively leverage
visual cues, Visual Referring Prompting Yang et al. (2023b) is proposed by
editing the pixels of input images to include elements like arrows, boxes, and
circles. To unlock GPT-4V’s ability to model fine-grained visual grounding,
Yang et al. (2023a) introduce a new prompting mechanism called Set-of-Mark
prompting. This mechanism involves adding visual marks ((i.e., numeric or
alphabetic labels)) to image regions so that GPT-4V can better understand and
interpret these edited regions.

To further utilize visual cues, GPT-4V can employ in-context learning to
enhance its capabilities Yang et al. (2023b). In-context learning is widely
used to enable LLMs to adapt to new NLP tasks by learning from a few
demonstration examples Brown et al. (2020); Liu et al. (2021); Min et al.
(2022); Dong et al. (2022). To transfer in-context learning from solving NLP
tasks to multimodal tasks, a common approach is to convert images into textual
descriptions using image-to-text models Yang et al. (2022); Guo et al. (2022).
To eliminate the need for additional models, Flamingo Alayrac et al. (2022) is
trained to directly encode interleaved visual-text demonstration examples by
incorporating visual inputs to LLMs. This raises the question: can we
incorporate all information, including demonstrations, visual cues, and other
valuable information, into a single image instead of using additional image-
to-text models or incorporating visual inputs to LLMs, to enable LMMs (such as
GPT-4V) to perform in-context learning effectively?

In this work, we introduce a new in-context learning mechanism called In-Image
Learning (I2L), which incorporates demonstration examples, visual cues, chain-
of-thought reasoning, and instructions into a single image to unlock the
ability of GPT-4V. Consolidating valuable information into a single image
offers several advantages. Firstly, I2L relies on image modeling, avoiding
inaccurately described certain complex images by text Hu et al. (2022).
Secondly, I2L can position demonstration examples anywhere within the image
and organize them using visual cues, which may avoid sensitivity to the
ordering of demonstration examples Lu et al. (2021). Thirdly, the input
consists of only one image, eliminating the need for multiple images and
lengthy text. This reduces the input burden and avoids exceeding input limits
Jiang et al. (2023); Liu et al. (2023e). We observe that I2L is good at
handling complex images, while VT-ICL is better for images that can be easily
described by text. To combine the strengths of these two methods for
multimodal tasks, we propose using GPT-4V as a selector to determine the
appropriate ICL method for a given multimodal data example in a task.

For experiments, We start by conducting an empirical study on MathVista Lu et
al. (2023) to test if GPT-4V can perform better on complex reasoning images,
some of which are difficult to accurately describe with text alone, using I2L.
Secondly, Hallusionbench Liu et al. (2023a) is motivated by the observation
that strong language bias often overshadows visual information (“Language
Hallucination”) and visual inputs can be misinterpreted (“Visual Illusion”).
Thus, we experiment on Hallusionbench to check if I2L can mitigate language
hallucination and visual illusion. Lastly, I2L may be sensitive to image
resolution, the number of demonstration examples in the input image, and the
position of demonstrations in the input image. We thus experiment on a subset
of VQA Goyal et al. (2017) to further explore factors that may influence I2L.

##  2 Related Work

###  2.1 Prompting

Recent advancements in the field of LLMs have received significant attention
Brown et al. (2020); Chowdhery et al. (2022), particularly following the
success of models like ChatGPT OpenAI (2022). This success has led to the
development of various open-source LLMs Zhang et al. (2022a); Touvron et al.
(2023a, b) and innovative prompting techniques Wei et al. (2022); Zhou et al.
(2022); Kojima et al. (2022); Zhang et al. (2022b); Wang et al. (2023).
Following ChatGPT, a powerful and versatile LMM called GPT-4 OpenAI (2023a)
with vision capabilities (GPT-4V OpenAI (2023b)) has been developed. GPT-4V
can process and interpret both textual and visual inputs, making it highly
regarded for its ability to understand visual elements in images Yang et al.
(2023b). Yang et al. (2023b) highlight GPT-4V’s proficiency in recognizing and
interpreting visual signals, such as arrows, boxes, circles, and hand-drawn
shapes, directly from images, and introduce visual referring prompting, which
involves modifying image pixels to enhance visual cues. To unleash GPT-4V’s
ability to model fine-grained visual grounding, Yang et al. (2023a) introduce
a new prompting mechanism called Set-of-Mark (SoM) prompting. This mechanism
involves adding visual marks ((i.e., numeric or alphabetic labels)) to image
regions so that GPT-4V can better understand and interpret these edited
regions. In this work, we expand on these approaches by integrating visual
cues into demonstration examples rather than test queries. This allow us to
leverage in-context learning and improve GPT-4V’s performance on multimodal
tasks, demonstrating its potential to understand and respond to complex
inputs.

###  2.2 In-Context Learning

Recent advancements in large language models Brown et al. (2020); Chowdhery et
al. (2022) have introduced a new capability, where models can adapt to a new
NLP task by utilizing a few demonstration examples. This adaptation, known as
in-context learning (ICL) Dong et al. (2022), relies on learning from task-
relevant demonstrations. The performance of ICL is greatly influenced by the
wording of instructions Madaan and Yazdanbakhsh (2022), label design Yoo et
al. (2022), demonstration selection Liu et al. (2021); Shi et al. (2022) and
ordering Lu et al. (2021). In the fields of multimodality, there have been
early attempts at in-context learning. One example is Flamingo Alayrac et al.
(2022), which is trained by integrating visual inputs into LLMs. This enables
the in-context learning of visual-linguistic tasks such as image captioning
and OCR through language-based interfacing. Otter Li et al. (2023a), a multi-
modal model based on OpenFlamingo Awadalla et al. (2023), trained on a multi-
modal in-context instruction tuning dataset and showcasing improved in-context
learning ability. To adapt LLMs from NLP tasks to more multimodal tasks,
another common strategy is to leverage strong closed-source LLMs like ChatGPT,
without pre-training and fine-tuning. This involves converting the
corresponding images into textual descriptions, treating multimodal tasks as
normal NLP tasks Yang et al. (2022); Guo et al. (2022); Hu et al. (2022); He
et al. (2023). This work introduces a new in-context learning mechanism called
In-Image Learning. It incorporates demonstrations and useful information into
a single image, which is then fed into GPT-4V. Compare to previous in-context
learning approaches for multimodal tasks, which convert images to textual
descriptions and rely on the strong text processing ability of LLMs, in-image
learning consolidates all information into one image and primarily leverages
the image processing ability of LMMs.

##  3 Approach

In this section, we first describe prompting with visual cues and text-only
in-context learning (T-ICL). Next, we present text-only in-context learning
with additional image-to-text models (T-ICL-Img). We then describe visual-text
interleaved in-context learning (VT-ICL). Finally, we introduce the proposed
in-image learning (I2L).

###  3.1 Prompting with Visual Cues

GPT-4V is highly proficient in understanding visual cues, such as symbols and
numbers in the image. These cues are used to develop visual prompting methods,
including Visual Referring Prompting (Yang et al., 2023b) and Set-of-Mark
prompting (Yang et al., 2023a). Visual cues can be obtained by the operation
$f_{\text{vc}}$ that edits the pixels of the corresponding image. Prediction
by this prompting can be formulated as follows:

$\underbrace{x^{\text{img}}_{\text{vc},q},x^{\text{txt}}_{q}}_{\text{query
}},\underbrace{\hat{y}_{q}^{\text{txt}}}_{\text{prediction }},$ (1)

where
$x^{\text{img}}_{\text{vc},q}=f_{\text{vc}}\left(x^{\text{img}}_{q}\right)$
and $x^{\text{txt}}_{q}$ are the image with visual cues and text information
of the test query $x_{q}$ respectively, and $\hat{y}_{q}^{\text{txt}}$ is the
prediction generated by GPT-4V.

###  3.2 T-ICL

Several previous works Wei et al. (2022); Dong et al. (2022) describe T-ICL as
a learning paradigm that allows LLMs to solve tasks by learning from only a
few demonstration examples. In this paradigm, given $k$ text demonstration
examples
$x^{\text{txt}}_{1},x^{\text{txt}}_{2},\ldots,x^{\text{txt}}_{k}\in\mathcal{D}$
of a new task along with their corresponding ground truth labels
$y^{\text{txt}}_{1},y^{\text{txt}}_{2},\ldots,y^{\text{txt}}_{k}$, the goal of
ICL is to ask LLMs to generate $\hat{y}^{\text{txt}}_{q}$ as the predicted
answer to a query data $x^{\text{txt}}_{q}$. T-ICL can be formulated as
follows:

$\underbrace{x^{\text{txt}}_{1},y^{\text{txt}}_{1},\ldots,x^{\text{txt}}_{k},y^{\text{txt}}_{k}}_{\text{in-
context examples (left-to-right)
}},\underbrace{x^{\text{txt}}_{q}}_{\text{query
}},\underbrace{\hat{y}_{q}^{\text{txt}}}_{\text{prediction }}.$ (2)

###  3.3 T-ICL-Img

To adapt LLMs from NLP tasks to multimodal tasks, a common strategy is to
convert the corresponding images into textual descriptions using image-to-text
models $f_{\text{i2t}}$ (Yang et al., 2022; Guo et al., 2022). Formally, given
$k$ input image-text data pairs
$(x^{\text{txt}}_{1},x^{\text{img}}_{1}),(x^{\text{txt}}2,x^{\text{img}}2),\ldots,(x^{\text{txt}}{k},x^{\text{img}}{k})\in\mathcal{D}$
with respective ground truth labels
$y^{\text{txt}}_{1},y^{\text{txt}}2,\ldots,y^{\text{txt}}{k}$, T-ICL-Img aims
to output $\hat{y}q^{\text{txt}}$ for the query
$(x^{\text{txt}}{q},x^{\text{img}}_{q})$ using LLMs based on the knowledge
gained from the given $k$ data examples, formulated as:

$\begin{split}\underbrace{x^{\text{cap}}_{1},x^{\text{txt}}_{1},y^{\text{txt}}_{1}\ldots
x^{\text{cap}}_{k},x^{\text{txt}}_{k},y^{\text{txt}}_{k}}_{\text{in-context
examples (left-to-right)
}},\underbrace{x^{\text{cap}}_{q},x^{\text{txt}}_{q}}_{\text{query
}},\underbrace{\hat{y}_{q}^{\text{txt}}}_{\text{prediction }},\end{split}$ (3)

where $x^{\text{cap}}_{i}=f_{\text{i2t}}\left(x^{\text{img}}_{i}\right)$ and
$f_{\text{i2t}}$ represents the image-to-text mdoel.

###  3.4 VT-ICL

While the results of T-ICL-Img are promising, there is a potential risk of
losing information when converting visual inputs into a textual description
Yang et al. (2022); Hu et al. (2022). To avoid the need for additional image-
to-text models, such as generic caption models, interleaved image-text pairs
can be prepared for in-context learning, and visual inputs can be directly
incorporated into LLMs. Formally, given $k$ input image-text data pairs
$(x^{\text{txt}}_{1},x^{\text{img}}_{1}),(x^{\text{txt}}_{2},x^{\text{img}}_{2}),\ldots,(x^{\text{txt}}_{k},x^{\text{img}}_{k})\in\mathcal{D}$
with their ground truth labels
$y^{\text{txt}}_{1},y^{\text{txt}}_{2},\ldots,y^{\text{txt}}_{k}$, VT-ICL aims
to output $\hat{y}^{\text{txt}}_{q}$ for the query data
$(x^{\text{txt}}_{q},x^{\text{img}}_{q})$ using LLMs based on the knowledge
gained from the given $k$ demonstration examples, formulated as:

$\underbrace{x^{\text{img}}_{1},x^{\text{txt}}_{1},y^{\text{txt}}_{1},\ldots,x^{\text{img}}_{k},x^{\text{txt}}_{k},y^{\text{txt}}_{k}}_{\text{in-
context examples (left-to-right)
}},\underbrace{x^{\text{img}}_{q},x^{\text{txt}}_{q}}_{\text{query
}},\underbrace{\hat{y}_{q}^{\text{txt}}}_{\text{prediction }}$ (4)

###  3.5 The Proposed I2L

This section presents the proposed In-Image Learning (I2L), which combines
visual-text demonstration examples, visual cues, and instructions into a
single image to enhance the capabilities of GPT-4V. Consolidating valuable
information into a single image offers three main benefits. Firstly, it
effectively conveys complex images that cannot be accurately described by text
alone. Secondly, it allows demonstration examples to be positioned anywhere
within the image, unlike previous in-context learning methods that are
sensitive to the ordering of examples. Thirdly, using only one image as input
reduces the need for lengthy input, thereby reducing the input burden and
avoiding exceeding the input limits of LMMs. Formally, given $k$ input image-
text data pairs
$(x^{\text{txt}}_{1},x^{\text{img}}_{1}),(x^{\text{txt}}_{2},x^{\text{img}}_{2}),\ldots,(x^{\text{txt}}_{k},x^{\text{img}}_{k})\in\mathcal{D}$
with their ground truth labels
$y^{\text{txt}}_{1},y^{\text{txt}}_{2},\ldots,y^{\text{txt}}_{k}$, the
inference of I2L aims to output $\hat{y}^{\text{txt}}_{q}$ for the query data
$(x^{\text{txt}}_{q},x^{\text{img}}_{q})$ using LMMs (i.e., GPT-4V) based on
the knowledge gained from the given $k$ data examples with visual cues in a
position permutation order $\pi$ within one single image, where images with
visual cues $x^{\text{img}}_{\text{vc},i}$ can be obtained by editing pixels
using the operation $f^{\text{vc}}$, formulated as:

$\underbrace{x^{\text{img}}_{\text{vc},1},x^{\text{txt}}_{1},y^{\text{txt}}_{1}\ldots
x^{\text{img}}_{\text{vc},1},x^{\text{txt}}_{1},y^{\text{txt}}_{1},\overbrace{x^{\text{img}}_{q},x^{\text{txt}}_{q}}^{\text{query}}}_{\text{demonstrations
in any permutation order in an
image}},\underbrace{\hat{y}_{q}^{\text{txt}}}_{\text{prediction }},$ (5)

where $x_{\text{vc}}^{\text{img}}=f_{vc}\left(x^{\text{img}}_{\pi,1}\right)$.
Note that we do not include any visual cues or other annotations for the test
query image $x^{\text{img}}_{q}$.

Figure 2: Overview of GPT-4V-Selector.

###  3.6 Automatic Selection of ICLs

The proposed I2L excels at handling complex images that cannot be accurately
described using text alone, while VT-ICL can better leverage text information
to enhance performance for images that can be easily described by text. To
combine the strengths of these two ICL methods for multimodal tasks, we draw
inspiration from the GPT-4V-as-a-Generalist-Evaluator Zhang et al. (2023c) and
propose using GPT-4V as a selector called GPT-4V-Selector to determine the
appropriate ICL method for a specific multimodal data example in a given task.
Specifically, we first prompt GPT-4V to generate a description for a given
data example’s image. Then, we ask GPT-4V to rate this data example from 1 to
4 based on the comparison between the generated description and the image. A
score of 1 or 2 indicates that the image is difficult to describe, making it
more suitable for I2L. Conversely, a score of 3 or 4 suggests that the image
is easy to describe, making VT-ICL more suitable for this data example. Figure
2 provides an overview of the selection process. The detailed prompt for
GPT-4V-Selector can be found in Figure 9 in the Appendix.

##  4 Experiment

We evaluate the proposed I2L on multimodal tasks to demonstrate its
effectiveness in enhancing the model’s comprehension and extraction of
knowledge from image-only instances. The following sections will detail the
methods used to create in-image demonstrations and the results of our
experiments on multimodal tasks.

###  4.1 Experimental Setup

Datasets. We use three benchmark datasets for multimodal tasks: MathVista Lu
et al. (2023), HallusionBench Liu et al. (2023a), and VQA Goyal et al. (2017).
MathVista comprises 6,141 instances from 28 multimodal datasets, including new
datasets like IQTest, FunctionQA, and PaperQA. Mastering these tasks requires
understanding complex visual details and compositional reasoning, posing
challenges for advanced models. HallusionBench explores “Language
Hallucination” and “Visual Illusion” errors in vision-language models. VQA
contains open-ended questions about images, demanding understanding of vision,
language, and common sense. For VQA, we randomly select 50 examples from three
subsets: VQA(YorN), VQA(Counting), and VQA(others).

###  4.2 Demonstration Construction

####  4.2.1 The proposed I2L

GPT-4V shows a strong understanding of visual cues, such as symbols, numbers,
and contextual text, especially within specific regions Yang et al. (2023b).
To improve GPT-4V’s ability to understand and extract information from purely
visual instances, this work proposes I2L that combines visual-text
demonstrations, visual cues, and instructional content into a single image. In
the following sections, we’ll explain the techniques and processes used to
create in-image demonstrations, using specific examples from HallusionBench.

Figure 4 illustrates the methodology employed to construct in-image
demonstrations specifically tailored for the HallusionBench dataset. Given the
diverse nature of tasks within HallusionBench, it becomes imperative to create
demonstrations for each task. This is accomplished through a process of
decomposition, wherein we extract key visual components and subsequently
incorporate visual cues and textual demonstrations. In the provided example,
we present separate images featuring Xuanzang’s three disciples, namely
Wujing, Wukong, and Bajie, within the demonstration example section, each
delineated by visual cue boxes. Additionally, a task instruction is
prominently positioned at the top of the image. The inclusion of this task
instruction is of paramount importance as it serves as a crucial guide for
Visual Language Models (VLMs) to generate predictions in the expected format.
Figure 5 and 6 illustrate the method used to construct in-image demonstrations
for MathVista and VQA datasets. It is worth noting that our approach differs
from recent works Yang et al. (2023a, b), we exclusively incorporate visual
cues into the demonstration examples, rather than applying them to test
examples. This distinction is significant, as it necessitates the model to
learn from the demonstration examples in order to successfully tackle the test
examples.

Model | ALL | FQA | GPS | MWP | TQA | VQA | ALG | ARI | GEO | LOG | NUM | SCI | STA
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Heuristics baselines |
Random chance | 17.9 | 18.2 | 21.6 | 3.8 | 19.6 | 26.3 | 21.7 | 14.7 | 20.1 | 13.5 | 8.3 | 17.2 | 16.3
Frequent guess | 26.3 | 22.7 | 34.1 | 20.4 | 31.0 | 24.6 | 33.1 | 18.7 | 31.4 | 24.3 | 19.4 | 32.0 | 20.9
LLMs (Input: $x^{\text{txt}}$) |
Zero-shot ChatGPT OpenAI (2022) | 23.5 | 21.9 | 26.9 | 9.1 | 38.6 | 23.5 | 27.7 | 15.9 | 25.7 | 21.6 | 9.9 | 41.5 | 20.5
Zero-shot GPT-4 OpenAI (2023a) | 26.1 | 22.3 | 37.0 | 7.0 | 39.2 | 27.4 | 33.6 | 17.4 | 35.6 | 16.2 | 9.2 | 45.8 | 19.5
Zero-shot Claude-2 Anthropic (2023) | 26.4 | 21.9 | 34.1 | 13.4 | 36.1 | 29.1 | 32.8 | 20.4 | 33.3 | 13.5 | 12.1 | 36.4 | 20.5
2-shot CoT Claude-2 Wei et al. (2022) | 24.4 | 18.6 | 29.8 | 9.7 | 33.5 | 34.1 | 29.2 | 19.0 | 28.0 | 5.4 | 13.9 | 36.9 | 18.9
2-shot CoT ChatGPT Wei et al. (2022) | 26.8 | 20.1 | 36.5 | 8.6 | 44.9 | 28.5 | 35.6 | 17.0 | 33.5 | 21.6 | 14.6 | 45.9 | 17.9
2-shot CoT GPT-4 Wei et al. (2022) | 29.2 | 20.1 | 44.7 | 8.6 | 46.2 | 31.3 | 41.6 | 19.3 | 41.0 | 18.9 | 13.9 | 47.5 | 18.9
2-shot PoT ChatGPT Chen et al. (2022) | 25.1 | 19.0 | 30.8 | 16.1 | 38.0 | 25.7 | 29.9 | 19.8 | 29.3 | 24.3 | 19.4 | 38.5 | 16.9
2-shot PoT GPT-4 Chen et al. (2022) | 26.0 | 20.1 | 33.2 | 8.1 | 44.9 | 28.5 | 32.7 | 16.7 | 31.0 | 24.3 | 13.2 | 48.4 | 18.3
Augmented-LLMs (Input: $x^{\text{txt}}$, $x^{\text{cap}}$) |
2-shot CoT Claude-2 Wei et al. (2022) | 33.2 | 26.0 | 31.7 | 35.5 | 48.1 | 30.2 | 32.4 | 32.3 | 33.0 | 16.2 | 17.4 | 54.9 | 36.2
2-shot CoT ChatGPT Wei et al. (2022) | 33.2 | 27.5 | 29.3 | 36.0 | 49.4 | 29.1 | 31.0 | 32.9 | 31.0 | 16.2 | 17.4 | 50.8 | 37.2
2-shot CoT GPT-4 Wei et al. (2022) | 33.2 | 27.9 | 31.7 | 31.2 | 51.9 | 28.5 | 33.5 | 30.9 | 32.2 | 13.5 | 12.5 | 58.2 | 37.9
2-shot PoT ChatGPT Chen et al. (2022) | 26.8 | 24.5 | 26.4 | 23.7 | 33.5 | 27.9 | 27.8 | 26.1 | 28.0 | 18.9 | 13.2 | 33.6 | 29.9
2-shot PoT GPT-4 Chen et al. (2022) | 33.9 | 30.1 | 39.4 | 30.6 | 39.9 | 31.3 | 37.4 | 31.7 | 41.0 | 18.9 | 20.1 | 44.3 | 37.9
LMMs (Input: $x^{\text{txt}}$, $x^{\text{img}}$) |
IDEFICS-9B-Instruct Laurençon et al. (2023) | 19.8 | 21.6 | 21.1 | 6.5 | 25.9 | 24.0 | 22.1 | 15.0 | 19.8 | 18.9 | 9.9 | 24.6 | 18.1
mPLUG-Owl-LLaMA-7B Ye et al. (2023a) | 22.2 | 22.7 | 23.6 | 10.2 | 27.2 | 27.9 | 23.6 | 19.2 | 23.9 | 13.5 | 12.7 | 26.3 | 21.4
miniGPT-4-LLaMA-2-7B Zhu et al. (2023) | 23.1 | 18.6 | 26.0 | 13.4 | 30.4 | 30.2 | 28.1 | 21.0 | 24.7 | 16.2 | 16.7 | 25.4 | 17.9
LLaMA-Adapter-V2-7B Zhang et al. (2023a) | 23.9 | 21.2 | 25.5 | 11.3 | 32.3 | 31.8 | 26.3 | 20.4 | 24.3 | 24.3 | 13.9 | 29.5 | 18.3
LLaVAR Zhang et al. (2023d) | 25.2 | 21.9 | 25.0 | 16.7 | 34.8 | 30.7 | 24.2 | 22.1 | 23.0 | 13.5 | 15.3 | 42.6 | 21.9
InstructBLIP-Vicuna-7B Dai et al. (2023) | 25.3 | 23.1 | 20.7 | 18.3 | 32.3 | 35.2 | 21.8 | 27.1 | 20.7 | 18.9 | 20.4 | 33.0 | 23.1
LLaVA-LLaMA-2-13B Liu et al. (2023d) | 26.1 | 26.8 | 29.3 | 16.1 | 32.3 | 26.3 | 27.3 | 20.1 | 28.8 | 24.3 | 18.3 | 37.3 | 25.1
Multimodal Bard Google (2023) | 34.8 | 26.0 | 47.1 | 29.6 | 48.7 | 26.8 | 46.5 | 28.6 | 47.8 | 13.5 | 14.9 | 47.5 | 33.0
GPT-4V (Playground) OpenAI (2023b) | 49.9 | 43.1 | 50.5 | 57.5 | 65.2 | 38.0 | 53.0 | 49.0 | 51.0 | 21.6 | 20.1 | 63.1 | 55.8
Our Implementation (GPT-4V) |
1-shot T-ICL-Img | 49.1 | 45.3 | 49.0 | 56.4 | 61.7 | 36.8 | 49.8 | 47.3 | 50.2 | 21.6 | 27.7 | 63.8 | 57.5
1-shot VT-ICL | 51.6 | 49.8 | 42.3 | 60.7 | 65.6 | 44.1 | 46.9 | 53.2 | 43.9 | 24.3 | 29.1 | 67.2 | 61.2
1-shot VT-ICL w/ cues | 51.6 | 50.1 | 48.5 | 56.9 | 65.8 | 39.4 | 51.6 | 48.3 | 48.9 | 27.0 | 25.6 | 67.5 | 61.2
1-shot I2L | 51.5 | 49.6 | 40.8 | 58.0 | 67.7 | 45.8 | 46.9 | 50.9 | 43.9 | 29.7 | 32.6 | 65.2 | 59.0
1-shot ICL-Mix (GPT-4V-Selector) | 52.9 | 52.0 | 50.4 | 58.0 | 65.8 | 40.7 | 53.3 | 49.3 | 50.6 | 29.7 | 25.7 | 69.1 | 62.8
Human |
Human performance | 60.3 | 59.7 | 48.4 | 73.0 | 63.2 | 55.9 | 50.9 | 59.2 | 51.4 | 40.7 | 53.8 | 64.9 | 63.9

Table 1: Accuracy on the testmini subset of MathVista. ALL: overall accuracy.
Task types: FQA: figure question answering, GPS: geometry problem solving,
MWP: math word problem, TQA: textbook question answering, VQA: visual question
answering. Mathematical reasoning types: ALG: algebraic reasoning, ARI:
arithmetic reasoning, GEO: geometry reasoning, LOG: logical reasoning, NUM:
numeric commonsense, SCI: scientific reasoning, STA: statistical reasoning.
The highest scores across all models are bolded.

####  4.2.2 Baseline: T-ICL-Img

In prior research (Yang et al., 2022; Guo et al., 2022), a common approach
involved converting images into textual descriptions through the use of image-
to-text models. This conversion process aimed to adapt LLMs originally
designed for NLP tasks to the demands of multimodal tasks. In accordance with
this strategy, we have employed GPT-4V to generate textual descriptions for
both the demonstration images and test images. Figures 7 illustrate the
incorporation of ICL along with additional image-to-text examples, pertaining
to the HallusionBench dataset. In our evaluation process, we exclusively
utilize text-only inputs to assess the performance of GPT-4.

####  4.2.3 Baseline: VT-ICL

Despite the promising outcomes observed in the context of in-context learning
with supplementary image-to-text models, there remains a noteworthy concern
regarding the potential loss of information during the transformation of
visual inputs into textual descriptions for the given task, as discussed in
previous studies (Yang et al., 2022; Hu et al., 2022). Consequently, we adopt
an approach that takes both images and textual demonstrations as input,
thereby allowing the model to leverage both language and visual information in
order to address multimodal tasks effectively. Figures 8 illustrate the ICL
with interleaved image-text demonstration examples, as applied to the
HallusionBench dataset.

###  4.3 Results

| Yes/No Bias | Consistency | Language and Vision Diagnosis
---|---|---|---
Method | Evaluation | Pct. Diff | FP Ratio | Correct | Inconsistent | Wrong | Lang. Hallus. | Visual Illus. | Mixed
Random Chance | GPT-4-Assisted | 0.08 | 0.57 | 18.20 | 57.51 | 24.28 | - | - | -
LLaVA-1.5 Liu et al. (2023c) | Human | 0.27 | 0.76 | 25.43 | 42.49 | 32.08 | 25.63 | 51.42 | 22.95
GPT-4-Assisted | 0.26 | 0.75 | 24.86 | 45.38 | 29.77 | 26.71 | 51.09 | 22.20
BLIP2-T5 Li et al. (2023b) | GPT-4-Assisted | 0.08 | 0.58 | 20.52 | 59.54 | 19.94 | 41.64 | 40.44 | 17.92
Qwen-VL Bai et al. (2023) | GPT-4-Assisted | 0.12 | 0.60 | 6.65 | 50.29 | 43.06 | 0.87 | 88.06 | 11.06
Open-Flamingo Awadalla et al. (2023) | GPT-4-Assisted | 0.33 | 0.77 | 11.27 | 59.83 | 28.90 | 30.07 | 48.06 | 21.87
MiniGPT5 Zheng et al. (2023) | GPT-4-Assisted | 0.28 | 0.71 | 9.83 | 56.36 | 33.82 | 10.09 | 73.44 | 16.47
MiniGPT-4 Zhu et al. (2023) | GPT-4-Assisted | 0.19 | 0.65 | 10.12 | 57.80 | 32.08 | 23.59 | 56.55 | 19.86
InstructBLIP Dai et al. (2023) | GPT-4-Assisted | -0.13 | 0.38 | 10.12 | 68.50 | 21.39 | 29.29 | 54.53 | 16.18
BLIP2 Li et al. (2023b) | GPT-4-Assisted | 0.18 | 0.65 | 12.43 | 63.01 | 24.57 | 39.14 | 43.45 | 17.41
mPLUG_Owl-v2 Ye et al. (2023b) | GPT-4-Assisted | 0.25 | 0.77 | 19.94 | 58.09 | 21.97 | 28.24 | 50.42 | 21.34
mPLUG_Owl-v1 Ye et al. (2023a) | GPT-4-Assisted | 0.32 | 0.79 | 10.40 | 60.12 | 29.48 | 3.95 | 78.36 | 17.69
LRV_Instruction Liu et al. (2023b) | GPT-4-Assisted | 0.26 | 0.73 | 13.01 | 53.47 | 33.53 | 4.49 | 76.47 | 19.04
GIT Wang et al. (2022) | GPT-4-Assisted | 0.04 | 0.53 | 6.36 | 53.76 | 39.88 | 30.90 | 58.30 | 10.80
GPT-4V OpenAI (2023b) | Human | 0.07 | 0.60 | 44.22 | 32.66 | 23.12 | 21.86 | 46.17 | 31.97
GPT-4-Assisted | 0.06 | 0.58 | 39.88 | 38.15 | 21.97 | 22.19 | 45.66 | 32.14
1-shot T-ICL (GPT-4V) | Human | -0.02 | 0.33 | 24.63 | 53.62 | 26.73 | 49.01 | 58.75 | 56.39
GPT-4-Assisted | -0.03 | 0.31 | 24.63 | 57.97 | 17.39 | 54.90 | 60.00 | 58.76
1-shot VT-ICL (GPT-4V) | Human | -0.10 | 0.26 | 20.28 | 66.66 | 13.04 | 49.01 | 59.37 | 56.87
GPT-4-Assisted | -0.10 | 0.27 | 20.28 | 66.66 | 13.04 | 49.01 | 58.12 | 55.92
1-shot I2L (GPT-4V) | Human | -0.13 | 0.23 | 21.73 | 65.21 | 13.04 | 62.74 | 56.25 | 57.81
GPT-4-Assisted | -0.13 | 0.23 | 20.28 | 68.11 | 11.59 | 62.74 | 56.25 | 57.81

Table 2: Analytical Evaluation Results on HallusionBench with various LVLMs:
Pct. Diff ranges from [-1, 1]. The model is more biased when Pct. Diff is
close to -1 or 1. FP Ratio ranges from [0, 1]. The model is more robust when
FP Ratio is close to 0.5. All the other metrics are presented in %, and the
full score is 100%. We highlight the Top 3 models with the GPT-4-assisted
evaluation.  Figure 3: In-depth analysis for I2L: (a): the impact of
resolution ratio. (b): the impact of number of shots. (c): the impact of
example placing in a image. T2B is top-to-bottom placement. L2R is left-to-
right placement. T2B-Rot is top-to-bottom placement and $45^{\circ}$
counterclockwise rotation. Rand is random placement.

Method | Acc. |
---|---|---
I2L w/o chain-of-thought | 49.0 |
I2L w/o visual cues | 49.0 |
I2L | 57.0 |

Table 3: An ablation study of the impact of different components in the
proposed I2L on a subset of 100 data examples from MathVista.

| VQA(YorN) | VQA(Counting) | VQA(others) | Average
---|---|---|---|---
T-ICL-Img | 0.64 | 0.46 | 0.44 | 0.51
VT-ICL | 0.78 | 0.56 | 0.68 | 0.64
I2L | 0.76 | 0.66 | 0.74 | 0.72

Table 4: In-image learning: we test on datasets with different question types.
For VQA, we test with specific counting questions, yes-or-no questions, and
random sampling without specific questions.

####  4.3.1 MathVista

Table 1 shows the performance results of various models on the MathVista
minitest dataset. All our implementations use a one-shot evaluation, designed
to compare the effectiveness of different in-context learning paradigms.
Notably, I2L achieves an average accuracy of 51.5% on MathVista, almost
matching VT-ICL’s 51.6%. On the other hand, T-ICL-Img, which solely relies on
textual input from GPT-4V, attains a lower average accuracy of 49.1%. This
highlights the potential information loss during the caption generation
process for T-ICL-Img, even when using the GPT-4V model.

Performance variation across different MathVista subsets is apparent. T-ICL-
Img shows superior performance in the GPS and GEO subsets. VT-ICL excels in
the FQA, ALG, SCI, and STA subsets. In contrast. I2L outperforms other methods
in the MWP, TQA, VQA, ARI, LOG, and NUM subsets. The efficacy of Integrated
Information Integration Learning (I2L) in harnessing comprehensive visual data
from individual images contributes significantly to the superior performance
attained by GPT4-V in subsets characterized by challenges in textually
articulating visual content. The proposed Mixed method, which integrates
elements of I2L and VT-ICL, produces promising results. Using GPT-4V for
selection, ICL-Mix achieves the highest average accuracy of 52.9% on
MathVista. However, it’s worth noting that the I2L model’s performance
declines specifically within the GPS subset of MathVista. This drop is due to
vision-large language models’ reliance on textual information for solving
geometric problems, reinforcing the need for the Mixed method of I2L and VT-
ICL.

####  4.3.2 HallusionBench

We adhere to the methodology outlined in HallusionBench Liu et al. (2023a) for
the evaluation of LLMs across various assessment metrics, including the Yes/No
bias test, Consistency test, and Language and Vision Diagnosis. The results of
these evaluations are summarized in Table 2. Notably, both human evaluation
and GPT-4-Assisted evaluation metrics are reported, in accordance with the
protocol established by Liu et al. (2023a). Upon examination of the table, it
becomes apparent that the GPT-4 model continues to exhibit bias in automatic
evaluations, as indicated by discrepancies between the GPT-4-Assisted
evaluation and human evaluation results. Consequently, our analysis primarily
focuses on the human evaluation outcomes, with GPT-4-Assisted results serving
as supplementary evidence.

In the context of the Yes/No bias test, T-ICL achieves the most favorable Yes
Percentage Difference (Pct. Diff) score of -0.02, while I2L demonstrates the
lowest False Positive Ratio (FP Ratio) of 0.23, as evidenced by both human
evaluation and GPT-4-Assisted evaluation. In the Consistency test, T-ICL
attains the highest Correct and Inconsistent score at 24.63% and 53.62%,
respectively, while I2L achieves the superior Wrong score at 13.04%. Of
particular note is the performance of the proposed I2L model in the Language
and Vision Diagnosis, wherein it surpasses other models with a Mixed score of
57.81%. This outcome signifies that I2L exhibits efficacy in mitigating
language hallucination and visual illusion, thereby enhancing diagnostic
accuracy in multimodal contexts.

####  4.3.3 VQA

In order to assess the general vision understanding capabilities of the
proposed I2L method, we conducted experiments on the Visual Question Answering
(VQA) dataset. Table 4 provides an overview of the results obtained by I2L,
T-ICL-Img, and VT-ICL on the VQA dataset. Notably, the I2L method surpasses
T-ICL-Img and VT-ICL, achieving an average accuracy of 72%. This outcome
underscores the superior ability of I2L to enhance the overall vision
understanding capabilities of the models.

Additionally, we performed experiments to investigate the influence of various
factors, including resolution ratio, number of shots, and the positioning of
demonstration and test examples. Figure 3 (a) illustrates the performance
variation in response to changes in resolution ratio. Notably, the model’s
performance tends to decline as the resolution ratio increases or decreases.
Figure 3 (b) demonstrates the impact of the number of demonstration examples
on performance, revealing that the model achieves optimal performance with two
demonstration examples. Furthermore, Figure 3 (c) explores the performance
variation based on the positioning of demonstration and test examples,
indicating that the model performs most effectively when the demonstration and
test examples are arranged from top to bottom.

####  4.3.4 Ablation Study

In this section, we conduct an ablation study on the proposed I2L using a
subset of 100 data examples from MathVista. Table 3 demonstrates that adding
chain-of-thought as the target in the image demonstration and incorporating
visual cues in the image both contribute to the final performance. This
suggests that in-image learning is flexible and has the potential to enhance
GPT-4V’s ability by incorporating additional useful and relevant information
into a single image.

##  5 Conclusion

In this paper, we propose a novel approach called In-Image Learning to enhance
the capabilities of GPT-4V. In-Image Learning combines demonstration examples,
visual cues, and instructions into a single image, providing an in-context
learning experience. In-Image Learning excels at handling complex images,
while interleaved visual-text in-context learning is better suited for images
that can be easily described by text. To leverage the strengths of both
methods for multimodal tasks, we propose using GPT-4V as a selector to
determine the appropriate method for each multimodal data example in a given
task. Through comprehensive experiments on MathVista and Hallusionbench, we
demonstrate the effectiveness of our proposed method in complex reasoning
tasks and in mitigating language hallucination and visual illusions. We also
examine the impact of factors such as image resolution and the positioning of
demonstration examples, further highlighting the versatility and potential of
in-image learning.

##  6 Limitation

This paper presents In-Image Learning as a means to enhance the capabilities
of GPT-4V for multimodal tasks. One limitation of the proposed method is its
sensitivity to the position of demonstrations. Additionally, we did not
conduct experiments on open-source large multimodal models. In future work, we
will explore methods to reduce the sensitivity to position in image
demonstrations and investigate the implementation of in-image learning on more
open-source large multimodal models.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736.
  * Anthropic (2023) Anthropic. 2023.  Claude 2.
  * Awadalla et al. (2023) Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _arXiv preprint arXiv:2308.01390_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  _arXiv preprint arXiv:2211.12588_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  InstructBLIP: Towards general-purpose vision-language models with instruction tuning.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * Google (2023) Google. 2023.  Bard.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Guo et al. (2022) Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. 2022.  From images to textual prompts: Zero-shot vqa with frozen large language models.  _arXiv preprint arXiv:2212.10846_.
  * He et al. (2023) Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. 2023.  Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction.  _arXiv preprint arXiv:2303.05063_.
  * Hu et al. (2022) Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022.  Promptcap: Prompt-guided task-aware image captioning.  _arXiv preprint arXiv:2211.09699_.
  * Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023.  Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.  _arXiv preprint arXiv:2310.06839_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023.  OBELICS: An open web-scale filtered dataset of interleaved image-text documents.
  * Li et al. (2023a) Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Liu et al. (2023a) Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023a.  Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models.  _arXiv preprint arXiv:2310.14566_.
  * Liu et al. (2023b) Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023b.  Aligning large multi-modal model with robust instruction tuning.  _arXiv preprint arXiv:2306.14565_.
  * Liu et al. (2023c) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023c.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_.
  * Liu et al. (2023d) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023d.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021.  What makes good in-context examples for gpt-$3$?  _arXiv preprint arXiv:2101.06804_.
  * Liu et al. (2023e) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023e.  Lost in the middle: How language models use long contexts.  _arXiv preprint arXiv:2307.03172_.
  * Lu et al. (2023) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  _arXiv preprint arXiv:2310.02255_.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _arXiv preprint arXiv:2104.08786_.
  * Madaan and Yazdanbakhsh (2022) Aman Madaan and Amir Yazdanbakhsh. 2022.  Text and patterns: For effective chain of thought, it takes two to tango.  _arXiv preprint arXiv:2209.07686_.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.  https://openai.com/blog/chatgpt.
  * OpenAI (2023a) OpenAI. 2023a.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * OpenAI (2023b) OpenAI. 2023b.  Gpt-4v(ision) system card.
  * Shi et al. (2022) Peng Shi, Rui Zhang, He Bai, and Jimmy Lin. 2022.  Xricl: Cross-lingual retrieval-augmented in-context learning for cross-lingual text-to-sql semantic parsing.  _arXiv preprint arXiv:2210.13693_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2022) Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022.  Git: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_.
  * Wang et al. (2023) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  _arXiv preprint arXiv:2305.04091_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Yang et al. (2023a) Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023a.  Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.  _arXiv preprint arXiv:2310.11441_.
  * Yang et al. (2022) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022.  An empirical study of gpt-3 for few-shot knowledge-based vqa.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 3081–3089.
  * Yang et al. (2023b) Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023b.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  _arXiv preprint arXiv:2309.17421_ , 9.
  * Ye et al. (2023a) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023a.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_.
  * Ye et al. (2023b) Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023b.  mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.  _arXiv preprint arXiv:2311.04257_.
  * Yoo et al. (2022) Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022.  Ground-truth labels matter: A deeper look into input-label demonstrations.  _arXiv preprint arXiv:2205.12685_.
  * Zhang et al. (2023a) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023a.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_.
  * Zhang et al. (2022a) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhang et al. (2023b) Xiang Zhang, Senyu Li, Zijun Wu, and Ning Shi. 2023b.  Lost in translation: When gpt-4v (ision) can’t see eye to eye with text. a vision-language-consistency analysis of vllms and beyond.  _arXiv preprint arXiv:2310.12520_.
  * Zhang et al. (2023c) Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. 2023c.  Gpt-4v (ision) as a generalist evaluator for vision-language tasks.  _arXiv preprint arXiv:2311.01361_.
  * Zhang et al. (2023d) Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023d.  LLaVAR: Enhanced visual instruction tuning for text-rich image understanding.  _arXiv preprint arXiv:2306.17107_.
  * Zhang et al. (2022b) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022b.  Automatic chain of thought prompting in large language models.  _arXiv preprint arXiv:2210.03493_.
  * Zheng et al. (2023) Kaizhi Zheng, Xuehai He, and Xin Eric Wang. 2023.  Minigpt-5: Interleaved vision-and-language generation via generative vokens.  _arXiv preprint arXiv:2310.02239_.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.

##  Appendix A Appendix

The demonstration construction examples for I2L, T-ICL-Img, and VT-ICL for
different datasets including MathVista, HallusionBench, and VQA.

Figure 4:  Example of I2L for HallusionBench. With adding visual cues of
different character images and corresponding names, the GPT4-V model is able
to distinguish the characters and generated the correct answer. Figure 5:
Example of I2L for MathVista. GPT4-V is not able to generate the correct
answer without demonstrate. With adding visual cues and Chain-of-Thought
rationales on the demonstration example, GTP4-V is able learn from the
demonstration to solve the test example, even there is no additional
information for the test example. Figure 6:  Example of I2L for the counting
subset of VQA dataset. Counting questions are challenging for vision-language
models even for GPT4-V. We add bounding box as the visual cues to locate
objects in demonstration images and also the Chain-of-Thought rationales. With
the visual cues and Chain-of-Thought rationales, GPT4-V is able to generate
the correct answer for the test examples. Figure 7: The prompt used for T-ICL-
Img. We use GPT4-V to generate the captions for both demonstration and test
images. Figure 8:  The prompt template of VT-ICL. We also add the visual cues
and Chain-of-Thought rationales to enhance the capabilities of VT-ICL for a
stronger baseline. Figure 9: Example of GPT-4V-Selector for ICL method
selection. In this example, since GPT-4V outputs a ranking score of 1, the I2L
method will be chosen. Figure 10: Example of GPT-4V-Selector for ICL method
selection. In this example, since GPT-4V outputs a ranking score of 2, the VT-
ICL method will be chosen.