#  Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using
Self-Imagination

Syeda Nahida Akter1, Aman Madaan1, Sangwu Lee2, Yiming Yang1, Eric Nyberg1
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA,
United States1
Department of Computer Science, University of Rochester, Rochester, NY, United
States2
<EMAIL_ADDRESS>
###### Abstract

The potential of Vision-Language Models (vlms) often remains underutilized in
handling complex text-based problems, particularly when these problems could
benefit from visual representation. Resonating with humans’ ability to solve
complex text-based problems by (1) creating a visual diagram from the problem
and (2) deducing what steps they need to take to solve it, we propose Self-
Imagine. We leverage a single Vision-Language Model (vlm) to generate a
structured representation of the question using HTML, then render the HTML as
an image, and finally use the same vlm to answer the question using both the
question and the image. Our approach does not require any additional training
data or training. We evaluate our approach in three mathematics tasks and nine
general-purpose reasoning tasks using state-of-the-art vlm. Our approach
boosts the performance of vlm on all math tasks (gsm8k: +4.62%; asdiv: +4.49%;
svamp: +9.30%) and the majority of the general-purpose reasoning tasks by 0.4%
to 13.20% while achieving comparable performance in other tasks111Code and
data at https://github.com/snat1505027/self-imagine.

##  1 Introduction

Figure 1: Generating an image from a question via a single VLM through HTML.

Vision Language Models (vlm) are getting increasingly adept at solving a wide
range of reasoning tasks (Liu et al., 2023a; b; You et al., 2023; Ye et al.,
2023; Chen et al., 2023b; Zhang et al., 2023; Chen et al., 2023a; Dai et al.,
2023; Lu et al., 2023). As these capabilities advance, vlms are set to replace
the current text-only language models for general-purpose interfaces like BARD
(GoogleAI, 2023) and ChatGPT (OpenAI, 2021). In such scenarios, the deployed
vlm would be required to handle a wide variety of end-user queries. Crucially,
this includes queries that are not inherently multimodal, such as math-
reasoning problems or program synthesis (Cobbe et al., 2021).

A key question arises in these situations: How should a vlm, capable of
functioning in a text-only mode like a Language Language Model (llm), handle
text-based queries? While the default approach is to process these queries
purely as text, this method does not fully exploit the vlm’s capabilities in
image processing. Recent studies on human problem-solving provide a clue to
addressing this gap: humans often draw visual representations to better
understand and solve problems (Boonen et al., 2014; van Garderen et al., 2012;
Krawec, 2014).

Building on this insight, we propose Self-Imagine–a technique designed to
enhance the reasoning abilities of vlms on text-only tasks through
visualization (Figure 1). Self-Imagine initially generates a graphical
representation of the text query using the vlm. Then, the same vlm is used to
solve the problem using both the original question and the self-generated
image.

One inherent challenge is that advanced vlms are not typically equipped for
direct image generation. To circumvent this, we utilize the vlm’s code
generation capabilities to generate HTML code visually representing the query
information. This HTML is then rendered as an image, which, when used in
conjunction with the original text query, allows the vlm to operate with both
textual and visual information. With Self-Imagine, the vlm efficiently serves
dual purposes: generating visual representations and solving the problem. This
strategy effectively reduces reliance on separate image generation models such
as DALL-E (Shi et al., 2020), streamlining the problem-solving process.

We test our approach across three mathematical reasoning tasks and nine
general-purpose reasoning tasks. We find that our approach is particularly
effective when the generated image demonstrates the information in a
structured way that corresponds to the reasoning steps needed to be performed
to solve the question. We show that our approach improves the performance of
all mathematics tasks namely gsm8k (Cobbe et al., 2021) (+4.62%), asdiv (Miao
et al., 2020) (+4.49%) and svamp (Patel et al., 2021) (+9.30%), and achieves
superior performance (ranging from 0.40% to 13.2% improvement) in five out of
nine general-purpose reasoning tasks while receiving comparable accuracy to
question only setup in other tasks.

##  2 Self-Imagine

Figure 2: [Left] Reasoning using vlm without Self-Imagine: Given a question
(0), the vlm generates an answer (1). [Right] Reasoning using vlm with Self-
Imagine: Given a question (0), the vlm generates a structured representation
of the question using HTML (1). The HTML is rendered as an image (2) which is
then passed along with the question to the vlm again (3). The vlm finally
generates the answer by combining both vision and language modalities (4).

Unlike Large Language Models (llm), Vision Language Models (vlm) can combine
multiple modalities in the same representation space and perform complex
reasoning. However, when it comes to unimodal downstream tasks (e.g., math-
reasoning), vlms are not fully leveraged due to the absence of additional
modalities. In Self-Imagine, we circumvent this by generating a visual
representation for a given reasoning question using the vlm in the form of an
image. Then, the same vlm is fed both the question and the generated image to
answer the question. In the following section, we expand on the image
generation from the question.

###  2.1 Generate Image from Question

While vlm cannot generate images directly, they are pre-trained on large
corpus of programs and thus are proficient in code generation. Thus, we
utilize code generation capabilities of these models to create an image for
the question. While there are several choices for choosing a representation
(SVG (St.Laurent et al., 2001), Tikz (Tantau, 2022)), we use HTML due to its
prevalence and its ability to easily generate structured information present
in the question using tables, lists, flow charts, etc.

#### Generate HTML from Question.

To convert natural language questions into HTML, we use a Large Multimodal
Model, specifically LLaVA-1.5 (Liu et al., 2023a), as our Vision Language
Model (vlm) due to its impressive performance on a wide range of reasoning
tasks. Since multimodal models are not traditionally trained for HTML
generation, we approach this using a few-shot prompt, interleaving natural
language questions with HTML codes. For each natural language question
$q_{i}$, we generate a corresponding HTML code $h_{i}$. These are paired as
$\langle q_{i},h_{i}\rangle$ to form a prompt $p=\\{q_{j},h_{j}\\}_{j=1}^{K}$,
where $K=5$ represents the number of in-context examples chosen for diversity
in reasoning tasks. Given a new question $q_{t}$, we combine it with the
prompt $p$ and a placeholder image $I_{d}$, and input these into the vlm to
generate the HTML $h_{t}$ for $q_{t}$ as shown in Equation 1.

$h_{t}=\textsc{vlm}(p\,||\,q_{t},I_{d})$ (1)

#### Convert HTML to Image.

After generating HTML from questions, we use the ‘imgkit’ API to render these
HTML codes into images. To evaluate the role of images in reasoning tasks, we
conduct experiments both with and without the generated images. We append
task-specific prompts to the questions, as detailed in Table 3. In the image-
inclusive experiments, we use the HTML-generated images alongside the
concatenated prompts and questions, inputting these into the vlm for
processing.

$\begin{split}{I}_{g}=f(h_{t})\\\
y_{t}=\textsc{vlm}(p||q_{t},{I}_{g})\end{split}$ (2)

Here, $f$ represents the HTML renderer function, and $I_{g}$ represents the
final generated image from the question. $y_{t}$ is the answer generated using
the question with the prompt ($p||q_{t}$) and the image ($I_{g}$).

##  3 Experimental Setup

Figure 3: Self-Imagine main results: Self-Imagine improves accuracy over a
diverse range of mathematical and symbolic reasoning tasks.

#### Benchmarks.

We explore two kinds of reasoning tasks to evaluate our approach: (1) math
word problems consisting of gsm8k (Cobbe et al., 2021), asdiv (Miao et al.,
2020), and svamp (Patel et al., 2021) and (2) symbolic reasoning consisting of
Navigate, Geometric Shapes, Tracking Shuffled Objects, Penguins in a Table,
Colored Objects, Date Understanding, and Object Counting tasks from BIG-Bench
Hard (Suzgun et al., 2022).

#### Baselines.

For the baseline, we consider zero-shot prompting where we only pass a basic
prompt (Table 3) and the question. We performed greedy decoding from the
language model using a temperature of 0. Note that this is a realistic setup
for current open-source multimodal LLMs, which cannot accept a prompt
interleaved with text and images.

#### Vision Language Models.

We use LLaVA-1.5 (Liu et al., 2023a) as our vlm throughout the HTML generation
phase to the question-answering phase. LLaVA-1.5 uses a CLIP ViT-L (Radford et
al., 2021) as a vision encoder and Vicuna 13B (Chiang et al., 2023) as the
llm.

#### Evaluation

During the evaluation, we slightly modified the standard evaluation protocol
(Gao et al., 2023a), which consisted of matching the words “The answer is”
followed by a numerical output. We found that the vlm sometimes fails to
follow this sentence verbatim even when it produces the correct answer. To
accommodate these cases, we simply take the last number/option of the
generated text as the answer to the question.

##  4 Results

We summarize our results across three math and nine reasoning tasks in Table
1. We define the baseline setup as ‘Question Only’ when we only feed the
question with the basic prompt to the vlm. Self-Imagine is indicated by the
‘Question + Image’ setup where we generate the HTML from the question at first
and pass the rendered image from HTML along with the basic prompt and question
to the vlm as input (Equation 2).

Self-Imagine improves the vlm’s performance in all math reasoning tasks: gsm8k
(+4.62%), asdiv (+4.49%) and svamp (+9.30%), in arithmetic reasoning task,
Object Counting (+5.60%) and in four symbolic reasoning tasks: Navigate
(+13.2%), Colored Objects (+5.20%), Tracking Shuffled Objects of five objects
(+0.40%) and Tracking Shuffled Objects of seven objects (+4.80%). All these
tasks require diverse reasoning abilities, and the improvement across these
tasks represents the generalizability of Self-Imagine.

However, Self-Imagine hurts the performance of vlm in some of the symbolic
reasoning tasks, namely Date Understanding (-4.80%), Geometric Shapes
(-1.20%), and Tracking Shuffled Objects of three objects (-2.80%). These tasks
are easier to solve using only the question rather than having an image. The
reason behind degradation with an image is two-fold: (1) the generated images
are incorrect and visually not informative given the question (Date
Understanding, Geometric Shapes), (2) HTML cannot visually portray terms like
swap between objects (Tracking Shuffled Objects). For the rest of the task
(Penguins in a Table), the vlm performance is similar with and without image
setup. These results indicate that stronger image generation capabilities that
capture consecutive progression of reasoning might help to boost the
performance of the vlm.

In the following section, we demonstrate that the improvement is highly
correlated with the quality of the generated image, underscoring the
dependency on the ease of converting text into an image. In addition, an image
that appropriately captures the flow of reasoning always guides the vlm to the
correct reasoning path.

Task | Dataset | Accuracy (%)
---|---|---
Question Only | Question + Image
Math Reasoning | gsm8k | 26.69 |  31.31 (+4.62)
asdiv | 52.24 |  56.73 (+4.49)
svamp | 43.50 |  52.80 (+9.30)
Symbolic Reasoning | Object Counting | 31.20 |  36.80 (+5.60)
Navigate | 44.80 |  58.00 (+13.2)
Colored Objects | 44.80 |  50.00 (+5.20)
Date Understanding | 50.00 | 45.20 (-4.80)
Penguins in a Table | 41.10 | 41.10 (0.00)
Geometric Shapes | 14.40 | 13.20 (-1.20)
Tracking Shuffled Objects (3) | 33.60 | 30.80 (-2.80)
Tracking Shuffled Objects (5) | 18.00 |  18.40 (+0.40)
Tracking Shuffled Objects (7) | 12.00 |  16.80 (+4.80)

Table 1: Comparison of accuracy between ‘Question Only’ and ‘Question + Image’
across diverse reasoning tasks where the image has been generated using Self-
Imagine.

##  5 Analysis

###  5.1 Math Reasoning

Figure 4: Example from math world problem tasks.

For all math reasoning tasks, we analyze the performance of models with and
without image support. This analysis includes examining performance variations
based on question complexity, the length of the reasoning chain, and specific
instances where images contribute positively or negatively to problem-solving.
The generated images, as depicted in Figure 4, predominantly feature boxes,
each labeling a variable and its value, designed to simplify and clarify the
information presented in the question.

#### Why does image help?

The primary advantage of using images lies in their ability to distill complex
information into a more manageable format. In several tasks, particularly
those involving substantial irrelevant data (e.g., gsm8k, asdiv), an image
serves as a focused reference point, enabling the model to concentrate on key
variables and their values (see Table 2 for examples). Additionally, images
often include variable names marked with question marks, as shown in Figure 4,
which guide the model in identifying the critical elements necessary for
multi-step reasoning.

Table 2: Example of Image improving reasoning in gsm8k task.  Question |  Stephen placed an online order for groceries. His final bill came to $40.00. Because this was through a delivery vendor, they tacked on a 25% fee to his final total and charged him $3.00 in delivery fees. Stephen also added a $4.00 tip. After the extra fees, what was the final price of Stephen’s groceries?
---|---
Image |  Total$40.0025% Fee?Delivery Fee$3.00Tip$4.00Final Price?
Label |  57
Output With Image |  To find the final price of Stephen’s groceries, we need to calculate the total cost after the extra fees and tip have been added.
1\. Calculate the 25% fee: $40.00 (final bill) * 0.25 = $10.00 (25% fee) 2\.
Add the delivery fee to the final bill: $40.00 (final bill) + $10.00 (25% fee)
+ $3.00 (delivery fee) = $53.00 (total cost with fees) 3\. Add the tip to the
total cost: $53.00 (total cost with fees) + $4.00 (tip) = $57.00 (final price
of groceries)
The answer is $57.00.
Output Without Image |  The final price of Stephen’s groceries was $40.00 + 25% fee + $3.00 delivery fee + $4.00 tip = $40.00 + 0.25*40 + 3 + 4 = $40.00 + 10 + 3 + 4 = $40.00 + 13 = $53.00. The answer is $53.00.

#### Image helps solve moderately complex questions.

In general, longer questions tend to be complex. Here, we examine the
performance variation regarding question length as detailed in Figure 7. We
find that image helps in longer and more complex questions in asdiv and svamp
tasks. This finding also aligns with the previous explanation, i.e., the image
removes unnecessary verbose from the question, making the reasoning process
easier. However, we can also observe that for more complex questions in the
gsm8k task (question length more than 70), performance with images
deteriorates compared to performance without images. This decline stems from
the inadequate HTML generated by longer questions, which often fail to
encapsulate all the necessary information. Therefore, images generated from
those HTMLs confuse the vlm rather than help. This observation also holds for
questions with longer reasoning chains depicted in Figure 8 for the gsm8k
task. Questions that require a longer chain of thoughts are not better
represented with images. This analysis also presents an opportunity for future
research. It suggests that the most challenging questions, which intuitively
could benefit the most from the structural and contextual support provided by
images, are precisely where current methodologies for image generation fall
short.

#### Why does the image hurt?

While images generally enhance the vlm’s reasoning, specific scenarios lead to
diminished performance. A notable issue arises during HTML generation, where
the vlm occasionally pre-solves arithmetic sequences, embedding them into the
image (Table 4). This can mislead the model if the embedded calculations are
incorrect. Furthermore, certain concepts like ‘trade/exchange’ or ‘add/delete’
are challenging to represent visually, leading to inaccuracies in questions
involving these terms. Another complication involves questions with fractions,
such as ‘Shelly ate 3/4 of the eggs from a dozen.’ The corresponding images
often depict these fractions in a simplified form (e.g., a box labeled
‘Already ate: 3/4 $\times$ 12’), which the model struggles to compute
accurately as it requires the execution of multiple operations (i.e., division
and multiplication) simultaneously. Similarly, when the vlm tries to imitate
executing multiple operations mentioned in the image, it usuallyt generates
the incorrect answer. For example, in Table 5, with the image, the vlm
executes four operations in a single line (i.e., $10*1/2+15*1/3$) and ends up
generating the wrong answer. But without an image, the calculation is broken
down even further, producing the correct answer. This problem might be solved
with an improved image that breaks down each step as a single operation
consisting of two numbers.

###  5.2 Symbolic Reasoning

In this category, we focus on nine diverse reasoning sub-tasks from BIG-Bench
Hard benchmark (Suzgun et al., 2022) to observe the importance of image. We
break down the overall accuracy by sub-tasks and analyze the performance by
question complexity and answer types. The images generated with HTML for the
sub-tasks are images with labeled/colored boxes (5(b)), tables (5(a), 5(c)).
Occasionally, we find that the generated image simply contains the text (as in
Table 6).

(a) Object Counting

(b) Colored Objects

(c) Penguins in a Table

Figure 5: Examples from some BIG-Bench Hard sub-tasks.

#### Why and when does image help?

The overall accuracy indicates a decent improvement (2.38%) with the presence
of an image (as Figure 3). We further break down the results across the sub-
tasks. As shown in Figure 3, adding an image helps in Navigate, Colored
Objects, Object Counting, Tracking Shuffled Objects of five and seven objects
sub-tasks. In the Penguins in a Table task, the performance with the image is
somewhat similar to that without the image.

For Colored Objects and Object Counting tasks, the vlm generates well-
structured tables or multiple boxes in rows with variable names in one column
and corresponding values in another column. Therefore, while solving the
question with an image, the reasoning problem comes down to finding the
summation of the elements of a column.

On the other hand, Tracking Shuffled Objects task requires tracking multiple
objects after their consecutive swaps. As mentioned in subsection 5.1, vlm is
not good at depicting swaps between objects using HTML, which is the reason
behind the degradation in performance with the image in Tracking Shuffled
Objects of three objects task. However, Tracking Shuffled Objects of five and
seven objects tasks are notably more complex than the three objects task as
they require tracking more objects and their swaps. Having an image that logs
each object attribute and their swaps, helps the vlm to solve the question
rather than having no image. Hence, we can see the improvement with the
increasing number of variables for Tracking Shuffled Objects tasks.

Finally, for the Navigate task, we get significant improvement in the presence
of an image. The Navigate task is quite different than the other tasks and can
not be easily depicted using HTML. Therefore, most of the images for this task
contain texts either showing the question or showing the necessary reasoning
steps in natural language (Table 6). Without an image, the vlm performs poorly
on this task. However, with the inclusion of images, the vlm executes
additional reasoning during HTML generation, thereby increasing the likelihood
of accurately predicting the answer in the presence of an image.

#### Image helps with shorter and more complex questions.

Following subsection 5.1, we investigate the impact of the image in the
reasoning process with increasing question length. As shown in Figure 9, the
image helps when the corresponding question is simpler and shorter and when
the questions are more complex. Generating high-quality HTML is also easier
for simpler and shorter questions, which benefits the vlm during question
answering with the appropriate image. However, with increasing question
length, the generated HTMLs tend to ignore some of the information or can not
summarize all information in a structured manner. This results in lower
performance compared to without image setup. In contrast, Figure 9 shows
higher performance for with-image setup when the question length exceeds 120
words. This category includes the Tracking Shuffled Objects of seven objects
task, which requires tracking seven objects during multiple swaps. As
mentioned earlier, the image depicts the role of each object in the question
and their swaps, which simplifies the reasoning process.

#### Why does the image hurt?

Apart from tasks that benefit from having an image, some tasks also perform
worse after incorporating an image into the reasoning process. We observe that
the reason behind the performance drop-off is two-fold: (1) images generated
from HTML are incorrect or missing information, and (2) generated images
cannot depict the reasoning process.

As mentioned in the previous sections, vlm is not good at showing swaps,
additions, or deletions in the HTML. Therefore, without images, responses are
better when the questions have swaps, insertions, and deletions of elements
(Tracking Shuffled Objects of three objects, Penguins in a Table).

For Date Understanding and Geometric Shapes tasks, the images generated from
HTMLs most of the time are incorrect given the question. One reason is HTMLs
can not depict the progression of forming a shape (Geometric Shapes) mentioned
in the question text. Another reason is that HTML can not fully maintain the
date, month, and year pattern mentioned in the question text (Table 7) which
further confuses the vlm while performing reasoning with the image.

#### Image helps a different subset of a particular task.

We further investigate the performance of with-image and without-image setups
to find out when having an image is beneficial and when having an image hurts.
As shown in Figure 6, we break down the performance by subtasks and count the
number of times the vlm produces a correct answer with an image and gets the
same question wrong without an image (Image Improves). Then, we reverse the
conditions, i.e., count the number of times the vlm produces an incorrect
answer with an image and generates the correct answer for the same question
without an image (Image Hurts). We can see that for all subtasks, having
images helps solve some questions that can not be solved without images and
vice versa.

Figure 6: Number of Instances from each subtask impacted by Image. Here ‘Image
Hurts’ represents instances that achieved correct answers without image and
got incorrect with image. Similarly ‘Image Improves’ shows data points getting
the correct answers with image and getting incorrect without image.

##  6 Related Works

#### Visual Problem Solving.

Visual problem-solving is an inherent human ability while performing complex
multi-step reasoning. Humans tend to draw a mental image of a question to
understand and ground the problem, which helps to plan subsequent steps to
solve the question. Mental images provide a simplified representation of the
content of the cognitive task. Thus, the involvement of visual images in
problem-solving may result in a notable degree of success (Bauer & Johnson-
Laird, 1993; Antonietti, 1991). Studies also show that an accurate visual
representation of a problem enhances the chances of solving the problem rather
than having no representation at all (van Garderen et al., 2018; Krawec,
2014).

#### Reasoning with llms and vlms.

In recent years, several Large Language Models (llms) and Vision-Language
Models (vlms) have been introduced, which are showing impressive performance
in complex reasoning tasks (OpenAI, 2023; Touvron et al., 2023; Chowdhery et
al., 2022; Liu et al., 2023a; Zhu et al., 2023; Li et al., 2023; Dai et al.,
2023; Liu et al., 2023b). However, when it comes to solving math word problems
(Cobbe et al., 2021; Koncel-Kedziorski et al., 2016; Patel et al., 2021) or
symbolic reasoning tasks (Suzgun et al., 2022), the vlm can not fairly compete
with the llm as the nature of these tasks is unimodal. While considerable
efforts have been invested in improving the performance of llms on these
reasoning tasks during inference (Madaan et al., 2023; Wang et al., 2023; Gao
et al., 2023b; Wei et al., 2023; Poesia et al., 2023; Hao et al., 2023), fewer
endeavors have been made to tackle these challenges from the perspective of a
vision-language model (Lee et al., 2023; Hsu et al., 2023). A very relevant
work to ours is Hsu et al. (2023), which leverages llm to generate drawing
commands and reads out abstractions from the resulting picture. However, it
relies on a fine-tuned visual foundation model (Lee et al., 2023) to interpret
abstractions from the drawn diagram, requiring additional training data. In
addition, diagrams can only benefit specific tasks, limiting their
applicability to diverse reasoning types. In this paper, we study these text-
only benchmarks using a vlm by proposing a simple idea to leverage the full
potential of a vlm on diverse reasoning tasks.

##  7 Conclusion

In this work, we present Self-Imagine, an approach that maximizes the
capabilities of Vision Language Models (vlms) in solving text-only reasoning
tasks. Our method draws on a common human problem-solving technique, creating
visual representations of problems to aid in reasoning. Our approach is self-
sufficient, requiring no additional data, supervision, or training. Through
our intensive experiments with diverse reasoning tasks, we find that Self-
Imagine significantly improves the performance of vlm (llava-1.5) using self-
generated images. We also find that the extent of improvement relies heavily
on the quality of the generated image. We present cases where image improves
and hurts the performance of the vlm, motivating future research on better
image generation approaches.

## References

  * Antonietti (1991) Alessandro Antonietti.  Chapter 15 why does mental visualization facilitate problem-solving?  In Robert H. Logie and Michel Denis (eds.), _Mental Images in Human Cognition_ , volume 80 of _Advances in Psychology_ , pp. 211–227. North-Holland, 1991.  doi: https://doi.org/10.1016/S0166-4115(08)60515-4.  URL https://www.sciencedirect.com/science/article/pii/S0166411508605154.
  * Bauer & Johnson-Laird (1993) Malcolm I. Bauer and P. N. Johnson-Laird.  How diagrams can improve reasoning.  _Psychological Science_ , 4(6):372–378, 1993.  ISSN 09567976, 14679280.  URL http://www.jstor.org/stable/40062565.
  * Boonen et al. (2014) A.J.H. Boonen, F. van Wesel, J. Jolles, and M. van der Schoot.  The role of visual representation type, spatial ability, and reading comprehension in word problem solving: An item-level analysis in elementary school children.  _International Journal of Educational Research_ , 68(4):15–26, 2014.  ISSN 0883-0355.  doi: 10.1016/j.ijer.2014.08.001.
  * Chen et al. (2023a) Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang.  Visual instruction tuning with polite flamingo.  _arXiv preprint arXiv:2307.01003_ , 2023a.
  * Chen et al. (2023b) Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.  _arXiv preprint arXiv:2310.09478_ , 2023b.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Gao et al. (2023a) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 12 2023a.  URL https://zenodo.org/records/10256836.
  * Gao et al. (2023b) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  PAL: Program-aided language models.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 10764–10799. PMLR, 23–29 Jul 2023b.  URL https://proceedings.mlr.press/v202/gao23f.html.
  * GoogleAI (2023) GoogleAI.  Bard, 2023.
  * Hao et al. (2023) Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu.  Reasoning with language model is planning with world model.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 8154–8173, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.507.  URL https://aclanthology.org/2023.emnlp-main.507.
  * Hsu et al. (2023) Joy Hsu, Gabriel Poesia, Jiajun Wu, and Noah D. Goodman.  Can visual scratchpads with diagrammatic abstractions augment LLM reasoning?  In _I Can’t Believe It’s Not Better Workshop: Failure Modes in the Age of Foundation Models_ , 2023.  URL https://openreview.net/forum?id=YlhKbQ0zF3.
  * Koncel-Kedziorski et al. (2016) Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi.  MAWPS: A math word problem repository.  In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 1152–1157, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/N16-1136.  URL https://aclanthology.org/N16-1136.
  * Krawec (2014) Jennifer L. Krawec.  Problem representation and mathematical problem solving of students of varying math ability.  _Journal of Learning Disabilities_ , 47(2):103–115, 2014.  doi: 10.1177/0022219412436976.  URL https://doi.org/10.1177/0022219412436976.  PMID: 22392891.
  * Lee et al. (2023) Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  In _International Conference on Machine Learning_ , pp. 18893–18912. PMLR, 2023.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , 2023.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023a.
  * Liu et al. (2023b) Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li.  Llava-plus: Learning to use tools for creating multimodal agents, 2023b.
  * Lu et al. (2023) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.  Chameleon: Plug-and-play compositional reasoning with large language models.  _arXiv preprint arXiv:2304.09842_ , 2023.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback, 2023.
  * Miao et al. (2020) Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su.  A diverse corpus for evaluating and developing english math word problem solvers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 975–984, 2020.
  * OpenAI (2021) OpenAI.  Openai gpt-3.5 api [gpt-3.5-turbo], 2021.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are NLP models really able to solve simple math word problems?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2080–2094, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.168.  URL https://aclanthology.org/2021.naacl-main.168.
  * Poesia et al. (2023) Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D. Goodman.  Certified reasoning with language models.  2023\.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 8748–8763. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/radford21a.html.
  * Shi et al. (2020) Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu.  Improving image captioning with better use of captions, 2020.
  * St.Laurent et al. (2001) Simon St.Laurent, Murata Makoto, and Dan Kohn.  XML Media Types.  RFC 3023, January 2001.  URL https://www.rfc-editor.org/info/rfc3023.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_ , 2022.
  * Tantau (2022) Till Tantau.  _The TikZ and PGF Packages_ , April 2022.  URL https://github.com/pgf-tikz/pgf.  Manual for version 3.1.9a.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * van Garderen et al. (2012) Delinda van Garderen, Amy Scheuermann, and Christa Jackson.  Developing representational ability in mathematics for students with learning disabilities: A content analysis of grades 6 and 7 textbooks.  _Learning Disability Quarterly_ , 35(1):24–38, 2012.  doi: 10.1177/0731948711429726.  URL https://doi.org/10.1177/0731948711429726.
  * van Garderen et al. (2018) Delinda van Garderen, Amy Scheuermann, Apryl Poch, and Mary M. Murray.  Visual representation in mathematics: Special education teachers’ knowledge and emphasis for instruction.  _Teacher Education and Special Education_ , 41(1):7–23, 2018.  doi: 10.1177/0888406416665448.  URL https://doi.org/10.1177/0888406416665448.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=1PL1NIMMrw.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality, 2023.
  * You et al. (2023) Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang.  Ferret: Refer and ground anything anywhere at any granularity.  _arXiv preprint arXiv:2310.07704_ , 2023.
  * Zhang et al. (2023) Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo.  Gpt4roi: Instruction tuning large language model on region-of-interest, 2023.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.

##  Appendix A Appendix

##  Appendix B Prompts

The prompt used for image generation is shown in 1. Please see the code
repository for the complete prompt.

⬇

1

2Q: Alfie, the albatross, flies 400 kilometers every day. If the circumference
of the earth is 40,000 kilometers, how many days will it take Alfie to fly a
distance equal to half of the way around the earth?

3

4# HTML code:

5

6

7<!DOCTYPE html>

8<html lang="en">

9<head>

10<meta charset="UTF-8">

11<meta name="viewport" content="width=device-width, initial-scale=1.0">

12<title>Alfie’s Journey</title>

13<style>

14 .diagram-container {{

15 display: flex;

16 align-items: center;

17 justify-content: center;

18 flex-direction: column;

19 font-family: Arial, sans-serif;

20 }}

21

22 .earth {{

23 position: relative;

24 width: 200px;

25 height: 200px;

26 border: 3px solid blue;

27 border-radius: 50%;

28 overflow: hidden;

29 }}

30

31

32

33 .text {{

34 margin: 10px;

35 text-align: center;

36 }}

37

38 .stat {{

39 display: flex;

40 justify-content: space-around;

41 margin-top: 20px;

42 }}

43

44 .stat > div {{

45 text-align: center;

46 }}

47</style>

48</head>

49<body>

50

51<div class="diagram-container">

52 <div class="earth">

53 <div class="albatross-flight"></div>

54 </div>

55 <div class="text">Alfie’s Journey Around the Earth</div>

56 <div class="stat">

57 <div>

58 <strong>Alfie’s Daily Distance:</strong><br>

59 400 km

60 </div>

61 <div>

62 <strong>Earth’s Circumference:</strong><br>

63 40,000 km

64 </div>

65 <div>

66 <strong>Target Distance:</strong><br>

67 20,000 km (halfway around the Earth)

68 </div>

69 </div>

70</div>

71

72</body>

73</html>

Listing 1: Prompt for generating HTML using vlm Task | Prompt
---|---
Question Only |  Question + Image
gsm8k, asdiv, svamp, mawps |  Solve the math problem. Think step-by-step. Always end your answer with ‘The answer is ⟨answer⟩’. |  Solve the math problem using the image. Think step-by-step. Always end your answer with ‘The answer is ⟨answer⟩’.
Penguins in a Table |  Answer questions about a table of penguins and their attributes. |  Answer questions about a table of penguins and their attributes using the image.
Colored Objects |  Answer extremely simple questions about the colors of objects on a surface. |  Answer extremely simple questions about the colors of objects on a surface using the image.
Object Counting |  Questions that involve enumerating objects and asking the model to count them. |  Questions that involve enumerating objects and asking the model to count them using the image.
Navigate |  Given a series of navigation instructions, determine whether one would end up back at the starting point. |  Given a series of navigation instructions, determine whether one would end up back at the starting point using the image.
Date Understanding |  Infer the date from context. |  Infer the date from context using the image.
Geometric Shapes |  Name geometric shapes from their SVG paths. |  Name geometric shapes from their SVG paths and using the image.
Temporal Sequences |  Answer questions about which times certain events could have occurred. |  Answer questions about which times certain events could have occurred using the image.
Tracking Shuffled Objects |  A task requiring determining the final positions of a set of objects given their initial positions and a description of a sequence of swaps. |  A task requiring determining the final positions of a set of objects given their initial positions and a description of a sequence of swaps using the image.
Table 3: Prompts used for both reasoning and mathematics tasks. For all
reasoning tasks, we also add Please think step-by-step, and finally answer by
selecting an option using the format “The answer is ⟨option⟩” after adding the
question to the above mentioned prompts.

(a) gsm8k

(b) svamp

(c) asdiv

Figure 7: Accuracy by question length across three mathematical reasoning tasks. In the cases of asdiv and svamp, accuracy is notably higher when utilizing images for longer and more intricate questions compared to scenarios without images. However, in the context of more complex questions, such as those found in gsm8k, the limitations of the vlm become apparent. In this scenario, the inability to generate effective HTML results in erroneous image generation, consequently leading to decreased accuracy, particularly with longer questions. Figure 8: gsm8k accuracy by chain-of-thought length. Similar to the findings in Figure 7, image representations for complex questions are not efficient and structured. Therefore, the inclusion of images does not enhance the representation of questions that demand longer chains of thought.  Figure 9: Accuracy by Question Length for a subset of BIG-Bench-Hard benchmark. Incorporating images helps when the corresponding question is simpler and shorter and when the questions are more complex. Table 4: Example of Image hurting reasoning in asdiv task. Question |  Our class got fifty-four books from the library. Then we got twenty-three more books from the library. How many books did our class get from the library?
---|---
Image |
Label |  77
Output With Image |  To solve this problem, we need to add the initial number of books (50) and the number of books received from the library (23). 50 + 23 = 73 The answer is 73.
Output Without Image |  To solve this problem, we need to add the number of books the class got from the library initially (54) to the number of books they got from the library later (23). Step 1: Add 54 and 23 to find the total number of books the class got from the library. 54 + 23 = 77 The answer is 77.
Table 5: Example of Image hurting reasoning in gsm8k task. Question |  A pound of almonds costs $10 while a pound of walnuts costs $15. How much more does it cost for a mixture of 1/2 pound almonds and 1/3 pound walnuts than a mixture of 1/5 pound almonds and 1/3 pound walnuts?
---|---
Image |
Label |  3
Output With Image |  To solve this problem, we need to find the cost of each mixture and then compare them. Cost of 1/2 pound almonds and 1/3 pound walnuts: We know that a pound of almonds costs $10, and a pound of walnuts costs $15. To find the cost of 1/2 pound almonds and 1/3 pound walnuts, we need to multiply the cost per pound by the amount of each nut in the mixture: (10/lb) * (1/2 lb almonds) + (15/lb) * (1/3 lb walnuts) = $10 + $5 = $15 Cost of 1/5 pound almonds and 1/3 pound walnuts: We can find the cost of this mixture in a similar way: (10/lb) * (1/5 lb almonds) + (15/lb) * (1/3 lb walnuts) = $2 + $5 = $7 Comparison: Now we can compare the costs of the two mixtures: $15 - $7 = $8 The answer is $8.
Output Without Image |  To solve this problem, we need to find the cost of the mixture of 1/2 pound almonds and 1/3 pound walnuts and compare it to the cost of the mixture of 1/5 pound almonds and 1/3 pound walnuts. First, we need to find the cost of 1/2 pound of almonds and 1/3 pound of walnuts. We can do this by multiplying the cost per pound of almonds and walnuts by the respective weights: 1/2 pound of almonds * $10/pound = $1/2*10=5 1/3 pound of walnuts * $15/pound = $1/3*15=5 The total cost of the mixture is the sum of the costs of the almonds and walnuts: $5 + $5 = $5+5=10 Next, we need to find the cost of 1/5 pound of almonds and 1/3 pound of walnuts. We can do this in the same way: 1/5 pound of almonds * $10/pound = $1/5*10=2 1/3 pound of walnuts * $15/pound = $1/3*15=5 The total cost of the mixture is the sum of the costs of the almonds and walnuts: $2 + $5 = $2+5=7 Now we can compare the costs of the two mixtures: $10 - $7 = $10-7=3 The answer is $3.
Table 6: Example of Image improving reasoning in Navigate task. Question |  If you follow these instructions, do you return to the starting point? Always face forward. Take 5 steps forward. Take 8 steps backward. Take 4 steps forward. Take 4 steps right.
Options:
\- Yes
\- No
---|---
Image |
Label |  No
Output With Image |  The answer is No
Output Without Image |  To determine whether one would end up back at the starting point, we need to analyze the given navigation instructions step-by-step. Take 5 steps forward. Take 8 steps backward. Take 4 steps forward. Take 4 steps right. Let’s analyze each step: Take 5 steps forward: After taking 5 steps forward, the person would be 5 steps away from the starting point. Take 8 steps backward: After taking 8 steps backward, the person would be 8 steps away from the starting point, in the opposite direction. Take 4 steps forward: After taking 4 steps forward, the person would be 4 steps away from the starting point, but now they are moving in the correct direction. Take 4 steps right: After taking 4 steps right, the person would be 4 steps away from the starting point, but now they are at a right angle to the starting point. Since the person is now at a right angle to the starting point and moving in the correct direction, they will eventually return to the starting point by continuing to move forward. The answer is: Yes
Table 7: Example of Image hurting reasoning in Date Understanding task. Question |  TIn the US, Thanksgiving is on the fourth Thursday of November. Today is the US Thanksgiving of 2001. What is the date yesterday in MM/DD/YYYY?
Options:
(A) 09/12/2002
(B) 11/30/2002
(C) 11/21/2002
(D) 11/21/2076
(E) 11/07/2002
(F) 11/15/2002
---|---
Image |
Label |  (C)
Output With Image |  The answer is (E) 11/07/2002.
Output Without Image |  To infer the date from context, we need to determine the date of US Thanksgiving in 2001. Step 1: Determine the date of US Thanksgiving in 2001. The US Thanksgiving holiday is celebrated on the fourth Thursday of November. In 2001, November has 30 days. Step 2: Calculate the date of US Thanksgiving in 2001. 4th Thursday of November 2001: Thursday = 22 (since it’s a Thursday) November = 30 22 + 30 = 52 (the date in numerical format) Step 3: Convert the numerical date to MM/DD/YYYY format. 52 = 11/22/2001 The answer is (C) 11/21/2002.