(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Shanghai AI Laboratory 22institutetext: Peking University
33institutetext: Xi’an Jiaotong University 44institutetext: University of
Wisconsin-Madison 55institutetext: The Chinese University of Hong Kong
55email<EMAIL_ADDRESS>55email<EMAIL_ADDRESS>55email:
<EMAIL_ADDRESS>

https://Draw-and-Understand.github.io/

# Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend
What You Want

Weifeng Lin Equal contribution.11 Xinyu Wei∗ 22 Ruichuan An 33 Peng Gao∗∗ 11
Bocheng Zou 44
Yulin Luo 22 Siyuan Huang 11 Shanghang Zhang 22 Hongsheng Li Corresponding
authors.55

###### Abstract

The interaction between humans and artificial intelligence (AI) is a crucial
factor that reflects the effectiveness of multimodal large language models
(MLLMs). However, current MLLMs primarily focus on image-level comprehension
and limit interaction to textual instructions, thereby constraining their
flexibility in usage and depth of response. In this paper, we introduce the
Draw-and-Understand project: a new model, a multi-domain dataset, and a
challenging benchmark for visual prompting. Specifically, we propose SPHINX-V,
a new end-to-end trained Multimodal Large Language Model (MLLM) that connects
a vision encoder, a visual prompt encoder and an LLM for various visual
prompts (points, bounding boxes, and free-form shape) and language
understanding. To advance visual prompting research for MLLMs, we introduce
MDVP-Data and MDVP-Bench. MDVP-Data features a multi-domain dataset containing
1.6M unique image-visual prompt-text instruction-following samples, including
natural images, document images, OCR images, mobile screenshots, web
screenshots, and multi-panel images. Furthermore, we present MDVP-Bench, a
comprehensive and challenging benchmark to assess a model’s capability in
understanding visual prompting instructions. Our experiments demonstrate
SPHINX-V’s impressive multimodal interaction capabilities through visual
prompting, revealing significant improvements in detailed pixel-level
description and question-answering abilities.

###### Keywords:

Multimodal Large Language Model Visual Prompting

Figure 1: An Overview of the Draw-and-Understand Project. (a) Illustrating the
task of pixel-level referring understanding, where visual prompts and an input
image are processed by the model to generate a targeted response. (b) The
architecture of SPHINX-V, which consists of an image encoder, a visual prompt
encoder, and an LLM. (c) The data generation process for training, which
involves two components: reconstruction of open-source data and data
generation assisted by GPT-4V.

##  1 Introduction

Recent works [35, 78, 33, 2, 34] have enhanced Multimodal Large Language
Models (MLLMs) with visual perception, facilitating image-related
communication and a deeper understanding of the world. However, current MLLMs
tend to focus on comprehending entire images by aligning them with textual
information at the image level. This method restricts users from highlighting
specific areas within images for more detailed inquiries. Therefore, enabling
models to understand at the pixel level presents a significant research
challenge.

To meet this challenge, recent advancements have concentrated on employing
visual prompting to enhance pixel-level comprehension. Specifically, research
works [74, 6, 7] utilize textual representations to delineate coordinates
within images, thereby enhancing interactions with models. Another group of
studies [46, 73, 77] employ positional embeddings to achieve enhanced spatial
recognition. Meanwhile, efforts by [51, 73, 67, 70, 4] focus on Regions of
Interest (ROI) to augment focus on specific areas within images. Each of these
methods has shown promising results, yet they come with their respective
limitations. ROI-based methods, for instance, rely on fine-grained masks for
precise object representation. However, they often require pre-existing
segmentation models or externally provided ground truth masks, which limits
user flexibility and model scalability. Furthermore, some methods rely on
fixed-format visual prompts such as bounding boxes, which are not user-
friendly. A critical shortfall is the inability of most existing methods to
support referring to multiple objects simultaneously. This limitation
restricts usage flexibility and hampers the achievement of more complex
understandings, such as discerning the nuanced interrelations and spatial
dynamics among surrounding entities and the backdrop.

To address these shortcomings, we introduce SPHINX-V, an end-to-end multimodal
large language model designed to enhance pixel-level image understanding. As
depicted in Fig.1(b), the SPHINX-V architecture comprises a vision encoder, a
visual prompt encoder, and a Large Language Model (LLM). We leverage pre-
trained MLLMs such as SPHINX[18, 32] as the base model, chosen for their
robust vision-language understanding. To align vision and language features at
the pixel level more effectively, we propose a two-stage training strategy:
$(S1)$ pre-training for image-visual prompt-text alignment and $(S2)$
supervised fine-tuning following instructions. Furthermore, unlike other
methods, we integrate continuous visual prompts (like strokes and scribbles)
with bounding boxes during inference, avoiding separate modeling. To
effectively adapt the model to this approach, we introduce a noise-based
training augmentation strategy. This involves adding noise to sparse visual
prompts to simulate the region area for free-form shaped inputs. With these
methods in place, users can interact with SPHINX-V using their native
language, as well as referring actions such as clicking and drawing, to obtain
desired answers about the region of interest as shown in Fig.2.

Figure 2: (a) SPHINX-V has achieved unprecedented performance in text
recognition, surpassing other visual prompting MLLMs. Meanwhile, it excels in
understanding implicit relationships and conducting complex reasoning. (b)
SPHINX-V demonstrates the capability to perceive objects at a pixel level
granularity, such as ’a girl holding a pen’. Furthermore, it can perform
detailed captioning and inter-relationship analysis on arbitrarily shaped
Regions Of Interest.

To further achieve fine-grained pixel-level understanding, high-quality
instruction-following training datasets are essential. Currently, several
works [70, 67, 74] have developed datasets focused on region-text referring
instructions. However, these datasets are mainly limited to natural images,
which restricts the model’s applicability after training. Additionally, their
scope is often narrow, typically covering only regional descriptions and
simple Q&A tasks. To overcome these limitations, we introduce the Multi-domain
Visual-Prompting Instruction Dataset (MDVP-Data). Unlike previous datasets,
MDVP-Data covers a wider range of image types and interactions, providing a
more comprehensive solution. The dataset includes approximately 0.9 million
images and 1.6 million query-answer pairs, primarily featuring image-point-
text and image-region-text pairings. We assembled this dataset by enhancing
existing datasets with segmentation masks or bounding box annotations and by
utilizing the capabilities of GPT-4V for data generation. MDVP-Data includes
detailed attributes for objects identified by visual prompts, their
relationships with nearby entities, and the context within the background. By
covering a broad spectrum of pixel-level understanding across diverse image
domains, MDVP-Data significantly improves MLLMs’ adaptability and accuracy in
various visual scenarios.

To evaluate the effectiveness and robustness of visual prompting models, we
introduce MDVP-Bench, a benchmark tailored for assessing models’ pixel-level
comprehension. MDVP-Bench encompasses a variety of tasks, including point-
level and region-level captioning, which test the models’ ability to describe
specific visual elements; inter-relationship analysis, assessing the
understanding of connections between entities; and complex reasoning,
challenging the models’ ability to infer and deduce from visual information.
We anticipate that MDVP-Bench will lay a solid foundation for future research
in the realm of visual prompting and multimodal models.

In particular, our paper makes the following contributions:

  * $\bullet$

We introduce SPHINX-V, a new multimodal large language model designed for
visual prompting, equipped with a novel visual prompt encoder and a two-stage
training strategy. SPHINX-V supports multiple visual prompts simultaneously
across various types, significantly enhancing user flexibility and achieve a
fine-grained and open-world understanding of visual prompts.

  * $\bullet$

We construct MDVP-Data, a comprehensive dataset for multi-domain visual-prompt
instruction tuning. This dataset encompasses data for both point-level and
region-level understanding, designed to enhance a model’s comprehension
ability and robustness.

  * $\bullet$

We introduce MDVP-Bench, a challenging benchmark designed to evaluate tasks
that require a combination of detailed description referrals, inter-
relationship analysis, and complex reasoning.

  * $\bullet$

In a broad range of pixel-level understanding tasks across established
benchmarks, our experimental results reveal that our model consistently
outperforms other visual prompting models, showcasing SPHINX-V’s exceptional
pixel-level understanding.

##  2 Related Works

###  2.1 Multimodal large language models (MLLMs)

In recent years, the development of Large Language Models (LLMs) has marked a
significant milestone in the field of Natural Language Processing (NLP). LLMs
such as the GPT series [49], PaLM [12], and LLaMA [58] have not only achieved
remarkable results in text processing but also laid the foundation for the
development of multimodal learning. Following this, the emergence of
Multimodal Large Language Models (MLLMs), including BLIP-2 [29], Flamingo [1],
and PaLM-E [14], further expanded the application scope of LLMs. Moreover,
ongoing research into LLMs has explored extending their capabilities beyond
text to other modalities. Notably, a fine-grained understanding of vision has
emerged as a new focus. Works such as VisionLLM [62] employ language-guided
tokenizers to extract vision features at specific granularities to obtain
relevant visual features, showcasing the potential in this direction.

###  2.2 Visual and Multimodal Prompting

Visual and multimodal prompting in deep learning [24, 70, 67, 3, 74] is an
emerging area of study. Techniques using visual prompts (e.g., boxes, masks)
aim to enhance model performance on specific visual tasks. Key developments
include SAM [24] and its enhanced versions, which support a broad range of
prompts. However, SAM initially struggled to generate semantic labels vital
for complex scene analysis. This gap led to innovations like SEEM [80], HIPIE
[63], and Semantic SAM [27] to improve semantic prediction. However, for real-
world applications, models require multidimensional semantic analysis to fully
grasp and reason about visual scenes.

Recent studies such as GPT4RoI [73], Kosmos-2 [46], Shikra [7], Ferret [67],
and GLaMM [51] have enabled MLLMs to achieve region-based image understanding.
Colorful Prompting Tuning (CPT)[66] and RedCircle[55] pioneer the use of color
overlays and targeted visual cues to enhance models’ interpretative abilities.
ViP-LLaVA [3] also allows users to intuitively mark images and interact with
the model using natural prompts. Osprey [70] incorporates fine-grained mask
regions into language instruction, achieving precise pixel-wise visual
understanding. However, this mask-based technology relies on pre-attached
segmentation models or externally provided ground truth masks, which limits
its flexibility and scope of application. Furthermore, most existing methods
do not support referring to multiple targets simultaneously, which
significantly limits the model’s ability to perform more complex understanding
and reasoning tasks.

Figure 3: Illustrative Example from the MDVP Dataset: This diagram displays
the distribution of images drawn from various domains, including nature
scenes, texts in the wild, web content, mobile interfaces, documents, and
multi-pane graphics. It also features a sample from the GPT-assisted MDVP
dataset, emphasizing the diversity and richness of its point-based and region-
based instruction-following data.

##  3 MDVP: Multi-domain Visual-Prompt Instruction Data Generation

In this section, we introduce MDVP-Data, an instruction dataset designed to
foster fine-grained pixel-level and open-world image understanding in MLLMs,
encompassing approximately 1.6 million multimodal dialogues. MDVP-Data
integrates both point-level and region-level instruction data derived from
public datasets. It consists of two types of data: $(1)$ a restructured public
grounding dataset formatted for visual prompt-based instruction following
(Sec. 3.1); and $(2)$ high-quality training pairs developed using meticulously
crafted prompt templates, produced through the GPT-4V model (Sec. 3.2).
Additionally, we present MDVP-Bench, a challenging benchmark suite designed to
evaluate comprehensive pixel-level understanding capabilities under various
visual prompts.(Sec. 3.3)

###  3.1 Inverse-Grounding Visual Prompt-based Dataset

We observe that the grounding capability, which encompasses the ability to
localize visual content and link each entity referred to by a noun phrase in
the caption to a specific region in the image, essentially represents the
inverse process of visual prompt proficiency. Building on this insight, we
reorganize existing datasets for Referring Expression Comprehension (REC),
Phrase Grounding, and Grounding QA to construct our dataset for visual
prompting.

For captioning tasks, we have inverted the inputs and outputs of existing REC
and Phrase Grounding datasets, such as Flickr30K [48] RefCOCO/+[23, 69]
GCG[51], and GRIT [46], treating the ground truth boxes as input visual
prompts, with the corresponding natural language descriptions serving as the
answers. For visual prompt-based conversations, we utilize grounding QA pairs
from point-QA [40], Visual 7W [79] and VCR [72] datasets. The bounding boxes
mentioned in the questions are treated as visual prompt inputs, with the
positions of bounding boxes in both questions and answers being substituted
with <Mark i> or <Region i>.

###  3.2 GPT4V-assisted Visual Instruction Data Generation

To construct a more comprehensive and diverse instruction-following dataset,
we have collected various multi-domain images along with their unique
annotations from publicly available datasets. These encompass natural images,
Optical Character Recognition (OCR) in the wild, documents, webpage
screenshots, mobile screen captures, and multi-panel images. For each distinct
image domain, we have meticulously crafted prompts to facilitate the GPT-4V’s
adoption of various roles in the generation of instructional data.
Specifically, we task GPT-4V with four distinct assignments: brief captioning,
detailed captioning, inter-relationship analysis, and the generation of
complex reasoning. Uniquely, the captions created for each image type possess
their own distinctive characteristics. For instance, in mobile screen
captures, objects can be identified not only as relevant components—icons,
text, search bars, URLs, etc.—but information about potential actions is also
provided. This includes whether a component is clickable and the expected
outcome upon interaction. This diversity significantly enhances the richness
and presents challenges for our dataset, covering a broader and more
comprehensive range of knowledge domains.

Furthermore, as shown in Fig. 3, to enable GPT-4V to better recognize the
objects referred to by visual prompts, we have employed Set-of-Marks (SoM)
[65] prompting. This method involves directly marking the highlighted objects
on the input images, ensuring that the data created is strongly associated
with the referenced objects. Concurrently, we propose incorporating the
category information of each object within the prompts, which significantly
enhances the quality of the generated data. (For more details on the specific
prompts and the methods used for marking the objects, please refer to the
Appendix. 0.C.2.)

###  3.3 MDVP-Bench

To evaluate the proficiency of the MLLM in complex pixel-level image
understanding tasks and its versatility across various domains, we initially
curated a subset of our MDVP-Data. This subset was subjected to a thorough
manual content review and filtering process, culminating in the creation of
MDVP-Bench. MDVP-Bench is a comprehensive and challenging benchmark
encompassing a wide array of tasks, including concise descriptions, elaborate
narratives, analyses of interconnections among different regions, and complex
reasoning.

Existing evaluations typically follow the approach of LLaVA [35], utilizing
GPT-4 for open-ended evaluation and employing textual descriptions to depict
image content. These textual descriptions, often based on image annotations,
can lead to scenarios where GPT-4 may not recognize objects or backgrounds not
annotated in the scene. Additionally, inherent domain gaps between textual
descriptions and images can lead to GPT-4 misunderstanding the image content.
To address these limitations, we employ GPT-4V for evaluation in MDVP-Bench.
Following the pipeline described in Sec.3.2, we directly annotate images with
SoM prompting, submitting both the image and textual questions to GPT-4V for
scoring. This approach helps to circumvent potential misinterpretations of
image content by GPT-4V. The scoring adheres to the LLaVA-bench guidelines,
with scores ranging from 1 to 10, where higher scores indicate superior model
performance.

##  4 Model Architecture

Figure 4: Overview of the SPHINX-V model architecture. (a & b) The proposed
two-stage training strategy. (c) Overall model architecture.

In this section, we present SPHINX-V, a model designed for pixel-level
referring understanding. As illustrated in Fig. 4(c), SPHINX-V comprises three
components, : a mixed vision encoder, a versatile visual prompt encoder, and a
large language model (LLM). Given an image alongside coordinates of specific
points or regions and instruction language, we perform tokenization and
encoding to derive three distinct embeddings: image embeddings $Z_{i}$, visual
prompt embeddings $Z_{v}$, and text embeddings $Z_{t}$. Subsequently, we
employ two projection matrixs to convert $Z_{i}$ and $Z_{v}$ into language
embedding tokens, aligning them with the word embedding space in the language
model. Finally, we integrate the image, visual prompt, and language
representations, forwarding them collectively to the LLM to attain
pixel/region-level understanding capability.

Vision Encoder.  Driven by the scalability and effective pretraining of MLLMs,
we incorporate the Mixed Vision Encoder (MVE) from the pretrained SPHINX [18,
32]. This encoder generates the image embeddings $Z_{i}=MVE(X_{i})$, operating
at an image resolution of $448\times 448$. It expands the scope of image
embeddings to encompass multiple scales and high-resolution sub-images,
facilitating the efficient encoding of high-resolution images.

Visual Prompt Encoder.  In SPHINX-V, we mainly consider two types of visual
prompts: points and bounding boxes. As shown in Fig. 4(c), we introduce a new
visual prompt encoder. Inspired by SAM [24], initially, we utilizes positional
encodings [57] for points and boxes, then adds learnable embeddings for each
prompt type, and finally processes them through a linear layer to obtain
unified output embeddings. Additionally, our visual prompt encoder
accommodates a dynamic number of visual prompts as input. Specifically, we
first set a fixed number of input tokens for visual prompts (e.g., 10). Then,
based on the validity of the actual input tokens, we equip both valid and
invalid tokens with a set of learnable vectors to assist the model in
discerning their effective features. (For more details on the visual prompt
encoder, please refer to the Appendix. 0.A.)

Large Lanuage Model.  We consider LLaMA 2-13B [58] as our large language model
(LLM), a transformer-based decoder-only LLM.

Table 1: Statistics of Data in Stage 1.

Type | Raw Data | #Samples
---|---|---
Natural |  COCO [31]/LVIS [19] Visual Genome [25] OpenImages [26] Obj365 [54]/V3Det [60] ADE20k [76] Cityscapes [13] Pascal VOC [41] Flickr30k [48] | 4.6M
Document layout |  Docbank [30] PublayNet [75] DoclayNet [47]/M6Doc [9] | 3.1M
OCR Spoting |  icdar13 [22]/icdar15 [21] ctw1500 [71]/mlt2017 [43] mlt2019 [42]/totaltext [11] CurvedSynText150k [36] | 550K
Android & Web |  AITW [52]/SeeClick [10] | 300K

Table 2: Statistics of Data in Stage 2.

Task | Raw Data | #Samples
---|---|---
Category Identification |  Subset in Stage-1 | 2.0M
Brief Caption |  RefCOCO [23, 69] RefCOCO+ [23, 69] RefCOCOg [23, 69] Visual Genome [25] MDVP-Data | 1.2M
Detailed Caption |  MDVP-Data | 752K
Relationship Analysis |  Visual Genome [25] MDVP-Data | 1.1M
Inverse-Grounding Description |  GRIT [46] Flicker30K [48] OpenPsgGCG [51] | 1.2M
Whole Image Caption |  shareGPT4v [8] laionGPT4v [5] | 1.2M
Q&A and Resoning |  VCR [25]/Visual7W [79] Osprey-724K [70] MDVP-Data | 2.3M

Training Strategy. The training process of our SPHINX-V model consists of two
important stages.

$\bullet$ Stage 1: Image-Visual Prompt-Text Alignment Pre-training. By
employing a visual prompt encoder, we initially freeze both the pre-trained
vision encoder and the pre-trained LLM. Then, we focus on training the
features of visual prompts to align with the features of image-visual prompt-
text. Following the approach in LLaVA [35], we implement an MLP as the
connector between visual prompts and language to enhance the multimodal
prompting capabilities of the model. At the first stage, we gather and
reconstruct open-source datasets, including those for object detection,
semantic segmentation, layout detection, spotting, and more, to build our
training data. Specifically, for point inputs, we initially randomly sample
from semantic segmentation images, where each point has a corresponding pixel-
level label annotation. Similarly, for instance segmentation, we can sample
points from the mask of each object, with these points corresponding to the
label annotation of that object. For box inputs, we directly utilize the
ground truth bounding boxes provided by detection datasets as visual prompt
inputs, allowing the model to recognize their corresponding labels. The
structure of our stage-1 pre-training data is as follows:

As illustrated in Table 2, we have gathered a diverse array of detection and
segmentation datasets covering multiple image domains. These datasets
encompass a wide range of label types, including elements of the natural world
(persons, animals, stuff, etc.), document elements (titles, paragraphs,
images, tables, etc.), OCR (text recognition), and screenshots (icons, text,
search bars, etc.). With such rich and extensive data for pre-training,
SPHINX-V is equipped to develop capabilities for visual prompting and
preliminary category identification.

Furthermore, to support free-form shape visual prompts, we propose a noise-
based training augmentation in the pre-training stage to simulate the region
area required when inputting free-form shaped visual prompts. Specifically,
for box-type visual prompts, we introduce a certain degree of noise based on
the size of the box itself. This noise may cause the ground truth bounding box
to be slightly larger than the target object or only cover part of the target
object, thereby approximating a free-form outer rectangle. For points, we
sample several pixels within the mask of a target object. These pixels are
directed towards the same target object during training, thus achieving
precise point-based referring.

$\bullet$ Stage 2: Multi-Task Supervised Finetuning. At the second stage, we
load the weights trained during stage 1 and keep the vision encoder and visual
prompt encoder weights frozen. We then fine-tune the visual prompt projector
and the LLM model. Our focus is on enhancing SPHINX-V’s ability to accurately
interpret user instructions and handle diverse pixel-level understanding
tasks, such as detailed captioning, inter-relationship analysis, and complex
reasoning. Table 2 outlines all the data utilized during stage 2 fine-tuning,
which includes our proposed MDVP dataset, Visual Genome (VG)[25], Visual
Commonsense Reasoning (VCR)[72], Visual7w [79], and Osprey-724k [70].
Furthermore, we reformat the grounding dataset for compatibility with visual
prompt-based instructions. Lastly, we ensure the model retains the capability
to provide detailed captions for the entire image when given a visual prompt
that encompasses most of the image’s content. After the instruction fine-
tuning in stage 2, SPHINX-V becomes proficient in comprehending complex
scenarios based on user instructions and specific regions of interest.

Inference.  Although SPHINX-V is limited to modeling point-level and box-level
features of visual prompts, the noise-based training augmentation proposed in
the stage 1 enables users to freely draw on the image, greatly increasing the
flexibility of user usage. Specifically, by pre-processing these free-form
visual prompts, we can obtain their corresponding bounding boxes, and then
feed these box coordinates into SPHINX-V for inference.

##  5 Experiments

###  5.1 Implementation Details

We utilized AdamW [38] as our optimizer and employed flash attention [16] to
improve computational efficiency. During the first stage of training, we set
the initial learning rate to $4e-5$. For the second stage, the learning rate
was adjusted to $1e-5$. The input images were processed at a resolution of 448
x 448 pixels, and the maximum sequence length for the Large Language Model
(LLM) was established at 2048. The entire training procedure was conducted on
8 NVIDIA A100 GPUs, each equipped with 80GB of memory. For all evaluation
experiments, we opted for a zero-shot testing approach, refraining from
further fine-tuning on specific datasets.

###  5.2 Referring Object Classification

This task is defined as follows: the question targets a specific area within
the image, requiring the model to identify the object within that designated
region. Following [70], we employ two semantic relevance indicators—Semantic
Similarity (SS) and Semantic Intersection over Union (S-IOU)[53]—to assess a
model’s classification prowess on the validation sets of object-level LVIS
[19] and part-level PACO [50] datasets. As the results shown in Table 3
indicate, SPHINX-V achieves 83.16% in SS and 58.64% in S-IoU based on point-
level visual prompts, and 87.06% in SS and 62.90% in S-IoU based on box-level
visual prompts on the LVIS dataset. Both sets of results significantly
outperform the state-of-the-art method by a notable margin. Furthermore,
SPHINX-V surpasses the previous best, Osprey, by 6.82% in SS and 2.32% in
S-IoU on the PACO dataset, demonstrating its robust capability in fine-grained
part-level classification.

We also conducted tests on traditional closed-set object classification tasks.
To ensure the model outputs names of categories within the closed set, we
followed the approach described in Ferret [67] and adapted our evaluation to a
binary-choice format. We posed questions such as ’Please identify the label of
the marked region in the image. Is it (Class A) or (Class B)?’. The results,
as shown in Table 4, further demonstrate SPHINX-V’s excellent classification
ability.

Table 3: The results of referring object classification on LVIS and PACO. Method | LVIS | PACO
---|---|---
Semantic Similarity |  Semantic IoU |  Semantic Similarity |  Semantic IoU
LLaVA-1.5 [34] | 48.95 | 19.81 | 42.20 | 14.56
Kosmos-2 [46] | 38.95 | 8.67 | 32.09 | 4.79
Shikra-7B [7] | 49.65 | 19.82 | 43.64 | 11.42
GPT4RoI-7B [73] | 51.32 | 11.99 | 48.04 | 12.08
Ferret-7B [67] | 63.78 | 36.57 | 58.68 | 25.96
Osprey-7B [70] | 65.24 | 38.19 | 73.06 | 52.72
SPHINX-V(Point) | 83.16 | 58.64 | 76.18 | 51.13
SPHINX-V(Box) | 87.06 | 62.90 | 79.88 | 55.04
Table 4: Results of referring object classification and optical character
recognition (OCR) on COCO text. We randomly perturb the center positions and
scale the dimensions of box visual prompts to simulate free-form prompts.

Method | LVIS |  OCR (COCO Text)
---|---|---
Point |  Box |  Free-Form |  Box
LLaVA [34] | 50.1 | 50.3 | - | -
Kosmos-2 [46] | - | 60.3 | - | -
Shikra-7B [7] | 57.8 | 67.7 | - | -
GPT4RoI-7B [73] | - | 61.8 | - | -
Ferret-7B [67] | 67.9 | 79.4 | 69.8 | -
Ferret-13B [70] | 68.4 | 80.5 | 71.0 | -
ChatSpot-7B | - | 64.5 | - | 31.8
SPHINX-V(Ours) | 86.46 | 89.82 | 88.96 | 45.44

###  5.3 Regional Optical Character Recognition.

Optical character recognition (OCR) is a task focused on identifying text in
images, serving as a fundamental aspect of visual entity recognition.
Following [74], we utilize the COCO-Text dataset [59] to gauge the regional
text recognition prowess of SPHINX-V. Employing the ground-truth bounding
boxes supplied within the dataset, we prompt the model with requests such as
"Please provide the ocr results of each marked region in the image." SPHINX-V
then analyzes and response the textual content present within the specified
regions. Given that most visual prompt-based models do not support OCR, we
compare our results with ChatSpot under identical zero-shot settings. As
illustrated in Table 4, our model surpasses the ChatSpot by 13.64%, showcasing
the promising regional capability of text recognition.

Moreover, as shown in the Fig. 5, SPHINX-V is capable of not only recognizing
text in the referring region but also possesses enhanced understanding and
descriptive abilities. For instance, it can further describe characteristics
such as the font type, color, and background of the text, and understand the
role of the text within the image area. This fully demonstrates SPHINX-V’s
exceptional pixel-level understanding capabilities.

###  5.4 Region-Level Captioning

Brief Region Description.  We provide quantitative comparisons for the region-
level captioning task using both mask- and box-based approaches [46, 51, 15,
70]. Specifically, we employ a box visual prompt and a text prompt, such as
"Please provide a brief description of each marked region in the image," to
prompt SPHINX-V to concisely describe the content of the targeted region.
Experiments were conducted on the validation sets of RefCOCOg. The comparison
results, shown in Table 6, demonstrate promising outcomes when compared to
other methods. (More experimental results can be found in Appendix. 0.B.1)

Table 5: Detailed region-level captioning performance on the RefCOCOg
validation set.

Method | Detailed Description
---|---
LLaVA-1.5 [34] | 71.11
Kosmos-2 [46] | 40.89
Shikra-7B [7] | 40.97
GPT4RoI-7B [73] | 49.97
Osprey-7B [70] | 77.54
SPHINX-V(Ours) | 92.19

Table 6: Brief region captioning performance evaluated on the validation set
of RefCOCOg.

Method | Type | METEOR | CIDEr
---|---|---|---
GRIT [67] | Box | 15.2 | 71.6
Kosmos-2 [46] | Box | 14.1 | 62.3
GLaMM [51] | Box | 16.2 | 105.0
Osprey-7B [70] | Mask | 16.6 | 108.3
SPHINX-V(Ours) | Box | 23.9 | 162.5

Detailed Region Description.  To further evaluate the detailed area
description capabilities of our model and other visual prompting MLLMs,
inspired by [70, 34], we leverage GPT-4 to comprehensively measure the quality
of the responses generated by the model to the input referring regions.
Specifically, in the same manner as Osprey [70], we sample 80 images from the
validation set of RefCOCOs [23, 69] for generating detailed region caption,
using box visual prompt and text prompt like "Please provide a detailed
description of each marked region in the image.". Subsequently, GPT-4 was
utilized to assess the detailed captions generated by MLLMs. The evaluation
scores span from 1 to 10, with higher scores reflecting enhanced performance.
To evaluate the efficacy of MLLMs, we determined the ratio of the predicted
answer score to that of GPT-4, expressing this ratio as a percentage. The
results shown in Table 6 indicates that SPHINX-V achieves the best performance
with 92.19% accuracy, significantly outperforming other region-based and mask-
based methods.

Table 7: Results on LLaVA-Bench and Ferret-Bench.

| LLaVA-Bench | Ferret-Bench
---|---|---
|  Conversation |  Detail Description |  Complex Reasoning |  Referring Description |  Referring Reasoning
LLaVA [35] | 85.4 | 68.3 | 92.1 | 41.4 | 31.7
Kosmos-2 [46] | 71.7 | 63.4 | 74.9 | 51.8 | 33.7
Shikra-7B [7] | 80.6 | 70.7 | 88.1 | 46.0 | 41.6
Ferret-7B [67] | 84.4 | 79.4 | 96.3 | 68.7 | 67.3
Ferret-13B [67] | 85.2 | 80.9 | 96.4 | 70.6 | 68.7
SPHINX-V(Ours) | 85.0 | 86.4 | 91.3 | 76.4 | 70.4

Table 8: Results on different domains of MDVP-bench. *Osprey accepts the ground truth mask of the referring object as visual prompt input. Domain | Natural | OCR | Multi-panel | Screenshot
---|---|---|---|---
| Box | Point | Box | Point | Box | Point | Box | Point
Osprey* [70] | 86.40 | - | 18.33 | - | 48.56 | - | 28.84 | -
Ferret-7B [67] | 84.92 | 81.42 | 21.24 | 12.20 | 44.42 | 32.08 | 26.40 | 10.62
Ferret-13B [67] | 86.67 | 83.40 | 27.68 | 12.20 | 49.83 | 37.65 | 30.92 | 14.40
SPHINX-V(Ours) | 88.82 | 92.95 | 78.49 | 76.29 | 77.30 | 79.08 | 80.07 | 54.05
Figure 5: Visual examples of SPHINX-V.

###  5.5 Comprehensive Assessment: LLaVA-Bench, Ferret-Bench and MDVP-Bench

We use LLaVA-Bench to compare SPHINX-V with previous models [34, 7, 46, 70] to
evaluate whole image-level understanding capability. Given that the questions
in LLaVA-Bench are based on the whole image, we employ bounding boxes of
nearly full-image size as visual prompts. Furthermore, we use Ferret-Bench and
our proposed MDVP-Bench to evaluate pixel-level understanding capability.
Experiments were conducted on the Referring Description and Referring
Reasoning tasks within Ferret-Bench.

As the results presented in Table 7, since we did not incorporate open-source
image-level VQA data into our training dataset, it is observable that in
LLaVA-Bench, our model performs slightly below the current state-of-the-art
methods in conversation and complex reasoning evaluations. However, it still
significantly leads in the capability of detailed captioning. Moreover, in the
Ferret-Bench assessment focused on pixel-level image understanding, whether in
referring description or referring reasoning, our model achieved the best
performance. This demonstrates SPHINX-V’s powerful fine-grained part-level
understanding and reasoning abilities.

For MDVP-Bench, we mixed different tasks together to obtain a comprehensive
assessment of the model’s overall capabilities, applying evaluation process
mentioned in Sec 3.3. As the results shown in Table 8, compared to Osprey and
Ferret, our SPHINX-V significantly leads in evaluations across multiple
domains. In addition, it can be observed that the scores for box prompts are
sometimes lower than those for point prompts. Upon reviewing the responses to
each question, we discovered that for tasks with fewer visual prompts on an
image, box-based prompts perform better, while in tasks involving more visual
prompts on an image, simple point prompts demonstrate significantly better
performance. This finding reveals the modeling function of our visual prompt
encoder and their capability to model different visual prompts, inspiring
future research and improvement of their functionality.

Figure 6: Comparative visual examples from various methods.

###  5.6 Visualization

We provide visual examples to further illustrate the pixel-level spatial
understanding capabilities of our SPHINX-V across a diverse range of visual
prompts. Fig. 5 displays a variety of visual scenarios from different domains,
such as natural scenes, mobile and web interfaces, optical character
recognition (OCR) in natural settings, document layouts, and multi-panel
images. In these complex situations, SPHINX-V showcases its proficiency in
accurately responding to queries, highlighting its robust and versatile
capabilities. (More visualizations could be found in Appendix. 9)

Furthermore, Fig.6 provides comparisons with existing image-level and region-
level methods. Our methodology yields superior scene comprehension results,
offering a nuanced and detailed analysis when compared to existing image-level
and region-level approaches.

###  5.7 Ablation Study

To evaluate the effectiveness of the key elements of our design, we conduct
the following ablation experiments. Given the extensive amount of training
data, our comparison was limited to the first 50k training iterations.

Table 9: Comparisons across different modeling format of visual prompts and
training strategy.

Method | LVIS | PACO
---|---|---
|  Semantic Similarity |  Semantic IoU |  Semantic Similarity |  Semantic IoU
Alpha Blending | 45.5 | 21.7 | 39.6 | 14.2
Text | 47.3 | 25.5 | 43.47 | 18.3
Encoder(Ours) | 55.9 | 30.1 | 52.3 | 22.4
One-Stage | 43.1 | 19.2 | 39.2 | 12.5
Two-Stage(Ours) | 55.9 | 30.1 | 52.3 | 22.4

The Effectiveness of Visual Prompt Encoder.  To assess the effectiveness of
our visual prompt encoder, we explored two alternative methods for
incorporating visual prompts into MLLMs: (1) overlaying the visual prompts on
the original image via alpha blending with an alpha value of 0.5; and (2)
explicitly including the coordinates of the visual prompt in the instructional
text prompt. As shown in Table 9, while these alternative methods enabled the
MLLM to successfully identify the regions indicated by the visual prompts,
their performance was not as strong as that achieved with our visual prompt
encoder. This suggests that our visual prompt encoder possesses superior
capabilities for processing visual prompts.

The Two-stage Training Strategy.  To validate the effectiveness of our
proposed two-stage training strategy, we conducted experiments in which we
bypassed the pre-training phase (stage 1) and directly set both the visual
prompt encoder and the LLM to a trainable state for model training. As
depicted in Table 9, the results clearly demonstrate that, within the same
training duration and using identical data, the two-stage training approach
significantly outperforms the single-stage method. This improvement is due to
the two-stage training strategy’s notable acceleration of the model’s
convergence speed, thereby underscoring the effectiveness and necessity of the
two-stage training paradigm.

##  6 Conclusion

In summary, we introduce a new multimodal large language model for visual
prompting named SPHINX-V, which exhibits promising capabilities in pixel-level
image understanding. With our proposed visual prompt encoder and two-stage
training strategy, SPHINX-V is able to support various types of visual
prompts, including points, boxes, and free-form shapes, significantly
enhancing user flexibility. Additionally, we have curated the MDVP dataset,
comprising 1.6M high-quality image-point-text and image-region-text pairs for
model training, and the MDVP benchmark, which is both comprehensive and
challenging, for evaluation purposes. Our superior performance in established
visual prompting tasks—including referring object classification, region-level
captioning, regional OCR, and referring reasoning—underscores the efficacy and
robustness of SPHINX-V. We anticipate that our contributions will lay a solid
foundation for further exploration in the field of intelligent visual
interaction systems.

## References

  * [1] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [2] Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)
  * [3] Cai, M., Liu, H., Mustikovela, S.K., Meyer, G.P., Chai, Y., Park, D., Lee, Y.J.: Making large multimodal models understand arbitrary visual prompts. arXiv preprint arXiv:2312.00784 (2023)
  * [4] Chen, C., Qin, R., Luo, F., Mi, X., Li, P., Sun, M., Liu, Y.: Position-enhanced visual instruction tuning for multimodal large language models. arXiv preprint arXiv:2308.13437 (2023)
  * [5] Chen, G.H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., Wang, B.: Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684 (2024)
  * [6] Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)
  * [7] Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)
  * [8] Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 (2023)
  * [9] Cheng, H., Zhang, P., Wu, S., Zhang, J., Zhu, Q., Xie, Z., Li, J., Ding, K., Jin, L.: M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15138–15147 (2023)
  * [10] Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., Wu, Z.: Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935 (2024)
  * [11] Ch’ng, C.K., Chan, C.S.: Total-text: A comprehensive dataset for scene text detection and recognition. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 935–942. IEEE (2017)
  * [12] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24(240), 1–113 (2023)
  * [13] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223 (2016)
  * [14] Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)
  * [15] Duckworth, A.L., Quinn, P.D.: Development and validation of the short grit scale (grit–s). Journal of personality assessment 91(2), 166–174 (2009)
  * [16] Ford, S., Cova, M., Kruegel, C., Vigna, G.: Analyzing and detecting malicious flash advertisements. In: 2009 Annual Computer Security Applications Conference. pp. 363–372. IEEE (2009)
  * [17] Gan, Z., Chen, Y.C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems 33, 6616–6628 (2020)
  * [18] Gao, P., Zhang, R., Liu, C., Qiu, L., Huang, S., Lin, W., Zhao, S., Geng, S., Lin, Z., Jin, P., et al.: Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935 (2024)
  * [19] Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5356–5364 (2019)
  * [20] Huang, X., Wang, J., Tang, Y., Zhang, Z., Hu, H., Lu, J., Wang, L., Liu, Z.: Segment and caption anything. arXiv preprint arXiv:2312.00869 (2023)
  * [21] Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V.R., Lu, S., et al.: Icdar 2015 competition on robust reading. In: 2015 13th international conference on document analysis and recognition (ICDAR). pp. 1156–1160. IEEE (2015)
  * [22] Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., i Bigorda, L.G., Mestre, S.R., Mas, J., Mota, D.F., Almazan, J.A., De Las Heras, L.P.: Icdar 2013 robust reading competition. In: 2013 12th international conference on document analysis and recognition. pp. 1484–1493. IEEE (2013)
  * [23] Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 787–798 (2014)
  * [24] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [25] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123, 32–73 (2017)
  * [26] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision 128(7), 1956–1981 (2020)
  * [27] Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Yang, J., Li, C., Zhang, L., Gao, J.: Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767 (2023)
  * [28] Li, G., Duan, N., Fang, Y., Gong, M., Jiang, D.: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 11336–11344 (2020)
  * [29] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [30] Li, M., Xu, Y., Cui, L., Huang, S., Wei, F., Li, Z., Zhou, M.: Docbank: A benchmark dataset for document layout analysis. arXiv preprint arXiv:2006.01038 (2020)
  * [31] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [32] Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al.: Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575 (2023)
  * [33] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 (2023)
  * [34] Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Lee, Y.J.: Llava-next: Improved reasoning, ocr, and world knowledge (January 2024), https://llava-vl.github.io/blog/2024-01-30-llava-next/
  * [35] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems 36 (2024)
  * [36] Liu, Y., Chen, H., Shen, C., He, T., Jin, L., Wang, L.: Abcnet: Real-time scene text spotting with adaptive bezier-curve network. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9809–9818 (2020)
  * [37] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)
  * [38] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [39] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems 32 (2019)
  * [40] Mani, A., Yoo, N., Hinthorn, W., Russakovsky, O.: Point and ask: Incorporating pointing into visual question answering. arXiv preprint arXiv:2011.13681 (2020)
  * [41] Mottaghi, R., Chen, X., Liu, X., Cho, N.G., Lee, S.W., Fidler, S., Urtasun, R., Yuille, A.: The role of context for object detection and semantic segmentation in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 891–898 (2014)
  * [42] Nayef, N., Patel, Y., Busta, M., Chowdhury, P.N., Karatzas, D., Khlif, W., Matas, J., Pal, U., Burie, J.C., Liu, C.l., et al.: Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019. In: 2019 International conference on document analysis and recognition (ICDAR). pp. 1582–1587. IEEE (2019)
  * [43] Nayef, N., Yin, F., Bizid, I., Choi, H., Feng, Y., Karatzas, D., Luo, Z., Pal, U., Rigaud, C., Chazalon, J., et al.: Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 1454–1459. IEEE (2017)
  * [44] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [45] Pan, T., Tang, L., Wang, X., Shan, S.: Tokenize anything via prompting (2023)
  * [46] Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F.: Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023)
  * [47] Pfitzmann, B., Auer, C., Dolfi, M., Nassar, A.S., Staar, P.: Doclaynet: A large human-annotated dataset for document-layout segmentation. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 3743–3751 (2022)
  * [48] Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: Proceedings of the IEEE international conference on computer vision. pp. 2641–2649 (2015)
  * [49] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [50] Ramanathan, V., Kalia, A., Petrovic, V., Wen, Y., Zheng, B., Guo, B., Wang, R., Marquez, A., Kovvuri, R., Kadian, A., Mousavi, A., Song, Y., Dubey, A., Mahajan, D.: PACO: Parts and attributes of common objects. In: arXiv preprint arXiv:2301.01795 (2023)
  * [51] Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R.M., Xing, E., Yang, M.H., Khan, F.S.: Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03356 (2023)
  * [52] Rawles, C., Li, A., Rodriguez, D., Riva, O., Lillicrap, T.: Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088 (2023)
  * [53] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: A metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 658–666 (2019)
  * [54] Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., Sun, J.: Objects365: A large-scale, high-quality dataset for object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 8430–8439 (2019)
  * [55] Shtedritski, A., Rupprecht, C., Vedaldi, A.: What does clip know about a red circle? visual prompt engineering for vlms. arXiv preprint arXiv:2304.06712 (2023)
  * [56] Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019)
  * [57] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33, 7537–7547 (2020)
  * [58] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [59] Veit, A., Matera, T., Neumann, L., Matas, J., Belongie, S.: Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140 (2016)
  * [60] Wang, J., Zhang, P., Chu, T., Cao, Y., Zhou, Y., Wu, T., Wang, B., He, C., Lin, D.: V3det: Vast vocabulary visual detection dataset. arXiv preprint arXiv:2304.03752 (2023)
  * [61] Wang, W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, L., Chen, Z., Li, H., Zhu, X., Cao, Z., et al.: The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907 (2023)
  * [62] Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al.: Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems 36 (2024)
  * [63] Wang, X., Li, S., Kallidromitis, K., Kato, Y., Kozuka, K., Darrell, T.: Hierarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems 36 (2024)
  * [64] Wu, J., Wang, J., Yang, Z., Gan, Z., Liu, Z., Yuan, J., Wang, L.: Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280 (2022)
  * [65] Yang, J., Zhang, H., Li, F., Zou, X., Li, C., Gao, J.: Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441 (2023)
  * [66] Yao, Y., Zhang, A., Zhang, Z., Liu, Z., Chua, T.S., Sun, M.: Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797 (2021)
  * [67] You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.F., Yang, Y.: Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 (2023)
  * [68] Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., Wang, H.: Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 3208–3216 (2021)
  * [69] Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pp. 69–85. Springer (2016)
  * [70] Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., Zhu, J.: Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10032 (2023)
  * [71] Yuliang, L., Lianwen, J., Shuaitao, Z., Sheng, Z.: Detecting curve text in the wild: New dataset and new solution. arXiv preprint arXiv:1712.02170 (2017)
  * [72] Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: Visual commonsense reasoning. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [73] Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Chen, K., Luo, P.: Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601 (2023)
  * [74] Zhao, L., Yu, E., Ge, Z., Yang, J., Wei, H., Zhou, H., Sun, J., Peng, Y., Dong, R., Han, C., et al.: Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning. arXiv preprint arXiv:2307.09474 (2023)
  * [75] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1015–1022. IEEE (2019)
  * [76] Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., Torralba, A.: Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127, 302–321 (2019)
  * [77] Zhou, Q., Yu, C., Zhang, S., Wu, S., Wang, Z., Wang, F.: Regionblip: A unified multi-modal pre-training framework for holistic and regional comprehension. arXiv preprint arXiv:2308.02299 (2023)
  * [78] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)
  * [79] Zhu, Y., Groth, O., Bernstein, M., Fei-Fei, L.: Visual7w: Grounded question answering in images. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4995–5004 (2016)
  * [80] Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Wang, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. Advances in Neural Information Processing Systems 36 (2024)

## Appendix

##  Appendix 0.A More Details of Architecture

Image Encoder.  In general, the image encoder can be any network that outputs
a $C\times H\times W$ image embedding. Motivated by scalability and access to
strong pre-training, we employ a Mixture of Visual Experts (MoV) with mixed
scales and high-resolution sub-images to process high-resolution inputs [18,
32]. Specifically, we use two complementary vision encoders: DINOv2 [44] and
CLIP-ConvNeXt [37], to capture a diverse visual representation. During
training, all images are by default scaled and zero-padded to a high
resolution of $448\times 448$, and then divided into four $224\times 224$ sub-
images. For images with large aspect ratios, such as $2:1$, this results in
fully-padded sub-images that are filled entirely with zero-value pixels. To
address this issue, we follow a strategy [18] that involves using a learnable
skip token to replace the fully-padded sub-image. This provides explicit
relative positional information, enabling Language Learning Models (LLMs) to
identify the positions between useful sub-images more effectively.

Visual Prompt Encoder.  In Fig. 7, we present our approach for the integration
of multiple visual prompts (VPs) into the visual prompt encoder. Specifically,
for point-type inputs, we represent VPs as the sum of a positional encoding
[57] of the point’s coordinates. For box-type inputs, inspired by [24], VPs
are conceptualized as the aggregate of positional encodings for the
coordinates of both the ’top-left corner’ and the ’bottom-right corner’. We
then proceed to dynamically assess the efficacy of the feature set associated
with each VP, applying one of two learned embeddings to indicate the VP’s
validity. To further delineate and model the positions of the center point,
the ’top-left corner’, and the ’bottom-right corner’, we allocate three
distinct learned embeddings for each. The culmination of this process involves
the incorporation of an MLP as the output layer, tasked with generating the
final embeddings of the visual prompts with a dimensionality of 512.

Figure 7: Details of the visual prompt encoder.

##  Appendix 0.B More Experiments

###  0.B.1 Brief Region Description

To further evaluate the captioning capability of SPHINX-V, we conducted tests
on the evaluation set of Visual Genome [25]. As the comparison results shown
in Tab. 10, our SPHINX-V model exhibits promising performance compared to
other models that utilize various types of prompts. It achieves a METEOR score
of 24.3 and a CIDEr score of 197.3, significantly outperforming the state-of-
the-art TAP(ViT-L) [45] approach by 6.8 and 46.6, respectively. These results
highlight the superior capability of our model in generating semantically
relevant descriptions for object regions.

Table 10: Comparison of various methods on the Visual Genome dataset. Method | Prompt Type | Visual Genome
---|---|---
|  | METEOR | CIDEr
GRiT [64] | Bounding Box | 17.1 | 142.0
GPT4ROI-7B [73] | Bounding Box | 17.4 | 145.2
GPT4ROI-13B [73] | Bounding Box | 17.6 | 146.8
ASM [61] | Bounding Box | 18.0 | 145.1
SCA(GPT2-774M) [20] | (8 Caption + 6 Task) Tokens | 17.4 | 148.8
SCA(LLama-3B) [20] | (8 Caption + 6 Task) Tokens | 17.4 | 149.8
TAP(ViT-B) [45] | 1 Semantic Token | 17.4 | 149.2
TAP(ViT-L) [45] | 1 Semantic Token | 17.5 | 150.7
SPHINX-V(ours) | Bounding Box |  24.3 | 197.3
Table 11: Validation Accuracy on VCR dataset. Model | Q $\rightarrow$ A (%) | QA $\rightarrow$ R (%) | Q $\rightarrow$ AR (%)
---|---|---|---
ViLBERT [39] | 72.4 | 74.5 | 54.0
Unicoder-VL [28] | 72.6 | 74.5 | 54.5
VLBERT-L [56] | 75.5 | 77.9 | 58.9
ERNIE-ViL-L [68] | 78.52 | 83.37 | 65.81
VILLA-L [17] | 78.45 | 82.57 | 65.18
GPT4RoI-7B [73] | 87.4 | 89.6 | 78.6
ViP-LLaVA-Base-7B [3] | 87.66 | 89.80 | 78.93
SPHINX-V(Ours) | 89.65 | 90.23 | 89.63

###  0.B.2 Region-Level Reasoning

To evaluate the reasoning capabilities of SPHINX-V, we utilized the Visual
Commonsense Reasoning (VCR) dataset [72], a challenging benchmark designed to
assess a model’s high-level cognitive and commonsense reasoning abilities
within context. The VCR dataset comprises multiple-choice questions that
require an understanding of the scene depicted in an image. Each question (Q)
is accompanied by four possible answers (A), necessitating the model to not
only identify the correct answer but also provide a rationale (R) supporting
its selection. This process underscores the model’s proficiency in
interpreting and justifying visual elements within specific contexts, with
accuracy serving as the evaluation metric. Following the evaluation approach
employed in LLaVA-ViP [34] and GPT4RoI [73], we fine-tuned our SPHINX-V model
using the training set of VCR. As shown in the comparison results in Tab. 11,
our model achieves the highest performance scores of 89.65%, 90.23%, and
89.63% across three distinct evaluation methods, respectively, showcasing its
proficiency in visual commonsense reasoning.

##  Appendix 0.C More Details of Dataset

In this section, we outline our methodology for reconstructing public
grounding datasets and our application of GPT-4V to produce instruction-based
data spanning a variety of domains, leading to the development of the MDVP
Dataset. Fig.8 depicts an example from our Inverse-Grounding dataset.
Sec.0.C.2 showcase the prompts used to generate data across these diverse
domains with GPT-4V.

###  0.C.1 Examples of Inverse Grounding

Using two sample images from the Flickr30k dataset [48] as illustrations, the
phrase grounding task is defined as identifying the bounding boxes in an image
that correspond to various referring phrases, based on a given ground truth
(gt) sentence. In a twist on this task, we invert the annotation process: the
model, provided with bounding boxes, is tasked with generating the
corresponding gt sentence. This process is depicted in Fig. 8.

Figure 8: Examples of Inverse-Grounding data.

###  0.C.2 Prompts for Data Generation

We carefully designed prompts to facilitate data generation by GPT-4V,
assigning it various roles and embedding category-specific information within
these prompts. Table 12 outlines a universal prompt template applied across
different domains. Meanwhile, Tables 13, 15, 15, 17, and 17 detail prompts
tailored to specific domains. An illustrative example of GPT-4V’s response
during data generation is provided in Table 18.

image = messages =  {"role":"system", "content": In the image, I have marked
each visual object with a green point, and each is identified by a white
numeric ID against a black background. The categories of visual objects are as
follows: <Mark 1>: dog <Mark 2>: bed <Mark 3>: mattress <Mark 4>: pillow
Your analysis will revolve around four main functions: <Role>
I will now supply you with specific output templates for content corresponding
to the four roles. Please adhere strictly to these templates when generating
content, refraining from making any additional alterations, including the
insertion of extra spaces or line breaks. Adhering to this guideline is
extremely crucial. Format: <Format>
Proceed with your analysis, keeping the language natural and clear.}

Table 12: The public prompt template used to feed to GPT-4V for data
generation. Public prompts for different domains exhibit slight variations.
For instance, in the screen-shot domain, "In the image" is modified to "In the
screen-shot". The highlighted <Role> and <Format> in red are domain-specific.

Role
<Role 1 (Short Description)> Provide a succinct and clear description for each
marked region, ensuring each description stands independently without
reference or comparison to others.
<Role 2 (Detailed Description)> For each marked region in the image, please
provide as detailed a description as possible using natural language.
Highlight the object’s category, type, color, and additional attributes such
as location, condition, and any relevant details. Envision yourself observing
each region directly and convey your observations as thoroughly and promptly
as possible. A minimum of 30 words is required in your description. Treat each
mark as a unique and separate entity, requiring a full description exclusive
to that mark only.
<Role 3 (Inter-Relationship Analysis)> Delve into and analyze the
relationships between the marked regions. In your analysis, reference specific
areas using identifiers like <Mark 1>, <Mark 2>, etc. Elucidate the links and
common features among these areas, which might encompass aspects such as their
spatial arrangement, inherent qualities, underlying principles, resemblances,
variances, contextual ties, or notable discrepancies. Should you find certain
relational aspects insignificant or lacking in noteworthy content, omit them
from your discussion. In instances where a marked region stands apart without
clear ties to others, highlight its distinctiveness and provide a detailed
description of this unique object or area.
<Role 4 (Q&A and Conversations)> Dive deeper into the detailed content and
intricacies of every marked regions, and interconnections among multiple
marked regions. Employ identifiers such as <Mark 1>, <Mark 2>, etc., to
specify each area in your inquiries and responses. Assist in formulating
question-answer pairs that focus on either a single target area or multiple
target areas, aiming to develop a rich dialogue dataset (comprising at least 4
Q&A pairs). Ensure that the questions you craft inquire about one or more
specified <Mark>s.
Format
Role 1
<Mark 1>: Your Short Description
…
<Mark N>: Your Short Description
Role 2
<Mark 1>: Your Comprehensive Description
…
<Mark N>: Your Comprehensive Description
Role 3
<Mark 1><Mark 2>…<Mark N>: Your Detailed Analysis
…
<Mark 1><Mark 2>…<Mark N>: Your Detailed Analysis
Role 4
{"question": [Your created Question], "Answer": [Your created answer]}
…
{"question": [Your created Question], "Answer": [Your created answer]} Table
13: The <Role> and <Format> prompt template used for natural images domain.

Role
<Role 1 (Detailed Analysis and Description)> First, you need to deeply
understand the page content presented by the entire screenshot, and then
thoroughly examine and explain the content and purpose of the highlighted
area. Provide a comprehensive description of its features, layout and
functionality. Specify if the area is interactive (such as a clickable button
or search bar), and describe the result or action that occurs upon
interaction. If the highlighted area contains text, analyze whether it is a
hyperlink, and the meaning of the text and its intended function. To ensure
precise and detailed descriptions for the marked regions, please observe the
following guidelines. Avoid mentioning the green rectangle or numeric ID of
the marks I placed on the image, as they are not significant. Treat each mark
as a unique and separate entity, requiring a full description exclusive to
that mark only. A minimum of 30 words is required in your description.
<Role 2 (Q&A and Conversations)> Dive deeper into the detailed content and
intricacies of every marked regions, and interconnections among multiple
marked regions. Employ identifiers such as <Region 1>, <Region 2>, etc., to
specify each area in your inquiries and responses. Assist in formulating
question-answer pairs that focus on either a single target area or multiple
target areas, aiming to develop a rich dialogue dataset (comprising at least 4
Q&A pairs). Ensure that the questions you craft inquire about one or more
specified <Region>s.

Table 14: The <Role> prompt template used for screenshots domain. The <Format>
can be referenced from natural images domain in Tab.13.

Role
<Role 1 (Detailed Region Description)> For each marked region in the image,
provide a thorough description using natural language. First, referring to the
categories of visual objects I provided you with, you need to tell me what the
marked region is. Then, do your best to describe the content, characteristics,
and function of the marked area. If the marked area is a text paragraph, you
should first understand the content of the text, and then summarize the main
idea. If the marked area is an image, you need to describe the content of the
image in detail, as well as why this image is included in the document.
To ensure precise and detailed descriptions for the marked regions, please
observe the following guidelines. Avoid mentioning the green rectangle or
numeric ID of the marks I placed on the image, as they are not significant.
Treat each mark as a unique and separate entity, requiring a full description
exclusive to that mark only.
<Role 2 (Q&A and Conversations)> Dive deeper into the detailed content and
intricacies of every marked regions, and interconnections among multiple
marked regions. Employ identifiers such as <Region 1>, <Region 2>, etc., to
specify each area in your inquiries and responses. Assist in formulating
question-answer pairs that focus on either a single target area or multiple
target areas, aiming to develop a rich dialogue dataset (comprising at least 4
Q&A pairs). Ensure that the questions you craft inquire about one or more
specified <Region>s.

Table 15: The <Role> prompt template used for document domain. The <Format>
can be referenced from natural images domain in Tab.13.

Role
<Role 1 (Detailed Region Description)> For each marked region in the image,
I’d like you to give a detailed description, as if you were examining it in
person. First refer to the OCR results of the specified area I provided to
clarify the text content. Then, delve into the characteristics of the text,
including the font style and its exact location within the image. I’d also
appreciate insights into the background context of the region and an analysis
of why the text is present—its intended purpose. Please ensure your
explanation is clear and straightforward.
To ensure precise and detailed descriptions for the marked regions, please
observe the following guidelines. Avoid mentioning the red polygan or numeric
ID of the marks I placed on the image, as they are not significant. Treat each
mark as a unique and separate entity, requiring a full description exclusive
to that mark only.
<Role 2 (Q&A and Conversations)> Dive deeper into the detailed content and
intricacies of every marked regions, and interconnections among multiple
marked regions. Employ identifiers such as <Region 1>, <Region 2>, etc., to
specify each area in your inquiries and responses. Assist in formulating
question-answer pairs that focus on either a single target area or multiple
target areas, aiming to develop a rich dialogue dataset (comprising at least 4
Q&A pairs). Ensure that the questions you craft inquire about one or more
specified <Region>s.

Table 16: The <Role> prompt template used for OCR-spotting domain. The
<Format> can be referenced from natural images domain in Tab.13.

Role
<Role 1 (Detailed Region Description)> For each marked region in the image,
please provide as detailed a description as possible using natural language.
Carefully observe and analyze the content and details of each panel I have
marked. Tell me as best you can what the content and purpose of each marked
area is, including the content of figure and the text. Envision yourself
observing each region directly and convey your observations as thoroughly and
promptly as possible. Treat each mark as a unique and separate entity,
requiring a full description exclusive to that mark only. A minimum of 30
words is required in your description.
<Role 2 (Inter-Relationship Analysis)> Delve into and analyze the
relationships between the marked regions. In your analysis, reference specific
areas using identifiers like <Region 1>, <Region 2>, etc. Elucidate the links
and common features among these areas, which might encompass aspects such as
their spatial arrangement, reading order and contextual ties.
<Role 2 (Q&A and Conversations)> Dive deeper into the detailed content and
intricacies of every marked regions, and interconnections among multiple
marked regions. Employ identifiers such as <Region 1>, <Region 2>, etc., to
specify each area in your inquiries and responses. Assist in formulating
question-answer pairs that focus on either a single target area or multiple
target areas, aiming to develop a rich dialogue dataset (comprising at least 4
Q&A pairs). Ensure that the questions you craft inquire about one or more
specified <Region>s.

Table 17: The <Role> prompt template used for multi-panels images domain. The
<Format> can be referenced from natural images domain in Tab.13.

Response
<Role 1 (Short Description)>
<Mark 1>: Light brown dog sleeping peacefully on a bed.
<Mark 2>: A neatly arranged bed with light blue floral patterns.
<Mark 3>: Cream-colored mattress exposed at the edge of the bed.
<Mark 4>: Soft grey pillow partially tucked under a sleeping dog’s head.
<Role 2 (Detailed Description)>
<Mark 1>: The image displays a light brown adult dog with a closed eyes and
relaxed ears, lying on its side, comfortably asleep. The dog appears to be in
a serene state of rest, with its head gently resting on a pillow, hinting at
the trust and safety it feels in its environment.
<Mark 2>: This mark points to a well-kept bed covered with light blue sheets
that exhibit a delicate floral design. The sheets seem smooth and unwrinkled,
suggesting the bed was recently made or is well-maintained, providing a cozy
resting place.
<Mark 3>: An exposed corner of a cream-colored mattress is visible here,
revealing its thick, sturdy build. The mattress looks to be in good condition,
with a soft upper layer that likely contributes to a comfortable sleeping
experience.
<Mark 4>: Here, we see a soft grey pillow with a smooth texture. It appears to
be of a thickness that provides ample support for comfort. The pillow is
partially tucked under the sleeping dog’s head, indicating its use for rest.
<Role 3 (Inter-Relationship Analysis) >
<Mark 1><Mark 2>: These marks indicate a relationship of proximity and usage.
The dog <Mark 1> is resting on the bed <Mark 2>, which implies the bed’s
purpose in the household as a place for relaxation and sleep.
<Mark 1><Mark 4>: There is a direct interaction here; the dog <Mark 1>)has its
head on the pillow <Mark 4>, showcasing the pillow’s function as a support for
comfort and the dog’s apparent enjoyment of this provision.
<Mark 2><Mark 3>: The bed <Mark 2> is made up with sheets that likely cover
the mattress <Mark 3>, indicating that the mattress is part of the bed setup,
essential for providing a comfortable sleeping surface.
<Mark 3><Mark 4>: There is no direct relationship between the mattress <Mark
3> and the pillow <Mark 4> other than they both contribute to the overall
comfort of the sleeping area. Each serves a different aspect of comfort and
support.
<Role 4 (Q&A and Conversations)>
{"question": "What color is the dog at <Mark 1>?", "answer": "The dog at <Mark
1> is light brown."}
{"question": "Describe the design of the bed’s sheets at <Mark 2>.", "answer":
"The sheets at <Mark 2> have a light blue color with a floral design."}
{"question": "What can be said about the condition of the mattress at <Mark
3>?", "answer": "The mattress at <Mark 3> appears to be in good condition and
sturdy."}
{"question": "How does the pillow at <Mark 4> contribute to the dog’s
comfort?", "answer": "The pillow at <Mark 4> provides support and comfort for
the dog’s head while it sleeps."} Table 18: An example response from GPT-4V
during data generation.  Figure 9: Additional visual examples of SPHINX-V
utilizing arbitrary-shaped visual prompts across multiple domains.