#  OmChat: A Recipe to Train Multimodal Language Models with Strong Long
Context and Video Understanding

Tiancheng Zhao, Qianqian Zhang, Kyusong Lee, Peng Liu, Lu Zhang,
Chunxin Fang, Jiajia Liao, Kelei Jiang, Yibo Ma and Ruochen Xu
Linker Technology Research
Binjiang Institute of Zhejiang University
<EMAIL_ADDRESS>
###### Abstract

We introduce OmChat, a model designed to excel in handling long contexts and
video understanding tasks. OmChat’s new architecture standardizes how
different visual inputs are processed, making it more efficient and adaptable.
It uses a dynamic vision encoding process to effectively handle images of
various resolutions, capturing fine details across a range of image qualities.
OmChat utilizes an active progressive multimodal pretraining strategy, which
gradually increases the model’s capacity for long contexts and enhances its
overall abilities. By selecting high-quality data during training, OmChat
learns from the most relevant and informative data points. With support for a
context length of up to 512K, OmChat demonstrates promising performance in
tasks involving multiple images and videos, outperforming most open-source
models in these benchmarks. Additionally, OmChat proposes a prompting strategy
for unifying complex multimodal inputs including single image text, multi-
image text and videos, and achieving competitive performance on single-image
benchmarks. To further evaluate the model’s capabilities, we proposed a
benchmark dataset named Temporal Visual Needle in a Haystack. This dataset
assesses OmChat’s ability to comprehend temporal visual details within long
videos. Our analysis highlights several key factors contributing to OmChat’s
success: support for any-aspect high image resolution, the active progressive
pretraining strategy, and high-quality supervised fine-tuning datasets. This
report provides a detailed overview of OmChat’s capabilities and the
strategies that enhance its performance in visual understanding.

##  1 Introduction

In recent years, the ability to process and understand multimodal data has
become increasingly critical for developing advanced AI systems Liu et al.
(2024f; d); Bai et al. (2023b); AI et al. (2024). Models that can handle both
textual and visual inputs are essential for a wide range of applications, from
video analysis to complex image processing tasks. One of the key challenges in
this domain is efficiently managing and leveraging long-context data, which
includes sequences of images and video frames that can span significant
temporal lengths.

To address these challenges, we introduce OmChat, a strong and efficient model
designed to excel in handling long multimodal contexts and understanding video
data. OmChat employs an active progressive multimodal pretraining strategy,
which gradually scales the model’s capacity for processing long contexts and
enhances its overall capabilities. By selectively utilizing high-quality data
during training, OmChat is able to learn from the most relevant and
informative data points, ensuring robust performance across various tasks.

OmChat supports a context length of up to 512K tokens, making it well-suited
for tasks involving multiple images and videos. In benchmarks for these tasks,
OmChat consistently outperforms all other open-source models, demonstrating
its superior ability to manage and interpret complex visual data.
Additionally, OmChat 8B model achieves competitive performance on single-image
benchmarks, often surpassing larger size models.

To further evaluate OmChat’s capabilities, we proposed a benchmark dataset
named Temporal Visual Needle in a Haystack. This dataset is designed to assess
the model’s ability to comprehend and process temporal visual details within
videos, challenging OmChat to locate and interpret key information embedded
within long video sequences.

Our analysis identifies several key factors that contribute to OmChat’s
success, including the support for higher image resolutions, the active
progressive pretraining strategy, and the incorporation of high-quality
supervised fine-tuning (SFT) datasets. These elements collectively enhance
OmChat’s efficiency, adaptability, and overall performance in visual
understanding tasks.

In this paper, we provide a comprehensive overview of OmChat’s capabilities
and the innovative strategies. We detail the model’s architecture, training
methodology, and performance across various benchmarks. Our findings highlight
the importance of higher image resolutions, progressive multimodal
pretraining, and high-quality data selection in achieving state-of-the-art
performance in multimodal large language models.

The structure of this paper is as follows: Section 2 details the overall
architecture and training methods of OmChat, including the vision tower and
dynamic vision encoding processes. Section 3 presents the training data recipe
and Section 4 shows the results of our evaluations on single-image, multi-
image, and video benchmarks. Section 5 discusses the ablation and analysis.

##  2 Method

Figure 1: OmChat model overall structure and training steps.

The overall OmChat architecture and training method is depicted in Figure 1.
OmChat processes both visual and textual data inputs. In the context of
multimodal tasks, the visual input can vary from a single image to multiple
images, image-text interleaved data, and video frames. Conversely, for
language tasks, the visual component may be absent. These diverse inputs are
subsequently fed into a large language model for processing. Ultimately, the
output generated by OmChat manifests in textual form.

Unified Multimodal Processing OmChat implements a unified approach to
processing various types of visual inputs. Regardless of the input format,
OmChat standardizes the procedure by first decomposing the inputs into images
before channeling them into the vision tower. This systematic method ensures
that all input variations, whether they involve single images, multiple
images, image-text combinations, or video frames, undergo a consistent
transformation process. This unified process not only enhances the model’s
efficiency in handling different types of visual data but also underscores the
model’s adaptability and robustness in accommodating a wide range of input
modalities.

Dynamic Vision Encoding: In order to effectively address images with varying
resolutions, OmChat has implemented a dynamic vision encoding process inspired
by AnyRes Liu et al. (2024e). Our innovative approach ensures that the model
can adeptly handle images of different resolutions without overlooking small
objects that may be present in high-resolution images. By incorporating this
dynamic vision encoding mechanism, OmChat enhances its capability to capture
fine details and nuances across a spectrum of image resolutions, thereby
improving the overall accuracy and robustness of its vision capabilities.

Multi-Stage Training: OmChat’s training process unfolds in three distinct
steps to optimize its capabilities effectively. During the initial phase, the
vision tower and the language model remain frozen. The focus is on training
the projector that bridges the visual and textual modalities. By isolating
this component for training, OmChat optimizes the connections between vision
and text inputs, ensuring seamless integration and effective communication
between the two domains. The second step involves multimodal generative
training, where both the vision encoder, the language model and the projector
are optimized. In this stage, the training objective revolves around
minimizing the cross-entropy of the text tokens. By updating the vision
encoder, the language model and the projector simultaneously, OmChat enhances
its ability to generate coherent and contextually relevant responses across
different modalities. This comprehensive training approach strengthens the
model’s multimodal understanding and generation capabilities.

Active Progressive Multimodal Pretraining: We implement a progressive training
strategy from 4K to 512K to scale up the model’s long context capacity and
capabilities gradually. Additionally, the implementation of high-quality
selection during training is a crucial step in ensuring that the model learns
from the most informative and relevant data points.

We further detail the key components essential for enhancing OmChat: 1)
support for high resolutions with dynamic vision encoding, 2) a progressive
training strategy for long contexts, and 3) the selection of high-quality
instruction tuning data.

###  2.1 Dynamic Vision Encoding

The vision encoder is a crucial component of multimodal systems. Previous
research has demonstrated that supporting various resolutions can lead to
significant improvements in multimodal training Liu et al. (2024e); Dong et
al. (2024); She et al. (2024). Our findings also show that dynamic vision
encoders greatly enhance performance. Additionally, we employ specific data
formats and delimiters to differentiate between image patches and various
types of visual inputs. For example, a single image is processed as an
individual entity, while videos are treated as sequences of frames. Delimiters
mark the beginning and end of each frame in a video sequence, enabling the
model to effectively understand and process the temporal aspects of video
data.

We adopted the AnyRes technique, which enables our vision tower to support
images and videos of any resolution. AnyRes dynamically adjusts the processing
pipeline to handle varying resolutions, ensuring that the model can process
high-resolution inputs efficiently and accurately. Our dynamic image encoding
method is based on AnyRes. Additionally, we incorporate delimiters to help the
model differentiate between patches, images, and video frames, thereby
enhancing its ability to understand dynamic visual inputs.

###  2.2 Progressive Training Strategy to Long Context

To enhance the model’s ability to process longer contexts effectively, a
progressive training strategy is implemented (Liu et al., 2024a). Initially,
the language base model is expanded to 512K by leveraging a text pretraining
dataset comprising diverse source data. This expansion builds upon our
original language model, i.e., OmBase. The training strategy follows a
sequential context length of 4k, 32K, 128K, and finally 512K. Additionally, we
also train OmChat using Qwen2-7B (Bai et al., 2023a), which initially enables
a context length of 32k. In this case, we extend the context length to 128K
and then 512K. By extending the context length successively, OmChat retains
its proficiency in processing short contexts while developing the capacity to
handle longer contexts at a relatively lower cost. Notably, the positional
encoding parameters are adjusted by scaling up the RoPE $\theta$ to 50M. In
situations where there is minimal or no data available for extensive context
lengths, we employ a specific strategy to ensure our model is adequately
trained. This involves the concatenation of shorter samples to generate
samples that align with the desired context length. Subsequently, leveraging
the language model with a 512k context length, omchat is transformed into a
multimodal model through a multimodal pretraining process. After the projector
alignment training, the model’s context length was progressively extended by
incrementally increasing the context length from 4K, 32K, 128K, to 512K.
Details regarding the training data are elaborated on in Subsection 3.1,
where, akin to text pretraining, training samples that are shorter than the
intended context are concatenated to form a single sample. The RoPE $\theta$
is maintained at 50M.

During the training phase for contexts exceeding 128K in length, RingAttention
(Liu et al., 2024b) is implemented for computing QKV attention. This
specialized method is tailored for calculating attention over long contexts,
addressing memory constraints associate with the quadratic complexity of
attention weight computations. The fundamental concept involves segmenting QKV
along the seq_len dimension into blocks of n*block_size, facilitating the
iterative derivation of complete attention by calculating attention for each
block.

###  2.3 High-Quality multimodal Pretrain Data Selection

Enhancing model performance through multimodal pretraining heavily relies on
the quality of the training dataset. In this pursuit, we leverage an
innovative technology known as Rho-1 Lin et al. (2024b) for the purpose of
high-quality data selection. Expanding upon the original Rho-1 methodology,
initially tested in the context of language-only pretraining, we extend its
application to the realm of multimodal fine-tuning. In alignment with our
perspective that “Not all tokens in a corpus hold equal importance for
multimodal training,” tokens within multimodal data can be categorized into
three distinct types Xiao et al. (2024):

  * •

Type 1: Text highly related to images: entities (e.g., people, animals,
objects), quantities, colors, text, etc. These tokens directly correspond to
image information and are crucial for multimodal alignment.

  * •

Type 2: Text with low relevance to images: transitional words or content that
can be inferred from the preceding text. These tokens primarily serve to train
the pure text capabilities of MLLM.

  * •

Type 3: Text conflicting with image content: These tokens are inconsistent
with image information, potentially providing misleading information and
negatively impacting the multimodal alignment process.

We propose a Selective Visual Language Modeling (SVLM), to prioritize type 1
text and disregard type 3 text. SVLM starts with training a reference model on
high-quality multimodal instruction tuning data. It then computes the
reference loss for all text tokens in multimodal pretraining based on the
logarithmic probability derived from the reference model. To distinguish the
three types of tokens, we generalize the excess loss in Rho-1, which is the
difference between the pretrain loss and the reference loss, to a multimodal
setting. By selectively retaining tokens with high excess loss, the focus can
be efficiently directed towards type 1 tokens while filtering out type 3
tokens. We compute the reference loss offline for the multimodal corpus and
incorporate it into the batched data for real-time computation of excess loss
during training. During step-2 multimodal pretraining, we prioritize tokens
based on their excess loss values within a batch, focusing solely on the top
percentile of tokens for loss computation. The additional steps of loading the
reference loss and conducting ranking have minimal impact on the training
process. The efficacy of SVLM is validated across six benchmarks following the
second phase of multimodal pretraining. Specifically, we evaluated the
performance on CMMLU Li et al. (2023c), CEval Huang et al. (2024), GSM8K Cobbe
et al. (2021), MATH Hendrycks et al. , HumanEval Chen et al. (2021a), and BBH
Suzgun et al. (2022). The average score exhibited an improvement from 32.7 to
38 after the implementation of SVLM.

###  2.4 High-quality Instruction Tuning Data Selection

Figure 2: Illustration of the continuous training strategy. The black dot
represents the step at which a checkpoint is saved using a constant learning
rate for the current data portion. Based on each of these checkpoints and the
corresponding data portions, we perform additional training for an extra 10%
of the steps on the same data using linear decay scheduler, as indicated by
the red lines. The green dot represents the checkpoint used to evaluate the
performance of this data portion.

To enhance the multimodal capabilities of our model, we meticulously curate
over 60 datasets from a diverse array of vision-language tasks for the fine-
tuning stage. These datasets encompass a broad spectrum of key areas,
including general visual question answering, OCR-related tasks, chart and
diagram understanding, image quality assessment, image and video captioning,
document-related queries, mathematical and computational tasks, multi-image
analysis, and video comprehension, among others. From another perspective,
these datasets can be classified into four categories: text-only tasks,
single-image tasks, multi-image tasks, and video-related tasks. Unlike other
models, we have specifically included tasks that utilize multi-image or video
data, such as image comparison and video question answering. This approach
helps the model learn to comprehend multi-image scenarios and the temporal
relationships among video frames, thereby enhancing its performance in multi-
image and video understanding.

One challenge of using these datasets for fine-tuning is the disparity in
their sizes. For instance, OpenHermes-2.5 contains approximately 1 million
samples, whereas InfoVQA has only 2,000 samples. Directly using all datasets
may lead to an imbalance in task variety, making it crucial to find an optimal
approach for data combination. Inspired by the findings in Hu et al. (2024),
we employ a continuous training strategy to select the best instruction tuning
data combination. As shown in Figure 2, we first manually design the
proportion of each dataset in percentage based on the importance of the tasks.
Then, based on the total number of tokens for the answers, we sample a large
combination from the original datasets according to the designed percentages.
For datasets with insufficient data, we perform repeated sampling to meet the
required amount. Afterward, we divide this a large combination into multiple
parts, ensuring that each part maintains the designed percentages. Next, we
apply a continuous training strategy. The model is initially trained on the
first portion of the large data combination using a constant learning rate of
2e-5 with warm up. Subsequently, the second portion of the large data
combination is trained starting from the last checkpoint of the first portion
using the same constant learning rate, and this process continues for the
remaining portions. At the same time, we take the last checkpoint from each
portion and perform additional training on the same data used in constant
phase, followed by a linear decay scheduler for an extra 10% of the total
steps. In this way, we are able to evaluate the performance of instruction
tuning datasets at different scales without repeatedly training the previous
portions. This allows us to select the best combination of portions from all
the instruction tuning datasets. Detailed results are presented in Subsection
5.2.

##  3 Training Data Recipe

The utilisation of diverse and large-scale datasets represents a pivotal
component in the training of multimodal models. Our datasets are categorized
into two main parts: pretraining data and instruction tuning data. The
pretraining data includes both vision-text and text-only data from various
sources, aimed at enhancing the model’s foundational cross-modal understanding
and knowledge. Besides, the instruction tuning data is smaller in scale and is
primarily used to train the model for specific downstream tasks, further
refining and optimizing its performance in real-world applications. This two-
stage data construction approach ensure that the model not only possesses a
broad range of universal capabilities but also excels in specific tasks.

###  3.1 Pretraining data

The pretraining data used in our study encompasses a variety of publicly
available sources along with some proprietary data. The pie chart to the right
of Figure 1 visually represents these data as a percentage of the whole.

Specifically, we divide the whole pretraining dataset into the following
categories:

  * •

Image interleaved with text: This type of data enables the model to improve
its contextual learning for multimodal inputs. We use MMC4-Core (Zhu et al.,
2024) and in-house interleaved data (e.g. news data), which help the model
understand and process contexts where images alternate text.

  * •

Optical Character Recognition (OCR): These data enhance the model’s OCR
capabilities at both the document and image levels. Publicly available
datasets including CTW (Yuan et al., 2019), LSVT (Sun et al., 2019), ReCTS
(Yao et al., 2012), ICDAR2019-ArT (Chng et al., 2019), and pdfa-eng-wds (Pablo
Montalvo, 2024) are used in the pretraining stage.

  * •

Video: This category of data empowers the model to comprehend the sequence of
multiple images and frames, thereby facilitating the processing of continuous
actions and events in videos. We utilize public datasets such as InternVideo
(Wang et al., 2022) and ActivityNet (Caba Heilbron et al., 2015), as well as
various in-house video data from sources like movies and news. The appropriate
frame rate for frame extraction is selected based on the duration of each
video, with frame rates ranging from 0.5 FPS to 30 FPS.

  * •

Caption: The data used consist of several public datasets and a few in-house
datasets. The major public datasets include CC12M (Changpinyo et al., 2021),
ShareGPT4V (Chen et al., 2023), VizWiz-Captions (Gurari et al., 2020), SBU
(Ordonez et al., 2011), Flickr30K (Young et al., 2014), Microsoft COCO
Captions (Chen et al., 2015), Taisu (Liu et al., 2022), ALLaVA-4V (Chen et
al., 2024a), Laion-400M (Schuhmann et al., 2021), CC3M (Sharma et al., 2018),
TextCaps (Sidorov et al., 2020). These datasets assist the model in creating
associations between images and descriptive text.

  * •

Text-only: These datasets, which include RedPajama (Computer, 2023), Alpaca
(Taori et al., 2023), Belle (BELLEGroup, 2023), DeepCtrl-sft-data (DeepCtrl,
2024), Wudao (BAAI, 2023), Telechat (Wang et al., 2024), Firefly (Yang, 2023),
pCLUE (CLUEbenchmarkGroup, 2022), Guanaco (GuanacoGroup, 2023), are primarily
utilized to enhance the language capabilities of our model.

  * •

Visual Question Answering (VQA): This category of datasets includes VQAv2
(Goyal et al., 2017), M3IT (Li et al., 2023e), VizWiz-VQA (Gurari et al.,
2018), OCR-VQA (Mishra et al., 2019), Visual Spatial Reasoning (VSR) (Liu et
al., 2023a), OK-VQA (Marino et al., 2019), Visual Dialog (VisDial) (Das et
al., 2017), A-OKVQA (Schwenk et al., 2022), TextVQA (Singh et al., 2019), GQA
(Hudson & Manning, 2019), along with several in-house VQA datasets. These
datasets facilitate the model handle tasks that require joint visual and
textual reasoning.

  * •

Grounding and Localization: To guide our model to better interpret and
interact with the visual world, we incorporate grounding and localization
datasets during the pretraining stage. The grounding data are sourced from
high-quality in-house data and GoldG Li et al. (2022), and localization data
primarily derived from RefCOCO, RefCOCO+, and RefCOCOg (Yu et al., 2016).
These datasets enable the model to accurately localize and identify specific
objects and regions within images.

To ensure the quality of training during the pretraining stage, we implement a
series of data cleaning procedures on the pretraining datasets. Firstly,
sensitive information is meticulously screened and removed to safeguard
privacy and comply with relevant regulations. Secondly, we apply data
deduplication techniques to eliminate redundant entries, thereby enhancing the
diversity and effectiveness of the dataset. Additionally, we conduct thorough
damage detection and cleaning of image data, removing corrupted images to
ensure the model receives high-quality inputs during pretraining. These
comprehensive cleaning operations not only improve the purity and reliability
of the dataset but also establish a robust foundation for subsequent model
training.

###  3.2 Instruction tuning data

The instruction tuning data employed in our study encompass a wide range of
vision-text and text-only datasets, addressing a variety of challenging
downstream tasks. Additionally, the pie chart on the right side of the Figure
1 visually represents the proportion of each of task data within the whole
dataset.

To be more precise, we divide the entire dataset into the following categories
based on downstream tasks:

  * •

Multi-images task: This type of tasks focuses on visual question answering
involving multiple images. For this purpose, we use the Mantis-Instruct (Jiang
et al., 2024), which covers a diverse array of multi-image skills such as co-
reference, reasoning, comparing, temporal understanding.

  * •

General VQA: The primary distinction from the multi-images task is that
general VQA typically involves questions and answers pertaining to individual
images. For this task, we use datasets such as COCO-QA (Ren et al., 2015),
Visual7W (Zhu et al., 2016), VQAv2 (Goyal et al., 2017), TallyQA (Acharya et
al., 2019), HatefulMemes (Kiela et al., 2020), VQA-RAD (Lau et al., 2018),
LLaVA-Instruct-150K (llava_v1_5_mix665k) (Liu et al., 2024c), M3IT (Li et al.,
2023e), and ALLaVA-Instruct-4V (Chen et al., 2024a).

  * •

Text-only task: This task involves handling not only general textual dialogues
but also mathematical problems and arithmetic calculations. For this purpose,
we primarily rely on publicly available datasets including OpenHermes-2.5
(Teknium, 2023), Dolly (Conover et al., 2023), MetaMathQA (Yu et al., 2023),
MathInstruct (Yue et al., 2023), CamelAIMath (Li et al., 2023b), AtlasMathSets
(AtlasMathSetsGroup, 2023), Goat (Liu & Low, 2023), CoT (Qingyi Si, 2023),
pCLUE (CLUEbenchmarkGroup, 2022), Firefly (Yang, 2023), COIG (Zhang et al.,
2023), and Alpaca (Taori et al., 2023).

  * •

Video-related task: This task primarily designed to enable the model to
understand videos. The specific datasets used include Video Instruction (Maaz
et al., 2024), InternVideo (Wang et al., 2022), and Video-ChatGPT-100K (Maaz
et al., 2024). Note that we select the appropriate frame rate (ranging from
0.5FPS to 30FPS) based on the duration of each video for frame extraction, so
that videos of different length have similar number of frames.

  * •

Chart/figure understanding: The data utilized for this type of task are all
derived from publicly available datasets, including Chart2Text (Obeid & Hoque,
2020), DVQA (Kafle et al., 2018), ChartQA (Masry et al., 2022), FigureQA
(Kahou et al., 2017), MapQA (Chang et al., 2022), and MMC-Instruction (Liu et
al., 2023b).

  * •

OCR-related task: The scope of this task extends beyond mere OCR to encompass
document understanding and text transcription. The datasets employed in our
study include RenderedText (RenderedTextGroup, 2023), DocVQA (Mathew et al.,
2021), TextVQA (Singh et al., 2019), ST-VQA (Biten et al., 2019), VisualMRC
(Tanaka et al., 2021), IAM (Marti & Bunke, 2002), InfoVQA (Mathew et al.,
2022), Diagram image-to-text (image-to textGroup, 2023), RCTW-17 (Shi et al.,
2017), and ReCTS (Yao et al., 2012).

  * •

Table understanding: The datasets utilized for this type of task comprise
TabMWP (Lu et al., 2022b), TAT-QA (Zhu et al., 2021), HiTab (Cheng et al.,
2021), MultiHiertt (Zhao et al., 2022), FinQA (Chen et al., 2021b), WikiSQL
(Zhong et al., 2017), SQA (Iyyer et al., 2017), and WTQ (Pasupat & Liang,
2015).

  * •

Logical reasoning and Math: This part of datasets covers GeomVerse (Kazemi et
al., 2023), CLEVR-Math (Lindström & Abraham, 2022), CLEVR (Johnson et al.,
2017), IconQA (Lu et al., 2021b), RAVEN (Zhang et al., 2019), and Inter-GPs
(Lu et al., 2021a).

  * •

Image captioning: As one of the prevalent downstream tasks for multimodal
models, we primarily utilize the LNarratives (Pont-Tuset et al., 2020),
Screen2Words (Wang et al., 2021), ShareGPT4V (Chen et al., 2023), and some in-
house caption data.

  * •

Image comparison: This task involves employing models to compare two images
and analyze the differences between them. Public datasets such as NLVR2 (Suhr
et al., 2018), GSD (Li et al., 2023a), and Spot the diff (Jhamtani & Berg-
Kirkpatrick, 2018) are utilized for this purpose.

  * •

Knowledge question answering (Knowledge QA): This task involves using models
to answer intellectual questions originated from textbooks or academic texts.
For this purpose, We utilize AI2D (Kembhavi et al., 2016), TQA (Kembhavi et
al., 2017), and ScienceQA (Lu et al., 2022a).

  * •

Code generation: As the name implies, this task focuses on the utilization of
models to generate code. We select WebSight (Laurençon et al., 2024b) and
DaTikz (Belouadi et al., 2023) as the primary datasets for this task.

To avoid data contamination, rigorous processing and validation are conducted
to guarantee that the data in the evaluation benchmarks is entirely excluded
from the pretraining and instruction tuning stages. This strict separation
ensures no overlap between the training and test datasets, upholding the
fairness and credibility of the comparison results.

##  4 Experiments

###  4.1 Benchmarks for Evaluation

We evaluate our models using a variety of publicly available multimodal
benchmarks to assess the performance across different tasks.

1\. General Single-image Datasets:

  * •

MMBench V1.1 (Liu et al., 2023c) (MMBench-CN v1.1 and MMBench-EN v1.1) is a
comprehensive benchmark includes over 3,000 multiple-choice questions covering
20 distinct ability dimensions, including object localization, social
reasoning, and image emotion recognition. It evaluates cognitive abilities and
linguistic competencies in both Chinese and English.

  * •

MMStar (Chen et al., 2024b) evaluates MLLM’s abilities in Coarse Perception
(CP), Fine-grained Perception (FP), Logical Reasoning (LR), Instance Reasoning
(IR), Science & Technology (ST), and Mathematics (MA). It includes 1,500
challenging samples selected by humans, evaluating the model’s understanding
and reasoning with visual content across different complexity levels.

  * •

MMMU (Yue et al., 2024) probes multi-disciplinary competencies with 11.5K
curated questions from college resources. Covers Art & Design, Business,
Science, Health & Medicine, Humanities & Social Science, and Tech &
Engineering. Includes diverse question types for comprehensive evaluation.

  * •

HallusionBench (Guan et al., 2023) assesses the language hallucination and
visual illusion of MLLMs with 455 visual-question control pairs. It includes
346 unique figures and a total of 1129 questions spanning diverse topics and
formats.

  * •

AI2D (Kembhavi et al., 2016) is a dataset for understanding science diagrams,
consisting of over 5,000 diagrams representing grade school science topics.
Each diagram is annotated with constituent segmentations, their relationships
to each other, and their relationships to the diagram canvas.

  * •

MMVet(Yu et al., 2024) evaluates a wide range of abilities, including
recognition, OCR, knowledge retention, language generation, spatial awareness,
and mathematical proficiency. It features 16 tasks for quantitative evaluation
and includes a curated collection of 200 images and 218 samples, each paired
with their corresponding ground truths. MMVet integrates to solve various
complex multimodal tasks.

2\. Math Reasoning Datasets:

  * •

MathVista (Lu et al., 2024) is a comprehensive benchmark that evaluates a
model’s understanding of mathematical and visual tasks by combining 28
multimodal datasets, comprising 9 MathQA datasets and 19 VQA datasets. It
introduces three new datasets (IQTest, FunctionQA, PaperQA) to assess logical
reasoning through puzzle test figures, algebraic reasoning with functional
plots, and scientific reasoning using academic paper figures.

3\. OCR-related Image Understanding Datasets:

  * •

OCRBench (Liu et al., 2024g) is a comprehensive OCR evaluation benchmark
featuring 29 datasets and five key components: text recognition, Scene Text-
Centric VQA, Document-Oriented VQA, KIE, and HMER. It includes 1000 question-
answer pairs, making it a thorough assessment of a model’s OCR capabilities.

4\. General Multi-image Datasets:

  * •

Mantis-Eval (Jiang et al., 2024) is a challenging dataset featuring 217 multi-
image reasoning examples covering various skills such as size perceptions and
weight comparisons. The dataset includes both multiple-choice and short-answer
questions.

  * •

Q-Bench(Wu et al., 2024) assesses the capability of MLLMS in evaluating and
comparing visual quality. It focuses on testing MLLMs’ low-level visual
skills, particularly their ability to assess image quality. The evaluation is
conducted on the Qbench2-A2-pair development set, which presents 1000
multiple-choice questions based on various image contents.

  * •

MileBench (Song et al., 2024) rigorously evaluates MLLMs across diverse
challenges. It comprises two evaluation sets: a diagnostic evaluation
emphasizing long-context recall tasks like needle-in-a-haystack and image
retrieval, and a realistic evaluation simulating real-world conditions with
temporal and semantic multi-image tasks.

5\. General Video Understanding Datasets:

  * •

MVBench (Li et al., 2023d) is a varied benchmark for multimodal video
understanding, encompassing 20 intricate video tasks that demand analysis of
image sequences for precise solutions, rather than relying solely on a single
image.

###  4.2 Single Image Results

As shown in Table 1, we evaluated our model on the OpenCompass multimodal
benchmarks Contributors (2023), including MMBench, MMStar, MMMU, MathVista,
etc. We also compared the evaluation scores of OmChat to other MLLM models on
OpenCompass leaderboard. Our OmChat 8B model demonstrates promising results in
single image inference, outperforming much larger models such as LLaVA-Next-
Yi-34B Liu et al. (2024e), 360VL-70B qihoo360 (2024), CogVLM-17B-Chat Wang et
al. (2023),XVERSE-V xverse (2024), IDEFICS2 Laurençon et al. (2024a) and Yi-VL
34B AI et al. (2024).

Other models such as MiniCPM-V2.5 Hu et al. (2024) and InternLM-
XComposer2-VL-4kHD (internLM-XC-HD) Dong et al. (2024) also show strong
performance in specific benchmarks but do not maintain consistently high
scores across all tasks. Notably, OmChat achieves the highest score on MMBench
(78.8), outperforming other models across several benchmarks.

Table 1: Single image performance.

Model | Params | Avg. | MMBench | MMStar | MMMU | MathVista | Hallusion | AI2D | OCRBench | MMVet
---|---|---|---|---|---|---|---|---|---|---
MiniCPM-V2.5 | 8B | 58.8 | 72.0 | 51.8 | 45.8 | 54.3 | 42.4 | 78.4 | 725 | 52.8
InternLM-XC-HD | 7B | 58.8 | 76.5 | 55.3 | 39.7 | 59.4 | 42.5 | 81.0 | 675 | 48.2
LLaVA-Next-Yi-34B | 34B | 55.0 | 77.8 | 51.6 | 48.8 | 40.4 | 34.8 | 78.9 | 574 | 50.7
IDEFICS2 | 8B | 53.0 | 68.9 | 49.5 | 45.2 | 52.2 | 39.1 | 72.3 | 626 | 34.0
XVERSE-V | 13B | 49.4 | 66.3 | 49.3 | 44.1 | 45.3 | 33.3 | 70.6 | 489 | 37.8
360VL | 70B | 48.2 | 75.0 | 48.1 | 53.4 | 38.0 | 34.8 | 71.9 | 397 | 24.7
CogVLM-17B-Chat | 17B | 47.9 | 58.8 | 39.9 | 37.3 | 35.0 | 35.4 | 63.3 | 590 | 54.5
LLaVA-Next-Vicuna | 13B | 47.6 | 66.5 | 40.4 | 37.3 | 34.1 | 31.8 | 72.2 | 537 | 44.9
LLaVA-Next | 7B | 45.8 | 63.1 | 38.4 | 37.0 | 34.6 | 29.1 | 69.0 | 531 | 42.2
Qwen-VL-Chat | 9B | 45.2 | 59.1 | 34.5 | 37.0 | 34.9 | 36.8 | 63.0 | 488 | 47.3
Yi-VL | 34B | 43.5 | 67.8 | 40.5 | 45.1 | 31.5 | 35.3 | 65.9 | 290 | 32.7
OmChat(Ours) | 8B | 55.9 | 78.8 | 53.8 | 45.9 | 48.3 | 40 | 77.5 | 637 | 39.6

###  4.3 Long Context Results

####  4.3.1 Text Needle-in-the-Haystack

Figure 3: Text needle retrieval performance of OmChat.

As illustrated in Figure 3, we evaluate our 512k OmChat model on the widely
used Needle in a Haystack task (HaystackGroup, 2023). Specifically, the model
is evaluated with a single needle setting, where it needs to retrieve and
answer a question based on a fact or statement randomly placed within a long
context. OmChat demonstrates nearly perfect performance across context lengths
ranging from 4k to 256k tokens, highlighting the success and effectiveness of
our long context training strategy. The minor performance decrease observed at
the 512k context length is likely due to fine-tuning the model with much
shorter contexts. Incorporating more long context data during the fine-tuning
stage should help improve this performance.

####  4.3.2 Temporal Visual Needle-in-the-Haystack

(a) LLaVa-1.5

(b) GPT-4o

(c) Random

(d) OmChat

Figure 4: TV Needle performance on LLaVa-1.5, GPT-4o, Random, and OmChat.

We propose a benchmark dataset, named as Temporal Visual Needle in a Haystack
(TV Needle), to assess the MLLMs’ ability to comprehend temporal visual
details within videos. Inspired by “Needle in Haystack” (gkamradt, 2023), we
extend the concept of the needle task from text-based to a multimodal version.
The TV Needle dataset retains the original objective of locating the “text
needle” information from long documents but introduces an additional challenge
by requiring the identification of visual needles, including temporal
information from long videos.

Specially, the long videos sourced from ActivityNet (Caba Heilbron et al.,
2015) are utilized as the “haystack” for the TV Needle dataset. A subset of 10
to 16 videos is randomly selected and subsequently concatenated to create a
single testing video. This process is repeated to yield a total of 20 distinct
testing videos. This methodology ensures a comprehensive and varied evaluation
of the system’s performance. These testing videos have duration ranging from
938 to 1,461 seconds. To create the “needle” for the task, emojis are posted
into the videos. Three emojis are randomly selected from a pool of 200 emojis
and inserted into three consecutive frames within the video. This process
introduces a temporal multimodal challenge where the model need to identify
and locate these emoji sequences within the video content. Figure 5 provides
an illustrative example of the TV Needle dataset, showcasing how the emojis
are integrated into the video frames. Following the “Needle in Haystack”
approach (gkamradt, 2023), emojis are inserted at varying depths within the
video frames, ranging from 0% to 100%. We create subsets of varying number of
frames, ranging from a minimum of 13 to a maximum of 444. With each frame
contributing 576 tokens, this corresponds an input length of 8k to 256k tokens
for the MLLM being evaluated. By incorporating emojis into the video frames at
varying depths and challenging the model to locate and identify these
sequences, the TV Needle dataset offers a unique and engaging task that
measure the ability of MLLMs to understand and extract temporal visual details
from videos.

In Figure 4, we present a comparative analysis of the performance of OmChat,
LLaVa-1.5, and GPT-4o. It is noteworthy that GPT-4o, despite being the top
performer, does not perfectly solve the task. Its performance begins to
significantly deteriorate on videos comprising 222 frames (equivalent to 128k
tokens), and it fails to process 444 frames (256k tokens) due to API
constraints. Our model, OmChat, demonstrates consistent and commendable
performance across all input lengths and needle positions. It surpasses both
the random baseline and LLaVa-1.5 by a substantial margin. We attribute this
robust capability of understanding long videos to our progressive training
strategy, which gradually equipped our model with long context capability.

Figure 5: Illustration of the TV Needle. Three emojis are strategically
embedded within three consecutive frames amidst a plethora of frames.

####  4.3.3 Multi-Image and Video Benchmarks

Table 2: Evaluation results on multi-image and video benchmarks

Model | Model Type | Mantis-Eval | Q-Bench | MileBench Real | MileBench Diag | MVBench | Avg
---|---|---|---|---|---|---|---
LLaVA-1.5-7B | Single Image | 31.3 | 49.3 | 38.0 | 2.1 | 36.0 | 31.4
LLaVA-1.6-7B | Single Image | 45.6 | 54.8 | 38.1 | 7.2 | 40.9 | 37.3
Qwen-VL-Chat | Single Image | 39.2 | 45.9 | 39.1 | 37.2 | 42.2 | 40.7
VILA | Multi-image | 51.2 | 45.7 | 44.4 | 12.8 | 49.4 | 40.7
Mantis-CLIP | Multi-image | 55.8 | 66.0 | 47.5 | 19.9 | 48.3 | 47.5
OmChat-8B(Ours) | Multi-image | 56.2 | 74.8 | 51.7 | 41.7 | 51.2 | 55.1
GPT-4V | Multi-image | 62.7 | 76.5 | 53.0 | 99.4 | 43.5 | 67.0

In Table 2, we compare our model, OmChat, on multi-image and video benchmarks
with other state-of-the-art (SOTA) open-source and closed-source models. For
MVBench, OmChat is evaluated using 16 frames to get the best performance.

Firstly, Comparing with models trained on single-image tasks, such as
LLaVA-1.5 and Qwen-VL-Chat Bai et al. (2023a), OmChat demonstrates a superior
performance across all benchmarks, scoring more than 10 points higher on
average. Additionally, when compared with SOTA multi-image models like Mantis
Jiang et al. (2024) and VILA Lin et al. (2024a), OmChat still outperforms
them, likely due to our carefully designed multi-image and video data format.

Furthermore, when compared with the closed-source model GPT-4V, OmChat
achieves a better score on MVBench, which requires a strong temporal video
understanding capability.

##  5 Ablation & Analysis

###  5.1 Multi-image and Video Format Ablation

Table 3: Ablation study on multi-image data format. We compare five different
data formats for handling multi-image and video data, labeled from F0 to F4.
The symbols <im_start> and <im_end> are special tokens used as image
delimiters. The term “image/frame” indicates that the text prompt “image” is
used for multi-image data, while the text prompt “frame” is used for video
data.

|  Format |  Mantis-Eval |  Q-Bench |  MileBench Real. |  MileBench Diag. |  MVBench |  Average
---|---|---|---|---|---|---|---
F0 |  <image><image>... |  41.0 |  47.3 |  45.8 |  32.5 |  42.0 |  41.7
F1 |  <im_start><image><im_end> |  39.2 |  50.8 |  45.9 |  34.7 |  44.7 |  43.1
F2 |  image {i}: <image> |  52.1 |  73.5 |  48.0 |  21.8 |  47.5 |  48.6
F3 |  image {i}: <im_start><image><im_end> |  56.2 |  73.8 |  48.9 |  32.8 |  48.6 |  52.1
F4 |  image/frame {i}: <im_start><image><im_end> |  56.2 |  74.8 |  51.7 |  41.7 |  50.2 |  54.9

As shown in Table 3, we conduct an ablation study to explore different input
formats for improving the model’s ability to handle multi-image and video
scenarios. Various combinations of text prompts and special tokens are tested,
with five different input formats labeled from F0 to F4. For this experiment,
we use the same multi-image and single-image fine-tuning datasets and add an
extra video instruction dataset for format F4.

Format F0 simply concatenates all the image tokens together, similar to how
most single-image models operate. Despite this straightforward approach, our
model using format F0 achieved higher scores on multi-image and video
benchmarks than other single image models, benefiting from the utilization of
interleaved data and multi-image and video data during the pretraining phase.

According to the experiment results, input formats F3 and F4, which use the
text prompt “image {i}” and special tokens <im_start> and <im_end> as image
delimiters, proves to be effective for multi-image scenarios. Among these,
format F4 delivers the best performance across all benchmarks.

A detailed analysis of the benchmark results for format F4 reveals that the
text prompt “image {i}” performs better for semantic multi-image tasks such as
Mantis-Eval and Q-Bench. Additionally, the text prompt “frame {i}” yields high
scores on temporal video tasks, such as certain components in MileBench
Realistic Evaluation and MVBench. This demonstrates that the combination of
text prompts “image {i}” and “frame {i}” is an effective approach to help the
model understand both multi-image and video scenarios.

###  5.2 Data Selection Ablation

Table 4: Evaluation results on continuous training for instruction tuning Ddata selection. The evaluation results for models trained with varying data portions using a continuous training strategy are presented. The data portion size is determined based on the total number of tokens in the answers from the fine-tuning dataset. The total average scores encompass performance metrics from both text benchmarks and multimodal benchmarks. Data Portion Size | Total Average
---|---
10M | 56.05
20M | 57.09
40M | 56.59
60M | 56.64
80M | 56.01

To validate the effectiveness of our continuous training strategy and select
an optimal combination of fine-tuning datasets, we conduct fine-tuning
experiments using the approach introduced in Section 2.5. Initially, a large
dataset comprising 0.2B tokens in answers is sampled based on manually
designed percentages. This large dataset is then divided into multiple data
portions, each maintaining the designed percentages.

We use our language model, OmBase, as the base for MLLM and perform a data
selection ablation. After applying the continuous training strategy to these
data portions, we evaluate the fine-tuned models on multiple multimodal and
text-only benchmarks, using a total average score to compare performances.

As shown in Table 4, the total average score increases from the data portion
of 10M to 20M tokens, indicating that adding more samples from each dataset
improves model performance. However, from the data portion of 40M to 80M
tokens, the total average score decreases. Detailed analysis of each
benchmark’s scores reveals that the decline is primarily in text-only
benchmarks. This drop in text performance may be due to excessive training
steps on multimodal tasks. Conversely, the multimodal scores only slightly
increase with data portions from 40M to 80M tokens.Therefore, considering the
trade-off between multimodal and text-only capabilities, we select the data
portion size of 20M tokens as the best instruction tuning data combination.

###  5.3 Qualitative Examples

Figure 6: Qualitative comparison samples of OmChat against open source models.

In our study, we present a series of examples derived from benchmarks, which
include multi-image or video data (displayed via extracted frames) as shown in
Figure 6. The OmChat model demonstrates superior performance in comparison to
other robust MLLMs. Its proficiency is particularly evident in its ability to
predict future actions in a video, make judgements rooted in common sense, and
compare visual features across consecutive images. The emergent capability of
the OmChat model to understand long-context visuals broadens the scope of its
potential applications significantly. This underscores the model’s versatility
and effectiveness, positioning it as a leading tool in the field of multimodal
large language models. More examples of OmChat for various visual tasks are
depicted in Figure 7 and Figure 8 in Appendix.

##  6 Conclusion

This paper introduced OmChat, a multimodal model designed for handling long
contexts and video understanding tasks. OmChat’s active progressive multimodal
pretraining strategy, combined with high-quality supervised fine-tuning
datasets and support for any-aspect high image resolutions, ensures
exceptional performance across various benchmarks.

OmChat performs outstandingly in tasks involving multiple images and videos,
managing complex visual data effectively and supporting a context length of up
to 512K tokens. Our analysis highlights the importance of higher image
resolutions, progressive multimodal pretraining, and high-quality training
data in achieving state-of-the-art performance.

OmChat sets a new benchmark for multimodal large language models. Future work
will enhance its capabilities, explore efficient training techniques, and
expand its application to more multimodal tasks.

## References

  * Acharya et al. (2019) Manoj Acharya, Kushal Kafle, and Christopher Kanan.  Tallyqa: Answering complex counting questions.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 8076–8084, 2019.
  * AI et al. (2024) 01\. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.  Yi: Open foundation models by 01.ai, 2024.
  * AtlasMathSetsGroup (2023) AtlasMathSetsGroup.  Atlasmathsets: An open dataset for mathematical computations, 2023.  URL https://huggingface.co/datasets/AtlasUnified/atlas-math-sets.
  * BAAI (2023) BAAI.  An open source large-scale dataset.  https://data.baai.ac.cn/details/WuDaoCorporaText, 2023.
  * Bai et al. (2023a) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023a.
  * Bai et al. (2023b) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  2023b.
  * BELLEGroup (2023) BELLEGroup.  Belle: Be everyone’s large language model engine.  https://github.com/LianjiaTech/BELLE, 2023.
  * Belouadi et al. (2023) Jonas Belouadi, Anne Lauscher, and Steffen Eger.  Automatikz: Text-guided synthesis of scientific vector graphics with tikz.  _arXiv preprint arXiv:2310.00367_ , 2023.
  * Biten et al. (2019) Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas.  Scene text visual question answering.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 4291–4301, 2019.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ , pp. 961–970, 2015.
  * Chang et al. (2022) Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao.  Mapqa: A dataset for question answering on choropleth maps.  _arXiv preprint arXiv:2211.08545_ , 2022.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 3558–3568, 2021.
  * Chen et al. (2024a) Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang.  Allava: Harnessing gpt4v-synthesized data for a lite vision-language model.  _arXiv preprint arXiv:2402.11684_ , 2024a.
  * Chen et al. (2023) Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  _arXiv preprint arXiv:2311.12793_ , 2023.
  * Chen et al. (2024b) Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al.  Are we on the right way for evaluating large vision-language models?  _arXiv preprint arXiv:2403.20330_ , 2024b.
  * Chen et al. (2021a) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  2021a.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chen et al. (2021b) Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al.  Finqa: A dataset of numerical reasoning over financial data.  _arXiv preprint arXiv:2109.00122_ , 2021b.
  * Cheng et al. (2021) Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang.  Hitab: A hierarchical table dataset for question answering and natural language generation.  _arXiv preprint arXiv:2108.06712_ , 2021.
  * Chng et al. (2019) Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al.  Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pp. 1571–1576. IEEE, 2019.
  * CLUEbenchmarkGroup (2022) CLUEbenchmarkGroup.  pclue: Large-scale prompt-based dataset for multi-task and zero-shot learning in chinese.  https://github.com/CLUEbenchmark/pCLUE, 2022.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Computer (2023) Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset.  https://github.com/togethercomputer/RedPajama-Data, 2023.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.  _Company Blog of Databricks_ , 2023.
  * Contributors (2023) OpenCompass Contributors.  Opencompass: A universal evaluation platform for foundation models.  https://github.com/open-compass/opencompass, 2023.
  * Das et al. (2017) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra.  Visual dialog.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 326–335, 2017.
  * DeepCtrl (2024) DeepCtrl.  Deepctrl-sft-data: An open dataset cleaned from open source.  https://modelscope.cn/datasets/deepctrl/deepctrl-sft-data, 2024.
  * Dong et al. (2024) Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al.  Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd.  _arXiv preprint arXiv:2404.06512_ , 2024.
  * gkamradt (2023) gkamradt.  Llmtest needle in a haystack - pressure testing llms.  https://github.com/gkamradt/LLMTest_NeedleInAHaystack, 2023.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 6904–6913, 2017.
  * Guan et al. (2023) Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou.  Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models, 2023.
  * GuanacoGroup (2023) GuanacoGroup.  An open source multilingual dataset.  https://huggingface.co/datasets/JosephusCheung/GuanacoDataset, 2023.
  * Gurari et al. (2018) Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3608–3617, 2018.
  * Gurari et al. (2020) Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya.  Captioning images taken by people who are blind.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16_ , pp. 417–434. Springer, 2020.
  * HaystackGroup (2023) HaystackGroup.  Needle in a haystack - pressure testing llms.  https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main, 2023.
  * (36) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  _Sort_ , 2(4):0–6.
  * Hu et al. (2024) Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al.  Minicpm: Unveiling the potential of small language models with scalable training strategies.  _arXiv preprint arXiv:2404.06395_ , 2024.
  * Huang et al. (2024) Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al.  C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Hudson & Manning (2019) Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 6700–6709, 2019.
  * image-to textGroup (2023) image-to textGroup.  image-to-text: An open dataset, 2023.  URL https://huggingface.co/datasets/Kamizuru00/diagram_image_to_text.
  * Iyyer et al. (2017) Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang.  Search-based neural structured learning for sequential question answering.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1821–1831, 2017.
  * Jhamtani & Berg-Kirkpatrick (2018) Harsh Jhamtani and Taylor Berg-Kirkpatrick.  Learning to describe differences between pairs of similar images.  _arXiv preprint arXiv:1808.10584_ , 2018.
  * Jiang et al. (2024) Dongfu Jiang, Xuan He, Huaye Zeng, Con Wei, Max Ku, Qian Liu, and Wenhu Chen.  Mantis: Interleaved multi-image instruction tuning.  _arXiv preprint arXiv:2405.01483_ , 2024.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2901–2910, 2017.
  * Kafle et al. (2018) Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan.  Dvqa: Understanding data visualizations via question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5648–5656, 2018.
  * Kahou et al. (2017) Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio.  Figureqa: An annotated figure dataset for visual reasoning.  _arXiv preprint arXiv:1710.07300_ , 2017.
  * Kazemi et al. (2023) Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut.  Geomverse: A systematic evaluation of large models for geometric reasoning.  _arXiv preprint arXiv:2312.12241_ , 2023.
  * Kembhavi et al. (2016) Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi.  A diagram is worth a dozen images.  In _European conference on computer vision_ , pp. 235–251. Springer, 2016.
  * Kembhavi et al. (2017) Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi.  Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern recognition_ , pp. 4999–5007, 2017.
  * Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  _Advances in neural information processing systems_ , 33:2611–2624, 2020.
  * Lau et al. (2018) Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman.  A dataset of clinically generated visual questions and answers about radiology images.  _Scientific data_ , 5(1):1–10, 2018.
  * Laurençon et al. (2024a) Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.  What matters when building vision-language models?  _arXiv preprint arXiv:2405.02246_ , 2024a.
  * Laurençon et al. (2024b) Hugo Laurençon, Léo Tronchon, and Victor Sanh.  Unlocking the conversion of web screenshots into html code with the websight dataset.  _arXiv preprint arXiv:2403.09029_ , 2024b.
  * Li et al. (2023a) Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu.  Mimic-it: Multi-modal in-context instruction tuning.  _arXiv preprint arXiv:2306.05425_ , 2023a.
  * Li et al. (2023b) Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.  Camel: Communicative agents for” mind” exploration of large language model society.  _Advances in Neural Information Processing Systems_ , 36:51991–52008, 2023b.
  * Li et al. (2023c) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin.  Cmmlu: Measuring massive multitask language understanding in chinese.  _arXiv preprint arXiv:2306.09212_ , 2023c.
  * Li et al. (2023d) Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao.  Mvbench: A comprehensive multi-modal video understanding benchmark, 2023d.
  * Li et al. (2023e) Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al.  M3 it: A large-scale dataset towards multi-modal multilingual instruction tuning.  _arXiv preprint arXiv:2306.04387_ , 2023e.
  * Li et al. (2022) Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al.  Grounded language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10965–10975, 2022.
  * Lin et al. (2024a) Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han.  Vila: On pre-training for visual language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 26689–26699, 2024a.
  * Lin et al. (2024b) Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al.  Rho-1: Not all tokens are what you need.  _arXiv preprint arXiv:2404.07965_ , 2024b.
  * Lindström & Abraham (2022) Adam Dahlgren Lindström and Savitha Sam Abraham.  Clevr-math: A dataset for compositional language, visual and mathematical reasoning.  _arXiv preprint arXiv:2208.05358_ , 2022.
  * Liu et al. (2023a) Fangyu Liu, Guy Emerson, and Nigel Collier.  Visual spatial reasoning.  _Transactions of the Association for Computational Linguistics_ , 11:635–651, 2023a.
  * Liu et al. (2023b) Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu.  Mmc: Advancing multimodal chart understanding with large-scale instruction tuning.  _arXiv preprint arXiv:2311.10774_ , 2023b.
  * Liu et al. (2024a) Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel.  World model on million-length video and language with blockwise ringattention, 2024a.  URL https://arxiv.org/abs/2402.08268.
  * Liu et al. (2024b) Hao Liu, Matei Zaharia, and Pieter Abbeel.  Ring attention with blockwise transformers for near-infinite context.  _International Conference on Learning Representations_ , 2024b.
  * Liu et al. (2024c) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 26296–26306, 2024c.
  * Liu et al. (2024d) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, 2024d.
  * Liu et al. (2024e) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024e.  URL https://llava-vl.github.io/blog/2024-01-30-llava-next/.
  * Liu et al. (2024f) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024f.
  * Liu & Low (2023) Tiedong Liu and Bryan Kian Hsiang Low.  Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks.  _arXiv preprint arXiv:2305.14201_ , 2023.
  * Liu et al. (2023c) Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.  Mmbench: Is your multi-modal model an all-around player?  _arXiv preprint arXiv:2307.06281_ , 2023c.
  * Liu et al. (2024g) Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai.  On the hidden mystery of ocr in large multimodal models, 2024g.
  * Liu et al. (2022) Yulong Liu, Guibo Zhu, Bin Zhu, Qi Song, Guojing Ge, Haoran Chen, GuanHui Qiao, Ru Peng, Lingxiang Wu, and Jinqiao Wang.  Taisu: A 166m large-scale high-quality dataset for chinese vision-language pre-training.  _Advances in Neural Information Processing Systems_ , 35:16705–16717, 2022.
  * Lu et al. (2021a) Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu.  Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning.  _arXiv preprint arXiv:2105.04165_ , 2021a.
  * Lu et al. (2021b) Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu.  Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.  _arXiv preprint arXiv:2110.13214_ , 2021b.
  * Lu et al. (2022a) Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  _Advances in Neural Information Processing Systems_ , 35:2507–2521, 2022a.
  * Lu et al. (2022b) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  _arXiv preprint arXiv:2209.14610_ , 2022b.
  * Lu et al. (2024) Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  In _International Conference on Learning Representations (ICLR)_ , 2024.
  * Maaz et al. (2024) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)_ , 2024.
  * Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pp. 3195–3204, 2019.
  * Marti & Bunke (2002) U-V Marti and Horst Bunke.  The iam-database: an english sentence database for offline handwriting recognition.  _International journal on document analysis and recognition_ , 5:39–46, 2002.
  * Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  _arXiv preprint arXiv:2203.10244_ , 2022.
  * Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.  Docvqa: A dataset for vqa on document images.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pp. 2200–2209, 2021.
  * Mathew et al. (2022) Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar.  Infographicvqa.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1697–1706, 2022.
  * Mishra et al. (2019) Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In _2019 international conference on document analysis and recognition (ICDAR)_ , pp. 947–952. IEEE, 2019.
  * Obeid & Hoque (2020) Jason Obeid and Enamul Hoque.  Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model.  _arXiv preprint arXiv:2010.09142_ , 2020.
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  _Advances in neural information processing systems_ , 24, 2011.
  * Pablo Montalvo (2024) Ross Wightman Pablo Montalvo.  pdfa-eng-wds dataset, 2024.  URL https://huggingface.co/datasets/pixparse/pdfa-eng-wds.
  * Pasupat & Liang (2015) Panupong Pasupat and Percy Liang.  Compositional semantic parsing on semi-structured tables.  _arXiv preprint arXiv:1508.00305_ , 2015.
  * Pont-Tuset et al. (2020) Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.  Connecting vision and language with localized narratives.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pp. 647–664. Springer, 2020.
  * qihoo360 (2024) qihoo360.  360vl-70b, 2024.  URL https://huggingface.co/qihoo360/360VL-70B.
  * Qingyi Si (2023) Zheng Lin Qingyi Si.  Alpaca-cot: An instruction fine-tuning platform with instruction data collection and unified large language models interface.  https://github.com/PhoebusSi/alpaca-CoT, 2023.
  * Ren et al. (2015) Mengye Ren, Ryan Kiros, and Richard Zemel.  Exploring models and data for image question answering.  _Advances in neural information processing systems_ , 28, 2015.
  * RenderedTextGroup (2023) RenderedTextGroup.  Renderedtext: An open dataset, 2023.  URL https://huggingface.co/datasets/wendlerc/RenderedText.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schwenk et al. (2022) Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In _European conference on computer vision_ , pp. 146–162. Springer, 2022.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2556–2565, 2018.
  * She et al. (2024) Qi She, Junwen Pan, Xin Wan, Rui Zhang, Dawei Lu, and Kai Huang.  Mammothmoda: Multi-modal large language model.  _arXiv preprint arXiv:2406.18193_ , 2024.
  * Shi et al. (2017) Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai.  Icdar2017 competition on reading chinese text in the wild (rctw-17).  In _2017 14th iapr international conference on document analysis and recognition (ICDAR)_ , volume 1, pp. 1429–1434. IEEE, 2017.
  * Sidorov et al. (2020) Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , pp. 742–758. Springer, 2020.
  * Singh et al. (2019) Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8317–8326, 2019.
  * Song et al. (2024) Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang.  Milebench: Benchmarking mllms in long context.  _arXiv preprint arXiv:2404.18532_ , 2024.
  * Suhr et al. (2018) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi.  A corpus for reasoning about natural language grounded in photographs.  _arXiv preprint arXiv:1811.00491_ , 2018.
  * Sun et al. (2019) Yipeng Sun, Jiaming Liu, Wei Liu, Junyu Han, Errui Ding, and Jingtuo Liu.  Chinese street view text: Large-scale chinese text reading with partially supervised learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9086–9095, 2019.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _arXiv preprint arXiv:2210.09261_ , 2022.
  * Tanaka et al. (2021) Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida.  Visualmrc: Machine reading comprehension on document images.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 13878–13888, 2021.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.
  * Teknium (2023) Teknium.  Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023.  URL https://huggingface.co/datasets/teknium/OpenHermes-2.5.
  * Wang et al. (2021) Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li.  Screen2words: Automatic mobile ui summarization with multimodal learning.  In _The 34th Annual ACM Symposium on User Interface Software and Technology_ , pp. 498–510, 2021.
  * Wang et al. (2023) Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  Cogvlm: Visual expert for pretrained language models.  _arXiv preprint arXiv:2311.03079_ , 2023.
  * Wang et al. (2022) Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al.  Internvideo: General video foundation models via generative and discriminative learning.  _arXiv preprint arXiv:2212.03191_ , 2022.
  * Wang et al. (2024) Zihan Wang, Xinzhang Liu, Shixuan Liu, Yitong Yao, Yuyao Huang, Zhongjiang He, Xuelong Li, Yongxiang Li, Zhonghao Che, Zhaoxi Zhang, et al.  Telechat technical report.  _arXiv preprint arXiv:2401.03804_ , 2024.
  * Wu et al. (2024) Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin.  Q-bench: A benchmark for general-purpose foundation models on low-level vision.  In _ICLR_ , 2024.
  * Xiao et al. (2024) Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Xun Zhou, and Haoyuan Guo.  Seeing the image: Prioritizing visual correlation by contrastive alignment.  _arXiv preprint arXiv:2405.17871_ , 2024.
  * xverse (2024) xverse.  Xverse-v-13b, 2024.  URL https://huggingface.co/xverse/XVERSE-V-13B.
  * Yang (2023) Jianxin Yang.  Firefly: Chinese dialogic large language models.  https://github.com/yangjianxin1/Firefly, 2023.
  * Yao et al. (2012) Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu.  Detecting texts of arbitrary orientations in natural images.  In _2012 IEEE conference on computer vision and pattern recognition_ , pp. 1083–1090. IEEE, 2012.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78, 2014.
  * Yu et al. (2016) Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pp. 69–85. Springer, 2016.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint arXiv:2309.12284_ , 2023.
  * Yu et al. (2024) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  In _International conference on machine learning_. PMLR, 2024.
  * Yuan et al. (2019) Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min Hu.  A large chinese text dataset in the wild.  _Journal of Computer Science and Technology_ , 34:509–521, 2019.
  * Yue et al. (2023) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Yue et al. (2024) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  In _Proceedings of CVPR_ , 2024.
  * Zhang et al. (2019) Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu.  Raven: A dataset for relational and analogical visual reasoning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 5317–5327, 2019.
  * Zhang et al. (2023) Ge Zhang, Yemin Shi, Ruibo Liu, Ruibin Yuan, Yizhi Li, Siwei Dong, Yu Shu, Zhaoqun Li, Zekun Wang, Chenghua Lin, Wenhao Huang, and Jie Fu.  Chinese open instruction generalist: A preliminary release, 2023.
  * Zhao et al. (2022) Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang.  Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data.  _arXiv preprint arXiv:2206.01347_ , 2022.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _arXiv preprint arXiv:1709.00103_ , 2017.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua.  Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance.  _arXiv preprint arXiv:2105.07624_ , 2021.
  * Zhu et al. (2024) Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal c4: An open, billion-scale corpus of images interleaved with text.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Zhu et al. (2016) Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei.  Visual7w: Grounded question answering in images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4995–5004, 2016.

##  Appendix A Appendix

Figure 7: Examples of OmChat Handling Various Tasks Figure 8: More Examples of
OmChat Handling Various Tasks