# Ovis: Structural Embedding Alignment for
Multimodal Large Language Model

Shiyin Lu1 Yang Li1 Qing-Guo Chen1 Zhao Xu1
Weihua Luo1 Kaifu Zhang1 Han-Jia Ye2,3
1AI Business, Alibaba Group 2School of Artificial Intelligence, Nanjing
University
3National Key Laboratory for Novel Software Technology, Nanjing University
Corresponding author, email<EMAIL_ADDRESS>
###### Abstract

Current Multimodal Large Language Models (MLLMs) typically integrate a pre-
trained LLM with another pre-trained vision transformer through a connector,
such as an MLP, endowing the LLM with visual capabilities. However, the
misalignment between two embedding strategies in MLLMs — the structural
textual embeddings based on an embedding look-up table and the continuous
embeddings generated directly by the vision encoder — makes challenges for a
more seamless fusion of visual and textual information. We propose Ovis, a
novel MLLM architecture designed to structurally align visual and textual
embeddings. Ovis integrates an additional learnable visual embedding table
into the visual encoder’s process. To capture rich visual semantics, each
image patch indexes the visual embedding table multiple times, resulting in a
final visual embedding that is a probabilistic combination of the indexed
embeddings. This structural approach mirrors the method used for generating
textual embeddings. Empirical evaluations on various multimodal benchmarks
demonstrate that Ovis outperforms open-source MLLMs of similar parameter
scales and even surpasses the proprietary model Qwen-VL-Plus overall. These
results highlight the potential of Ovis’ structured visual representation for
advancing MLLM architectural design and promoting more effective multimodal
learning. Both the source code and the training dataset of Ovis will be made
publicly available.

##  1 Introduction

The development of Large Language Models (LLMs) is advancing rapidly [64, 65,
6, 60, 58, 59], illuminating the path toward Artificial General Intelligence
(AGI). These sophisticated models excel at understanding and generating text
with remarkable proficiency [76, 77, 14, 74]. However, to approach the
complexity and versatility of human intelligence, LLMs must transcend mere
textual comprehension. The ability to interpret and understand visual
information becomes a critical feature on this journey toward AGI.
Consequently, there has been a surge of interest in developing Multimodal
Large Language Models (MLLMs) — models that meld the power of language
comprehension and visual perception [37, 38, 18, 91, 48, 41, 40, 4, 86].

Instead of directly training the entire MLLMs, current open-source MLLMs
primarily derive their visual ability from a pre-trained LLM and a pre-trained
vision encoder. The visual and textual components have different tokenization
and embedding strategies. Textual embeddings are indexed from the LLM’s
embedding look-up table, where each “word” is mapped to an embedding, via one-
hot textual tokens. In contrast, visual embeddings are generated directly by
the vision encoder in an unstructured manner. To align the dimensions between
these two types of embeddings, cross-modal connectors such as MLPs project
embeddings into a joint space, allowing all embeddings to serve as inputs to
the LLM [48, 47, 49, 41, 40]. Although this architecture only aligns the
dimensions of visual and textual embeddings, it has shown promising
performance across various vision-language tasks. Nevertheless, the inherent
discrepancy in tokenization and embedding strategies may lead to a potential
limitation in the connector-based architecture, so an intuitive question is

> Could we achieve further improvement in MLLMs if we generate visual
> embeddings in a structured manner to match the textual embedding strategy in
> LLMs?

We propose a novel MLLM architecture, dubbed “Ovis”, which assimilates the
insights from LLMs to establish structured embeddings of visual input. As
illustrated in Figure 1, Ovis introduces an additional learnable visual
embedding look-up table to transform continuous visual tokens, thus
paralleling the structural integrity of its textual counterpart. Figure 2
demonstrates that Ovis outperforms open-source MLLMs within the same parameter
tier across various benchmarks, and Ovis-14B also surpasses the high-resource
proprietary model Qwen-VL-Plus overall.

Figure 1: Comparison between different embedding strategies in MLLM. In the
connector-based approach (a), the connector transforms the visual embeddings
into the same dimensional as the textual embedding, where the latter is
indexed from a textual embedding table. As illustrated in (b), our Ovis
leverages an additional visual embedding table to produce structural visual
embeddings and align the embedding strategies of two modalities.

(a) 7B tier

(b) 14B tier and Qwen-VL-Plus

Figure 2: Ovis outperforms open-source MLLMs within the same parameter tier in
various benchmarks, and Ovis-14B also surpasses the high-resource proprietary
model Qwen-VL-Plus overall.

In particular, Ovis incorporates a visual embedding table whose rows
correspond to unique visual words, representing distinct visual patterns.
Given the continuous token of a visual patch output by the visual encoder
[22], Ovis first maps the token into a probabilistic token, revealing its
similarity among the entire visual vocabulary set. The probabilistic token
captures the rich semantics within a single visual patch, which may contain
patterns from multiple visual words, effectively treating the visual token as
if it were sampled from the visual embedding table based on the distribution.
Ovis subsequently indexes the visual embedding table multiple times based on
the probabilistic token, resulting in a final visual embedding that is a
combination of the indexed embeddings, i.e., the expectation of embeddings
over the whole embedding table. Therefore, Ovis aligns the visual embedding
strategy with the structured nature of its textual counterpart.

The optimization of the visual embedding table and the parameters for
generating the probabilistic tokens significantly influence the performance of
the MLLM. Instead of using an additional autoencoder with vector quantization
over images and various other losses, as utilized in previous methods [78, 23,
33], Ovis leverages a joint textual generation loss and optimizes the
parameters in a three-stage manner. This learning process of Ovis avoids the
risk of falling short in vision-language tasks due to the absence of textual
guidance.

We implement Ovis with open-source vision transformer and LLM models as
backbones and evaluate its performance in diverse multimodal benchmarks. The
outcomes demonstrate that Ovis outperforms popular open-source MLLMs within
the same parameter tier in the majority of these benchmarks. Specifically,
Ovis-8B exhibits a large margin over its competitors, and Ovis-14B
consistently surpasses the compared open-source MLLMs. Impressively, Ovis-14B
also performs better than the high-resource proprietary model Qwen-VL-Plus [4]
overall, and its performance is even on par with the stronger proprietary
model Qwen-VL-Max [4] in the general multimodal benchmarks MMStar [12] and
MMBench [50] and several specialized multimodal benchmarks including MathVista
[55], HallusionBench [46], and RealWorldQA [84]. These results underscore the
superiority and potential of the Ovis architecture. We believe that the
demonstrated effectiveness and advantages of Ovis will enhance further
investigations into MLLM architectural designs, moving beyond the confines of
the connector-based architecture.

##  2 Related Work

#### Large Language Models.

In recent years, the development of Large Language Models (LLMs) has
significantly advanced the field of natural language processing. The debut of
GPT-3 [6] marked a notable surge in performance, especially in few-shot and
zero-shot learning scenarios, underscoring the substantial promise of LLMs.
This potential was further demonstrated by subsequent enhancements in models
such as ChatGPT [58], GPT-4 [59], Gemini [75, 68], and Claude [2].
Concurrently, open-source models have been rapidly evolving, including the
LLaMA [76, 77] series, Vicuna [14], Baichuan [85], Qwen [3], Mistral [32], and
Yi [87]. Notably, the open-source models Llama3 [56] and Mistral-MOE [57] have
approached and, in some cases, surpassed the performance of closed-source
models. Despite these advancements, LLMs inherently lack the capability to
process or interpret multimodal data, limiting their application in scenarios
requiring an understanding of more than just textual information.

#### Multimodal Large Language Models.

Multimodal Large Language Models (MLLMs) enhance the capabilities of LLMs by
not only understanding and generating text but also interpreting and relating
visual elements to textual descriptions. Most open-source MLLMs consist of
several components, namely a vision encoder [66, 73, 24, 89], a connector, and
an LLM. The type of the connector can be roughly divided into three
categories. The cross-attention-based methods isolate and integrate visual and
text modalities within the LLM, as seen in models like Flamingo [1] and CogVLM
[79]. The query-based methods query visual embeddings via a transformer-like
architecture and send the obtained visual embeddings along with the text to
the LLM, exemplified by Blip-2 [38], Instruct-Blip [18], and Qwen-VL [4]. The
projection-based methods directly project the visual embeddings, align them to
the text modality, and uniformly feed the mixed embeddings into the LLM for
understanding and generation. This approach is used by models such as LLaVA
[48], Mini-GPT4 [91], DeepSeek-VL [51], and Mini-Gemini [40]. In addition to
architecture design, current MLLM research focuses on high-resolution
capabilities [44, 21, 80], miniaturization of MLLMs [31, 15, 42], specialized
models (e.g., medical MLLMs [36], document MLLMs [10, 30]), and the
integration of other modalities [83, 61, 52]. Our Ovis serves as a new MLLM
architecture that departs from the connector-based framework and involves a
novel visual tokenizer for structured visual embeddings.

#### Visual Tokenization.

Tokenizing visual input has been explored in various visual tasks [35]. VQVAE
[78] encodes visual input into discrete latent variables, combining the
principles of variational autoencoders with vector quantization. This approach
facilitates the generation of high-quality and diverse outputs, making it
effective for tasks such as image generation and compression. Based on VQVAE,
VQGAN [23] incorporates the adversarial training framework of PatchGAN [19],
enhancing the realism of generated images. Leveraging a visual tokenization
strategy similar to VQVAE, BEIT [5, 63] uses discrete visual tokens during its
pre-training phase. In this phase, portions of the input image are masked, and
the model predicts the discrete tokens for these masked patches, similar to
the masked language modeling in BERT [20]. Due to the lack of joint modeling
with the linguistic modality, there has been scant work combining discretized
visual tokens with MLLMs. The discretization of visual tokens has been
investigated to link visual output to the input of diffusion models [28, 29,
33], where additional reconstruction loss and decoders are used during
training. A recent method [62] employs a linear head layer to tokenize visual
information, which differs from our approach. Specifically, the head layer in
[62] is trained solely on vision data in a distilled manner, whereas we
optimize the visual head layer using gradients backward from the LLM on
vision-language data. Additionally, we propose learning a distinct visual
embedding table tailored specifically for visual information, rather than
directly using the LLM’s textual embedding table to retrieve embeddings for
visual tokens as done in [62].

##  3 Ovis

In this section, we first review the differences in visual and textual
embedding strategies in MLLMs. We then introduce our proposed architecture,
Ovis, which incorporates a linear mapping for probabilistic tokens and an
additional visual embedding look-up table within LLM.

(a) Overall Architecture

(b) Visual Tokenizer

(c) From visual patch to visual embedding

Figure 3: Illustration of Ovis. Figure (a) shows the whole architecture of
Ovis, which contains two embedding tables for visual and textual inputs.
Figure (b) illustrates how a visual patch is first mapped to a probabilistic
token. Figure (c) demonstrates that the probabilistic token helps select
multiple embeddings from the embedding table and output their weighted
combination.

###  3.1 Difference between Visual and Textual Tokens

Both images and texts are input into the MLLM, and they have diverse
tokenization strategies.

Let $\mathcal{I}\in\mathbb{R}^{C\times W\times H}$ be the pixel value tensor
of an image, where $C,W,H$ denote the channel number, width, and height of the
image, respectively. The image is first divided into a sequence of visual
patches $\\{\mathcal{P}_{i}\in\mathbb{R}^{C\times w\times
h}\\}_{i=1,2,\ldots,n}$, where $w$ and $h$ denote the width and height of the
patch, respectively, and $n=\lceil\frac{W}{w}\rceil\lceil\frac{H}{h}\rceil$ is
the number of patches. Given a pre-trained vision transformer (ViT) backbone
$g_{\theta}$ with parameters $\theta$, we then transform the patches into a
sequence of visual representations
$\\{{\bm{r}}_{i}\in\mathbb{R}^{d}\\}_{i=1}^{n}$.

For the textual input, let $\\{\bm{t}_{i}\\}_{i=1}^{m}$ be the input sequence
of textual tokens, which are further processed by an LLM $f_{\phi}$,
parameterized by $\phi$. In MLLM, both visual ($\\{{\bm{r}}_{i}\\}_{i=1}^{n}$)
and textual ($\\{\bm{t}_{i}\\}_{i=1}^{m}$) tokens should be transformed into
the same form, and then LLM processes all tokens into an output sequence of
textual tokens. We use $\lambda$ to denote the index of the image indicator
token, i.e., $\bm{t}_{\lambda}=\text{\textless image\textgreater}$, and the
multimodal input tokens become

$[\bm{t}_{1},\ldots,\bm{t}_{\lambda-1},\text{\textless
image\textgreater},\ldots,\bm{t}_{m}]\;.$ (1)

Since the visual and textual tokens have diverse dimensions, it is difficult
to substitute <image> directly with $\\{{\bm{r}}_{i}\\}_{i=1}^{n}$. Previous
approaches introduce additional linear projection [48], MLP [47], or
transformer [1, 79, 38, 4] architectures to map visual tokens into the same
form as textual ones.

###  3.2 Probabilistic Visual Tokens

Instead of using continuous visual tokens in Equation 1, we align the internal
tokenization strategies between images and texts to inspire the potential of
the MLLM.

To mimic the discrete textual tokens, we use a linear head
$\mathbb{R}^{K\times d}$ to transform the concrete visual tokens. Assuming $K$
is the visual vocabulary size, i.e., the number of unique visual words, then
given a visual token ${\bm{r}}_{i}$, we first transform ${\bm{r}}_{i}$ into a
$(K-1)$-dimensional probability simplex $\Delta^{K}$ by a linear projection
followed by a softmax normalization:

$\bm{v}_{i}=\text{softmax}(\bm{W}{\bm{r}}_{i})\quad\bm{W}\in\mathbb{R}^{K\times
d}.$ (2)

We set $\bm{v}_{i}\in\Delta^{K}$ as a kind of probabilistic token, which is a
probability distribution over the visual vocabulary containing $K$ visual
words. If ${\bm{r}}_{i}$ is more related to certain patterns, the
corresponding elements in $\bm{v}_{i}$ should be larger.

Remark. Given a visual embedding table, we associate each visual word with its
prototype $\\{\bm{w}_{i}\in\mathbb{R}^{d}\\}_{i=1}^{K}$. To match a continuous
visual token with the $K$ visual words in the embedding table, we leverage the
inner product to calculate their similarity value. Equation 2 is the
normalized similarity between ${\bm{r}}_{i}$ and all visual words.

###  3.3 Visual Embedding Table

In LLMs, it is a common practice to employ a textual embedding table, which
maps each word in the vocabulary to an embedding vector. For each textual
token $\bm{t}_{i}$ in the one-hot form, its embedding
$T_{i}\in\mathbb{R}^{d^{\prime}}$ is the row of the textual embedding table
indicated by the non-zero index in $\bm{t}_{i}$.

Analogously, we introduce an additional visual embedding table, where each
visual word (each row) is associated with an embedding vector
${\bm{e}}_{k}\in\mathbb{R}^{d^{\prime}}$ with $d^{\prime}$ being the embedding
dimension. To make the embeddings of visual and textual tokens have compatible
shapes, we simply set the dimension of the visual embedding table to be the
same as that of the textual embedding table.

Accordingly, the embedding of each visual token $\bm{v}_{i}$ can be derived
based on the probabilistic token:

$V_{i}=\sum_{k=1}^{K}v_{i,k}{\bm{e}}_{k}\in\mathbb{R}^{d^{\prime}}\;,$ (3)

where $v_{i,k}$ denotes the $k$-th component of $\bm{v}_{i}$. On the other
hand, since $\bm{v}_{i}\in\Delta^{K}$, the above formula can be rewritten as
$V_{i}=\mathbb{E}_{k\sim\bm{v}_{i}}[{\bm{e}}_{k}]$, which is an expectation of
the visual word’s embedding, with the visual word drawn from $\bm{v}_{i}$. In
other words, we assume that the visual embedding could be sampled from the
discrete visual embedding table based on the probabilistic token $\bm{v}_{i}$
of the patch.

Remark. Considering the polysemous nature of a visual patch ${\bm{r}}_{i}$,
assigning it only one visual word from the embedding table indexed by
$\arg\max_{j\in\\{1,\ldots,K\\}}\bm{v}_{ij}$ may neglect the rich semantics
within the patch. To address this, we link the patch with multiple visual
words at a time, as indicated by the non-zero elements in $\bm{v}_{i}$, which
represent the correlation between the patch and $K$ visual words. We then use
the weighted combination $V_{i}$ of the selected visual words as the final
patch embedding. In other words, multiple visual embeddings are indexed from
the embedding table based on the values in $\bm{v}_{i}$, and the weighted
average of these embeddings serves as the final output of the visual embedding
module. Experiments in Appendix E validate the sparsity of $\bm{v}_{i}$. The
visual embedding $V_{i}$ captures the rich visual semantics and keeps its
generation process similar to its textual counterpart simultaneously.

###  3.4 Training Strategy of Ovis

Both visual and textual embeddings are combined as the input into the LLM. In
particular, we feed the following multimodal input embedding sequence

$[T_{1},\ldots,T_{\lambda-1},V_{1},\ldots,V_{n},T_{\lambda+1},\ldots,T_{m}]$
(4)

to the LLM. All tokens not only have the same dimensionality but are also
generated in a similar manner with embedding tables. LLM will output a textual
token sequence $o_{1},\ldots,o_{l}$.

Ovis is trained in a three-stage fashion and optimized throughout with the
textual generation loss, i.e., the cross-entropy between the textual tokens of
Ovis output and that of the ground-truth text. The stages differ in their
trainable parameters and the types of training data.

Stage 1. We freeze all parameters of LLM, as well as most parameters in the
visual encoder $g$, an open-source pre-trained ViT backbone. We randomly re-
initialize the parameters within the last block of $g$, and using visual
caption datasets such as COYO [7] to train the re-initialized parameters as
well as the projection $\bm{W}$ and the visual embedding table
$\\{{\bm{e}}_{k}\\}_{k=1}^{K}$ of Ovis. For each image in the caption dataset,
we construct a training sample with input as “<image>’s caption: ” and label
it as “CAPTION”, where CAPTION denotes the image’s caption.

Stage 2. In this stage, we advance the training of Ovis’s $\bm{W}$, the visual
embedding table $\\{{\bm{e}}_{k}\\}_{k=1}^{K}$, and all parameters in the
vision encoder $g$. The LLM is still frozen. In contrast to the caption
samples used in the first stage, we leverage visual description datasets such
as ShareGPT4V-Pretrain [11], which consist of training samples structured as
dialogues that describe images.

Stage 3. After endowing Ovis with visual perception capabilities through
training in Stage 1 and Stage 2, this stage focuses on multimodal instruction
learning. The goal is to equip Ovis with the ability to follow multimodal
instructions. To this end, we unfreeze the LLM module and train Ovis’s entire
set of parameters on multimodal instruction datasets, such as LLaVA-Finetune
[47].

##  4 Experiments

In this section, we provide empirical results to demonstrate the effectiveness
of the proposed MLLM architecture Ovis.111Qualitative results are presented in
Appendix A.

###  4.1 Experimental Setup

#### Implementation Details.

Ovis encompasses three configurations: the LLM module, the ViT backbone, and
the visual vocabulary size. We incorporate popular open-source LLMs
(Qwen1.5-Chat [3] and Llama3-Instruct [56]) and ViTs (Clip-ViT-L/14@336px
[66]) into Ovis. The visual vocabulary size is set to $2^{17}=131,072$, a
value comparable to LLMs’ textual vocabulary size. To facilitate community use
and future innovation, the Ovis architecture and its training code are built
upon the widely-used Transformers [81] and DeepSpeed [67] packages. We detail
the training hyper-parameters for each stage in Table 4 of Appendix B.

#### Training Datasets.

Ovis is trained predominantly on open-source datasets, supplemented by a
smaller proportion of in-house datasets. The datasets employed can be
categorized into three groups: visual captions, visual descriptions, and
multimodal instructions, which are respectively utilized in Stage 1, 2, and 3
of the training process. The visual caption dataset is extracted from the COYO
dataset based on the similarity between the image and its caption. We
leveraged the COYO dataset’s provided “clip-similarity-vitb32” and “clip-
similarity-vitl14” scores for this purpose. Specifically, we select all
entries from the COYO dataset with both similarity metrics exceeding $0.36$.
The visual description datasets and multimodal instruction datasets are all
converted into the same format as LLaVA-Finetune [47]. Our in-house datasets
will be made publicly available alongside the source code of Ovis. We describe
the constructions and present several samples from the in-house datasets in
Appendix C and Appendix D. Statistics of the training dataset are reported in
Table 5 of Appendix B.

###  4.2 Main Results

Table 1: Comparison with popular open-source and proprietary MLLMs on general multimodal benchmarks. MMBench is shortened to MMB due to width limitations. MMMU-V and MMMU-T denote the validation and test splits, respectively. The GPT4V-HR denotes GPT4V in the high-resolution mode. Within each parameter tier, the top-performing model is highlighted in bold, while the runner-up is marked with an underscore. MLLM | MMStar | MMB-EN | MMB-CN | MMMU-V | MMMU-T
---|---|---|---|---|---
open-source models in 7B tier
InstructBLIP-7B [18] | 32.7 | 33.9 | 23.9 | 30.6 | 33.8
VW-LMM-PIF-7B [62] | - | 65.2 | 53.5 | - | -
LLaVA-v1.5-7B [47] | 33.1 | 66.5 | 59.0 | 35.7 | -
ShareGPT4V-7B [11] | 35.7 | 67.6 | 60.7 | 37.2 | -
Monkey [41] | 37.0 | 59.6 | 54.7 | 38.9 | -
Monkey-Chat [41] | 40.7 | 72.4 | 67.5 |  40.7 | -
DeepSeek-VL-7B [51] | 40.5 | 73.8 |  71.4 | 38.3 | -
Yi-VL-6B [87] | 37.7 | 68.4 | 66.6 | 40.3 |  37.8
Qwen-VL-Chat [4] | 34.5 | 61.8 | 56.3 | 37.0 | 32.9
Mini-Gemini-7B [40] | - | 69.3 | - | 36.1 | 32.8
Mini-Gemini-HD-7B [40] | - | 65.8 | - | 36.8 | 32.9
LLaVA-Next-Vicuna-7B [49] | 37.6 | 69.2 | 62.3 | 37.6 | -
LLaVA-Next-Mistral-7B [49] | 38.4 | 69.6 | 63.3 | 37.0 | -
LLaVA-Llama3-8B [17] |  46.1 | 71.7 | 63.2 | 39.2 | -
VILA1.5-Llama3-8B [43] | - | 72.3 | 66.2 | 36.9 | 36.0
Ovis-Qwen1.5-7B | 44.3 |  75.1 | 70.2 | 39.7 | 37.7
Ovis-Llama3-8B | 49.5 | 77.4 | 72.8 | 44.7 | 39.0
open-source models in 14B tier
PandaGPT-13B [72] | 25.6 | 42.5 | 32.0 | 32.9 | -
LLaVA-v1.5-13B [47] | 34.3 | 69.2 | 65.0 | 37.0 | 33.6
ShareGPT4V-13B [11] | 38.3 | 69.8 | 65.1 | 36.6 | -
Mini-Gemini-13B [40] | - | 68.5 | - |  38.1 | 33.5
Mini-Gemini-HD-13B [40] | - | 68.6 | - | 37.3 |  35.1
LLaVA-Next-Vicuna-13B [49] |  40.4 | 70.0 |  68.5 | 37.3 | -
VILA1.5-13B [43] | - |  74.9 | 66.3 | 37.9 | 33.6
Ovis-Qwen1.5-14B | 48.5 | 78.4 | 76.6 | 46.7 | 40.7
proprietary models
GPT4V [59] | 52.9 | 80.8 | 79.1 | 62.3 | 55.7
GPT4V-HR [59] | 56.0 | 81.0 | 80.2 | 61.7 | -
Gemini-Pro [75] | 38.6 | 73.6 | 74.3 | 49.0 | -
Qwen-VL-Plus [4] | 39.7 | 67.0 | 70.7 | 39.8 | 40.8
Qwen-VL-Max [4] | 49.5 | 77.6 | 75.7 | 52.0 | 46.8
Table 2: Comparison with popular open-source and proprietary MLLMs on specialized multimodal benchmarks. The GPT4V-HR denotes GPT4V in the high-resolution mode. We report the sum of perception and cognition scores for MME and the QuestionAcc score for HallusionBench. Within each parameter tier, the top-performing and runner-up models are in bold and underscored, respectively. MLLM | MathVista-Mini | MME | HallusionBench | RealWorldQA
---|---|---|---|---
open-source models in 7B tier
InstructBLIP-7B [18] | 24.4 | 1391 | 53.6 | 36.9
LLaVA-v1.5-7B [47] | 25.6 | 1808 | 48.8 | 54.8
ShareGPT4V-7B [11] | 26.5 |  1915 | 48.8 | 54.9
Monkey [41] | 33.5 | 1760 | 55.1 | 51.6
Monkey-Chat [41] | 35.9 | 1887 |  58.4 | 52.4
DeepSeek-VL-7B [51] | 36.9 | 1765 | 53.9 | 54.2
Yi-VL-6B [87] | 29.7 |  1915 | 55.7 | 53.5
Qwen-VL-Chat [4] | 34.9 | 1860 | 56.4 | 49.3
Mini-Gemini-7B [40] | - | 1839 | - | -
Mini-Gemini-HD-7B [40] | - | 1865 | - | -
LLaVA-Next-Vicuna-7B [49] | 31.5 | 1769 | 47.2 | 57.8
LLaVA-Next-Mistral-7B [49] | 34.6 | 1821 | 47.9 | 60.0
LLaVA-Llama3-8B [17] | 40.0 | 1826 | 48.6 | 56.7
Ovis-Qwen1.5-7B | 41.4 | 1882 | 56.4 | 60.0
Ovis-Llama3-8B |  40.8 | 2009 | 61.1 |  57.9
open-source models in 14B tier
PandaGPT-13B [72] | 25.0 | 1072 | 43.1 | 32.8
LLaVA-v1.5-13B [47] | 27.7 | 1781 | 45.3 | 55.3
ShareGPT4V-13B [11] | 29.3 | 1853 | 48.8 | 57.0
Mini-Gemini-13B [40] | - | 1887 | - | -
Mini-Gemini-HD-13B [40] | - |  1917 | - | -
LLaVA-Next-Vicuna-13B [49] |  34.1 | 1746 |  51.5 |  57.6
Ovis-Qwen1.5-14B | 43.4 | 1961 | 57.6 | 62.7
proprietary models
GPT4V [59] | 51.4 | 2038 | 60.1 | 61.4
GPT4V-HR [59] | 54.7 | 2070 | 62.1 | 68.0
Gemini-Pro [75] | 46.5 | 2149 | 63.7 | 60.4
Qwen-VL-Plus [4] | 37.6 | 2230 | 57.6 | 44.6
Qwen-VL-Max [4] | 43.4 | 2282 | 57.7 | 61.3

We evaluate Ovis across a variety of benchmarks, covering both general
multimodal capabilities benchmarks (MMMU [88], MMBench-EN [50], MMBench-CN
[50], and MMStar [12]), as well as benchmarks for more specialized multimodal
tasks (MathVista-Mini [54], MME [25], HallusionBench [46], and RealWorldQA
[84]). The evaluation is performed using the VLMEvalKit package [16]. The
comparison between Ovis’ benchmark performance with that of popular open-
source MLLMs and leading proprietary models is summarized in Table 1 and Table
2, where the benchmark scores of the compared models are mainly derived from
VLMEvalKit for consistency. MLLMs with no specialized multimodal benchmark
performance reported are not included in Table 2 for conciseness.

It can be seen that Ovis-8B outperforms the open-source models of similar size
across the majority of benchmarks. Ovis-14B not only excels in all benchmarks
but also surpasses the high-resource proprietary model Qwen-VL-Plus in most
benchmarks. In the vision-indispensable multi-modal benchmark MMStar, Ovis-8B
exhibits a large margin over the compared open-source MLLMs, highlighting its
advantage in utilizing visual information. Ovis also achieves leading results
in the highly challenging college-level MMMU benchmark, demonstrating strong
visual comprehension and reasoning abilities. The MMBench-EN and MMBench-CN
benchmarks differ only in the language [50]. While Ovis’ training dataset
contains very few non-English samples, Ovis performs well in both versions.
Ovis-14B achieves consistently outstanding performance in MMBench-EN and
MMBench-CN, suggesting that Ovis’ advantage in multimodal capabilities is not
limited to English but can extend to another language like Chinese as well.

Turning attention to the specialized multimodal benchmarks, we find that Ovis
enjoys better multimodal capabilities in math and logical reasoning than open-
source competitors, as demonstrated by its notable performance in the
MathVista-Mini benchmark. While Ovis only employs a 336px ViT backbone and is
not equipped with high-resolution-boosted techniques such as the dynamic high
resolution used in LLaVA-Next [49] and the dual vision encoders used in Mini-
Gemini-HD [40], Ovis exhibits impressive performance in the RealWorldQA
benchmark, which is comprised of real-world visual tasks with high-resolution
images (e.g., 1080P). Notably, Ovis-14B’s RealWorldQA score is even higher
than the leading proprietary model GPT4V, illustrating its outstanding
multimodal capabilities in solving practical visual tasks. In the MME and
hallucination benchmarks, Ovis-8B and Ovis-14B perform the best within the 7B
and 14B tier, respectively. This implies that Ovis’ strong visual
understanding and reasoning abilities are accompanied by a lower rate of
hallucination, a highly desirable trait for the deployment of MLLMs in
critical scenarios such as medicine.

###  4.3 Ablation Study

Table 3: Comparison between Ovis and the conventional connector-based architecture, both employing Qwen1.5-7B-Chat and Clip-ViT-L/14@336px as backbones and trained on the same datasets. Due to width limitations, MMBench-EN and MMBench-CN are merged into a single column, as are MMMU-V and MMMU-T. MathVista-Mini, HallusionBench, and RealWorldQA are shortened to Math, HB, and RWQA, respectively. Architecture | MMStar | MMBench | MMMU | Math | MME | HB | RWQA
---|---|---|---|---|---|---|---
Connector | 41.1 | 71.0 / 65.2 | 34.8 / 33.8 | 36.3 | 1757 | 54.0 | 56.1
Ovis | 44.3 | 75.1 / 70.2 | 39.7 / 37.7 | 41.4 | 1882 | 56.4 | 60.0
Improvement | 7.8% | 5.8% / 7.7% | 14.1% / 11.5% | 14.0% | 7.1% | 4.4% | 7.0%

To further elucidate the advantages of Ovis’ architectural design, we conduct
a comparative experiment between Ovis-7B and a connector-based MLLM utilizing
identical LLM and ViT backbones as Ovis-7B. Following [47], we implement the
connector as a two-layer MLP with GELU activation. The hidden size of the MLP
is configured to match Ovis-7B’s visual vocabulary size, ensuring parity in
parameter count between the connector-based MLLM and Ovis-7B. We train the
connector-based MLLM on the same datasets as Ovis-7B, adhering to the training
paradigm outlined in [47]. The experimental results are summarized in Table 3.
Remarkably, Ovis consistently outperforms the connector-based architecture
across all benchmark evaluations, achieving an impressive 8.8% performance
margin on average. Given the identical parameter counts, backbones, and
training datasets, the results compellingly advocate for the efficacy of Ovis’
architectural design.

##  5 Conclusion

We emphasize the necessity of structurally aligning visual embeddings with the
textual counterparts, considering their different tokenization and embedding
strategies in MLLMs. In Ovis, we introduce an additional visual embedding
look-up table. Image patches are mapped into probabilistic tokens, which then
index the visual embedding table and are transformed into a structural manner
similar to textual embeddings. Empirical evaluations across various multimodal
benchmarks validate Ovis’ effectiveness, demonstrating that it outperforms
open-source MLLMs of similar parameter scales as well as the proprietary model
Qwen-VL-Plus.

##  6 Broader Impact and Limitations

#### Broader Impact.

As a powerful multimodal large language model architecture, Ovis has the
potential to benefit a wide range of users through enhanced interactions
between visual content and textual analysis. However, it is crucial to
acknowledge the potential negative impacts associated with Ovis, such as the
risk of hallucination, wherein Ovis may generate misleading or incorrect
information, potentially leading to misinformation. Furthermore, Ovis also
suffers from biases and potential harms, a common issue among generative
models. These potential adverse effects could be mitigated through content
moderation mechanisms and transparent model developments.

#### Limitations.

While Ovis has demonstrated promising performance, its efficacy in handling
visual tasks with high-resolution images is limited due to the absence of
high-resolution-boosted techniques. Moreover, Ovis is trained solely with
single-image samples, posing challenges when confronted with scenarios
requiring visual understanding across multiple images. Considerable research
efforts have been dedicated to these areas [49, 40, 43], primarily within the
connector-based framework. Drawing inspiration from these researches, we plan
to enhance Ovis’ capacity to better handle high-resolution images and process
multi-image inputs in future iterations.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _NeurIPS_ , 2022.
  * anthropic [2024] anthropic.  Introducing the next generation of claude.  Technical report, anthropic, 2024.  URL https://www.anthropic.com/news/claude-3-family.
  * Bai et al. [2023a] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv:2309.16609_ , 2023a.
  * Bai et al. [2023b] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv:2308.12966_ , 2023b.
  * Bao et al. [2021] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv:2106.08254_ , 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NeurIPS_ , 2020.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _CVPR_ , 2021.
  * Chen et al. [2024a] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang.  Allava: Harnessing gpt4v-synthesized data for a lite vision-language model.  _arXiv:2402.11684_ , 2024a.
  * Chen et al. [2024b] Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang.  Onechart: Purify the chart structural extraction via one auxiliary token.  _arXiv:2404.09987_ , 2024b.
  * Chen et al. [2023] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  _arXiv:2311.12793_ , 2023.
  * Chen et al. [2024c] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al.  Are we on the right way for evaluating large vision-language models?  _arXiv:2403.20330_ , 2024c.
  * Chen et al. [2024d] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang.  How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.  _arXiv:2404.16821_ , 2024d.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, 2023.  URL https://vicuna.lmsys.org.
  * Chu et al. [2024] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al.  Mobilevlm v2: Faster and stronger baseline for vision language model.  _arXiv:2402.03766_ , 2024.
  * Contributors [2023a] OpenCompass Contributors.  Opencompass: A universal evaluation platform for foundation models.  https://github.com/open-compass/opencompass, 2023a.
  * Contributors [2023b] XTuner Contributors.  Xtuner: A toolkit for efficiently fine-tuning llm.  https://github.com/InternLM/xtuner, 2023b.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv:2305.06500_ , 2023.
  * Demir and Unal [2018] Ugur Demir and Gozde Unal.  Patch-based image inpainting with generative adversarial networks, 2018.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv:1810.04805_ , 2018.
  * Dong et al. [2024] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al.  Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd.  _arXiv:2404.06512_ , 2024.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv:2010.11929_ , 2020.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Fang et al. [2023] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva-02: A visual representation for neon genesis.  _arXiv:2303.11331_ , 2023.
  * Fu et al. [2023a] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Zhenyu Qiu, Wei Lin, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _arXiv:2306.13394_ , 2023a.
  * Fu et al. [2023b] Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, and Xing Sun.  A challenger to gpt-4v? early explorations of gemini in visual expertise.  _arXiv preprint arXiv:2312.12436_ , 2023b.
  * Gao et al. [2023] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al.  G-llava: Solving geometric problem with multi-modal large language model.  _aarXiv:2312.11370_ , 2023.
  * Ge et al. [2023a] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan.  Planting a seed of vision in large language model.  _arXiv preprint arXiv:2307.08041_ , 2023a.
  * Ge et al. [2023b] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan.  Making llama see and draw with seed tokenizer.  _arXiv:2310.01218_ , 2023b.
  * Hu et al. [2024a] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al.  mplug-docowl 1.5: Unified structure learning for ocr-free document understanding.  _arXiv:2403.12895_ , 2024a.
  * Hu et al. [2024b] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al.  Minicpm: Unveiling the potential of small language models with scalable training strategies.  _arXiv:2404.06395_ , 2024b.
  * Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _arXiv:2310.06825_ , 2023.
  * Jin et al. [2023] Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al.  Unified language-vision pretraining with dynamic discrete visual tokenization.  _arXiv:2309.04669_ , 2023.
  * Kim et al. [2022] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.  Ocr-free document understanding transformer.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv:1312.6114_ , 2013.
  * Li et al. [2023a] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv:2306.00890_ , 2023a.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , 2022.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv:2301.12597_ , 2023b.
  * Li et al. [2024a] Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu.  Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models, 2024a.
  * Li et al. [2024b] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia.  Mini-gemini: Mining the potential of multi-modality vision language models.  _arXiv:2403.18814_ , 2024b.
  * Li et al. [2023c] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai.  Monkey: Image resolution and text label are important things for large multi-modal models.  _arXiv:2311.06607_ , 2023c.
  * Lin et al. [2024] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan.  Moe-llava: Mixture of experts for large vision-language models.  _arXiv:2401.15947_ , 2024.
  * Lin et al. [2023] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han.  Vila: On pre-training for visual language models, 2023.
  * Liu et al. [2024a] Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu.  Hrvda: High-resolution visual document assistant.  _arXiv:2404.06918_ , 2024a.
  * Liu et al. [2023a] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Aligning large multi-modal model with robust instruction tuning.  _arXiv:2306.14565_ , 2023a.
  * Liu et al. [2024b] Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou.  Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models.  In _CVPR_ , 2024b.
  * Liu et al. [2023b] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv:2310.03744_ , 2023b.
  * Liu et al. [2023c] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv:2304.08485_ , 2023c.
  * Liu et al. [2024c] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.  Llava-next: Improved reasoning, ocr, and world knowledge, January 2024c.  URL https://llava-vl.github.io/blog/2024-01-30-llava-next/.
  * Liu et al. [2023d] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.  Mmbench: Is your multi-modal model an all-around player?  _arXiv:2307.06281_ , 2023d.
  * Lu et al. [2024a] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al.  Deepseek-vl: towards real-world vision-language understanding.  _arXiv:2403.05525_ , 2024a.
  * Lu et al. [2023] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi.  Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action.  _arXiv:2312.17172_ , 2023.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  _Advances in Neural Information Processing Systems_ , 35:2507–2521, 2022.
  * Lu et al. [2024b] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  In _International Conference on Learning Representations (ICLR)_ , 2024b.
  * Lu et al. [2024c] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.  Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.  In _ICLR_ , 2024c.
  * meta [2024] meta.  Build the future of ai with meta llama 3.  Technical report, meta, 2024.  URL https://llama.meta.com/llama3/.
  * Mistral [2024] Mistral.  Mistral-8×22b.  Technical report, Mistral, 2024.  URL https://mistral.ai/news/mixtral-8x22b/.
  * OpenAI [2023a] OpenAI.  Chatgpt: A language model for conversational ai.  Technical report, OpenAI, 2023a.  URL https://www.openai.com/research/chatgpt.
  * OpenAI [2023b] OpenAI.  Gpt-4 technical report.  _arXiv:2303.08774_ , 2023b.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _NeurIPS_ , 2022.
  * Panagopoulou et al. [2023] Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles.  X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning.  _arXiv:2311.18799_ , 2023.
  * Peng et al. [2024] Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, and Bo Du.  Multi-modal auto-regressive modeling via visual words.  _arXiv:2403.07720_ , 2024.
  * Peng et al. [2022] Z Peng, L Dong, H Bao, Q Ye, and F Wei.  Beit v2: Masked image modeling with vector-quantized visual tokenizers. arxiv 2022.  _arXiv:2208.06366_ , 2022.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Rasley et al. [2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20, page 3505–3506, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450379984.  doi: 10.1145/3394486.3406703.  URL https://doi.org/10.1145/3394486.3406703.
  * Reid et al. [2024] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv:2403.05530_ , 2024.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision_ , 115(3):211–252, 2015.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _NeurIPS_ , 2022.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8317–8326, 2019.
  * Su et al. [2023] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  _arXiv:2305.16355_ , 2023.
  * Sun et al. [2023] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv:2303.15389_ , 2023.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv:2312.11805_ , 2023.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv:2307.09288_ , 2023b.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  Cogvlm: Visual expert for pretrained language models.  _arXiv:2311.03079_ , 2023.
  * Wei et al. [2023] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang.  Vary: Scaling up the vision vocabulary for large vision-language models.  _arXiv:2312.06109_ , 2023.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
  * Wu et al. [2023a] Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, Geng Xue, Wenxiu Sun, Qiong Yan, and Weisi Lin.  Q-instruct: Improving low-level visual abilities for multi-modality foundation models, 2023a.
  * Wu et al. [2023b] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv:2309.05519_ , 2023b.
  * x.ai [2024] x.ai.  Grok-1.5v.  https://x.ai/blog/grok-1.5v, 2024.
  * Yang et al. [2023a] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al.  Baichuan 2: Open large-scale language models.  _arXiv:2309.10305_ , 2023a.
  * Yang et al. [2023b] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  _arXiv:2309.17421_ , 2023b.
  * Young et al. [2024] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al.  Yi: Open foundation models by 01. ai.  _arXiv:2403.04652_ , 2024.
  * Yue et al. [2024] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  In _Proceedings of CVPR_ , 2024.
  * Zhai et al. [2023] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11975–11986, 2023.
  * Zhang et al. [2024] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al.  Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?  _arXiv preprint arXiv:2403.14624_ , 2024.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv:2304.10592_ , 2023.

##  Appendix A Qualitative Results

As shown in Figure 4 and Figure 5, Ovis-Llama3-8B performs well in various
multimodal tasks, where the images and prompts are sourced from literature.

(a) Perception [40, 41, 26]

(b) Reasoning [75]

(c) Coding [84]

Figure 4: Qualitative results with Ovis-Llama3-8B, part I

(a) Math [90]

(b) Science [13]

(c) Cooking scenario [75]

Figure 5: Qualitative results with Ovis-Llama3-8B, part II

##  Appendix B Training Details

We will release the source code and the whole training dataset of Ovis in the
near future.

Table 4: Training hyper-parameters Hyper-parameter | Stage 1 | Stage 2 | Stage 3
---|---|---|---
batch size | 8192 | 1024 | 1024
learning rate (Ovis-Qwen1.5-7B/14B) | 1e-4 | 1e-4 | 2e-5
learning rate (Ovis-Llama3-8B) | 1e-4 | 1e-4 | 1e-5
learning rate schedule | cosine | cosine | cosine
learning rate warm-up ratio | 0.1 | 0.1 | 0.05
weight decay | 0 | 0 | 0
grad norm clipping | 1.0 | 1.0 | 1.0
epoch | 1 | 1 | 1
optimizer | AdamW | AdamW | AdamW
float precision | bfloat16 | bfloat16 | bfloat16
deepspeed configuration (Ovis-7B/8B) | zero2 | zero3 | zero3
deepspeed configuration (Ovis-14B) | zero3 | zero3 | zero3
training hours (Ovis-7B/8B, 128 H100 GPUs) | 6 | 2 | 7
training hours (Ovis-14B, 128 H100 GPUs) | 10 | 6 | 21
Table 5: Statistics of the training dataset with $\star$ denoting in-house data Dataset Category | Dataset Name | #Samples | Total Size
---|---|---|---
Visual Caption | COYO-10M [7] | 10M | 10M
Visual Description | LLaVA-Pretrain [47] | 558K | 2M
ShareGPT4V-Pretrain [11] | 82K
ALLaVA-Caption-Laion-4V [9] | 485K
ALLaVA-Caption-Vflan-4V [9] | 203K
Laion-Description⋆ | 11K
CC12M-Description⋆ | 1M
Multimodal Instruction | ScienceQA-Train-Val [53] | 17K | 3M
TextVQA-Train [71] | 35K
ALLaVA-Instruct-Laion-4V [9] | 485K
ALLaVA-Instruct-Vflan-4V [9] | 203K
ArXivQA [39] | 100K
Q-Instruct [82] | 198K
LLaVA-Finetune [47] | 665K
Geo [27] | 177K
LRV-Instruction [45] | 300K
Chart-Instruction [45] | 43K
Synthdog-EN-OCR [34] | 200K
ALLaVA-Evol-Instruct [9] | 143K
CC12M-QA⋆ | 387K

##  Appendix C In-house Visual Description Dataset

We sample images from the Laion [70] and CC12M [8] datasets, which cover
various categories, including nature, lifestyle, humanities, architecture,
cartoons, and abstract art, as shown in Figure 6. For each image, we call the
Gemini-Pro or GPT-4V API with a unified prompt to generate the image’s
descriptions. The unified prompt explicitly requires the API to reply with
concise and clear visual information about the image, as well as performing
OCR recognition if relevant, while avoiding embellishments and
interpretations.

Figure 6: Samples from our in-house visual description dataset

##  Appendix D In-house Visual Instruction Dataset

We create visual instruction samples for images from the CC12M dataset [8] in
a similar way to [9], using Gemini-Pro and GPT-4V to carry out a self-
questioning and answering task. This method produces diverse questions and
high-quality answers, as illustrated in Figure 7.

Figure 7: Samples from our in-house visual instruction dataset

##  Appendix E Sparsity of Ovis’ Visual Tokenizer

To assess the sparsity of Ovis’ visual tokenizer, we conduct an experiment
using 10,000 images sampled from the ImageNet-1K dataset [69]. Each image is
tokenized by the visual tokenizer of Ovis-Llama3-8B, resulting in a sequence
of visual tokens, each of which is a probability distribution over the visual
vocabulary. We then employ thresholds of 1e-4, 1e-5, and 1e-6 to categorize
the probability values and calculate the ratio of values falling within each
interval across the 10,000 images. As shown in Figure 8, the probability
distributions characterized by the visual tokens are highly sparse, with only
0.22% probability values exceeding the threshold of 1e-4.

Figure 8: Statistics of the probability distributions characterized by the
visual tokens across 10,000 images from ImageNet-1K. The visual tokens are
obtained using Ovis-Llama3-8B’s visual tokenizer.