#  Dragonfly: Multi-Resolution Zoom Supercharges Large Visual-Language Model

Kezhen Chen1 Rahul Thapa1,211footnotemark: 1
Rahul Chalamala1,3 Ben Athiwaratkun1 Shuaiwen Leon Song1 James Zou1,2
1Together AI 2Stanford University 3 Caltech
{kezhen, rahul, ben<EMAIL_ADDRESS>
{rthapa84<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Recent advances in large multimodal models (LMMs) suggest that higher image
resolution enhances the fine-grained understanding of image details, crucial
for tasks such as visual commonsense reasoning and analyzing biomedical
images. However, increasing input resolution poses two main challenges: 1) It
extends the context length required by the language model, leading to
inefficiencies and hitting the model’s context limit; 2) It increases the
complexity of visual features, necessitating more training data or more
complex architecture. We introduce Dragonfly, a new LMM architecture that
enhances fine-grained visual understanding and reasoning about image regions
to address these challenges. Dragonfly employs two key strategies: multi-
resolution visual encoding and zoom-in patch selection. These strategies allow
the model to process high-resolution images efficiently while maintaining
reasonable context length. Our experiments on eight popular benchmarks
demonstrate that Dragonfly achieves competitive or better performance compared
to other architectures, highlighting the effectiveness of our design.
Additionally, we finetuned Dragonfly on biomedical instructions, achieving
state-of-the-art results on multiple biomedical tasks requiring fine-grained
visual understanding, including 92.3% accuracy on the Path-VQA dataset
(compared to 83.3% for Med-Gemini) and the highest reported results on
biomedical image captioning. To support model training, we curated a visual
instruction-tuning dataset with 5.5 million image-instruction samples in the
general domain and 1.4 million samples in the biomedical domain. We also
conducted ablation studies to characterize the impact of various architectural
designs and image resolutions, providing insights for future research on
visual instruction alignment. The codebase and model are available at
https://github.com/togethercomputer/Dragonfly.

##  1 Introduction

Recent progress in large multimodal models (LMMs) has underscored the
importance of effectively integrating visual information into large language
models (LLMs). Impressive performance is achieved through sophisticated visual
encoding strategies, which are pivotal in projecting complex real-world visual
information into the latent space of LLMs. One of the prominent architectures
in LMMs is vision instruction alignment, which encodes image inputs into
patch-level tokens using a pretrained image encoder. Then, these visual tokens
are projected into the latent space of an LLM through visual instruction-
tuning (Liu et al., 2023b, a; Yang et al., 2023; Li et al., 2023; Xu et al.,
2023; McKinzie et al., 2024; Laurençon et al., 2024; You et al., 2023; Zhang
et al., 2024).

Despite these advancements, most existing LMMs process images at a fixed
aspect ratio and low resolution, causing high-resolution images to be
downsampled before being passed to the model. The downsampling often results
in shape distortion and the loss of fine details (Xu et al., 2023). However,
many visual reasoning or biomedical tasks require models to capture fine-
grained details in image regions. Recent studies suggest that encoding higher
resolution images could provide more detailed information, thereby improving
fine-grained visual understanding(McKinzie et al., 2024; Laurençon et al.,
2024; Liu et al., 2023a; Zhang et al., 2024). One strategy is using better
image encoders trained on high-resolution images. For example, Liu et al.
(2023a) and Laurençon et al. (2024) replaced the low-resolution image encoder
trained on 224x224 images with high-resolution image encoders trained on
336x336 or 384x384 images, resulting in performance improvements on multiple
benchmarks. Xu et al. (2023) and Zhang et al. (2024) further enhance the
vision encoding by resizing original images into two different resolutions to
encode both abstract and detailed information.

However, existing methods face two key challenges when encoding high-
resolution images: filling more of the context window leads to lower model
efficiency, and higher resolutions may have more redundant patches. An image
with a resolution of 336x336 is encoded into 576 visual tokens in the
architecture from Liu et al. (2023a). In the biomedical domain, images are
much higher resolution and may require more visual tokens to capture detailed
information. Also, more patches in higher-resolution encoding may be
irrelevant to the given query, which leads to more complexity and noise. Some
tasks may only require focus on high-resolution regions of the images.

To address these challenges, we introduce a new LMM architecture, Dragonfly,
focusing on fine-grained understanding and reasoning on image regions while
maintaining high efficiency. Dragonfly incorporates two key techniques: multi-
resolution visual encoding and zoom-in patch selection. Inspired by multiple
works (Liu et al., 2023a; Xu et al., 2023; Laurençon et al., 2024), we encode
multi-resolution information of the images by adjusting the original input
image into three distinct resolutions—low, medium, and high—to capture
information at multiple granularities. Dragonfly also employs a zoom-in patch
selection method to enhance fine-grained understanding of image regions by
selecting the most informative visual tokens from high-resolution images. With
this approach, Dragonfly focuses on relevant high-resolution regions of the
image instead of using all the visual tokens of high-resolution images. This
new technique enables Dragonfly to demonstrate its fine-grained processing
capabilities on multiple popular benchmarks and biomedical tasks. Furthermore,
we conducted comprehensive ablation studies focusing on patch selection
strategies and image resolution. To support these studies, we curated a visual
instruction-tuning dataset with 5.5 million general-domain and 1.4 million
biomedical-domain samples. The model, codebase, and training data will be
available upon acceptance.

Figure 1: Overview of the proposed Dragonfly model architecture. The original
image is resized into low, medium, and high resolutions. The medium and high-
resolution images are divided into sub-images, aligning with the encoder’s
training resolution. All sub-images pass through a shared vision encoder to
produce visual tokens. The projection layer maps the visual tokens to the
language space, and the mean of these projected token embeddings serves as the
summary token embeddings. For each mid-resolution embedding, we select the top
3 high-resolution sub-images from the corresponding regions based on the dot
product similarity of the projected tokens, resulting in 12 non-overlapping
high-resolution sub-images. Finally, visual tokens from the top high-
resolution sub-images, along with all tokens from low and mid-resolution
images, are flattened and passed to the LLM.

In summary, our contributions are as follows:

  * •

We introduce Dragonfly, an innovative LMM architecture that utilizes multi-
resolution visual encoding and zoom-in patch selection, enabling Dragonfly to
process fine-grained image details efficiently. Trained on our curated dataset
of 5.5 million image-instruction samples, Dragonfly outperforms the baselines
on five popular benchmarks, providing evidence of the architecture’s
effectiveness.

  * •

We demonstrate our model’s understanding and reasoning capabilities on
biomedical tasks, which often require a fine-grained understanding of high-
resolution image regions. By finetuning our general-domain model on a
biomedical instruction-tuning dataset with 1.4 million biomedical image-text
pairs, Dragonfly-Med, our biomedical version of the model, achieves state-of-
the-art or competitive performance on multiple biomedical benchmarks,
including visual question answering (VQA), image captioning, and radiology
report generation. For example, Dragonfly-Med achieves 92.3% accuracy on the
Path-VQA dataset(He et al., 2020), a 58.8 CIDEr score on the IU X-Ray
dataset(Demner-Fushman et al., 2016), and a 179.9 CIDEr score on the Peir
Gross dataset(Jing et al., 2017).

  * •

We conducted extensive ablation studies to investigate the impacts of patch
selection strategies and image resolution. These studies enhance our
understanding of the factors influencing model training and provide valuable
insights that inform our design choices.

##  2 Related Work

The advancement of LLMs has significantly impacted vision-language research by
facilitating the integration of visual representations into language models,
allowing LLMs to utilize their extensive language knowledge while
incorporating newly emerging visual capabilities (Wei et al., 2022). Visual
feature alignment is one of the predominant methods to introduce multimodal
capabilities into LLMs. This approach maps visual embeddings into the language
space, effectively merging vision and language through visual instruction-
tuning (Liu et al., 2023b, a; Dai et al., 2023; Yang et al., 2023; Li et al.,
2023; Xu et al., 2023; McKinzie et al., 2024; Laurençon et al., 2024; You et
al., 2023; Awadalla et al., 2023). For instance, Liu et al. (2023b) employs a
fully connected layer to project image embeddings generated by a pretrained
CLIP encoder (Radford et al., 2021) into the large language model’s embedding
space. Similarly, Awadalla et al. (2023) and Li et al. (2023) adopt the
Flamingo framework Alayrac et al. (2022), utilizing a Perceiver Resampler with
cross-attention (Jaegle et al., 2022) to integrate visual tokens into the
language domain. Models utilizing embedding alignment methods have
demonstrated considerable success across various benchmarks, showcasing the
efficacy of embedding alignment in vision-language tasks. However, a
limitation among these models is their reliance on downscaling input images to
a fixed low resolution, which sacrifices visual detail. McKinzie et al. (2024)
and Laurençon et al. (2024) have provided evidence that higher-resolution
visual inputs can enhance the model performance and reasoning on fine-grained
visual contents. Liu et al. (2023a) and Xu et al. (2023) have refined this
approach by segmenting native-resolution images into smaller, variable-sized
slices that closely align with the traditional pretraining configurations of
image encoders. Recently, Fuyu (AdeptAI, 2023), a decoder-only model that
supports arbitrary image resolutions, has been used as an alternative to
circumvent the drawbacks of downscaling. By linearly projecting image patches
into the transformer’s first layer, Fuyu treats the sequence of image patches
similar to text tokens AdeptAI (2023). Despite these improvements, the
contextual constraints of LLMs still pose challenges. These methods often
require excessive visual tokens to encode higher-resolution images, leading to
inefficiencies. In contrast, Dragonfly applies a zoom-in patch selection
method, choosing a subset of visual tokens from high-resolution images to
achieve more efficient processing while focusing on regional fine-grained
details. Unlike McKinzie et al. (2024) and Laurençon et al. (2024), our paper
focuses only on the visual instruction-tuning stage, aiming to refine and
optimize the design of the Dragonfly architecture.

Advancements in LLMs and LMMs have also significantly propelled the
development of biomedical vision-language models, integrating medical imaging,
literature, and clinical notes. Notable examples include BiomedGPT (Zhang et
al., 2023a), a multi-task biomedical foundation model that combines language
modeling and masked image infilling objectives. Similarly, LLaVA-Med (Li et
al., 2024a) leverages PubMed and GPT-4 to curate a multimodal instruction-
following dataset and finetuning a LLaVA model. Additionally, general-purpose
LLMs have been adapted for biomedical applications, as seen in Med-PaLM (Tu et
al., 2024), Med-Flamingo (Moor et al., 2023), and Med-Gemini (Saab et al.,
2024). These models highlight the potential of leveraging LMMs to address
specialized biomedical challenges. Biomedical tasks require models with strong
and fine-grained understanding and reasoning capabilities for image regions.
For example, identifying a condition might depend on analyzing a specific area
of chest X-ray. In this paper, we evaluate the fine-grained image
understanding of Dragonfly architecture by finetuning it on biomedical data.
With 1.4 million high-quality biomedical visual instruction pairs, our model
achieves state-of-the-art or competitive performance on multiple biomedical
tasks, all while using a smaller backbone than other baselines.

##  3 Multi-resolution Zoom Architecture for Dragonfly

We introduce Dragonfly, an innovative LMM architecture that uniquely
incorporates multi-resolution visual encoding and zoom-in patch selection to
enhance the visual understanding of regional details. Figure 1 illustrates the
Dragonfly architecture, which consists of an image encoder $E$, a projection
layer $P$, and an LLM $L$. Given a pair of an image $I$ and a text instruction
$T$, the image encoder $E$ encodes the image into $n$ visual tokens
$\\{v_{i}\\}_{i=1}^{n}$, and the projection layer maps the visual tokens to
the language space to construct the input tokens $\\{t_{i}\\}_{i=1}^{n}$.
Then, the projected visual and text tokens are concatenated to construct the
sequence of embeddings passed to the LLM. This section presents the technical
details of multi-resolution visual encoding and zoom-in patch selection.

###  3.1 Multi-resolution Visual Encoding

Dragonfly leverages multi-resolution visual encoding through a shared image
encoder $E$ trained on a fixed image resolution: $R\times R$. Drawing on
strategies from prior works Liu et al. (2023a); Xu et al. (2023), Dragonfly
handles larger images by dividing them into multiple sub-images, each matching
the encoder’s training resolution. Specifically, given an image $I$, Dragonfly
resizes it into three distinct resolutions: a low-resolution image $I^{l}$
sized $R\times R$, a medium-resolution image $I^{m}$ sized $x^{m}R\times
y^{m}R$, and a high-resolution image $I^{h}$ sized $x^{h}R\times y^{h}R$. The
medium and high-resolution images are segmented into two sets of sub-images,
$\\{I^{m}_{i}\\}_{i=1}^{x^{m}\times y^{m}}$ and
$\\{I^{h}_{j}\\}_{j=1}^{x^{h}\times y^{h}}$, respectively, aligning with the
training resolution $R\times R$ of the encoder $E$.

The image encoder $E$ encodes each sub-image into a series of visual tokens
$\\{v_{1}...v_{n}\\}$. These tokens from various sub-images are then projected
into the latent space of the language model via the projection layer $P$, and
a sequence of projected tokens $\\{t_{1}...t_{n}\\}$ are generated. The
projected tokens from different sets of sub-images are concatenated to form a
comprehensive representation of the image and are used for LLM understanding.
However, the number of sub-images, particularly from the high-resolution set,
can be extensive. As previously mentioned, utilizing all these sub-images
could lead to a longer context length and introduce noisy features during
training. Dragonfly adopts a selective approach, using a novel zoom-in patch
selection method to choose only a subset of sub-images that contain richer
visual information. Thus, Dragonfly enhances the fine-grained understanding of
image regions while keeping efficiency. The following section details the
mechanics of the zoom-in patch selection method.

###  3.2 Zoom-in Patch Selection

Dragonfly employs a multi-resolution visual encoding strategy to efficiently
process images while capturing both abstract and fine-grained visual
information. The model processes all the visual tokens encoded for low and
medium-resolution images. Even when segmented into smaller sub-images, these
images do not produce excessive visual tokens and mainly provide a more
abstract representation of the visual content. However, the sub-images from
the high-resolution set are treated differently due to their potential to
generate many sub-images, some of which may be redundant. To address this,
Dragonfly introduces the zoom-in patch selection method only to select a
subset of high-resolution sub-images, effectively reducing redundancy. Each
high-resolution sub-image is a magnified view of a specific area of the
original image. We only identify and retain patches that correspond
semantically to the medium-resolution sub-images. This targeted selection
helps enrich the image representation with detailed information on critical
content areas and drop redundant sub-images.

To facilitate this selection, we construct a summary embedding $a$ for each
sub-image in both the medium-resolution and high-resolution sets. For the
medium-resolution sub-images, the summary embedding is denoted as $a^{m}$, and
for the high-resolution sub-images, as $a^{h}$. We compute the mean vector of
the projected sequence of tokens $\\{t_{1}...t_{n}\\}$ from the projection
layer $P$ as the summary embedding. As the projection layer $P$ projects
visual tokens $\\{v_{1}...v_{n}\\}$ to the tokens $\\{t_{i}...t_{n}\\}$ in the
latent space of LLMs, the summary embedding $a$ is a vector that encapsulates
the overall semantic information from the language space of the corresponding
sub-image.

Given a summary embedding $a_{i}^{m}$ for a medium-resolution sub-image
$I^{m}_{i}$, Dragonfly identifies a set of $k$ most similar high-resolution
sub-images $I^{h}_{i1}...I^{h}_{ik}$ from the high-resolution set of the
corresponding region. We use the $a_{i}^{m}$ to match the summary embeddings
$\\{a^{h}_{ij}\\}_{j=1}^{k}$ from the high-resolution set of the corresponding
region by calculating their dot-product embedding similarity. Note that the
number of sub-images of the high-resolution set of the corresponding region is
$s=\frac{x^{h}\times y^{h}}{x^{m}\times y^{m}}$. The corresponding high-
resolution sub-images, identified by their matched embeddings, are then
integrated into the final sequence of tokens for the LLM. This selective
process effectively minimizes redundancy while concentrating on the most
visually significant parts of the image. The mathematical representation of
this process is outlined as follows:

$a^{m}_{i}=\frac{1}{N}\sum_{i=1}^{N}P({v^{m}_{i}})\quad\quad
a^{h}_{j}=\frac{1}{N}\sum_{j=1}^{N}P({v^{h}_{i}})$ (1)

$\\{I^{h}_{ij}\\}_{j=1}^{k}=\operatorname{argmax}_{i_{1}...i_{k}}(\operatorname{sim}(a^{m}_{i},a^{h}_{j})),a^{m}_{i}\in\\{a^{m}\\}_{i=1}^{x^{m}\times
y^{m}},a^{h}_{j}\in\\{a^{h}\\}_{j=s\times i}^{s\times(i+1)}$ (2)

$V=\operatorname{Concat}(\\{t_{1}^{l},...,t_{n}^{l}\\},\\{t^{m}_{i1},...,t^{m}_{in}\\}_{i=1}^{x^{m}\times
y^{m}},\\{\\{t^{h}_{ij1},...,t^{h}_{ijn}\\}_{j=1}^{k}\\}_{i=1}^{x^{m}\times
y^{m}})$ (3)

Following the zoom-in patch selection, the projected tokens
$\\{t_{1}...t_{n}\\}$ from the resized low-resolution image, the sub-images of
the medium-resolution image, and the selected sub-images of the high-
resolution are concatenated to form the latent embeddings $\\{T^{V}\\}$ of the
original image $I$ for the LLM. Dragonfly merges the processed tokens for the
images, $\\{T^{V}\\}$, with the text tokens $\\{T^{T}\\}$ generated by the
LLM’s embedding layer and forwards them to the LLM to produce the final
output.

##  4 Experiments

To evaluate the effectiveness of our architecture, we construct various
baselines with different settings and conduct experiments on multiple popular
multimodal benchmarks. In this section, we first introduce our implementation.
Then, we describe our experiment results on general domain benchmarks.
Finally, we evaluate Dragonfly’s fine-grained understanding of biomedical
tasks.

###  4.1 Implementation

We adopt the popular two-stage visual instruction-tuning framework introduced
in Liu et al. (2023b). In the first stage, the LLM and the vision encoder are
frozen, allowing only the projection layer to be trained. This stage enables
the projection layer to learn how to map vision tokens effectively into the
language space while preserving the established language alignment of the LLM.
The entire model undergoes finetuning in the subsequent stage using a high-
quality visual instruction-tuning dataset. This step is crucial as it enhances
the alignment of visual features with the language space, optimizing the
model’s overall performance in handling vision-language tasks. Dragonfly uses
Llama3-8B-chat (Meta AI, 2024) as the backbone and CLIP-vit-base-patch32
(Radford et al., 2021) as the image encoder. Although there are many visual
encoders with higher resolution and better performance, we chose this encoder
to reduce the effects of the visual encoder and better understand our model
architecture. Could Dragonfly achieve competitive performance compared with
other LMMs with better visual encoders? For both stages, the low resolution
corresponds to 224x224 to match the resolution of the image encoder. We apply
the same any-resolution method from Xu et al. (2023) to segment the image into
multiple sub-images. The any-resolution chooses the resolution from a set of
possible resolution grids closest to the original image aspect ratio. The
medium resolution uses a set of possible grids {(2,2),(1,4),(4,1)}, resulting
in the four sub-images. In the current implementation, the high resolution
uses the {(6,4), (4,6), (3,8), (8,3)} grids, resulting in 24 sub-images, and
we select 12 sub-images from the 24 sub-images in the zoom-in patch selection.
For the first training stage, the model is trained on LLaVA-Pretrain (Liu et
al., 2023b) with 558K image-text pairs using the global batch size 64 with the
learning rate 2e-5. The global batch size in the second training stage is 16,
and the learning rate is 2e-6. Both stages only train the model one epoch on
the training dataset. To support the model training and experiments, we
curated a data mixture with 5,580,560 image-instruction samples from multiple
sources, containing detailed image descriptions, complicated reasoning, and
question-answering tasks. Besides the dataset for the general domain tasks, we
also collect 1.4 million visual instruction samples for the biomedical domain
to evaluate the domain adaption capability of our model. Further details about
the instruction-tuning dataset are provided in Appendix sections A and C. The
training utilized 8 NVIDIA H100 GPUs with DeepSpeed ZeRO stage 1 and lasted
approximately 73 hours.

###  4.2 Results

We construct several baselines to evaluate the effectiveness of the Dragonfly
architecture. For a fair comparison, all baselines utilize the same image
encoder and LLM backbone and are trained for 100,000 steps in training stage 2
on the same data mixture using the same random seed. We use LLaVA-1.6 (Liu et
al., 2023a) as the baseline architecture because this framework resizes the
image into two distinct resolutions—low and medium—to capture different levels
of visual information, aligning with the goals of Dragonfly. The first
baseline (Baseline-1) utilizes grid settings of (1, 1) for low resolution and
{(2, 2), (1, 4), (4,1)} for medium resolution, resulting in a total of five
sub-images. The second baseline (Baseline-2) adopts grid settings of (1, 1)
for low resolution and {(4, 4), (2, 8), (8,2)} for medium resolution,
resulting in 17 sub-images. While Dragonfly uses higher image resolutions and
zoom-in patch selection, it encodes the image with the same number of sub-
images as Baseline-2. The final baseline (Baseline-3) uses grid settings of
(1, 1) for low resolution and {(6, 4), (4, 6), (3, 8), (8, 3)} for medium
resolution, matching the highest resolution of Dragonfly. However, without the
zoom-in patch selection, Baseline-3 generates 25 sub-images. We assess the
baselines and our model across eight popular benchmarks covering commonsense
reasoning QA, image captioning, chart understanding, and hallucination
detection. Table 1 shows the comparison results on these eight benchmarks,
with detailed descriptions of each benchmark provided in Appendix Section B.

Table 1: Comparison results between Dragonfly and baselines on the eight benchmarks. The best results are in bold font, and the second-best performances are underlined. AI2D and ScienceQA are mainly focused on commonsense reasoning. ChartQA evaluates the chart understanding capability. MMMU and MMVet evaluate the unified LMM capabilities. Pope and Hallucination-Bench evaluates the hallucination of the object or image details. COCO tests the image captioning. Benchmark |  Metric |  Baseline-1 |  Baseline-2 |  Baseline-3 |  Ours
---|---|---|---|---|---
AI2D |  Acc |  58.1 |  49.48 |  60.81 |  62.37
ScienceQA |  Acc |  65.79 |  61.63 |  70.17 |  73.14
ChartQA |  Relaxed Acc |  28.6 |  46.24 |  45.92 |  37.43
MMMU-val |  Acc |  33.4 |  35.1 |  36.6 |  36.2
MMVet |  GPT-Eval-Avg |  28.8 |  30.2 |  29.7 |  31.7
Pope-f1 |  F1 Score |  81.81 |  77.06 |  71.59 |  83.23
COCO-val |  CIDEr |  89.02 |  91.20 |  86.84 |  94.07
Hallucination-Bench |  Acc |  41.9 |  41.12 |  44.11 |  41.37
Table 2: Comparison between Dragonfly and existing LMMs. Given the limited space, we only present a subset of popular LMMs with a LLM backbone between 7B - 13B parameters, including InstructBLIP(Dai et al., 2023), Qwen-VL(Bai et al., 2023), VILA(Lin et al., 2024), LLaVA-v1.5(Liu et al., 2023b), LLaVA-NeXT(Liu et al., 2023a), MM1(McKinzie et al., 2024), Idefics2(Laurençon et al., 2024). The LLM backbone, visual encoder, training data, and architecture significantly influence the results. Dragonfly achieves promising performance on AI2D, ScienceQA, MMMU-val, and POPE. The model has lower performance on ChartQA as our training dataset has limited chart data. Note that the results for LLaVA-series models are reproduced using the evaluation code, and other LMMs’ results are reported from their technical papers or reports. Model |  Backbone |  AI2D |  ScienceQA |  ChartQA |  MMMU-val |  MMVet |  POPE |  COCO-Cap
---|---|---|---|---|---|---|---|---
InstructBLIP |  Vicuna-7B |  - |  60.5 |  - |  - |  26.2 |  - |  -
Qwen-VL-Chat |  Qwen-7B |  62.3 |  68.2 |  65.7 |  35.9 |  - |  - |  -
LLaVA-v1.5 |  Vicuna-7B |  54.8 |  70.4 |  18.2 |  35.3 |  30.5 |  85.9 |  -
VILA |  Llama2-7B |  - |  68.2 |  - |  - |  34.9 |  85.5 |  -
LLaVA-NeXT |  Mistral-7B |  60.8 |  72.8 |  38.8 |  33.4 |  44.8 |  86.7 |  109.2
LLaVA-NeXT |  Vicuna-7B |  66.6 |  70.1 |  54.8 |  35.8 |  43.9 |  86.5 |  99.9
LLaVA-NeXT |  Llama3-8B |  60.9 |  73.3 |  - |  36.8 |  - |  86.4 |  -
MM1 |  MM1-7B |  - |  72.6 |  - |  37.0 |  42.1 |  86.6 |  -
Idefics2 |  Mistral-7B |  - |  - |  - |  43.0 |  - |  - |  116.0
InstructBLIP |  Vicuna-13B |  - |  63.1 |  - |  - |  - |  78.9 |  -
LLaVA-v1.5 |  Vicuna-13B |  59.4 |  74.9 |  18.2 |  34.8 |  35.2 |  85.9 |  115.6
VILA |  Llama2-13B |  - |  73.7 |  - |  - |  38.8 |  84.2 |  -
LLaVA-NeXT |  Vicuna-13B |  - |  73.6 |  - |  36.2 |  48.8 |  86.2 |  -
Dragonfly (Ours) |  Llama3-8B |  63.6 |  80.5 |  29.9 |  37.8 |  35.9 |  91.2 |  95.8

Based on the results, Dragonfly outperforms all the baselines in five
benchmarks and achieves competitive performance in the other three. The
results demonstrate the effectiveness of our architecture, particularly in
image understanding and reasoning. Additionally, we note two intriguing
observations from the experiment. First, although Baseline-3 utilizes all sub-
images at the same resolution as Dragonfly, its performance does not
significantly exceed that of Dragonfly or even Baseline-2, suggesting that
some high-resolution sub-images might introduce excessive complexity and
noise. A more comprehensive approach, such as zoom-in patch selection, may be
required to enhance visual feature alignment. Second, Dragonfly underperforms
compared to other baselines on tasks that require OCR or chart understanding.
These tasks typically demand visual information from the entire image, and
zoom-in patch selection may accidentally omit crucial sub-images. To overcome
this limitation, exploring adjustments to the drop ratio or employing a more
capable visual encoder could be considered in future works. After training
Dragonfly on all samples in our processed dataset, Dragonfly achieves
performance that is competitive with or superior to off-the-shelf LMMs across
multiple benchmarks. Table 2 shows the comparison results between Dragonfly
and other LMMs. Many existing models use different image encoders, LLM
backbones, and datasets, making comparing results challenging to interpret.
Despite only being trained via visual instruction tuning and using a smaller
visual encoder than many of the LMMs, Dragonfly still achieved good
performance on AI2D (Kembhavi et al., 2016), ScienceQA (Lu et al., 2022),
MMMU-val(Xiang et al., 2023), and POPE(Yifan et al., 2023), highlighting the
promising potential of its new architecture.

###  4.3 Biomedical Domain Adaptation

Biomedical tasks require LMMs to capture fine-grained information on image
regions, and we hypothesize that this is especially well-suited for
Dragonflyarchitecture. Here, we employed a domain adaptation strategy to
assess our model’s ability to generalize to the biomedical domain and evaluate
its fine-grained image understanding. Starting with a model checkpoint
instruction tuned on a general domain dataset, we implemented a three-step
training process tailored specifically for the biomedical domain to create
Dragonfly-Med.

Figure 2: Examples of Biomedical Visual Question Answering (VQA). The figure
shows three questions along with their gold standard answers and the
corresponding responses from the Dragonfly model.

Figure 3: Examples of Biomedical captioning and radiology report generation
tasks. The figure shows three questions along with their gold standard answers
and the corresponding responses from the Dragonfly model.

Stage 1: Vision Encoder Tuning The initial stage involved tuning the vision
encoder, a crucial step given the limited exposure of the standard CLIP vision
encoder to biomedical images. The training dataset for this phase
predominantly consisted of short caption datasets from sources like LLaVA-Med
(Li et al. (2024a)), Openpath (Huang et al. (2023a)), and MedICaT (Subramanian
et al. (2020)), supplemented by datasets from the general domain LLaVA-
Pretrain (Liu et al. (2024)). This phase included approximately 1.16 million
image-text pairs, split roughly evenly between the general and biomedical
domains. Stage 1 took about 12 hours to train on 8 NVIDIA H100 GPUs.

Stage 2: Comprehensive Model Tuning The second stage involved jointly training
the vision encoder, the language model, and the projection layer. We utilized
a diverse set of datasets including LLaVA-Med-Instruct (Li et al. (2024a)),
MIMIC-III-CXR (Johnson et al. (2019)), Openpath (Huang et al. (2023a)), ROCO
(Pelka et al. (2018)), Kaggle DR, and DDR (Li et al. (2019)). Additionally, we
incorporated training sets from benchmark datasets such as VQA-RAD (Lau et al.
(2018)), SLAKE (Liu et al. (2021)), Path-VQA (He et al. (2020)), IU X-Ray, and
Peir Gross (Demner-Fushman et al. (2016)). The total dataset encompassed 723K
image-text pairs, with the composition being approximately 15% from the
general domain and 85% from the biomedical domain. We additionally included
70K general domain text-only instruction-tuning dataset. The general domain
datasets included Scale up Visual Instruction Tuning (SVIT) Zhao et al.
(2023b), ShareGPT4V Chen et al. (2023a), and ArXivCap Li et al. (2024b). Stage
2 took about 9 hours on 8 NVIDIA H100 GPUs.

Stage 3: Supervised Finetuning The final stage involved supervised finetuning
using combined training datasets from our benchmark tasks: VQA-RAD, SLAKE,
Path-VQA, IU X-Ray, Peir Gross, and subsets of ROCO and MIMIC-CXR. We
finetuned a single model end-to-end on this aggregated training data to
optimize performance across all tasks simultaneously. Stage 3 required about
an hour of training on 8 NVIDIA H100 GPUs.

The results, as reported in Table 3 and 4, are based on this finetuned model
and evaluated against the official held-out test sets of the respective
benchmarks (the details of the biomedical benchmarks are described in Appendix
Section D). For VQA tasks, we use accuracy and token-level F1 (Tu et al.,
2024), while for image captioning and radiology report generation tasks, we
use metrics such as ROUGE-L (Lin, 2004), METEOR (Banerjee & Lavie, 2005), and
CIDEr (Vedantam et al., 2015). These metrics evaluate the fluency of text, the
sequence of content, and the recognition of synonyms and word stems, with
CIDEr specifically tailored for assessing text descriptions of images.

Dragonfly-Med achieves competitive performance across multiple benchmarks,
outperforming existing state-of-the-art methods on the Path-VQA dataset for
both accuracy (closed) and token F1 metrics. Our model attains an accuracy of
90.4% on the SLAKE dataset, which is close to the current state-of-the-art of
91.6%. Notably, Dragonfly-Med outperforms Med-Gemini, a much larger model, on
all VQA tasks. On the image captioning task, Dragonfly achieves state-of-the-
art or competitive results on several metrics across these datasets. Notably,
on the Peir Gross and ROCO datasets, Dragonfly outperforms existing methods on
all three metrics: ROUGE-L, METEOR, and CIDEr. Some of the baseline models are
much larger than our current implementation. Results on biomedical tasks
further support the fine-grained understanding and reasoning capabilities on
image regions of Dragonfly architecture. We present a few examples from our
evaluation tasks, along with Dragonfly’s responses, in Figures 3 and 3.

Table 3: Biomedical VQA evaluation results. Dataset | Metric | LLaVA-Med | Med-Gemini | SOTA |  Dragonfly-Med (Ours)
---|---|---|---|---|---
VQA-RAD | Acc (closed) | 84.2 | 69.7 |  87.1 (Tanwani et al., 2022) |  77.4
Token F1 | - | 50.1 |  62.1 (Tu et al., 2024) |  59.6
SLAKE | Acc (closed) | 83.2 | 84.8 |  91.6 (Yuan et al., 2023) |  90.4
Token F1 | - | 75.8 |  89.3 (Tu et al., 2024) |  88.8
Path-VQA | Acc (closed) | 91.7 | 83.3 |  91.7 (Li et al., 2024a) |  92.3
Token F1 | - | 58.7 |  62.7 (Tu et al., 2024) |  67.6
Table 4: Medical image captioning and clinical report generation evaluation results. For MIMIC-CXR, we specifically focus on generating the findings section of the radiology report. Dataset | Metric | BiomedGPT | SOTA |  Dragonfly-Med (Ours)
---|---|---|---|---
IU X-Ray | ROUGE-L | 28.5 |  44.8 (Zhou et al., 2021) |  28.5
METEOR | 12.9 |  24.2 (Huang et al., 2023b) |  29.7
CIDEr | 40.1 |  43.5 (Wang et al., 2023) |  58.8
Peir Gross | ROUGE-L | 36.0 |  36.0 (Zhang et al., 2023a) |  40.3
METEOR | 15.4 |  15.4 (Zhang et al., 2023a) |  38.3
CIDEr | 122.7 |  122.7 (Zhang et al., 2023a) |  179.9
ROCO | ROUGE-L | 18.2 |  18.2 (Zhang et al., 2023a) |  19.3
METEOR | 7.8 |  7.8 (Zhang et al., 2023a) |  15.1
CIDEr | 24.2 |  24.2 (Zhang et al., 2023a) |  40.1
MIMIC CXR | ROUGE-L | 23.8 |  33.5 (Zhou et al., 2021) |  24.2
METEOR | 14.2 |  19.0 (Zhou et al., 2021) |  22.6
CIDEr | 14.7 |  50.9 (Miura et al., 2020) |  47.2

##  5 Ablation Studies

We conducted several ablation studies to better understand the effects of
architecture design and image resolutions. First, we explored two different
zoom-in selection strategies, which provided insights for our final design.
Second, we investigated the image resolutions in zoom-in patch selection and
discovered that different tasks may require different optimal resolutions.

###  5.1 Zoom-in Selection Strategy

We evaluate two zoom-in selection strategies to select subsets of the sub-
images segmented from the resized high-resolution image. The first strategy,
global selection, uses the summary token from each medium-resolution sub-image
to directly retrieve 12 sub-images from the entire high-resolution set. This
strategy aims to capture the most relevant high-resolution patches based on
image-level embedding, focusing on collecting patches containing the original
image’s most crucial visual features. The second strategy, region selection,
uses the summary token from each medium-resolution sub-image to extract three
sub-images from the corresponding high-resolution regions. Unlike global
selection, region selection focuses on high-resolution patches from four
specific image regions, dividing the medium-resolution image into four sub-
images. We evaluate both strategies by training 200K steps on our data
mixture. Table 5 shows the results on several benchmarks for the two selection
strategies, using the same evaluation metrics as in Table 1. Based on the
ablation studies, region selection slightly outperforms global selection,
although performance varies across different tasks. In our final
implementation, we opted for the region selection strategy because it
surpasses global selection in more benchmarks.

Table 5: Results of the ablation studies on zoom-in selection strategy. Strategy | AI2D | ScienceQA | ChartQA | MMMU-val | Pope-f1
---|---|---|---|---|---
Global selection | 59.45 | 74.60 | 33.12 | 33.4 | 81.93
Region selection | 62.37 | 73.14 | 37.43 | 36.2 | 83.23

###  5.2 Image Resolution

Table 6: Results of the ablation studies on the image resolution for zoom-in patch selection. Resolution Grid | AI2D | ScienceQA | ChartQA | MMMU-val | Pope-f1
---|---|---|---|---|---
{[4,4],[2,8],[8,2]} | 62.24 | 69.59 | 38.44 | 35.8 | 73.67
{[6,4],[4,6],[3,8],[8,3]} | 62.37 | 73.14 | 37.43 | 36.2 | 83.23
{[6,8],[8,6],[4,12],[12,4]} | 58.23 | 75.43 | 31.72 | 37.0 | 60.69

Here, we explore the effects of image resolution. Specifically, we selected
three resolution grid settings for the resized high-resolution image:
{[4,4],[2,8],[8,2]}, {[6,4],[4,6],[3,8],[8,3]}, and
{[6,8],[8,6],[4,12],[12,4]}. Table 6 presents the results. We do not observe a
consistent improvement in performance with higher image resolution. A key
reason is that the resolution of the most original images in these benchmarks
is lower than the resolutions evaluated. Consequently, upsampling the image
may degrade quality and introduce more noise. Additionally, we note that
increasing the resolution may enhance commonsense reasoning capabilities (as
seen in ScienceQA and MMMU), but it also tends to induce more hallucinations
in the model. We plan to conduct further ablation studies on high-resolution
images in the future.

##  6 Conclusion

We present Dragonfly, a novel LMM architecture for fine-grained understanding
and reasoning within image regions. Dragonfly leverages multi-resolution
visual encoding and zoom-in patch selection to focus on high-resolution image
areas of images efficiently. Our results across multiple popular benchmarks
underscore the effectiveness of the architecture, with Dragonfly achieving
competitive performance compared to state-of-the-art LMMs in both general and
biomedical tasks. The performance of Dragonfly-Medis particularly exciting, as
it achieved the best-reported results for pathology QA and several medical
image captioning tasks. We believe that our architecture will benefit the LMM
research community, and we make our models open-source. Looking ahead,
exploring more advanced visual encoders and improved selection strategies
presents an exciting research opportunity. Additionally, exploring how zoom-in
patch selection is integrated during the vision-language pretraining stage
presents another avenue for future research. In the Appendix Section 7, we
further discuss our model’s broader impact and limitations.

##  7 Broader Impact and Limitation

The development of Dragonfly, which focuses on fine-grained image region
understanding and reasoning, has significant implications across multiple
fields. The enhanced performance of Dragonfly on image understanding and
reasoning tasks opens new avenues for applications in numerous domains, mainly
where detailed image analysis is critical. Dragonfly finetuned on biomedical
data achieves promising performance on various benchmarks, highlighting its
ability to interpret accurately and reason about medical images, which can
lead to improved diagnostic tools, aiding healthcare professionals in early
detection and treatment planning, potentially enhancing patient outcomes, and
reducing misdiagnosis. Moreover, Dragonfly can also be applied to scientific
research, education, or industrial applications to benefit the broader
community. Despite the decent performance, some limitations of this work
should also be acknowledged. Firstly, although our architecture enhances
performance in tasks requiring fine-grained details on specific image regions,
it may omit some essential information for tasks that require the high-
resolution details of the whole image, such as optical character recognition
and chart understanding tasks. Exploring the selection ratio is one of our
future research directions. Secondly, the patch selection process itself may
introduce biases. While our method aims to identify the most relevant patches
for specific tasks, the criteria for relevance are predefined and may not
capture all important features of an image. Finally, this work only focuses on
visual instruction-tuning instead of pretraining. Some claims made in this
work may not apply to multimodal pretraining stages. We will explore how
Dragonfly performs during multimodal pretraining in the future.

## References

  * AdeptAI (2023) AdeptAI.  Fuyu-8b: A multimodal architecture for ai agents, 2023.
  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Awadalla et al. (2023) Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _arXiv preprint arXiv:2308.01390_ , 2023.
  * Bai et al. (2023) Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  _arXiv preprint arXiv:2308.12966_ , 2023.
  * Banerjee & Lavie (2005) Banerjee, S. and Lavie, A.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pp. 65–72, 2005.
  * Chen et al. (2024) Chen, G. H., Chen, S., Zhang, R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., and Wang, B.  Allava: Harnessing gpt4v-synthesized data for a lite vision-language model.  _arXiv preprint arXiv:2402.11684_ , 2024.
  * Chen et al. (2023a) Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D.  Sharegpt4v: Improving large multi-modal models with better captions.  _arXiv preprint arXiv:2311.12793_ , 2023a.
  * Chen et al. (2023b) Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D.  Sharegpt4v: Improving large multi-modal models with better captions.  _arXiv preprint arXiv:2311.12793_ , 2023b.
  * Dai et al. (2023) Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2305.06500_ , 2023.
  * Demner-Fushman et al. (2016) Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., and McDonald, C. J.  Preparing a collection of radiology examinations for distribution and retrieval.  _Journal of the American Medical Informatics Association_ , 23(2):304–310, 2016.
  * Guan et al. (2024) Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., Manocha, D., and Zhou, T.  Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models.  _arXiv preprint arXiv:2310.14566_ , 2024.
  * He et al. (2020) He, X., Zhang, Y., Mou, L., Xing, E., and Xie, P.  Pathvqa: 30000+ questions for medical visual question answering.  _arXiv preprint arXiv:2003.10286_ , 2020.
  * Huang et al. (2023a) Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T., and Zou, J.  Leveraging medical twitter to build a visual–language foundation model for pathology ai.  _bioRxiv_ , pp. 2023–03, 2023a.
  * Huang et al. (2023b) Huang, Z., Zhang, X., and Zhang, S.  Kiut: Knowledge-injected u-transformer for radiology report generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 19809–19818, 2023b.
  * Jaegle et al. (2022) Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., Hénaff, O., Botvinick, M. M., Zisserman, A., Vinyals, O., and Carreira, J.  Perceiver io: A general architecture for structured inputs & outputs.  _arXiv preprint arXiv:2107.14795_ , 2022.
  * Jing et al. (2017) Jing, B., Xie, P., and Xing, E.  On the automatic generation of medical imaging reports.  _arXiv preprint arXiv:1711.08195_ , 2017.
  * Johnson et al. (2019) Johnson, A. E., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Mark, R. G., and Horng, S.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ , 6(1):317, 2019.
  * Kembhavi et al. (2016) Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A.  A diagram is worth a dozen images.  _arXiv preprint arXiv:1603:07396_ , 2016.
  * Lau et al. (2018) Lau, J. J., Gayen, S., Ben Abacha, A., and Demner-Fushman, D.  A dataset of clinically generated visual questions and answers about radiology images.  _Scientific data_ , 5(1):1–10, 2018.
  * Laurençon et al. (2024) Laurençon, H., Tronchon, L., Cord, M., and Sanh, V.  What matters when building vision-language models?  _arXiv preprint arXiv:2403.11703_ , 2024.
  * Li et al. (2023) Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., and Liu, Z.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_ , 2023.
  * Li et al. (2024a) Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, J.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _Advances in Neural Information Processing Systems_ , 36, 2024a.
  * Li et al. (2024b) Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., and Liu, Q.  Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models.  _arXiv preprint arXiv:2403.00231_ , 2024b.
  * Li et al. (2019) Li, T., Gao, Y., Wang, K., Guo, S., Liu, H., and Kang, H.  Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening.  _Information Sciences_ , 501:511–522, 2019.
  * Lin (2004) Lin, C.-Y.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pp. 74–81, 2004.
  * Lin et al. (2024) Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S.  Vila: On pre-training for visual language models.  _arXiv preprint arXiv:2312.07533_ , 2024.
  * Liu et al. (2021) Liu, B., Zhan, L.-M., Xu, L., Ma, L., Yang, Y., and Wu, X.-M.  Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering.  In _2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)_ , pp. 1650–1654. IEEE, 2021.
  * Liu et al. (2023a) Liu, H., Li, C., Li, Y., and Lee, Y. J.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023a.
  * Liu et al. (2023b) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning.  _NeurIPS 2023_ , 2023b.
  * Liu et al. (2024) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Lu et al. (2022) Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Masry et al. (2022) Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E.  Chartqa: A benchmark for question answering about charts with visual and logical reasoning.  _ACL 2022 Findings_ , 2022.
  * McKinzie et al. (2024) McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Weers, F., Belyi, A., Zhang, H., Singh, K., Kang, D., Jain, A., Hè, H., Schwarzer, M., Gunter, T., Kong, X., Zhang, A., Wang, J., Wang, C., Du, N., Lei, T., Wiseman, S., Yin, G., Lee, M., Wang, Z., Pang, R., Grasch, P., Toshev, A., and Yang, Y.  Mm1: Methods, analysis & insights from multimodal llm pre-training.  _arXiv preprint arXiv:2403.09611_ , 2024.
  * Meta AI (2024) Meta AI.  Introducing meta llama 3: The most capable openly available llm to date.  _https://ai.meta.com/blog/meta-llama-3/_ , 2024.
  * Miura et al. (2020) Miura, Y., Zhang, Y., Tsai, E. B., Langlotz, C. P., and Jurafsky, D.  Improving factual completeness and consistency of image-to-text radiology report generation.  _arXiv preprint arXiv:2010.10042_ , 2020.
  * Moor et al. (2023) Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C., Reis, E. P., and Rajpurkar, P.  Med-flamingo: a multimodal medical few-shot learner.  In _Machine Learning for Health (ML4H)_ , pp. 353–367. PMLR, 2023.
  * Pelka et al. (2018) Pelka, O., Koitka, S., Rückert, J., Nensa, F., and Friedrich, C. M.  Radiology objects in context (roco): a multimodal image dataset.  In _Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3_ , pp. 180–189. Springer, 2018.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Saab et al. (2024) Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., et al.  Capabilities of gemini models in medicine.  _arXiv preprint arXiv:2404.18416_ , 2024.
  * Subramanian et al. (2020) Subramanian, S., Wang, L. L., Mehta, S., Bogin, B., van Zuylen, M., Parasa, S., Singh, S., Gardner, M., and Hajishirzi, H.  Medicat: A dataset of medical images, captions, and textual references.  _arXiv preprint arXiv:2010.06000_ , 2020.
  * Tanwani et al. (2022) Tanwani, A. K., Barral, J., and Freedman, D.  Repsnet: Combining vision with language for automated medical reports.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pp. 714–724. Springer, 2022.
  * Tsung-Yi et al. (2015) Tsung-Yi, L., Michael, M., Serge, B., Lubomir, B., Ross, G., James, H., Pietro, P., Deva, R., C. Lawrence, Z., and Piotr, D.  Microsoft coco: Common objects in context.  _arXiv preprint arXiv:1405:0312_ , 2015.
  * Tu et al. (2024) Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al.  Towards generalist biomedical ai.  _NEJM AI_ , 1(3):AIoa2300138, 2024.
  * Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., and Parikh, D.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4566–4575, 2015.
  * Wang et al. (2023) Wang, Z., Liu, L., Wang, L., and Zhou, L.  Metransformer: Radiology report generation by transformer with multiple learnable expert tokens.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 11558–11567, June 2023.
  * Wei et al. (2022) Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022.
  * Weihao et al. (2023) Weihao, Y., Zhengyuan, Y., Linjie, L., Jianfeng, W., Kevin, L., Zicheng, L., Xinchao, W., and Lijuan, W.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _arXiv preprint arXiv:2308:02490_ , 2023.
  * Xiang et al. (2023) Xiang, Y., Yuansheng, N., Kai, Z., Tianyu, Z., Ruoqi, L., Ge, Z., Samuel, S., Dongfu, J., Weiming, R., Yuxuan, S., Cong, W., Botao, Y., Ruibin, Y., Renliang, S., Ming, Y., Boyuan, Z., Zhenzhu, Y., Yibo, L., Wenhao, H., Huan, S., Yu, S., and Wenhu, C.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  _arXiv preprint arXiv:2311:16502_ , 2023.
  * Xu et al. (2023) Xu, R., Yao, Y., Guo, Z., Cui, J., Ni, Z., Ge, C., Chua, T.-S., Liu, Z., Sun, M., and Huang, G.  Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images.  _arXiv preprint arXiv:2403.11703_ , 2023.
  * Yang et al. (2023) Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., and Wang, L.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  _arXiv preprint arXiv:2303.11381_ , 2023.
  * Yifan et al. (2023) Yifan, L., Yifan, D., Kun, Z., Jinpeng, W., Wayne, X. Z., and Ji-Rong, W.  Evaluating object hallucination in large vision-language models.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.  URL https://openreview.net/forum?id=xozJw0kZXF.
  * You et al. (2023) You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F., and Yang, Y.  Ferret: Refer and ground anything anywhere at any granularity.  _arXiv preprint arXiv:2310.07704_ , 2023.
  * Yuan et al. (2023) Yuan, Z., Jin, Q., Tan, C., Zhao, Z., Yuan, H., Huang, F., and Huang, S.  Ramm: Retrieval-augmented biomedical visual question answering with multi-modal pre-training.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pp. 547–556, 2023.
  * Zhang et al. (2024) Zhang, H., You, H., Dufter, P., Zhang, B., Chen, C., Chen, H.-Y., Fu, T.-J., Wang, W. Y., Chang, S.-F., Gan, Z., and Yang, Y.  Ferret-v2: An improved baseline for referring and grounding with large language models.  _arXiv preprint arXiv:2403.11703_ , 2024.
  * Zhang et al. (2023a) Zhang, K., Yu, J., Yan, Z., Liu, Y., Adhikarla, E., Fu, S., Chen, X., Chen, C., Zhou, Y., Li, X., et al.  Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks.  _arXiv preprint arXiv:2305.17100_ , 2023a.
  * Zhang et al. (2023b) Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., Valluri, N., et al.  Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs.  _arXiv preprint arXiv:2303.00915_ , 2023b.
  * Zhao et al. (2023a) Zhao, B., Wu, B., He, M., and Huang, T.  Svit: Scaling up visual instruction tuning.  _arXiv preprint arXiv:2307.04087_ , 2023a.
  * Zhao et al. (2023b) Zhao, B., Wu, B., and Huang, T.  Svit: Scaling up visual instruction tuning.  _arXiv preprint arXiv:2307.04087_ , 2023b.
  * Zhou et al. (2021) Zhou, Y., Huang, L., Zhou, T., Fu, H., and Shao, L.  Visual-textual attentive semantic consistency for medical report generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 3985–3994, 2021.

## Supplementary Material

##  Appendix A General Training Data Description

We collected and curated a vision instruction tuning dataset from ShareGPT4V
(Chen et al., 2023b), ALLaVA (Chen et al., 2024), SVIT (Zhao et al., 2023a),
and a subset tasks from the Cauldron (Laurençon et al., 2024). We initially
combined all the samples from these four resources, resulting in nearly 9
million samples. During experiments on the training data, we have several
lessons:

  * •

Increasing the number of training samples during the visual instruction tuning
improves the accuracy of commonsense reasoning tasks but also introduces more
hallucination. To reduce hallucinations, the model should be trained on
specialized data.

  * •

Deduplicating the training samples is important. Duplicated samples may
introduce additional bias during the training.

  * •

Question-answering data can improve the performance on benchmarks but reduce
the details and length of the generated text

Based on the lessons learned, we first deduplicate the image and instruction
pairs. SVIT and ShreGPT4V share the same image set, and to scale up the
samples, SVIT generates multiple instructions for each image. Therefore, for
each image, we only randomly pick eight instructions. The Cauldron dataset is
a massive collection of 50 high-quality datasets converted to the
user/assistant format. However, some of the datasets include math or coding.
We observe some misalignment during the training if we include these datasets.
Thus, we remove five datasets from the Cauldron dataset. After the processing
and deduplication, our final training set has 5,580,560 image-instruction
pairs. Besides the image-instruction pairs, we also add text-only data to keep
the zero-shot capability of the model. We use the data from OpenHermes and
MathInstruct.

##  Appendix B General Benchmarks

Here, we introduce the evaluation benchmarks we used in our experiments.

#### AI2D

AI2D dataset (Kembhavi et al., 2016) is a dataset of illustrative diagrams for
research on diagram understanding and associated question answering. The
dataset covers over 5,000 grade school science diagrams with over 150,000 rich
annotations. We use the test split of AI2D dataset, which contains 3,088
samples.

#### ScienceQA

ScienceQA dataset (Lu et al., 2022) aims to evaluate the multimodal
commonsense reasoning capabilities of the models and is collected from
elementary and high school science curricula, and contains 21,208 multimodel
multiple-choice science questions. This dataset has richer domain diversity
from three subjects: natural science, language science, and social science. In
this paper, we use the popular evaluation split with 4,114 samples.

#### ChartQA

ChartQA dataset (Masry et al., 2022) is a benchmark for question answering
about charts with visual and logical reasoning. This dataset requires the
model to have understanding and reasoning capabilities across the information
of the whole image. We use the test split with 2,500 samples to evaluate our
model.

#### MMMU

Massive Multi-discipline Multimodal Understanding and Reasoning benchmark
(MMMU) (Xiang et al., 2023) is a benchmark to access the expert-level
multimodal understanding capability of foundation models across a broad scope
of tasks. The dataset covers subjects and disciplines including Art, Business,
Health & Medicine, Science, Humanities & Social Science, and Tech &
Engineering and over subfields. This benchmark measures three essential skills
in LMMs: perception, knowledge, and reasoning. Dragonfly performs
competitively on this benchmark, providing evidence of strong multimodal
capabilities.

#### MMVet

MMVet dataset (Weihao et al., 2023) is an evaluation benchmark that examines
LMMs on complicated multimodal tasks. MMVet defines six core vision-language
capabilities and examines the 16 integrations of interest derived from the
capability combination. This dataset uses GPT4 to evaluate the generated open-
ended outputs and provides a unified scoring metric.

#### POPE

POPE dataset (Yifan et al., 2023) aims to evaluate the object hallucination of
LMMs using a polling-based query method. This dataset is widely used to assess
object hallucination more stably and flexibly. We use the test split of this
dataset with 9,000 samples and the F1 score as the evaluation metric.

#### COCO Caption

COCO dataset (Tsung-Yi et al., 2015) is a large-scale object detection,
segmentation, key-point detection, and captioning dataset. The dataset
consists of 328K images. This dataset has multiple tasks, including
captioning, question-answering, object detection, and segmentation. We use the
COCO-2017 validation set to evaluate the model’s captioning capability.

#### Hallucination-Bench

Hallucination-Bench (Guan et al., 2024) is a comprehensive benchmark designed
to evaluate image-context reasoning. It emphasizes nuanced understanding and
interpretation of visual data, which presents significant challenges to
advanced LMMS. This benchmark uses the GPT4 to compute a unified evaluation
score.

##  Appendix C Biomedical Training Data Description

Many public datasets were used in the training and evaluation of Dragonfly.
All datasets were de-identified. Open datasets were used following their
existing licenses.

###  C.1 LLaVA-Med

LLaVA-Med is a dataset for instruction-following tasks involving multi-round
conversations about biomedical images, generated using the language-only model
GPT-4 (Li et al. (2024a)). Specifically, the model is prompted to generate
questions and answers in multi-round formats based on an image caption, as if
it could view the image itself. To assemble the image captions and their
contexts, LLaVA-Med utilizes PMC-15M (Zhang et al. (2023b)) to select images
that contain a single plot. From these, it samples 60,000 image-text pairs
from the five most prevalent imaging modalities: CXR (chest X-ray), CT
(computed tomography), MRI (magnetic resonance imaging), histopathology, and
gross pathology. The dataset also extracts sentences referencing the image
from the original PubMed articles to provide additional context to the
captions. LLaVA-Med offers two primary versions of the dataset: (i) 60K-IM,
which includes inline mentions as context, and (ii) 60K, a similar-sized
dataset that excludes inline mentions in its self-instruct generations.
Furthermore, a supplementary dataset of 500,000 image-caption pairs is
available for alignment purposes. Data link:
https://github.com/microsoft/LLaVA-Med

###  C.2 Medicat

Medicat (Subramanian et al. (2020)) is a dataset of medical figures, captions,
subfigures/subcaptions, and inline references that enables the study of these
figures in context. It consists of 217,000 images from 131,000 open-access
PubMed Central and includes captions, inline references for 74% of figures,
and manually annotated subfigures and subcaptions for a subset of figures.
Data link: https://github.com/allenai/medicat.

###  C.3 MIMIC-III-CXR

The MIMIC-III-CXR dataset (Johnson et al. (2019)) is a substantial publicly
available collection of chest radiographs, containing 377,110 images derived
from 227,827 imaging studies conducted at the Beth Israel Deaconess Medical
Center from 2011 to 2016. Each image in the dataset is paired with structured
labels extracted from free-text radiology reports. The dataset is organized
into training, validation, and testing subsets, with 368,960 images allocated
for training, 2,991 for validation, and 5,159 for testing. To ensure patient
confidentiality, all images have been de-identified. Data link:
https://physionet.org/content/mimic-cxr-jpg/2.1.0/

###  C.4 Openpath

OpenPath dataset is an expansive collection of 208,414 pathology image-text
pairs, making it the largest publicly available pathology image dataset
annotated with text descriptions (Huang et al. (2023a)). This dataset was
meticulously curated using popular pathology-related hashtags recommended by
the United States and Canadian Academy for Pathology (USCAP) and the Pathology
Hashtag Ontology projects. It spans images gathered from Twitter and other
internet sites, including the LAION dataset, collected between March 21, 2006,
and November 15, 2022. The dataset consists of three main components: (1)
Tweets, with 116,504 image-text pairs; (2) Replies, comprising 59,869 pairs
from highly liked responses; and (3) PathLAION, which adds 32,041 pairs from
broader internet sources. Data link:
https://github.com/PathologyFoundation/plip.

###  C.5 Kaggle DR (Diabetic Retinopathy)

The Kaggle website organized a DR detection challenge in 2015 Li et al.
(2019). The California Healthcare Foundation sponsored the competition. The
Kaggle DR dataset consists of 88,702 color fundus images, including 35,126
samples for training and 53,576 samples for testing. Different devices
captured the images under various conditions (e.g., resolutions) at multiple
primary care sites throughout California and elsewhere. For each subject, two
images of the left and right eyes were collected with the same resolution.
Clinicians rate each image for the presence of DR on a scale of 0–4 according
to the ETDRS scale. Data link: https://www.kaggle.com/c/diabetic-retinopathy-
detection.

###  C.6 DDR

DDR is a diabetic retinopathy dataset (Li et al. (2019)) that comprises 13,673
color fundus images collected from 147 hospitals across 23 provinces in China
between 2016 and 2018, ensuring a broad demographic spread by including images
from patients aged 1 to 100, averaging 54.13 years, and almost evenly split
between males (48.23%) and females (51.77%). These images, derived from 9,598
patients and captured using 42 types of fundus cameras, adhere to stringent
photographic standards to ensure clarity and appropriate exposure, focusing on
crucial retinal structures and lesions. All images have been desensitized for
widespread usage and graded for diabetic retinopathy (DR) severity by seven
trained graders using the International Classification of Diabetic
Retinopathy, supplemented by consensus and consultation with experienced
specialists where necessary. Data link: https://github.com/nkicsl/DDR-dataset.

###  C.7 ROCO

The Radiology Objects in Context (ROCO) dataset is a comprehensive collection
of over 81,000 radiology images derived from PubMedCentral’s open-access
biomedical literature (Pelka et al. (2018)). The dataset focuses on analyzing
visual elements and semantic relationships within radiological imagery. It
includes a variety of medical imaging modalities such as Computer Tomography
(CT), Ultrasound, X-ray, Fluoroscopy, Positron Emission Tomography (PET),
Mammography, Magnetic Resonance Imaging (MRI), and Angiography. Each image is
accompanied by detailed metadata, including captions, keywords, and
identifiers from the Unified Medical Language System (UMLS). The ROCO dataset
also features an out-of-class set of approximately 6,000 images, ranging from
synthetic radiology figures to digital art, to aid in improving prediction and
classification tasks. The dataset is split into training, validation, and test
sets with 70,308, 8,782, and 8,786 images, respectively.

###  C.8 VQA-RAD

The VQA-RAD dataset (Lau et al. (2018)) contains 314 radiology images and
2,244 question-answer pairs obtained from CT, MRI, and X-ray examinations,
covering three anatomical regions: the head, abdomen, and chest. It features a
diverse range of question styles, categorized into 11 types: modality, plane,
organ system, abnormalities, etc. Among these, 58% of the question-answer
pairs are closed-ended (yes/no), with the remaining 42% being open-ended. The
dataset is segmented into a training set of 1,790 QA pairs and a testing set
of 451 QA pairs. Our model was trained on the official training set and
evaluated on the official test set. Data link:
https://huggingface.co/datasets/flaviagiammarino/vqa-rad.

###  C.9 SLAKE

The Slake-VQA dataset, annotated by expert physicians (Liu et al. (2021)), is
a comprehensive bilingual (English and Chinese) VQA dataset. It includes 642
images and 14,028 question-answer pairs across three imaging modalities: CXR,
CT, and MRI. This dataset spans various radiological areas, covering body
regions such as the brain, neck, chest, abdomen, and pelvic cavity. It
contains 9,849 VQA samples designated for training, 2,109 for validation, and
2,070 for testing. The questions vary widely, featuring both open-ended (free-
form) and closed-ended (yes/no) types that assess different image
characteristics like plane, quality, position, organ, abnormality, size,
color, shape, and pertinent medical knowledge. We utilized only the English-
language examples from the official dataset divisions, comprising 4,919
training, 1,053 validation, and 1,061 test examples. Our model was trained on
the official training set and evaluated on the official test set. Data link:
https://www.med-vqa.com/slake/

###  C.10 Path-VQA

This dataset comprises question-answer pairs relating to pathology images (He
et al. (2020)). It encompasses a variety of question formats, including open-
ended and closed-ended (yes/no) questions. The dataset is constructed through
automated techniques and draws from two open-access pathology textbooks and a
digital library. It encompasses a total of 32,632 question-answer pairs
derived from 4,289 images. The dataset is partitioned into official training,
validation, and test subsets, containing 19,654, 6,259, and 6,719 QA pairs,
respectively. Our model was trained on the official training set and evaluated
on the official test set. Data link: https://github.com/UCSD-
AI4H/PathVQA/tree/master/data

###  C.11 IU X-ray

The IU X-ray dataset, detailed in Demner-Fushman et al. (2016), is available
through the Open Access Biomedical Image Search Engine (OpenI). This
collection includes radiological exams or cases, each associated with one or
more images, a radiology report, and two sets of tags. The reports consist of
four sections: Comparison, Indication, Findings, and Impression, with the
latter two sections useful for image captioning. The dataset features two
types of tags: MTI tags derived automatically from the report text by the
Medical Text Indexer and manual tags assigned by two trained coders. Overall,
it comprises 3,955 reports and 7,470 frontal and lateral X-ray images. The
dataset is divided into 6,698 samples in the training set and 745 samples in
the test set. Data link: https://github.com/nlpaueb/bioCaption

###  C.12 Peir Gross

The Peir Gross dataset, initially utilized for captioning in research by Jing
et al. (2017), features photographs from medical cases sourced from the
Pathology Education Informational Resource (PEIR) digital library intended for
educational purposes in pathology. This dataset includes 7,443 images from the
Gross collections across 21 pathology sub-categories in PEIR, with each image
paired with a descriptive single-sentence caption. It is organized into two
subsets: 5,172 images for training and 1,289 for testing. Data link:
https://github.com/nlpaueb/bioCaption

##  Appendix D Biomedical Benchmarks

The details of our evaluation benchmarks are discussed in Section C. A
benchmark summary table is also included in 7.

Table 7: Summary of the biomedical evaluation benchmark, which includes vision question answering, image captioning, and report generation across radiology and pathology modalities. We finetuned the model using a subset of the official training set and evaluated it on the official testing set. It should be noted that for MIMIC-CXR and ROCO, we utilized only a portion of the training dataset. Furthermore, for MIMIC-CXR, we selected only those subsets of the test set, including a findings section. Task Type |  Modality |  Dataset | Split
---|---|---|---
|  |  | Train | Test
Visual Question Answering |  Radiology |  VQA-RAD | 1,790 | 451
Radiology |  Slake-VQA | 4,919 | 1,053
Pathology |  Path-VQA | 19,654 | 6,719
Report Generation |  Chest X-ray |  MIMIC-CXR | 25,000 | 3,513
Image Captioning |  Radiology |  ROCO | 25,000 | 8,786
Radiology |  IU X-RAY | 6,698 | 745
|  Pathology |  Peir Gross | 5,172 | 1,289