#  Silkie: Preference Distillation for Large Visual Language Models

Lei Li1222Equal Contribution. Zhihui Xie122footnotemark: 2 Mukai Li1 Shunian
Chen2 Peiyi Wang3 Liang Chen3
Yazheng Yang1 Benyou Wang2 Lingpeng Kong1
1The University of Hong Kong 2The Chinese University of Hong Kong (Shenzhen)
3Peking University
{nlp.lilei, zhxieml, kaikiaia3<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

This paper explores preference distillation for large vision language models
(LVLMs), improving their ability to generate helpful and faithful responses
anchoring the visual context. We first build a vision-language feedback
(VLFeedback) dataset utilizing AI annotation. Specifically, responses are
generated by models sampled from 12 LVLMs, conditioned on multi-modal
instructions sourced from various datasets. We adopt GPT-4V to assess the
generated outputs regarding helpfulness, visual faithfulness, and ethical
considerations. Furthermore, the preference supervision is distilled into
Qwen-VL-Chat through the direct preference optimization (DPO) method. The
resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME
benchmark regarding the perception and cognition capabilities, respectively.
Silkie also demonstrates reduced hallucination by setting a new state-of-the-
art score of 3.02 on the MMHal-Bench benchmark. Further analysis shows that
DPO with our VLFeedback dataset mainly boosts the fine-grained perception and
complex cognition abilities of LVLMs, leading to more comprehensive
improvements compared to human-annotated preference datasets. Project page:
https://vlf-silkie.github.io.

##  1 Introduction

In recent years, there has been a rapid evolution in the development of large
vision language models (LVLMs), epitomized by the remarkable achievements of
the GPT-4V model (OpenAI, 2023a). The integration of language and visual
components through modality alignment (Alayrac et al., 2022; Awadalla et al.,
2023), coupled with supervised fine-tuning on instruction tuning datasets (Liu
et al., 2023c; Li et al., 2023c; Zhu et al., 2023), has endowed LVLMs with
promising perceptual and cognitive abilities (Yang et al., 2023). This
augmentation enables them to engage in nuanced reasoning over images and
deliver responses according to user queries. Despite these advancements, open-
sourced LVLMs still exhibit misalignment issues that pose a potential risk of
generating misleading content without anchoring to the provided visual context
(Li et al., 2023d; Sun et al., 2023), or biased responses against minority
groups (OpenAI, 2023a). Consequently, addressing and rectifying the
misalignment behavior of LVLMs is crucial to enhance their reliability and
mitigate the risk of generating biased outputs.

In this paper, inspired by the success of reinforcement learning from AI
feedback (RLAIF) in improving LLMs alignment with AI preference (Tunstall et
al., 2023; Lee et al., 2023; Cui et al., 2023), we enhance LVLMs by distilling
the preference annotation from a powerful LVLM, i.e., GPT-4V (OpenAI, 2023a).
We first construct a high-quality instruction set by collecting from various
multi-modal instruction tuning sources, including general conversation and
reasoning instructions from LLaVA (Liu et al., 2023c) and SVIT (Zhao et al.,
2023a), academic task instructions from M3IT (Li et al., 2023c), special
datasets such as medical image understanding from PMC-VQA (Zhang et al.,
2023a) and text-rich understanding from LLaVaR and complex visual reasoning
from ComVint (Du et al., 2023). There are 80k instructions in total, covering
65k unique images. We further built a pool of 12 LVLMs including BLIP-family
Li et al. (2023b); Dai et al. (2023), LLaVA-series (Liu et al., 2023c; b; Sun
et al., 2023), Fuyu-8B (Bavishi et al., 2023), Qwen-VL (Bai et al., 2023) and
GPT-4V (OpenAI, 2023a), to obtain corresponding responses conditioned on our
collected instructions. We randomly sample 4 models for each instruction to
obtain the corresponding outputs, resulting in 28k multi-modal instruction and
response pairs for preference annotation. As annotating response preferences
with human annotators is infeasible regarding large-scale responses and can be
cumbersome due to the subjectivity in the annotation process, GPT-4V is
adopted to assess the quality of different model outputs. To achieve this, we
define evaluation templates to evaluate LVLMs outputs from three aspects: (i)
_Helpfulness_ , aims to evaluate whether the response is relevant for the user
query and improves the user understanding for the visual content; (ii) _Visual
Faithfulness_ , focuses on the consistency between the visual clues and
response, aiming for detecting potential ungrounded hallucination; and (iii)
_Ethical Considerations_ , targets to examine whether the response contains
any offensive or biased content. GPT-4V is then queried with the annotation
templates, the image, and corresponding instructions to assess the quality of
different model outputs. The annotated preference ranking dataset, named
vision-language feedback (VLFeedback), enables us to explore the effect of
preference distillation for LVLMs.

With the VLFeedback dataset, we use direct preference optimization (DPO)
(Rafailov et al., 2023a) to build our Silkie model based on Qwen-VL-Chat (Bai
et al., 2023). Our experimental results reveal that the preference
distillation effectively improves the overall capabilities of multi-modal
tasks. For example, on the MME benchmark (Fu et al., 2023), Silkie outperforms
the base model by 6.9% and 9.5% in perception and cognition tasks,
respectively. Furthermore, with visual faithfulness as an annotation aspect,
Silkie could produce responses that are more aligned with the visual context,
indicated by the new state-of-the-art score of 3.02 on the hallucination
evaluation benchmark MMHal-Bench (Sun et al., 2023). Additional in-depth
investigation of performance improvements shows that VLFeedback provides more
pronounced improvements on fine-grained perception tasks such as Optical
Character Recognition (OCR) and complex cognition-level reasoning tasks such
as code reasoning. Intriguingly, we also find that the AI-annotated preference
boosts the LVLMs more consistently than the human-annotated preference dataset
(Yu et al., 2023), potentially due to the comprehensive coverage of our
preference dataset. These findings substantiate the value of our VLFeedback
dataset for future studies towards building LVLMs better aligned with humans.

Figure 1: The annotation framework of our vision-language feedback
(VLFeedback) dataset. We collect instructions from various sources and decode
the corresponding responses using 4 models sampled from the LVLM pool. The
GPT-4V model assesses these responses regarding three aspects, providing
ratings and rationales for the scores.

In summary, the contribution of this work is two-fold: (i) We construct
VLFeedback, a large-scale multi-modal preference dataset annotated by GPT-4V
on three curated aspects, covering 80k multi-modal instructions and decoded by
12 performant LVLMs; (ii) Experimental results and analysis demonstrate that
performing DPO on our VLFeedback dataset improves LVLMs comprehensively,
providing insights for future LVLM alignment research.

##  2 Visual-Language Feedback Dataset

In this section, we elaborate on the construction process of our visual-
language feedback (VLFeedback) dataset, as illustrated in the Figure 1. We
first introduce the multi-modal instructions sources (§2.1), followed by the
details of selected LVLMs for decoding (§2.2) and the annotation with GPT-4V
(§2.3). Finally, we present the statistics of our VLFeedback dataset (§2.4).

###  2.1 Instruction Source

We curate instruction sources from diverse datasets that span various
capabilities of LVLMs across different domains. Our selection encompasses:

  * •

General Vision-Language Instructions: Featuring datasets such as LLaVA (Liu et
al., 2023c) and SVIT (Zhao et al., 2023a), these datasets are constructed by
inputting textual descriptions of images to ChatGPT/GPT-4. They prompt the
generation of visual-related instructions that encompass diverse types,
including detailed descriptions, reasoning processes, and interactive
conversations.

  * •

Academic Vision-Language Instructions: Drawn from 20 samples of each task in
M3IT (Li et al., 2023c), this set offers comprehensive coverage of previous
academic vision-language tasks such as visual question answering, image
captioning and classification.

  * •

Robustness-oriented Vision-Language Instructions: Challenging instructions
from datasets like LRV (Liu et al., 2023a), demanding complex visual reasoning
from LVLMs, and ComVint (Du et al., 2023), which introduces misleading queries
in the instructions, are incorporated to enrich the coverage of our dataset.

  * •

Domain-specific Vision-Language Instructions: We incorporate LLaVAR (Zhang et
al., 2023b), emphasizing text-rich images like documents and logos; PMC-VQA
(Zhang et al., 2023a) for medical images; LLaVAMed (Li et al., 2023a) for
biomedical images; and PCA-EVAL (Chen et al., 2023a), designed for visual
decision-making instructions in embodied environments. These instructions
require domain knowledge that is potentially useful for downstream
applications.

Dataset | Dataset Description | # Sampled Instructions
---|---|---
LLaVA | Visual Instruction Synthesized by GPT-4 | 19,614
SVIT | Scaled-up Visual Instruction Synthesized by GPT-4 | 22,823
LLaVAR | Text-rich Image Understanding | 13,770
LRV | Robust Visual Instruction | 12,357
LLaVAMed | Biomedical Vision-Language Instruction | 5,861
ComVint | Complex Visual Reasoning Instruction | 2,384
PMC-VQA | Medical Image Question Answering | 2,364
M3IT | Academic Vision-Language Tasks | 687
PCA-EVAL | Embodied Decision-making Instruction | 398
Total | Visual instruction in multi-domains | 80,258
Table 1: Descriptions and statistics of multi-modal instructions in our
VLFeedback dataset.

Table 1 summarizes the characteristics and statistics of instruction sources
sampled in our paper.

###  2.2 Model Pool

We have curated a diverse model pool comprising 12 LVLMs to cover recent
advancements, including:

  * •

GPT-4V (OpenAI, 2023a), the proprietary vision language models developed by
OpenAI, which are shown to be powerful on various multi-modal tasks (Yang et
al., 2023).

  * •

LLaVA-series models, which adopt Vicuna models as the backbone and are trained
on the GPT-4 (text-only) synthesized multi-modal dataset. We select the
enhanced version LLaVA-v1.5-7B and LLaVA-v1.5-13B (Liu et al., 2023b), and the
RLHF version with visual faithfulness alignment, LLaVA-RLHF (Sun et al., 2023)
with different image resolutions LLaVA-RLHF-7b-v1.5-224 and LLaVA-
RLHF-13b-v1.5-336.

  * •

Qwen-VL-Chat (Bai et al., 2023), which show promising capabilities on various
vision-language benchmarks with scaled-up multi-modal pre-training and
supervised fine-tuning on curated datasets.

  * •

IDEFICS-9b-Instruct (Laurençon et al., 2023), which is a open-sourced
implementation of Flamingo (Alayrac et al., 2022), supporting interleaved
image-text inputs. After training on publicly available image-text alignment
pairs and instruction tuning datasets, it demonstrates comparable results with
the original closed-source model on various image-text benchmarks.

  * •

Fuyu-8B (Bavishi et al., 2023), which introduces a novel architecture by
segmenting images into patches and training a conditional language model from
scratch, showcasing the great potential to deal with high-resolution images.

  * •

InstructBLIP (Dai et al., 2023), which employs an instruction-aware visual
feature extraction module based on BLIP2 (Li et al., 2023b). We select
InstructBLIP-Vicuna-7B and InstructBLIP-Vicuna-13B with different LLMs as the
backbone models.

  * •

VisualGLM-6B (Du et al., 2022) is an open-sourced, multi-modal dialog language
model supporting images, Chinese, and English.

  * •

MM-ICL (Zhao et al., 2023b), which is built on BLIP2 (Li et al., 2023b) and
has been further enhanced via training on a curated interleaved image-text
dataset to enhance the in-context learning ability. We adopt MMICL-Vicuna-13B
for decoding.

For each instruction, we randomly sample four models for decoding. The
decoding hyper-parameters adhere to the recommendations provided in the
original implementations.

Visual Faithfulness Assessment Definition: Evaluate whether the generated
response is aligned with the image content, avoiding ungrounded statements.
Guidelines: \- Ensure that the generated response accurately reflects the
visual elements present in the image. \- Flag instances where the model
provides ungrounded statements that do not align with the content of the
image. \- Assess the level of consistency between the generated text and the
visual information. Scoring: Rate outputs 1 to 5 based on the following
criteria: 1\. Significantly Inaccurate: The response is significantly
inaccurate and does not align with the image content. 2\. Some Inaccuracy /
Minor Deviations: The response contains some inaccuracies or minor deviations
from the image content. 3\. Moderately Faithful: The response is moderately
faithful but may have subtle inaccuracies. 4\. Faithful: The response is
faithful to the visual elements present in the image. 5\. Highly Faithful: The
response is highly faithful, accurately reflecting the image content. Table 2:
Visual faithfulness assessment annotation guideline for GPT-4V model.

###  2.3 GPT-4V Aided Preference Annotation

Inspired by the recent progress in alignment from AI Feedback (Bai et al.,
2022b; Lee et al., 2023; Cui et al., 2023), we define _Helpfulness_ for
judging whether the response is relevant and helps the user, and _Ethical
Considerations_ to avoid potential inappropriate responses that may contain
toxic content such as biases or violence. Furthermore, considering the
characteristics of LVLMs involving the interaction between modalities, we
design a special _Visual Faithfulness_ criterion to evaluate the response
consistency between modalities. Specifically, we ask the GPT-4V model to
assess the response quality given the original image and instruction, rating
the visual faithfulness from 1 to 5. The annotation template for visual
faithfulness can be found in Table 2, and we include the annotation templates
for helpfulness and harmlessness in Appendix A.

###  2.4 Preference Statistics

We present statistics on the annotated results to elucidate the distribution
of the annotation scores.

Figure 2: Rating distribution of different aspects. Helpfulness and Visual
Faithfulness share similar score distributions, with the majority of decoded
responses evaluated without Ethical Considerations.

#### Score Distribution in Different Aspects

In Figure 2, we illustrate the score distributions for three distinct aspects.
(1) Helpfulness: The majority of samples garnered scores exceeding 4, while a
notable portion of samples received the lowest score. This suggests the
general effectiveness of LVLMs in meeting the intended objectives of the
annotations, indicating the successfully performed instruction tuning. (2)
Visual Faithfulness: Scores for visual faithfulness closely mirror the
distribution observed in the helpfulness evaluation, implying a potential
correlation between these two aspects during the annotation process. The
similarity in distributions suggests that the perceived helpfulness of the
content likely influences judgments on visual faithfulness. (3) Ethical
Considerations: Interestingly, only a limited portion of the annotated
instructions exhibit potential ethical considerations. This observation may be
attributed to the predominant nature of the sampled instructions, which may
not be primarily geared toward red-teaming prompts (Perez et al., 2022)
designed to elicit harmful results from the LVLMs. Notably, this finding
prompts consideration for a more targeted preference annotation focused
explicitly on ethical considerations in future endeavors.

#### Score Differences between Models

Model | Helpfulness | Visual Faithfulness | Ethical Considerations | Average
---|---|---|---|---
GPT-4V | 4.54 | 4.59 | 4.96 | 4.70
LLaVA-1.5-13B | 3.47 | 3.63 | 4.84 | 3.98
Qwen-VL-Chat | 3.33 | 3.62 | 4.86 | 3.94
LLaVA-1.5-7B | 3.40 | 3.54 | 4.83 | 3.92
LLaVA-RLHF-13b-v1.5-336 | 3.49 | 3.40 | 4.75 | 3.88
IDEFICS-9B-Instruct | 3.12 | 3.40 | 4.90 | 3.81
LLaVA-RLHF-7b-v1.5-224 | 3.35 | 3.27 | 4.75 | 3.79
InstructBLIP-Vicuna-7B | 2.71 | 2.96 | 4.80 | 3.49
InstructBLIP-Vicuna-13B | 2.71 | 2.94 | 4.80 | 3.49
Fuyu-8B | 2.53 | 2.82 | 4.77 | 3.37
VisualGLM-6B | 2.24 | 2.27 | 4.55 | 3.02
MMICL-Vicuna-13B | 1.51 | 1.51 | 4.01 | 2.34
Table 3: Average score in three aspects and the overall performance. GPT-4V
shows an evident advantage over open-sourced LVLMs, motivating us to adopt
GPT-4V as a proxy of human annotators.

Table 3 lists the scores of different models regarding three aspects. As the
evaluated LVLMs may adopt the annotated instructions as the training data, we
would like to note that this score comparison could be unfair for certain
models. Nevertheless, GPT-4V demonstrates a clear advantage over open-sourced
LVLMs, showcasing its great potential to serve as a proxy for human annotators
to provide feedback. We further select two representative models, GPT-4V and
Qwen-VL-Chat, to delve into the distribution of annotated scores. Figure 3
depicts the distinctions between these models. Notably, GPT-4V consistently
obtains higher ratings across all three facets, evidenced by a prevalence of
samples with scores equal to or greater than 4, echoing the results in the
average ratings. It is important to acknowledge that GPT-4V’s dominance may
stem from its role as the annotator, introducing a potential bias towards its
own characteristics and proclivity for detailed responses. Despite this, Qwen-
VL-Chat still exhibits better results in the helpfulness and visual
faithfulness evaluation than in the overall performance of all models as
presented in Figure 2. This suggests Qwen-VL-Chat’s commendable competence in
addressing diverse user queries, motivating us to adopt it as a backbone model
for future explorations.

#### Preference Agreement between GPT-4V and Human Annotators

Given that the efficacy of RLHF hinges on accurately rated human preferences
and the AI evaluator can become unstable (Wang et al., 2023), we undertake a
validation experiment by calculating the agreement rate between human
annotators and GPT-4V. We asked three human annotators to compare the overall
quality of two responses given the same annotation guide for GPT-4V. The
experiment is conducted on a subset of 100 randomly sampled comparisons from
our VLFeedback dataset, revealing an impressive average agreement rate of
83.1%. This finding further underscores the reliability of employing GPT-4V
for annotating preference data, substantiating its credibility in this crucial
role.111In Appendix C, we provide examples of human-GPT disagreements,
demonstrating instances where GPT-4V generates incorrect annotations due to
misjudgment regarding visual contents or conflicting rationales.

##  3 Preference Distillation for LVLMs

Previous results have shown that performant open-sourced LVLMs have been
equipped with promising abilities after sufficient instruction tuning.
Therefore, in this work, we explore whether learning from the preference data
can improve LVLMs regarding helpfulness and visual faithfulness. Our method
builds upon the VLFeedback dataset and distills vision-language AI preferences
with direct preference optimization (DPO) (Rafailov et al., 2023b).

#### Task Formulation

Let $x$ be a prompt containing both images and text inputs, and $y_{i}$
denotes the corresponding response generated by model $\pi_{i}$, with scores
annotated by GPT-4V in three aspects: $s_{i}^{h}$ for helpfulness, $s_{i}^{v}$
for visual faithfulness and $s_{i}^{e}$ for ethical consideration,
respectively. To fully utilize the fine-grained annotations in various
aspects, we average the scores of three aspects into an overall rating $s_{i}$
to compare model responses for the same prompt, resulting in an ordered list
of responses $\\{y_{1},\dots,y_{K}\\}$. Following InstructGPT (Ouyang et al.,
2022), the list of $K$ responses is then mapped into $K(K-1)/2$ comparisons.
Pairs with tied scores are disregarded. The final preference dataset
$\mathcal{D}$ used for fine-tuning consists of triples of one prompt and two
responses $\left(x,y_{w},y_{l}\right)$, where $y_{w}$ is the chosen response
with a higher score and $y_{l}$ is the response labeled as rejected.

#### Preference Alignment Optimization

To align models with preference data, the prevalent RLHF pipeline is to
optimize the following objective (Stiennon et al., 2020a):

$\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y\mid
x)}\left[r(x,y)\right]-\beta\mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta}(y\mid
x)\|\pi_{\mathrm{ref}}(y\mid x)\right],$

where $r$ is the reward model and the KL term penalizes deviations of the
current model $\pi_{\theta}$ from the initial model $\pi_{\mathrm{ref}}$. This
optimization can be done in a two-stage manner, by first learning a reward
model $r_{\phi}(x,y)$ on comparison pairs under the Bradley-Terry (BT) model
(Bradley & Terry, 1952) and then using online RL algorithms (e.g., proximal
policy optimization (PPO) (Schulman et al., 2017b)) to optimize the model with
respect to rewards. However, this approach necessitates an additional reward
model and iterating fine-tuning the model and extensive sampling, leading to
training instability and high computational cost. Direct preference
optimization (DPO) mitigates these issues by directly fine-tuning the model on
the preference data, bypassing the reward modeling stage. The key insight
behind DPO is that the optimal policy $\pi^{*}$ has a closed-form solution
with regard to a reward function $r$ and initial policy $\pi_{\mathrm{ref}}$:

$r(x,y)=\beta\frac{\pi^{*}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}+\beta\log
Z(x),$

where $Z$ is the partition function. Under the BT preference model, the
objective becomes:

$\max_{\pi_{\theta}}\mathbb{E}_{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}\log\sigma\left(\beta\log\frac{\pi_{\theta}\left(y_{w}\mid
x\right)}{\pi_{\mathrm{ref}}\left(y_{w}\mid
x\right)}-\beta\log\frac{\pi_{\theta}\left(y_{l}\mid
x\right)}{\pi_{\mathrm{ref}}\left(y_{l}\mid x\right)}\right),$ (1)

where $\sigma$ denotes the sigmoid function. By iterating over the preference
dataset, calculating the objective, and backpropagate Eq. 1 to update the
model parameters, we can distill preference alignment into the target model
$\pi_{\theta}$ to enhance overall capabilities.

Figure 3: Score distribution comparison between GPT-4V and Qwen-VL-Chat.

##  4 Experiments

In this section, we first introduce the experimental setups (§4.1), including
training details, evaluated benchmarks and baseline models for comparison. We
further present the main results and discuss the findings (§4.2), followed by
analysis explorations and a case study (§4.3).

###  4.1 Experimental Settings

#### Training Details

We adopt Qwen-VL-Chat-v1.1 (7B) (Bai et al., 2023) as our backbone model for
experiments.222The evaluation score slightly differs from the scores reported
in Qwen-VL-Chat, which is based on v1.0 and is not publicly accessible. Our
Silkie model and the baseline methods are trained for 3 epochs with the AdamW
optimizer (Loshchilov & Hutter, 2019), $\beta_{1}=0.9$, $\beta_{2}=0.98$, eps
$=10^{-6}$, and a weight decay of 0.05. We apply a cosine learning rate
schedule with a warmup ratio of 0.1 and a peak learning rate of $10^{-5}$. We
use a global batch size of 256. To facilitate efficient training, we utilize
LoRA tuning (Hu et al., 2022). Every single training can be finished within 30
hours with 16 NVIDIA-A100 GPUs.

#### Evaluation Benchmarks

We adopt various multi-modal evaluation benchmarks for comprehensively
understanding the performance. For a comprehensive evaluation of the model
capability, we evaluate LVLMs on MME, consisting of two splits, where MMEP
measures perception abilities through tasks such as and MMEC for assessing
cognition capabilities such as coding and math problems. We further adopt
MMHal-Bench (Sun et al., 2023) to measure visual faithfulness, where the GPT-4
rated score is reported.

#### Compared Models

We include various performant open-sourced LVLMs for comparison. Specifically,
we compare with LLaVA-series models based on Vicuna-7B (Chiang et al., 2023)
for a comparable base LLM scale, including the original LLaVA (Liu et al.,
2023c) trained with GPT-4 (text-only) annotated multi-modal instructions
dataset, LLaVA-1.5 (Liu et al., 2023b) with enhanced image resolution and
further fine-tuning on high-quality datasets, and LLaVA-RLHF (Sun et al.,
2023) trained with a factually augmented RLHF method on a human-annotated
hallucination preference dataset. We also incorporate ShareGPT4V, which
enhances LLaVA-1.5 by training on GPT4V annotated image-caption pairs. To
eliminate the effect of the base LLMs, we compare the preference distillation
performance with the original Qwen-VL-Chat and incorporate two variants of our
method with the same backbone: (i) _Length as Best_ , which utilizes the
longest response in a comparison to serve as the positive pairs and randomly
choose a shorter response to serve as a negative. (ii) _GPT-4V as Best_ ,
which always adopts the response from GPT-4V as the positive sample and the
negative is chosen from the responses of other models.

###  4.2 Results

Model | MMEP | MMEC | MMHal-Bench | MM-Vet
---|---|---|---|---
LLaVA† | 807.0 | 247.9 | - | -
LLaVA-RLHF∗ | - | - | 2.05 | -
LLaVA-v1.5† | 1510.7 | 316.1 | 2.42 | 30.5
\+ SFT (ShareGPT4V)† | 1567.4 | 376.4 | 2.28 | 37.6
Qwen-VL-Chat | 1440.5 | 362.5 | 2.89 | 45.7
\+ DPO (Longest as Best) | 1393.8 | 355.4 | 2.59 | 44.5
\+ DPO (GPT-4V as Best) | 1460.9 | 353.6 | 2.81 | 45.9
\+ SFT (ShareGPT4V)† | 1527.4 | - | - | 45.9
Silkie (Ours) | 1539.6 | 397.1 | 3.02 | 49.9
Improvement (w.r.t. Qwen-VL-Chat) | 6.9%$\uparrow$ | 9.5%$\uparrow$ | 4.5%$\uparrow$ | 9.2%$\uparrow$
Table 4: Performance evaluation on multi-modal benchmarks. The best results
are shown in bold and the second best are in underline. Results with † and ∗
are from Chen et al. (2023b) and Sun et al. (2023), respectively.

#### Main Results

Table 4 illustrates the evaluation results of various models on benchmarks.
After learning the preference data annotated by GPT-4V, Silkie consistently
outperforms the original Qwen-VL-Chat model across all evaluated benchmarks.
Notably, on the MME benchmark, the perception score exhibits a substantial
improvement, rising from 1440.5 to 1539.6, while the cognitive score
experiences a notable increase from 362.5 to 397.1. Similarly, the score on
MM-Vet demonstrates a commendable 9.1% relative enhancement. These performance
advancements underscore the significant benefits of preference distillation on
the overall capabilities of LVLMs. Furthermore, the score for hallucination
evaluation sees an enhancement from 2.89 to 3.02, underscoring the efficacy of
preference distillation in enhancing the visual faithfulness of LVLMs.

#### Comparison to Heuristic Preference Baselines

In comparison to the two baselines, Length as Best yields inferior results
compared to the original base model, suggesting that reward hacking through
the production of lengthy responses (Shen et al., 2023) may not be prevalent
in LVLMs cases. Additionally, selecting the GPT-4V output as the chosen
response (GPT-4V as Best) does not consistently improve performance as much as
preference distillation. For instance, while perception scores increase,
cognition scores decrease on the MME benchmark. Besides, compared with the
training the base model directly on the ShareGPT4V captioning (Chen et al.,
2023b) data, Silkie performs better on MM-Vet and MME perception evaluation.
These findings suggest that direct preference optimization with annotated
pairs could be more beneficial for improving LVLMs comprehensively.

Figure 4: (Left) In-depth analysis on the MME benchmark for the performance
improvements. Our VLFeedback dataset brings clearer gains in OCR recognition
and code reasoning tasks. (Right) Relative performance improvement by
performing DPO with RLHF-V preference data and a subset of our VLFeedback
dataset. Our GPT-4V annotated preference dataset brings more consistent
improvements on four benchmarks.

###  4.3 Analysis

#### In-Depth Analysis of Performance Enhancement

We further perform a breakdown analysis to delve into the improvements in
different aspects to understand the effect of DPO training better. As
illustrated in the left of Figure 4, Silkie consistently outperforms the
original model across various tasks, confirming the effectiveness of our
VLFeedback dataset. Among the perception tasks, i.e., the first 10 groups in
the bar plot, performing DPO brings more pronounced improvements on the OCR
task and fine-grained perception tasks such as artwork understanding. For
cognition capability evaluation tasks, i.e., the last 4 groups, Silkie’s
advantage is more evident on code reasoning and text translation tasks. These
findings suggest that using DPO with our VLFeedback dataset mainly boosts
fine-grained perception abilities and complex cognition-level tasks, rather
than basic visual understanding like recognizing colors and positions.

#### Comparison with Human-annotated Preference Dataset

To assess whether GPT-4V can annotate high-quality preferences in lieu of
human annotators, we compare the performance of two models fine-tuned on
RLHF-V (Yu et al., 2023) and a subset of VLFeedback. RLHF-V encompasses 1.4k
instances of human-annotated preference data, with the goal of mitigating the
model’s hallucination. To match the volume of RLHF-V, we randomly select 1.4k
prompts from the original dataset and create a comparison pair by choosing the
highest-ranked and lowest-ranked responses for each prompt. Our training
protocol mirrors that of our primary experiments, albeit with a reduced 1k
fine-tuning steps to account for the limited data. The outcomes, illustrated
in right of Figure 4, reveal that our VLFeedback dataset significantly
enhances the model’s perceptual capabilities on the MME benchmark and
contributes to improvements in MM-Vet. The performance on MME Cognition and
MMHal-Bench remains consistent, potentially due to the small coverage of the
downsampled pairs. Conversely, while the RLHF-V dataset successfully addresses
hallucination issues on MMHal-Bench, it adversely affects the performance in
MME cognition and MM-Vet evaluations. This discrepancy is attributed to the
narrow scope of RLHF-V, given the time-consuming nature of human annotation.
In contrast, our VLFeedback dataset is annotated automatically, enabling
scalability for comprehensive task coverage to improve the model’s
capabilities substantially.

#### Case Studies

We provide a comparison sample of our Silkie and the original Qwen-VL-Chat
model for an intuitive understanding of the improvement. As shown in the left
part of Figure 5, the original Qwen-Chat-VL produces a misleading assertion,
i.e., _There is no existence of a vase with a red flower on any of the wooden
stools._. Our Silkie instead identifies the wooden stool with a red flower
correctly. In the right part, Silkie shows improved cognition reasoning
ability by correctly answering the scientific-related question. We provide
another comparison of a challenging report generation given a process diagram
in Appendix B, where Silkie composites a well-structured report satisfying the
query better. These results again validate previous conclusions that
performing DPO with our VLFeedback dataset improves the overall capabilities
of LVLMs.

Figure 5: Case studies on evaluation samples from MMHal-Bench (left) and MM-
Vet (right). Our Silkie locates the wooden stools with a red flower without
giving misleading assertions, and correctly answers the scientific-related
question.

##  5 Related Works

#### Large Vision Language Models

The development of large vision language models has been accelerated recently.
Flamingo (Alayrac et al., 2022; Awadalla et al., 2023) and IDEFICS (Laurençon
et al., 2023) have showcased the effectiveness of consolidating LLMs with
vision encoders. The Q-Former from BLIP-2 (Li et al., 2023b) has helped bridge
the gap between the visual and text modalities. InstructBLIP (Dai et al.,
2023) and MM-ICL (Zhao et al., 2023b) further integrate instructions into the
visual-text alignment process for improved in-context learning ability (Dong
et al., 2022). MiniGPT-4 (Zhu et al., 2023) and LLaVA (Liu et al., 2023c; b)
use a single projection layer, while mPLUG-Owl (Ye et al., 2023) adopts LoRA
tuning (Hu et al., 2022), have shown promising results in aligning visual
encoders and LLMs. Qwen-VL-Chat (Bai et al., 2023) has scaled up multi-modal
pre-training. Fuyu-8 (Bavishi et al., 2023) proposes a new architecture by
segmenting images into pixel patches, treating them as visual tokens to train
a condition multi-modal language model directly. We refer readers to Yin et
al. (2023) for a detailed survey of LVLMs. In this paper, we build VLFeedback
by ranking the response of various LVLMs leveraging GPT-4V as an annotator. We
then adopt Qwen-VL-Chat as the base model to perform preference distillation
to build our Silkie, demonstrating the effectiveness of preference
distillation for LVLMs.

#### Preference Alignment

Learning human preferences has become the core topic for building well-behaved
LLMs (OpenAI, 2022; 2023b), where explorations mainly focus on instruction
tuning (Mishra et al., 2022) (alias supervised fine-tuning, SFT) and
reinforcement learning from either human feedback (Stiennon et al., 2020b; Bai
et al., 2022a) or AI feedback (Bai et al., 2022b; Lee et al., 2023).
Instruction tuning trains LLMs to generate desired responses given certain
user queries, demonstrating promisingly results (Mishra et al., 2022; Longpre
et al., 2023) and are highly scalable with techniques such as self-instruction
(Wang et al., 2022). However, humans may hold subtle preference differences
that are easy to elicit but difficult to formalize and automate. Reinforcement
Learning from Human Feedback (RLHF) (Bai et al., 2022b; OpenAI, 2022) has
emerged as a preferred approach to modeling this comparative preference, with
PPO (Schulman et al., 2017a) and DPO (Rafailov et al., 2023a) becoming
representative implementations. There are preliminary explorations with RLHF
for LVLMs. LLaVA-RLHF (Sun et al., 2023) proposes building a human-annotated
factually oriented preference dataset and reduces LLaVA hallucination. RLHF-V
(Yu et al., 2023) enhances LLaVA-RLHF by collecting a more fine-grained
preference annotation dataset on hallucination. In this work, we instead
resort to GPT-4V to explore the effect of AI feedback for LVLMs, showcasing
the great potential of our VLFeedback dataset.

##  6 Conclusions

This paper explores preference distillation for large vision language models
to improve the alignment with humans. To achieve this, we build a vision-
language preference dataset, VLFeedback, consisting of 80k multi-modal
instructions from various sources, accompanied by the responses decoded by 12
LVLMs and the preference annotation by GPT-4V. Our experimental findings
underscore the substantial performance boost achieved by the Silkie model,
distilled with our preference dataset, across various benchmarks when compared
to the original base model. Further analysis demonstrates our our dataset is
particularly advantageous for refining the fine-grained perception and
cognition abilities of LVLMs. Notably, it exhibits greater scalability and
delivers more comprehensive improvements compared to human-annotated
preference datasets. We hope our VLFeedback dataset and the Silkie model will
serve as valuable resources for future endeavors aimed at refining and
aligning large vision language models with human preferences.

## Limitations

One main limitation is the lack of safety-oriented feedback. While we have
designed the annotation aspect to address potential ethical considerations in
responses, as highlighted in the main paper, it is important to note that the
VLFeedback dataset currently lacks sufficient supervision for safety
alignment. To enhance this aspect, future work can involve incorporating
safety-oriented instructions generated through red-teaming techniques. This
will contribute to a more thorough understanding of the alignment coverage and
further strengthen the ethical dimensions of our approach.

Additionally, our work focuses on a limited range of LVLMs and instruction
datasets. The landscape of LVLMs is evolving rapidly, with numerous promising
models and multi-modal instruction datasets emerging during our project.
Despite this, our results on the VLFeedback dataset demonstrate the
effectiveness of the current model and instruction selections. Acknowledging
the dynamism in the field, we recognize the need to expand our coverage to
incorporate the latest LVLMs and diverse instruction datasets. Future
iterations of our work will involve integrating these advancements and
exploring the effects of selection strategies for LVLMs and instructions.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.  Flamingo: a visual language model for few-shot learning.  _ArXiv preprint_ , abs/2204.14198, 2022.
  * Awadalla et al. (2023) Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _ArXiv preprint_ , abs/2308.01390, 2023.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _ArXiv preprint_ , abs/2308.12966, 2023.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Bavishi et al. (2023) Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar.  Introducing our multimodal models, 2023.  URL https://www.adept.ai/blog/fuyu-8b.
  * Bradley & Terry (1952) Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Chen et al. (2023a) Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang.  Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond.  _ArXiv_ , 2023a.
  * Chen et al. (2023b) Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions, 2023b.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
  * Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback, 2023.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _ArXiv preprint_ , abs/2305.06500, 2023.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui.  A survey for in-context learning, 2022.
  * Du et al. (2023) Yifan Du, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Jinpeng Wang, Chuyuan Wang, Mingchen Cai, Ruihua Song, and Ji-Rong Wen.  What makes for good visual instructions? synthesizing complex visual reasoning instructions for visual instruction tuning, 2023.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 320–335, 2022.
  * Fu et al. (2023) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _arXiv preprint arXiv:2306.13394_ , 2023.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_ , 2022.
  * Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh.  Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Li et al. (2023a) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv preprint arXiv:2306.00890_ , 2023a.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ArXiv preprint_ , abs/2301.12597, 2023b.
  * Li et al. (2023c) Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu.  M3IT: A large-scale dataset towards multi-modal multilingual instruction tuning.  _ArXiv preprint_ , abs/2306.04387, 2023c.
  * Li et al. (2023d) Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  _ArXiv preprint_ , abs/2305.10355, 2023d.
  * Liu et al. (2023a) Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Aligning large multi-modal model with robust instruction tuning.  _arXiv preprint arXiv:2306.14565_ , 2023a.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023b.
  * Liu et al. (2023c) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _ArXiv preprint_ , abs/2304.08485, 2023c.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _ArXiv preprint_ , abs/2301.13688, 2023.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3470–3487, 2022.
  * OpenAI (2022) OpenAI.  Introducing chatgpt, 2022.
  * OpenAI (2023a) OpenAI.  Gpt-4v(ision) system card, 2023a.
  * OpenAI (2023b) OpenAI.  Gpt-4 technical report, 2023b.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Perez et al. (2022) Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.  Red teaming language models with language models.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 3419–3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.225.  URL https://aclanthology.org/2022.emnlp-main.225.
  * Rafailov et al. (2023a) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023a.
  * Rafailov et al. (2023b) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=HPuSIXJaa9.
  * Schulman et al. (2017a) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017a.
  * Schulman et al. (2017b) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017b.
  * Shen et al. (2023) Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang.  Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback.  _arXiv preprint arXiv:2310.05199_ , 2023.
  * Stiennon et al. (2020a) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020a.
  * Stiennon et al. (2020b) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020b.
  * Sun et al. (2023) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell.  Aligning large multimodal models with factually augmented rlhf.  _ArXiv preprint_ , abs/2309.14525, 2023.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_ , 2023.
  * Wang et al. (2023) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.  Large language models are not fair evaluators.  _arXiv preprint arXiv:2305.17926_ , 2023.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 5085–5109, 2022.
  * Yang et al. (2023) Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  _arXiv preprint arXiv:2309.17421_ , 9, 2023.
  * Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality, 2023.
  * Yin et al. (2023) Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.  A survey on multimodal large language models.  _arXiv preprint arXiv:2306.13549_ , 2023.
  * Yu et al. (2023) Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua.  Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.  _arxiv_ , 2023.
  * Zhang et al. (2023a) Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Pmc-vqa: Visual instruction tuning for medical visual question answering.  _arXiv preprint arXiv:2305.10415_ , 2023a.
  * Zhang et al. (2023b) Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun.  Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2023b.
  * Zhao et al. (2023a) Bo Zhao, Boya Wu, and Tiejun Huang.  Svit: Scaling up visual instruction tuning.  _ArXiv preprint_ , abs/2307.04087, 2023a.
  * Zhao et al. (2023b) Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang.  Mmicl: Empowering vision-language model with multi-modal in-context learning.  _ArXiv preprint_ , abs/2309.07915, 2023b.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _ArXiv preprint_ , abs/2304.10592, 2023.

##  Appendix A Annotation Templates

Assessment Guidelines Helpfulness Assessment Definition: Carefully read the
user prompt and ensure that the generated response directly addresses the
user’s request. Guidelines: Consider whether the generated text provides
valuable insights, additional context, or relevant information that
contributes positively to the user’s comprehension of the image. Assess
whether the language model accurately follows any specific instructions or
guidelines provided in the prompt. Evaluate the overall contribution of the
response to the user experience. Scoring: Rate outputs 1 to 5 based on the
following criteria: 1\. Not Helpful The response is not relevant or helpful in
addressing the user prompt. 2\. Some Relevance / Minor Helpfulness The
response contains some relevant information but lacks significant helpfulness.
3\. Moderately Helpful The response is moderately helpful but may have some
minor issues. 4\. Helpful The response is helpful and addresses the user
prompt effectively. 5\. Highly Helpful The response is very helpful, providing
valuable insights and enhancing the user’s understanding. Table 5: Helpfulness
assessment annotation guideline for GPT-4V model.  Ethical Considerations
Assessment (Safety, Privacy, Fairness, and Harmfulness) Definition Assess the
model’s ethical considerations in terms of safety, privacy, fairness, and
harmfulness. Guidelines \- Identify if the model gives appropriate warnings or
avoids providing advice on sensitive topics, such as medical images (Safety).
\- Ensure the model refrains from stating identification information in the
image that could compromise personal privacy (Privacy). \- Evaluate the
language model’s responses for fairness in treating individuals and
communities, avoiding biases (Fairness). \- Assess for harmfulness, ensuring
the avoidance of content that may potentially incite violence, be classified
as NSFW (Not Safe For Work), or involve other unmentioned ethical
considerations. Consider any content that could be deemed offensive,
inappropriate, or ethically problematic beyond the explicitly listed criteria.
Scoring Rate outputs 1 to 5 based on the following criteria: 1\. Highly Unsafe
/ Violates Privacy / Unfair / Harmful Content The response is highly unsafe,
violates privacy, exhibits unfair treatment, or contains harmful content. 2\.
Unsafe / Privacy Concerns / Unfair / Potentially Harmful The response raises
safety concerns, privacy issues, displays unfair treatment, or contains
potentially harmful content. 3\. Moderately Ethical The response is moderately
ethical but may have some ethical concerns or subtleties. 4\. Ethical The
response adheres to ethical guidelines, considering safety, privacy, fairness,
and harmfulness. 5\. Highly Ethical / Safe / Privacy-Respecting / Fair /
Harmless The response is highly ethical, safe, respects privacy, exhibits
fairness, and is free from harmful content. Table 6: Ethical consideration
annotation guideline for GPT-4V model.

##  Appendix B Case Study

Figure 6: Case study on a challenging report composition query. The Silkie
model generates a comprehensive report satisfying the word requirement and
provides a better layout for the user to read.

As illustrated in Figure 6, the test sample is a challenging generation query
asking LVLMs to identify the key processes and technologies in the image and
compose a report with word number requirements. While Qwen-VL-Chat generates
the relevant report, it fails to meet the word requirement. Instead, Silkie
generates a comprehensive report satisfying the word requirement and provides
a better layout to improve readability.

##  Appendix C Human Evaluation

We present two examples where all human annotators have different preferences
compared to GPT-4V. In the case shown in Table 7, all human annotators agree
that the rejected answer accurately describes the presence of an analog clock
with a white frame and its location. However, GPT-4V disagrees and harshly
penalizes visual faithfulness by claiming it is not present in the image.
Another case is presented in Table 8 where all human annotators believe the
chosen answer contains hallucinations, such as the existence of ear tags, and
is of poor language quality. However, the GPT-4V annotations fail to capture
these subtle details. The two examples also demonstrate that GPT-4V may have
inconsistent criteria for determining helpfulness, as reflected by how visual
faithfulness contributes to the helpfulness scores of rejected answers.

Question Is there an analog clock with a white frame on the wall? Chosen
Answer There is no existence of an analog clock with a white frame in the
image description. GPT-4V Annotations 1\. Helpfulness (Rating: 5): The
response directly addresses the user’s inquiry by confirming that an analog
clock with a white frame is not present in the image, which is accurate and
thus fully helpful. 2\. Ethical Considerations (Rating: 5): The response is
ethical, does not raise privacy concerns, and contains no potentially harmful
content. 3\. Visual Faithfulness (Rating: 5): The text is highly faithful to
the visual content by correctly asserting that there is no clock with a white
frame visible in the image. Rejected Answer Yes, there is an analog clock with
a white frame on the wall to the right side of the image. GPT-4V Annotations
1\. Helpfulness (Rating: 5) The response is straightforward, confirming the
presence and location of an analog clock with a white frame, which is exactly
what the user asked for, making it fully helpful. 2\. Ethical Considerations
(Rating: 5) The text is ethical and does not contain any content that would
raise safety, privacy, or fairness concerns. 3\. Visual Faithfulness (Rating:
1) The response is not faithful to the image content, as it incorrectly
confirms the presence of an analog clock with a white frame on the wall, which
is not in the image. Table 7: An example of GPT-4V annotation for a sample
from LRV (Liu et al., 2023a). The annotations have been reformatted for
improved readability. Question What are the main objects in this image? Chosen
Answer Upon analyzing the image, I can assert that the main objects are cows.
It is a black and white picture of two cows in a barn stall with one looking
at you. There is also some hay on the ground behind them. The background
appears to be wooden bars like those found in livestock pens or barns. In
addition there is an object hanging from above which could possibly be used
for feeding the animals. On top of this it seems as though there may be other
cattle visible through windows in the back wall but they aren’t clearly
identifiable due to their position relative to the camera angle. Overall the
scene looks rather dull and ordinary - just two cows standing around inside a
barn. However, if you take time to look closely enough, you will see more
details such as the tags attached to each cow ’s ears indicating ownership and
perhaps health information about the animal. This would make the photo quite
interesting even though it doesn ’ t seem so initially. To sum up, the primary
elements captured here are cows and a barn. We can also see tags on the ear of
the cow closest to us. Finally, we have a dark shadow cast over part of the
image. All these GPT-4V Annotations 1\. Helpfulness (Rating: 4) The response
is detailed and describes the image with depth, mentioning not only the cows
but also the barn setting and additional elements like hay, wooden bars, and
ear tags. It loses a point for the incorrect assertion that the image is black
and white and for the unnecessary subjective commentary on the dullness of the
scene. 2\. Ethical Considerations (Rating: 5) This response is safe,
respectful of privacy, fair, and contains no harmful content. It includes
details that are all ethical to discuss, such as observable items in a public
setting. 3\. Visual Faithfulness (Rating: 3) The response is moderately
faithful to the image content, correctly identifying the cows and the barn
setting, but inaccurately describes the image as black and white and includes
speculative details not confirmed by the image. Rejected Answer The main
objects in this image are a cow and a calf, both standing in a barn. GPT-4V
Annotations 1\. Helpfulness (Rating: 1) This response incorrectly identifies a
calf in the image, which is not present; therefore, it does not address the
user’s prompt effectively. 2\. Ethical Considerations (Rating: 5) Despite the
inaccuracy in the content, the response does not raise any ethical concerns
regarding safety, privacy, fairness, or harmfulness. 3\. Visual Faithfulness
(Rating: 1) This response is significantly inaccurate as it incorrectly
identifies a calf in the image, which is not visible in the provided
photograph. Table 8: An example of GPT-4V annotation for a sample from SVIT
(Zhao et al., 2023a). The annotations have been reformatted for improved
readability.