# Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework
for Large Vision Language Models

Chaoya Jiang1, Wei Ye1, Mengfan Dong1, Hongrui Jia1, Haiyang Xu2,
Ming Yan2, Ji Zhang2, Shikun Zhang1
1 National Engineering Research Center for Software Engineering, Peking
University
2 DAMO Academy, Alibaba Group
{jiangchaoya, wye<EMAIL_ADDRESS>
{shuofeng.xhy<EMAIL_ADDRESS>corresponding author.

###### Abstract

Large Vision-Language Models (LVLMs) exhibit remarkable capabilities but
struggle with "hallucinations"—inconsistencies between images and their
descriptions. Previous hallucination evaluation studies on LVLMs have
identified hallucinations in terms of objects, attributes, and relations but
overlooked complex hallucinations that create an entire narrative around a
fictional entity. In this paper, we introduce a refined taxonomy of
hallucinations, featuring a new category: Event Hallucination. We then utilize
advanced LLMs to generate and filter fine-grained hallucinatory data
consisting of various types of hallucinations, with a particular focus on
event hallucinations, laying the groundwork for integrating discriminative and
generative evaluation methods within our universal evaluation framework. The
proposed benchmark distinctively assesses LVLMs’ ability to tackle a broad
spectrum of hallucinations, making it a reliable and comprehensive tool for
gauging LVLMs’ efficacy in handling hallucinations. We will release our code
and data.

Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for
Large Vision Language Models

Chaoya Jiang1, Wei Ye1††thanks:  corresponding author., Mengfan Dong1, Hongrui
Jia1, Haiyang Xu2, Ming Yan2, Ji Zhang2, Shikun Zhang1 1 National Engineering
Research Center for Software Engineering, Peking University 2 DAMO Academy,
Alibaba Group {jiangchaoya, wye<EMAIL_ADDRESS>{shuofeng.xhy,
<EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) such as GPT-4 OpenAI (2023), LLaMA Touvron et al.
(2023a), and LLaMA2 Touvron et al. (2023b) have markedly enhanced capabilities
in natural language understanding (NLU) and generation (NLG). Building on
these advancements, recent Large Vision-Language Models (LVLMs) have shown
increased proficiency in handling both textual and visual information,
sparking significant interest among researchers. Zhu et al. (2023); Liu et al.
(2023e, c); Anil et al. (2023); Dai et al. (2023); Ye et al. (2023); Chen et
al. (2023a); Li et al. (2023d).

Figure 1: Different types of hallucination. Event hallucination, which
involves more complex vision-language discrepancy compared to other types of
hallucination, is commonly overlooked by previous efforts.

Despite the promising developments in LVLMs, they broadly face the pivotal
obstacle of hallucination, which refers to the discrepancy between factual
content in images and the associated generated textual descriptions. As
hallucination poses significant concerns for the LVLMs’ reliability and
robustness Liu et al. (2024); Li et al. (2023e); Liu et al. (2023a); Sun et
al. (2023a); Guan et al. (2023); Zhai et al. (2023), researchers have devised
strategies for hallucinations evaluation to bolster the practical deployment
of LVLMs, including discriminative and generative methods. The former method
directly prompts candidate LVLMs to determine the presence of a particular
hallucination, whereas the latter assesses the text produced by these
candidate LVLMs.

Prior research Liu et al. (2024); Jing et al. (2023); Gunjal et al. (2023) has
delineated vision-language mismatches as issues of non-existent objects,
incorrect object attributes, or inaccurate object relations, yet it does not
encompass the full spectrum of hallucinations observed in LVLMs. For example,
as depicted in Figure 1, LVLM outputs can exhibit more intricate
hallucinations, such as "At the same time, a kangaroo is hopping near the
zebras, seemingly foraging." This type of hallucination invents a fictional
target and weaves an entire narrative around it, including its attributes,
relationships, and actions. We categorize these intricate narratives as event
hallucinations. Presently, there is an absence of fine-grained hallucination
evaluation benchmarks for LVLMs that comprehensively address the various types
of hallucinations—such as artificial objects, relationships, attributes, and
events—while also accommodating both discriminative and generative evaluation
methods.

To this end, we first devise a strategy for creating datasets with fine-
grained hallucination instances. By leveraging the advanced capabilities of
LLMs for generating and filtering hallucinatory data, we develop an efficient
and cost-effective pipeline to assemble samples featuring hallucinations of
objects, relationships, attributes, and events. This pipeline then serves as a
solid foundation for unifying discriminative and generative evaluation
methodologies in our framework:

  * •

For the discriminative evaluation, we construct a dataset that features image
captions with hallucinations generated through our pipeline. Candidate LVLMs
are presented with uniform question templates to determine if a given caption,
produced by us, manifests a specific type of hallucination relative to the
image content.

  * •

For the generative evaluation, our pipeline facilitates the creation of a
large-scale hallucinatory dataset. This dataset serves to fine-tune an LVLM
into a specialized evaluator, namely Hal-Evaluator. This evaluator assesses
LVLM-generated descriptions and their associated images, identifying various
hallucination types without needing reference captions.

We conduct thorough experiments and analysis with five leading LLMs across in-
and out-of-domain datasets within our framework, assessing their performance
in terms of hullucination under both discrimitive and generative paradigm. Our
key findings are:

  * •

Our method mitigates the observed bias in evaluated LVLMs, which often default
to "Yes" responses to discriminative questions in benchmarks like POPE,
promoting more balanced responses.

  * •

Utilizing Chain-of-Thought (COT) significantly helps models minimize
hallucinations during discriminative evaluations, particularly those involving
relationships and events.

  * •

The incidence of hallucinations, especially event hallucinations, increases
with the length of the output. Length control becomes a crucial aspect of
generative evaluations, affecting comparative performance trends among LVLMs
under varied output lengths.

  * •

The hallucinatory samples used to train our evaluator also serve as effective
supervised fine-tuning data for LVLMs, contributing to the reduction of
hallucinations and enhancement of their benchmark performance.

In summary, we introduce a novel hallucination category (event hallucination)
of LVLMs, a fine-grained evaluation framework for LVLMs that spans various
hallucination types and unify discriminative and generative approaches, along
with some groundbreaking insights to guide future research on vision-language
hallucination.

Figure 2: The sub-figure (a) illustrates the pipeline of AFHA. The lower left
sub-figure (b) visualizes the prompt used for generating Object Hallucination
Annotations (Please refer to Appendix B.2 for the prompt of other
hallucination annotations.), while the lower right sub-figure visualizes
examples of annotations for four types of hallucinations.

##  2 AFHA: Automatic Fine-grained Hallucination Annotation Pipline

The existing multimodal hallucination research suffers from a lack of large-
scale datasets with fine-grained annotations specific to hallucinations. To
address this issue, we design AFHA, an Automatic Fine-grained Hallucination
Annotation pipline featuring annotations for four hallucination types and
specific hallucination content.

Data Annotation. We annotated image-text paired data using the GPT-4 prompt
method. We initially established a rigorous definition for various types of
hallucinations. Building upon this groundwork, we engaged GPT-4 to rephrase
the collated image-text pairs in line with the diverse classifications of
hallucinations. This step involved inject distinctive hallucinatory elements
into the original captions. An example of a prompt designed to generate
’object’ annotations is illustrated in Figure 2(b). The outcome of this
procedure was a collection of image descriptions enriched with specified
hallucination categories. Moreover, we delegated to GPT-4 the responsibility
of concocting specific hallucinatory content. Consequently, this strategy
yielded an extensively annotated dataset facilitated by GPT-4, with samples of
annotations across different types of hallucinations displayed in Figure 2(c).
For more details, please refer to the Appendix B.2.

Data Filtering. Following the initial annotation phase, we identified that the
quality of the labeled data remained unsatisfactory. Random sampling revealed
that approximately 30% of the annotated dataset still harbored noise that
failed to meet our stringent labeling criteria. Hence, we proceeded to craft a
tailored prompt to commission GPT-4 for the task of purging and refining the
noisy annotations, a process thoroughly outlined in the Appendix B.3.
Subsequent to GPT-4’s meticulous cleanup operation, a manual verification
process ascertained that over 97% of the data accorded with the stipulated
annotation standards.

##  3 Hal-Eval: A Universal Hallucination Evaluation Benchmark

Building upon AFHA, we have proposed a comprehensive and universal
hallucination evaluation benchmark named Hal-Evaluator. As shown in Figure 3,
Hal-Evaluator includes both Discriminative Evaluation and Generative
Evaluation, and is capable of effectively evaluating different types of
hallucinations. In the below subsections, we will first introduce the
evaluation datasets used in discriminative evaluation, then we will introduce
detailed descriptions of discriminative evaluation. Finally, we will introduce
generative evaluation.

###  3.1 Discrimniative Evaluation

####  3.1.1 Constructing Evaluation Dataset

Data Collection: In the past, evaluation benchmarks such as POPE Li et al.
(2023e) have mainly relied on datasets from manually annotated sources like
COCO Lin et al. (2014). However, these datasets are extensively used in the
training and fine-tuning of many LVLMs, resulting in the evaluation data being
in-domain with these models’ training data. Consequently, the models’ zero-
shot hallucination capabilities cannot be genuinely assessed. To address this,
our evaluation dataset is split into two parts. One part consists of in-domain
evaluation data, composed of image-text pairs from COCO 2014 validation and
COCO 2017 test sets. The other part is sourced from web-based datasets such as
CC Changpinyo et al. (2021), SBU Ordonez et al. (2011), and LAION Schuhmann et
al. (2022), providing out-of-domain data sampled randomly.

Data Annotation: We leveraged the automatic annotation process AFHA scripted
in Section 2 to annotate both in-domain and out-of-domain evaluation data.
This process resulted in 5,000 detailed annotations for in-domain evaluation
data and 5,000 for out-of-domain evaluation data, each delineating
hallucination types and content, as visually depicted in the sample in Sub-
Figure 2 (c). A formalized annotated sample is represented as
$S=\\{I,C^{T},C^{O},C^{R},C^{E},C^{A}\\}$ where I denotes the image, $C^{T}$
represents the correct image caption, and $C^{O},C^{R},C^{E},C^{A}$
respectively denote captions containing Object hallucinations, Relation
hallucinations, Event hallucinations, and Attribute hallucinations.

####  3.1.2 Evaluation Process

In prior work, the discriminative evaluation method proposed for evaluating a
certain type of hallucination directly asked LVLMs if the content of that type
existed in the image. For instance, evaluating object hallucinations directly
inquired about the presence of a specific object. In contrast, we have
proposed a more natural questioning method as following.

Figure 3: This figure provides a schematic of the discriminative evaluation
and generative evaluation used in Hal-Eval.

Prompting LVLMs. Assuming a sample as S, the form of the prompt is as follows:
<Image> I
Caption: $C\in\\{C^{T},C^{O},C^{R},C^{E},C^{A}\\}.$
Question: Does the description in the caption accurately reflect the content
of the image?

By controlling the different types of caption $C$, we can evaluate different
types of hallucinations based on a unified prompt template. For example, we
can set $C=C^{A}$ to evaluate Attribute-type hallucinations.

Evaluation Metric. Similar to POPE Li et al. (2023e), we also use Accuracy,
Precision, Recall, F1 score, and "Yes" ratio as the evaluation metrics. Here,
Accuracy represents the number of correctly answered cases, while Precision
and Recall respectively indicate the proportion of correctly answered
questions with responses "Yes" or "No." F1 score integrates the outcomes of
Precision and Recall, which we select as the primary evaluation metric. The
"Yes ratio" serves as a reference for analyzing model behaviors.

###  3.2 Generative Evaluation

####  3.2.1 Overview

Regarding generative evaluation, current evaluation methods either rely on
proprietary models that require subscription fees, such as GPT-4, or depend on
fine-tuned large language models (LLMs) that necessitate additional ground
truth annotations—a process that is prohibitively expensive. This
significantly restricts the scalability of evaluating models. In response, we
propose Hal-Evaluator, a reference-free, open-source evaluation model designed
specifically to detect hallucinatory content. Hal-Evaluator is fine-tuned on
LLaVA 1.5 Liu et al. (2023c) which is also a LVLM; as illustrated in Figure 3,
it takes as input the description of an image provided by the LVLMs under
evaluation, as well as the corresponding image itself. Following this, it
evaluate whether the description contains hallucinations. If hallucinations
are detected, it provides the specific content and categorization of the
hallucinations. Ultimately, it can even modify the hallucinated information in
the description to output an accurate depiction. In this way, our generative
evaluation eliminates the need for additional reference annotation, enabling
hallucination evaluation based solely on the content of the image.

To train the Hal-Evaluator which capable of effectively identifying different
types of hallucinations, a large-scale, fine-grained hallucinatory image-text
dataset is necessary as they facilitate the refinement of training for Hal-
Evaluator intended to detect and correct hallucinatory content. However, there
currently exists no dataset of this scale with detailed annotations.
Therefore, we initially constructed Hal-Data, the first large-scale, fine-
grained dataset with hallucination annotations, based on the AFHA pipeline.

####  3.2.2 Construction of Hal-Data

This dataset named Hal-Data which is generated based on the automatic
hallucination annotation pipline AFHA and comprises 2 million instances. Hal-
Data comprises two parts: 130K GPT-4 annotated and curated image-text pairs
named Hal-Data 130k, with each pair consisting of an image, a valid image
caption, and a description containing hallucinations. The second part consists
of 2M image-text pairs named Hal-Data 2M, crafted by our caption model trained
on the 130K high-quality captions. In the following section, we provide a
detailed exposition of the process involved in creating the Hal-Data.

Data Collection for Hal-Data 130k To maximize the diversity and
comprehensiveness of our data, we initially compiled approximately 200K images
from various sources, including 80K images from the in-domain COCO dataset Lin
et al. (2014) and 80K web images, such as those from CC Changpinyo et al.
(2021), SBU Ordonez et al. (2011), and LAION Schuhmann et al. (2022).
Additionally, to better align with the style of LVLM outputs, we also
collected 40K image-text datasets from ShareGPT4-V Chen et al. (2023b).
Subsequently, we employed AFHA to annotate this portion of the data, resulting
in a final collection of 130K GPT4 meticulously annotated instances and named
it as Hal-Data 130k.

Generation for Hal-Data 2M. Building upon the Hal-Data 130k dataset, we
endeavored to further expand the scale of our dataset. Due to the high cost
associated with using GPT-4, we leveraged the Hal-Data 130k dataset to fine-
tune the currently open-source large-scale language model LLaMA2 13B Touvron
et al. (2023a), resulting in a hallucination data annotation model named Hal-
Annotator. Thanks to its training on diverse and comprehensive data, the Hal-
Annotator is capable of generating highly quality and content-related
annotations. This approach allows the data scaling phase to proceed without
the need for using the paid GPT-4. To accumulate a substantial volume of high-
quality image-text pairs, we selected a subset of 2 million image-caption
pairs from current public datasets (see Appendix A.1 for more details) and
employed our pre-trained Hal-Annotator to modify the image captions by
introducing different types of hallucinations and annotating them.

####  3.2.3 Instruction tuning of Hal-Evaluator.

We futher developed a dedicated dataset named Hal-Data-SFT for instruction
fine-tuning, focused on the detection and modification of hallucinatory
content in image-text instructions based on Hal-Data. We begin by presenting
an image alongside a description that may contain hallucinations, and we query
whether the description includes hallucinatory elements. During this phase,
30$\%$ of the fine-tuning data consists of image descriptions without
hallucinations, thus forming a single-round dialogue. For the remaining 70$\%$
with hallucinatory descriptions, we engage in multiple rounds of dialogue,
continuing to inquire about the type and content of the hallucinations
present. In the final round, we instruct Hal-Evaluator to revise the image
description to eliminate any hallucinatory content, ultimately providing an
image description devoid of hallucinations.

####  3.2.4 Evaluation Metric

To evaluate the generative hallucination of LVLMs, we prompt them to describe
images from both an in-domain 5K dataset and an out-of-domain 5K dataset
mentioned in Subsection 3.1.1 with short length and longer length. These
descriptions, coupled with the respective images, are then fed into the pre-
trained Hal-Evaluator. Our procedure involves prompting Hal-Evaluator to
evaluate the existence and category of any hallucinatory content. Accuracy
serves as the principal metric for our evaluation, which measures the
proportion of outputs correctly identified as free from hallucinations.
Suppose the number of all outputs is $N$, the outputs that contains
hallucinations is $N_{h}$, the accuracy is calculated as
$A=\frac{N-N_{h}}{N}$. Moreover, we track the probability of various types of
hallucinations, encapsulated in the hallucination ratio. For instances, the
number of output contains the object hallucination is $N^{o}_{h}$, the object
ratio $r_{o}$ is calculated as $r_{O}=\frac{N^{o}_{h}}{N_{h}}$.

##  4 Experiments

Hal-Eval is divided into two distinct segments: Discriminative Evaluation and
Generative Evaluation. We have opted to assess five widely utilized open-
source LVLMs: MiniGPT-4 Zhu et al. (2023), InstructBLIP Dai et al. (2023),
mPLUG-owl Ye et al. (2023), LLaVA Liu et al. (2023e), LLaVA1.5 Liu et al.
(2023c).

Dataset | Type | Model | Accuracy | Precision | Recall | F1 | Yes (%)
---|---|---|---|---|---|---|---
In-domain | Object | mPLUG-Owl | 49.8 | 49.8 | 44.7 | 47.1 | 44.1
LLaVA | 52.6 | 55.5 | 26.3 | 35.7 | 23.6
MiniGPT-4 | 50.4 | 50.3 | 46.5 | 48.3 | 40.2
InstructBLIP | 50.0 | 50.0 | 99.0 | 66.5 | 98.0
LLaVA 1.5 | 62.2 | 76.1 | 35.6 | 48.5 | 23.3
Attribute | mPLUG-Owl | 49.9 | 49.9 | 44.7 | 47.2 | 44.6
LLaVA | 52.8 | 55.9 | 26.3 | 35.8 | 23.5
MiniGPT-4 | 51.1 | 51.1 | 46.5 | 48.7 | 39.4
InstructBLIP | 49.8 | 49.8 | 99.0 | 66.3 | 98.1
LLaVA 1.5 | 62.2 | 76.1 | 35.6 | 48.5 | 23.3
Relation | mPLUG-Owl | 50.4 | 50.5 | 44.7 | 47.4 | 44.7
LLaVA | 52.7 | 55.7 | 26.3 | 35.8 | 23.7
MiniGPT-4 | 50.4 | 50.1 | 46.5 | 48.2 | 40.0
InstructBLIP | 49.8 | 49.9 | 99.0 | 66.3 | 97.7
LLaVA 1.5 | 55.4 | 59.1 | 35.6 | 44.4 | 22.1
Event | mPLUG-Owl | 49.7 | 49.7 | 44.6 | 47.0 | 44.8
LLaVA | 51.5 | 53.0 | 26.3 | 35.1 | 24.8
MiniGPT-4 | 49.6 | 50.0 | 46.5 | 48.2 | 40.3
InstructBLIP | 49.6 | 49.7 | 99.0 | 66.2 | 84.3
LLaVA 1.5 | 62.7 | 77.9 | 35.6 | 48.9 | 22.8
Out-of-domain | Object | mPLUG-Owl | 50.3 | 50.4 | 43.6 | 46.8 | 43.4
LLaVA | 50.7 | 52.7 | 9.0 | 15.3 | 7.2
MiniGPT-4 | 50.3 | 51.7 | 53.6 | 52.6 | 25.0
InstructBLIP | 50.0 | 50.0 | 100.0 | 66.6 | 100.0
LLaVA 1.5 | 59.2 | 86.2 | 21.9 | 35.0 | 18.2
Attribute | mPLUG-Owl | 50.4 | 50.5 | 43.6 | 46.8 | 42.9
LLaVA | 51.8 | 66.5 | 9.0 | 15.8 | 6.2
MiniGPT-4 | 50.0 | 51.5 | 53.6 | 52.6 | 24.7
InstructBLIP | 50.0 | 50.0 | 100.0 | 66.6 | 100.0
LLaVA 1.5 | 58.1 | 79.4 | 21.9 | 34.4 | 13.8
Relation | mPLUG-Owl | 50.0 | 50.0 | 43.6 | 46.6 | 43.1
LLaVA | 50.8 | 57.1 | 9.0 | 15.5 | 7.8
MiniGPT-4 | 49.7 | 50.9 | 53.6 | 52.2 | 24.6
InstructBLIP | 50.0 | 50.0 | 100 | 66.6 | 100.0
LLaVA 1.5 | 53.7 | 60.2 | 21.9 | 32.2 | 12.7
Event | mPLUG-Owl | 50.1 | 50.1 | 43.6 | 46.6 | 43.3
LLaVA | 46.2 | 31.2 | 9.0 | 14.0 | 13.2
MiniGPT-4 | 49.3 | 52.3 | 53.6 | 53.0 | 24.3
InstructBLIP | 50.0 | 50.0 | 100 | 66.6 | 99.9
LLaVA 1.5 | 57.7 | 77.2 | 21.9 | 34.2 | 14.2
Table 1: Results of LVLMs under evaluation of four hallucination types on the in-domain dataset and out-of-domain dataset. Yes denotes the proportion of answering “Yes” to the given question.  Model | Length | In-domain | Out-of-domain
---|---|---|---
Object Ratio | Relation Ratio | Attribute Ratio | Event Ratio | Acc | Object Ratio | Relation Ratio | Attribute Ratio | Event Ratio | Acc
MiniGPT-4 | 28.7 | 36.6 | 30.6 | 16.5 | 10.6 | 69.3 | 45.5 | 20.8 | 19.2 | 14.6 | 66.5
79.6 | 46.2 | 22.5 | 8.0 | 23.4 | 61.4 | 53.7 | 9.7 | 7.2 | 29.6 | 50.1
InstructBLIP | 10.3 | 34.2 | 45.2 | 10.3 | 8.3 | 89.1 | 47.6 | 27.4 | 13.2 | 10.2 | 91.0
80.6 | 25.7 | 12.6 | 16.8 | 51.3 | 35.5 | 19.6 | 11.4 | 15.2 | 59.3 | 41.3
mPLUG-owl | 28.3 | 45.5 | 24.6 | 16.3 | 13.4 | 45.4 | 40.5 | 21.2 | 18.5 | 19.4 | 43.5
78.3 | 46.2 | 9.5 | 12.5 | 31.7 | 27.3 | 45.9 | 9.3 | 4.6 | 40.2 | 29.5
LLaVA | 37.3 | 40.1 | 18.5 | 4.5 | 37.1 | 47.4 | 34.9 | 23.2 | 24.4 | 17.8 | 46.3
88.3 | 45.7 | 9.4 | 3.1 | 42.1 | 23.3 | 38.3 | 7.2 | 2.2 | 52.6 | 26.3
LLaVA1.5 | 10.3 | 23.7 | 58.8 | 10.6 | 7.0 | 55.7 | 30.0 | 48.4 | 11.6 | 10.2 | 49.5
84.5 | 42.2 | 13.0 | 3.6 | 41.4 | 44.6 | 34.6 | 8.8 | 2.7 | 54.3 | 46.4
Table 2: Generative Hallucination Evaluation for LVLMs.

###  4.1 Discriminative Evaluation

####  4.1.1 Main Results

As shown in the Table 1, we evaluate the performance of five models, on
different types of hallucinations, following the method outlined in Subsection
3. First, we noticed that the results derived from the POPE Li et al. (2023e)
metric indicate a significant tendency among most models to favor "yes"
responses. In contrast, within our discriminative evaluations, such a penchant
is exclusively noted in the InstructBLIP model. This distinction serves to
underscore the efficiency of our evaluation strategy in attenuating the
inclination towards "yes" answers that is frequently encountered in LVLMs.
Moreover, LLaVA1.5 and LLaVA exhibit a more pronounced predilection for
hallucinations when tested against out-of-domain datasets as opposed to in-
domain datasets. This trend could possibly be ascribed to the prevalent
incorporation of in-domain imagery during the instruction tuning phase of the
models.

####  4.1.2 Analysis of Discriminative Evaluation

Data Reliability Analysis. Our proposed evaluation dataset comprises 5,000 in-
domain and 5,000 out-of-domain images, which we annotated based on the AFHA
framework. To verify the accuracy of the annotations, we randomly sampled 100
cases each from both the in-domain and out-of-domain data for manual
validation. As shown in the table, we found that after GPT-4’s annotation and
filtering process, the annotation accuracy rate in the in-domain dataset
reached 98%. Meanwhile, the annotation accuracy for the out-of-domain dataset
stood at 97%. This high level of accuracy in both datasets underscores the
effectiveness of our annotation process.

Figure 4: Comparison of LLaVA1.5 and LLaVA 1.5-COT. We report the F1 score for
both of them.

Effectiveness of Chain-Of-Thought (COT) for Mitigating Discriminative
Hallucination. For discriminative evaluation, we employed a chain of thought
(COT) approach to systematically evaluate whether the LVLM matches the content
of images with their respective captions (Refer to Appendix B.4 for more
details). As shown in the Figure 4, we observed a significant reduction in
discriminative hallucinations on both in-domain and out-of-domain datasets
after employing COT to LLaVA 1.5. Our experiments indicate that  utilizing COT
is particularly effective in reducing discriminative hallucinations for LVLMs,
especially for those related to relationships and events. We posit that the
complexity inherent in relationship and event type hallucinations arises from
their intrinsic reliance on contextual and inferential understanding.

###  4.2 Generative Evaluation

####  4.2.1 Main Results

As indicated in Table 2, our investigation has revealed that contemporary
models continue to have a significant inclination toward producing
hallucinations. MiniGPT-4 and InstructBLIP, displayed robust in-domain
accuracy, with the latter achieving 89.1% accuracy when the average output
length was approximately 10 tokens. Both mPLUG-owl and LLaVA showed moderate
performance across all evaluated metrics, whether tested on in-domain or out-
of-domain data. Furthermore, we found that (1) When models generate shorter
outputs, the majority of hallucination content produced by the models
predominantly consisted of objects. (2).As the length of the output increases,
all models became more prone to producing hallucination content, with the
incidence of event hallucinations rising markedly.

Metric | type |  $r$ (%) |  $\rho$ (%) |  $\tau$ (%)
---|---|---|---|---
BLEU-4 | Gen | -1.3 | -7.1 | -4.8
ROUGE-L | Gen | -6.7 | -8.5 | -7.4
GPT4-V | Gen | 42.2 | 38.5 | 31.3
CHAIR | Gen | 17.8 | 19.2 | 18.8
Hal-EML | Gen | 29.8 | 21.6 | 33.7
Hal-Eval-Gen | Gen | 47.34 | 37.20 | 43.43
Table 3: Correlation between each evaluation metric and human judgment on LVLM
hallucinations, measured by Pearson’s $r$, Spearman’s $\rho$, and Kendall’s
$\tau$.  Figure 5: The left sub-figure displays the results of the
discriminative evaluation for GPT-4V and Hal-Evaluator. The right sub-figure
compares the ROUGE-L between hallucination content detected by GPT-4V and Hal-
Evaluator with the annotated hallucination content. Figure 6: The figure
depicts the proportions of different types of hallucinations in the outputs of
InstructBLIP and LLaMA1.5, as well as the gray line illustrating the variation
in accuracy.

####  4.2.2 Analysis of Generative Evaluation

Correlations with Human Evaluations. To verify the correlation between the
generative evaluation and human judgment, we following experiments. We firstly
select the test dataset from COCO 2014 Lin et al. (2014) for human evaluation.
This test set comprises 50 images. Each image is supplemented with five
reference captions and object labels provided by the COCO dataset. We selected
three LVLMs – LLaVA Liu et al. (2023e), mPLUG-owl Ye et al. (2023), and
instructBLIP Dai et al. (2023) – to describe the content of the test set, and
we sought the annotation and evaluation from 15 human annotators to evaluate
the presence of hallucinations in these data. We made a comparison among five
benchmarks: ROUGE-L Lin (2004), BLEU-4 Papineni et al. (2002), CHAIR Rohrbach
et al. (2018), Hal-EML Wang et al. (2023b), GPT4-V along with the module of
Hal-Eval – Hal-Eval-Generative. Table 3 delineates the correlation between
various evaluation metrics and human judgment regarding LVLM faithfulness,
gauged using Pearson’s $r$, Spearman’s $\rho$, and Kendall’s $\tau$. Among the
diverse array of metrics, our generative evalutioin metric distinctly stand
out which demonstrate a robust positive correlation, underlining the superior
alignment with human perceptions.

Effectiveness of Hal-Evaluator. To further verify the effectiveness of Hal-
Evaluator for hallucination detection, we evaluated Hal-Evaluator and GPT-4V
(as candidate LVLMs instead of evaluator here) based on the discriminative
approach scripted in subsection 3.1.2, evaluating the detection of different
types of hallucinations in 5K in-domain data. The results disclosed that Hal-
Evaluator outperforms GPT-4V in hallucination detection ability, as shown in
the Sub-Figure 5 (a).

Analysis of Event Hallucination. We tasked GPT-4V and Hal-Evaluator with
identifying hallucination content in image descriptions of in-domain
evaluation dataset of Hal-Eval. (Noted that our evaluation dataset was also
carefully annotated to mark hallucination content within the descriptions.) We
evaluated the overlap between the hallucination content as identified by
GPT-4V and Hal-Evaluator and the actual hallucination information that exists
in the dataset containing hallucinations. The overlap was quantified using the
ROUGE-L score, as shown in Sub-figure 5 (b). The experiment result indicates
that, for the first three types of hallucinations (object, attribute,
relation), both GPT-4V and Hal-Evaluator are capable of accurately identifying
the majority of hallucination content from image descriptions. However, for
event hallucinations, GPT-4V fails to precisely locate the hallucination
content, whereas Hal-Evaluator exhibits an accurate identifying ability. This
discrepancy underscores the complexity inherent to event hallucinations and
concurrently attests to the reliability of Hal-Evaluator. Additionally, Hal-
Evaluator possesses enhanced evaluation capabilities, allowing it to reduce
reliance on external APIs such as GPT-4V. Furthermore, Hal-Evaluator also can
mitigate hallucinations in LVLMs by correcting hallucination content (See
Appendix C.3).

Impact of LVLM output length on Generative Hallucination. In this section, we
investigate the potential correlation between the output length of LVLMs and
the occurrence of generative hallucinations. We conducted the following
experiments on LLaVA 1.5 and InstructBLIP. Experiments were carried out on
LLaVA 1.5 and InstructBLIP wherein both models were prompted to describe the
content of images with outputs of varying token lengths, using the Hal-
Evaluator to detect and analyze the outputs at each length. The proportions of
different types of hallucinations and the accuracy of outputs without
hallucinations are visualized in the Figure 6. The following findings were
observed: (1) The probability of hallucinations occurring in brief responses
is generally lower. (2) With the increase in output length, the occurrence of
event hallucinations significantly rises. These results shed light on the
relationship between output length and the incidence of event hallucinations.

###  4.3 Utilizing Hal-Data for Supervised Fine-tuning

Figure 7:  Radar Chart of Different Models on Multiple General Benchmarks Model | Dis | Gen
---|---|---
in-domain | out-of-domain | in-domain | out-of-domain | length
LLaVA1.5 | 47.6 | 34.0 | 47.4 | 46.3 | 10.3
Hal-VL | 69.7 | 71.85 | 70.9 | 60.4 | 10.5
Table 4: The evaluation results of Hal-VL on Hal-Eval. For the discriminative
evaluation, we have only listed the average F1 scores for different types of
hallucination. For the generative evaluation, we list the accuracy with short
output length.

To validate whether Hal-Data can assist LVLM in eliminating hallucinations and
improving performance through instructional fine-tuning, we conducted the
following experiment: we constructed instructional data from the high-quality
data labeled for GPT-4, totaling 130K, and after combining this instructional
data with that of LLaVA1.5, we conducted joint fine-tuning training and get
the varient of LLaVA1.5 named Hal-VL. As shown in Table 4 , we evaluate Hal-VL
on Hal-Eval and achieve the best performance which demonstrate that Hal-Data
can aid in the elimination of hallucinations within LVLMs. In addition to
evaluations on Hal-Eval, we also other general benchmarks such as VQA Goyal et
al. (2017), GQA Hudson and Manning (2019), MMBench Liu et al. (2023f), MME Fu
et al. (2023). Seed Bench Li et al. (2023b) and so on. As shown in Figure 7,
the results demonstrate that Hal-Data can also enhances their performance
across general benchmarks. (Refer to Appendix C.1 for more results.)

##  5 Related Work

Large Vision Language Model: Unlike earlier vision-language pre-training
models Li et al. (2021, 2022); Jiang et al. (2023e, b, 2022, c, d), that are
relatively small in scale, based on LLMs, there are three principal approaches
to constructing LVLMs, all demonstrating potential for robust zero-shot
generalization in the vision-language field . Flamingo Alayrac et al. (2022)
leads the way by combining a static vision encoder with a large language model
that has special cross-attention features for linking images and text. On the
other hand, PaLM-E Driess et al. (2023) merges visual features into the
already powerful PaLM Chowdhery et al. (2022) and this method is also used by
models such as LLaVA Liu et al. (2023d) and Shikra Chen et al. (2023a).
However, creating long visual sequences is a limitation here. BLIP-2 Li et al.
(2023c) addressed this by shortening visual feature sequences with a method
inspired by DETR Carion et al. (2020), and this improvement is reflected in
subsequent models like Kosmos-1 Huang et al. (2023), mPLUG-Owl Ye et al.
(2023).

Hallucination Evaluation: LVLM hallucination benchmarks specifically aim at
non-hallucinatory generation or hallucination discernment. These benchmarks
are classified according to the type of evaluation approach they follow:
Discriminative (Dis) or Generative (Gen). POPE Li et al. (2023e), NOPE Lovenia
et al. (2023), and CIEM Hu et al. (2023) are examples of discriminative
benchmarks. Each of these benchmarks exclusively directs attention towards
object hallucinations and utilizes accuracy as their primary evaluation
metric. The metric is calculated by querying the presence of objects within
images and comparing the model’s responses with the ground truth. More recent
research predominantly accentuates generative benchmarks over discriminative
ones. Whilst discriminative benchmarks focus mainly on object-level
hallucinations, generative benchmarks widen their scope to encompass a more
extensive range of hallucinations, such as attribute and relation
hallucinations Gunjal et al. (2023); Liu et al. (2023a); Jing et al. (2023);
Wang et al. (2023b); Sun et al. (2023b); Jiang et al. (2023a).

##  6 Conclusion

We introduce a new category, Event Hallucination, into the study of LVLMs and
develop a unique evaluation framework leveraging advanced LLMs for data
analysis. This approach enhances the understanding and mitigation of
hallucinations in LVLMs, marking a significant step forward in assessing and
improving model performance.

## Limitation

We acknowledge several limitations within our study. Firstly, the annotations
for hallucination types, while extensive, may not encapsulate every possible
variant, suggesting room for expansion in terms of the dataset’s coverage.
Secondly, the Hal-Eval framework, although comprehensive, may still benefit
from further refinement to encompass emerging hallucination patterns as LVLMs
continue to evolve. Lastly, while efforts were made to reduce biases within
LVLMs using our dataset, the potential for residual biases remains,
necessitating continuous monitoring and updating of these models. Future work
should focus on addressing these limitations by expanding the scope of
hallucination types, refining evaluation frameworks, and further exploring the
nuances of hallucination content for the continued betterment of LVLMs.

## Ethics Statement

The datasets used for constructing Hal-Data include COCO Lin et al. (2014),
Conceptual Captions (CC) Changpinyo et al. (2021), SBU Ordonez et al. (2011),
LAION Schuhmann et al. (2022), and ShareGPT4v Chen et al. (2023b). These are
all publicly available datasets, and we have not utilized any private datasets
in our research. We have obtained the unanimous consent and permission from
all annotators involved in the human relevance evaluation experiments. Our
methodology is not specifically engineered to filter out potentially harmful
or biased image descriptions; furthermore, these biases may also be prevalent
in the models that constitute part of our pipeline.

## References

  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736.
  * Anil et al. (2023) Gemini Team Google Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, and Johan Schalkwyk… 2023.  Gemini: A family of highly capable multimodal models.  _ArXiv_ , abs/2312.11805.
  * Bai et al. (2023a) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023a.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _ArXiv_ , abs/2308.12966.
  * Bai et al. (2023b) Jinze Bai, Shuai Bai, Shusheng Yang, et al. 2023b.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568.
  * Chen et al. (2023a) Ke Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023a.  Shikra: Unleashing multimodal llm’s referential dialogue magic.  _ArXiv_ , abs/2306.15195.
  * Chen et al. (2023b) Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023b.  Sharegpt4v: Improving large multi-modal models with better captions.  _ArXiv_ , abs/2311.12793.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.  _J. Mach. Learn. Res._ , 24:240:1–240:113.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, et al. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2305.06500_.
  * Driess et al. (2023) Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. 2023.  Palm-e: An embodied multimodal language model.  In _International Conference on Machine Learning_.
  * Fu et al. (2023) Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. 2023.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _arXiv preprint arXiv:2306.13394_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Guan et al. (2023) Tianrui Guan, Fuxiao Liu, Xiyang Wu, et al. 2023.  Hallusionbench: An advanced diagnostic suite for entangled language hallucination & visual illusion in large vision-language models.  _arXiv preprint arXiv:2310.14566_.
  * Gunjal et al. (2023) Anisha Gunjal, Jihan Yin, and Erhan Bas. 2023.  Detecting and preventing hallucinations in large vision language models.  _arXiv preprint arXiv:2308.06394_.
  * Hu et al. (2023) Hongyu Hu, Jiyuan Zhang, Minyi Zhao, et al. 2023.  Ciem: Contrastive instruction evaluation method for better instruction tuning.  In _NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following_.
  * Huang et al. (2023) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023.  Language is not all you need: Aligning perception with language models.  _ArXiv_ , abs/2302.14045.
  * Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709.
  * Jiang et al. (2023a) Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Mingshi Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. 2023a.  Hallucination augmented contrastive learning for multimodal large language model.  _ArXiv_ , abs/2312.06968.
  * Jiang et al. (2022) Chaoya Jiang, Haiyang Xu, Chenliang Li, Ming Yan, Wei Ye, Shikun Zhang, Bin Bi, and Songfang Huang. 2022.  TRIPS: Efficient vision-and-language pre-training with text-relevant image patch selection.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4084–4096, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Jiang et al. (2023b) Chaoya Jiang, Haiyang Xu, Wei Ye, Qinghao Ye, Chenliang Li, Ming Yan, Bin Bi, Shikun Zhang, Fei Huang, and Songfang Huang. 2023b.  Bus: Efficient and effective vision-language pre-training with bottom-up patch summarization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2900–2910.
  * Jiang et al. (2023c) Chaoya Jiang, Haiyang Xu, Wei Ye, Qinghao Ye, Chenliang Li, Ming Yan, Bin Bi, Shikun Zhang, Fei Huang, and Ji Zhang. 2023c.  Copa: Efficient vision-language pre-training through collaborative object-and patch-text alignment.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 4480–4491.
  * Jiang et al. (2023d) Chaoya Jiang, Wei Ye, Haiyang Xu, Miang yan, Shikun Zhang, Jie Zhang, and Fei Huang. 2023d.  Vision language pre-training by contrastive learning with cross-modal similarity regulation.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Jiang et al. (2023e) Chaoya Jiang, Wei Ye, Haiyang Xu, Qinghao Ye, Mingshi Yan, Ji Zhang, and Shikun Zhang. 2023e.  Timix: Text-aware image mixing for effective vision-language pre-training.  _ArXiv_ , abs/2312.08846.
  * Jing et al. (2023) Liqiang Jing, Ruosen Li, Yunmo Chen, et al. 2023.  Faithscore: Evaluating hallucinations in large vision-language models.  _arXiv preprint arXiv:2311.01477_.
  * Li et al. (2023a) Bo Li, Yuanhan Zhang, Liangyu Chen, et al. 2023a.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_.
  * Li et al. (2023b) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023b.  Seed-bench: Benchmarking multimodal llms with generative comprehension.  _arXiv preprint arXiv:2307.16125_.
  * Li et al. (2023c) Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023c.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ArXiv_ , abs/2301.12597.
  * Li et al. (2023d) Junnan Li, Dongxu Li, Silvio Savarese, et al. 2023d.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR.
  * Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705.
  * Li et al. (2023e) Yifan Li, Yifan Du, Kun Zhou, et al. 2023e.  Evaluating object hallucination in large vision-language models.  In _EMNLP_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer.
  * Liu et al. (2023a) Fuxiao Liu, Kevin Lin, Linjie Li, et al. 2023a.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  _arXiv preprint arXiv:2306.14565_.
  * Liu et al. (2024) Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rong-Zhi Li, and Wei Peng. 2024.  A survey on hallucination in large vision-language models.  _ArXiv_ , abs/2402.00253.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b.  Improved baselines with visual instruction tuning.  _ArXiv_ , abs/2310.03744.
  * Liu et al. (2023c) Haotian Liu, Chunyuan Li, Yuheng Li, et al. 2023c.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_.
  * Liu et al. (2023d) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023d.  Visual instruction tuning.  _ArXiv_ , abs/2304.08485.
  * Liu et al. (2023e) Haotian Liu, Chunyuan Li, Qingyang Wu, et al. 2023e.  Visual instruction tuning.  In _NeurIPS_.
  * Liu et al. (2023f) Yuan Liu, Haodong Duan, Yuanhan Zhang, et al. 2023f.  Mmbench: Is your multi-modal model an all-around player?  _arXiv preprint arXiv:2307.06281_.
  * Lovenia et al. (2023) Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, et al. 2023.  Negative object presence evaluation (nope) to measure object hallucination in vision-language models.  _arXiv preprint arXiv:2310.05338_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011.  Im2text: Describing images using 1 million captioned photographs.  In _Neural Information Processing Systems_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020.  Zero: Memory optimizations toward training trillion parameter models.
  * Rohrbach et al. (2018) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, et al. 2018.  Object hallucination in image captioning.  In _EMNLP_.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294.
  * Sun et al. (2023a) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023a.  Aligning large multimodal models with factually augmented rlhf.  _ArXiv_ , abs/2309.14525.
  * Sun et al. (2023b) Zhiqing Sun, Sheng Shen, Shengcao Cao, et al. 2023b.  Aligning large multimodal models with factually augmented rlhf.  _arXiv preprint arXiv:2309.14525_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023a) Junyang Wang, Yuhang Wang, Guohai Xu, et al. 2023a.  An llm-free multi-dimensional benchmark for mllms hallucination evaluation.  _arXiv preprint arXiv:2311.07397_.
  * Wang et al. (2023b) Junyang Wang, Yiyang Zhou, Guohai Xu, et al. 2023b.  Evaluation and analysis of hallucination in large vision-language models.  _arXiv preprint arXiv:2308.15126_.
  * Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_.
  * Yu et al. (2023) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _arXiv preprint arXiv:2308.02490_.
  * Zhai et al. (2023) Bohan Zhai, Shijia Yang, Xiangchen Zhao, et al. 2023.  Halle-switch: Rethinking and controlling object existence hallucinations in large vision language models for detailed caption.  _arXiv preprint arXiv:2310.01779_.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, et al. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.

Name | COCO | LCS | ShareGPT4V
---|---|---|---
Hal-Data 130K | 40% | 40% | 20%
Hal-Data 2M | 0% | 100 % | 0%
Table 5:  The proportion of data sources in Hal-Data 130K and Hal-Data 2M.  Name | Data Source | Visible | Annotated by | Samples
---|---|---|---|---
M-HalDetect | COCO | ✓ | Human | 4K
HaELM | COCO | ✗ | Human,GPT3.5 | 15K
Hal-Data 130K | LCS, COCO, ShareGPT4V | ✓ | GPT4 | 130K
Hal-Data 2M | LCS | ✓ | Hal-Annotator | 1958K
Table 6:  Comparison of hallucinatory datasets and Hal-Data. ‘LCS’ abbreviates
the LAION Schuhmann et al. (2022), CC Changpinyo et al. (2021), and SBU
Ordonez et al. (2011) datasets. The ‘Visible’ column denotes the image
visibility during captioning, and the last column shows the average character
number of the caption.

##  Appendix A Dataset Analysis for Hal-Data

###  A.1 Data source

For the compilation of the Hal-Data 130K dataset, data were sourced from
several established datasets, including: (1) COCO, (2) Conceptual Captions
(CC), (3) SBU, (4) LAION, and (5) Share-GPT4. Table 5 and Table 6 provides the
detailed distribution ratio of these sources. In the scale-up process of Hal-
Data to 2 million entries, priority was given to incorporating 558,000 data
points from the LLaVA dataset, which constitutes a curated collection derived
from LAION, CC, and SBU. The remaining 1400K entries were likewise obtained
from this trio of datasets, ensuring a consistent and diverse array of data
for robust model training. As shown in Table 6, we also compared with other
hallucinatory datasets like M-HalDetectGunjal et al. (2023) and HaELM Wang et
al. (2023a).

###  A.2 Visualization

as shown in Figure 11, we tokenized the annotations of hallucination content
in Hal-Data 130K, and visualized them via a word cloud. The results shown in
the word cloud align with the definitions of the hallucination content. For
instance, the words in the word cloud for object hallucinations are
distinguishably nouns, and for attribute hallucinations, they largely consist
of features such as colors. This supports the reliability and validity of our
labeled dataset.

##  Appendix B Experiments Settings

###  B.1 Training Setting for Hal-Evaluator and Hal-VL

We followed the original approach of LLaVA 1.5 Liu et al. (2023b), we used the
complete pre-training dataset of LLaVA 1.5 during the first stage of pre-
training. We also keeping the same hyperparameter settings with LLaVA 1.5 .
Our experiments were conducted using 16 NVIDIA A100 GPUs with 80G of memory.
We used Deepspeed Rajbhandari et al. (2020) for Hal-Evaluator and Hal-VL, with
a batch size of 64 on a single GPU.

###  B.2 Setting for GPT-4 Annotation of AFHA

As shown in the Figure 9, we present the prompts used during the annotation
phase for the Hal-Data-130K dataset. We employed a unified prompt template for
different types of hallucinations, only modifying the definition of
hallucination within it. This standardized approach ensures consistency across
the dataset annotations. Through this methodology, each instance of potential
hallucination—be it an object, event, or relationship that does not exist
within the image—is flagged with a corresponding definition tailored to the
type of discrepancy encountered. This provides a structured framework for
subsequently evaluating the frequency and nature of hallucinations produced by
LVLMs when generating image descriptions.

###  B.3 Setting for GPT-4 Filtering of AFHA

Figure 10 provides a clear overview of the designed prompts that are
instrumental in discerning and eliminating aberrant data points classified as
noise within the hallucination annotations. These prompts are meticulously
crafted to navigate through the complex nuances of linguistic and contextual
interpretations that the model may generate, systematically identifying and
separating those annotations that do not adhere to the objective standards of
our dataset. This filtration process is crucial for maintaining the integrity
and reliability of our annotations, thereby ensuring that subsequent analyses
are based on high-quality, noise-free data.

###  B.4 Experiment Setting of Chain-of-Thought for Discriminative Evaluation.

In order to test the impact of Chain-of-Thought on discriminative evaluation,
we have made a simple modification to the discriminative prompt template. The
new prompt template is as follows:

<Image> I
Caption: $C\in\\{C^{T},C^{O},C^{R},C^{E},C^{A}\\}.$
Question: Does the description in the caption accurately reflect the content
of the image?
Please conduct a step-by-step analysis of the image and its associated
caption, thereafter providing an answer to the query.

Method | Overall | Hallucination |  Score in Each Question Type $\uparrow$
---|---|---|---
Score $\uparrow$ |  Rate $\downarrow$ | Attribute | Adversarial | Comparison | Counting | Relation | Environment | Holistic | Other
LLaVA-RLHF${}_{\textsc{7B}}$ Sun et al. (2023a) | 2.05 | 0.68 | 2.92 | 1.83 | 2.42 | 1.92 | 2.25 | 2.25 | 1.75 | 1.08
LLaVA${}_{\textsc{7B}}$ Liu et al. (2023d) | 1.55 | 0.76 | 1.33 | 0.00 | 1.83 | 1.17 | 2.00 | 2.58 | 1.67 | 1.83
LLaVA${}_{\textsc{7B}}$-Hal-Evaluator | 2.12 ($\uparrow 0.56$) | 0.60 ($\downarrow 0.16$) | 2.84 | 2.11 | 2.17 | 1.74 | 2.05 | 2.44 | 1.66 | 1.62
miniGPT-4${}_{\textsc{7B}}$ Liu et al. (2023d) | 1.39 | 0.71 | 0.75 | 1.83 | 2.16 | 0.91 | 1.25 | 1.33 | 0.91 | 1.91
miniGPT-4${}_{\textsc{7B}}$-Hal-Evaluator | 1.89 ($\uparrow 0.40$) | 0.56 ($\downarrow 0.15$) | 1.23 | 2.05 | 2.43 | 1.84 | 2.21 | 2.38 | 1.13 | 1.88
Table 7:  Evaluation results for different MLLMs on MMHal-Bench.  Dataset | Type | Model | Accuracy | Precision | Recall | F1 | Yes (%)
---|---|---|---|---|---|---|---
In-domain | Object | LLaVA 1.5 | 62.2 | 76.1 | 35.6 | 48.5 | 23.3
Hal-VL | 72.1 | 89.1 | 58.3 | 70.3 | 40.4
Attribute | LLaVA 1.5 | 62.2 | 76.1 | 35.6 | 48.5 | 23.3
Hal-VL | 72.1 | 83.2 | 60.3 | 69.9 | 38.2
Relation | LLaVA 1.5 | 55.4 | 59.1 | 35.6 | 44.4 | 22.1
Hal-VL | 66.0 | 79.1 | 60.2 | 68.3 | 28.2
Event | LLaVA 1.5 | 62.7 | 77.9 | 35.6 | 48.9 | 22.8
Hal-VL | 74.2 | 86.2 | 55.3 | 70.22 | 26.1
Out-of-domain | Object | LLaVA 1.5 | 59.2 | 86.2 | 21.9 | 35.0 | 18.2
Hal-VL | 78.0 | 94.4 | 59.6 | 73.1 | 37.6
Attribute | LLaVA 1.5 | 58.1 | 79.4 | 21.9 | 34.4 | 13.8
Hal-VL | 77.9 | 93.9 | 59.6 | 72.9 | 31.7
Relation | LLaVA 1.5 | 53.7 | 60.2 | 21.9 | 32.2 | 12.7
Hal-VL | 72.0 | 79.2 | 59.6 | 68.0 | 31.6
Event | LLaVA 1.5 | 57.7 | 77.2 | 21.9 | 34.2 | 14.2
Hal-VL | 78.4 | 95.5 | 59.6 | 73.4 | 31.2
Table 8: Results of LLaVA 1.5 and Hal-VL models under discriminative evaluation of four hallucination types on the in-domain dataset and out-of-domain dataset. Yes denotes the proportion of answering “Yes” to the given question.  Model | In-domain | Out-of-domain | Length
---|---|---|---
Object Ratio | Relation Ratio | Attribute Ratio | Event Ratio | Acc | Object Ratio | Relation Ratio | Attribute Ratio | Event Ratio | Acc
LLaVA1.5 | 23.7 | 58.8 | 10.6 | 7.0 | 55.7 | 30.0 | 48.4 | 11.6 | 10.2 | 49.5 | 10.3
LLaVA1.5 | 42.2 | 13.0 | 3.6 | 41.4 | 44.6 | 34.6 | 8.8 | 2.7 | 54.3 | 46.4 | 84.5
Hal-VL | 35.0 | 37.5 | 21.5 | 6.2 | 70.9 | 29.4 | 30.4 | 20.4 | 10.1 | 60.4 | 10.5
Hal-VL | 31.3 | 22.9 | 23.8 | 22.2 | 64.1 | 27.8 | 17.7 | 22.2 | 32.5 | 56.9 | 85.5
Table 9: Generative Hallucination Evaluation for Hal-VL and LLaVA 1.5.

##  Appendix C More Experiments

###  C.1 More result of Hal-VL.

In this section, we provide detailed evaluations of Hal-VL on more general
benchmarks and Hal-Eval (Table 8 and Table 9). As indicated in the Table 10
and Table 11, we found that Hal-VL achieved significant advantages on the vast
majority of these benchmarks. This demonstrates its robustness and
generalizability across different tasks and datasets, underlining its
effectiveness in handling hallucination problems in LVLMs.

|  | General VQA | General VQA (Zero-shot)
---|---|---|---
Method | #Params | VQAv2 | GQA | VizWizQA | TextVQA | SciQA (IMG)
BLIP-2 Li et al. (2023d) | 8.2B | 65.0 | 41.0 | 19.6 | 42.5 | 61.0
InstructBLIP Dai et al. (2023) | 8.2B | - | 49.2 | 34.5 | $50.1^{\dagger}$ | 60.5
Unified-IO${}_{\text{XL}}$ | 2.9B | 77.9 | - | $57.4^{\ddagger}$ | - | -
PaLM-E-12B | 12B | 76.2 | - | - | - | -
Shikra | 7.2B | 77.4 | - | - | - | -
Qwen-VL-Chat Bai et al. (2023b) | 9.6B | 78.2 | 57.5 | 38.9 | $61.5^{\ddagger}$ | 68.2
LLaVA Liu et al. (2023e) | 7.2B | 71.3 | 41.3 | 36.7 | $50.2^{\dagger}$ | 61.5
MiniGPT-4 Zhu et al. (2023) | 7.2B | 65.2 | 30.8 | 30.2 | $52.3^{\dagger}$ | 58.4
LLaVA1.5 Liu et al. (2023c) | 7.2B | 78.5 | 62.0 | 50.0 | $58.2^{\dagger}$ | 66.8
Hal-VL | 7.2B | 79.3 | 62.8 | 50.7 | $60.1^{\dagger}$ | 68.1
Table 10: Performance comparison on visual question answering. For VQA,
accuracy is reported. Note that specialists are fine-tuned on each individual
dataset. † denotes OCR inputs are utilized. ‡ indicates the model has trained
on the dataset. We gray out those specialists’ methods which are individually
fine-tuned on the dataset as well as those fine-tuned results of generalists.
Figure 8:  Evaluation results on Hal-Eval-Dis after fine-tuning Hal-Evaluator
using SFT data of varying sizes. We have calculated the average F1 value
across all types of hallucinations.

###  C.2  Impact of SFT Data Scales for Hal-Evaluator

To investigate the effect of different data scales on the efficacy of Hal-
Evaluator, we enlarged the scale of the SFT fine-tuning data from 130K to 2M
and evaluated Hal-Evaluator on Hal-Eval-Dis. The experimental outcomes reveal
that the application of 2M-scale data resulted in the highest accuracy in
identifying hallucinations. This highlights the importance of data scale in
enhancing the performance of such evaluators.

Method | MME | MMBench | MM-Vet | SEED-Bench
---|---|---|---|---
BLIP-2 Li et al. (2023d) | 1293.84 | - | 22.4 | 46.4
mPLUG-Owl Ye et al. (2023) | 967.34 | 46.6 | - | 34.0
InstructBLIP Dai et al. (2023) | 1212.82 | 36.0 | 26.2 | 53.4
Otter Li et al. (2023a) | 1292.26 | 48.3 | 24.6 | 32.9
Qwen-VL-Chat Bai et al. (2023a) | 1487.58 | 60.6 | - | 58.2
LLaVA Liu et al. (2023e) | 502.82 | 36.2 | 28.1 | 33.5
MiniGPT-4 Zhu et al. (2023) | 581.67 | 23.0 | 22.1 | 42.8
LLaVA-1.5 Liu et al. (2023c) | 1510.70 | 64.3 | 30.5 | 58.6
Hal-VL | 1620.10 | 66.3 | 31.3 | 59.4
Table 11: Zero-shot multi-modal evaluation on multi-modal benchmarks including MME Fu et al. (2023), MMBench Liu et al. (2023f), MM-Vet Yu et al. (2023), SEED-Bench Li et al. (2023b). The overall scores are reported for evaluation. For MMBench, we report test results. Benchmark | Tasks | Discriminative Hallucination | Generative Hallucination
---|---|---|---
Dis | Gen | Object | Attribute | Relation | Event | Object | Attribute | Relation | Event
POPE Li et al. (2023e) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
NOPE Lovenia et al. (2023) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
CIEM Hu et al. (2023) | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
M-HalDetect Gunjal et al. (2023) | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | ✓ | ✓ | ✓ | $\times$
GAVIE Liu et al. (2023a) | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | ✓ | ✓ | $\times$ | $\times$
FAITHScore Jing et al. (2023) | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | ✓ | ✓ | ✓ | $\times$
HaELM Wang et al. (2023b) | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | - | - | - | - | $\times$
MMHal-Bench Sun et al. (2023b) | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | - | - | - | - | $\times$
AMBER Wang et al. (2023a) | ✓ | ✓ | ✓ | ✓ | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$
Hal-Eval | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 12:  Comparison of Hallucination Evaluation Benchmarks for LVLMs.

###  C.3 Effectiveness of Hall-evaluator for Mitigating Generative
Hallucination.

The evaluation model Hal-Evaluator, utilized for generative hallucination
assessment, can not only detect and evaluate model hallucinations but also
modify the hallucinatory content within model outputs to aid in hallucination
elimination. To validate the effectiveness of Hal-Evaluator in eliminating
hallucinations, we conducted the following experiment: we used the output from
MiniGPT-4 and LLaVA as the input for Hal-Evaluator, allowing Hal-Evaluator to
detect and rectify any hallucinatory content. Subsequently, we re-evaluated
the corrected output with MMHal-Bench Sun et al. (2023b) as shown in Table 7.
Our findings indicate that Hal-Evaluator can effectively eliminate model
hallucinations.

###  C.4 Compared with Other Hallucination Benchmarks

As illustrated in the Table 12, we have compared current mainstream LVLMs
hallucination evaluation methods, highlighting that our approach ensures the
most extensive coverage of hallucination.

Figure 9: The image shows the prompt we used to elicit hallucination
annotations from GPT-4. Figure 10: The figure illustrates the tailored prompt
designed to guide GPT-4 in distinguishing and discarding labels that are
inconsistent with the predefined criteria of hallucination types or are marked
inaccurately.  Figure 11: We tokenized the annotations of hallucination
content in Hal-Data 130K, and visualized them via a word cloud.

###  C.5 Demo for Hal-Evaluator

As shown in Figure 12, through this visualization, we aim to provide a more
intuitive understanding of the model’s behavior and its ability to
appropriately respond to different types of hallucinations.

Figure 12: This figure presents the responses of the Hal-Evaluator when
confronted with various descriptions of hallucinations.

##  Appendix D Annotation Portal

The annotators are provided with an image, LM-generated detailed description
of the image. For each description, the annotators mark parts of the sentence
into appropriate cateogies: object, attribute, relation, event hallucination.
We provide the explanation and examples for those annotators with Figure 12
and Figure 9

###  D.1 Annotation Agreement

The annotators involved in this research are internal members of our
organization, with a good understanding of natural language processing and
multimodal field studies. They have all consented to the use of their
annotation information for this research and for it to be made public.