# Mitigating Object Hallucinations in Large Vision-Language Models through
Visual Contrastive Decoding

Sicong Leng1,2, Hang Zhang1,3,11footnotemark: 1 Guanzheng Chen1,3 Xin Li1,3,
Shijian Lu2 Chunyan Miao2 Lidong Bing1,3
1DAMO Academy, Alibaba Group 2Nanyang Technological University
3Hupan Lab, 310023, Hangzhou, China

https://github.com/DAMO-NLP-SG/VCD Equal contribution. Sicong Leng is under
the joint PhD program between Alibaba and NTU.Correspondence:
<EMAIL_ADDRESS>
###### Abstract

Large Vision-Language Models (LVLMs) have advanced considerably, intertwining
visual recognition and language understanding to generate content that is not
only coherent but also contextually attuned. Despite their success, LVLMs
still suffer from the issue of object hallucinations, where models generate
plausible yet incorrect outputs that include objects that do not exist in the
images. To mitigate this issue, we introduce Visual Contrastive Decoding
(VCD), a simple and training-free method that contrasts output distributions
derived from original and distorted visual inputs. The proposed VCD
effectively reduces the over-reliance on statistical bias and unimodal priors,
two essential causes of object hallucinations. This adjustment ensures the
generated content is closely grounded to visual inputs, resulting in
contextually accurate outputs. Our experiments show that VCD, without either
additional training or the usage of external tools, significantly mitigates
the object hallucination issue across different LVLM families. Beyond
mitigating object hallucinations, VCD also excels in general LVLM benchmarks,
highlighting its wide-ranging applicability. Codes will be released.

##  1 Introduction

Large Vision-Language Models (LVLMs) have become integral in the intersection
of computer vision and natural language processing, enabling a range of
applications due to their ability to generate contextually relevant textual
descriptions from visual inputs. These models are characterized by their
effectiveness in capturing and translating complex visual patterns into
coherent linguistic representations [45, 78, 70, 33, 12, 18, 49, 73, 5]. The
evolution of LVLMs is marked by ongoing improvements in model architecture,
training methodologies, and data diversity, leading to enhanced performance
and application versatility. Despite these advancements, specific challenges
persist, with the issue of object hallucination [38, 20, 43, 48] being a
prominent concern that impacts the reliability and applicability of LVLMs
across domains.

Figure 1: An illustration of Visual Contrastive Decoding. The hallucinated
object “Surfboards” is highlighted in red, and it is eliminated during the
generative process by contrasting with the output distribution that favors
hallucinations.

Object Hallucination in this context refers to the phenomenon where LVLMs
generate textual content that is semantically coherent but inconsistent with
ground-truth objects in the given image. This challenge not only reveals
fundamental issues of LVLMs, such as over-reliance on statistical bias [1, 2,
19, 38] and unimodal priors [69, 75, 22, 67, 21, 51], but also has direct
implications for the practical deployment of LVLMs. In applications where
precision and reliability of generated content are paramount, object
hallucinations can lead to misinformation, misinterpretation, and subsequent
erroneous decision-making. In domains like healthcare [65, 26], autonomous
systems [8, 68], and robotics [50, 46], such inaccuracies are not just
undesirable but could have significant consequences. Addressing the
hallucination issue is therefore essential to enhance the integrity,
reliability, and broad applicability of LVLMs in various real-world scenarios.

Various approaches have been explored to curb object hallucinations in VLMs.
Early works made attempts on small-scale VLMs by either performing fine-
grained modality alignment [6] or reducing the statistical bias of object co-
occurrence with data augmentation [54, 30]. However, the behaviors of LVLMs
differ significantly from small-scale VLMs, making related methods impractical
to generalize and scale up [29, 66]. Several recent studies address this issue
by proposing hallucination-targeted datasets for fine-tuning [42, 20],
training a post-hoc revisor to reconstruct less hallucinatory outputs [77] or
adapting factually augmented Reinforcement Learning from Human Feedback (RLHF)
[59]. While existing interventions for object hallucination in LVLMs have
shown effectiveness, the incurred human effort and computational cost
highlight a pressing need for a simpler but efficient approach.

In this work, we analyze the effect of visual uncertainty on the two primary
causes of object hallucinations in LVLMs, namely statistical bias and unimodal
priors (i.e., language priors). Building on the analysis above, we introduce
Visual Contrastive Decoding (VCD), a training-free technique designed to
mitigate object hallucination in LVLMs. As shown in Figure 1, VCD is grounded
in the principle of contrasting output distributions from original and
distorted visual inputs. Hence, it acts as a corrective mechanism and
calibrates the model’s over-reliance on language priors from integrated LLMs
and statistical bias of LVLMs’ pretraining corpus. In the realm of efficiency,
VCD stands out due to its minimal computational overhead compared with
previous studies [42, 20, 77, 59], circumventing the need for additional
training or the usage of external tools (e.g., other pretrained models). Our
experiments demonstrate VCD’s effectiveness, with consistent improvements on
multiple object hallucination benchmarks (e.g., up to $+7.4$ F1 score boost on
POPE [38] and $+18\%$ improvement on MME [16]) across different LVLM families,
including LLAVA-1.5 [45, 44], InstructBLIP [12], and Qwen-VL [5]. In addition,
our method is also beneficial to the general perception capacities of LVLMs as
evidenced by benchmarking on MME and LLaVA-
Bench111https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild,
indicating its potential applicability beyond the scope of object
hallucination mitigation.

To sum up, our main contributions are as follows:

  1. 1.

We conduct an in-depth analysis of the effect of visual uncertainty on object
hallucinations in LVLMs, particularly from the aspects of statistical bias and
unimodal priors.

  2. 2.

Inspired by the analysis above, we design VCD, a training-free technique that
can effectively mitigate object hallucinations in LVLMs. It calibrates the
model’s outputs by contrasting output distributions derived from original and
distorted visual inputs, ensuring more consistent content generation.

  3. 3.

Through comprehensive experiments, we demonstrate the efficacy of the proposed
VCD in alleviating object hallucination and enhancing general perception
capability. Our method yields notable improvements without the need for
additional training or external tools.

##  2 Related Work

###  2.1 Visual-Language Models

The development of Vision-Language Models (VLMs) has transitioned from being
rooted in BERT-based language decoders [13, 47, 31] for merging visual and
textual data [36, 58, 63, 34], to a notable advancement ushered by the
integration of Large Language Models (LLMs) [17, 62, 61, 53, 7, 11, 60, 10,
4]. The advent of LLMs heralded the emergence of Large Vision-Language Models
(LVLMs) [3, 9, 14, 34], characterized by enhanced capabilities and
performance. In this phase, LVLMs, supported by end-to-end training
techniques, demonstrated unified decoding of visual and textual tokens,
marking a significant enhancement in their performances and adaptability.
Recent developments have seen a focus on Visual Instruction Fine-tuning [45],
showcasing adaptability to a variety of vision-language tasks. The
methodologies adopted, ranging from integrating cross-modal alignment networks
to fine-tuning LLaMA models, underscore a trend of diversification and
specificity in the approach [12, 33, 70, 5].

###  2.2 Hallucination in VLMs

Prior to the advent of LLMs, the NLP community has primarily defined
“hallucination” as the generation of nonsensical content or content that
deviates from its sources [32, 76, 39, 28, 74, 57]. In the realm of VLMs,
“object hallucination” is also well-documented, referring to models producing
plausible outputs that include objects that do not match or are missing from
images [54, 6, 38]. Mitigating object hallucination in VLMs has typically
involved strategies such as fine-grained contrastive learning [72], ROI
feature fusion [6], and the curtailment of co-occurrence patterns via data
augmentation [30]. However, with the distinct training paradigms and model
architectures that characterize traditional VLMs and contemporary LVLMs,
adapting these strategies to the newer auto-regressive approaches in LVLMs
poses significant challenges [29, 66].

Recent efforts have sought to navigate these complexities, with studies
delving into the evaluation and detection of object hallucinations within the
domain of LVLMs [64, 42, 38, 48]. For example, POPE [38] converts the
hallucination into a binary classification problem to probe the model’s
awareness of whether a specific object exists in the image. Concurrently,
there has been a notable push towards the development of refined datasets
tailored for fine-tuning existing LVLMs [20, 35, 42], training a post-hoc
revisor to detect and reconstruct less hallucinatory outputs [77], and
adapting factually augmented RLHF [59]. Nevertheless, existing approaches that
acquire additional datasets, conduct fine-grained tuning on original or newly
introduced models, or utilize other off-the-shell pretrained models can be
time-consuming, labor-intensive, and computationally costly. Instead, we
propose a conceptually different and training-free approach, VCD, that
contrasts the output distributions with original and distorted visual inputs
to calibrate the model’s over-reliance on unimodal priors and statistical
bias, without utilizing external models.

##  3 Method

###  3.1 Decoding of Vision-Language Models

We consider an LVLM parametrized by $\theta$. The model takes as input a
textual query ${x}$ and a visual input ${v}$, where ${v}$ provides contextual
visual information to assist the model in generating a relevant response ${y}$
to the textual query. The response ${y}$ is sampled auto-regressively from the
probability distribution conditioned on the query ${x}$ and the visual context
${v}$. Mathematically, this can be formulated as:

$\displaystyle y_{t}$ $\displaystyle\sim
p_{\theta}\left(y_{t}\mid{v},{x},{y}_{<t}\right),$ (1)
$\displaystyle\propto\exp\operatorname{logit}_{\theta}\left(y_{t}\mid{v},{x},{y}_{<t}\right),$

where $y_{t}$ denotes the token at time step $t$, and ${y}_{<t}$ represents
the sequence of generated tokens up to the time step ($t-1$). In the decoding
phase of LVLMs, object hallucinations often emerge when probabilities are
erroneously allocated to tokens that do not align with the presented visual
input $v$. Previous studies have identified two primary causes of this
problem: (1) statistical biases inherent in training data (e.g., prevalent but
superficial object correlations) [1, 2, 19], and (2) over-reliance on language
priors embedded within the powerful LLMs used as decoders [38, 69, 75, 22].
Our approach to mitigate object hallucinations first amplifies these
undesirable behaviors with vague inputs and subsequently contrasts with them
in the decoding process.

Figure 2: An illustration of visual uncertainty amplifying language priors.
Given an image featuring a black banana among other colorful fruits, LVLMs
favor more conventional banana colors—such as ”yellow” and ”green”, with
increasing visual uncertainty. The ground-truth color ”black” diminishes in
probability ($logp(y|x,v^{\prime})$) as the distortion escalates, making LVLMs
over-reliant on the language priors from LLM pre-training that typically
associate bananas with being yellow or green.

###  3.2 Visual Uncertainty Amplifies Hallucinations

The fidelity of visual input is pivotal for LVLMs to accurately encode visual
features and generate outputs faithfully. Yet, the introduction of uncertainty
in visual inputs can tilt the equilibrium. This section delves into a
comprehensive analysis aiming to validate the assumption that increased visual
uncertainty can amplify the language priors and statistical biases in LVLMs,
thus exacerbating object hallucination.

Introduction of Visual Uncertainty In this paper, we propose to adopt the most
elementary method—applying a Gaussian noise mask to the original image—to
introduce visual uncertainty. This method, although straightforward, provides
an initial benchmark to estimate the baseline effects of visual uncertainty on
model outputs. Following the forward diffusion process in image generation
[24], the distorted image is modeled as follows:

$\displaystyle
q\left({v}_{t}\mid{v}_{t-1}\right)=\mathcal{N}\left({v}_{t};\sqrt{1-\gamma}{v}_{t-1},\gamma\mathbf{I}\right)$
(2) $\displaystyle
q\left({v}_{T}\mid{v}_{0}\right)=\prod_{t=1}^{T}q\left({v}_{t}\mid{v}_{t-1}\right),$

where $v_{0}$ denotes the original visual input (i.e., original image) and
$\mathbf{I}$ refers to an identity matrix. We incrementally add a small amount
of Gaussian noise for $T$ steps, producing a sequence of distorted images
$v_{1},\dots,v_{T}$. The original image $v_{0}$ gradually loses its
distinguishable features as step $t$ goes larger, where the amount of noise
added in each step is controlled by $\gamma$. Eventually, when
$T\rightarrow\infty$, visual uncertainty reaches the maximum and $v_{T}$ will
become indistinguishable from Gaussian noise.

Visual Uncertainty Amplifies Language Priors Figure 2 shows that visual
uncertainty can compel LVLMs to overlook visual evidence and overly exploit
language priors for decision-making. However, this tendency is not entirely
unexpected, as LLMs are designed to predict next-word probabilities based on
vast textual corpora. When confronted with ambiguous visual stimuli, an LVLM
might misinterpret these conventional, text-based predictions as a “safety
net”. These priors, while generally useful, can introduce biases or
assumptions that are inconsistent with the actual visual content, particularly
when the visual input lacks clarity.

Visual Uncertainty Amplifies Statistical Bias The construction of most vision-
language pretraining datasets is predominantly based on MSCOCO [40], which
inherently suffers from an unbalanced object distribution and biased object
correlations. Previous works [38, 77] point out that LVLMs, trained on such
data, may inherit those statistical biases to generate descriptions with
hallucinated objects. To further examine the hypothesis that visual
uncertainty may amplify statistical biases from pretraining, we designed two
targeted experiments to verify (1) if LVLMs hallucinate frequent objects more
with distorted visual inputs and (2) if LVLMs are more prone to hallucinate
objects that frequently co-occur with ground-truth objects in the image with
distorted visual inputs. Figure 3 shows an evident tendency that LVLMs are
more prone to hallucinate frequent and co-occurring objects, attributing to
the imbalanced object distributions and spurious object correlations inherited
from the training data.

###  3.3 Visual Contrastive Decoding

####  3.3.1 Contrasting the Predictions

Our observations in the previous section reveal that visual uncertainty not
only amplifies reliance on language priors but also makes LVLMs more likely to
be biased by superficial object correlations present in pretraining datasets,
leading to more severe hallucinations. In light of this, we introduce Visual
Contrastive Decoding (VCD). VCD is formulated to counteract the statistical
biases and language priors in LVLMs by contrasting model outputs generated
from original and distorted visual inputs. This is achieved without
necessitating additional training or external pretrained models, making VCD a
cost-effective and efficient solution.

Specifically, given a textual query ${x}$ and a visual input ${v}$, the model
generates two distinct output distributions: one conditioned on the original
${v}$ and the other on the distorted visual input ${v^{\prime}}$, which is
derived by applying pre-defined distortions (i.e., Gaussian noise mask) to
${v}$. Then, a new contrastive probability distribution is computed by
exploiting the differences between the two initially obtained distributions.
The new contrastive distribution $p_{vcd}$ is formulated as:

$\begin{gathered}p_{vcd}\left(y\mid
v,v^{\prime},x\right)=\operatorname{softmax}\left[(1+\alpha)\operatorname{logit}_{\theta}\left(y\mid
v,x\right)\right.\\\ \left.-\alpha\operatorname{logit}_{\theta}\left(y\mid
v^{\prime},x\right)\right],\end{gathered}$ (3)

where larger $\alpha$ values indicate a stronger amplification of differences
between the two distributions ($\alpha=0$ reduces to regular decoding). From
the adjusted output distribution $p_{vcd}$, we can apply various sampling
strategies, such as nucleus sampling [25] and beam search [15].

Essentially, VCD serves as a corrective mechanism, reducing hallucinations by
contrasting against a distribution predisposed to favoring them.
Alternatively, VCD can also be interpreted as a form of contrastive ensemble
that differentiates between the logits of $p_{\theta}\left(y\mid v,x\right)$
and $p_{\theta}\left(y\mid v^{\prime},x\right)$. This method echoes the
contrastive objective commonly employed in image generation. For instance,
classifier-free diffusion models [23] estimate diffusion noise using
$(1+\alpha)\epsilon_{\theta}(x,c)-\alpha\epsilon_{\theta}(x)$, where $c$
serves as a controlling factor. In the realm of text generation, several
studies have also exploited contrastive decoding for more faithful generation
[41, 37, 52, 56].

Figure 3:  The left subfigure shows the correlation between frequent objects
in MSCOCO and their propensity to be hallucinated in the validation set.
Objects with a higher occurrence rate in the dataset are more likely to be
hallucinated by LVLMs under distorted visual scenarios. The right subfigure
charts three objects that often appear alongside ”dining table”, where they
are also more frequently hallucinated when presented with distorted visual
inputs.

####  3.3.2 Adaptive Plausibility Constraints

According to the formation of the contrastive distribution $p_{vcd}$ in
Equation 3, a challenge may arise as it penalizes the model’s entire output
behaviors influenced by distorted visual inputs. However, this is not
universally correct – the output distributions with distorted visual inputs
can still uphold fundamental linguistic standards and common sense reasoning.
Indiscriminate penalization could inaccurately punish these valid outputs and
promote the generation of implausible tokens. To address this issue, we follow
Li et al. [37] to implement an adaptive plausibility constraint that is
contingent upon the confidence level associated with the output distribution
with original visual inputs:

$\begin{gathered}\begin{aligned} &\mathcal{V}_{\text{head
}}\left(y_{<t}\right)=\\{y_{t}\in\mathcal{V}:\\\ &p_{\theta}\left(y_{t}\mid
v,x,y_{<t}\right)\geq\beta\max_{w}p_{\theta}\left(w\mid
v,x,y_{<t}\right)\\},\end{aligned}\\\ \begin{aligned} p_{vcd}\left(y_{t}\mid
v,v^{\prime},x\right)=0,\text{ if }y_{t}\notin\mathcal{V}_{\text{head
}}\left(y_{<t}\right),\end{aligned}\end{gathered}$ (4)

where $\mathcal{V}$ is the output vocabulary of LVLMs and $\beta$ is a
hyperparameter in $[0,1]$ for controlling the truncation of the next token
distribution. Larger $\beta$ indicates more aggressive truncation, keeping
only high-probability tokens.

Combining the visual contrastive decoding and the adaptive plausibility
constraint, we obtain the full formulation:

$\begin{gathered}y_{t}\sim\operatorname{softmax}\left[(1+\alpha)\operatorname{logit}_{\theta}\left(y_{t}\mid
v,x,y_{<t}\right)\right.\\\
\left.-\alpha\operatorname{logit}_{\theta}\left(y_{t}\mid
v^{\prime},x,y_{<t}\right)\right],\\\ {subject\ to}\
y_{t}\in\mathcal{V}_{\text{head }}\left(y_{<t}\right)\end{gathered}$ (5)

Incorporating adaptive plausibility constraints refines the contrastive
distribution, bolstering confidence in straightforward decisions. This ensures
that when the model is highly confident in its outputs associated with the
original inputs, the candidate pool is streamlined, often retaining a singular
token with high probability. Such an approach effectively neutralizes
potential adverse effects of VCD, preventing it from inadvertently promoting
the generation of implausible tokens and maintaining the integrity of the
generated content.

##  4 Experiments

Dataset | Setting | Model | Decoding | Accuracy$\uparrow$ | Precision | Recall | F1 Score$\uparrow$
---|---|---|---|---|---|---|---
MSCOCO | Random | LLaVA1.5 | Regular | $83.29_{(\pm 0.35)}$ | $92.13_{(\pm 0.54)}$ | $72.80_{(\pm 0.57)}$ | $81.33_{(\pm 0.41)}$
VCD | $\textbf{87.73}_{(\pm 0.40)}$ | $91.42_{(\pm 0.55)}$ | $83.28_{(\pm 0.42)}$ | $\textbf{87.16}_{(\pm 0.41)}$
Qwen-VL | Regular | $84.73_{(\pm 0.36)}$ | $95.61_{(\pm 0.45)}$ | $72.81_{(\pm 0.38)}$ | $82.67_{(\pm 0.41)}$
VCD | $\textbf{88.63}_{(\pm 0.10)}$ | $94.64_{(\pm 0.25)}$ | $81.91_{(\pm 0.19)}$ | $\textbf{87.81}_{(\pm 0.11)}$
InstructBLIP | Regular | $80.71_{(\pm 0.73)}$ | $81.67_{(\pm 0.67)}$ | $79.19_{(\pm 1.14)}$ | $80.41_{(\pm 0.80)}$
VCD | $\textbf{84.53}_{(\pm 0.38)}$ | $88.55_{(\pm 0.54)}$ | $79.32_{(\pm 0.44)}$ | $\textbf{83.68}_{(\pm 0.40)}$
Popular | LLaVA1.5 | Regular | $81.88_{(\pm 0.48)}$ | $88.93_{(\pm 0.60)}$ | $72.80_{(\pm 0.57)}$ | $80.06_{(\pm 0.05)}$
VCD | $\textbf{85.38}_{(\pm 0.38)}$ | $86.92_{(\pm 0.53)}$ | $83.28_{(\pm 0.42)}$ | $\textbf{85.06}_{(\pm 0.37)}$
Qwen-VL | Regular | $84.13_{(\pm 0.18)}$ | $94.31_{(\pm 0.43)}$ | $72.64_{(\pm 0.45)}$ | $82.06_{(\pm 0.23)}$
VCD | $\textbf{87.12}_{(\pm 0.07)}$ | $91.49_{(\pm 0.10)}$ | $81.85_{(\pm 0.19)}$ | $\textbf{86.40}_{(\pm 0.09)}$
InstructBLIP | Regular | $78.22_{(\pm 0.84)}$ | $77.87_{(\pm 1.03)}$ | $78.85_{(\pm 0.52)}$ | $78.36_{(\pm 0.76)}$
VCD | $\textbf{81.47}_{(\pm 0.42)}$ | $82.89_{(\pm 0.64)}$ | $79.32_{(\pm 0.44)}$ | $\textbf{81.07}_{(\pm 0.39)}$
Adversarial | LLaVA1.5 | Regular | $78.96_{(\pm 0.52)}$ | $83.06_{(\pm 0.58)}$ | $72.75_{(\pm 0.59)}$ | $77.57_{(\pm 0.57)}$
VCD | $\textbf{80.88}_{(\pm 0.33)}$ | $79.45_{(\pm 0.29)}$ | $83.29_{(\pm 0.43)}$ | $\textbf{81.33}_{(\pm 0.34)}$
Qwen-VL | Regular | $82.26_{(\pm 0.30)}$ | $89.97_{(\pm 0.33)}$ | $72.61_{(\pm 0.50)}$ | $80.37_{(\pm 0.37)}$
VCD | $\textbf{84.26}_{(\pm 0.39)}$ | $85.84_{(\pm 0.45)}$ | $82.05_{(\pm 0.39)}$ | $\textbf{83.90}_{(\pm 0.39)}$
InstructBLIP | Regular | $75.84_{(\pm 0.45)}$ | $74.30_{(\pm 0.63)}$ | $79.03_{(\pm 0.68)}$ | $76.59_{(\pm 0.40)}$
VCD | $\textbf{79.56}_{(\pm 0.41)}$ | $79.67_{(\pm 0.59)}$ | $79.39_{(\pm 0.50)}$ | $\textbf{79.52}_{(\pm 0.38)}$
A-OKVQA | Random | LLaVA1.5 | Regular | $83.45_{(\pm 0.48)}$ | $87.24_{(\pm 0.68)}$ | $78.36_{(\pm 0.54)}$ | $82.56_{(\pm 0.50)}$
VCD | $\textbf{86.15}_{(\pm 0.23)}$ | $85.18_{(\pm 0.34)}$ | $87.53_{(\pm 0.14)}$ | $\textbf{86.34}_{(\pm 0.21)}$
Qwen-VL | Regular | $86.67_{(\pm 0.48)}$ | $93.16_{(\pm 0.55)}$ | $79.16_{(\pm 0.59)}$ | $85.59_{(\pm 0.53)}$
VCD | $\textbf{89.22}_{(\pm 0.14)}$ | $90.77_{(\pm 0.04)}$ | $87.32_{(\pm 0.34)}$ | $\textbf{89.01}_{(\pm 0.16)}$
InstructBLIP | Regular | $80.91_{(\pm 0.34)}$ | $77.97_{(\pm 0.59)}$ | $86.16_{(\pm 0.88)}$ | $81.86_{(\pm 0.32)}$
VCD | $\textbf{84.11}_{(\pm 0.27)}$ | $82.21_{(\pm 0.35)}$ | $87.05_{(\pm 0.53)}$ | $\textbf{84.56}_{(\pm 0.28)}$
Popular | LLaVA1.5 | Regular | $79.90_{(\pm 0.33)}$ | $80.85_{(\pm 0.31)}$ | $78.36_{(\pm 0.54)}$ | $79.59_{(\pm 0.37)}$
VCD | $\textbf{81.85}_{(\pm 0.44)}$ | $78.60_{(\pm 0.58)}$ | $87.53_{(\pm 0.14)}$ | $\textbf{82.82}_{(\pm 0.36)}$
Qwen-VL | Regular | $85.56_{(\pm 0.35)}$ | $90.44_{(\pm 0.56)}$ | $79.53_{(\pm 0.84)}$ | $84.63_{(\pm 0.42)}$
VCD | $\textbf{87.85}_{(\pm 0.30)}$ | $88.10_{(\pm 0.36)}$ | $87.53_{(\pm 0.47)}$ | $\textbf{87.81}_{(\pm 0.31)}$
InstructBLIP | Regular | $76.19_{(\pm 0.80)}$ | $72.16_{(\pm 0.69)}$ | $85.28_{(\pm 0.79)}$ | $78.17_{(\pm 0.73)}$
VCD | $\textbf{79.78}_{(\pm 0.47)}$ | $76.00_{(\pm 0.52)}$ | $87.05_{(\pm 0.53)}$ | $\textbf{81.15}_{(\pm 0.42)}$
Adversarial | LLaVA1.5 | Regular | $74.04_{(\pm 0.34)}$ | $72.08_{(\pm 0.53)}$ | $78.49_{(\pm 0.38)}$ | $75.15_{(\pm 0.23)}$
VCD | $\textbf{74.97}_{(\pm 0.39)}$ | $70.01_{(\pm 0.40)}$ | $87.36_{(\pm 0.15)}$ | $\textbf{77.73}_{(\pm 0.29)}$
Qwen-VL | Regular | $79.57_{(\pm 0.31)}$ | $79.77_{(\pm 0.34)}$ | $79.23_{(\pm 0.73)}$ | $79.50_{(\pm 0.38)}$
VCD | $\textbf{81.27}_{(\pm 0.09)}$ | $77.79_{(\pm 0.20)}$ | $87.53_{(\pm 0.34)}$ | $\textbf{82.38}_{(\pm 0.10)}$
InstructBLIP | Regular | $70.71_{(\pm 0.76)}$ | $65.91_{(\pm 0.74)}$ | $85.83_{(\pm 0.80)}$ | $75.56_{(\pm 0.57)}$
VCD | $\textbf{74.33}_{(\pm 0.67)}$ | $69.46_{(\pm 0.73)}$ | $86.87_{(\pm 0.27)}$ | $\textbf{77.19}_{(\pm 0.47)}$
GQA | Random | LLaVA1.5 | Regular | $83.73_{(\pm 0.27)}$ | $87.16_{(\pm 0.39)}$ | $79.12_{(\pm 0.35)}$ | $82.95_{(\pm 0.28)}$
VCD | $\textbf{86.65}_{(\pm 0.45)}$ | $84.85_{(\pm 0.59)}$ | $89.24_{(\pm 0.34)}$ | $\textbf{86.99}_{(\pm 0.41)}$
Qwen-VL | Regular | $80.97_{(\pm 0.32)}$ | $88.07_{(\pm 0.34)}$ | $71.64_{(\pm 0.57)}$ | $79.01_{(\pm 0.40)}$
VCD | $\textbf{85.59}_{(\pm 0.38)}$ | $86.88_{(\pm 0.44)}$ | $83.84_{(\pm 0.36)}$ | $\textbf{85.33}_{(\pm 0.38)}$
InstructBLIP | Regular | $79.65_{(\pm 0.24)}$ | $77.14_{(\pm 0.43)}$ | $84.29_{(\pm 0.36)}$ | $80.56_{(\pm 0.18)}$
VCD | $\textbf{83.69}_{(\pm 0.11)}$ | $81.84_{(\pm 0.42)}$ | $86.61_{(\pm 0.48)}$ | $\textbf{84.16}_{(\pm 0.01)}$
Popular | LLaVA1.5 | Regular | $78.17_{(\pm 0.17)}$ | $77.64_{(\pm 0.26)}$ | $79.12_{(\pm 0.35)}$ | $78.37_{(\pm 0.18)}$
VCD | $\textbf{80.73}_{(\pm 0.47)}$ | $76.26_{(\pm 0.68)}$ | $89.24_{(\pm 0.34)}$ | $\textbf{82.24}_{(\pm 0.35)}$
Qwen-VL | Regular | $75.99_{(\pm 0.33)}$ | $78.62_{(\pm 0.41)}$ | $71.40_{(\pm 0.38)}$ | $74.84_{(\pm 0.34)}$
VCD | $\textbf{81.83}_{(\pm 0.27)}$ | $80.45_{(\pm 0.47)}$ | $84.09_{(\pm 0.32)}$ | $\textbf{82.23}_{(\pm 0.22)}$
InstructBLIP | Regular | $73.87_{(\pm 0.58)}$ | $69.63_{(\pm 0.54)}$ | $84.69_{(\pm 0.68)}$ | $76.42_{(\pm 0.52)}$
VCD | $\textbf{78.57}_{(\pm 0.14)}$ | $74.62_{(\pm 0.22)}$ | $86.61_{(\pm 0.48)}$ | $\textbf{80.17}_{(\pm 0.16)}$
Adversarial | LLaVA1.5 | Regular | $75.08_{(\pm 0.33)}$ | $73.19_{(\pm 0.49)}$ | $79.16_{(\pm 0.35)}$ | $76.06_{(\pm 0.24)}$
VCD | $\textbf{76.09}_{(\pm 0.43)}$ | $70.83_{(\pm 0.45)}$ | $88.75_{(\pm 0.56)}$ | $\textbf{78.78}_{(\pm 0.36)}$
Qwen-VL | Regular | $75.46_{(\pm 0.63)}$ | $77.92_{(\pm 0.73)}$ | $71.07_{(\pm 0.97)}$ | $74.33_{(\pm 0.71)}$
VCD | $\textbf{80.01}_{(\pm 0.27)}$ | $77.86_{(\pm 0.24)}$ | $83.85_{(\pm 0.35)}$ | $\textbf{80.75}_{(\pm 0.27)}$
InstructBLIP | Regular | $70.56_{(\pm 0.53)}$ | $66.12_{(\pm 0.32)}$ | $84.33_{(\pm 1.05)}$ | $74.12_{(\pm 0.58)}$
VCD | $\textbf{75.08}_{(\pm 0.13)}$ | $70.59_{(\pm 0.16)}$ | $85.99_{(\pm 0.10)}$ | $\textbf{77.53}_{(\pm 0.08)}$

Table 1: Results on POPE. Regular decoding denotes direct sampling, whereas
VCD refers to sampling from our proposed contrastive distribution $p_{vcd}$.
The best performances within each setting are bolded.

Model | Decoding | Object-level | Attribute-level | Total Scores$\uparrow$
---|---|---|---|---
Existence$\uparrow$ | Count$\uparrow$ | Position$\uparrow$ | Color$\uparrow$
LLaVA1.5 | Regular | $175.67_{(\pm 7.51)}$ | $124.67_{(\pm 19.59)}$ | $114.00_{(\pm 9.32)}$ | $151.00_{(\pm 10.45)}$ | $565.33_{(\pm 32.92)}$
VCD | $\textbf{184.66}_{(\pm 6.81)}$ | $\textbf{138.33}_{(\pm 15.68)}$ | $\textbf{128.67}_{(\pm 7.21)}$ | $\textbf{153.00}_{(\pm 7.58)}$ | $\textbf{604.66}_{(\pm 18.76)}$
Qwen-VL | Regular | $155.00_{(\pm 3.54)}$ | $127.67_{(\pm 13.36)}$ | $\textbf{131.67}_{(\pm 7.73)}$ | $173.00_{(\pm 9.75)}$ | $587.33_{(\pm 31.06)}$
VCD | $\textbf{156.00}_{(\pm 6.52)}$ | $\textbf{131.00}_{(\pm 6.19)}$ | $128.00_{(\pm 3.61)}$ | $\textbf{181.67}_{(\pm 5.14)}$ | $\textbf{596.67}_{(\pm 11.61)}$
InstructBLIP | Regular | $141.00_{(\pm 13.97)}$ | $75.33_{(\pm 14.16)}$ | $\textbf{66.67}_{(\pm 3.91)}$ | $97.33_{(\pm 16.94)}$ | $380.33_{(\pm 40.20)}$
VCD | $\textbf{168.33}_{(\pm 11.55)}$ | $\textbf{92.33}_{(\pm 8.47)}$ | $64.00_{(\pm 6.73)}$ | $\textbf{123.00}_{(\pm 11.27)}$ | $\textbf{447.67}_{(\pm 13.36)}$

Table 2: Results on the hallucination subset of MME. Regular decoding denotes
direct sampling, whereas VCD refers to sampling from our proposed contrastive
distribution $p_{vcd}$. The best performances within each setting are bolded.

This section details our assessment of the proposed Visual Contrastive
Decoding across various LVLMs.

###  4.1 Experimental Settings

Datasets & Evaluation Metrics

POPE, the Polling-based Object Probing Evaluation [38], presents a streamlined
approach to assess object hallucination. Within this benchmark, LVLMs are
queried to answer if a specific object exists in the given image. The ratio
between queries probing existent objects and non-existent objects is balanced
(i.e.,$50$% vs. $50$%). It encompasses three sampling settings: random,
popular, and adversarial, each distinct in constructing negative samples. In
the random setting, objects absent from the image are chosen randomly. The
popular setting selects missing objects from a high-frequency pool, while in
the adversarial setting, co-occurring objects not present in the image are
prioritized. The POPE benchmark aggregates data from three distinct sources:
MSCOCO [40], A-OKVQA [55], and GQA [27]. It involves $500$ images from each
dataset under each sampling setting and formulates $6$ questions per image,
culminating in a total of $27,000$ query-answer pairs from the development
sets of these datasets222Given the absence of ground-truth object annotations
in A-OKVQA and GQA, SEEM [79] is applied for image segmentation and object
identification. . The evaluation pivots on four key metrics: Accuracy,
Precision, Recall, and the F1 score.

MME [16] serves as an extensive benchmark tailored to assess LVLMs across
multiple dimensions. It comprises ten perception-related subtasks and four
cognition-focused ones. Following Yin et al. [71], except for adapting the
whole dataset, we additionally leverage the existence and count subsets for
object-level hallucination evaluation, and the position and color subsets for
attribute-level hallucination assessment. Performance is quantified via the
combined metric of accuracy and accuracy+ as the official
implementation333https://github.com/BradyFU/Awesome-Multimodal-Large-Language-
Models/tree/Evaluation.

LLaVA-Bench444https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-
wild features a collection of $24$ images, accompanying $60$ questions that
span a range of contexts including indoor and outdoor scenes, memes,
paintings, and sketches. This dataset is crafted to assess the capability of
LVLMs in tackling more challenging tasks and their adaptability to new
domains. We conduct case studies on this dataset to qualitatively demonstrate
the effectiveness of our proposed VCD.

LVLM Baselines We evaluate the effectiveness of our VCD on three state-of-the-
art LVLMs. Concretely, we apply our VCD to LLaVA-1.5 and InstructBLIP, which
employ Vicuna 7B as language decoder [44, 12], and Qwen-VL, built on top of
Qwen 7B backbone [5]. For a more convincing comparison, we report the averaged
results as well as the standard deviation over $5$ runs on POPE and MME
benchmarks.

Implementation Details Throughout our experiments, we set $\alpha=1$,
$\beta=0.1$, and $\gamma=0.1$ unless explicitly stated otherwise. For a
consistent comparative analysis, our baseline decoding strategy employs direct
sampling (i.e., denoted as “Regular” in all experimental tables), where the
next token is directly sampled from the post-softmax
distribution555Optimization of $\alpha$, $\beta$, $T$, and applying other
sampling strategies as detailed in the ablation studies in Supplementary
Materials may yield better results. The current settings serve as a constant
baseline to demonstrate the efficacy of our approach.. Conversely, instances
labeled as“VCD” in the decoding column of all experimental tables refer to our
proposed Visual Contrastive Decoding strategy, which also directly samples
from the modified post-softmax distribution after applying VCD. Comprehensive
parameter configurations can be found in Supplementary Materials.

###  4.2 Experimental Results

Figure 4: MME full set results on LLaVA-1.5. VCD leads to consistent
enhancement in LVLMs’ perception capacities while preserving their recognition
competencies.

Results on POPE Experimental results on POPE under the random, popular, and
adversarial settings are summarized in Table 1. A notable observation is the
robust effect of our proposed VCD. Specifically, under different sampling
settings, the performances of our VCD consistently surpass the baseline
results by large margins (up to +5.8 accuracy and +7.4 F1) on all of the
LVLMs. This suggests its pivotal role in counteracting statistical biases and
language priors in LVLMs, thereby reducing instances of object hallucination.
In addition, all LVLMs display a clear performance degradation as we move from
the random setting to popular and experience a further decline while moving to
the adversarial setting. This trend verifies our hypothesis that statistical
biases inherent in LVLMs substantially contribute to the object hallucination
problem. In a more detailed model-specific analysis, VCD demonstrates varied
effects across different LVLMs. For LLaVA-1.5 and Qwen-VL, the F1 score
elevation is predominantly driven by a recall boost (e.g., up to $10$ points),
showcasing its enhanced ability to accurately detect object presences.
Conversely, InstructBLIP’s F1 score improvement is largely due to improved
precision, signifying its enhanced capability to accurately filter out false
positives. This highlights VCD’s ability to accentuate distinct attributes of
various model architectures in binary decision scenarios of POPE.

Figure 5: Illustration of hallucination correction by our proposed VCD with
two samples from LLaVA-Bench. Hallucinated objects from LVLM’s regular
decoding are highlighted in red.

Results on MME Hallucination Subset The MME subset evaluations extend beyond
POPE’s scope, encompassing both object-level and attribute-level
hallucinations. Results in Table 2 show that implementing VCD leads to a
uniform enhancement in addressing object-level hallucinations for all models.
Additionally, VCD demonstrates an overall positive impact on attribute-level
Color scores, contributing to substantial overall performance gains. These
improvements emphasize VCD’s strength in addressing the embedded statistical
bias and language priors of LVLMs, thus bringing a positive impact on a
broader range of hallucination challenges. In contrast, the Position score is
relatively low across four metrics, with minimal uplift from VCD, suggesting
the relatively weak ability of LVLMs in position reasoning.

Results on MME Full Set As shown in Figure 4, we also include the evaluation
of VCD on MME Full Set to assess its impact on the general capability of
LVLMs. With all models exhibiting comparable performance trajectories, we
present the results of LLaVA-1.5 as a representative666Comprehensive results
for all three LVLMs on the MME full set are provided in Supplementary
Materials.. The implementation of VCD leads to a consistent enhancement in
perception-based tasks, while the original recognition competencies of the
LVLMs are preserved. This may be attributed to VCD’s reduction of statistical
bias and language priors, which improves LVLMs’ general perception capacities
by ensuring a visually grounded analysis.

###  4.3 Further Discussions

Figure 6: Performance of LLaVA-1.5 on the POPE benchmark across varying noise
levels with regular decoding. We visualize the distorted visual inputs
subjected to different levels of Gaussian noise at the bottom.

Effect of Visual Uncertainty on Hallucinations We further study how the object
hallucination of LLaVA-1.5 changes along with visual uncertainty. Figure 6
depicts a clear performance drop on the POPE benchmark with the increase of
noise steps, suggesting that the object hallucination will become more severe
as visual uncertainty goes larger. This observation aligns with our previous
findings in Section 3.2 that visual uncertainty will exacerbate object
hallucination issues in LVLMs’ generative process. Our proposed VCD emerges as
a correction mechanism by contrasting model outputs with original and
distorted visual inputs.

GPT-4V Aided Evaluation of Open-Ended Generation Beyond the “Yes-or-No”
question format employed in our POPE and MME evaluations, we extend our
analysis to open-ended captioning tasks in the LLaVA-Bench using the recently
released LVLM, GPT-4V777https://openai.com/research/gpt-4v-system-card,
following Yin et al. [71]888The prompt used for evaluation and an evaluation
case is provided in Supplementary Materials.. Results in Table 3 show
consistent improvements in VCD over regular decoding. The observed enhancement
in accuracy points to VCD’s ability to mitigate hallucinations effectively.
Simultaneously, VCD’s counteraction of statistical biases and language priors
enhances the perceptual capabilities of LVLMs, as evidenced by the marked
improvement in the detailedness of the responses.

Case Study on LLaVA-Bench Figure 5 demonstrates two case studies on how, given
identical prompts and images, regular decoding can yield object hallucinations
influenced by the statistical bias and language priors inherent during
pretraining. For instance, in the displayed examples, objects such as “dining
table” and “fork”, which often co-occur with the likely ground-truth object
“chair”, are hallucinated. In contrast, the implementation of VCD notably
mitigates these hallucination issues and simultaneously preserves the
coherence and informativeness of the output text. Due to the page limit,
please refer to Supplementary Materials for more cases and ablation
studies999Ablation studies in Supplementary Materials include effects of total
noise steps $T$, hyper-parameters $\alpha$, $\beta$, and effect of VCD on
larger LVLM variants and with other sampling strategies..

Model | Decoding | Accuracy$\uparrow$ | Detailedness$\uparrow$
---|---|---|---
LLaVA-1.5 | Regular | $3.23$ | $3.54$
VCD | 4.15 | 3.85
InstructBLIP | Regular | $3.84$ | $4.07$
VCD | 4.23 | 4.69
Qwen-VL | Regular | $4.76$ | $3.46$
VCD | 6.69 | 4.46

Table 3: Results of GPT-4V-aided evaluation on open-ended generation. Accuracy
measures the response’s alignment with the image content, and Detailedness
gauges the richness of details in the response. Both metrics are on a scale of
10.

##  5 Conclusion and Limitation

In this paper, we tackle the object hallucination issue in LVLMs. We conducted
an in-depth analysis of how visual uncertainty influences hallucinations,
particularly from the aspect of statistical biases and language priors. Our
findings indicate that visual uncertainty amplifies these factors,
contributing to more hallucinations. In light of this, we introduced Visual
Contrastive Decoding (VCD), a novel, training-free method that employs
contrastive distributions to calibrate the model’s output without the usage of
external tools. Our extensive experiments across multiple benchmarks and LVLM
families confirm VCD’s efficacy in reducing hallucinations and also
demonstrate its potential to enhance the overall perception capabilities of
LVLMs.

Limitation While this study employs a basic Gaussian noise approach to
introduce visual uncertainty, more fine-grained techniques, like object-level
blurring, hold the potential for improved outcomes. In addition, our focus was
limited to LVLMs processing images and text, not encompassing their emerging
applications in video understanding. Future research directions include
exploring diverse image distortion methods and extending the Visual
Contrastive Decoding (VCD) framework to a broader range of LVLMs.

## References

  * Agarwal et al. [2020] Vedika Agarwal, Rakshith Shetty, and Mario Fritz.  Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9690–9698, 2020.
  * Agrawal et al. [2016] Aishwarya Agrawal, Dhruv Batra, and Devi Parikh.  Analyzing the behavior of visual question answering models.  _arXiv preprint arXiv:1606.07356_ , 2016.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Bai et al. [2023a] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023a.
  * Bai et al. [2023b] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_ , 2023b.
  * Biten et al. [2022] Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas.  Let there be a clock on the beach: Reducing object hallucination in image captioning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1381–1390, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2023] Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton.  Driving with llms: Fusing object-level vector modality for explainable autonomous driving.  _arXiv preprint arXiv:2310.01957_ , 2023.
  * Chen et al. [2022] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_ , 2022.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2306.04387_ , 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Freitag and Al-Onaizan [2017] Markus Freitag and Yaser Al-Onaizan.  Beam search strategies for neural machine translation.  _arXiv preprint arXiv:1702.01806_ , 2017.
  * Fu et al. [2023] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _arXiv preprint arXiv:2306.13394_ , 2023.
  * Gilardi et al. [2023] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _arXiv preprint arXiv:2303.15056_ , 2023.
  * Gong et al. [2023] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.  Multimodal-gpt: A vision and language model for dialogue with humans.  _arXiv preprint arXiv:2305.04790_ , 2023.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6904–6913, 2017.
  * Gunjal et al. [2023] Anisha Gunjal, Jihan Yin, and Erhan Bas.  Detecting and preventing hallucinations in large vision language models.  _arXiv preprint arXiv:2308.06394_ , 2023.
  * Gupta et al. [2022] Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille.  Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5078–5088, 2022.
  * Han et al. [2022] Yudong Han, Liqiang Nie, Jianhua Yin, Jianlong Wu, and Yan Yan.  Visual perturbation-aware collaborative learning for overcoming the language prior problem.  _arXiv preprint arXiv:2207.11850_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Holtzman et al. [2019] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_ , 2019.
  * Hu et al. [2023] Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang.  Advancing medical imaging with language models: A journey from n-grams to chatgpt.  _arXiv preprint arXiv:2304.04920_ , 2023.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709, 2019.
  * Ji et al. [2023] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38, 2023.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kim et al. [2023] Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata.  Exposing and mitigating spurious correlations for cross-modal retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2584–2594, 2023.
  * Koroteev [2021] MV Koroteev.  Bert: a review of applications in natural language processing and understanding.  _arXiv preprint arXiv:2103.11943_ , 2021.
  * Lee et al. [2018] Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo.  Hallucinations in neural machine translation.  _OpenReview_ , 2018.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_ , 2023a.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022a.
  * Li et al. [2023b] Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al.  A large-scale dataset towards multi-modal multilingual instruction tuning.  _arXiv preprint arXiv:2306.04387_ , 2023b.
  * Li et al. [2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_ , 2019.
  * Li et al. [2022b] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.  Contrastive decoding: Open-ended text generation as optimization.  _arXiv preprint arXiv:2210.15097_ , 2022b.
  * Li et al. [2023c] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  _arXiv preprint arXiv:2305.10355_ , 2023c.
  * Lin et al. [2021] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_ , 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2021] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi.  Dexperts: Decoding-time controlled text generation with experts and anti-experts.  _arXiv preprint arXiv:2105.03023_ , 2021.
  * Liu et al. [2023a] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Aligning large multi-modal model with robust instruction tuning.  _arXiv preprint arXiv:2306.14565_ , 2023a.
  * Liu et al. [2023b] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  _arXiv preprint arXiv:2306.14565_ , 2023b.
  * Liu et al. [2023c] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  _arXiv preprint arXiv:2310.03744_ , 2023c.
  * Liu et al. [2023d] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023d.
  * Liu et al. [2023e] Haokun Liu, Yaonan Zhu, Kenji Kato, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa Hasegawa.  Llm-based human-robot collaboration framework for manipulation tasks.  _arXiv preprint arXiv:2308.14972_ , 2023e.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Lovenia et al. [2023] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung.  Negative object presence evaluation (nope) to measure object hallucination in vision-language models.  _arXiv preprint arXiv:2310.05338_ , 2023.
  * Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models.  _arXiv preprint arXiv:2306.05424_ , 2023.
  * Mai et al. [2023] Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard Ghanem.  Llm as a robotic brain: Unifying egocentric memory and control.  _arXiv preprint arXiv:2304.09349_ , 2023.
  * Niu et al. [2021] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen.  Counterfactual vqa: A cause-effect look at language bias.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12700–12710, 2021.
  * O’Brien and Lewis [2023] Sean O’Brien and Mike Lewis.  Contrastive decoding improves reasoning in large language models.  _arXiv preprint arXiv:2309.09117_ , 2023.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Rohrbach et al. [2018] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  _arXiv preprint arXiv:1809.02156_ , 2018.
  * Schwenk et al. [2022] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In _European Conference on Computer Vision_ , pages 146–162. Springer, 2022.
  * Shi et al. [2023a] Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih.  Trusting your evidence: Hallucinate less with context-aware decoding.  _arXiv preprint arXiv:2305.14739_ , 2023a.
  * Shi et al. [2023b] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_ , 2023b.
  * Sun et al. [2019] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7464–7473, 2019.
  * Sun et al. [2023] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.  Aligning large multimodal models with factually augmented rlhf.  _arXiv preprint arXiv:2309.14525_ , 2023.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Tay et al. [2022] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al.  Ul2: Unifying language learning paradigms.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. [2022] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_ , 2022.
  * Wang et al. [2023a] Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al.  Evaluation and analysis of hallucination in large vision-language models.  _arXiv preprint arXiv:2308.15126_ , 2023a.
  * Wang et al. [2023b] Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen.  Chatcad: Interactive computer-aided diagnosis on medical image using large language models.  _arXiv preprint arXiv:2302.07257_ , 2023b.
  * Wei et al. [2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022.
  * Wu et al. [2022] Yike Wu, Yu Zhao, Shiwan Zhao, Ying Zhang, Xiaojie Yuan, Guoqing Zhao, and Ning Jiang.  Overcoming language priors in visual question answering via distinguishing superficially similar instances.  _arXiv preprint arXiv:2209.08529_ , 2022.
  * Wu et al. [2023] Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan.  Embodied task planning with large language models.  _arXiv preprint arXiv:2307.01848_ , 2023.
  * Yan et al. [2023] Hong Yan, Lijun Liu, Xupeng Feng, and Qingsong Huang.  Overcoming language priors with self-contrastive learning for visual question answering.  _Multimedia Tools and Applications_ , 82(11):16343–16358, 2023.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_ , 2023.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen.  Woodpecker: Hallucination correction for multimodal large language models.  _arXiv preprint arXiv:2310.16045_ , 2023.
  * Zeng et al. [2021] Yan Zeng, Xinsong Zhang, and Hang Li.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  _arXiv preprint arXiv:2111.08276_ , 2021.
  * Zhang et al. [2023a] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023a.
  * Zhang et al. [2023b] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_ , 2023b.
  * Zhibo et al. [2023] Ren Zhibo, Wang Huizhen, Zhu Muhua, Wang Yichao, Xiao Tong, and Zhu Jingbo.  Overcoming language priors with counterfactual inference for visual question answering.  In _Proceedings of the 22nd Chinese National Conference on Computational Linguistics_ , pages 600–610, 2023.
  * Zhou et al. [2020] Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad.  Detecting hallucinated content in conditional neural sequence generation.  _arXiv preprint arXiv:2011.02593_ , 2020.
  * Zhou et al. [2023] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao.  Analyzing and mitigating object hallucination in large vision-language models.  _arXiv preprint arXiv:2310.00754_ , 2023.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.
  * Zou et al. [2023] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023.

##  Appendix A Detailed Experimental Settings

In all experimental setups, the hyper-parameters $\gamma$, $\alpha$ and
$\beta$, as specified in Equations 2, 3 and 4, are fixed at values of $0.1$,
$1$ and $0.1$, respectively. For the total number of noise steps $T$
delineated in Equation 2, we set a value of $500$ for experiments involving
the MME and LLaVA-Bench, while for those evaluating on POPE, the $T$ value is
set at $999$.

##  Appendix B Ablation Studies

For the Ablation Studies section, the default configuration for hyper-
parameters $\alpha$, $\beta$, and $\delta$ is set to $1$, $0.1$, and $500$,
respectively. These values are retained across all experiments unless an
individual study specifies an alternative parameter adjustment for
investigation. Across all the experiments, we use LLaVA-1.5 as the
representative LVLM baseline to demonstrate the effect of tuning different
hyper-parameters.

###  B.1 Effect of Total Noise Steps $T$

$T$ |  | Hallucination Subset
---
Total Scores
| Perception Subset
---
Total Scores
| Recognition Subset
---
Total Scores
200 | $586.67_{\pm 11.67}$ | $1311.47_{\pm 4.33}$ | $338.69_{\pm 19.87}$
500 | $591.67_{\pm 36.06}$ | $1340.89_{\pm 55.91}$ | $323.45_{\pm 5.89}$
700 | $578.89_{\pm 19.17}$ | $1339.04_{\pm 40.40}$ | $320.95_{\pm 14.18}$
999 | $557.78_{\pm 1.92}$ | $1345.81_{\pm 36.31}$ | $321.90_{\pm 10.19}$

Table 4: An ablation study of total noise steps $T$ on the MME benchmark.

Figure 4 presents an ablation study examining the impact of varying noise
levels, denoted as $\delta$, using the LLaVA-1.5 model on the MME benchmark.
In alignment with the experimental configuration, MME is subdivided into three
subsets: hallucination, perception, and recognition. The hallucination subset
includes tasks related to Existence, Count, Position, and Color, while the
perception subset encompasses these and additional perception-focused tasks.
The recognition subset, conversely, involves tasks that challenge LVLMs’
cognitive reasoning abilities.

The study reveals a pronounced sensitivity to different $\delta$ values within
the hallucination subset, where optimal noise levels correlate with
substantially enhanced overall scores. In the realm of perception tasks, a
surpassing of a specific noise threshold ($\delta>500$) showcases VCD’s
capability to consistently yield improvements. For recognition tasks, VCD
maintains steady performance across the spectrum of tested noise values.

###  B.2 Effect of $\alpha$ in Visual Contrastive Decoding

$\alpha$ |  | Hallucination Subset
---
Total Scores
| Perception Subset
---
Total Scores
| Recognition Subset
---
Total Scores
0.25 | $583.89_{\pm 19.32}$ | $1322.25_{\pm 32.58}$ | $330.24_{\pm 13.60}$
0.5 | $580.56_{\pm 17.11}$ | $1315.49_{\pm 27.28}$ | $333.45_{\pm 5.77}$
0.75 | $578.33_{\pm 29.49}$ | $1312.93_{\pm 39.31}$ | $330.95_{\pm 13.58}$
1.0 | $591.67_{\pm 36.06}$ | $1340.89_{\pm 55.91}$ | $323.45_{\pm 5.89}$

Table 5: An ablation study of $\alpha$ on the MME benchmark.

Table 5 demonstrates the outcomes of an ablation study focusing on $\alpha$,
which modulates the level of amplification between output distributions from
original and distorted visual inputs, as formulated in Equation 3. The study
observes minimal variance in the aggregate scores across the three MME subsets
as $\alpha$ ranges from $0.25$ to $1.0$, showcasing a uniform improvement over
regular decoding. This consistency evidences the efficacy and stability of the
contrastive decoding strategy across a spectrum of $\alpha$ settings.

###  B.3 Effect of $\beta$ in Adaptive Plausible Constraint

$\beta$ |  | Hallucination Subset
---
Total Scores
| Perception Subset
---
Total Scores
| Recognition Subset
---
Total Scores
0 | $577.22_{\pm 11.10}$ | $1299.04_{\pm 39.30}$ | $302.98_{\pm 19.82}$
0.001 | $574.44_{\pm 6.31}$ | $1298.71_{\pm 40.24}$ | $289.88_{\pm 14.02}$
0.01 | $583.33_{\pm 18.78}$ | $1324.44_{\pm 37.84}$ | $327.38_{\pm 17.11}$
0.1 | $591.67_{\pm 36.06}$ | $1340.89_{\pm 55.91}$ | $323.45_{\pm 5.89}$
0.2 | $591.67_{\pm 7.26}$ | $1343.06_{\pm 13.06}$ | $328.57_{\pm 16.37}$
0.5 | $635.00_{\pm 7.64}$ | $1474.02_{\pm 15.53}$ | $331.43_{\pm 13.03}$

Table 6: Ablation studies of $\beta$ on the MME benchmark.

Table 6 presents the results of an ablation study on $\beta$, which controls
the adaptive plausible constraint in Equation 4, where larger $\beta$
indicates more aggressive truncation, keeping only high-probability tokens.
The table illustrates that a $\beta$ value of $0$, implying no constraint,
results in suboptimal performance, which validates our rationale for
implementing this constraint: the output distribution with distorted visual
inputs can still uphold fundamental linguistic standards and common sense
reasoning. Indiscriminate penalization could inadvertently sanction these
valid outputs and promote the generation of implausible tokens. As $\beta$
increases, improvements in total scores across the hallucination and
perception subsets are observed, highlighting the constraint’s critical role
in reducing hallucinations and improving LVLMs’ perception capacities.

###  B.4 Effect of Different Sampling Strategies

Sampling Strategy | w. VCD | Accuracy | Precision | Recall | F1 Score
---|---|---|---|---|---
Top P | No | $84.91_{\pm 0.25}$ | $94.73_{\pm 0.30}$ | $73.93_{\pm 0.52}$ | $83.05_{\pm 0.32}$
Yes | $\textbf{87.82}_{\pm 0.66}$ | $91.17_{\pm 0.57}$ | $83.76_{\pm 0.87}$ | $\textbf{87.31}_{\pm 0.72}$
Top K | No | $83.04_{\pm 0.16}$ | $91.84_{\pm 0.15}$ | $72.53_{\pm 0.44}$ | $81.05_{\pm 0.24}$
Yes | $\textbf{87.49}_{\pm 0.56}$ | $91.09_{\pm 0.53}$ | $83.11_{\pm 0.71}$ | $\textbf{86.92}_{\pm 0.60}$
Greedy | No | $87.10_{\pm 0.00}$ | $97.33_{\pm 0.00}$ | $76.29_{\pm 0.00}$ | $85.54_{\pm 0.00}$
Yes | $\textbf{88.49}_{\pm 0.28}$ | $91.78_{\pm 0.28}$ | $84.56_{\pm 0.44}$ | $\textbf{88.02}_{\pm 0.30}$
Top K+Temperature 0.7 | No | $85.17_{\pm 0.12}$ | $94.82_{\pm 0.12}$ | $74.40_{\pm 0.35}$ | $83.38_{\pm 0.17}$
Yes | $\textbf{87.94}_{\pm 0.51}$ | $91.21_{\pm 0.49}$ | $83.98_{\pm 0.60}$ | $\textbf{87.45}_{\pm 0.54}$
Top K+Temperature 1.5 | No | $79.28_{\pm 0.22}$ | $86.48_{\pm 1.12}$ | $69.42_{\pm 0.91}$ | $77.01_{\pm 0.22}$
Yes | $\textbf{86.97}_{\pm 0.50}$ | $90.96_{\pm 0.64}$ | $82.09_{\pm 0.41}$ | $\textbf{86.30}_{\pm 0.51}$
Table 7: An ablation study of different sampling strategies.

Table 7 presents an ablation study on various sampling strategies conducted on
the POPE-Random dataset using LLaVA-1.5. In addition to the direct sampling
approach discussed in the main paper, this experiment includes four additional
sampling strategies: Top P sampling (specifically, $p=0.9$), Top K sampling
(specifically, $k=50$), Greedy decoding, and Top K sampling with temperature
normalization ($k=50,temp=1.5/0.7$). The results indicate that applying VCD,
irrespective of the sampling strategy employed, consistently contributes to
hallucination mitigation and an enhancement of the general performance
capabilities of LVLMs. This consistency underscores the versatility and
effectiveness of VCD across different sampling strategies in the context of
LVLMs.

###  B.5 Effect of VCD when LVLMs Scale Up

Dataset | POPE | Model | Decoding | Accuracy | Precision | Recall | F1 Score
---|---|---|---|---|---|---|---
MSCOCO | Random | LLaVA1.5(13B) | Regular | $83.31_{\pm 0.32}$ | $91.46_{\pm 0.38}$ | $73.48_{\pm 0.75}$ | $81.49_{\pm 0.43}$
VCD | $\textbf{87.39}_{\pm 0.32}$ | $92.68_{\pm 0.36}$ | $81.19_{\pm 0.63}$ | $\textbf{86.55}_{\pm 0.41}$
InstructBLIP(13B) | Regular | $82.36_{\pm 0.59}$ | $86.93_{\pm 0.85}$ | $76.19_{\pm 1.05}$ | $81.20_{\pm 0.68}$
VCD | $\textbf{84.53}_{\pm 0.38}$ | $88.55_{\pm 0.54}$ | $79.32_{\pm 0.44}$ | $\textbf{83.68}_{\pm 0.40}$
Popular | LLaVA1.5(13B) | Regular | $82.47_{\pm 0.55}$ | $89.55_{\pm 0.92}$ | $73.53_{\pm 0.78}$ | $80.75_{\pm 0.61}$
VCD | $\textbf{85.74}_{\pm 0.25}$ | $89.33_{\pm 0.52}$ | $81.19_{\pm 0.63}$ | $\textbf{85.06}_{\pm 0.29}$
InstructBLIP(13B) | Regular | $79.07_{\pm 0.66}$ | $81.11_{\pm 0.70}$ | $75.79_{\pm 1.27}$ | $78.35_{\pm 0.78}$
VCD | $\textbf{81.47}_{\pm 0.42}$ | $82.89_{\pm 0.64}$ | $79.32_{\pm 0.44}$ | $\textbf{81.07}_{\pm 0.39}$
Adversarial | LLaVA1.5(13B) | Regular | $80.00_{\pm 0.52}$ | $84.46_{\pm 0.73}$ | $73.53_{\pm 0.76}$ | $78.62_{\pm 0.58}$
VCD | $\textbf{81.92}_{\pm 0.44}$ | $82.40_{\pm 0.42}$ | $81.17_{\pm 0.65}$ | $\textbf{81.78}_{\pm 0.47}$
InstructBLIP(13B) | Regular | $76.57_{\pm 0.75}$ | $77.00_{\pm 0.83}$ | $75.79_{\pm 0.80}$ | $76.39_{\pm 0.75}$
VCD | $\textbf{79.56}_{\pm 0.41}$ | $79.67_{\pm 0.59}$ | $79.39_{\pm 0.50}$ | $\textbf{79.52}_{\pm 0.38}$

Table 8: Results of 13B-sized LLaVA1.5 and InstructBLIP variants on the POPE
metric. The best performance of each setting is bolded.

Our evaluation extends to larger 13B variants of the LLaVA-1.5 and
InstructBLIP models101010Qwen-VL lacks larger variants., assessing the
scalability of our proposed VCD across different LVLM magnitudes. Table 8
reveals that the 7B and 13B variants of LLaVA-1.5 and InstructBLIP exhibit
comparable performances across POPE settings (e.g., $81.33$ and $81.49$ F1
scores for LLaVA-1.5 7B and 13B in Random setting), suggesting that increasing
the model parameters does not inherently resolve hallucination issues, thereby
underscoring the pertinence of addressing this challenge. Crucially, VCD
consistently boosts performance in all POPE configurations, reaffirming its
robustness independent of model scale.

##  Appendix C Detailed Experimental Results on MME

Model | Decoding | Existence | Count | Position | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR |  | Percetion
---
Total
LLaVA1.5 | Regular | $175.67_{\pm 7.51}$ | $124.67_{\pm 19.59}$ | $114.00_{\pm 9.32}$ | $151.00_{\pm 10.45}$ | $127.82_{\pm 7.13}$ | $113.59_{\pm 3.43}$ | $148.30_{\pm 3.49}$ | $129.95_{\pm 5.33}$ | $102.20_{\pm 4.70}$ | $92.00_{\pm 31.29}$ | $1279.19_{\pm 37.09}$
VCD | $\textbf{184.66}_{\pm 6.81}$ | $\textbf{138.33}_{\pm 15.68}$ | $\textbf{128.67}_{\pm 7.21}$ | $\textbf{153.00}_{\pm 7.58}$ | $\textbf{132.11}_{\pm 6.53}$ | $\textbf{120.94}_{\pm 7.57}$ | $\textbf{152.20}_{\pm 0.21}$ | $\textbf{140.45}_{\pm 6.73}$ | $\textbf{109.60}_{\pm 2.66}$ | $\textbf{104.00}_{\pm 30.96}$ | $\textbf{1363.96}_{\pm 40.58}$
Qwen-VL | Regular | $155.00_{\pm 3.54}$ | $127.67_{\pm 13.36}$ | $\textbf{131.67}_{\pm 7.73}$ | $173.00_{\pm 9.75}$ | $137.76_{\pm 2.49}$ | $116.24_{\pm 2.58}$ | $\textbf{150.17}_{\pm 2.80}$ | $158.00_{\pm 2.40}$ | $\textbf{125.75}_{\pm 5.74}$ | $\textbf{89.50}_{\pm 7.37}$ | $1364.74_{\pm 30.78}$
VCD | $\textbf{156.00}_{\pm 6.52}$ | $\textbf{131.00}_{\pm 6.19}$ | $128.00_{\pm 3.61}$ | $\textbf{181.67}_{\pm 5.14}$ | $\textbf{142.45}_{\pm 2.96}$ | $\textbf{137.35}_{\pm 2.45}$ | $149.10_{\pm 2.51}$ | $\textbf{163.95}_{\pm 1.77}$ | $127.65_{\pm 2.81}$ | $86.00_{\pm 3.35}$ | $\textbf{1403.17}_{\pm 14.57}$
InstructBLIP | Regular | $141.00_{\pm 13.97}$ | $75.33_{\pm 14.16}$ | $\textbf{66.67}_{\pm 3.91}$ | $97.33_{\pm 16.94}$ | $109.66_{\pm 6.21}$ | $87.50_{\pm 6.80}$ | $128.74_{\pm 3.13}$ | $100.55_{\pm 3.33}$ | $94.10_{\pm 5.05}$ | $83.50_{\pm 19.25}$ | $1223.72_{\pm 86.59}$
VCD | $\textbf{168.33}_{\pm 11.55}$ | $\textbf{92.33}_{\pm 8.47}$ | $64.00_{\pm 6.73}$ | $\textbf{123.00}_{\pm 11.27}$ | $\textbf{121.09}_{\pm 5.12}$ | $\textbf{118.71}_{\pm 3.93}$ | $\textbf{149.65}_{\pm 1.46}$ | $\textbf{123.65}_{\pm 1.89}$ | $\textbf{110.60}_{\pm 2.89}$ | $\textbf{96.50}_{\pm 8.94}$ | $\textbf{1447.19}_{\pm 25.43}$

Table 9: Results on all MME perception-related tasks. The best performance of
each setting is bolded.

In Table 9, we present the performance of three LVLM baselines on the
perception-related tasks of the MME benchmark. The baselines exhibit
consistent performance patterns, and the deployment of VCD uniformly improves
their perceptual competencies. This improvement is likely a consequence of
VCD’s capability to diminish statistical biases and language priors, thus
recalibrating the LVLMs to favor visual information over pre-existing biases
and priors.

Model | Decoding |  | Common Sense
---
Reasoning
| Numerical
---
Calculation
| Text
---
Translation
| Code
---
Reasoning
| Recognition
---
Total
LLaVA1.5 | Regular | $106.43_{\pm 9.04}$ | $\textbf{72.50}_{\pm 15.51}$ | $\textbf{95.50}_{\pm 12.80}$ | $78.50_{\pm 22.12}$ | $352.93_{\pm 27.98}$
VCD | $\textbf{111.29}_{\pm 7.06}$ | $68.50_{\pm 16.64}$ | $89.50_{\pm 5.97}$ | $\textbf{84.00}_{\pm 25.35}$ | $\textbf{353.29}_{\pm 36.19}$
Qwen-VL | Regular | $109.86_{\pm 10.31}$ | $\textbf{60.00}_{\pm 6.37}$ | $83.00_{\pm 11.91}$ | $\textbf{67.50}_{\pm 10.16}$ | $\textbf{320.36}_{\pm 26.00}$
VCD | $\textbf{114.39}_{\pm 5.83}$ | $54.00_{\pm 9.62}$ | $\textbf{85.00}_{\pm 7.29}$ | $64.50_{\pm 7.37}$ | $317.89_{\pm 11.59}$
InstructBLIP | Regular | $79.57_{\pm 6.81}$ | $62.86_{\pm 11.23}$ | $55.00_{\pm 10.75}$ | $70.00_{\pm 10.75}$ | $267.43_{\pm 10.42}$
VCD | $\textbf{109.71}_{\pm 7.31}$ | $\textbf{66.00}_{\pm 16.45}$ | $\textbf{69.00}_{\pm 11.54}$ | $\textbf{74.50}_{\pm 20.26}$ | $\textbf{319.21}_{\pm 20.60}$
Table 10: Results on all MME recognition-related tasks. The best performance
of each setting is bolded.

Furthermore, Table 10 showcases the LVLMs’ performances on recognition-related
tasks within the MME benchmark. The results indicate that the application of
VCD, while alleviating hallucination issues and augmenting perceptual
capabilities, does not compromise the inherent reasoning abilities of LVLMs,
as evidenced by the stable overall recognition scores.

##  Appendix D More Case Studies

Figure 7: More examples from LLaVA-Bench of our proposed VCD for hallucination
corrections. Hallucinated objects from LVLM’s regular decoding are highlighted
in red. Figure 8: More examples from LLaVA-Bench of our proposed VCD for
enhanced general perception and recognition capacities.

Additional case studies on the LLaVA-bench are presented to illustrate the
effectiveness of our approach across different LVLMs. Figure 7 provides
further instances of hallucination corrections by our method. Meanwhile,
Figure 8 offers supplemental examples of the enhancements brought by our
proposed VCD in bolstering the general perception and recognition abilities of
LVLMs.

Description: AI that scores image description accuracy and detailedness.
Instructions: You are an AI designed to evaluate and score the performance of
two AI assistants in describing a given image. Your primary focus is on the
accuracy and detailedness of their descriptions. You will assess the accuracy
by checking for hallucinations - any part of the description that is
inconsistent with the image content. For detailedness, you will consider how
rich the response is in necessary details, excluding any hallucinated parts.
You will provide scores on a scale from 1 to 10 for each assistant separately,
based on these criteria. After scoring, you will offer an explanation for your
evaluation, ensuring it is free from bias and not influenced by the order of
presentation of the responses. Input format: [Assistant 1] {Response 1} [End
of Assistant 1] [Assistant 2] {Response 2} [End of Assistant 2] Output format:
Accuracy: Scores of the two answers: Reason: Detailedness: Scores of the two
answers: Reason:

Table 11: The configuration to build an image-description evaluator with
GPT-4V

Figure 9: Case illustrating the evaluation of GPT-4V in open-ended generation
task. “Assistant 1” and “Assistant 2” correspond to “visual contrastive
decoding” and “regular decoding”.

##  Appendix E Prompt and Case for GPT-4V Aided Evaluation

To evaluate open-ended generation, we utilize GPT-4V to assess the accuracy
and detailedness of LVLMs’ responses. The specific configurations are detailed
in Table 11. Additionally, an illustrative evaluation case is presented in
Figure 9.