#  Don’t Miss the Forest for the Trees: Attentional Vision Calibration for
Large Vision Language Models

Sangmin Woo Donguk Kim$\ast$ Jaehyuk Jang$\ast$ Yubin Choi Changick Kim
KAIST
{smwoo95, kdu3613, jhyuk, choibinbin<EMAIL_ADDRESS>
https://sangminwoo.github.io/AvisC/ Equal Contribution

###### Abstract

This study addresses the issue observed in Large Vision Language Models
(LVLMs), where excessive attention on a few image tokens, referred to as blind
tokens, leads to hallucinatory responses in tasks requiring fine-grained
understanding of visual objects. We found that tokens receiving lower
attention weights often hold essential information for identifying nuanced
object details — ranging from merely recognizing object existence to
identifying their attributes (color, position, _etc_.) and understanding their
relationships. To counteract the over-emphasis on blind tokens and to
accurately respond to user queries, we introduce a technique called
Attentional Vision Calibration (AvisC). During the decoding phase, AvisC
identifies blind tokens by analyzing the image-related attention distribution.
It then dynamically adjusts the logits for the next token prediction by
contrasting the logits conditioned on the original visual tokens with those
conditioned on the blind tokens. This effectively lowers the dependency on
blind tokens and promotes a more balanced consideration of all tokens. We
validate AvisC on benchmarks such as POPE, MME, and AMBER, where it
consistently outperforms existing decoding techniques in mitigating object
hallucinations in LVLMs.

##  1 Introduction

Large Vision Language Models (LVLMs) [9, 50, 28, 27, 1] have demonstrated
remarkable capabilities in generating coherent and contextually relevant
descriptions from visual inputs. This success largely hinges on the models’
ability to interpret and integrate complex visual information with textual
data. However, a significant challenge that persists in these models is their
tendency towards "hallucinations" — producing inaccurate or fabricated
descriptions that do not accurately reflect the visual data. The phenomenon of
hallucination in LVLMs can significantly impede their reliability, especially
in applications requiring precise and trustworthy visual descriptions.

Figure 1:  Attention bias in LVLMs. Even when the image ($\mathcal{V}$) does
not contain information relevant to the query ($\mathcal{Q})$, LVLMs [9, 28]
exhibit a tendency for attention to be biased towards a few image tokens
(_i.e_., blind tokens). This phenomenon is observed by averaging the attention
weights across all layers when generating the first response token.

As shown in Fig. 1 LVLMs [28, 9] exhibit biased attention towards certain
image tokens, which we refer to as blind tokens. Even when all the pixels in
the image contain identical information and when the image does not contain
any information relevant to the query, LVLMs [9, 28] tend to focus their
attention on a few specific image tokens. This pattern of focusing on certain
tokens, despite their lack of meaningful content, highlights potential flaws
in the decoding processes of LVLMs.

Further analysis of the attention patterns in LVLMs is demonstrated in Fig. 2.
We examine the attention distribution of LLaVA-1.5 [28] in response to the
given image and query. The attention weights appear to align reasonably well
with object regions, indicating that the model’s attention is appropriately
targeted at relevant areas. However, deeper scrutiny into the functional
impact of these weights on response predictions reveals intriguing insights.
Zeroing out image tokens that receive the bulk of attention does not
significantly impact the original prediction logits. This suggests that LVLMs
might assign high attention weights to tokens that do not carry substantial
object-discriminative information. On the other hand, zeroing out image tokens
with low attention weights drastically changes the prediction logits, leading
to near-equal probabilities and indicating a loss of crucial object-
discriminative information. Such skewed attention disproportionately favors
blind tokens and often overlooks tokens that might contain finer details,
potentially resulting in misclassifications or entirely incorrect predictions.

In response to this challenge, we propose a novel method termed Attentional
Vision Calibration (AvisC) which recalibrates the model’s attention on the fly
during the decoding phase. Unlike existing approaches that typically require
extensive training [19, 34, 49, 43] or auxiliary models [47, 36, 10, 41, 23],
AvisC operates without these prerequisites. AvisC dynamically modifies the
decoding process in three steps: (i) Based on our finding that different LVLM
exhibit different attentional patterns (see Fig. 4), we first select relevant
layers that allocate a higher attention proportion to the image tokens. (ii)
Next, we identify blind tokens, which disproportionately monopolize attention.
These tokens are isolated, and all other image tokens are zeroed out, creating
a biased visual input. (iii) Finally, we employ a contrastive decoding [21,
11]. This technique contrasts the logits calculated from the original visual
input with those derived solely from the blind tokens. By doing so, it
amplifies the influence of tokens that exhibit significant differences between
the two distributions. The recalibration process aims to decrease the reliance
on blind tokens—those that occupy substantial attention but lack detailed
content—and increase focus on underemphasized tokens that may hold crucial
details. As a result, AvisC promotes a more balanced attention distribution
across the image. This recalibration not only reduces hallucinations but also
enhances the overall interpretative accuracy of LVLMs.

Through a series of experiments involving benchmarks like POPE [32], CHAIR
[24], MME [12], and AMBER [38], we demonstrate that AvisC significantly
mitigates hallucination while simultaneously improving the models’ ability to
capture and describe detailed image attributes more accurately.

Figure 2:  Impact of blind/non-blind tokens on prediction logits. (Left)
Zeroing out image tokens with attention weights higher than the $\mu+\sigma$
(mean + standard deviation), _i.e_., blind tokens, does not significantly
affect the original prediction logits, suggesting that LVLMs may assign high
attention weights to tokens that do not carry significant object-
discriminative information. Conversely, zeroing out non-blind tokens
drastically disrupts the logits, often leading to near 50:50 probabilities,
indicating a loss of object-discriminative information. (Right) Similarly,
examples demonstrate that zeroing out non-blind tokens results in a loss of
discriminative power for previously well-classified instances or produces
entirely incorrect predictions, causing a significant drop in performance.

##  2 Related Work

LVLMs [22, 9, 50, 28, 27, 4] are prone to generating hallucinations, _i.e_.,
misalignment between visual inputs and textual outputs. These hallucinations
manifest across various semantic dimensions such as incorrect object presence,
attributes, or relations.

To mitigate these, researchers have developed strategies across three levels:

Input-level. Efforts here focus on data quality improvement to reduce
hallucinations [13, 26, 37, 30], including the introduction of negative data
[26], counterfactual data [42] to challenge the model’s assumptions, dataset
cleansing to minimize noise and errors [39, 44].

Model-level. This includes increasing the resolution at which models process
visual data [6, 27, 28, 45], or enhancing perception abilities through
advanced vision encoders [15, 18, 35]. These are usually training-based [19,
44], and often involve auxiliary supervision from external datasets [7] and
reinforcement learning techniques [2, 48, 14, 34, 43] to better align model
outputs with accurate visual representations.

Output-level. Techniques like contrastive decoding [21, 11] directly contrast
incorrect predictions during decoding, helping models distinguish between
accurate and inaccurate descriptions. Guided decoding [47, 10, 5] leverages
external models like CLIP [31] or DETR [3] to enhance accuracy. Other
approaches include training-free methods [36, 46, 16] and post-hoc corrections
via self-feedback [20, 40].

Among these, we focus on contrastive decoding methods: (1) VCD [21] mitigates
statistical biases and language priors by contrasting output distributions
from original and distorted visual inputs, moderating decoding probabilities.
(2) M3ID [11] uses a similar approach where the reference image amplifies its
influence over the language prior, thereby enhancing the generation of tokens
with higher mutual information with the visual prompt.

Our approach belongs to the output-level category. AvisC analyzes attention
patterns to identify blind tokens during decoding steps. It then utilizes a
contrastive decoding technique to enhance token prediction. Our method does
not require additional training, external data or models, and costly self-
feedback mechanisms.

##  3 Approach: AvisC

Figure 3:  An overview of AvisC.

We propose a straightforward method, called AvisC, to enhance visual object
understanding in LVLMs during the decoding phase. AvisC dynamically calibrates
the over-emphasis on blind tokens on-the-fly at every token generation step.
The calibration is guided by the attention patterns of image tokens in
response to the given image and textual query. Importantly, AvisC operates
without additional training, external models, or complex self-feedback
mechanisms. A visual summary of our method is shown in Fig. 3. AvisC modifies
the decoding process in three steps: (1) Layer selection: choose layers
significantly influenced by image tokens, (2) Blind token identification:
detect non-relevant tokens in selected layers, and (3) Contrastive decoding:
adjust the decoding process to mitigate the influence of blind tokens.

###  3.1 LVLM Framework

#### Uni-modal encoding.

LVLM begins by encoding visual inputs and textual queries into compact
representations. Visual inputs provide contextual information that helps
generate responses relevant to the textual queries. The text data is
tokenized, turning it into a sequence of manageable pieces for further
processing. For visual data, a commonly used encoder is a pre-trained model
like CLIP [31], which is already semantically aligned with textual data
through extensive training on image-text pairs.

#### Cross-modal alignment.

As LLM inherently perceives only text, aligning text and vision modalities is
essential. Instead of retraining LLM, which would be prohibitively expensive,
a more viable approach is to use a learnable cross-modal alignment module.
This module, such as Q-Former [22] or a linear projection layer [28],
transforms visual features into a format compatible with the LLM’s input
space. This process results in a set of visual tokens,
$\mathcal{V}=\\{\nu_{0},\nu_{1},\dots,\nu_{N-1}\\}$, which are then
concatenated with the text tokens,
$\mathcal{Q}=\\{\sigma_{N},\sigma_{N+1},\dots,\sigma_{N+M-1}\\}$, to form a
unified input sequence of length $N+M$.

#### Next token prediction via LLM.

The concatenated sequence of visual and textual tokens is then processed by
LVLM, parametrized by $\theta$, which generates responses in an auto-
regressive manner. The model calculates logits for each potential next token:

$\ell_{t}=\log p(\xi_{t}|\mathcal{V},\mathcal{Q},\xi_{<t};\theta),$ (1)

where $\ell_{t}$ are the logits for the next token at timestep $t$, $\xi_{t}$
denotes the next token being predicted, and $\xi_{<t}$ represents the sequence
of tokens generated up to timestep $(t-1)$. From these logits, we apply a
softmax function to convert logits into a normalized probability distribution:

$p(\xi_{t})=\mathrm{Softmax}(\ell_{t}).$ (2)

The next token $\xi_{t}$ is sampled from this probability distribution, with
the model continuing this predictive process until the response sequence is
complete.

###  3.2 Attentional Vision Calibration for Alleviating Hallucinations

Visual hallucinations in LVLMs can emerge during the decoding phase when the
model selects tokens based on erroneous probability distributions that do not
align with the visual inputs. These discrepancies, as demonstrated in our
observations (refer to Figs. 1 and 2), often originate from an attentional
bias toward certain non-relevant tokens, referred to as blink tokens. Our
methodology aims to recalibrate these attention patterns to correct such
hallucinations.

Layer selection. As illustrated in Fig. 4, the attentional behavior of LVLM
layers varies depending on the model’s architecture or size. For example,
InstructBLIP [9] shows increasing attention levels in the later layers,
whereas LLaVA-1.5 [28] exhibits a concentration of attention in the earlier
layers. To adapt these diverse models, we initially focus on selecting layers
that exhibit a high proportion of image-related attention. Formally, we define
the attention weight matrix for $i$-th layer as follows:

$\mathbf{A}_{i}=\left[\mathbf{a}_{h,q,k}^{i}\right]_{(h,q,k)=(1,1,1)}^{(h,q,k)=(H,N+M,N+M)},$
(3)

where $\mathbf{a}_{h,q,k}^{i}$ represents the attention weight assigned by
head $h$, for query $q$, to key $l_{k}$ in layer $i$. The model handles two
types of tokens: image tokens ($\mathcal{V}\in\mathbb{R}^{N\times D}$) and
query tokens ($\mathcal{Q}\in\mathbb{R}^{M\times D}$). Next, we calculate the
proportion of attention dedicated to image tokens for each layer $i$ as:

$AP_{i}^{\text{layer}}=\frac{\sum_{h}\sum_{k=1}^{N}\mathbf{a}^{i}_{h,(N+M),k}}{\sum_{i,h}\sum_{k=1}^{N}\mathbf{a}^{i}_{h,(N+M),k}},$
(4)

where $H$ is the total number of attention heads, $N$ is the number of image
tokens, and $M$ is the number of query tokens. We sort the layers by this
proportion and employ top-P sampling based on a predefined threshold value
$\gamma$. The selected layers are:

$\\{\text{Selected
Layers}\\}=\text{top-P}(\\{AP_{i}^{\text{layer}}\\}_{i=1}^{L},\gamma).$ (5)

Here, top-P selects layers until the cumulative proportion of image attention
across these layers meets or exceeds $\gamma$. These selected layers are used
to analyze and adjust the attention at the token level and identify specific
image tokens that the model may over-rely on, _i.e_., blind tokens.

(a) InstructBLIP [9]

(b) LLaVA-1.5 [28]

Figure 4:  Layer-wise image attention proportion in LVLMs [27, 9]. This shows
the proportion of attention given to image tokens at each layer compared to
the overall attention. Different layers exhibit distinct attention patterns,
and these patterns vary between different models. The attention weights are
averaged across 60 questions from the LLaVA-bench [28].

Blind token identification. After selecting relevant layers, we calculate the
attention weights for each image token within these layers. The attention
proportion for image tokens, denoted as $AP^{\text{image}}$, is calculated by
averaging the attention weights across the selected layers and attention
heads:

$AP^{\text{image}}=\frac{\sum_{i\in\\{\text{Selected
Layers}\\}}\sum_{h=1}^{H}\mathbf{a}^{i}_{h,(N+M),[1:N]}}{|\\{\text{Selected
Layers}\\}|\times H}.$ (6)

To identify tokens that disproportionately capture the model’s attention,
_i.e_., blind tokens, we calculate the mean ($\mu$) and standard deviation
($\sigma$) of the image attention weights. Tokens with an attention proportion
exceeding $\mu+\lambda\sigma$ (where $\lambda$ is a hyperparameter) are
classified as blind tokens:

$\\{\text{Blind Token
Indices}\\}=\\{j|AP^{\text{image}}_{j}>\mu+\lambda\sigma\\}.$ (7)

Contrastive decoding. Our method seeks to reduce the influence of blind
tokens, thereby decreasing the incidence of hallucinations in LVLMs. Drawing
inspiration from recent successes in contrastive decoding [21, 11], which
effectively minimizes hallucinations by contrasting the differences between an
image and its distorted counterpart, we adopt a similar scheme. We construct a
new set of visual tokens $\mathcal{V}^{\ast}$ by zeroing out non-blind tokens
and only leaving blind tokens, which biases the input towards emphasizing
blind tokens:

$\mathcal{V}^{\ast}=\bigcup_{j=1}^{N}\vmathbb{1}_{\\{j\in\text{Blind Token
Indices}\\}}(j)\nu_{j}.$ (8)

Next, we compute the logits using both the original input ($\mathcal{V}$) and
the biased input ($\mathcal{V}^{\ast}$):

$\displaystyle\ell_{t}$ $\displaystyle=\log
p(\xi_{t}|\mathcal{V},\mathcal{Q},\xi_{<t};\theta),$ (9)
$\displaystyle\ell_{t}^{\ast}$ $\displaystyle=\log
p(\xi_{t}|\mathcal{V^{\ast}},\mathcal{Q},\xi_{<t};\theta),$

where $\ell_{t}$ and $\ell_{t}^{\ast}$ are the logits computed from the
original and the biased inputs, respectively. We adjust the logits by
contrasting the original and biased outputs, and then sample the next token
$\xi_{t}$ from the following softmax distribution:

$\xi_{t}\sim\mathrm{Softmax}((1+\alpha)\ell_{t}-\alpha\ell_{t}^{\ast}).$ (10)

Here, $\alpha$ is a hyperparameter that moderates the contrastive effect. This
balances the distribution of attention across tokens thereby mitigating the
likelihood of visual hallucinations in LVLMs.

##  4 Experiments

###  4.1 Evaluation Setup

In our experiments, we did not constrain the LVLMs to provide one-word answers
in discriminative tasks, which often require simple ‘Yes’ or ‘No’ responses.
For instance, we avoid instructions such as "Please answer in one word." in
the query text. We see that imposing a one-word response constraint on LVLMs
leads to notable changes in performance (see Appendix D). For the experiments,
we set $\text{P}=0.5$ in Eq. 5, $\lambda=1$ Eq. 7, $\alpha=3$ for InsturctBLIP
[9] and $\alpha=2.5$ for LLaVA-1.5 [28] in Eq. 10. 111Please refer to Appendix
for further experimental and implementation details.

| Setup | Method | InstructBLIP [9] | LLaVA 1.5 [28]
---|---|---|---|---
|  Acc. |  Prec. |  Rec. |  F1 |  Acc. |  Prec. |  Rec. |  F1
MS-COCO [25] | Random | base |  82.27 |  82.84 |  81.40 |  82.11 |  84.47 |  83.35 |  86.13 |  84.72
VCD |  83.37 |  83.39 |  82.60 |  83.24 |  84.80 |  83.00 |  87.53 |  85.20
M3ID |  84.37 |  84.62 |  84.00 |  84.31 |  86.00 |  85.11 |  87.27 |  86.18
AvisC |  88.73 |  93.88 |  82.87 |  88.03 |  87.93 |  88.24 |  87.53 |  87.88
Popular | base |  77.77 |  74.81 |  83.73 |  79.02 |  82.23 |  79.72 |  86.47 |  82.95
VCD |  78.00 |  75.12 |  83.73 |  79.19 |  82.27 |  79.19 |  87.53 |  83.15
M3ID |  77.30 |  74.10 |  83.93 |  78.71 |  82.83 |  79.62 |  88.27 |  83.72
AvisC |  83.90 |  81.33 |  88.00 |  84.53 |  84.33 |  81.71 |  88.47 |  84.96
Adversarial | base |  73.13 |  69.41 |  82.60 |  75.46 |  77.10 |  72.57 |  87.13 |  79.19
VCD |  75.87 |  72.85 |  82.47 |  77.36 |  76.10 |  71.50 |  86.80 |  78.41
M3ID |  76.03 |  72.47 |  83.93 |  77.79 |  77.70 |  73.23 |  87.33 |  79.66
AvisC |  81.57 |  80.37 |  83.53 |  81.92 |  77.53 |  72.82 |  87.87 |  79.64
A-OKVQA [33] | Random | base |  81.00 |  77.71 |  86.93 |  82.06 |  82.73 |  77.43 |  92.40 |  84.26
VCD |  81.73 |  78.67 |  87.07 |  82.66 |  81.30 |  75.45 |  92.80 |  83.23
M3ID |  82.33 |  77.81 |  90.47 |  83.66 |  83.57 |  77.86 |  93.80 |  85.09
AvisC |  88.47 |  87.66 |  89.53 |  88.59 |  84.60 |  79.29 |  93.67 |  85.88
Popular | base |  75.00 |  70.14 |  87.07 |  77.69 |  76.10 |  69.86 |  91.80 |  79.34
VCD |  75.33 |  70.52 |  87.07 |  77.92 |  75.43 |  68.58 |  93.87 |  79.26
M3ID |  75.60 |  70.40 |  88.33 |  78.36 |  76.80 |  70.20 |  93.13 |  80.06
AvisC |  81.77 |  77.82 |  88.87 |  82.98 |  78.83 |  72.10 |  94.07 |  81.63
Adversarial | base |  68.80 |  63.57 |  88.07 |  73.84 |  67.90 |  62.11 |  91.80 |  74.09
VCD |  69.70 |  64.54 |  87.47 |  74.27 |  67.43 |  61.50 |  93.20 |  74.11
M3ID |  69.57 |  64.21 |  88.40 |  74.39 |  68.10 |  61.99 |  93.60 |  74.58
AvisC |  72.53 |  67.12 |  88.33 |  76.28 |  68.97 |  62.70 |  93.67 |  75.11
GQA [17] | Random | base |  80.00 |  77.08 |  85.40 |  81.02 |  82.40 |  77.03 |  92.33 |  83.99
VCD |  81.73 |  79.35 |  85.80 |  82.45 |  82.27 |  75.85 |  94.67 |  84.22
M3ID |  80.57 |  76.77 |  87.67 |  81.85 |  82.83 |  76.64 |  94.47 |  84.62
AvisC |  86.47 |  85.89 |  87.27 |  86.57 |  85.00 |  78.81 |  95.73 |  86.45
Popular | base |  73.53 |  68.80 |  86.13 |  76.49 |  72.03 |  65.57 |  92.80 |  76.84
VCD |  74.10 |  69.45 |  86.07 |  76.87 |  71.77 |  64.90 |  94.80 |  77.05
M3ID |  74.57 |  69.45 |  87.83 |  77.53 |  72.83 |  66.04 |  94.00 |  77.58
AvisC |  78.00 |  73.68 |  87.13 |  79.84 |  74.80 |  67.46 |  95.80 |  79.17
Adversarial | base |  68.00 |  63.49 |  84.73 |  72.59 |  68.73 |  62.54 |  93.40 |  74.92
VCD |  70.27 |  65.43 |  85.93 |  74.29 |  68.27 |  62.00 |  94.40 |  74.84
M3ID |  68.90 |  64.06 |  86.13 |  73.47 |  68.13 |  61.88 |  94.47 |  74.78
AvisC |  73.07 |  67.80 |  87.87 |  76.54 |  69.20 |  62.61 |  95.33 |  75.58

Table 1:  POPE benchmark results. AvisC consistently outperforms base decoding
and other methods: VCD [21] and M3ID [11]. We reimplemented VCD and M3ID in
our evaluation setup.

#### LVLMs.

We evaluated AvisC on two state-of-the-art LVLMs: LLaVA-1.5 [28] and
InstructBLIP [9], both incorporating Vicuna 7B [8] as an LLM backbone.
LLaVA-1.5 synchronizes image and text modalities by applying linear projection
layers, while InstructBLIP uses the Q-Former [22] to efficiently link visual
and textual features using a fixed number of tokens (_e.g_., 32 tokens).
Notably, AvisC is model-agnostic and can integrate with various of LVLM
architectures.

#### Benchmarks.

(1) POPE [24] views hallucination evaluation as a binary classification task
(yes/no) with questions regarding object presence (_e.g_., "Is there a cat in
the image?"). It includes 500 images from MS-COCO and evaluates them based on
visible objects and imaginary ones across different object categories, using
three setups (random, popular, and adversarial). (2) MME [12] evaluates 14
subtasks including object hallucination by answering binary questions about
object existence, count, position, color, _etc_. (3) AMBER [38] includes both
generative and discriminative tasks, focusing on hallucinations related to
object existence, attributes, and relationships, with performance evaluated
using CHAIR for generative tasks and an F1 score for discriminative tasks. The
overall AMBER score is calculated as $((100-\text{CHAIR})+\text{F1})/2$.

#### Baselines.

AvisC aims to minimize hallucinations in LVLMs without the need for external
models, costly self-feedback mechanisms, or further training. We select
baseline methods that fulfill these conditions. We choose recent contrastive
decoding methods as baselines, notably VCD [21] and M3ID [11]. These methods
are designed to reduce object hallucinations by enhancing the influence of the
reference image over the language model’s prior or statistical bias, by
contrasting output distributions from both original and altered visual inputs.
We reimplemented VCD and M3ID within our evaluation framework.

###  4.2 Results on Benchmarks

POPE. Table 1 showcases the performance of different methods on the POPE
benchmark [24] across MS-COCO [25], A-OKVQA [33], and GQA [17] datasets,
evaluated under Random, Popular, and Adversarial setups. (AvisC) consistently
outperforms the baseline (base) and other decoding methods (VCD [21], M3ID
[11]) in most cases, achieving the highest Accuracy and F1 scores. It also
demonstrates balanced improvements in Precision and Recall, indicating a
reduction in errors and better information capture. For InstructBLIP, AvisC
shows a significant performance boost, particularly in mitigating
hallucinations related to object existence. However, LLaVA 1.5 exhibits less
pronounced improvements in Popular and Adversarial setups, highlighting its
limitations in more challenging scenarios. Yet, overall, AvisC proves to be
robust and effective across different datasets and query setups.

Model | Method | Object-level | Attribute-level |  Total Score
---|---|---|---|---
Existence |  Count |  Position |  Color |
InstructBLIP [9] | base |  170.19(±11.12) |  89.52(±11.04) |  67.62(±14.04) |  114.76(±9.60) |  442.09(±31.51)
VCD |  172.62(±8.92) |  98.33(±15.99) |  71.90(±13.42) |  117.14(±10.70) |  459.99(±16.56)
M3ID |  173.89(±10.52) |  89.72(±13.44) |  72.72(±14.77) |  110.56(±7.20) |  446.88(±28.54)
AvisC (Ours) |  184.76(±5.56) |  82.85(±12.16) |  74.76(±6.19) |  131.43(±4.76) |  473.80(±19.67)
LLaVA 1.5 [28] | base |  173.57(±8.16) |  110.00(±15.82) |  100.47(±18.78) |  125.24(±15.91) |  509.28(±30.57)
VCD |  172.14(±8.09) |  117.14(±8.76) |  103.33(±20.56) |  119.52(±8.58) |  512.14(±31.82)
M3ID |  178.33(±6.83) |  107.22(±14.78) |  96.39(±5.52) |  127.50(±8.28) |  509.44(±22.52)
AvisC (Ours) |  189.29(±1.82) |  104.76(±11.66) |  106.19(±13.93) |  127.86(±9.13) |  528.09(±24.70)

Table 2:  MME-Hallucination [12] benchmark results. Our method effectively
reduces hallucinations at both object and attribute levels, surpassing VCD
[21] and M3ID [11] in Total Score.

(a) InstructBLIP [9]

(b) LLaVA-1.5 [28]

Figure 5:  Performance comparison on MME-Fullset. AvisC achieves top
performance in 7 of 14 categories with InstructBLIP [9] and in 11 categories
with LLaVA-1.5 [28]. Beyond minimizing hallucinations, AvisC also boosts the
general functionality of LVLMs.

MME-Hallucination. Table 2 presents performance results for InstructBLIP [9]
and LLaVA 1.5 [28] on the MME-Hallucination benchmark [12], focusing on
object-level (Existence, Count) and attribute-level (Position, Color) metrics.
Both models exhibit significant improvements in the Existence category with
Ours, achieving the highest scores. While VCD [21] performs best in the Count
metric, AvisC excels in the Position and Color categories, attaining the top
scores for both models. AvisC demonstrates superior performance in Total Score
compared to other methods, affirming its effectiveness in reducing
hallucinations and improving accuracy across multiple metrics.

MME-Fullset. Figure 5 compares the performance of various decoding methods on
the MME-Fullset [12] across 14 categories. AvisC generally outperforms other
methods, achieving top performance in 7 categories for InstructBLIP and 11
categories for LLaVA 1.5. This demonstrates AvisC’s effectiveness in enhancing
understanding of visual information through attention calibration. However,
both models see a decline in performance for the Count category with AvisC,
and InstructBLIP shows poor OCR performance. Conversely, LLaVA 1.5 experiences
significant OCR improvement with AvisC, indicating the method’s variable
impact across different models. Overall, AvisC provides consistent and
superior results across most tasks compared to other methods.

| Metric | InstructBLIP [9] |  | LLaVA 1.5 [28]
---|---|---|---|---
|  base |  VCD [21] |  M3ID [11] |  AvisC |  |  base |  VCD [21] |  M3ID [11] |  AvisC
Generative |  CHAIR$\downarrow$ |  8.40(±0.57) |  7.60(±0.42) |  6.85(±0.07) |  6.70(±0.28) |  |  7.95(±0.64) |  6.70(±0.42) |  6.00(±0.14) |  6.25(±0.07)
Cover $\uparrow$ |  46.40(±1.27) |  47.65(±0.35) |  47.20(±0.71) |  46.65(±1.48) |  |  44.45(±0.21) |  46.50(±0.28) |  48.90(±0.28) |  46.55(±0.64)
Hal$\downarrow$ |  31.10(±0.64) |  29.90(±0.99) |  27.50(±0.71) |  28.00(±0.28) |  |  31.00(±2.83) |  27.80(±1.70) |  26.00(±0.28) |  25.60(±1.70)
Cog$\downarrow$ |  2.60(±0.05) |  2.20(±0.14) |  2.20(±0.14) |  2.55(±0.35) |  |  2.15(±0.35) |  1.95(±0.35) |  1.45(±0.07) |  2.00(±0.04)
Discriminative |  Acc. $\uparrow$ |  68.20(±0.14) |  69.65(±0.35) |  69.05(±0.35) |  72.60(±0.42) |  |  67.00(±0.71) |  67.30(±1.41) |  67.25(±0.21) |  70.70(±0.57)
Prec. $\uparrow$ |  79.00(±0.14) |  80.70(±0.42) |  79.70(±0.28) |  72.60(±0.42) |  |  85.45(±0.49) |  86.10(±1.70) |  86.50(±0.57) |  85.45(±0.21)
Rec. $\uparrow$ |  70.70(±0.42) |  71.60(±0.42) |  71.25(±0.35) |  76.10(±0.05) |  |  60.95(±1.20) |  60.55(±1.34) |  60.05(±0.07) |  67.55(±0.92)
F1 $\uparrow$ |  74.60(±0.14) |  75.90(±0.42) |  75.25(±0.07) |  78.60(±0.28) |  |  71.10(±0.99) |  71.10(±1.56) |  70.90(±0.14) |  75.45(±0.64)
AMBER $\uparrow$ |  83.10(±0.35) |  84.15(±0.05) |  84.20(±0.07) |  85.95(±0.05) |  |  81.58(±0.18) |  82.20(±0.99) |  82.45(±0.14) |  84.60(±0.35)

Table 3:  AMBER [38] benchmark results. AvisC outperforms contrastive decoding
baselines [21, 11] in both generative and discriminative tasks, achieving the
highest AMBER score.

AMBER. Table 3 presents the results of InstructBLIP [9] and LLaVA 1.5 [28] on
the AMBER benchmark [38], which includes both generative tasks (detailed image
descriptions) and discriminative tasks (answering questions about images).
Both models show significant improvements in Accuracy and F1 scores in
discriminative tasks using AvisC, outperforming the base, VCD [21], and M3ID
[11] methods. In generative tasks, AvisC continues to exhibit substantial
gains, indicating its effectiveness in generating detailed image descriptions.
Notably, there is a marked improvement in the Existence metric, highlighting
the method’s accuracy in detecting objects. Overall, both models achieve the
highest performance across most metrics with AvisC. AvisC stands out with the
highest AMBER score, indicating its comprehensive superiority in both
generative and discriminative tasks. Fig. 6 visualizes the performance of each
decoding method across discriminative tasks in the AMBER benchmark.

(a) InstructBLIP [9]

(b) LLaVA-1.5 [28]

Figure 6:  Performance comparison on AMBER discriminative tasks. Our
demonstrates superior performance overall, particularly excelling in the
Existence and Action categories in both InstructBLIP [9] and LLaVA-1.5 [28].
Figure 7:  Response comparison on LLaVA-Bench [28]. Hallucinations are colored
in red. AvisC demonstrates a robust understanding of images and reduces
hallucinations in responses.

###  4.3 Analysis

Qualitative comparison on LLaVA-Bench. Figure 7 shows responses from LLaVa-1.5
[28] given LLaVA-Bench [28] samples, highlighting the differences between
sentences generated by standard decoding (Base), VCD [21], and those produced
by AvisC. The results demonstrate the effectiveness of AvisC in dealing with a
variety of challenging visual contexts. Base and VCD often generate
descriptions that include errors or hallucinations where elements not present
in the image are described. In contrast, AvisC helps counteract these
hallucinations, generating sentences that reflect a more accurate
comprehension of the image.

| Case |  Acc. $\uparrow$ |  Prec. $\uparrow$ |  Rec. $\uparrow$ |  F1 $\uparrow$
---|---|---|---|---|---
InstructBLIP | Zeros |  88.50 |  93.00 |  83.27 |  87.86
Ones |  82.50 |  75.48 |  96.27 |  84.62
Noise |  86.77 |  84.71 |  89.73 |  87.15
Mask |  88.53 |  90.14 |  86.53 |  88.30
LLaVA 1.5 | Zeros |  87.87 |  88.12 |  87.53 |  87.83
Ones |  79.97 |  72.22 |  97.40 |  82.94
Noise |  88.47 |  93.19 |  83.00 |  87.80
Mask |  84.77 |  86.29 |  82.67 |  84.44

Table 4:  Design choices for non-blind image token deactivation.

Alternatives to zero-out. Table 4 presents the results of ablation experiments
on various deactivation schemes for non-blind image tokens using InstructBLIP
[9] and LLaVA 1.5 [28] models, evaluated on the POPE-COCO-random benchmark
[24]. We compare Zeros, Ones, Noise, and Mask. For InstructBLIP, Mask achieves
the highest Accuracy and F1 score, while Zeros excels in Precision. Ones shows
the highest Recall, and Noise provides balanced performance with high
Precision and Recall. For LLaVA 1.5, Noise achieves the highest Accuracy and
Precision, while Zeros shows balanced performance across all metrics. On
average, using Zeros was the most effective in improving model performance by
calibrating attention to image tokens.

##  5 Conclusion

Our study demonstrates that addressing the issue of blind tokens in LVLMs
significantly mitigates the problem of hallucinatory responses. By introducing
a novel technique, termed AvisC, which recalibrates attention distribution
during the decoding phase, without the need for extensive retraining, external
data or models, or costly self-feedback mechanisms. our approach effectively
enhances the model’s focus on underemphasized yet crucial tokens. This
recalibration leads to a more balanced consideration across all image tokens,
significantly reducing reliance on misleading information and minimizing
object hallucinations. The efficacy of AvisC is validated across several
benchmarks, where it consistently outperforms existing decoding techniques.
AvisC not only boosts reliability but also ensures more trustworthy
applications of LVLMs in real-world scenarios requiring fine-grained visual
understanding. We hope our contributions inspire further research into
refining and adapting LVLMs for increasingly complex multimodal tasks.

## References

  * [1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  2023\.
  * [2] Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elor.  Mocha: Multi-objective reinforcement mitigating caption hallucinations.  arXiv preprint arXiv:2312.03631, 2023.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [4] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.  Shikra: Unleashing multimodal llm’s referential dialogue magic.  arXiv preprint arXiv:2306.15195, 2023.
  * [5] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou.  Halc: Object hallucination reduction via adaptive focal-contrast decoding.  arXiv preprint arXiv:2403.00425, 2024.
  * [6] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.  Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.  arXiv preprint arXiv:2312.14238, 2023.
  * [7] Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang.  Mitigating hallucination in visual language models with visual supervision.  arXiv preprint arXiv:2311.16479, 2023.
  * [8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
  * [9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [10] Ailin Deng, Zhirui Chen, and Bryan Hooi.  Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding.  arXiv preprint arXiv:2402.15300, 2024.
  * [11] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto.  Multi-modal hallucination control by visual information grounding.  arXiv preprint arXiv:2403.14003, 2024.
  * [12] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  arXiv preprint arXiv:2306.13394, 2024.
  * [13] Anisha Gunjal, Jihan Yin, and Erhan Bas.  Detecting and preventing hallucinations in large vision language models.  arXiv preprint arXiv:2308.06394, 2023.
  * [14] Anisha Gunjal, Jihan Yin, and Erhan Bas.  Detecting and preventing hallucinations in large vision language models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143, 2024.
  * [15] Xin He, Longhui Wei, Lingxi Xie, and Qi Tian.  Incorporating visual experts to resolve the information loss in multimodal large language models.  arXiv preprint arXiv:2401.03105, 2024.
  * [16] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu.  Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation.  arXiv preprint arXiv:2311.17911, 2023.
  * [17] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * [18] Jitesh Jain, Jianwei Yang, and Humphrey Shi.  Vcoder: Versatile vision encoders for multimodal large language models.  arXiv preprint arXiv:2312.14233, 2023.
  * [19] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang.  Hallucination augmented contrastive learning for multimodal large language model.  arXiv preprint arXiv:2312.06968, 2023.
  * [20] Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo.  Volcano: mitigating multimodal hallucination through self-feedback guided revision.  arXiv preprint arXiv:2311.07362, 2023.
  * [21] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing.  Mitigating object hallucinations in large vision-language models through visual contrastive decoding.  arXiv preprint arXiv:2311.16922, 2023.
  * [22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In International conference on machine learning, pages 19730–19742. PMLR, 2023.
  * [23] Wei Li, Zhen Huang, Houqiang Li, Le Lu, Yang Lu, Xinmei Tian, Xu Shen, and Jieping Ye.  Visual evidence prompting mitigates hallucinations in multimodal large language models.  2023\.
  * [24] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.  Evaluating object hallucination in large vision-language models.  arXiv preprint arXiv:2305.10355, 2023.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [26] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  In The Twelfth International Conference on Learning Representations, 2023.
  * [27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv preprint arXiv:2310.03744, 2023.
  * [28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2023.
  * [29] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.  Mmbench: Is your multi-modal model an all-around player?  arXiv preprint arXiv:2307.06281, 2023.
  * [30] Jiaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang.  Evaluation and enhancement of semantic grounding in large vision-language models.  In AAAI-ReLM Workshop, 2024.
  * [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [32] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.  Object hallucination in image captioning.  arXiv preprint arXiv:1809.02156, 2018.
  * [33] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In European Conference on Computer Vision, pages 146–162. Springer, 2022.
  * [34] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al.  Aligning large multimodal models with factually augmented rlhf.  arXiv preprint arXiv:2309.14525, 2023.
  * [35] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  arXiv preprint arXiv:2401.06209, 2024.
  * [36] David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal.  Contrastive region guidance: Improving grounding in vision-language models without training.  arXiv preprint arXiv:2403.02325, 2024.
  * [37] Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al.  Vigc: Visual instruction generation and correction.  arXiv preprint arXiv:2308.12714, 2023.
  * [38] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang.  Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation.  arXiv preprint arXiv:2311.07397, 2023.
  * [39] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann.  Mitigating hallucinations in large vision-language models with instruction contrastive decoding.  arXiv preprint arXiv:2403.18715, 2024.
  * [40] Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan.  Logical closed loop: Uncovering object hallucinations in large vision-language models.  arXiv preprint arXiv:2402.11622, 2024.
  * [41] Dingchen Yang, Bowen Cao, Guang Chen, and Changjun Jiang.  Pensieve: Retrospect-then-compare mitigates visual hallucination.  arXiv preprint arXiv:2403.14401, 2024.
  * [42] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang.  Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data.  arXiv preprint arXiv:2311.13614, 2023.
  * [43] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al.  Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.  arXiv preprint arXiv:2312.00849, 2023.
  * [44] Zihao Yue, Liang Zhang, and Qin Jin.  Less is more: Mitigating multimodal hallucination from an eos decision perspective.  arXiv preprint arXiv:2402.14545, 2024.
  * [45] Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li.  Halle-switch: Controlling object hallucination in large vision language models.  arXiv e-prints, pages arXiv–2310, 2023.
  * [46] Yi-Fan Zhang, Weichen Yu, Qingsong Wen, Xue Wang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan.  Debiasing large visual language models.  arXiv preprint arXiv:2403.05262, 2024.
  * [47] Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu.  Mitigating object hallucination in large vision-language models via classifier-free guidance.  arXiv preprint arXiv:2402.08680, 2024.
  * [48] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He.  Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization.  arXiv preprint arXiv:2311.16839, 2023.
  * [49] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao.  Analyzing and mitigating object hallucination in large vision-language models.  arXiv preprint arXiv:2310.00754, 2023.
  * [50] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.

## Appendix

##  Appendix A Further Implementation Details

The text generation decoding process utilized cut-off sampling to assess the
effectiveness of logit distribution enhancements achieved through AvisC.
Following the experimental settings of VCD [21], tokens with probability
values below $\beta$ times the maximum generating token probability were
masked and excluded from sampling. Specifically, we only consider text tokens
that belong to $\mathcal{H}$ at the generation step $t$:

$\mathcal{H}(\xi_{<t})=\\{\xi_{t}\in\mathcal{H}:p\left(\xi_{t}\mid\mathcal{V},\mathcal{Q},\xi_{<t};\theta\right)\geq\beta\max_{w}p\left(w\mid\mathcal{V},\mathcal{Q},\xi_{<t};\theta\right)\\}.\\\
$ (11)

We set the balancing parameter $\beta$ to 0.1. We configured the LVLMs to
generate a maximum of 64 tokens for both generative and discriminative tasks.
During our experiments with the LLaVA-1.5 [28], we utilized the "llava_v1"
template provided by LLaVA for the conversation setup.

For reproducing the VCD [21], we referenced the official code provided by VCD
[21]. We set $\alpha$ for contrastive decoding to 1.0, the cut-off
hyperparameter $\beta$ to 0.1, and the diffusion noise step $T$ used for
generating noise images to 500. In the reproduction of the M3ID [11], we used
0.2 as the $\lambda$. The aforementioned token generation decoding method was
utilized to ensure a fair comparison with other methods.

##  Appendix B More Experimental Details

###  B.1 Benchmark

POPE We employed the official benchmark described in [24], which comprises
3,000 question-answer pairs across the random, popular, and adversarial
settings. Our queries followed the structure ‘Is there a [object] in the
image?’, where [object] is selected either at random, from the most common
objects in the dataset, or from objects that are often found alongside the
specified [object], tailored to the random, popular, and adversarial
scenarios, respectively. The model’s effectiveness was assessed by determining
if the model-generated response accurately matched the correct answer (’Yes’
or ’No’), using metrics such as accuracy, precision, recall, and mean
F1-score.222https://github.com/RUCAIBox/POPE

MME The MME dataset [12] is divided into 10 perceptual categories (existence,
count, position, color, posters, celebrity, scene, landmark, artwork, OCR) and
four cognitive categories (commonsense reasoning, numerical calculation, text
translation, code reasoning). While we utilized the official dataset, we
modified the prompt by eliminating the instruction (_i.e_. "Answer the
question using a single word or phrase.") that restricts LVLMs to response
length.333https://github.com/BradyFU/Awesome-Multimodal-Large-Language-
Models/tree/Evaluation

AMBER The AMBER dataset [38] comprises 1004 images along with their associated
generative task prompts (_i.e_. "Describe this image.") and questions
categorized into three discriminative task types (existence, attribute, and
relation). we randomly sampled 500 questions for the generative tasks and 5000
questions for the discriminative tasks, and the evaluation was established on
official protocols.444https://github.com/junyangwang0410/AMBER.git

LLaVA-Bench [28] features a collection of $24$ images, accompanying $60$
questions that span a range of contexts, including indoor and outdoor scenes,
paintings, and sketches. This dataset is crafted to assess the capability of
LVLMs in tackling more challenging tasks and their adaptability to new
domains.555https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild

###  B.2 Metrics

Metric on the MME The evaluation dataset, $\mathcal{D}$ of the MME bechmark
consists of two questions, {$q_{1}$, $q_{2}$} regarding the same visual
input,$\mathcal{V}$. Every question in $\mathcal{D}$ is a discriminating
question. Based on the answers ("Yes" or "No") provided by the LVLMs, we can
calculate the accuracy (ACC) for any $i$ as follows:

$\text{{ACC}}(\mathcal{V},q_{i})=\begin{cases}1&\text{if
LVLMs$(\mathcal{V},q_{i})$ = Answer$(\mathcal{V},q_{i})$},\\\
0&\text{otherwise}.\end{cases}$ (12)

ACC, which is calculated for each query corresponding to an individual image,
ACC\+ [12] is calculated only when both queries associated with a single image
are answered correctly. This metric is defined as follows:

$\text{{ACC+}}(\mathcal{V})=\begin{cases}1&\text{if LVLMs$(\mathcal{V},q_{i})$
= Answer$(\mathcal{V},q_{i})$ for any $i$},\\\ 0&\text{otherwise}.\end{cases}$
(13)

MME score for each evaluated category is the summation of ACC and ACC+.

Metrics on the Generative Task Considering $R$ as the response by LVLMs for
visual input, $V$, the following metrics can be delineated.

CHAIR [32, 38] The CHAIR evaluates the occurrence of hallucinatory objects in
responses to LVLMs. CHAIR uses an annotated list of objects
$A$={$a_{obj}^{1}$, $a_{obj}^{2}$, $\ldots$, $a_{obj}^{n}$} to calculate how
often hallucinated objects appear in the responses. Let $R$={$r_{obj}^{1}$,
$r_{obj}^{2}$, $\ldots$, $r_{obj}^{m}$} be the list of objects mentioned in
the response of LVLMs, the formula for CHAIR is given as:

$\text{{CHAIR}}=1-\frac{len(R\cap A)}{len(R)}.$ (14)

Cover [38] The Cover metric measures how completely the objects in the
response cover the identified objects in the image. Cover calculates the ratio
of objects mentioned in the response to the total objects listed. The formula
for Cover is:

$\text{{Cover}}=\frac{len(R\cap A)}{len(A)}.$ (15)

Hal [38] The Hal metric quantifies the presence of hallucinations by checking
if the CHAIR value is not zero, indicating the presence of hallucinations. The
Hal is presented by the following formula:

$\text{{Hal}}=\begin{cases}1&\text{if $\text{{CHAIR}}\neq 0$},\\\
0&\text{otherwise}.\end{cases}$ (16)

Cog [38] The Cog metric evaluates whether the hallucinations in LVLMs
responses resemble human cognition. The Cog calculates the ratio of the human
hallucinatory object targets, denoted as $H$={$h_{obj}^{1}$, $h_{obj}^{2}$,
$\ldots$, $h_{obj}^{n}$} to the objects mentioned in the response. The formula
for Cog is:

$\text{{Cog}}=\frac{len(R\cap H)}{len(R)}.$ (17)

AMBER Score [38] The AMBER Score metric evaluates the comprehensive
performance of LVLMs for generative tasks and discriminative tasks. This score
combines the CHAIR metric for generative tasks with the F1 metric for
discriminative tasks. The formula representing the AMBER Score is as follows:

$\text{{AMBER Score}}=\frac{1}{2}\times(1-\textit{CHAIR}+\textit{F1}).$ (18)

##  Appendix C License of Assets.

POPE [24] is made available under the MIT License. AMBER [38] and LLaVA-Bench
[28] is available under Apache-2.0 License. InstructBLIP [9] is under
BSD-3-Clause License and LLaVA [28] is licensed under the Apache-2.0 License.

##  Appendix D Additional Experiments

POPE [24] one word.

| Setup | Method | InstructBLIP [9] | LLaVA 1.5 [28]
---|---|---|---|---
|  Acc. |  Prec. |  Rec. |  F1 |  Acc. |  Prec. |  Rec. |  F1
MS-COCO [25] | Random | base |  81.53 |  82.71 |  79.73 |  81.19 |  83.77 |  92.31 |  73.67 |  81.94
VCD |  82.03 |  83.77 |  79.47 |  81.56 |  85.43 |  93.25 |  76.40 |  83.99
AvisC |  86.03 |  95.53 |  75.60 |  84.41 |  84.67 |  97.88 |  70.87 |  82.21
Popular | base |  78.47 |  77.73 |  79.80 |  78.75 |  82.57 |  89.62 |  73.67 |  80.86
VCD |  79.13 |  78.94 |  79.47 |  79.20 |  83.17 |  88.36 |  76.40 |  81.94
AvisC |  84.27 |  91.45 |  75.60 |  82.77 |  83.67 |  95.25 |  70.87 |  81.27
Adversarial | base |  77.43 |  76.09 |  80.00 |  78.00 |  79.77 |  83.85 |  73.73 |  78.47
VCD |  77.23 |  76.10 |  79.40 |  77.72 |  80.27 |  82.76 |  76.47 |  79.49
AvisC |  81.83 |  86.20 |  75.80 |  80.67 |  81.83 |  90.99 |  70.67 |  79.55
A-OKVQA [33] | Random | base |  81.33 |  78.52 |  86.27 |  82.21 |  84.93 |  89.16 |  79.53 |  84.07
VCD |  81.57 |  78.78 |  86.40 |  82.42 |  85.53 |  87.64 |  82.73 |  85.12
AvisC |  87.10 |  89.95 |  83.53 |  86.62 |  87.33 |  95.09 |  78.73 |  86.14
Popular | base |  76.87 |  72.69 |  86.07 |  78.82 |  80.90 |  81.77 |  79.53 |  80.64
VCD |  77.30 |  73.10 |  86.40 |  79.19 |  81.17 |  80.22 |  82.73 |  81.46
AvisC |  82.47 |  81.79 |  83.53 |  82.65 |  85.03 |  90.08 |  78.73 |  84.03
Adversarial | base |  71.40 |  66.67 |  85.60 |  74.96 |  74.80 |  72.63 |  79.60 |  75.95
VCD |  72.47 |  67.39 |  87.07 |  75.97 |  75.03 |  71.87 |  82.27 |  76.72
AvisC |  76.47 |  73.16 |  83.60 |  78.03 |  79.27 |  79.58 |  78.73 |  79.16
GQA [17] | Random | base |  80.57 |  77.47 |  86.20 |  81.60 |  84.80 |  87.88 |  80.73 |  84.16
VCD |  81.73 |  79.02 |  86.40 |  82.55 |  85.63 |  86.89 |  83.93 |  85.38
AvisC |  85.30 |  88.57 |  81.07 |  84.65 |  87.40 |  95.17 |  78.80 |  86.21
Popular | base |  74.67 |  70.17 |  85.80 |  77.20 |  79.37 |  78.59 |  80.73 |  79.64
VCD |  74.63 |  69.94 |  86.40 |  77.30 |  78.73 |  76.03 |  83.93 |  79.78
AvisC |  80.63 |  80.37 |  81.07 |  80.72 |  83.33 |  86.66 |  78.80 |  82.54
Adversarial | base |  72.63 |  67.78 |  86.27 |  75.92 |  76.00 |  74.13 |  79.87 |  76.89
VCD |  71.93 |  67.21 |  85.67 |  75.32 |  76.40 |  72.76 |  84.40 |  78.15
AvisC |  77.60 |  75.91 |  80.87 |  78.31 |  80.37 |  81.52 |  78.53 |  80.00

Table 5:  POPE [24] results with one-word constraint. We use the instruction
"Please answer in one word." at the end of the query text.

As shown in Tab. 5, we see that imposing a one-word response constraint on
LVLMs leads to notable changes in performance compared to Tab. 1. Despite the
change in query setup, AvisC shows the best performance on the POPE benchmark.
Specifically, precision and recall vary significantly in the COCO random setup
comparing scenarios with and without the instruction "Please answer this
question with one word." To mitigate these impacts and better evaluate
discriminative capabilities, we designed experiments that allow the LVLMs to
freely make judgments and provide explanations for these judgments rather than
restricting them to answers in one word.

(a) InstructBLIP [9] ($\lambda=1$)
---
| Object | Attribute |  Total Score
$\alpha$ |  Exist |  Count |  Position |  Color |
0.5 |  180 |  83.33 |  80.00 |  130 |  473.33
2 |  180 |  86.66 |  75 |  135 |  476.66
2.5 |  180 |  85.00 |  71.66 |  135 |  471.66
3 |  195 |  75.00 |  73.33 |  135 |  478.33

(b) InstructBLIP [9] ($\alpha=3$)
---
| Object | Attribute |  Total Score
$\lambda$ |  Exist |  Count |  Position |  Color |
0 |  180 |  75 |  60 |  115 |  430
0.1 |  185 |  60 |  65 |  123.33 |  433.33
1 |  195 |  75 |  73.33 |  135 |  478.33
1.5 |  195 |  75 |  73.33 |  135 |  478.33

(c) LLaVA-1.5 [27] ($\lambda=1$)
---
| Object | Attribute |  Total Score
$\alpha$ |  Exist |  Count |  Position |  Color |
0.5 |  185 |  111.66 |  103.33 |  115 |  514.99
2 |  180 |  103.33 |  101.66 |  120 |  504.99
2.5 |  180 |  105 |  111.66 |  120 |  516.66
3 |  180 |  105 |  111.66 |  120 |  516.66

Table 6:  Ablation on MME-Hallucination [12]. We set $\alpha=3$, $\lambda=1$
for InstructBLIP [9] and $\alpha=2.5$, $\lambda=1$ for LLaVA-1.5 [27].

Ablations on $\alpha$ and $\lambda$. $\lambda$ is a threshold for
discriminating blind tokens that excessively concentrate attention weight, as
detailed in Eq. 7. $\alpha$ is contrastive decoding hyperparameter defined in
Eq. 10. We conducted ablation experiments on the MME-Hallucination [29]
benchmark to evaluate how these hyperparameters influence the performance of
our AvisC. Tab. 4 parts (a) and (c) display the experimental results using
InstructBLIP [9] and LLaVA-1.5 [28], respectively, where we fixed fixed
$\lambda$=1 and varied $\alpha$ from 0.5 to 3. While there are variations
across evaluation categories, performance comprehensively improves with
increasing values of $\alpha$. Specifically, each LVLM achieved the highest
total score at $\alpha$=3 and $\alpha$=2.5. These results suggest that
enhancing the intensity of contrastive decoding can improve the robustness of
LVLMs against visual hallucinations. Table Tab. 4-(b) presents the
experimental results for the InstructBLIP [9] model using varying values of
$\lambda$. The result indicates that performance enhances as $\lambda$
increases, demonstrating that our AvisC yields better results when applied to
a smaller number of blind tokens with excessive attention weight.

##  Appendix E Additional Qualitative Results

We provide additional qualitative results on all benchmarks (POPE [24], MME
[12], AMBER [38], and LLaVA-Bench [28]) in Figs. 8, 9, 10 and 11.

Figure 8:  Qualitative examples on POPE [24].

Figure 9:  Qualitative examples on MME [12].

Figure 10:  Qualitative examples of InstructBLIP [9] on AMBER [38].

Figure 11:  Qualitative examples of LLaVA-1.5 [27] on AMBER [38] .

##  Appendix F Limitations

While AvisC demonstrated strong performance on the POPE [24], MME [12] and
AMBER [38] benchmarks, experiment results ( Tab. 2, Fig. 5, Fig. 6) exhibited
a limitations of AvisC. Specifically, the discriminative capability of LVLMs
with AvisC is diminished in tasks involving counting the number of objects
associated with an image. This limitation suggests that our approach
specifically discourages the performance of LVLMs in the count category of MME
and the Numbers category of AMBER. AvisC operates by identifying blind tokens
with disproportionately high attention weights during the text generation
process of LVLMs and modulating the impact of these tokens. In scenarios where
the prompt involves counting objects, blind tokens may hold significant
information.

##  Appendix G Broader Impacts

The release of our proposed AvisC for alleviating hallucinations in LVLMs
comes with a wide range of positive and negative impacts.

Positive impacts. By mitigating hallucination, LVLMs can become more accurate
and reliable tools for a wide range of applications, such as machine
translation, chatbot development, and news generation.

Negative impacts. Our approach, AvisC, aimed at reducing hallucination, could
heighten computational requirements, potentially resulting in higher expenses
and greater energy use.

Overall, the potential positive impacts of research on reducing hallucination
in LVLMs surpass the potential negative consequences. By addressing the
hallucination problem, we can enhance the trustworthiness of LVLMs.


# Conditions for the Absence of Infrared Sensitivity in Cosmological Probes in
Any Gravity Theories

Matteo Magi<EMAIL_ADDRESS>Center for Theoretical Astrophysics and
Cosmology, Institute for Computational Science, University of Zurich, CH–8057
Zurich, Switzerland  Jaiyul Yoo<EMAIL_ADDRESS>Center for Theoretical
Astrophysics and Cosmology, Institute for Computational Science, University of
Zurich, CH–8057 Zurich, Switzerland  Physics Institute, University of Zurich,
Winterthurerstrasse 190, CH–8057, Zurich, Switzerland

###### Abstract

Large-scale surveys allow us to construct cosmological probes such as galaxy
clustering, weak gravitational lensing, the luminosity distance, and cosmic
microwave background anisotropies. The gauge-invariant descriptions of these
cosmological probes reveal the presence of numerous relativistic effects in
the cosmological probes, and they are sensitive (or even divergent) to the
long wave-length fluctuations in the initial conditions. In the standard
$\Lambda$CDM model, this infrared sensitivity is absent due to subtle
cancellations among the relativistic contributions, once the Einstein equation
is used. Here we derive the most general conditions for the absence of
infrared sensitivity in the cosmological probes without committing to general
relativity. We discuss the implications of our results for gravity theories
beyond general relativity.

Introduction.— Cosmological observations in large-scale surveys allow us to
construct cosmological probes such as galaxy clustering, weak gravitational
lensing, the luminosity distance, and cosmic microwave background (CMB)
anisotropies with ever increasing precision (see, e.g., [1, 2, 3, 4, 5, 6] for
the upcoming surveys). In particular, the two-point statistics of these
cosmological probes are often used to constrain the cosmological models (see,
e.g., [7, 8, 9, 10, 11]). These two-point (and also $N$-point) statistics
contain critical information about the initial conditions and the evolution of
fluctuations, which are set by inflation and evolved according to general
relativity in the standard cosmological model. For instance, galaxy clustering
in a simple description measures the matter density fluctuations at the
redshift of a galaxy sample, and CMB anisotropies are a snapshot of the
gravitational potential fluctuations at the recombination epoch.

However, these cosmological probes are obtained by measuring light signals
from a source such as galaxies, CMB photons, and it involves light propagation
from the source to the observer, which are affected by relativistic effects
such as the gravitational redshift, the gravitational lensing effect and so on
[12, 13]. Hence, all the cosmological probes are in fact affected by subtle
relativistic effects, which in turn provide a great opportunity to probe the
early universe and gravity on large scales with the relativistic effects [14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], but at the same
time it can result in sensitivity to the presence of long wave-length
fluctuations or infrared sensitivity in the cosmological probes.

Consider the two-point correlation function of a cosmological probe ${\cal P}$
in a given survey, and the long wave-length contributions to the two-point
correlation is

$\displaystyle\langle{\cal P}(\eta,\bm{x}){\cal
P}(\eta,\bm{y})\rangle\simeq\int d\ln k~{}{\cal T}^{2}_{\cal
P}(\eta,k)\Delta^{2}_{\cal R}(k)~{},$ (1)

where ${\cal T}_{\cal P}$ is the transfer function and $\Delta^{2}_{\cal
R}:=A_{s}(k/k_{0})^{n_{s}-1}$ is the scale-invariant primordial power spectrum
with scalar spectral index $n_{s}\simeq 1$ and amplitude $A_{s}\simeq 10^{-9}$
at a given scale $k_{0}$ [30, 11]. The presence of gravitational potential in
the cosmological probe ${\cal P}$ implies an enhanced sensitivity to long
wave-length fluctuations: the integral is dominated by the contributions at
very low Fourier modes $k$ and may become logarithmically divergent in the
infrared limit $k\to 0$, as the transfer function for gravitational potential
on large scales is a constant.

Any cosmological probes are modulated by long wave-length fluctuations (see,
e.g., [31]), but their sensitivity is expected to gradually wane as the wave-
length increases beyond the characteristic scale, i.e., the distance between
the observer and the sources. In the case of gravitational potentials, such
sensitivity would amount to a violation of the equivalence principle, since
the leading contributions of long wave-length fluctuations correspond to a
spatially uniform gravitational field. In fact, it has been shown in the past
that this infrared sensitivity is absent for galaxy clustering [21, 32, 33,
34], luminosity distance [35, 36], and CMB temperature anisotropies [37] at
the linear order in perturbations by assuming a $\Lambda$CDM model and solving
the Einstein equations in general relativity.

However, these cosmological probes or their theoretical descriptions are not
limited to the underlying model of matter contents or gravity theories like
those in $\Lambda$CDM models adopted in the previous work. The only
assumptions behind the theoretical descriptions of the cosmological probes are
geodesic equation for photons and the Friedmann-Lemaître-Robertson-Walker
(FLRW) metric. So it is interesting to ask “is the absence of infrared
sensitivity specific to a $\Lambda$CDM model with general relativity? What
happens to the cosmological probes in terms of infrared sensitivity in gravity
theories beyond general relativity?” In this Letter, we investigate the
infrared sensitivity of the cosmological probes without assuming any gravity
model, and we derive generic conditions that any gravity theories should
satisfy to ensure the absence of infrared sensitivity.

Cosmological probes.— Here we consider theoretical descriptions for galaxy
clustering, the luminosity distance, and the CMB temperature anisotropies and
investigate their sensitivity to long wave-length fluctuations. These three
cosmological probes are extensively used in large-scale surveys to constrain
cosmological models. Though weak gravitational lensing is also a powerful
cosmological probe (e.g., [38, 39, 40, 41, 42]), the relativistic
contributions to the weak lensing signals come from at least two spatial
derivatives of gravitational potential, and hence it is rather insensitive to
long wave-length fluctuations of our interest here.

The cosmological probes are constructed in terms of basic observables, and in
cosmology these observables are the observed redshift (or spectral
information), the angular position of the light sources, and the number of the
observed sources or the observed flux in case the sources are not spatially
resolved. Under the minimal assumption that the light propagation follows null
geodesics, these observables can then be related to the physical quantities of
the sources by tracing back the null path from the observer position to the
source position [43, 17, 44, 45]. Since the observables are measured in our
rest frame and the physical quantities of the sources are defined in the rest
frame of the sources, their relation connected via a null path is completely
independent of the coordinate system or diffeomorphism invariant [46, 47].
While this description of the cosmological observables involves the matter
contents in terms of the observer and the source, the relation between the
matter and the light propagation (or metric fluctuations) is left free, such
that its validity is not limited to general relativity.

The theoretical description of the luminosity distance is essentially the
physical area of the source in the rest frame, corresponding to the observed
angular size at the observed redshift [14, 16, 48, 49, 22, 50, 35, 36], which
defines the luminosity distance (or the angular diameter distance) in case of
the standard candle, for which the luminosity in the rest frame is known.
Similar to the case for the luminosity distance, the theoretical description
of galaxy clustering is the physical volume of the source in the rest frame,
with the additional dimension corresponding to the observed redshift interval
[17, 18, 19, 20, 21, 51, 23, 24, 26, 25, 52]. Furthermore, galaxy clustering
contains an intrinsic fluctuation in relation to the matter density
fluctuations, as different galaxy samples cluster differently (or known as
galaxy bias [53, 54, 55, 56, 57]). Finally, the theoretical description of CMB
temperature anisotropies is essentially the same as the observed redshift,
under the assumption that the recombination takes place instantaneously at the
temperature set by the atomic physics [58, 59, 60, 37], since its ratio to the
observed temperature in the sky is the (observed) redshift.

Linear-order expressions— Here we adopt the spatially flat FLRW metric for our
cosmological model. In the Newtonian gauge the metric reads

$\displaystyle
ds^{2}=-a^{2}(1+2\psi)d\eta^{2}+a^{2}[(1+2\phi)\delta_{\alpha\beta}+2h_{\alpha\beta}]dx^{\alpha}dx^{\beta},$
(2)

where $a(\eta)$ is the scale factor as a function of conformal time,
$\psi(\eta,\bm{x})$ and $\phi(\eta,\bm{x})$ are the Newtonian-gauge potentials
while $h_{\alpha\beta}(\eta,\bm{x})$ is the transverse-traceless tensor
perturbation. Here we perform only linear-order computations and ignore the
vector contributions. Any time-like motion in this spacetime is described by a
four-velocity $u^{\mu}=a^{-1}(1-\psi,-\partial^{\alpha}v)^{\mu}$, where the
peculiar velocity potential $v$ arises from the inhomogeneities of the
universe. We assume that the observer and light sources move along geodesics
but we will relax this assumption later.

Now we present the linear-order expressions for the cosmological observables.
The expression for the luminosity distance fluctuation in the Newtonian gauge
is given by [50, 35, 36, 29] as

$\displaystyle\delta D_{L}=\delta z+\frac{\delta
r}{\bar{r}_{z}}-\kappa+\phi-\frac{1}{2}h_{\parallel\parallel}\,,$ (3)

and the expression for the observed galaxy number density fluctuation is given
by [17, 32, 33, 29] as

$\displaystyle\delta_{g}=b\left(\delta_{m}+3\mathcal{H}v_{m}\right)-e_{z}\left(\delta
z+\mathcal{H}v_{m}\right)+\delta V\,,$ (4)

where the distortion $\delta V$ in the volume occupied by the source is

$\displaystyle\delta V=2~{}\delta D_{L}+\delta
z+\phi-\partial_{\parallel}v+h_{\parallel\parallel}+\frac{d}{d\bar{r}}\delta
r\,,$ (5)

$\mathcal{H}=a^{\prime}/a$ is the conformal Hubble function, $\delta z$ is the
perturbation in the observed redshift, $\delta r$ is the radial distortion in
the observed source position at comoving distance $\bar{r}_{z}$, $\kappa$ is
the gravitational lensing convergence,
$\delta_{m}=\delta\rho_{m}/\bar{\rho}_{m}$ is the matter density fluctuation,
$b$ is the galaxy bias, $e_{z}$ is the evolution bias parameter of the mean
galaxy number density. We defined the line-of-sight derivative

$\displaystyle\frac{d}{d\bar{r}}=n^{\alpha}\partial_{\alpha}-\partial_{\eta}=:\partial_{\parallel}-\partial_{\eta}\,,$
(6)

and the subscript $\parallel$ denotes the contraction of a spatial index with
the observation direction $n^{\alpha}$. Finally, the expression for the
observed CMB temperature anisotropies is [60, 59, 37]

$\displaystyle\Theta=\left(\frac{1}{4}\delta_{\gamma}-\delta z\right)_{*}\,,$
(7)

where $\delta_{\gamma}=\delta\rho_{\gamma}/\bar{\rho}_{\gamma}$ is the
radiation density fluctuation and all the terms are evaluated at the
recombination position indicated by $*$. For the derivations and detailed
expressions of the redshift perturbation $\delta z$, lensing convergence
$\kappa$, and radial distortion $\delta r$, we refer the reader to the
aforementioned references (see, e.g., [45]). The structure of those quantities
at the linear order is rather simple, consisting of contributions at the
observer position, source position, and integrations along the line-of-sight
direction.

Infrared contributions in the cosmological probes— The observer-source system
in cosmology has a finite extent defined by the comoving distance
$\bar{r}_{z}$ out to the observed redshift of the light source. Since we want
to address the long wave-length behavior of the perturbations, consider a
comoving scale $R$ much larger than $\bar{r}_{z}$ (ideally larger than any
sound horizon of the involved propagating fields). Denoting any scalar or
tensor perturbation as $f$, we define the long wave-length part $f_{L}$ of the
perturbation $f$ by smoothing it over the scale $R$ (the exact form of the
smoothing kernel is irrelevant here). The leftover of this procedure is the
short wave-length contribution $f_{S}:=f-f_{L}$, which starts with at least
$k^{2}f$ in Fourier space for a Gaussian smoothing kernel and hence will not
be considered anymore. By definition, the long-mode $f_{L}$ varies very little
in space over the scale $R$, thus the successive terms in the following
spatial expansion around an arbitrary origin

$\displaystyle
f_{L}(\eta,\bm{x})=:f_{L}(\eta)+x^{\alpha}\partial_{\alpha}f_{L}(\eta)+\mathcal{O}(\bm{x}^{2})\,,$
(8)

can be made arbitrarily small according to the choice of $R$. The
perturbations of our interest are those on the line-of-sight of the observer
and hence their spatial dependence can be parameterized by the comoving
distance and angles, more conveniently in terms of

$\displaystyle f_{0}(\eta):=f_{L}(\eta)\,,\qquad\qquad
f_{1}(\eta,\bm{n}):=\partial_{\parallel}f_{L}(\eta)\,,$ (9)

where the subscripts 0 and 1 represent the zeroth and first-order terms in the
spatial expansion. The spatial expansion of $f_{L}$ in Eq. (8) corresponds in
Fourier space to

$\displaystyle
f_{L}(\eta,\bm{x})=\int\frac{d^{3}k}{(2\pi)^{3}}\left[1+i\bm{k}\cdot\bm{x}+\mathcal{O}(\bm{k}^{2})\right]f_{L}(\eta,\bm{k})\,,~{}~{}~{}~{}$
(10)

such that higher-order terms in the spatial expansion are multiplied by higher
powers of $k$ and the infrared behaviors are hence controlled by the lowest-
order terms $f_{0},f_{1}$ in the expansion. As evident in Eq. (1) these
infrared modes in the cosmological probes would lead to either infrared
divergences or sensitivity of the cosmological probes to super-horizon modes
especially given the scale-invariant fluctuations as the initial conditions.

To study the infrared contributions in the cosmological probes we can apply
the spatial expansion to all the perturbation variables in Eqs. (3)$-$(7) and
keep only the lowest-order terms. Such procedure is straightforward but
results in lengthy expressions. To simplify such expressions, we first
introduce the comoving gauge ($v\equiv 0$) curvature perturbation
$\mathcal{R}$ and the uniform-density gauge ($\delta\rho\equiv 0$) curvature
perturbation $\zeta$:

$\displaystyle\mathcal{R}:=\phi-\mathcal{H}v\,,\qquad\zeta:=\phi-\mathcal{H}\frac{\delta\rho}{\bar{\rho}^{\prime}}\,,$
(11)

here the peculiar velocity $v$ and the energy density
$\rho:=\bar{\rho}+\delta\rho$ are the total ones, i.e.,

$\displaystyle
v=\frac{3\bar{\rho}_{m}v_{m}+4\bar{\rho}_{\gamma}v_{\gamma}}{3\bar{\rho}_{m}+4\bar{\rho}_{\gamma}}\,,\qquad\delta\rho=\delta\rho_{m}+\delta\rho_{\gamma}\,,$
(12)

for models with matter ($m$) and radiation ($\gamma$) fluids. Second, we
exploit the covariant conservation of the energy-momentum tensor, which we
assume to be separately conserved for multiple non-interacting fluids such as
matter and photon fluids today. Mind that the conservation equations hold for
any gravity theory. The continuity equation for linear perturbations yields

$\displaystyle\delta^{\prime}_{m}+3\phi^{\prime}=\Delta
v_{m}\,,\qquad\delta^{\prime}_{\gamma}+4\phi^{\prime}=\Delta v_{\gamma}\,,$
(13)

where the Laplacians can be neglected in the infrared, leading to the
following expressions

$\displaystyle\zeta^{\prime}_{m}=\zeta^{\prime}_{\gamma}=0\,,\qquad\zeta_{\gamma}-\zeta_{m}=\frac{\delta_{\gamma}}{4}-\frac{\delta_{m}}{3}\,,$
(14)

obtained using the definition in Eq. (11) for each fluid. For the spatial
components of the conservation equations we derive

$\displaystyle\psi=\mathcal{H}v_{m}+v^{\prime}_{m}\,,\qquad\psi=v^{\prime}_{\gamma}-\frac{\delta
p_{\gamma}}{4\bar{p}_{\gamma}}\,,$ (15)

where $\delta p_{\gamma}$ is the radiation pressure fluctuation around the
background $\bar{p}_{\gamma}$. Utilizing Eq. (6) to replace the line-of-sight
integration in the infrared with the boundary terms at the observer
($\eta_{\bar{\rm o}}$) and source position ($\eta_{z}$)

$\displaystyle\int_{0}^{\bar{r}_{z}}d\bar{r}\,\partial_{\eta}f_{0,1}(\eta)=f_{0,1}(\eta_{\bar{\rm
o}})-f_{0,1}(\eta_{z})\,,$ (16)

and combining all previous equations, we obtain the zeroth-order terms of the
cosmological probes in the spatial expansion:

$\displaystyle\delta D_{L,0}$ $\displaystyle=$
$\displaystyle\left(\frac{1}{\bar{r}_{z}\mathcal{H}_{z}}-1\right)\left[\mathcal{R}_{m}+h_{\parallel\parallel}\right]^{z}_{\bar{\rm
o}}-\frac{3}{2}h_{\parallel\parallel}(\eta_{\bar{\rm o}})$ (17)
$\displaystyle-\frac{1}{2}h_{\parallel\parallel}+\mathcal{R}_{m}-\frac{1}{\bar{r}_{z}}\int_{0}^{\bar{r}_{z}}d\bar{r}\,\left(\mathcal{R}_{m}-2h_{\parallel\parallel}\right)\,,\qquad$
$\displaystyle\delta_{g,0}$ $\displaystyle=$ $\displaystyle
3b\left(\zeta_{m}-\mathcal{R}_{m}\right)+e_{z}\left[\mathcal{R}_{m}+h_{\parallel\parallel}\right]^{z}_{\bar{\rm
o}}+\delta V_{0}\,,$ (18) $\displaystyle\Theta_{0}$ $\displaystyle=$
$\displaystyle\zeta_{\gamma}-\mathcal{R}_{m}(\eta_{\bar{\rm
o}})+h_{\parallel\parallel}-h_{\parallel\parallel}(\eta_{\bar{\rm o}})\,,$
(19) $\delta V_{0}=2\delta
D_{L,0}-\left(1-\frac{\mathcal{H}^{\prime}_{z}}{\mathcal{H}^{2}_{z}}\right)\left[\mathcal{R}_{m}+h_{\parallel\parallel}\right]^{z}_{\bar{\rm
o}}-\frac{1}{\mathcal{H}_{z}}\left(\mathcal{R}^{\prime}_{m}+h^{\prime}_{\parallel\parallel}\right),\quad$
(20)

with all the variables in the expressions representing the zeroth-order terms
$f_{0}(\eta)$ in the spatial expansion, and the first-order terms

$\displaystyle\delta D_{L,1}$ $\displaystyle=$
$\displaystyle-\left(\frac{1}{\bar{r}_{z}\mathcal{H}_{z}}\right)\int_{0}^{\bar{r}_{z}}d\bar{r}\left(\mathcal{R}_{m}+h_{\parallel\parallel}\right)-\bar{r}_{z}\frac{3}{2}h_{\parallel\parallel}$
(21)
$\displaystyle+\frac{1}{\mathcal{H}_{z}}\left(\mathcal{R}_{m}+h_{\parallel\parallel}\right)-\int_{0}^{\bar{r}_{z}}d\bar{r}\,\left(1-\frac{5}{\bar{r}_{z}}\bar{r}\right)h_{\parallel\parallel}\,,\qquad$
$\displaystyle\delta_{g,1}$ $\displaystyle=$ $\displaystyle
3\bar{r}_{z}b\left(\zeta_{m}-\mathcal{R}_{m}\right)+e_{z}\bigg{[}\bar{r}_{z}\left(\mathcal{R}_{m}-h_{\parallel\parallel}\right)$
(22)
$\displaystyle+\int_{0}^{\bar{r}_{z}}d\bar{r}\left(\mathcal{R}_{m}+h_{\parallel\parallel}\right)\bigg{]}+\delta
V_{1}\,,$ $\displaystyle\delta V_{1}$ $\displaystyle=$ $\displaystyle
2~{}\delta
D_{L,1}-\frac{\bar{r}_{z}}{\mathcal{H}_{z}}\left(\mathcal{R}^{\prime}_{m}+h^{\prime}_{\parallel\parallel}\right)+\left(1-\frac{\mathcal{H}^{\prime}_{z}}{\mathcal{H}^{2}_{z}}\right)\times$
(23)
$\displaystyle\bigg{(}\int_{0}^{\bar{r}_{z}}d\bar{r}\,\left(\mathcal{R}_{m}+h_{\parallel\parallel}\right)-\bar{r}_{z}\left(\mathcal{R}_{m}+h_{\parallel\parallel}\right)\bigg{)}\,,$
$\displaystyle\Theta_{1}$ $\displaystyle=$
$\displaystyle\bar{r}_{*}\left(\zeta_{\gamma}+h_{\parallel\parallel}\right)+v_{\gamma}-v_{m}-\int_{0}^{\bar{r}_{*}}d\bar{r}\,\left(\mathcal{R}_{m}+h_{\parallel\parallel}\right),$

with all the perturbation variables representing the first-order terms
$f_{1}(\eta,\bm{n})$ in the spatial expansion. We emphasize that these
expressions are the leading infrared contributions in the cosmological probes,
for example, ${\cal P}_{L}(\eta,\bm{x})={\cal P}_{0}(\eta)+\bar{r}_{z}{\cal
P}_{1}(\eta,\bm{n})+\mathcal{O}(\bm{x}^{2})$, and they are obtained without
committing to any gravity theory. These equations are the main results of the
current work.

General conditions for the absence of infrared sensitivity.— With a closer
look to Eqs. (17)$-$(Conditions for the Absence of Infrared Sensitivity in
Cosmological Probes in Any Gravity Theories), it is possible to derive
necessary and sufficient conditions for the probes to be devoid of infrared
contributions or the lowest-order terms ${\cal P}_{0}$ and ${\cal P}_{1}$.
First of all, the tensor perturbations must be constant in time in the leading
spatial expansion:

$\displaystyle\left(h^{\prime}_{\alpha\beta}\right)_{0,1}=0\,,$ (25)

for all the cosmological probes. For the luminosity distance it is further
necessary that the comoving-gauge curvature perturbation is time independent
in the infrared

$\displaystyle\left(\mathcal{R}^{\prime}_{m}\right)_{0,1}=0\,.$ (26)

For galaxy clustering, the presence of galaxy bias in proportion to the matter
density fluctuation in the proper-time hypersurface imposes an extra condition
in addition to Eqs. (25) and (26):

$\displaystyle\mathcal{R}_{m,0}=\zeta_{m,0}\,\qquad\qquad\mathcal{R}_{m,1}=\zeta_{m,1}\,,$
(27)

i.e., the comoving-gauge curvature perturbations cannot be any arbitrary
constant in the infrared, but it needs to be equal to the uniform-density
gauge curvature perturbation. In the case of the CMB anisotropies, since both
matter and radiation fluids are involved as the observer and the source, the
conditions in the scalar sector now read

$\displaystyle
v_{m,1}(\eta_{*})=v_{\gamma,1}(\eta_{*})\,,\quad\mathcal{R}_{m,0}=\zeta_{\gamma,0}\,,\quad\mathcal{R}_{m,1}=\zeta_{\gamma,1}\,,\quad$
(28)

in addition to Eqs. (25) and (26). We can combine all the conditions stated so
far, such that any cosmological probes presented here are devoid of infrared
contributions, if and only if

$\begin{split}\left(\mathcal{R}^{\prime}_{m}\right)_{0,1}&=\left(h^{\prime}_{\alpha\beta}\right)_{0,1}=0\,,\qquad
v_{m,1}(\eta_{*})=v_{\gamma,1}(\eta_{*})\,,\\\
\mathcal{R}_{m,0}&=\zeta_{m,0}=\zeta_{\gamma,0}\,,\qquad\mathcal{R}_{m,1}=\zeta_{m,1}=\zeta_{\gamma,1}\,.\end{split}$
(29)

We can recast these equations in terms of fluid quantities via Eqs. (11) and
(14):

$\begin{split}\left(\frac{\delta\rho_{m}}{\bar{\rho}^{\prime}_{m}}-\frac{\delta\rho_{\gamma}}{\bar{\rho}^{\prime}_{\gamma}}\right)_{0,1}=0=\left(\frac{\delta\rho_{m}}{\bar{\rho}^{\prime}_{m}}-v_{m}\right)_{0,1}\,,\\\
v_{m,1}(\eta_{*})=v_{\gamma,1}(\eta_{*})\,,\qquad\qquad\left(h^{\prime}_{\alpha\beta}\right)_{0,1}=0\,.\end{split}$
(30)

These conditions are the adiabatic conditions for the fluids and the
conservation of $\mathcal{R}$ and $h_{\alpha\beta}$ in the infrared. Note,
however, that the equality condition between the velocity and the density is
in general determined not only by the properties of the fluids, but also by
the underlying gravity. We stress again that the derivation of Eq. (30) was
made without resorting to $\Lambda$CDM or general relativity.

Discussion on the general conditions.— The theoretical descriptions of the
cosmological probes and hence the general conditions are derived by assuming
that photons travel on a null geodesic and the matter components follow a
time-like geodesic. The matter components (the observer or the source) can, of
course, deviate from a geodesic motion, if they experience non-gravitational
forces. In general, these forces fall off sufficiently fast on large scales.
To be precise, if two lowest-order terms of the non-gravitational forces
vanish in the spatial expansion, our derivation of the general conditions is
not affected.

Infrared divergences or very large fluctuations in the cosmological probes
would arise, if the general conditions are broken at all $k$ on large scales.
If they are broken at some $k$, but not at all $k$, no infrared divergences
exist, but the cosmological probes are still sensitive to and modulated by
infrared modes. Such infrared sensitivity is not as strong, as the divergence
is only logarithmic, if $n_{s}\simeq 1$ on such large scales. However, the
sensitivity can be further enhanced, if the spectral index $n_{s}$ becomes
redder ($n_{s}\ll 1$) on larger scales.

Critical to our conclusions is the existence of fluctuations on all large
scales. A presence of a cut-off scale $k_{\rm IR}$ in the infrared, beyond
which there is no fluctuation at all, would simply eliminate the infrared
divergences or limit the sensitivity in the cosmological probes. Such cut-off
scales might naturally arise from the trans-Planckian conjecture [61, 62, 63].
While an inflationary expansion in the early Universe lasts for a finite
number of $e$-folding, it is somewhat odd to demand that the Universe is
perfectly homogeneous and isotropic without any fluctuations, even at the
level of $10^{-5}$ on scales larger than the largest scale with fluctuations
generated by inflation. Nonetheless, the status of such large scales is
governed by pre-inflationary physics and is highly uncertain.

Implications for cosmological models beyond the standard.— In the standard
$\Lambda$CDM model with general relativity, the adiabatic conditions are
assumed for the matter and radiation components presumably arising from the
decay of the inflaton field. With the adiabatic conditions, the conservation
of $\zeta$ in the infrared is guaranteed by the energy momentum conservation,
while the infrared conservation of $\mathcal{R}$ is guaranteed only by the
Einstein equations. The equivalence of $\mathcal{R}$ and $\zeta$ in the
infrared is, however, imposed only by the Einstein equations, independent of
the adiabatic conditions for the matter components. For the infrared
conservation of the tensor fluctuations $h_{\alpha\beta}$, the Einstein
equations are needed in conjunction with the absence of anisotropic pressure
in the infrared. It was shown [64] that this infrared insensitivity in the
cosmological probes holds non-linearly in $\Lambda$CDM models.

Our general conditions apply to any four-dimensional metric theory of gravity
with diffeomorphism symmetry. In such modified gravity theories, there exists
at least one additional degree of freedom (see, e.g., [65, 66] for recent
reviews), arising from higher-order derivatives in the equation of motion.
Despite these extra degrees of freedom, adiabatic conditions for matter and
radiation components can be assumed in the same way as in the $\Lambda$CDM
model, and the conservation of the energy-momentum tensor still leads to the
conservation of $\zeta$ in the infrared. The conservation of $\mathcal{R}$ in
the infrared is, on the other hand, dictated by the modified Einstein
equations and may be allowed. Unlike in general relativity, conservation of
both $\mathcal{R}$ and $\zeta$ may not result in their equivalence.
Consequently, our general conditions can constrain the space of physically
viable solutions in any gravity theories.

Furthermore, with extra degrees of freedom, the adiabaticity alone in the
matter sector would not be enough to meet the general conditions, as the extra
degrees of freedom will act as non-adiabatic fluctuations on large scales,
though a complete synchronization might be still possible. We note, however,
that any viable gravity theories often admit a solution that mimics general
relativity in some limit such as the Brans-Dicke model [67] and the Horndeski
theories [68]. If such solution exists on large scales, we expect that the
general conditions be met, but of course such solution is not the most
interesting one in any modified gravity theories.

Multi-field inflationary models [69, 70, 71, 72, 73, 74] or models with
cosmological defects (see, e.g., [75, 76] for review) often generate
isocurvature perturbations. The presence of isocurvature fluctuations on very
large scales would break the general conditions and lead to the infrared
sensitivity. Long-range non-gravitational forces would act in a similar way to
break the general conditions, and the cosmological probes would be modulated
by very long wave-length modes. These forces can arise from the presence of
primordial non-Gaussianity in the initial condition [77, 78], from an
interaction in the dark sector [79], or cosmological back-reactions [80, 81,
82, 83, 84, 85, 86]. However, the level of these non-adiabatic fluctuations is
already tightly constrained by CMB observations [87, 88], and hence the
infrared sensitivity from these models is expected to be negligible.

We acknowledge useful discussions with Robert Brandenberger, Ermis Mitsou, and
Marko Simonović. This work is support by the Swiss National Science Foundation
and a Consolidator Grant of the European Research Council.

## References

  * Stubbs et al. [2004] C. W. Stubbs, D. Sweeney, J. A. Tyson, and LSST Collaboration, in _American Astronomical Society Meeting Abstracts_ (2004), vol. 36 of _Bulletin of the American Astronomical Society_ , p. 108.02.
  * Dewdney et al. [2009] P. E. Dewdney, P. J. Hall, R. T. Schilizzi, and T. J. L. W. Lazio, IEEE Proceedings 97, 1482 (2009).
  * Laureijs et al. [2011] R. Laureijs, J. Amiaux, S. Arduini, J. . Auguères, J. Brinchmann, R. Cole, M. Cropper, C. Dabin, et al., ArXiv e-prints (2011), eprint 1110.3193.
  * Green et al. [2012] J. Green, P. Schechter, C. Baltay, R. Bean, D. Bennett, R. Brown, C. Conselice, M. Donahue, et al. (2012), eprint 1208.4012.
  * Levi et al. [2013] M. Levi, C. Bebek, T. Beers, R. Blum, R. Cahn, D. Eisenstein, B. Flaugher, K. Honscheid, R. Kron, O. Lahav, et al., ArXiv e-prints (2013), eprint 1308.0847.
  * Doré et al. [2014] O. Doré, J. Bock, M. Ashby, P. Capak, A. Cooray, R. de Putter, T. Eifler, N. Flagey, Y. Gong, S. Habib, et al., arXiv e-prints arXiv:1412.4872 (2014), eprint 1412.4872.
  * Percival et al. [2001] W. J. Percival, C. M. Baugh, J. Bland-Hawthorn, T. Bridges, R. Cannon, S. Cole, M. Colless, C. Collins, W. Couch, G. Dalton, et al., Mon. Not. R. Astron. Soc. 327, 1297 (2001), eprint astro-ph/0105252.
  * Eisenstein et al. [2005] D. J. Eisenstein, I. Zehavi, D. W. Hogg, R. Scoccimarro, et al., Astrophys. J. 633, 560 (2005), eprint arXiv:astro-ph/0501171.
  * Tegmark et al. [2006] M. Tegmark et al., Phys. Rev. D 74, 123507 (2006), eprint arXiv:astro-ph/0608632.
  * Komatsu et al. [2008] E. Komatsu, J. Dunkley, M. R. Nolta, C. L. Bennett, B. Gold, G. Hinshaw, N. Jarosik, D. Larson, M. Limon, L. Page, et al., ArXiv e-prints (2008), eprint 0803.0547.
  * Planck Collaboration et al. [2020a] Planck Collaboration, N. Aghanim, Y. Akrami, M. Ashdown, J. Aumont, C. Baccigalupi, M. Ballardini, A. J. Banday, R. B. Barreiro, N. Bartolo, et al., Astron. Astrophys. 641, A6 (2020a), eprint 1807.06209.
  * Sachs and Wolfe [1967] R. K. Sachs and A. M. Wolfe, Astrophys. J. 147, 73+ (1967).
  * Kaiser [1992] N. Kaiser, Astrophys. J. 388, 272 (1992).
  * Sasaki [1987] M. Sasaki, Mon. Not. R. Astron. Soc. 228, 653 (1987).
  * Futamase and Sasaki [1989] T. Futamase and M. Sasaki, Phys. Rev. D 40, 2502 (1989).
  * Bonvin et al. [2006a] C. Bonvin, R. Durrer, and M. A. Gasparini, Phys. Rev. D 73, 023523 (2006a), eprint arXiv:0511183.
  * Yoo et al. [2009] J. Yoo, A. L. Fitzpatrick, and M. Zaldarriaga, Phys. Rev. D 80, 083514 (2009), eprint arXiv:0907.0707.
  * Yoo [2010] J. Yoo, Phys. Rev. D 82, 083508 (2010), eprint arXiv:1009.3021.
  * Challinor and Lewis [2011] A. Challinor and A. Lewis, Phys. Rev. D 84, 043516 (2011), eprint arXiv:1105.5292.
  * Bonvin and Durrer [2011] C. Bonvin and R. Durrer, Phys. Rev. D 84, 063505 (2011), eprint arXiv:1105.5280.
  * Jeong et al. [2012] D. Jeong, F. Schmidt, and C. M. Hirata, Phys. Rev. D 85, 023504 (2012), eprint arXiv:1107.5427.
  * Ben-Dayan et al. [2014] I. Ben-Dayan, R. Durrer, G. Marozzi, and D. J. Schwarz, Phys. Rev. Lett. 112, 221301 (2014), eprint 1401.7973.
  * Umeh et al. [2014] O. Umeh, C. Clarkson, and R. Maartens, Classical and Quantum Gravity 31, 202001 (2014), eprint 1207.2109.
  * Yoo and Zaldarriaga [2014] J. Yoo and M. Zaldarriaga, Phys. Rev. D 90, 023513 (2014), eprint 1406.4140.
  * Bertacca et al. [2014] D. Bertacca, R. Maartens, and C. Clarkson, J. Cosmol. Astropart. Phys. 9, 037 (2014), eprint 1405.4403.
  * Di Dio et al. [2014] E. Di Dio, R. Durrer, G. Marozzi, and F. Montanari, J. Cosmol. Astropart. Phys. 12, 017 (2014), eprint 1407.0376.
  * Bonvin et al. [2015] C. Bonvin, C. Clarkson, R. Durrer, R. Maartens, and O. Umeh, J. Cosmol. Astropart. Phys. 6, 50 (2015), eprint 1503.07831.
  * Koyama et al. [2018] K. Koyama, O. Umeh, R. Maartens, and D. Bertacca, J. Cosmol. Astropart. Phys. 2018, 050 (2018), eprint 1805.09189.
  * Magi and Yoo [2022] M. Magi and J. Yoo, J. Cosmol. Astropart. Phys. 2022, 071 (2022), eprint 2204.01751.
  * Planck Collaboration et al. [2016] Planck Collaboration, P. A. R. Ade, N. Aghanim, M. Arnaud, M. Ashdown, J. Aumont, C. Baccigalupi, A. J. Banday, R. B. Barreiro, J. G. Bartlett, et al., Astron. Astrophys. 594, A13 (2016), eprint 1502.01589.
  * Hu and Kravtsov [2003] W. Hu and A. V. Kravtsov, Astrophys. J. 584, 702 (2003), eprint arXiv:0203169.
  * Scaccabarozzi et al. [2018] F. Scaccabarozzi, J. Yoo, and S. G. Biern, J. Cosmol. Astropart. Phys. 10, 024 (2018), eprint 1807.09796.
  * Grimm et al. [2020] N. Grimm, F. Scaccabarozzi, J. Yoo, S. G. Biern, and J.-O. Gong, J. Cosmol. Astropart. Phys. 2020, 064 (2020), eprint 2005.06484.
  * Castorina and Di Dio [2022] E. Castorina and E. Di Dio, J. Cosmol. Astropart. Phys. 2022, 061 (2022), eprint 2106.08857.
  * Biern and Yoo [2017a] S. G. Biern and J. Yoo, J. Cosmol. Astropart. Phys. 4, 045 (2017a), eprint 1606.01910.
  * Biern and Yoo [2017b] S. G. Biern and J. Yoo, J. Cosmol. Astropart. Phys. 026 (2017b), eprint 1704.07380.
  * Baumgartner and Yoo [2021] S. Baumgartner and J. Yoo, Phys. Rev. D 103, 063516 (2021), eprint 2012.03968.
  * Mellier [1999] Y. Mellier, Ann. Rev. Astron. Astrophys. 37, 127 (1999), eprint arXiv:9812172.
  * Bartelmann and Schneider [2001] M. Bartelmann and P. Schneider, Phys. Rep. 340, 291 (2001), eprint arXiv:9912508.
  * Refregier [2003] A. Refregier, Ann. Rev. Astron. Astrophys. 41, 645 (2003), eprint arXiv:0307212.
  * Heavens [2003] A. Heavens, Mon. Not. R. Astron. Soc. 343, 1327 (2003), eprint arXiv:0304151.
  * Munshi et al. [2008] D. Munshi, P. Valageas, L. van Waerbeke, and A. Heavens, Phys. Rep. 462, 67 (2008), eprint astro-ph/0612667.
  * Yoo [2009] J. Yoo, Phys. Rev. D 79, 023517 (2009), eprint arXiv:0808.3138.
  * Yoo [2014] J. Yoo, Class. Quant. Grav. 31, 234001 (2014), eprint arXiv:1409.3223.
  * Yoo et al. [2018] J. Yoo, N. Grimm, E. Mitsou, A. Amara, and A. Refregier, J. Cosmol. Astropart. Phys. 4, 029 (2018), eprint 1802.03403.
  * Yoo and Durrer [2017] J. Yoo and R. Durrer, J. Cosmol. Astropart. Phys. 9, 016 (2017), eprint 1705.05839.
  * Mitsou and Yoo [2020] E. Mitsou and J. Yoo, Springer Briefs in Physics arXiv:1908.10757 (2020), eprint 1908.10757.
  * Bonvin et al. [2006b] C. Bonvin, R. Durrer, and M. Kunz, Phys. Rev. Lett. 96, 191302 (2006b), eprint astro-ph/0603240.
  * Ben-Dayan et al. [2012] I. Ben-Dayan, M. Gasperini, G. Marozzi, F. Nugier, and G. Veneziano, J. Cosmol. Astropart. Phys. 4, 036 (2012), eprint 1202.1247.
  * Yoo and Scaccabarozzi [2016] J. Yoo and F. Scaccabarozzi, J. Cosmol. Astropart. Phys. 9, 046 (2016), eprint 1606.08453.
  * Bruni et al. [2012] M. Bruni, R. Crittenden, K. Koyama, R. Maartens, C. Pitrou, and D. Wands, Phys. Rev. D 85, 041301 (2012), eprint arXiv:1106.3999.
  * Maartens et al. [2020] R. Maartens, S. Jolicoeur, O. Umeh, E. M. De Weerd, C. Clarkson, and S. Camera, J. Cosmol. Astropart. Phys. 2020, 065 (2020), eprint 1911.02398.
  * Press and Schechter [1974] W. H. Press and P. Schechter, Astrophys. J. 187, 425 (1974).
  * Kaiser [1984] N. Kaiser, Astrophys. J. Lett. 284, L9 (1984).
  * Politzer and Wise [1984] H. D. Politzer and M. B. Wise, Astrophys. J. Lett. 285, L1 (1984).
  * Bardeen et al. [1986] J. M. Bardeen, J. R. Bond, N. Kaiser, and A. S. Szalay, Astrophys. J. 304, 15 (1986).
  * Bond et al. [1991] J. R. Bond, S. Cole, G. Efstathiou, and N. Kaiser, Astrophys. J. 379, 440 (1991).
  * Zibin and Scott [2008] J. P. Zibin and D. Scott, Phys. Rev. D 78, 123529 (2008), eprint 0808.2047.
  * Yoo et al. [2019a] J. Yoo, E. Mitsou, Y. Dirian, and R. Durrer, Phys. Rev. D 100, 063510 (2019a), eprint 1905.09288.
  * Yoo et al. [2019b] J. Yoo, E. Mitsou, N. Grimm, R. Durrer, and A. Refregier, J. Cosmol. Astropart. Phys. 2019, 015 (2019b), eprint 1905.08262.
  * Brandenberger and Martin [2013] R. H. Brandenberger and J. Martin, Classical and Quantum Gravity 30, 113001 (2013), eprint 1211.6753.
  * Bedroya et al. [2020] A. Bedroya, R. Brandenberger, M. Loverde, and C. Vafa, Phys. Rev. D 101, 103502 (2020), eprint 1909.11106.
  * Blamart et al. [2023] M. Blamart, S. Laliberte, and R. Brandenberger, arXiv e-prints arXiv:2301.02741 (2023), eprint 2301.02741.
  * Mitsou et al. [2023] E. Mitsou, J. Yoo, and M. Magi, arXiv e-prints arXiv:2302.00427 (2023), eprint 2302.00427.
  * De Felice and Tsujikawa [2010] A. De Felice and S. Tsujikawa, Living Reviews in Relativity 13, 3 (2010), eprint 1002.4928.
  * Clifton et al. [2012] T. Clifton, P. G. Ferreira, A. Padilla, and C. Skordis, Phys. Rep. 513, 1 (2012), eprint 1106.2476.
  * Brans and Dicke [1961] C. Brans and R. H. Dicke, Physical Review 124, 925 (1961).
  * Horndeski [1974] G. W. Horndeski, International Journal of Theoretical Physics 10, 363 (1974).
  * Mollerach [1990] S. Mollerach, Phys. Rev. D 42, 313 (1990).
  * Polarski and Starobinsky [1992] D. Polarski and A. A. Starobinsky, Nuclear Physics B 385, 623 (1992).
  * Bartolo et al. [2001] N. Bartolo, S. Matarrese, and A. Riotto, Phys. Rev. D 64, 123504 (2001), eprint astro-ph/0107502.
  * Byrnes and Wands [2006] C. T. Byrnes and D. Wands, Phys. Rev. D 74, 043529 (2006), eprint astro-ph/0605679.
  * Gordon et al. [2000] C. Gordon, D. Wands, B. A. Bassett, and R. Maartens, Phys. Rev. D 63, 023506 (2000), eprint astro-ph/0009131.
  * Peterson and Tegmark [2011] C. M. Peterson and M. Tegmark, Phys. Rev. D 83, 023522 (2011), eprint 1005.4056.
  * Brandenberger [1994] R. H. Brandenberger, International Journal of Modern Physics A 9, 2117 (1994), eprint astro-ph/9310041.
  * Brandenberger [2014] R. H. Brandenberger, Nuclear Physics B Proceedings Supplements 246, 45 (2014), eprint 1301.2856.
  * Komatsu and Spergel [2001] E. Komatsu and D. N. Spergel, Phys. Rev. D 63, 063002 (2001), eprint astro-ph/0005036.
  * Bartolo et al. [2004] N. Bartolo, E. Komatsu, S. Matarrese, and A. Riotto, Phys. Rep. 402, 103 (2004), eprint arXiv:0406398.
  * Archidiacono et al. [2022] M. Archidiacono, E. Castorina, D. Redigolo, and E. Salvioni, J. Cosmol. Astropart. Phys. 2022, 074 (2022), eprint 2204.08484.
  * Buchert and Ehlers [1997] T. Buchert and J. Ehlers, Astron. Astrophys. 320, 1 (1997), eprint astro-ph/9510056.
  * Abramo et al. [1997] L. R. W. Abramo, R. H. Brandenberger, and V. F. Mukhanov, Phys. Rev. D 56, 3248 (1997), eprint arXiv:9704037.
  * Buchert [2000] T. Buchert, General Relativity and Gravitation 32, 105 (2000), eprint gr-qc/9906015.
  * Geshnizjani and Brandenberger [2002] G. Geshnizjani and R. Brandenberger, Phys. Rev. D 66, 123507 (2002), eprint gr-qc/0204074.
  * Clarkson et al. [2011] C. Clarkson, G. Ellis, J. Larena, and O. Umeh, Reports on Progress in Physics 74, 112901 (2011), eprint 1109.2314.
  * Marozzi et al. [2013] G. Marozzi, G. P. Vacca, and R. H. Brandenberger, J. Cosmol. Astropart. Phys. 2, 027 (2013), eprint 1212.6029.
  * Comeau and Brandenberger [2023] V. Comeau and R. Brandenberger, arXiv e-prints arXiv:2302.05873 (2023), eprint 2302.05873.
  * Planck Collaboration et al. [2019] Planck Collaboration et al., arXiv e-prints arXiv:1905.05697 (2019), eprint 1905.05697.
  * Planck Collaboration et al. [2020b] Planck Collaboration et al., Astron. Astrophys. 641, A10 (2020b), eprint 1807.06211.