,

# Blinded by Generated Contexts: How Language Models
Merge Generated and Retrieved Contexts for Open-Domain QA?

Hexiang Tan♠♡, Fei Sun♠222Corresponding author. , Wanli Yang♠♢, Yuanzhuo
Wang♠, Qi Cao♠, Xueqi Cheng♠♡
♠CAS Key Laboratory of AI Safety & Security,
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
♡University of Chinese Academy of Sciences, Beijing, China
♢Nankai University, Tianjin, China
{tanhexiang21s, sunfei, wangyuanzhuo, caoqi<EMAIL_ADDRESS>
###### Abstract

While auxiliary information has become a key to enhance Large Language Models
(LLMs), relatively little is known about how well LLMs merge these contexts,
specifically generated and retrieved. To study this, we formulate a task
specifically designed to identify whether the answers, derived from the
integration of generated and retrieved contexts, are attributed to either
generated or retrieved contexts. To support this task, we develop a
methodology to construct datasets with conflicting contexts, where each
question is paired with both generated and retrieved contexts, yet only one of
them contains the correct answer. Our experiments reveal a significant bias in
LLMs towards generated contexts, as evidenced across state-of-the-art open
(Llama2-7b/13b) and closed (GPT 3.5/4) systems. We further identify two key
factors contributing to this bias: i) Contexts generated by LLMs typically
show greater similarity to the questions, increasing their likelihood of
selection; ii) The segmentation process used in retrieved contexts disrupts
their completeness, thereby hindering their full utilization in LLMs. Our
analysis enhances the understanding of how LLMs merge diverse contexts,
offering valuable insights for advancing current augmentation methods for
LLMs.

##  1 Introduction

Recent advancements in augmenting Large Language Models (LLMs) with auxiliary
information have significantly revolutionized their efficacy in knowledge-
intensive tasks Chang et al. (2023); Ram et al. (2023). In this evolving
landscape, existing works can be broadly categorized into two groups based on
information sources: generation-augmented and retrieval-augmented approaches.
To effectively harness the LLMs’ internal knowledge, generation-augmented
approaches Liu et al. (2022); Sun et al. (2023), e.g., GenRead Yu et al.
(2022), instruct LLMs to initially generate a background context tailored to
the given question, which is then employed as the basis for producing the
final answer. In contrast, retrieval-augmented approaches Lewis et al. (2020);
Ram et al. (2023) adopt an alternative strategy by incorporating relevant
passages from external corpora, e.g., Wikipedia, as context, thereby notably
enhancing LLMs’ capability to handle situations like knowledge updates Jang et
al. (2022) and long-tail knowledge Kandpal et al. (2023).

Figure 1: Instances correctly answered using solely the retrieved context in
the NQ dataset are represented in cyan. The subset of these instances that can
also be resolved by a hybrid approach is indicated in orange.

Building on the foundations laid by generation-augmented and retrieval-
augmented methods, recent hybrid approaches have attempted to integrate them
to further improve performance in tasks such as Question Answering (QA) Yu et
al. (2022); Abdallah and Jatowt (2023); Mallen et al. (2023). These hybrid
approaches face a significant challenge: conflicts between diverse sources can
impede the effectiveness of information integration Zhang et al. (2023). While
recent works have investigated conflicts within a single source of contexts
(Xie et al., 2023; Chen et al., 2022), there remains a gap in the
comprehension of how LLMs resolve conflicts between generated and retrieved
contexts. This study, therefore, aims to investigate the underlying mechanisms
by which LLMs process the two types of contexts, especially when they contain
conflicting information.

Our investigation was driven by a striking observation: in certain cases,
models relying solely on retrieval contexts succeeded, whereas, counter-
intuitively, hybrid approaches failed, as depicted in Figure 1. To uncover the
underlying reasons, we proposed a systematic framework to dissect the process
by which LLMs merge generated and retrieved contexts. For this purpose, we
curated tailored datasets in which each question is accompanied by two
incompatible contexts, i.e., they are not only inconsistent but, crucially,
only one contains the correct answer. These datasets were derived from
existing QA datasets, with questions extracted based on the aforementioned
criterion. Finally, the datasets, contrasting generated and retrieved
contexts, provide an innovative lens to unravel how LLMs discern and integrate
diverse contextual information in QA.

In this paper, we conducted a series of controlled experiments using our
uniquely designed datasets to empirically study this question, focusing on
several state-of-the-art open (Llama2-7b/13b) and closed (GPT 3.5/4) LLMs.
Surprisingly, our findings reveal a pronounced bias in LLMs to favor generated
contexts, regardless of whether the generated text was produced internally or
by other LLMs, which persists even when the generated contexts offer incorrect
information while the retrieval contexts hold the correct answer. Notably,
this tendency remains consistent even after accounting for potentially
confounding factors including the order and length of contexts, and the
confirmation bias (Xie et al., 2023). These findings highlight a critical
challenge for existing LLMs in effectively merging internal parametric
knowledge (i.e., generated contexts) and external information (i.e., retrieved
contexts), under increasingly common non-tunable settings, e.g., those
involving black-box APIs like GPT-4.

To delve deeper into the factors driving this phenomenon, we initially
conducted a comprehensive analysis examining the text similarity between the
contexts fed into LLMs and their corresponding questions (Section 5.2). We
discovered that generated contexts typically exhibit a higher degree of
similarity to the questions compared to retrieved contexts, even when they
contain incorrect information. Furthermore, our analysis reveals that samples
with smaller similarity gaps between generated and retrieved contexts
exhibited a reduced bias, although a noticeable preference for generated
contexts remains evident. This finding indicates text similarity as a
significant factor in this bias, and emphasizes the the need for caution with
LLM-generated contexts, to avoid being misled by highly relevant but
inaccurate information.

Besides the similarity factor, we further examined the role of semantic
completeness in the bias observed in LLMs (Section 5.3). This area of
exploration was driven by another key difference between these two types of
contexts: retrieved contexts often exhibit semantic incompleteness due to the
segmentation process, a prevalent practice in current retrieval methodologies.
By intentionally truncating the generated context to mimic the incompleteness
typically in retrieved contexts, we discerned a significant decrease in the
bias towards generated contexts. This finding suggests that LLMs exhibit
significant sensitivity to the semantic completeness of input contexts,
calling for improvements in passage segmentation strategies in current
retrieval-augmented systems.

Our analysis provides a better understanding of how LLMs use contexts for QA
tasks. This work represents a preliminary exploration, aimed at highlighting
the critical issue of context utilization in existing LLMs, a concern that
grows increasingly as LLM-generated content, including potential
misinformation, proliferates across the web. Furthermore, our findings offer
valuable guidance for enhancing existing retrieval-augmented methods, such as
optimizing passage segmentation in retrieval systems. Our main contributions
can be summarized as:

  * •

We uncover a critical bias in existing LLMs, where they heavily rely on
generated contexts regardless of correctness, indicating an insufficient use
of diverse information sources.

  * •

To facilitate controlled experiments, we develop a specialized process for
constructing tailored datasets.

  * •

Our extensive analyses have identified two key factors, i.e., text similarity
and semantic completeness, in the context utilization of LLMs.

##  2 Preliminary & Task Formulation

(a) Retrieval-Augmented Approach

(b) Generation-Augmented Approach

(c) Hybrid Approach

Figure 2: The frameworks of retrieval-augmented approach, generation-augmented
approach and hybrid approach.

In this section, we briefly review three categories of LLMs augmented with
auxiliary information for QA tasks: retrieval-augmented, generation-augmented,
and hybrid approaches. Additionally, we introduce the framework of our
investigation. Figure 2 presents high-level abstract frameworks for three
typical types of QA systems, each centered around an LLM as the reader
component, and potentially incorporating additional components like a
retriever, generator, or a blend of both, tailored to the specific
methodology.

Retrieval-Augmented Approach. As shown in fig. 2, for a given question
$q$111Here, we omit the subscript $i$ for notational simplicity. in a set of
questions $\mathbb{Q}$, these approaches initially use a retrieval model
$\gamma$ to select the top $k$ relevant documents
$D_{k}^{\gamma}{=}\gamma_{k}(q,\mathbb{C}){=}\\{d_{1}^{\gamma},\dots,d_{k}^{\gamma}\\}$
from a corpus $\mathbb{C}{=}\\{d_{1},\dots,d_{|\mathbb{C}|}\\}$. Subsequently,
a reader (often LLM) $\phi$ employs these documents $D^{\gamma}$ to generate
an answer $a_{\phi}^{\gamma}$, expressed as
$a_{\phi}^{\gamma}=\phi(q,D^{\gamma})$.

Generation-Augmented Approach. In contrast, as illustrated in fig. 2, these
works involve an LLM as a generator $\varrho$ to produce $k$ tailored
background contexts
$D_{k}^{\varrho}{=}\varrho_{k}(q){=}\\{d_{1}^{\varrho},\dots,d_{k}^{\varrho}\\}$
for a give question $q$, thereby enhancing the utilization of the LLM’s
internal knowledge. These LLM-generated contexts $D^{\varrho}$ form the input
for reader $\phi$ to produce the final answer:
$a_{\phi}^{\varrho}=\phi(q,D^{\varrho})$. These methodologies generally use
LLMs for the reader and generator functions, with implementation options
including a shared LLM with varied prompts (Wei et al., 2022b; Sun et al.,
2023), or distinct LLMs dedicated to each role (Luo et al., 2023; Feng et al.,
2023).

Hybrid Approach. They, as depicted in fig. 2, combine retrieved and generated
contexts to enhance performance Yu et al. (2022); Abdallah and Jatowt (2023),
as $a_{\phi}=\phi(q,D_{k}^{\gamma},D_{k}^{\varrho})$. Building upon the
foundations of previous approaches, hybrid approaches primarily focus on how
to effectively merge these two types of information to enhance overall
performance Zhang et al. (2023).

###  2.1 Answer Tracing

Departing from previous research, our study delves into the mechanisms by
which LLMs merge contexts from diverse sources, i.e., hybrid approach setting.
We specifically focus on non-tunable, i.e., zero-shot setting, LLMs acting as
the reader and generator, reflecting prevalent real-world use cases like
ChatGPT. This research direction is motivated by several considerations:  i)
Model architecture: LLMs represent an emerging trend for the future, in
contrast to the specially designed reader models like Fusion-in-Decoder (FiD)
(Izacard and Grave, 2021).  ii) Non-tunability: Due to the high cost and
limited accessibility of fine-tuning, the direct use of black-box non-tunable
LLMs has gained popularity in real-world applications.  iii) Impact: Given the
extensive use of LLMs, any bias or issue in their merging mechanisms could
lead to serious consequences, like pulling an LLM that prefers generated
context with LLM-generated misinformation (Pan et al., 2023).

Figure 3: The task to study LLMs’ merging mechanisms by tracing the sources of
the answers.

More specially, to reveal if an LLM has a bias towards generated contexts, we
design a task to ascertain whether an LLM-generated answer $a_{\phi}$
originates from generated contexts $D_{k}^{\varrho}$ or retrieved contexts
$D_{k}^{\gamma}$. As illustrated in Figure 3, our task mirrors the basic
process of hybrid approaches,while our focus is studying LLMs’ merging
mechanisms by tracing the sources of the answers instead of improving QA
performance. For a more controlled and simpler analysis, we limit the context
to a single instance from each source, i.e., $k{=}1$ and
$a_{\phi}{=}\phi(q,d^{\gamma}_{1},d_{1}^{\varrho})$. Subsequently, by
comparing the answer $a_{\phi}$ with answers derived from the retrieved
context $a_{\phi}^{\gamma}$ and the generated context $a_{\phi}^{\varrho}$, we
can determine its source222Here, we use a stricter criterion by assessing the
answer’s consistency with those generated from individual sources, rather than
directly comparing it with the input contexts., thereby analyzing the merging
mechanism of LLMs.

##  3 Experimental Setup

To facilitate our investigation into how LLMs merge generated and retrieved
contexts, this section elaborates on the construction of our context-
conflicting datasets and the evaluation metric.

###  3.1 Context-Conflicting Datasets

Figure 4: The framework of constructing context-conflicting datasets.

As depicted in Figure 4, in our dataset $\mathcal{D}_{\mathrm{cc}}$, each
sample $x$ is a quintet
$(q,d_{1}^{\gamma},d_{1}^{\varrho},a_{\phi}^{\gamma},a_{\phi}^{\varrho})$,
where $d_{1}^{\gamma}$ is the context returned by retriever $\gamma$ for
question $q$, $d_{1}^{\varrho}$ represents the context generated by LLM
$\varrho$, $a_{\phi}^{\gamma}$ and $a_{\phi}^{\varrho}$ are the candidate
answers provided by the reader $\phi$, each based solely on the respective
contexts $d_{1}^{\gamma}$ and $d_{1}^{\varrho}$. To guarantee that our dataset
is suitable for controlled experiments aimed at investigating the merging
mechanisms of LLMs, it must adhere to specific criteria:

  * •

Traceability: Both $a_{\phi}^{\gamma}$ and $a_{\phi}^{\varrho}$ must be solely
supported by their respective contexts, i.e., $d_{1}^{\gamma}$ and
$d_{1}^{\varrho}$.

  * •

Exclusivity: Only one of the contexts, $d_{1}^{\gamma}$ or $d_{1}^{\varrho}$
provides the correct answer, i.e., either $a_{\phi}^{\gamma}$ or
$a_{\phi}^{\varrho}$ matches the gold answer of question $q$.

Such constraints establish a solid basis to identify which context, generated
or retrieved, is selected by LLMs to produce answers in hybrid approaches.

Considering the specified prerequisites and cost considerations, we utilize
two open-domain QA benchmark datasets with golden answers, i.e.,
NaturalQuestions (NQ) Kwiatkowski et al. (2019) and TriviaQA (TQA) Joshi et
al. (2017), to assemble our experimental datasets. The overall pipeline for
dataset construction is depicted in Figure 4, with detailed steps outlined as
follows:

Context Preparation. Step 1 in Figure 4 illustrates the process of preparing
retrieved and generated contexts for each question.

For retrieved context, it is obtained from the top-$1$ ranked passage from
Wikipedia using a retrieval model, e.g., Contriever (Izacard et al., 2021), a
powerful off-the-shelf retrieval model that is extensively employed in various
retrieval-augmented generation systems Shi et al. (2023); Ram et al. (2023).

For generated context, we follow the GenRead Yu et al. (2022), instructing the
generator, i.e., LLM like GPT 4, with the prompt:  Generate a background
context from Wikipedia to answer the given question {#question} However, the
initial prompt often yields contexts much longer (${>}250$ words) than the
retrieved contexts (typically truncated to ${\sim}100$ words (Karpukhin et
al., 2020; Izacard et al., 2021)). The discrepancy in length could potentially
affect the merging mechanisms of LLMs, e.g., LLMs may tend to longer context,
as discussed in Xie et al. (2023). More detail can be found in Appendix A.1.
To mitigate this, we further append  Keep the length of the document around
$n$ words to the initial prompt to regulate the length of the generated
context333To precisely control the length, we vary the word count $n$ in the
prompt, and choose the results that best match the length of the retrieved
context..

Dataset Retrieved Generated GPT 4 GPT 3.5 Llama2-13b Llama2-7b NQ 107.3 108.0
106.0 110.1 104.0 TQA 106.3 107.2 104.9 105.5 102.6

Table 1: Average lengths of the generated and retrieved contexts. Length is
measured in the number of words after punctuation removal.

Table 1 confirms that the lengths of the final generated and retrieved
contexts are closely matched, with an average length discrepancy below $3$%.
All subsequent experiments and analyses, unless otherwise specified, employ
this method to eliminate the impact of length variations.

Sample Filtering for Traceability. With each question paired with a context
(either generated or retrieved) established in the initial stage, the reader
generates the answer using the following prompt:  Refer to the context below
and answer the following question with just one entity. context: {#contexts}
question: {#question} The answer is To unravel the mechanisms of LLMs in
context merging, it is essential to ensure the traceability, i.e., the output
answer is derived from the input context, rather than the intrinsic parametric
knowledge of the LLMs. To achieve this, we only keep samples in which both the
generated and retrieved contexts exactly include their respective generated
answers444Samples yielding responses like “unknown” that reflect the model
evading answering are also excluded from our dataset., exemplified by
$a_{\phi}^{\gamma}{\in}d_{1}^{\gamma}$, where $\in$ denotes $d_{1}^{\gamma}$
contains the substring $a_{\phi}^{\gamma}$. This practice is grounded in the
findings of Chen et al. (2022); Xie et al. (2023), which demonstrate that in
the presence of external context, LLMs tend to rely on external context rather
than their intrinsic parametric knowledge.

Building Context-Conflicting Dataset. Having obtained answers for each type of
context, we are now positioned to construct our context-conflicting (CC)
datasets, as depicted in Step 3 of Figure 4. Initially, We employ the exact
match metric Yu et al. (2022) to evaluate the correctness of candidate answers
derived from contexts, considering an answer correct if its normalized form
matches any of the golden answers. Subsequently, the conflicting context
datasets are composed of samples for which only one of the two types of
contexts, either generated or retrieved, yields the correct answer. Notably,
each dataset comprises two distinct subsets: AIG, consisting of samples with
correct answers in the generated context; and AIR comprising samples with
correct answers in the retrieved context.

###  3.2 Statistics of Datasets

Generator &Reader NQ (3610) TQA (11,313) NQ-AIG NQ-AIR TQA-AIG TQA-AIR GPT 4
331 238 946 358 GPT 3.5 379 279 1357 567 Llama2-13b 439 419 1648 1201
Llama2-7b 432 406 1737 1488 Avg. Prop. 10.9% 9.3% 12.6% 8.0%

Table 2: Dataset statistics across LLMs, “Avg. Prop.” shows average
proportions of subsets to original testsets.

For each reader-generator pair, we respectively constructs context-conflicting
datasets from the original NQ and TQA test sets, named NQ-CC (NQ-AIG + NQ-AIR)
and TQA-CC (TQA-AIG + TQA-AIR).

We initially adopt a typical and simple setting in which a singular LLM serves
as both the generator and reader. Table 2 provides detailed statistics for the
constructed subsets corresponding to various LLMs, including GPT 4
(gpt-4-0613), GPT 3.5 (gpt-3.5-turbo-0613), Llama2-7b/13b (Llama-2-7b/13b-chat
(Touvron et al., 2023)). The statistics reveal that context-conflicting
subsets constitute a significant portion of the overall test sets. This
observation indicates that numerous questions can be answered by only one type
of contexts, thus emphasizing the importance of exploring how LLMs merge such
varied contexts. Moreover, as shown in Table 2, GPT 4 has fewer conflicting
instances than other LLMs, because of its higher efficacy in answering
questions using either solely retrieved or generated contexts.

In Section 4.2, we also explore a more complex scenario in which the generator
and reader are distinct LLMs. Appendix A.2 provides detailed statistics for
the context-conflicting datasets created for each reader-generator pair, where
we observe similar conclusions .

###  3.3 Evaluation Metric

Besides datasets, we also develop metrics to study how LLMs merge generated
and retrieved contexts in hybrid approaches. Specifically, the selection of
LLMs towards either generated or retrieved context can be measured by the
proportion of answers that exactly match the answer produced solely by the
corresponding context, denoted as

$\rho_{\text{gen}}{=}\text{avg}(\mathtt{em}(a_{\phi},a_{\phi}^{\varrho})),\quad\rho_{\text{ret}}=\text{avg}(\mathtt{em}(a_{\phi},a_{\phi}^{\gamma}))$

where $\mathtt{em}(a,b)$ returns 1 if $a$ exactly match $b$, and 0 otherwise.
Thus, for a balanced dataset, an $\rho_{\text{gen}}>\rho_{\text{ret}}$
indicates a preference of LLMs for generated contexts over retrieved contexts,
and vice versa. To facilitate a simple and efficient experiment, we define a
synthesized metric as follows:

$\mathrm{DiffGR}=\frac{\rho_{\text{gen}}-\rho_{\text{ret}}}{\rho_{\text{gen}}+\rho_{\text{ret}}}$
(1)

The metric $\mathrm{DiffGR}$, ranging from $[-1,1]$, quantifies the extent of
LLMs’ preference for choosing generated contexts over retrieved contexts.
Using AIR as an example, where all correct answers come from retrieved
contexts, an ideal $\mathrm{DiffGR}$ value should be $-1$, and
$\mathrm{DiffGR}{>}0$ would indicate a significant bias towards generated
contexts.

##  4 How LLMs Merge Contexts?

In this section, we conduct experiments on the proposed context-conflicting
datasets to investigate the merging mechanism of the LLMs in hybrid approaches
in different settings. We first consider a typical setting where generator and
reader share a single LLM, to explore how LLMs merge retrieved and self-
generated contexts (Section 4.1). Then, we extend our experiments to more
flexible combinations of generator and reader (Section 4.2 ) to investigate
their effects.

###  4.1 LLMs Prefer Self-Generated Contexts

Our preliminary experiments, in which a single LLM serves both as generator
and reader, are designed to explore how LLMs integrate information from
retrieved and self-generated contexts. The LLMs under evaluation are tasked
with answering questions using both types of contexts on their corresponding
context-conflicting subsets, where either self-generated or retrieved contexts
provide correct answers. In all experiments, we employ a randomized input
sequence of contexts to mitigate the influence of order, which is further
discussed in Appendix A.3.

We begin our analysis by examining LLMs’ QA performance on context-conflicting
datasets to reveal how well can LLMs utilize both types of contexts. Table 3
presents the Exact Match (EM %) percentages, a typical QA metric, across
various LLMs. Surprisingly, LLMs demonstrate significantly low performance
($\leq 23.40\%$) on AIR subsets, despite the fact that the retrieved context
alone consistently yields the correct answer on these subsets. In contrast,
most LLMs exhibit strong performance on AIG subsets (near $90\%$ or more). An
exception is the Llama2-7B model, which performs relatively poorly, likely due
to its inferior instruction-following capabilities. Overall, all LLMs exhibit
a significant performance gap between AIR and AIG datasets, with a pronounced
decline in performance when the correct answers come from retrieved contexts.

Generator &Reader NQ-CC TQA-CC NQ-AIG NQ-AIR TQA-AIG TQA-AIR GPT 4 91.54 14.29
94.08 18.44 GPT 3.5 94.46 13.26 93.96 19.58 Llama2-13b 89.52 17.66 91.81 20.23
Llama2-7b 67.13 23.40 80.08 21.64

Table 3: The Exact Match (EM) scores (%) of hybrid approaches on corresponding
context-conflicting datasets. Figure 5: The $\mathrm{DiffGR}$ of LLMs on their
corresponding context-conflicting datasets.

To further reveal LLMs’ behavior underlying the QA performance, we trace the
source contexts of LLMs’ answers using the proposed $\mathrm{DiffGR}$ metric.
Figure 5 illustrates the $\mathrm{DiffGR}$ of LLMs on their corresponding
context-conflicting subsets. An ideal LLM that optimally leverages information
from both sides should always rely on retrieved contexts in case of AIR (with
$\mathrm{DiffGR=-1}$), and on generated contexts for AIG (with
$\mathrm{DiffGR=1}$). Contrary to expectations, Figure 5 illustrates that LLMs
fail to identify the correct information and consistently tend to rely on
generated contexts on both AIG and AIR subsets. This result indicates a
pronounced bias in LLMs to favor generated contexts, even when they provide
incorrect information. This bias leads to the insufficient utilization of
retrieved contexts mentioned above, and highlights a critical challenge for
existing LLMs in effectively merging generated and retrieved contexts.

As the bias on AIR subsets has more direct impact on the performance, the
following experiments and analysis will focus on the biases on these subsets
to conserve space. Results on the AIG subsets can be found in Appendix A.4.

###  4.2 LLMs Broadly Prefer Generated Contexts

Figure 6: $\mathrm{DiffGR}$ (%) with different (reader, generator) pairs on
their corresponding NQ-AIR datasets.

Above experiments reveal the bias in LLMs to favor the self-generated context.
Consequently, an intriguing question emerges: Are LLMs biased exclusively
towards self-generated contexts, or do they exhibit a similar preference for
contexts generated by other LLMs? To investigate this question, this section
extends the experiments to more flexible combinations of generators and
readers. We construct context-conflicting datasets for each (generator,
reader) pair and show the detail statistics of the constructed datasets in
Appendix A.2. Base on these datasets, we then compute $\mathrm{DiffGR}$ metric
to examine biases across various (generator, reader) pairs , as shown in
Figure 6. The results yield two notable conclusions and insights:

LLMs are also biased towards contexts generated by other LLMs. Figure 6
indicates that the bias towards generated context persists across various
combinations of readers and generators. This suggests that such bias in LLMs
is widespread and not limited to self-generated context.

LLMs exhibit a stronger preference for contexts generated by themselves.
Figure 6 also illustrates that readers exhibit a strongest preference when the
generator LLM is identical to themselves. The sole exception is Llama2-7b,
which shows the strongest preference when paired with Llama2-13b as its
generator. This phenomenon likely results from their highly similar model
structures and training processes (Touvron et al., 2023).

##  5 Why LLMs Prefer Generated Contexts

In this section, we investigate why LLMs prefer generated contexts rather than
retrieved contexts, from several perspectives: confirmation bias (Xie et al.,
2023) in Section 5.1, context similarity to the question in Section 5.2, and
context completeness in Section 5.3.

###  5.1 Effect of Confirmation Bias

Recent work (Xie et al., 2023) has uncovered a confirmation bias in LLMs,
which leads them to prefer the context consistent with their parametric
knowledge (or parametric memory), when presented with two conflicting
generated contexts. This finding naturally raises the question: does the
inherent consistency of generated contexts with parametric knowledge lead to
the observed preference for these contexts? To investigate this question, we
designed controlled experiments that manipulate the consistency of the
generated context with the LLMs’ parametric knowledge.

In this section, we adopt the setting where the reader and generator share a
single LLM555When the generator and reader are distinct LLMs, generated
contexts may not align with the reader’s parameterized knowledge. In this
setting, LLMs consistently exhibits a significant preference for generated
contexts, as observed in Section 4.2, indicating that confirmation bias is not
the primary factor., resulting in an inherent consistency between the
naturally generated context $d^{\varrho}_{1}$ and the parametric knowledge of
LLMs. In this setting, we can refer to $a^{\varrho}_{\phi}$, the answer
derived from $d^{\varrho}_{1}$, as the memory answer (the answer consistent
with parametric knowledge (Xie et al., 2023)). To explore the effect of the
confirmation bias, we enforce LLMs to generate a counter-memory context
$d^{\varrho^{\prime}}_{1}$ that supports a counter-memory answer
$a^{\varrho^{\prime}}_{1}$, which is completely different with the memory
answer. Specifically, we construct counter-memory answers and contexts based
on the original AIR context-conflicting subsets of GPT 3.5, as outlined below:

Counter-Memory Answers Preparation. For each question on the AIR subsets, we
substitute the original memory answer (e.g., “Canada”) with a same-type yet
distinct entity (e.g., “United States”), which serves as the counter-memory
answer, as detailed in Section A.5. Furthermore, to facilitate the calculation
of $\mathrm{DiffGR}$, we only retain those samples where the counter-memory
answer also diverged from $a_{\phi}^{\gamma}$, the answer provided by the
retrieved context. Formally, the counter-memory answer satisfies
$a_{\phi}^{\varrho^{\prime}}\neq a_{\phi}^{\varrho}$ and
$a_{\phi}^{\varrho^{\prime}}\neq a_{\phi}^{\gamma}$.

Counter-Memory Contexts Generation. We enforce LLMs to generate a counter-
memory context that supports the counter-memory answer, with a similar prompt
discussed in Pan et al. (2023):

Generate a background document in support of the given opinion to the
question. Keep the length of the document around $n$ words. Question:
{question} Opinion: {counter memory answer} Document:

Following Section 3, the prompt also incorporates a length constraint to
mitigate the influence of length, as evidenced by the length statistics
presented in Table 4.

Answer Consistency Checking. To verify that the counter-memory context
actually supports the counter-memory answer, we retain only those instances
where the predicted answer, derived exclusively from the counter-memory
context, exactly match the counter-memory answer:

$a_{\phi}^{\varrho^{\prime}}=\phi(q,d_{1}^{\varrho^{\prime}})$

Context NQ-AIR (149) TQA-AIR (330) Length Jaccard Length Jaccard Gen 105.9
0.194 104.9 0.385 Ctr 105.7 0.251 103.8 0.499 Ret 107.5 0.111 106.3 0.184

Table 4: Average length and Jaccard similarity of different contexts. Gen, Ret
and Ctr respectively represent generated contexts, retrieved contexts and
counter-memory contexts. Detailed discussion about Jaccard similarity is shown
in Section 5.2.

Context pair NQ-AIR (149) TQA-AIR (330) Gen vs. Ret 80.3 67.5 Ctr vs. Ret 81.6
83.0

Table 5: $\mathrm{DiffGR}$ (%) of different input context pairs. Gen, Ret and
Ctr respectively represent generated contexts, retrieved contexts and counter-
memory contexts.

Following the above procedures, we obtain 149 instances from NQ-AIR, and 330
instances from TQA-AIR, with each instance encompassing a question $q$, a
generated context $d_{1}^{\varrho}$, a retrieved context $d_{1}^{\gamma}$, a
counter-memory context $d_{1}^{\varrho^{\prime}}$ and the associated answers
for these contexts. Utilizing these instances, we initially employ the
$\mathrm{DiffGR}$ metric, as introduced in Section 3, to determine the LLMs’
preference for generated versus retrieved contexts. Then, we replace the
generated context $d^{\varrho}_{1}$ with the counter-memory context
$d^{\varrho^{\prime}}_{1}$ and recalculate $\mathrm{DiffGR}$ to examine shifts
in preference after excluding confirmation bias, as depicted in Table 5. The
results reveal that LLMs maintain a high $\mathrm{DiffGR}$ when the generated
contexts are replaced with counter-memory contexts. This suggests that LLMs
retain a preference for generated contexts, even when these contexts contain
information that contradicts the LLMs’ parametric memory.

Notably, our findings do not contradict the conclusions regarding confirmation
bias presented in Xie et al. (2023), due to the pronounced differences in the
experimental setups. While Xie et al. (2023) investigates the bias within two
generated contexts, our research focus on LLMs’ preferences between generated
and retrieved contexts. In our experimental setup, the generated and retrieved
contexts might present other, more pronounced disparities that could
overshadow the effects of confirmation bias.

###  5.2 Effect of Text Similarity

The text similarity between a context and a question can reflect the degree of
their relevance, thereby influencing LLMs’ preference for the context. This
section analyzes the text similarity of generated and retrieved contexts, and
investigates its role in the observed bias towards generated contexts. We
perform our analysis using the context-conflicting datasets with the
setup666Similar observation and conclusion are observed in other settings.
where the reader and generator share a single LLM. The detail framework to
constructe the context-conflicting datasets can be found in Section 3.

Similarity Metric. We employ Jaccard similarity to assess the term-based
overlap and BERTScore (Zhang et al., 2020) for evaluating the semantic
similarity between contexts and questions. To mitigate the effect of length
discrepancies between contexts and questions, we calculate the similarity at
the sentence level and then aggregate them to derive the overall context-
question similarity. In this section, we adopt a maximum aggregation strategy
due to the single-hop nature of the NQ and TQA datasets, where the majority of
questions can be answered using a small subset of sentences. We also try
average aggregation strategy and observe similar results, as shown in A.6.
Formally,

$\operatorname{sim}(q,d)=\max_{s\in
d}\left(\operatorname{sim}\left(q,s\right)\right)$

where $\operatorname{sim}$ is the similarity function, Jaccard similarity and
BERTScore in this paper, and $s$ represents a sentence in the context $d$.

Figure 7: Context-question similarity distribution of generated and retrieved
contexts on the union of AIR subsets for different LLMs (GPT 4, GPT 3.5,
Llama2-13b, Llama2-7b). A detailed distribution for each LLM, as well as the
distribution on AIG subsets, is presented in Appendix A.6. Figure 8: The
$\mathrm{DiffGR}$ in slices with different average $\Delta\operatorname{sim}$.
Llama2-13b is utilized as both the generator and reader here. Results for
other LLMs are presented in Appendix A.6.

Figure 7 shows the similarity distribution of generated and retrieved
contexts. We can observe that generated contexts exhibit a significant higher
similarity to questions even on AIR subsets, where generated contexts contain
incorrect information, whether assessed by term-based overlap or semantic
similarity.

Although we observe that the generated contexts exhibit a markedly higher
degree of similarity, the extent to which this difference in similarity
affects the bias remains unclear. To further investigate this question, we
evaluate LLMs’ preference in samples with different similarity gap
$\Delta\operatorname{sim}$ between generated and retrieved contexts.

$\Delta\operatorname{sim}=\frac{\operatorname{sim}(q,d^{\varrho})-\operatorname{sim}(q,d^{\gamma})}{\operatorname{sim}(q,d^{\varrho})+\operatorname{sim}(q,d^{\gamma})}$

Specifically, we rank the samples according to $\Delta\operatorname{sim}$ and
then divide the dataset into $n$ ($n=5$ here777Similar results and
observations are found with other $n$.) slices with an equal number of
samples. Subsequently, we calculate $\mathrm{DiffGR}$ for every slice.

Figure 8 illustrates the relationship between the average
$\Delta\operatorname{sim}$ within each slice and the corresponding
$\mathrm{DiffGR}$, using Llama2-13b as both generator and reader. Results for
other LLMs are presented in Appendix A.6, where similar conclusions are
observed. In Figure 8, we observe a general trend that on slices with a larger
average similarity gap, LLMs exhibit a increased preference for generated
contexts 888For example, on the TQA-AIR dataset, the slice exhibiting the
highest average Jaccard similarity gap demonstrates a $\mathrm{DiffGR}$ of
73.5%, compared to 48.2% in the slice with the lowest average Jaccard
similarity gap.. The stability of this trend is lower on NQ-AIR, which can be
attributed to the smaller number of instances on NQ-AIR (87 instances per
slice) as opposed to TQA-AIR (329 instances per slice). Overall, the notable
variations in $\mathrm{DiffGR}$ with respect to the similarity gap indicate
that text similarity is a significant factor in the preference for generated
contexts. These findings suggest that generated contexts should be applied
with greater caution to mitigate the influence of highly relevant but
misleading information.

To facilitate understanding of the impact of this similarity gap on LLMs’
preference, we include some examples about similarity in Appendix A.6.1. From
these case, we observe that the similarity gap can reflect the disparity in
the complexity of acquiring relevant information, which may partly leads to
LLMs’ preference. Concretely, contexts with greater similarity often support
candidate answers in a more straightforward manner, for instance, by mirroring
the phrasing used in the questions. Conversely, the contexts with low
similarity introduce more challenges, often necessitating an understanding of
synonyms and even some inferences.

###  5.3 Effect of Context Completeness

In all above experiments, there is a key difference between generated and
retrieved contexts that may affects the context preference: semantic and
sentence completeness. Concretely, current retrieval systems typically employ
fixed-length truncation to divide a complete article into multiple passages,
which serve as the fundamental units for retrieval tasks (Karpukhin et al.,
2020; Wang et al., 2019; Zhu et al., 2021). This truncation often results in
retrieved contexts with incomplete semantic meaning, as well as sentences that
are cut off at beginnings or endings. In contrast, generated contexts in above
experiments are naturally produced by LLMs, resulting in enhanced semantic and
sentence completeness.

Context Length Completeness Similarity Semantic Sentence Jaccard BERTScore
Retrieved 107.3 ✘ ✘ 0.1167 0.8537 Prompt 106.0 ✔ ✔ 0.1932 0.8753 Trunc. 106.2
✘ ✘ 0.1927 0.8753 S-Trunc. 105.1 ✘ ✔ 0.1903 0.8748

Table 6: Context statistics with different completeness on NQ. Three types of
generated contexts exhibit comparable average length, Jaccard and BertScore
similarities.

Context Pair NQ-AIR TQA-AIR Prompt vs. Retrieved 84.54 79.38 Trunc. vs.
Retrieved 65.58 65.56 S-Trunc. vs. Retrieved 65.06 64.64

Table 7: $\mathrm{DiffGR}$ (%) with different completeness in generated
context. “Prompt”, “Trunc.” and “S-Trunc.” represent three types of generated
contexts with different completeness. GPT 3.5 is used as both the reader and
generator. Random ordering is utilized when inputting context paris.

In this section, we conduct controlled experiments that vary the semantic and
sentence completeness of generated contexts to investigate the potential
effects of completeness on the observed bias. The following three methods are
employed to produce generated contexts with different completeness:

  * •

(a) Prompt: The prompt method, as used in all above experiments, incorporates
length constraints into the prompt to facilitate the natural generation of
contexts, leading to both semantic and sentence completeness.

  * •

(b) Truncation (Trunc.): To simulate the incompleteness of retrieved contexts,
we eliminate the length constraints from the prompt of Section 3, allowing
LLMs to generate extended contexts. These generated contexts are then
truncated to match the length of retrieved contexts, resulting in both
semantic and sentence incompleteness.

  * •

(c) Sentence Truncation (S-Trunc.): Based on method (b), we truncate generated
contexts only at the end of a sentence to preserve the sentence completeness,
while simulating the semantic incompleteness.

Table 6 demonstrates that three types of generated contexts have similar
average lengths999The minor difference in average length between truncated and
retrieved contexts results from a small samples (only 5 samples) where the
generated contexts are shorter. and similarities. This suggests that the
influences of similarity and length are mitigated, thereby highlighting the
principal disparities in semantic content and sentence completeness.

We evaluate LLMs’ preference between generated versus retrieval context,
varying the completeness of generated context, following the same pipeline in
Section 4.1. Table 7 presents the $\mathrm{DiffGR}$ with different semantic
and sentence completeness in generated contexts. A comparison between “Trunc.”
and “S-Trunc.” reveals that sentence completeness does not significantly
affect LLMs’ preference for generated contexts. In contrast, comparing
“Prompt” and “S-Trunc.”, we find a significant increase in preference for
generated contexts, when they are semantically more complete. These findings
indicate that LLMs are sensitive to the semantic completeness of the contexts,
underscoring the necessity to investigate improved passage segmentation
methods that maintain semantic completeness for current retrieval-augmented
LMs.

##  6 Related work

###  6.1 Generation-Augmented Approaches

Generation-augmented approaches prompt LLMs to generate intermediate contexts
for the final response, thereby leveraging their extensive parametric
knowledge acquired during the pre-training phase on vast text corpora (Roberts
et al., 2020; Petroni et al., 2019). Recent studies have demonstrated the
effectiveness of these methods across a range of tasks. Liu et al. (2022)
represents an early exploration into augmenting LLMs in commonsense reasoning
tasks using knowledge generated by LLMs themselves. Sun et al. (2023); Yu et
al. (2022) employ LLMs to produce background documents (or recitations) and
subsequently use these documents to enhance LLM performance in knowledge-
intensive tasks. Another researches, known as the chain-of-thought (Wei et
al., 2022a; Kojima et al., 2022), prompts LLMs to generate intermediate
reasoning steps to enhance LLMs’ reasoning abilities. Desipte the
effectiveness of these methods, the LLM-generated knowledge may contain
hallucinations (Ji et al., 2023) due to LLMs’ outdated memory (De Cao et al.,
2021) and limited memory for long-tail knowledge (Kandpal et al., 2023). The
LLM-generated inaccuracy information could potentially mislead current
retrieval model (Dai et al., 2023) and open-domain question answering systems
(Pan et al., 2023).

###  6.2 Retrieval-Augmented Approaches

The retrieval-augmented approaches (Guu et al., 2020; Lewis et al., 2020; Ram
et al., 2023; Gao et al., 2023) enhance LLMs by incorporating relevant
documents from external corpus. These approaches represent a promising
direction for addressing the knowledge limitations of LLMs, such as the need
for knowledge updates (Jang et al., 2022) and long-tail knowledge (Kandpal et
al., 2023). Early retrieval-augmented methods (Guu et al., 2020; Lewis et al.,
2020; Izacard et al., 2022) focused on the jointly training of LLMs and
retrieval modules to improve their cooperation. With the evolution of general-
purpose LLMs, recent studies (Ram et al., 2023; Shi et al., 2023) investigate
the strategy of appending relevant documents directly to the input while
keeping the LLMs frozen. Despite the effectiveness of these methods, the
retrieval-augmented approaches still face challenges due to irrelevant
retrieval results and incomplete knowledge coverage Yu et al. (2023); Mallen
et al. (2023). These noisy retrieval results can misguide the outcomes of LLMs
(Mallen et al., 2023; Anonymous, 2023; Ren et al., 2023).

###  6.3 Hybrid Approaches and Knowledge Conflicts

Recent works investigate to merge retrieved and generated contexts to leverage
both parametric knowledge and external knowledge Abdallah and Jatowt (2023);
Yu et al. (2022). These combination methods have shown improved performance
over those relying solely on a single information source in fully-supervised
setting. Furthermore, (Zhang et al., 2023) proposes an improved method to
effectively leverage the two sources of information, especially when conflicts
arise. While these works have focused on improving the efficacy of hybrid
approaches, the underlying mechanisms by which LLMs process conflicting
information from different types of contexts remain underexplored.

Current research on knowledge conflicts in LLMs primarily focuses on two
aspects: conflicts between input contexts and LLMs’ parametric memory, and
conflicts among the input contexts themselves. Regarding the former, Xie et
al. (2023); Chen et al. (2022) find that LLMs are highly receptive to the
input contexts rather than their internal memory. Concerning conflicts within
multiple input contexts, Chen et al. (2022) demonstrates that LLMs tend to
rely on a few most relevant retrieved contexts. Additionally, Xie et al.
(2023) reveals a confirmation bias, i.e., LLMs demonstrate a tendency to favor
contexts that align with their parametric knowledge, when confronted with both
supporting and opposing contexts. However, these studies are limited to
analyzing context conflicts within a single type of input contexts. Our work
complements these studies by considering conflicts between generated and
retrieved contexts, and reveals several key factors, such as semantic
completeness and text similarity.

##  7 Conclusion and Future work

In this study, we proposed a framework to investigate the underlying
mechanisms by which LLMs merge retrieved and generated contexts. Our results
reveal a pronounced bias towards generated contexts in several state-of-the-
art LLMs (GPT 3.5/4, Llama2-7b/13b). We further identify two key factors
contributing to this bias: higher similarity between generated contexts and
questions, as well as the semantic incompleteness of retrieved contexts.

Our insights highlight the critical need for advanced integration methods that
can validate and leverage information from both sources, moving beyond the
current overreliance on generated contexts. Additionally, we find that LLMs
display significant sensitivity to the semantic completeness of input
contexts. This sensitivity necessitates improved passage segmentation
strategies in current retrieval-augmented systems, thereby ensuring the
preservation of intended meaning and the maximization of utility. Finally,
addressing the challenges posed by highly relevant yet incorrect information
generated by LLMs is an important direction for future research. It is crucial
to develop methods for detecting and discounting misleading information
produced by LLMs, especially as the volume of such content continues to
escalate.

## References

  * Abdallah and Jatowt (2023) Abdelrahman Abdallah and Adam Jatowt. 2023.  Generator-retriever-generator: A novel approach to open-domain question answering.  _arXiv preprint arXiv:2307.11278_.
  * Anonymous (2023) Anonymous. 2023.  Making retrieval-augmented language models robust to irrelevant context.  In _Submitted to The Twelfth International Conference on Learning Representations_.  Under review.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_.
  * Chen et al. (2022) Hung-Ting Chen, Michael Zhang, and Eunsol Choi. 2022.  Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2292–2307, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Dai et al. (2023) Sunhao Dai, Yuqi Zhou, Liang Pang, Weihao Liu, Xiaolin Hu, Yong Liu, Xiao Zhang, and Jun Xu. 2023.  Llms may dominate information access: Neural retrievers are biased towards llm-generated texts.  _arXiv preprint arXiv:2310.20501_.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Feng et al. (2023) Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023.  Cook: Empowering general-purpose language models with modular and collaborative knowledge.  _arXiv preprint arXiv:2305.09955_.
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: retrieval-augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , pages 3929–3938.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022.  TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6237–6250, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Comput. Surv._ , 55(12).
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 15696–15707. PMLR.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  * Liu et al. (2022) Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022.  Generated knowledge prompting for commonsense reasoning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3154–3169, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2023) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the middle: How language models use long contexts.  _arXiv preprint arXiv:2307.03172_.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.  Augmented large language models with parametric knowledge guiding.  _arXiv preprint arXiv:2305.04757_.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.
  * Pan et al. (2023) Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Wang. 2023.  On the risk of misinformation pollution with large language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1389–1403, Singapore. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 11:1316–1331.
  * Ren et al. (2023) Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023.  Investigating the factual knowledge boundary of large language models with retrieval augmentation.  _arXiv preprint arXiv:2307.11019_.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_.
  * Sun et al. (2023) Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023.  Recitation-augmented language models.  In _The Eleventh International Conference on Learning Representations_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2019) Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019.  Multi-passage BERT: A globally normalized BERT model for open-domain question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5878–5882, Hong Kong, China. Association for Computational Linguistics.
  * Wei et al. (2022a) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022a.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 24824–24837. Curran Associates, Inc.
  * Xie et al. (2023) Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023.  Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts.  _arXiv preprint arXiv:2305.13300_.
  * Yu et al. (2022) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022.  Generate rather than retrieve: Large language models are strong context generators.  In _The Eleventh International Conference on Learning Representations_.
  * Yu et al. (2023) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023.  Chain-of-note: Enhancing robustness in retrieval-augmented language models.  _arXiv preprint arXiv:2311.09210_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2023) Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023.  Merging generated and retrieved knowledge for open-domain QA.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4710–4728, Singapore. Association for Computational Linguistics.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_.

##  Appendix A Appendix

###  A.1 Length Control for Generated Contexts

####  A.1.1 Length Distribution across LLMs

In our proposed framework, we regulate the length of generated contexts by
incorporating length constraint in the prompt:  Generate a background context
from Wikipedia to answer the given question {#question} Keep the length of the
document around $n$ words We observed that GPT 4 effectively controls the
output length, whereas other models struggle with this aspect. To address this
issue in the latter, we employ multiple values of $n$ and select the one that
best matched the retrieved context.

As the result, Table 9 shows the length distribution of retrieved contexts and
contexts generated by various LLMs. The length distribution of retrieved
contexts is more concentrated due to the fact that these contexts consist of
text truncated precisely at one hundred words, coupled with their titles
(Karpukhin et al., 2020). Variations in the length of different retrieved
articles are solely attributable to the differences in title lengths.

####  A.1.2 Length Distribution with Different Control Methods

Figure 10 illustrates the length distribution for generated contexts
corresponding to three different length control methods. All three methods
achieve a length distribution closely resembling that of the retrieved
contexts.

(a) NQ GPT 4

(b) NQ GPT 3.5

(c) NQ Llama2-13b

(d) NQ Llama2-7b

(e) TQA GPT 4

(f) TQA GPT 3.5

(g) TQA Llama2-13b

(h) TQA Llama2-7b

Figure 9: Length distribution of generated and retrieved contexts on different
datasets with different generator models.

(a) Truncation

(b) Sentence Truncation

(c) Prompt

Figure 10: Length distribution of generated and retrieved contexts on the NQ
dataset, with different methods to control the length of generated contexts.

###  A.2 Dataset Statistics

Table 8 presents the data size of context-conflicting datasets corresponding
to various generator-reader pairs. The statistics indicate that conflicting
data comprise a substantial proportion across all combinations of generators
and readers.

Reader Generator NQ (3610) TQA (11313) NQ-AIG NQ-AIR TQA-AIG TQA-AIR GPT 4 GPT
4 331 238 946 358 GPT 4 GPT 3.5 282 291 - - GPT 4 Llama2-13b 204 453 - - GPT 4
Llama2-7b 168 492 - - GPT 3.5 GPT 4 453 247 1506 414 GPT 3.5 GPT 3.5 379 279
1357 567 GPT 3.5 Llama2-13b 247 407 1000 1206 GPT 3.5 Llama2-7b 187 456 823
1492 Llama2-13b GPT 4 735 246 2672 404 Llama2-13b GPT 3.5 656 287 2363 576
Llama2-13b Llama2-13b 439 419 1648 1201 Llama2-13b Llama2-7b 381 484 1430 1571
Llama2-7b GPT 4 803 211 2976 459 Llama2-7b GPT 3.5 722 243 2702 632 Llama2-7b
Llama2-13b 474 355 1966 1170 Llama2-7b Llama2-7b 432 406 1737 1488

Table 8: The data quantities of the constructed subsets for different
(Generator, Reader) pairs. NQ and TQA refer to the original complete test
sets.

###  A.3 Effect of Context Order

Order | NQ-AIR | TQA-AIR
---|---|---
generated-first | 69.9 | 68.2
retrieved-first | 66.5 | 55.6
random | 69.1 | 58.6
Table 9: The generation ratio (%) with different context order on NQ-AIR and
TQA-AIR datasets. The backbone LLM is GPT 3.5.

In above experiments, retrieved and generated contexts are presented with
random order. Previous studies Xie et al. (2023); Liu et al. (2023); Lu et al.
(2022) have found that the model may be sensitive to the order of the input
contexts. In their experiments, the input context was either all retrieved Liu
et al. (2023) or all generated Xie et al. (2023). We conducted experiments to
investigate whether the context order impacts the preference for the generated
context. The generated and retrieved contexts are concatenated with three
different orders: generated-first, retrieved-first and random. We compute the
context preference with different context order respectively.

As shown in Table 9, across all context orders, LLMs consistently show a
strong tendency to generated contexts (generation ratio > 0). When the
retrieved context is positioned first, there is a slight reduction in the
generation ratio. This reduction may result from the LLMs’ preference for
generated contexts being partially offset by their bias towards the top
context (Liu et al., 2023; Xie et al., 2023).

###  A.4 More Results on AIG Datasets

Figure 11 shows the $\mathrm{DiffGR}$ (%) with different (reader, generator)
pairs on their corresponding NQ-AIG datasets. It can be observed that LLMs
show strong tendency to rely on generated contexts across various (reader,
generator) pairs.

Figure 11: $\mathrm{DiffGR}$ (%) with different (reader, generator) pairs on
their corresponding NQ-AIG datasets.

###  A.5 More Details about preparing Counter-Memory Answers

To generate a counter-memory answer distinct from a given memory answer, we
employ a LLM to associate a different entity of the same type. It is
noteworthy that the prompt does not incorporate the question, ensuring that
the LLM’s counter-memory answer is merely a categorical equivalent rather than
a memory related to the question. To better standardize the output of the LLM,
we have manually crafted several examples to be included within the prompt:
Give you a reference word, transform it into a analogous word.
Reference: Missouri River
Analogous Word: Mississippi River
Reference: {memory answer}
Analogous Word:

###  A.6 More Details about Similarity

Figure 12 and 13 show the similarity distribution of retrieved and generated
contexts across various generators. All LLM-generated contexts exhibit a
higher similarity over retrieved contexts.

Figure 14 illustrates the distribution of similarity when employing maximum
and average aggregation methods. It is observable that the generated contexts
exhibit a markedly higher degree of similarity regardless of the aggregation
method used. Furthermore, this disparity in similarity is more pronounced with
maximum aggregation, as contexts typically contain sentences that are
irrelevant, which dilute the similarity scores when an average aggregation is
applied.

Figure 15 demonstrates a general trend that on slices with a smaller average
similarity gap, LLMs exhibit a reduced preference for generated context. This
trend is not stable on NQ-AIR, which can be attributed to the smaller number
of instances on NQ-AIR, as shown in Table 8.

Figure 12: BERTScore distribution of retrieved context and context generated
by different LLMs. Figure 13: Jaccard Similarity distribution of retrieved
context and context generated by different LLMs. Figure 14: Similarity
distribution with maximum or mean aggregation strategies. Generated contexts
consistently exhibit higher similarity across two aggregation strategies.

(a) GPT 4

(b) GPT 3.5

(c) Llama2-13b

(d) Llama2-7b

Figure 15: The generation ratio in slices with different average
$\Delta\operatorname{sim}$. $\Delta\operatorname{sim}$ is the difference in
similarity between the generated context and the retrieved context.

####  A.6.1 Cases about Similarity

Table 10 shows examples that contain contexts with different similarity to the
question. The contexts with high similarity typically directly support
answering by repeating the phrasing in the question. Conversely, the contexts
with low similarity introduce more challenges, often necessitating an
understanding of synonyms and even some inferences. These observations
indicate that text similarity can partly reflect the relevance between a
question and a context, as well as the difficulty a LLM encounters in
identifying potential answers.

TQA-AIR Example TQA-AIG Example Question Between 1959 and 1967 which city was
the capital of Pakistan (Islamabad was being built)? Who is the most
successful UK solo artist in the USA? Golden Answer Rawalpindi Elton John
Generated Context Between 1959 and 1967, the capital of Pakistan was Karachi.
Karachi is the largest city in Pakistan and is located on the southern coast
of the country… Jaccard Similarity: 0.47 BertScore: 0.93 Elton John is the
most successful UK solo artist in the USA. Born Reginald Kenneth Dwight in
1947, he adopted the stage name Elton John in the late 1960s… Jaccard
Similarity: 0.69 BertScore: 0.93 Retrieved Context …was first shifted
temporarily to Rawalpindi in the early 60s, and then to Islamabad when
essential development work was completed in 1966… Jaccard Similarity: 0.16
BertScore: 0.85 …In 2009, Jay Sean’s single "Down" reached the number one spot
on the "Billboard" Hot 100 and sold millions in the United States, making him
ẗhe most successful male UK urban artist in US chart historyät the time…
Jaccard Similarity: 0.14 BertScore: 0.86 Model output Karachi Elton John

Table 10: Some examples where both the generator and reader are GPT-3.5. We
highlight the incorrect candidate answers in the context in pink, and the
correct answers in the context in green.