# Retrieval Augmented Generation or Long-Context LLMs?
A Comprehensive Study and Hybrid Approach

Zhuowan Li1 Cheng Li1 Mingyang Zhang1
Qiaozhu Mei2 Michael Bendersky1
1 Google DeepMind 2 University of Michigan
1<EMAIL_ADDRESS>2<EMAIL_ADDRESS>Visiting
researcher to Google DeepMind.

###### Abstract

Retrieval Augmented Generation (RAG) has been a powerful tool for Large
Language Models (LLMs) to efficiently process overly lengthy contexts.
However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities
to understand long contexts directly. We conduct a comprehensive comparison
between RAG and long-context (LC) LLMs, aiming to leverage the strengths of
both. We benchmark RAG and LC across various public datasets using three
latest LLMs. Results reveal that when resourced sufficiently, LC consistently
outperforms RAG in terms of average performance. However, RAG’s significantly
lower cost remains a distinct advantage. Based on this observation, we propose
Self-Route, a simple yet effective method that routes queries to RAG or LC
based on model self-reflection. Self-Route significantly reduces the
computation cost while maintaining a comparable performance to LC. Our
findings provide a guideline for long-context applications of LLMs using RAG
and LC.

Retrieval Augmented Generation or Long-Context LLMs?
A Comprehensive Study and Hybrid Approach

Zhuowan Li1 Cheng Li1 Mingyang Zhang1 Qiaozhu Mei2††thanks: Visiting
researcher to Google DeepMind. Michael Bendersky1 1 Google DeepMind 2
University of Michigan 1<EMAIL_ADDRESS>2
<EMAIL_ADDRESS>

##  1 Introduction

Retrieval augmented generation (RAG) has been shown to be a both effective and
efficient approach for large language models (LLMs) to leverage external
knowledge. RAG retrieves relevant information based on the query and then
prompts an LLM to generate a response in the context of the retrieved
information. This approach significantly expands LLM’s access to vast amounts
of information at a minimal cost.

However, recent LLMs like Gemini and GPT-4 have demonstrated exceptional
capabilities in understanding long contexts directly. For example, Gemini 1.5
can process up to 1 million tokens Reid et al. (2024). This prompts the need
for a systematic comparison between long-context (LC) LLMs and RAG: on one
hand, RAG conceptually acts as a prior, regularizing the attention of LLMs
onto retrieved segments, thus avoiding the distraction of the irrelevant
information and saving unnecessary attention computations; on the other hand,
large-scale pretraining may enable LLMs to develop even stronger long-context
capabilities. Therefore, we are motivated to compare RAG and LC, evaluating
both their performance and efficiency.

Figure 1: While long-context LLMs (LC) surpass RAG in long-context
understanding, RAG is significantly more cost-efficient. Our approach, Self-
Route, combining RAG and LC, achieves comparable performance to LC at a much
lower cost.

In this work, we systematically benchmark RAG and LC on various public
datasets, gaining a comprehensive understanding of their pros and cons, and
ultimately combining them to get the best of both worlds. Different from
findings in previous work Xu et al. (2023), we find that LC consistently
outperform RAG in almost all settings (when resourced sufficiently). This
demonstrates the superior progress of recent LLMs in long-context
understanding.

Despite the suboptimal performance, RAG remains relevant due to its
significantly lower computational cost. In contrast to LC, RAG significantly
decreases the input length to LLMs, leading to reduced costs, as LLM API
pricing is typically based on the number of input tokens. Google (2024);
OpenAI (2024b)111While retrieval may introduce extra cost, retrieval system is
much easier to set up and can be hosted on customer side.. Moreover, our
analysis reveals that the predictions from LC and RAG are identical for over
60% of queries. For these queries, RAG can reduce cost without sacrificing
performance.

Based on this observation, we propose Self-Route, a simple yet effective
method that routes various queries to RAG or LC based on model self-
reflection. With Self-Route, we significantly reduce the cost while achieving
overall performance comparable to LC. For example, the cost is reduced by 65%
for Gemini-1.5-Pro and 39% for GPT-4O.

Fig. 1 shows the comparisons of LC, RAG and Self-Route using three recent
LLMs: GPT-4O, GPT-3.5-Turbo and Gemini-1.5-Pro. In addition to quantitative
evaluation, we provide a comprehensive analysis comparing RAG and LC,
including common failure patterns of RAG, the trade-offs between cost and
performance, and the results on additional synthetic datasets. Our analysis
serves as a starting point, inspiring future improvements of RAG, and as a
empirical guide for building long-context applications using RAG and LC.

##  2 Related Work

Long-context LLMs. There has long been efforts for enabling LLMs to handle
long contexts Guo et al. (2022); Beltagy et al. (2020); Chen et al. (2023b).
While recent LLMs like Gemini-1.5 Reid et al. (2024), GPT-4 Achiam et al.
(2023), Claude-3 Anthropic (2024) achieve significantly larger context window
size, long-context prompting is still expensive due to the quadratic
computation cost of transformers regarding to the input token numbers. Recent
work proposes methods to reduce cost by prompt compression Jiang et al.
(2023), model distillation Hsieh et al. (2023), or LLM cascading Chen et al.
(2023a).

Retrieval-augmented generation. Augmenting LLMs with relevant information
retrieved from various sources Lewis et al. (2020), _i.e_., RAG, has been
successful in complementing LLMs with external knowledge. RAG achieves good
performance on various of tasks like language modeling Khandelwal et al.
(2019); Shi et al. (2023) and QA Guu et al. (2020); Izacard and Grave (2020),
with a significantly lower computation cost Borgeaud et al. (2022). Related to
but different from our work, recently works augment RAG with correction Yan et
al. (2024), critique Asai et al. (2023), or verification Li et al. (2023) to
improve retrieval quality on knowledge-intensive tasks.

Long-context evaluation. Evaluating long-context models is challenging due to
the difficulty in collecting and analyzing long texts. Recent researchers
propose both synthetic tests like needle-in-a-haystack Greg Kamradt (2023),
Ruler Hsieh et al. (2024), or Counting Stars Song et al. (2024), and real
datasets including LongBench Bai et al. (2023), $\infty$Bench Zhang et al.
(2024), L-Eval An et al. (2023), and others Shaham et al. (2022); Yuan et al.
(2024); Maharana et al. (2024). Evaluating on these datasets, recent works
study the performance degradation over various context lengths Levy et al.
(2024); Hsieh et al. (2024), the lost-in-the-middle phenomenon Liu et al.
(2024), and explore solutions Kuratov et al. (2024). Related to our work, Xu
et al. (2023) compare RAG and long-context prompting and find that long-
context models still lags behind RAG. This is different from our findings,
possibly due to consideration of stronger LLMs and longer contexts in our
work.

##  3 Benchmarking RAG versus LC

###  3.1 Datasets and metrics

We evaluate on a subset of datasets from LongBench Bai et al. (2023) and
$\infty$Bench Zhang et al. (2024), which are recent benchmarks containing a
collection of new and existing datasets for LLM evaluation, covering both
synthetic and real texts in multiple languages. LongBench contains a
collection of 21 datasets, with an average context length of 7k words.
$\infty$Bench consists of even longer contexts with an average length of 100k
tokens.

Among the datasets, we mainly focus on tasks that are (a) in English, (b)
real, and (c) query-based (_e.g_. summarization tasks do not contain queries
for retrieving relevant information). This results in 7 datasets from
LongBench including NarrativeQA Kočiskỳ et al. (2018), Qasper Dasigi et al.
(2021), MultiFieldQA Bai et al. (2023), HotpotQA Yang et al. (2018),
2WikiMultihopQA Ho et al. (2020), MuSiQue Trivedi et al. (2022), QMSum Zhong
et al. (2021); and 2 datasets from $\infty$Bench including En.QA and EN.MC.
Please refer to Appendix A for more details. Additionally, in Sec. 5.4, we
will provide an ablation a synthetic datasets PassKey from $\infty$Bench.

For evaluation metrics, we report F1 scores for the open-ended QA tasks,
accuracy for the multi-choice QA tasks, and ROUGE score for the summarization
tasks.

###  3.2 Models and Retrievers

Three latest LLMs are evaluated, including Gemini-1.5-Pro Reid et al. (2024),
GPT-4O OpenAI (2024a), and GPT-3.5-Turbo OpenAI (2023) 222gpt-3.5-turbo-0125,
gpt-4o-2024-05-13. Gemini-1.5-Pro is a recent long-context LLM from Google,
supporting up to 1 million tokens. GPT-4O, the newest lightweight yet strong
LLM from OpenAI, supports 128k tokens. GPT-3.5-Turbo supports 16k tokens.

Two retrievers are used in our study: Contriever Izacard et al. (2021), which
is a contrastively trained dense retriever outperforming BM25 on BEIR
datasets, and Dragon Lin et al. (2023), which is a recent generalizable dense
retriever achieving high performance in both supervised and zero-shot settings
without complex late interaction. Following Xu et al. (2023), we divide long
contexts into chunks of 300 words, and select the top $k$ chunks (default
$k=5$) based on the cosine similarity of the query embedding and the chunk
embeddings. The chunks are ordered by the similarity scores, with the chunk
index prepended at the beginning.

Since black-box LLMs are pretrained on unknown datasets, the leakage of
evaluation datasets may occur. Especially, some of the evaluation datasets are
based on Wikipedia, which has likely been seen by LLMs during during. In some
cases, we find that model may predict the correct answer using exactly the
same words as the groundtruth (_e.g_. “meticulously”), even when they do not
appear in the provided context. In our experiment, we try mitigating this
issue by prompting the model to answer ‘‘based only on the provided passage’’
for both RAG and LC. It remains an open question how to address the data
leakage issue in LLM evaluation.

###  3.3 Benchmarking results

We benchmark the performance of LC and RAG across the nine datasets, using
three recent LLMs: Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo. Tab. 1 presents
the results using the Contriever retriever, where rows *-1 and rows *-2
present the benchmarking results for LC and RAG respectively. Results using
the Dragon retriever will be discussed in Sec. 5.3 and Tab. 2.

As shown in Tab. 1, LC consistently outperforms RAG for all the three models,
with a significant margin. On average, LC surpasses RAG by 7.6% for
Gemini-1.5-Pro, 13.1% for GPT-4O, and 3.6% for GPT-3.5-Turbo. Noticeably, the
performance gap is more significant for the more recent models (GPT-4O and
Gemini-1.5-Pro) compared to GPT-3.5-Turbo, highlighting the exceptional long-
context understanding capacity of the latest LLMs.

However, there is an exception observed on the two longer datasets from
$\infty$Bench (_i.e_., En.QA and En.MC), where RAG achieves higher performance
than LC for GPT-3.5-Turbo. This result deviates from the overall trend, likely
due to the significantly longer context in these datasets (147k words on
average) compared with the limited context window (16k) of GPT-3.5-Turbo. This
finding highlights the effectiveness of RAG when the input text considerably
exceeds the model’s context window size, emphasizing a specific use case of
RAG.

|  |  | Avg | Narr | Qasp | Mult | Hotp | 2Wiki | Musi | Sum | En.QA | En.MC
---|---|---|---|---|---|---|---|---|---|---|---|---
| 1-1 | LC | 49.70 | 32.76 | 47.83 | 52.33 | 61.85 | 62.96 | 40.22 | 20.73 | 43.08 | 85.57
Gemini-1.5-Pro | 1-2 | RAG | 37.33 | 22.54 | 44.68 | 49.53 | 48.36 | 54.24 | 26.56 | 19.51 | 19.46 | 51.09
1-3 | Self-Route | 46.41 | 28.32 | 45.23 | 51.47 | 55.18 | 62.68 | 40.66 | 19.77 | 37.51 | 76.86
1-4 | answerable % | 76.78 | 73.00 | 85.00 | 96.67 | 84.50 | 81.00 | 58.50 | 93.50 | 56.41 | 62.45
1-5 | token % | 38.39 | 23.07 | 49.93 | 36.88 | 32.97 | 53.49 | 56.14 | 17.96 | 42.25 | 32.84
GPT-4O | 2-1 | LC | 48.67 | 32.78 | 44.54 | 55.28 | 62.42 | 70.69 | 41.65 | 21.92 | 32.36 | 76.42
2-2 | RAG | 32.60 | 18.05 | 46.02 | 50.74 | 36.86 | 50.21 | 16.09 | 19.97 | 14.43 | 41.05
2-3 | Self-Route | 48.89 | 31.36 | 47.99 | 53.17 | 62.14 | 70.14 | 41.69 | 21.31 | 34.95 | 77.29
2-4 | answerable % | 57.36 | 44.00 | 67.50 | 94.00 | 52.50 | 62.00 | 30.00 | 92.00 | 27.07 | 47.16
2-5 | token % | 61.40 | 66.40 | 72.25 | 39.65 | 65.79 | 77.05 | 85.00 | 20.26 | 73.01 | 53.21
GPT-3.5-Turbo | 3-1 | LC | 32.07 | 23.34 | 42.96 | 49.19 | 45.33 | 41.04 | 17.92 | 19.61 | 14.73 | 34.50
3-2 | RAG | 30.33 | 18.22 | 38.15 | 49.21 | 37.84 | 35.16 | 16.41 | 18.94 | 15.39 | 43.67
3-3 | Self-Route | 35.32 | 24.06 | 38.65 | 52.07 | 47.28 | 44.62 | 34.44 | 19.88 | 22.03 | 44.54
3-4 | answerable % | 74.10 | 71.50 | 80.00 | 91.33 | 68.50 | 69.00 | 47.00 | 93.50 | 50.43 | 95.63
3-5 | token % | 38.85 | 20.56 | 55.08 | 35.29 | 48.70 | 65.91 | 65.08 | 16.40 | 38.17 | 4.50

Table 1: Results of Gemini-1.5-Pro, GPT-3.5-Turbo, and GPT-4O using the
Contriever retriever. LC consistently outperforms RAG, while Self-Route
achieves performance comparable to LC using much less tokens.

##  4 Self-Route

###  4.1 Motivation

As demonstrated in Sec. 3, RAG lags behind long-context LLMs in terms of
performance. However, despite this performance gap, we surprisingly find a
high degree of overlap in their predictions, as illustrated in Fig. 2.

Figure 2: Distribution of the difference of prediction scores between RAG and
LC (computed w.r.t. groundtruth labels). RAG and LC predictions are highly
identical, for both correct and incorrect ones.

Fig. 2 displays the distribution of the differences between RAG prediction
scores $S_{RAG}$ and LC prediction scores $S_{LC}$, specifically
$S_{RAG}-S_{LC}$ (the scores are multiplied by 100 to be scaled to 1-100).
These scores $S$ represent the evaluation of model predictions against the
groundtruth. Notably, for most queries, RAG scores and LC scores are highly
similar. In fact, for 63% queries, the model predictions are exactly
identical; and for 70% queries, the score difference is less than 10 (absolute
value). Interestingly, the identical predictions are not necessarily correct,
as shown by the varying colors representing the average score, _i.e_.,
$(S_{RAG}+S_{LC})/2$. This observation suggests that RAG and LC tend to make
not only the same correct predictions but also similar errors.

This finding motivates us to leverage RAG for the majority of queries,
reserving computationally more expensive LC for a small subset of queries
where it truly excels. By doing so, RAG can significantly reduce computational
costs without sacrificing overall performance.

###  4.2 Self-Route

Based on the above motivation, we propose Self-Route, a simple yet effective
method combining RAG and LC to reduce cost while maintaining a performance
comparable to LC. Self-Route utilizes LLM itself to route queries based on
self-reflection, under the assumption that LLMs are well-calibrated in
predicting whether a query is answerable given provided context.

Concretely, our method consists of two steps: a RAG-and-Route step and a long-
context prediction step. In the first step, we provide the query and the
retrieved chunks to the LLM, and prompt it to predict whether the query is
answerable and, if so, generate the answer. This is similar to standard RAG,
with one key difference: the LLM is given the option to decline answering with
the prompt ‘‘Write unanswerable if the query can not be answered based on the
provided text’’. For the queries deemed answerable, we accept the RAG
prediction as the final answer. For the queries deemed unanswerable, we
proceed to the second step, providing the full context to the long-context
LLMs to obtain the final prediction (_i.e_., LC).

As our results will demonstrate, most queries can be solved by the first RAG-
and-Route step (_e.g_., 82% for Gemini-1.5-Pro), with only a small portion
requiring the following long-context prediction step. Since the RAG-and-Route
step only needs the retrieved chunks (_e.g_., 1.5k tokens) as input, which is
significantly shorter than the full contexts (_e.g_., 10k - 100k tokens), the
overall computation cost is substantially reduced. Detailed token count
analysis will be provided in the results.

###  4.3 Results

Rows *-3 to *-5 in Tab. 1 present the results of our method, utilizing the
three LLMs. Rows *-3 report the performance. Rows *-4 show the percentage of
answerable queries, as predicted in the RAG-and-Route step. Rows *-5 display
the percentage of tokens used by our method, compared to that of LC. In terms
of performance (rows *-3), Self-Route significantly outperforms RAG, achieving
results comparable to LC. Across all three models, Self-Route surpasses RAG
(rows *-2) by over 5%. Compared to LC (rows *-1), there is a slight
performance drop for GPT-4O (-0.2%) and Gemini-1.5-Pro (-2.2%), but an
improvement for GPT-3.5-Turbo (+1.7%).

All three LLMs consistently route more than half of queries towards RAG, as
shown in rows *-4. For Gemini-1.5-Pro, the answerable percentage even reaches
81.74% (row 1-4). This indicates that RAG may answer most queries without the
need for LC, confirming our initial motivation.

Due to the high answerable rate, the number of tokens required is
significantly reduced (rows *-5). For example, GPT-4O uses only 61% tokens
while achieving comparable performance (46.83) with LC (47.04), Gemini-1.5-Pro
uses 38.6% of the tokens. Since the computation cost of the transformer-based
LLMs is quadratic to token count, and most LLM APIs charge based on token
count OpenAI (2024b); Google (2024), this lower token count translates to
substantial cost savings.

On longer datasets, the advantage of our method is more pronounced for OpenAI
models, but less significant for Gemini. For instance, for GPT-4O, Self-Route
outperforms LC by 2.3% and 7.4% respectively on EN.QA and EN.MC, which contain
longer contexts. For GPT-3.5-Turbo, the advantage margins are even larger.
However, for Gemini-1.5-Pro, the performance is lower than LC. These different
behaviors are possibly due to the difference in LLM alignments, _i.e_., OpenAI
models are more likely to reject answering using RAG, leading to a lower
answerable percentage but higher accuracy, which results in a different
performance-cost trade-off compared with Gemini-1.5-Pro.

##  5 Analysis

###  5.1 Ablations of k

Both RAG and Self-Route relies on the top-$k$ retrieved text chunks. The
larger $k$ is, the longer context are fed into LLMs for RAG prediction as well
as routing, resulting in different costs versus performances. To study the
influence of $k$, in Fig. 3, we plot the performance and cost (_i.e_. input
token percentage) curves when different $k$s are used.

Figure 3: Trade-off curves between (a) model performance and (b) token
percentage as a function of $k$.

In terms of performance, for both RAG and Self-Route, a larger $k$ leads to
better performance. While $k$ increases, more and more chunks are fed into the
LLMs, thus the performance gradually improves to approach LC. As can be seen
in from the curves, the advantage of Self-Route is the most significant for
smaller $k$. For example, when $k=1$, RAG gets from 20.24% while Self-Route
gets 37.9%, while when $k$ is larger than 50, all three methods get similar
performance.

However, the trend of cost is not monotonous for Self-Route. As seen, the cost
reaches its minimum at $k=5$. This is because when $k$ increases, the cost of
RAG (and routing) increases, but more queries are routed to RAG from LC, thus
the overall cost may decrease. The sweet point of $k$ might be different for
each dataset, _e.g_. on average, $k=5$ has the lowest cost as shown in the
curves, but on some datasets, especially ones that contain extractive
questions which does not need multi-hop reasoning (like NarrativeQA and
QMSum), $k=1$ leads to the lowest cost. This indicates that the optimal $k$
depends on the nature of the task, as well as the performance requirement. We
encourage future researchers to look for different $k$s when applying our
method to various applications.

|  |  | Avg | Narr | Qasp | Mult | Hotp | 2Wiki | Musi | Sum | En.QA | En.MC
---|---|---|---|---|---|---|---|---|---|---|---|---
| 1 | LC | 49.70 | 32.76 | 47.83 | 52.33 | 61.85 | 62.96 | 40.22 | 20.73 | 43.08 | 85.57
Dragon | 2 | RAG | 38.09 | 21.91 | 44.33 | 53.08 | 51.61 | 50.05 | 30.47 | 19.93 | 21.25 | 50.22
3 | combine | 46.81 | 28.50 | 43.82 | 54.62 | 56.58 | 60.62 | 40.66 | 20.07 | 37.79 | 78.60
4 | RAG ratio | 77.88 | 74.00 | 84.00 | 97.33 | 86.00 | 77.00 | 66.00 | 95.50 | 61.25 | 59.83
5 | Token ratio | 37.87 | 19.31 | 54.15 | 34.78 | 32.64 | 55.65 | 48.16 | 16.64 | 38.71 | 40.83

Table 2: Results for Gemini-1.5-Pro using Dragon retriever.

###  5.2 Why does RAG fail?

To gain a better understanding of why RAG lags behind LC, we analyze the
failure reasons for the examples that cannot be answered by RAG. We first
manually check some examples for which our RAG-and-Route step predicts
“unanswerable” and summarize four typical failure reasons, then prompt LLM to
classify all the examples.

The four reasons include: (A) The query requires multi-step reasoning so the
results of previous steps are needed to retrieve information for later steps,
_e.g_. ‘‘What nationality is the performer of song XXX’’. (B) The query is
general, _e.g_. ‘‘What does the group think about XXX’’, which is challenging
for the retriever to formulate a good query. (C) The query is long and
complex, which is challenging for the retriever to understand. However,
answering this kind of questions is arguably, an advantage of LLMs. (D) The
query is implicit, demanding a thorough understanding of the entire context.
For instance, in a lengthy conversational narrative about a space voyage, a
question like ‘‘What caused the shadow behind the spaceship?’’ requires
readers to connect the dots and deduce the answer, as there is no explicit
mention of the shadow when the cause is revealed.

Figure 4: Distribution of typical RAG failure reasons.

Using these reasons, we prompt Gemini-1.5-Pro with few-shot in-context
examples that we manually annotated, to classify all the unanswerable examples
into these four categories, plus an “other" option. Fig. 4 shows the
distribution of failure reasons on the seven datasets in LongBench. Each
dataset may contain different number of RAG failure cases, resulting in
various bar heights. The distribution patterns are consistent with the nature
of the datasets. For example, the three Wikipedia-based multi-hop reasoning
datasets (HotpotQA, 2WikiMQA, MuSiQue) are challenging for RAG because of
multi-step retrieval as shown in blue. For NarrativeQA, which are long stories
containing a lot of dialogues, most failure cases are due to implicit queries
that requires understanding the whole context (shown in green). For QMSum,
which is a summarization dataset contains open-ended questions, failures are
mostly due to general queries (shown in red). We manually checked the examples
classified as “others” and find that most of them are actually multi-step
questions, often with ambiguities, which poses challenges for answering.

We hope this failure analysis inspires future improvements of RAG. For
example, engaging chain-of-thought Wei et al. (2022) into RAG may help address
the multi-step questions, and revisiting query understanding techniques like
query expansion Lv and Zhai (2009); Zhai and Lafferty (2001) may help with the
general queries and complex queries. We are also glad to see recent efforts
towards the direction Chan et al. (2024); Ma et al. (2023).

###  5.3 Different retrievers

The results using a retriever, Dragon, is shown in Tab. 2 based on
Gemini-1.5-Pro. As can be seen, the results are consistent with Contriever,
for all of LC, RAG, and Self-Route, showing that our findings are
generalizable across retrievers.

###  5.4 Results on synthetic data

In this study, we mainly focus on real datasets, with a consideration that
results on synthetic data, which are artificially created by researchers, may
subject to dataset artifacts. We notice some methods that researchers adopted
to create synthetic long context datasets may unconsciously, but largely,
influence the performance comparison between RAG and LC. For example, here we
describe the results on the “PassKey” dataset in $\infty$Bench and its
variations.

This “PassKey” dataset presents a needle-in-a-haystack test, where a sentence
with a passkey (_e.g_. ‘‘the passkey is 123456’’) is hidden within chunks of
irrelevant text, and the model is asked to answer the question ‘‘What is the
passkey’’. The task requires strong retrieval capability. On this dataset, RAG
achieves 80.34% accuracy, outperforming LC, which gets 65.25% using
Gemini-1.5-Pro. However, if the query is slightly modified as ‘‘What is the
special token hidden inside the texts’’, RAG accuracy sharply drops to only
4.58%, while LC keeps roughly the same (69.32%). Another example: if the
chunks contain two passkeys and the query is ‘‘Which passkey is larger? First
or second?’’, then RAG (47.63%) under-performs LC (64.24%) as well. These
examples demonstrate that the evaluation results highly subjects to artifacts
in dataset construction, showing limitation of synthetic testing.

##  6 conclusion

This paper presents a comprehensive comparison of RAG and LC, highlighting the
trade-offs between performance and computational cost. While LC demonstrate
superior performance in long-context understanding, RAG remains a viable
option due to its lower cost and advantages when the input considerably
exceeds the model’s context window size. Our proposed method, which
dynamically routes queries based on model self-reflection, effectively
combines the strengths of both RAG and LC, achieving comparable performance to
LC at a significantly reduced cost. We believe our findings contribute
valuable insights for the practical application of long-context LLMs and pave
the way for future research in optimizing RAG techniques.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * An et al. (2023) Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023.  L-eval: Instituting standardized evaluation for long context language models.  _arXiv preprint arXiv:2307.11088_.
  * Anthropic (2024) Anthropic. 2024.  Claude 3.5 sonnet.  https://www.anthropic.com/news/claude-3-5-sonnet/.
  * Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023.  Self-rag: Learning to retrieve, generate, and critique through self-reflection.  _arXiv preprint arXiv:2310.11511_.
  * Bai et al. (2023) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023.  Longbench: A bilingual, multitask benchmark for long context understanding.  _arXiv preprint arXiv:2308.14508_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Chan et al. (2024) Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024.  Rq-rag: Learning to refine queries for retrieval augmented generation.  _arXiv preprint arXiv:2404.00610_.
  * Chen et al. (2023a) Lingjiao Chen, Matei Zaharia, and James Zou. 2023a.  Frugalgpt: How to use large language models while reducing cost and improving performance.  _arXiv preprint arXiv:2305.05176_.
  * Chen et al. (2023b) Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023b.  Extending context window of large language models via positional interpolation.  _arXiv preprint arXiv:2306.15595_.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021.  A dataset of information-seeking questions and answers anchored in research papers.  _arXiv preprint arXiv:2105.03011_.
  * Google (2024) Google. 2024.  Gemini pricing.  https://ai.google.dev/pricing.
  * Greg Kamradt (2023) Greg Kamradt. 2023.  Needle in a haystack - pressure testing llms.  https://github.com/gkamradt/LLMTest_NeedleInAHaystack.
  * Guo et al. (2022) Mandy Guo, Joshua Ainslie, David C Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022.  Longt5: Efficient text-to-text transformer for long sequences.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 724–736.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020.  Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.  _arXiv preprint arXiv:2011.01060_.
  * Hsieh et al. (2024) Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024.  Ruler: What’s the real context size of your long-context language models?  _arXiv preprint arXiv:2404.06654_.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  _arXiv preprint arXiv:2305.02301_.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_.
  * Izacard and Grave (2020) Gautier Izacard and Edouard Grave. 2020.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_.
  * Jiang et al. (2023) Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023.  Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression.  _arXiv preprint arXiv:2310.06839_.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019\.  Generalization through memorization: Nearest neighbor language models.  _arXiv preprint arXiv:1911.00172_.
  * Kočiskỳ et al. (2018) Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The narrativeqa reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Kuratov et al. (2024) Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024.  In search of needles in a 10m haystack: Recurrent memory finds what llms miss.  _arXiv preprint arXiv:2402.10790_.
  * Levy et al. (2024) Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024.  Same task, more tokens: the impact of input length on the reasoning performance of large language models.  _arXiv preprint arXiv:2402.14848_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2023) Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023.  Llatrieval: Llm-verified retrieval for verifiable generation.  _arXiv preprint arXiv:2311.07838_.
  * Lin et al. (2023) Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023.  How to train your dragon: Diverse augmentation towards generalizable dense retrieval.  _arXiv preprint arXiv:2302.07452_.
  * Liu et al. (2024) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024.  Lost in the middle: How language models use long contexts.  _Transactions of the Association for Computational Linguistics_ , 12:157–173.
  * Lv and Zhai (2009) Yuanhua Lv and ChengXiang Zhai. 2009.  Adaptive relevance feedback in information retrieval.  In _Proceedings of the 18th ACM conference on Information and knowledge management_ , pages 255–264.
  * Ma et al. (2023) Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023.  Query rewriting for retrieval-augmented large language models.  _arXiv preprint arXiv:2305.14283_.
  * Maharana et al. (2024) Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024.  Evaluating very long-term conversational memory of llm agents.  _arXiv preprint arXiv:2402.17753_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-3.5-turbo.  https://platform.openai.com/docs/models/gpt-3-5-turbo.
  * OpenAI (2024a) OpenAI. 2024a.  Gpt-4o.  https://openai.com/index/hello-gpt-4o/.
  * OpenAI (2024b) OpenAI. 2024b.  Openai-api pricing.  https://platform.openai.com/docs/overview.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_.
  * Shaham et al. (2022) Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022.  Scrolls: Standardized comparison over long language sequences.  _arXiv preprint arXiv:2201.03533_.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_.
  * Song et al. (2024) Mingyang Song, Mao Zheng, and Xuan Luo. 2024.  Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models.  _arXiv preprint arXiv:2403.11802_.
  * Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022\.  Musique: Multihop questions via single-hop question composition.  _Transactions of the Association for Computational Linguistics_ , 10:539–554.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Xu et al. (2023) Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023.  Retrieval meets long context large language models.  _arXiv preprint arXiv:2310.03025_.
  * Yan et al. (2024) Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024.  Corrective retrieval augmented generation.  _arXiv preprint arXiv:2401.15884_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_.
  * Yuan et al. (2024) Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024.  Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k.  _arXiv preprint arXiv:2402.05136_.
  * Zhai and Lafferty (2001) Chengxiang Zhai and John Lafferty. 2001.  Model-based feedback in the language modeling approach to information retrieval.  In _Proceedings of the tenth international conference on Information and knowledge management_ , pages 403–410.
  * Zhang et al. (2024) Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024.  Infinity bench: Extending long context evaluation beyond 100k tokens.  _arXiv preprint arXiv:2402.13718_.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021.  Qmsum: A new benchmark for query-based multi-domain meeting summarization.  _arXiv preprint arXiv:2104.05938_.

##  Appendix A Dataset details

We evaluate on 7 datasets from LongBench Bai et al. (2023). NarrativeQA
Kočiskỳ et al. (2018) is a question answering dataset, where the context is a
long story like a novel or a movie script. Qasper Dasigi et al. (2021) focuses
on question answering over academic NLP papers and is annotated by NLP
practitioners. MultiFieldQA, originally proposed in LongBench, contains human-
annotated QA over documents and articles from multiple sources, including
legal documents, government reports, encyclopedias, academic papers, etc.
HotpotQA Yang et al. (2018) contains two-hop questions written by native
English speakers that requires reasoning over two related Wikipedia paragraphs
in the long context. 2WikiMultihopQA Ho et al. (2020) contains up to 5-hop
questions that are synthesized through manually designed templates, ensuring
that they cannot be solved through shortcuts. The questions in MuSiQue Trivedi
et al. (2022) are up to 4-hop, first constructed from single-hop question
compositions, and then paraphrased by annotators for linguistic diversity.
QMSum Zhong et al. (2021) is a query-based summarization dataset over meeting
scripts from multiple domains.

We evaluate on 2 datasets from $\infty$Bench Zhang et al. (2024). En.QA
contains human-annotated question-answer pairs for long novels, with key
entity names manually replaced in order to avoid knowledge leakage due to
model pretraining. EN.MC is annotated similarly to En.QA, but differs in that
the model is presented with four challenging answer choices written by the
annotators.

Tab. 3 shows the details of the datasets, including the number of queries in
each evaluation dataset and the average context length (_i.e_. number of
words).

|  | Num. Query | Avg. Length
---|---|---|---
LongBench Bai et al. (2023) | NarrativeQA | 200 | 18,395
Qasper | 200 | 3,599
MultiFieldQA | 150 | 4,539
HotpotQA | 200 | 9,133
2WikiMultihopQA | 200 | 4,873
MuSiQue | 200 | 11,196
QMSum | 200 | 10,533
$\infty$Bench Zhang et al. (2024) | En.QA | 351 | 150,374
En.MC | 229 | 142,622
Table 3: Dataset statistics.

##  Appendix B Ablations of k

Tab. 4 shows the performance and token ratio for different $k$, which
corresponds to Fig. 3. The performance of LC, which serves as an upper bound,
is 45.53. The token ratio is computed the token counts for RAG or Self-Route
divided the number of tokens required by LC.

| Performance | token ratio
---|---|---
top-k | RAG | Self-Route | RAG | Self-Route
1 | 20.24 | 41.35 | 5.26 | 39.64
5 | 37.92 | 43.33 | 17.02 | 38.63
10 | 41.20 | 44.38 | 42.42 | 53.66
50 | 44.06 | 45.19 | 95.29 | 102.97
100 | 44.12 | 45.23 | 100.32 | 106.59
Table 4: Performance and token ratio for different $k$. This table corresponds
to Fig. 3.

##  Appendix C Prompts

Tab. 5 shows the prompts for each dataset in our study. The prompts are
modified from the released prompts as in LongBench Bai et al. (2023) and
$\infty$Bench Zhang et al. (2024). Tab. 6 shows the prompts used in the
failure case study as in Sec. 5.2.

NarrativeQA |  You are given a story, which can be either a novel or a movie script, and a question. Answer the question as concisely as you can, using a single phrase if possible. Do not provide any explanation. If the question cannot be answered based on the information in the article, write “unanswerable”. Story: {context} Now, answer the question based on the story as concisely as you can, using a single phrase if possible. Do not provide any explanation. If the question cannot be answered based on the information in the article, write “unanswerable”. Question: {input} Answer:
---|---
Qasper |  You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write “unanswerable”. If the question is a yes/no question, answer “yes”, “no”, or “unanswerable”. Do not provide any explanation. Article: {context} Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write “unanswerable”. If the question is a yes/no question, answer “yes”, “no”, or “unanswerable”. Do not provide any explanation. Question: input Answer:
MultiFQA |  Read the following text and answer briefly. {context} Now, answer the following question based on the above text, only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. Question: {input} Answer:
HotpotQA |  Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. Question: {input} Answer:
2WikiMQA |  Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. Question: {input} Answer:
MuSiQue |  Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. Question: {input} Answer:
QMSum |  You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences. If the question cannot be answered based on the information in the article, write “unanswerable”. Transcript: {context} Now, answer the query based on the above meeting transcript in one or more sentences. If the question cannot be answered based on the information in the article, write “unanswerable”. Query: {input} Answer:
EN.QA |  Read the book and answer the question. Be very concise in your answer. If the question cannot be answered based on the information in the article, write “unanswerable”. {context} Question: {input} Only give me the answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. Answer:
EN.MC |  Read the book and answer the question. If the question cannot be answered based on the information in the article, write “unanswerable”. {context} Question: {input} {all_classes} Only output the letter of the correct answer and do not output any other words. If the question cannot be answered based on the information in the article, write “unanswerable”. The letter of the correct answer is

Table 5: Prompts for each dataset.

You are given some text chunks from an article, and a question. The text
chunks are retrieved by an external retriever. Now: (1) Tell whether the
question can be answered based only on the provided text chunks. (2) If the
question can be answered, answer the question based on the texts as concisely
as you can, using a single phrase if possible. (3) If the question cannot be
answered, choose the reason from the following: A. The question needs
multistep reasoning, thus it is hard to retrieve all the relevant chunks. For
example, "What nationality is the performer of song You Can?" contains two
steps: find the performer, then find the nationality of the performer. Other
examples include "Where does the director of film Wine Of Morning work at?",
"What is another notable work made by the author of Miss Sara Sampson?" B. The
question is a general query, thus it is hard to retrieve relevant chunks. For
example, "What did the group think about Dave leaving?" is general because the
group may include multiple persons, and they can have different thinkings. C.
The question is long and complex, which is hard for the retriever to encode it
to retrieve relevant chunks. For example, "What did Julie Morgan elaborate on
the online survey when talking about the evaluations on the legitimacy of the
children’s rights, protection and demands?", "The Huskies football team were
invited to the Alamo Bowl where they were defeated by a team coached by Art
Briles and who played their home games at what stadium?" D. The question is
not explicit and requires comprehensive understanding of the whole story and
cannot be solved using retrieval-augmented generation. For example, "What
caused the shadow behind Koerber’s ship?" needs a comprehensive understanding
of the whole story. Another example like "How many words are there in the
article" also requires the complete article. E. Others. Keep the above reasons
in mind, and choose the most possible reason if you think the question cannot
be answered based on the text. Output the results in JSON format.
{in_context_examples} Text: {context} Question: {input} Answer:
---

Table 6: Prompt for the failure case analysis.