# The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented
Generation

Eric Yang $\dagger$<EMAIL_ADDRESS>
Verily Jonathan Amar $\dagger$<EMAIL_ADDRESS>
Verily Jong Ha Lee<EMAIL_ADDRESS>
Verily Bhawesh Kumar<EMAIL_ADDRESS>
Verily Yugang Jia<EMAIL_ADDRESS>
Verily

###### Abstract

Digital health chatbots powered by Large Language Models (LLMs) have the
potential to significantly improve personal health management for chronic
conditions by providing accessible and on-demand health coaching and question-
answering. However, these chatbots risk providing unverified and inaccurate
information because LLMs generate responses based on patterns learned from
diverse internet data. Retrieval Augmented Generation (RAG) can help mitigate
hallucinations and inaccuracies in LLM responses by grounding it on reliable
content. However, efficiently and accurately retrieving most relevant set of
content for real-time user questions remains a challenge.

In this work, we introduce Query-Based Retrieval Augmented Generation (QB-
RAG), a novel approach that pre-computes a database of potential queries from
a content base using LLMs. For an incoming patient question, QB-RAG
efficiently matches it against this pre-generated query database using vector
search, improving alignment between user questions and the content. We
establish a theoretical foundation for QB-RAG and provide a comparative
analysis of existing retrieval enhancement techniques for RAG systems.
Finally, our empirical evaluation demonstrates that QB-RAG significantly
improves the accuracy of healthcare question answering, paving the way for
robust and trustworthy LLM applications in digital health.

Keywords: RAG, LLM, Healthcare, Digital Health.

11footnotetext: $\dagger$ Equal contribution.

##  1 Introduction

Large Language Models (LLMs), such as OpenAI’s GPT (OpenAI, 2024), Meta’s
Llama (Touvron et al., 2023) and Google’s Palm (Anil et al., 2023) and Gemini
(Team et al., 2023) suite, have demonstrated remarkable capabilities across a
diverse set of natural language understanding and generation tasks. This has
led to several novel applications of language models across different domains.
In healthcare, LLMs hold immense promise for developing conversational AI
systems that can answer patient questions, offer personalized health advice,
and potentially improve access to care, particularly for underserved
populations (Clusmann et al., 2023; Peng et al., 2023; Thirunavukarasu et al.,
2023; Alowais et al., 2023; Nori et al., 2023; Singhal et al., 2022, 2023; Tu
et al., 2024).

However, applying LLMs in healthcare presents significant challenges in
ensuring the accuracy, reliability, safety, and adherence to the latest
healthcare practices in the information they provide. The probabilistic nature
of LLMs, combined with limitations and potential biases in their training
data, can lead to the generation of fabricated or nonsensical information,
known as hallucinations. This poses significant risks in a healthcare context
where accurate information is paramount. Additionally, the knowledge embedded
within an LLM is limited to the data it was trained on and may not reflect the
most current medical guidelines and best practices. To address these
challenges, further fine-tuning, instruction tuning (Wei et al., 2022) and
additional reinforcement learning (Ouyang et al., 2022) approaches can be
considered. Common approaches involve training the models on specific datasets
containing the most up-to-date information and/or preferred responses.
However, fine-tuning has its limitations: datasets can be difficult and
expensive to acquire, especially in healthcare and the computation cost may be
prohibitive.

Retrieval Augmented Generation (RAG) (Lewis et al., 2020) offers a promising
alternative by grounding LLM responses in a curated and reliable knowledge
base of vetted information. Unlike relying solely on the LLM’s internal
knowledge, RAG systems retrieve relevant information for a specific user query
from this external knowledge base to inform the LLM’s answer generation
process. This grounding helps to mitigate hallucinations, promotes factual
accuracy, and allows developers to incorporate the latest knowledge.
Applications using RAG systems have been shown to be quite compelling compare
to traditional LLM fine tuning (Gupta et al., 2024; Ovadia et al., 2024).
However, the effectiveness of RAG hinges critically on its ability to
accurately retrieve the most pertinent information from the knowledge base, a
task complicated by the inherent semantic gap between natural language user
queries and the way information is structured and stored within a knowledge
base (Ma et al., 2023). This “retrieval challenge” is a significant bottleneck
for building effective RAG systems.

Several approaches have attempted to overcome this retrieval challenge by
improving the alignment between user queries and the knowledge base. Some
methods leverage the generative capabilities of LLMs themselves. For instance,
Query2Doc (Wang et al., 2023) and HyDE (Gao et al., 2022) propose generating a
hypothetical document that would ideally answer the user’s query. This
synthetic document, more semantically aligned with the knowledge base content,
is then used as the retrieval key. Similarly, QA-RAG (Kim and Min, 2024) takes
a two-pronged approach: first generating a candidate answer using a fine-tuned
LLM, and then leveraging both the initial query and this candidate answer to
retrieve relevant information from the knowledge base. (Ma et al., 2023)
proposes duplicating and/or splitting user queries and then retrieving content
similar to either rewritten query. Alternatively, some methods focus on fine-
tuning embedding models to create semantically aligned representations of
queries and documents (Karpukhin et al., 2020). Despite these efforts,
efficiently bridging the semantic gap between user queries and knowledge bases
for accurate and reliable information retrieval in RAG systems remains an
active area of research.

###  1.1 Contributions

This paper introduces a novel framework for analyzing and enhancing the
retrieval phase of RAG systems. We propose that effective retrieval hinges on
aligning user queries with the knowledge base across distinct semantic
representations: query, answer, and content. To the best of our knowledge, we
are the first to explicitly recognize these distinct “semantic spaces” for
improving retrieval. While prior methods have explored alignment based on
answer or content representations, our work proposes direct alignment within
the query space itself. Existing approaches often implicitly combine these
distinct representations, leading to sub-optimal retrieval. In contrast, we
systematically disentangle these semantic spaces and demonstrate the
advantages of our query based alignment strategy. Specifically, we make the
following contributions:

  1. 1.

QB-RAG: Query-Based Retrieval Augmented Generation: We propose QB-RAG, a novel
retrieval method for RAG systems that addresses the misalignment between user
queries and content bases by pre-generating a comprehensive set of questions
from the knowledge base. Distinct from the methods that rely on generate
hypothetical documents or candidate answers, QB-RAG directly maps incoming
queries to this offline-generated question set using efficient vector search,
enabling efficient, direct query-to-query comparison for enhanced retrieval.

  2. 2.

Theoretical Foundation for Query-Space Alignment: We develop a theoretical
framework that formalizes the retrieval challenge in RAG systems and
elucidates the advantages of query-space alignment. Our analysis underscores
the inherent semantic gap between user queries and content embeddings,
demonstrating how QB-RAG effectively bridges this gap by operating directly
within the query space.

  3. 3.

Survey and Comparative Analysis of Retrieval Strategies: We present a
comprehensive survey of prominent retrieval strategies for RAG systems,
including naive RAG, QA-RAG, HyDE, and our proposed QB-RAG method. We analyze
how each approach addresses the semantic gap between user queries and the
knowledge base, comparing their effectiveness through quantitative evaluation
on our healthcare dataset. This analysis provides insights into the trade-offs
involved in selecting a particular retrieval method for a specific
application.

  4. 4.

Comprehensive Benchmark and Empirical Validation: To evaluate the performance
of various retrieval methods, we conduct rigorous experiments on a healthcare
question-answering dataset generated from our proprietary healthcare content
database. Our evaluation encompasses both retrieval accuracy (using metrics
like exact recovery rate and relevance scores) and the quality of the
downstream generated answers (assessed using faithfulness, relevancy, and
accuracy metrics). This comprehensive benchmark provides valuable insights
into the strengths and limitations of each retrieval strategy, showcasing QB-
RAG’s superiority on our question-answering dataset. While our evaluation
focuses on healthcare question-answering, QB-RAG’s underlying principles and
advantages may be broadly applicable to various domains and RAG applications.

The remainder of this paper is structured as follows. Section 2 introduces our
healthcare-focused content base and details our methodology for extracting a
comprehensive set of questions from these materials. In Section 3, we
formalize the RAG objective, highlighting the challenges of retrieval
alignment. We then present QB-RAG as a solution, theoretically motivating its
effectiveness in bridging the semantic gap between queries and content.
Section 4 outlines our benchmark setup, encompassing a range of prominent
retrieval methods and evaluation metrics. Finally, Section 5 presents the
results of our empirical evaluation, demonstrating QB-RAG’s superiority in
retrieval accuracy and downstream answer quality.

##  2 Application and Dataset

###  2.1 Description

Digital health programs represent a paradigm shift in managing chronic
conditions such as Type 2 diabetes (T2D) and Hypertension (HTN). They can
improve health markers associated with chronic conditions by delivering
personalized care and support directly to patients (Majithia et al. (2020)).
These programs often utilize a variety of digital tools, including mobile
apps, messaging, wearable devices, and telehealth services, to facilitate
remote monitoring, provide educational content, and offer guidance on
lifestyle modifications. The overarching goal is to equip patients with the
knowledge and tools necessary to effectively self-manage their condition,
enhance their overall quality of life, and potentially reduce the need for
frequent traditional healthcare visits.

An integral component of these digital health programs is the messaging
platform they incorporate. This platform is not merely a channel for
communication; it is an integrated system designed to provide timely,
personalized, and interactive support to patients. The effectiveness of these
platforms hinges on their ability to engage patients and provide relevant,
timely information in response to their queries. However, the increasing
demand for immediate healthcare information often strains the capacity of
human healthcare providers to deliver timely support through messaging.

To overcome this scalability challenge, digital health platforms are
increasingly turning to meticulously curated content repositories designed to
provide readily available answers to common patient questions. This study
utilizes such a content repository, referred to as “Content Cards,” developed
to address frequently asked questions (FAQs) from patients with T2D and HTN
participating in our proprietary digital health programs. This dataset,
encompassing 630 English-language content cards, covers a comprehensive range
of topics pertinent to managing T2D and HTN. This includes general health
information, detailed guidance on using platform-specific features and
connected devices (e.g., blood glucose monitors), and personalized dietary
recommendations. Figure 1 illustrates how these Content Cards are presented
within our mobile application, while the excerpt below provides a
representative sample of the content:

If there is not enough blood on the test strip, you may not get an accurate
blood glucose reading. Some meters won’t even give you a reading at all. So
here are some tips to help you get a big enough drop of blood: Rub your hands
in warm water to get the blood to your fingertips. Shake your hand to help
force blood to your fingertips. Hold your hand down by your side for 30
seconds to help blood run to your fingertips. Set your lancing device to
puncture just deep enough to get the size drop you need. (This may take some
trial and error.) Take blood from the side of your finger. There are fewer
nerves there, so it doesn’t hurt as much. If your lancet isn’t going deep
enough, dial the number up higher on the lancing device. Once you’ve lanced,
squeeze your finger where it meets your palm and move toward the tip of your
finger.

Figure 1: Illustration of Content Cards in Verily’s Onduo mobile application,
these cover broad range of topics, from operational health management,
nutrition and physical advice, etc.

###  2.2 Offline Question Generation

The core premise of our work is to improve content retrieval in RAG systems by
directly aligning incoming user queries with a pre-computed set of questions
derived from our content base. This requires generating a comprehensive set of
questions that are answerable by our content, which we achieve through a two-
step process. First, we employ a base prompt to generate a preliminary set of
questions for each content card. Subsequently, we leverage an LLM-based
answerability model (detailed in Section 2.2.2) to filter out irrelevant or
nonsensical questions, ensuring the quality and relevance of our question set.

####  2.2.1 Base prompt

To generate an initial set of questions, we design a prompt (shown in listing
1) with clear instructions for question generation and few-shot examples to
illustrate the desired style and format. This prompt incorporates a
num_questions parameter to specify the target number of questions per content
card. While we set this parameter to 20, the actual number of questions
generated by the LLM may vary. Applying this prompt to our content base of 630
cards resulted in over 8,000 potential questions.

⬇

1***SYSTEM:***

2You are a Teacher / Professor. Your task is to setup {num_questions}
questions for an upcoming quiz/examination. The questions should be both
diverse and exhaustive in nature across the document. Restrict the questions
to the context information provided.

3

4***INSTRUCTIONS:***

5You are presented with a text authored by healthcare professionals, offering
advice and strategies for managing conditions such as diabetes and high blood
pressure. Your task is to formulate relevant questions that the text is
written to address. Closely follow the example questions for style and
structure when formulating your own question for the provided text. Your
generated questions should be in first person with details, but only at a high
school reading level. Your questions should be answerable from the text, but
do not copy the text verbatim. MAKE SURE to generate at least {num_questions}
questions. Format the generated questions separated by comma in the following
JSON format with "questions" as its key: {{"questions":
["...","...","...","..."]}}

6

7***EXAMPLE OUTPUTS:***

8Sample Text: {example content}

9Generated Questions: {example JSON with list of num_question questions}

10

11\--------------------------------------------

12Given the context information and no prior knowledge, generate the relevant
questions.

13Text:

14{cc_text}

15Generated Questions:

Listing 1: Base prompt for question generation

####  2.2.2 Answerability model

After generating questions using an LLM for our content cards, we noticed that
our output questions were sometimes not relevant to the content, or were
formatted inappropriately. To mitigate this issue and filter out irrelevant
questions, we developed an LLM-based answerability model. Specifically, for
each content card and generated question pair, we prompt the LLM to assess
whether or not the content contains relevant information to answer the
question. Leveraging the LLM’s reasoning abilities, we structure the prompt to
elicit a step-by-step explanation of the answerability judgment. The LLM is
instructed to first identify the key information required to answer the
question and then determine if that information is present within the content.
The LLM also outputs a “Yes” or “No” label which we use to filter out
questions that are deemed irrelevant to the corresponding content card.

Applying this answerability model effectively filters out irrelevant or
nonsensical questions generated in the initial phase. This filtering process
resulted in a refined set of approximately 4,800 answerable questions derived
from our 630 content cards. Each question in this curated set is directly
mapped back to its source content card, which serves as the “golden” source
for a correct answer.

To validate the effectiveness and reliability of our answerability model, we
randomly selected 100 question and content card pairs and evaluated them using
both our automated model and three clinical experts. The clinical experts
rated the answerability of each pair on a three-point scale: “Content answers
completely,” “Content answers partially,” or “Content does not answer.” The
results demonstrated strong agreement between our model’s assessments and the
clinical expert’s judgments. Specifically, 90% of the pairs received the same
answerability rating, with an additional 9% categorized as “partially
answerable,” indicating some degree of inherent subjectivity in the evaluation
task.

This high level of agreement, coupled with the increasing recognition of LLMs
as reliable auto-evaluation tools Lee et al. (2023), supports the validity of
our answerability model for filtering irrelevant questions. We provide the
prompt used for our answerability model in Listing 2.

⬇

1***SYSTEM***

2You are a health coach providing support for members living with diabetes.
You have some basic healthcare and nutrition knowledge.

3

4***INSTRUCTIONS***

5Given a pair of user query and a paragraph of content, determine if the
content contains relevant information to infer an answer to the query. Think
step by step. First provide an explanation, then generate a "Yes" or "No"
label. Put the results in a Python dictionary format with keys "Explanation"
and "Source relevant".

6

7***EXAMPLES***

8For example, given the following query and content as inputs: {positive and
negative examples, with explanations}

9

10\--------------------------------------------

11Provide the output for the following query and content:

12Question: {question}

13Content: {content}

Listing 2: Prompt for Answerability Auto-Evaluator

####  2.2.3 Analysis of Generated Question Similarity

To assess the potential of our question-based retrieval approach, we analyzed
the semantic similarity among the questions generated using the prompt in
Listing 1. Our goal was to determine if questions derived from the same
content card (intra-content similarity) exhibit higher similarity than
questions generated from different cards (inter-content similarity).

We calculated pairwise cosine similarity between question embeddings generated
using Google’s textembedding-gecko model. As shown in Table 1, mean intra-
content similarity (0.871) is indeed higher than mean inter-content similarity
(0.827). Similarly, questions are, on average, more similar to their source
content card (0.837) than to other content cards (0.762). We note that the
relatively high similarity between questions and content from different cards
likely stems from the inherent thematic overlap within our dataset, as all
content and questions focus on T2D and HTN management.

These results provide initial evidence supporting the premise of our question-
based retrieval strategy. While the specific similarity values are influenced
by our dataset and embedding model, the consistent trend of higher intra-
content similarity suggests that aligning incoming user queries with this
question set can improve retrieval accuracy in RAG systems.

Average Cosine Similarity b/w | Value
---|---
Questions generated from the same content | .871
Questions generated from a different content | .827
Questions and their associated content | .837
Questions and other contents | .762
Table 1: Cosine similarities between questions and contents combination

##  3 Mathematical Formulation

This section presents a mathematical framework for analyzing retrieval within
RAG systems, highlighting the limitations of conventional approaches and
motivating our proposed QB-RAG method. We formalize the retrieval objective,
elucidating the misalignment between user queries and content representations.
Subsequently, we introduce QB-RAG and provide a theoretical rationale for its
efficacy in addressing this misalignment.

###  3.1 Notation

Consider the set of $M$ content documents
$\mathcal{C}=\\{c_{1},\cdots,c_{M}\\}$ appropriately chunked for embedding.
Similarly, let $\mathcal{Q}=\\{q_{1},\cdots,q_{N}\\}$ represent the set of $N$
questions generated from the content documents as described in section 2.2.
Unless otherwise specified, $c\in\mathcal{C}$ and $q\in\mathcal{Q}$ will refer
to their respective embedding representations.

We note that in much of the RAG related literature, the embedder for the query
and content base are often the same. As mentioned, this typically leads to
misaligned retrieval when comparing query embeddings to content embeddings
through cosine similarity. Given cosine similarity as the metric used in our
retrieval experiments, we will assume without loss of generality that all
embeddings are normalized, i.e. $\forall c\in\mathcal{C},\forall
q\in\mathcal{Q}:\>\lVert c\rVert_{2}=\lVert q\rVert_{2}=1$. We also define the
distance function as the cosine distance $d(x,y)=1-\frac{x^{t}y}{\lVert
x\rVert\lVert y\rVert}$.

The question generation process in section 2.2 is a one to many process, that
is from one content we generate multiple questions. To encode this
relationship, we denote the matrix $A\in\\{0,1\\}^{M\times N}$ where
$A_{ij}=\mathds{1}[c_{i}\text{ generated }q_{j}]$, where $\mathds{1}$ denotes
the indicator function. Per our generation process and the relevance
evaluation, it is understood that $A_{ij}=1\Rightarrow q_{j}\text{ can be
answered using }c_{i}$. However the converse is not necessarily true as
potentially different contents may be able to answer the same question. Thus
we define $A^{*}\in\\{0,1\\}^{M\times N}$ which is the dense unobserved matrix
such that $A^{*}_{ij}=\mathds{1}[q_{j}\text{ can be answered using }c_{i}]$.
This implies that $A$ is a partial observation of $A^{*}$, which we call the
oracle matrix. We point out that following our question generation process
described in section 2, we have $\forall c_{j}\in\mathcal{C}:\>\exists
q_{i}\in\mathcal{Q}\text{ s.t. }A_{ij}=1$.

For a new user question $q_{0}$, traditional RAG systems retrieve content that
maximize some measure of similarity $\arg\max_{c\in\mathcal{C}}{c^{t}q_{0}}$
(or equivalently $\arg\min_{c\in\mathcal{C}}d(c,q_{0})$). More advanced RAG
systems rely on a specific functional form $f$, ideally aligned with the
objective of downstream generation and retrieves content maximizing
$\arg\max_{c\in\mathcal{C}}f({c^{t},q_{0}})$. We acknowledge that in general,
even for traditional RAGs, retrieving such a content may be challenging
especially for very large content bases. This has led to various key highly
efficient retrieval algorithms (e.g. ScaNN111
https://blog.research.google/2020/07/announcing-scann-efficient-vector.html ),
and our work takes the above as a solved problem. Further in RAG systems, we
usually retrieve a subset $\mathcal{S}\subset\mathcal{C}$ content pieces of
size $\lvert\mathcal{S}\rvert=k$ solving
$\arg\min_{\mathcal{S}}\sum_{c\in\mathcal{S}}{d(c,q_{0})}$. Oftentimes,
different retrieval methods encourage the retrieved set to be diverse to
improve the downstream retrieval; these ad-hoc methods are not the focus of
our work.

For simplicity in our mathematical formulation, we will assume the retrieval
to mean a single piece of content in this section, where RAG retrieves the
most similar content to the incoming user query. We use Google’s
textembedding-gecko in the current work for generating embeddings. We denote
the embedding matrices for content base and questions, respectively, with
capital letters $C\in\mathbb{R}^{d\times M}$ and $Q\in\mathbb{R}^{d\times N}$
where $d$ is the dimension of the chosen embedder, in our case 768.

As a general note, we usually denote sets as calligraphic, vectors as lower
case, matrices as upper case, and numbers/indices lower or upper case.

###  3.2 Ideal Retrieval Objective

Ideally during the retrieval phase, for a new user query $q_{0}$, one would
evaluate all documents $c\in\mathcal{C}$ by asking the retrieval question “Can
the query $q_{0}$ be answered using the content $c$?”. We call this the
retrieval task and define the retrieval function $f^{*}(c,q_{0})\in\\{0,1\\}$.
Note this is the exact formulation of the matrix $A^{*}$ for the generated
questions, i.e. $A^{*}=(f^{*}(c_{i},q_{j}))_{ij}$.

State-of-the-art LLMs have shown their ability to accurately approximate the
retrieval function (which we validate in our experimental section), which
means that we can in-principle solve the retrieval task above at scale for any
incoming query. However, in most practical application, using LLMs for
approximating the retrieval function is both time and cost prohibitive, as it
would involves $M$ LLM predictions for an online user. This necessitates
exploring alternate schemes.

Rather than pursuing this exact operation for every entry, RAG systems
typically approximate the retrieval task with some proxy. Below we list a few
of the most common attempts at approximating the retrieval task at scale. The
underlying challenge is how should one best approximate the retrieval function
using practically scalable operations, for instance, using embedding
similarities.

  * •

The vanilla version of RAG approximates $f^{*}(c,q_{0})$ (and in particular
its induced ranking) by $c^{t}q$. We will show below that this approximation,
albeit very wide spread, is typically misaligned and can lead to retrieval
failures.

  * •

Methods like HyDE (Gao et al., 2022), Query2Doc (Wang et al., 2023), and QA-
RAG (Kim and Min, 2024) try to address the alignment issue by leveraging LLMs
to rewrite the user query into a form more aligned with the content space.
These approaches implicitly approximate the retrieval function as
$f^{*}(c,q_{0})\approx c^{t}p_{LLM}(q_{0})$, where $p_{LLM}(q_{0})$ represents
the LLM’s output conditioned on the original query $q_{0}$.The effectiveness
of these methods to indirectly align query and content space hinges on the
LLM’s ability to generate representations that are both semantically rich and
well-aligned with the content embeddings, which can be inconsistent and
resource-intensive.

  * •

Adapter based methods (e.g. LlamaIndex222
https://docs.llamaindex.ai/en/stable/examples/finetuning/embeddings/finetune_embedding_adapter.html)
approximates the retrieval function by $c^{t}U(q)$ where $U$ is an adapter
function which must be learnt online, for example, with a single layer linear
layer or even a deep neural net. However, acquiring sufficient training data
for optimal adapter performance can be challenging. An alternative approach
involves training separate, specialized embedders for queries and content
(Karpukhin et al., 2020), aiming to learn inherently aligned representations.

  * •

Similar to adapter-based methods, this approach involves training or fine-
tuning embedding models to generate representations specifically optimized for
the retrieval task. Given a dataset of query-content pairs, these methods
learn an embedding function $e(.)$ that maximizes the similarity between
relevant queries and content. The retrieval function is then approximated as
$f^{*}(c,q_{0})\approx e(c)^{t}e(q_{0})$. Commercially available tools, such
as those on VertexAI333https://cloud.google.com/vertex-ai/generative-
ai/docs/models/tune-embeddings, offer pre-trained or fine-tuned embedding
models for specific domains and tasks. This method essentially tries to handle
alignment in a supervised way. However, this approach requires a substantial
amount of labeled data for training, which can be costly and time-consuming to
acquire, especially in specialized domains like healthcare.

We have intentionally excluded methods like re-ranking from this analysis.
While re-ranking techniques can be applied subsequent to any of the
aforementioned retrieval strategies, they don’t fundamentally alter the
initial embedding-based retrieval process. Re-rankers typically leverage LLMs
to evaluate the retrieval function $f^{*}(c,q_{0})$ on a smaller subset of
documents $\mathcal{\tilde{C}}\subset\mathcal{C}$ that have been pre-selected
using embedding similarity. This allows for a more precise approximation of
$f^{*}(c,q_{0})$ but only within the confines of the initially retrieved
subset. Determining the optimal subset for re-ranking remains an open
challenge. Various frameworks, such as those offered by
cohereAI444https://cohere.com/rerank/, implement re-ranking mechanisms.

Finally, in the context of RAG – i.e. identifying relevant content for a
downstream generation – we need only worry about finding some $c$ s.t.
$f^{*}(c,q_{0})=1$. This simplifies the problem as we need to find contents
that score high values following the chosen approximation function, rather
than accurately estimating the entire function.

###  3.3 Misalignment

As many have previously noted (Gao et al., 2022; Wang et al., 2023; Kim and
Min, 2024), directly using cosine similarity for retrieval within RAG systems
can be problematic since these representations often reside in different
semantic spaces, leading to misaligned retrieval. To clearly illustrate this,
consider the following sentences and their corresponding embeddings:

  * •

$q_{1}=$ “Where was Queen Elizabeth born?”

  * •

$q_{2}=$ “Where was the Queen of Spain born?”

  * •

$c_{1}=$ “The former monarch of the British empire was born in London. Corgis
are amazing fun dogs.”

  * •

$c_{2}=$ “The Spanish monarch was born in Madrid.”

  * •

$c_{3}=$ “The Queen of Spain was born in May.”

In this example clearly $c_{1}$ answers $q_{1}$ and $c_{2}$ answers $q_{2}$.
Yet after calculating embeddings similarities using our text embedder,
$d(q_{1},q_{2})=.12<d(q_{1},c_{2})=.28<d(q_{1},c_{1})=.32$. This implies that
the query $q_{1}$ is geometrically closer to the semantically similar query
$q_{2}$ than it is to the content $c_{1}$ that actually contains the
answer.mplying that the query $q_{1}$ is geometrically closer to the
semantically similar query $q_{2}$ than it is to the content $c_{1}$ that
actually contains the answer. A similar issue arises with
$d(q_{1},q_{2})=.12<d(q_{2},c_{3})=.30<d(q_{2},c_{2})$. This simple toy
illustration underscores the limitations of traditional retrieval based solely
on embedding similarity.

To overcome this alignment issue, some methods propose leveraging an LLM to
draft a fake document or a candidate answer from the LLM’s implicit knowledge
– see HyDE (Gao et al., 2022), Query2Doc (Wang et al., 2023), see QA-RAG (Kim
and Min, 2024) – with a hope that these transformations and their embeddings
would better align with those in $\mathcal{C}$ given we are now comparing
documents to documents (or similarly answers to documents). These approaches
are valuable as they recognize the misalignment, but are challenging to
evaluate in an offline setting in a principled way. Their success depends not
only on the LLM’s ability to comprehend the query and generate a relevant
response (which may be challenging in specialized applications) but also on
its capacity to produce answers that are semantically aligned with the
specific content within the RAG system’s knowledge base.

We now connect this back with the ideal retrieval objective defined above. As
we illustrated, when the inherent similarity between query and content
embeddings fails to reflect their true relevance, the retrieval process is
fundamentally hindered, often resulting in the retrieval of irrelevant
content. This underscores the need for a more robust alignment strategy within
RAG systems, one that can accurately capture the semantic relationship between
queries and content, regardless of their inherent embedding representations.
In the following section, we introduce QB-RAG, a novel approach that directly
addresses this challenge by aligning incoming queries with a pre-computed set
of questions derived from the content base and performing retrieval within a
shared query space.

###  3.4 QB-RAG

To address the semantic misalignment inherent in traditional RAG retrieval, we
propose Query-Based Retrieval Augmented Generation (QB-RAG). Our approach
leverages a pre-computed set of questions, $\mathcal{Q}$, derived from the
content base $\mathcal{C}$. These questions are generated offline as described
in Section 2.2, mitigating concerns about online computational overhead. We
not that efficient retrieval algorithms render the increased knowledge base
size introduced by incorporating $\mathcal{Q}$ negligible. In contrast to
methods relying on online LLM calls for query rewriting, QB-RAG shifts this
computational burden offline. This distinction is significant, as online
rewriting necessitates serial LLM invocation, potentially introducing latency
detrimental to user experience. Furthermore, QB-RAG’s direct alignment within
the query space offers a more transparent and interpretable retrieval process
compared to the implicit alignment strategies of LLM-based rewriting
techniques.

####  3.4.1 Vanilla QB-RAG

Our vanilla approach of QB-RAG first generates an extensive set of questions
that are known to be answered by the content by initializing $\mathcal{Q}$ and
$A$. This operation can happen offline and upon uploading new documents to our
content base. Second, for an online query, QB-RAG searches for similar
question (resp. questions) within $\mathcal{Q}$ by finding
$\arg\min_{\mathcal{Q}}d(q_{0},q)$ (resp.
$\arg\min_{\mathcal{S}\subset\mathcal{Q}}\sum_{q\in\mathcal{S}}{d(q,q_{0})}$).
Given we are now comparing questions to questions, we expect the distance
measure to be calibrated given the comparison is also aligned. Third upon
retrieving similar questions, the associated contents are fetched, and fed to
the generative LLM like other RAG systems (after dropping duplicate contents
if necessary). We provide the full details in Algorithm 1.

Algorithm 1 Vanilla QB-RAG

$\mathcal{C},\mathcal{Q},A$ (or $A^{*},\hat{A}$), new query $q_{0}$, target
number of retrievals $k\leq M$

Similarities $z\leftarrow Q^{t}q_{0}\in\mathbb{R}^{N}$

Sort $z$ by descending values and sort $\mathcal{Q}\&A$’s columns accordingly.

$\mathcal{S}\leftarrow\\{\\}$

for $j\in[1..N]$ do

$i\leftarrow i$ s.t. $A_{ij}=1$ identify the content associated associated to
the $j$-th question.

Note, if $A$ is non-sparse, we break ties by favoring new content, then by
number of associated questions, then at random.

$\mathcal{S}\leftarrow\mathcal{S}\cup\\{c_{i}\\}$ if $c_{i}\notin\mathcal{S}$

If $\lvert\mathcal{S}\rvert=k$ then break

end for

Return $\mathcal{S}$

####  3.4.2 Oracle and Approximate QB-RAG

We now turn our attention to the best possible retrieval system using a query
based alignment and retriever. We previously noted that the matrix $A$
indicating which content generated which question, is actually a sparse
partial observation of $A^{*}$ indicating which content is relevant to a
question. For this section we will assume that we have access to the oracle
matrix $A^{*}$ or some approximation of it denoted $\hat{A}$. For simplicity,
we describe the algorithms with ${A}^{*}$, but is also applicable to
$\hat{A}$.

Before getting into the algorithmic details, let us first motivate the use of
this matrix $A^{*}$. We note that while computing the matrix completely would
involve $N\times M$ LLM calls, these can all happen offline. Further depending
on the dataset at hand, this may not even prohibitive in cost. E.g. in our
dataset, this would lead to more than 5000 offline calls. Let us point out
though that exactly computing $A^{*}$ is not necessary, and some approximation
of it could also lead to improved retrieval. In fact, we propose two ways of
computing the estimate $\hat{A}$:

  1. 1.

Only compute matrix entries $A^{*}_{ij}$ such that $c_{i}^{t}q_{j}\geq\lambda$
where lambda is some threshold, which can be chosen in order to compute only
some percentile of entries. This typically filters out combinations of
contents and queries that are likely not relevant to each other.

  2. 2.

After computing a set of entries (whether chosen at random or according to the
above rule), we can rely on matrix completion techniques to infer and
approximate the remainder of the matrix. Matrix completion is highly efficient
on moderate to large matrix sizes.

Given the oracle $A^{*}$ or some estimate $\hat{A}$ thereof, we can adapt
Algorithm 1 to incorporate the non-sparse nature of $A^{*}$ or $\hat{A}$.
Specifically, since multiple content pieces may be associated with a single
question, we introduce tie-breaking rules within the algorithm that first
prioritize newer content (especially relevant to healthcare). We then
prioritizes contents capable of answering a diverse range of questions within
$\mathcal{Q}$. This reflects our intuition that a such content is likely to be
more broadly relevant.

One could also explore incorporating the many-to-many relationship by
leveraging a weighted majority vote into the retrieval step, rather than
simple tie breaking rules. We defer this to future research as well.

##  4 Benchmark

###  4.1 Methods

This section outlines the retrieval methods evaluated in our benchmark. To
ensure a fair comparison, each method retrieves the same number of documents,
denoted by $k$, for a given query (the value of $k$ is varied across
experiments). All LLM generations utilize Gemini-Pro and text embeddings are
generated using textembedding-gecko – both are commercially available tools.

  1. 1.

Naive RAG (Lewis et al., 2020). This baseline approach retrieves content based
on the cosine similarity between the embedded query and candidate content
using identical embedder, a standard practice in RAG systems Finally the query
and retrieved contents are fed into an answer generator LLM and the LLM
generates a response.

  2. 2.

HyDE (Gao et al., 2022). HyDE addresses query-content misalignment by first
generating a hypothetical document that answers the query. Specifically, an
instruction tuned LLM is prompted to generate a hypothetical pseudo-document
by setting the instruction “Write a paragraph that answers the question …” –
this pseudo-document is not real. Second this pseudo-document and its
embeddings are used for retrieval by searching for similar real contents,
offloading the relevance modeling to the LLM’s generation.
Query2Doc (Wang et al., 2023). Similarly to HyDE, Query2Doc employs few-shot
prompting to generate a hypothetical pseudo-document that would answer the
query. It concatenates the initial query with the synthetic document
(generated similar to HyDE) and then performs dense retrieval using the
embeddings of this concatenated text. Given the similarity with HyDE, we only
implement HyDE in our benchmark.

  3. 3.

QA-RAG (Kim and Min, 2024). In their methodology, they fine tune a foundation
LLM offline using a golden set of Q&A. Upon receiving a new query online, they
generate a pseudo-answer using the fine tuned LLM. They then retrieve contents
using the query embeddings only, and other contents using the pseudo-answer’s
embeddings only. A Reranker is then deployed to select only the most relevant
contents among this dual set. In our implementation of QA-RAG, we use Gemini-
Pro out of the box to generate a pseudo-answer, and retrieve $k$ documents
using the pseudo-answer only, without a downstream reranker.

  4. 4.

Vanilla QB-RAG: We run the simplest version of our method (see Algorithm 1)
using the matrix $A$ which directly encodes “which content generated which
question”. We believe that having access to the oracle matrix $A^{*}$ or some
approximation of it would be even more promising. Furthermore, given this
method is strongly dependent on the questions generated offline (in section
2), we parameterize QB-RAG-${\bar{m}}$ where $\bar{m}$ measures the average
number of questions generated per content. Notably, $\bar{m}$ indicates the
“coverage” extracted from each content, implying that higher coverage should
lead to improved retrieval quality and, consequently, better-generated
answers. We decrease $\bar{m}$ by down-sampling the set of questions in our
experiments, to effectively reduce the coverage of our question base. The
maximum value from our generation is $\bar{m}=8$ which includes our entire
question set.

We point out that the methods presented above are not mutually exclusive and
can be combined to potentially achieve improved performance, as illustrated by
QA-RAG (Kim and Min, 2024). Specifically, we could retrieve contents using a
combination of above methods and/or associated approached for generating
embeddings, and subsequently leverage a re-ranker to select the most relevant
documents. However, our experiments focus on isolating and evaluating the
efficacy of each method in isolation.

Finally, we also highlight several other advancement introduced in RAG:  1.
Modifications prior to retrieval by rewriting or breaking down the query (e.g.
ITER-RETGEN (Shao et al., 2023)) in order to incorporate contextual
information;  2. enhancements during retrieval by fine tuning the embedder
(Karpukhin et al., 2020), training an adapter, retrieving diverse documents
using MMR;  3. post retrieval refinements by reranking (or filtering, see
(Nogueira et al., 2019)) the contents retrieved using an evaluator LLM,
alternatively building a mixture LLM distributions (see REPLUG (Shi et al.,
2023)), or even fine tuning the generator LLM to better utilize the retrieved
contents (see RA-DIT (Lin et al., 2023)).  While these techniques hold
potential for enhancing retrieval and answer generation across all evaluated
methods, we omit them from our analysis to maintain a focused comparison. This
approach ensures an independent evaluation of the core retrieval mechanisms,
as these enhancements can be incorporated without fundamentally altering the
underlying methods.

###  4.2 Answer Generation

After retrieving relevant content using the methods described above, we employ
an LLM (Gemini-Pro) for answer generation. For each question, the retrieved
content is provided as context to the LLM along with the following prompt to
generate the answer:

⬇

1Use only the provided pieces of context to answer the question at the end.
Think step-by-step and then answer. Respond in 3 to 6 short sentences.

2

3Do not try to make up an answer:

4- If the context does not contain enough relevant information to determine an
answer to the query, say "I cannot determine the answer to that."

5- If the context is empty, just say "I do not know the answer to that."

6

7Answer in an empathetic and positive tone. Do not use phrases such as
"According to the context", and directly answer the question.

8

9Contexts: {contexts}

10Question: {question}

11Answer:

Listing 3: Base prompt for answer generation

###  4.3 Test Data Generation

We construct two distinct test sets designed to assess the performance of the
various retrieval methods under different conditions. Each test set consists
of questions answerable by our content base, ensuring relevance to the task.
Crucially, no test question appears verbatim within the pre-generated question
set used for retrieval. All test questions are generated using Gemini-Pro..

  1. 1.

Rephrase: To generate this test set, we prompted an LLM to rephrase each of
the 4.8k questions in the knowledge base. Then, 500 of those were randomly
sampled to yield the first test set. This approach was done to simulate
scenarios where the knowledge base contains a comprehensive set of questions.
In such scenarios, the intents of new incoming questions are more likely to be
represented in the knowledge base.

  2. 2.

Out-of-Distribution: To generate the second test set, we prompted an LLM to
generate a new question for each of the 630 contents. The LLM was instructed
to not generate a question that already exists in the question knowledge base
for that content. Then, the 630 newly generated questions were filtered via
the answerability model from Section 2 to ensure the new questions were indeed
answerable by the corresponding content. After filtering, 305 newly generated
questions made up the second test set.

###  4.4 Metrics

To rigorously assess the performance of our proposed QB-RAG method, we employ
two distinct sets of metrics: those evaluating the quality of content
retrieval and those assessing the quality of the generated answers. While QB-
RAG’s primary objective is to enhance retrieval, we hypothesize that this
improvement will translate to better-generated answers. Given the emphasis on
reliability and safety in healthcare applications, we carefully selected
answer quality metrics tailored to this domain. Since our content base
represents a vetted source of trusted information, we prioritize answer
faithfulness, penalizing any unwarranted extrapolation or information not
grounded in the provided content. Additionally, we recognize the critical
importance of a model’s ability to decline answering when the content lacks
clarity, a crucial aspect of mitigating potential risks in healthcare
settings.

####  4.4.1 Retrieval Evaluation

  * •

Exact Recovery Rate: This metric measures the percentage of test questions for
which the retrieved set of $k$ documents includes the exact content piece used
to generate the original question. We acknowledge that while exact recovery
represents retrieval of “golden content”, multiple contents might offer
relevant information for a single question. Therefore, we incorporate
additional relevance measures described below.

  * •

Auto-evaluator Relevancy Rate: This metric addresses the limitations of
relying solely on exact matches by leveraging the answerability model (Section
2) to gauge content relevance. Specifically, it calculates the percentage of
test questions for which at least one retrieved document is deemed relevant by
the answerability model. This automated assessment has demonstrated strong
correlation with human judgments as as described in section 2.

  * •

Average Re-ranker Relevancy Score: To evaluate the relevance of each retrieved
document individually, we employ the BGE re-ranker (Xiao et al., 2023). This
method assigns a relevance score (higher indicating greater relevance) between
each retrieved document and its corresponding test question. For each test
question, we take the maximum relevance score among the $k$ retrieved
documents and average these maximum scores across all test questions. The
effectiveness of the BGE re-ranker in relevance scoring has been previously
established in RAG research (Kim and Min, 2024).

####  4.4.2 Answer Evaluation

  * •

Answer Guideline Adherence Rate: This metric measures how well a generated
answer aligns with a predefined guideline derived from a “golden answer.” The
assessment involves a three-step process. Initially, an LLM generates a
“golden answer” for each test question using the associated content.
Subsequently, another LLM analyzes this golden answer to create a guideline
outlining the key elements an accurate response should contain. Finally, a
third LLM evaluates the candidate answer against this extracted guideline,
assigning a score from 0 to 1 based on the extent to which the answer covers
the guideline’s key points. While this guideline-based approach aims to
capture the nuances of answer quality, it relies heavily on the accuracy of
both the golden answer and the extracted rubric. To provide a more robust and
multifaceted evaluation, we introduce additional metrics that directly assess
distinct aspects of answer quality below.

  * •

Answer Relevancy Rate: This metric evaluates whether a generated answer
directly address the user’s question and provides a self-contained response.
An LLM classifies each answer as either relevant (YES) or not relevant (NO) to
the corresponding test question, focusing solely on the relevance, not the
accuracy or factual correctness of the answer.

  * •

Answer Faithfulness Rate: This evaluates weather the content supplied to the
LLM during generation supports the generated answer. An answer is deemed
faithful if any portion of the provided content supports it, regardless of the
presence of irrelevant content. This component is crucial to ensure that the
answer generated is grounded on our content, a crucial feature for many
healthcare applications. Ensuring the answer is grounded on the supplied
content will mitigate the hallucination risk. Note that declined answers are
considered not faithful in our definition.

  * •

Answer Declined Rate: This metric assesses whether the LLM declined to answer
the question. This may occur because our answer generation couldn’t find
relevant information in the content (we specifically prompt the LLM to only
answer questions when there is a retrieved content.)

##  5 Experimental Results

In this section we discuss the performance of our methods, QB-RAG-8 (resp. QB-
RAG-2) where the knowledge base has an average of 8 (resp. 2) queries per
content, which are compared against the incoming query for retrieval. We
assess the impact of QB-RAG on both retrieval efficacy and the quality of
generated answers.

###  5.1 Effect of QB-RAG on Retrieval Efficacy

On the Rephrase test set, where the knowledge base is expected to contain
questions semantically similar to the test questions, QB-RAG-8 consistently
outperforms all benchmark methods. As shown in table 2, QB-RAG-8 nearly
doubles the exact recovery rate compared to other methods (e.g., from 45% to
89% when retrieving a single document). This suggests that a comprehensive,
query-aligned knowledge base, as constructed by QB-RAG-8, substantially
improves the retrieval of the exact source document. These impressive gains
highlight the scenario where our knowledge base is comprehensive and covers
quite broadly the extent of questions our documents can answer.

The exact recovery rate does not present the full picture. When analyzing the
Auto-evaluator Relevancy and the Re-ranker score (refer section 4.4.1), our
method still dominates the baselines when retrieving a single content. As
measured by the LLM, the content we retrieve are relevant 68% of the time,
whereas traditional methods retrieve relevant content around 40% of the time.
In table 2, we also include the average re-ranker relevancy score. QB-RAG-8
yields the highest average relevance score using the BGE re-ranker.

Increasing the number of retrieved documents to 3 further illustrates the
efficacy of QB-RAG-8. The exact recovery rate reaches a near-optimal 97%,
substantially higher than the baseline approaches. QB-RAG-8 also achieves
almost 20% higher relevancy rate compared to the baselines. Similarly, the
average re-ranker relevancy score is significantly higher for QB-RAG. These
findings underscore the robustness of QB-RAG-8 in retrieving both the target
document and a set of relevant documents.

Retrieval assessment Rephrase |  Exact Recovery Rate |  Auto-evaluator Relevancy Rate |  Avg. Re-ranker Relevancy Score
---|---|---|---
Number of retrieved docs | 1 | 3 | 1 | 3 | 1 | 3
QB-RAG-8 | 0.89 | 0.97 | 0.68 | 0.76 | 1.21 | 1.65
QB-RAG-2 | 0.59 | 0.75 | 0.53 | 0.66 | –0.25 | 0.92
Naive RAG | 0.44 | 0.61 | 0.39 | 0.56 | -0.96 | 0.35
QA-RAG | 0.45 | 0.60 | 0.41 | 0.56 | -1.52 | -0.38
HyDE | 0.47 | 0.63 | 0.41 | 0.56 | -1.14 | -0.09
Table 2: Retrieval performance (higher is better) of methods on Rephrase,
where documents were retrieved given rephrased questions from the content
base.

On the more challenging Out-of-Distribution test set, QB-RAG-8 maintains its
advantage, albeit with smaller gains. Table 3 shows that QB-RAG-8 improves the
exact recovery rate by 1.3% to 6.2% compared to benchmark methods. Despite the
adversarial nature of this test set, where incoming questions are
intentionally dissimilar to the training set, QB-RAG-8 consistently retrieves
the correct source document more often.

Furthermore, QB-RAG-8 demonstrates a significant improvement in relevancy. The
auto-evaluator relevancy rate shows gains of 8% to 15% over baseline methods,
indicating that QB-RAG-8 effectively identifies relevant content even when the
query distribution shifts. These improvements in retrieval quality directly
benefit downstream answer generation, as more relevant content is likely to
lead to more accurate and informative answers. This is further supported by
the consistent gains observed in the average relevance scores from the BGE re-
ranker.

Retrieval assessment Out-of-Dist |  Exact Recovery Rate |  Auto-evaluator Relevancy Rate |  Avg. Re-ranker Relevancy Score
---|---|---|---
Number of retrieved docs | 1 | 3 | 1 | 3 | 1 | 3
QB-RAG-8 | 0.53 | 0.72 | 0.58 | 0.75 | 0.47 | 1.68
QB-RAG-2 | 0.42 | 0.57 | 0.50 | 0.64 | -0.54 | 0.78
Naive RAG | 0.50 | 0.68 | 0.50 | 0.64 | -0.03 | 1.25
QA-RAG | 0.51 | 0.66 | 0.47 | 0.60 | -0.48 | 0.53
HyDE | 0.52 | 0.70 | 0.48 | 0.65 | -0.01 | 1.20
Table 3: Retrieval performance (higher is better) of methods on Out-of-
Distribution, where documents were retrieved given newly generated questions.

###  5.2 Effect of QB-RAG on Generated Answer Quality

We now examine how the improved retrieval accuracy of QB-RAG translates to the
quality of generated answers. Recall that the Rephrase test set favors our
query-based approach as test questions are semantically similar to those in
our generated knowledge base.

As shown in Table 4, both QB-RAG-8 and QB-RAG-2 consistently outperform the
benchmark methods on all answer quality metrics for the Rephrase test set.
Notably, QB-RAG-8 achieves an 84% answer faithfulness rate, significantly
surpassing the 62%-68% rates of the baseline methods. This suggests that by
accurately retrieving the most relevant content, QB-RAG enables the LLM to
generate answers that are well-grounded in the provided information.
Additionally, QB-RAG achieves the highest accuracy rate, indicating its
answers effectively address the key elements outlined in the pre-defined
guidelines (refer to Section 4.4.2).

Answer assessment Rephrase |  Declined Rate |  Faithfulness Rate |  Relevancy Rate |  Accuracy Rate
---|---|---|---|---
QB-RAG-8 | .12 | .84 | .83 | .79
QB-RAG-2 | .21 | .74 | .73 | .73
Naive RAG | .30 | .67 | .66 | .68
QA-RAG | .33 | .62 | .62 | .62
HyDE | .30 | .68 | .66 | .67
Table 4: Answer quality of methods on Rephrase, where documents were retrieved
given rephrased questions from the content base. 3 documents retrieved.

These patterns hold on the more challenging Out-of-Distribution test set, as
seen in table 5. There QB-RAG-8 achieves 78% faithfulness, fairing higher than
the benchmarks 68%-74%. However QB-RAG-2 under-performs, highlighting its
inability to generalize to new questions.

Answer assessment Out-of-Dist |  Declined Rate |  Faithfulness Rate |  Relevancy Rate |  Accuracy Rate
---|---|---|---|---
QB-RAG-8 | .17 | .78 | .77 | .69
QB-RAG-2 | .29 | .66 | .65 | .63
Naive RAG | .27 | .72 | .70 | .64
QA-RAG | .29 | .68 | .67 | .63
HyDE | .24 | .74 | .72 | .68
Table 5: Answer quality of methods on Out-of-Distribution, where documents
were retrieved given newly generated questions. 3 documents retrieved.

An answer can be deemed unfaithful either because the answer is not grounded,
or because the LLM declined to answer. We note that the higher faithfulness
rate of QB-RAG-8 is largely due to improved retrieval, resulting in a decline
rate of only 12% on the Rephrase test set and 17% on the Out-of-Distribution
test set, compared to significantly higher rates for the baselines. We observe
a slight increase in answers that are not grounded, from $\approx 2\%-3\%$ on
the Rephrase test set to $\approx 3\%-5\%$ on the Out-of-Distribution test set
for QB-RAG. Importantly, we achieve the lowest unfaithfulness rate (3%) on the
Rephrase test set with QB-RAG-8, further underscoring the value of generating
a comprehensive question set for optimal coverage.

While QB-RAG significantly improves answer faithfulness through better
retrieval, it’s worth noting that the groundedness aspect of faithfulness rate
is ultimately determined by the capabilities of the answer generation module
itself. Further enhancements to answer faithfulness could involve techniques
like preference modeling and RLHF where the LLM is specifically for this
objective (Bai et al., 2022).

###  5.3 Effect of Coverage of Generated Questions

To examine the relationship between the breadth of the generated question set
and QB-RAG’s effectiveness, we compare the performance of QB-RAG-2 and QB-
RAG-8. This analysis reveals a strong dependence on question coverage.

On the Rephrase test set, QB-RAG-2, despite its reduced question set, still
surpasses other retrieval methods (table 2). However, the magnitude of
improvement is noticeably smaller compared to QB-RAG-8. For instance, QB-RAG-2
shows a 10-15% improvement in exact recovery rate over baselines, whereas QB-
RAG-8 nearly doubles the exact recovery rate. This pattern also holds for
relevancy rate and average re-ranker scores, indicating that a larger, more
comprehensive question set translates to more effective retrieval.

The Out-of-Distribution test set (table 3) reveals a more pronounced impact of
question base coverage. Here, QB-RAG-2’s performance drops below that of some
benchmark methods, with the exact recovery rate decreasing by 7-12%.
Interestingly, QB-RAG-2 still achieves comparable performance on relevancy-
based metrics, suggesting that even a limited question base can partially
capture relevant content, but may not pinpoint the exact source document as
effectively.

This sensitivity analysis clearly shows that the performance of QB-RAG methods
are very much tied to our ability to generate a comprehensive set of questions
from RAG knowledge base. As we increase the number of generated questions in
our database, we expect to see better retrieval, higher exact matches and
higher relevance. Beyond simply increasing the number of questions, improving
their diversity may be another way for further improving QB-RAG’s performance.
By generating a more diverse set of questions for each content piece, we can
capture a broader range of semantic nuances and user intents. This is
particularly relevant given that modern retrieval algorithms efficiently
handle large knowledge bases, making quantity less of a limiting factor in
doing efficient retrieval.

This sensitivity to question coverage is further evident in the quality of the
generated answers (Tables 4 and 5). QB-RAG-8 consistently leads to more
faithful, relevant, and accurate answers compared to QB-RAG-2, directly
reflecting the differences observed in their retrieval performance. These
findings highlight that QB-RAG’s success in downstream tasks is fundamentally
linked to its ability to construct and leverage a comprehensive and diverse
question set that effectively captures the content and semantic nuances of
incoming queries within the RAG knowledge base.

###  5.4 Case Study

To more concretely demonstrate the advantages of QB-RAG in our application, we
present an example query from the out-of-distribution test case. The results
of Naive RAG, QA-RAG and HyDE for the given test query are as follows.

Test query:

What is the difference between soluble and insoluble fiber and how do they
affect the body?

Retrieved content from all non QB-RAG methods:

Fiber is the “roughage” in plants that gives them their shape. Your body can’t
digest it (it just passes through), so why do we need to eat it? You’ve
probably heard that fiber helps keep you regular. Fiber can also help lower
your cholesterol, manage your blood glucose, and keep you feeling full longer.
And it can help you lose weight. How much fiber is enough? For women, experts
recommend 25 grams of fiber each day. Men need about 35 grams. Ready to add
more fiber to your diet? Try a couple of these tips this week: Choose 100%
whole-grain bread or tortillas instead of white bread or white-flour
tortillas. Eat brown rice instead of white rice. Fill half your plate with
low-carb vegetables. Grab an apple instead of applesauce. Swap in black bean
or navy bean soup instead of chicken noodle soup. Choose oatmeal or oat bran
instead of corn flakes or puffed rice cereal. Munch on popcorn or kale chips
instead of pretzels. Consider trying a fiber-enriched food. A fiber-enriched
breakfast bar has around one-third of your daily fiber! A quick note: Fiber is
wonderful, but add it to your diet slowly. Your body may need a little time to
adjust. As you add fiber, drink more fluids to keep it moving through. If you
have any problems with gas or bloating, there are over-the-counter that can
help. Which sources of fiber look good to you?

Answer generated using retrieved content from all non QB-RAG methods:

I cannot determine the answer to that. The provided context does not mention
the difference between soluble and insoluble fiber or how they affect the
body.

Given the query about soluble and insoluble fiber, Naive RAG, QA-RAG and HyDE
all retrieved a content about fiber as the top content. While on topic, the
content does not contain information to differentiate the effect of soluble
and insoluble fiber on the human body, leading to the answer generator
declining to answer. The results of QB-RAG for the same test query are as
follows.

Retrieved query by QB-RAG:

How does soluble fiber help the body?

Mapped content of retrieved query by QB-RAG:

Steel-cut oatmeal (also called Irish oatmeal) is an easy, healthy, high-fiber
food. Steel-cut oats have more fiber than rolled or instant oats. Steel-cut
oats have both soluble and insoluble fiber. Soluble fiber helps lower
cholesterol and keeps blood glucose from rising quickly. Insoluble fiber helps
move food through your body, which can keep your bowels regular. The fiber in
oatmeal also makes it a very filling food. You can eat a small portion size
and feel full longer, which is great for weight loss. Want to boost your
oatmeal flavor? Try these ideas: Cinnamon Slivered almonds or other nuts (2
Tbsp) Pumpkin pie spices without sugar Almond butter (1 Tbsp) or other nut
butter Powdered peanut butter Powdered protein powder (mix in after cooking
for a protein and flavor boost) Stevia (sugar-free sweetener) A dash of
vanilla extract Tips: Oatmeal has carbs so keep a small portion size in mind.
Milk adds carbs. Consider using a cup of regular milk, unsweetened almond
milk, or unsweetened coconut milk as low-carb options. Try eating oatmeal with
a protein food (like eggs, nuts, or peanut butter) for a balanced meal.

Answer generated using mapped content of retrieved query by QB-RAG:

Soluble fiber helps lower cholesterol and keeps blood glucose from rising
quickly, while insoluble fiber helps move food through your body, which can
keep your bowels regular.

Given the same test query, QB-RAG first retrieved a pre-generated query about
soluble fiber stored in the knowledge base. It then maps the retrieved query
to the content that was used to generate the query. Given the content, the
answer generator was able to provide a response to the test query. Note that
while the retrieved content is largely about steel-cut oatmeal, it contains
information about soluble and insoluble fiber and their effect on the body. On
the surface, it may seem that the content retrieved by the other methods is
more on topic, QB-RAG was able to pinpoint the exact content that contains the
needed specific information with the help of the pre-generated query.

##  6 Discussion

This paper introduces QB-RAG, a novel approach for enhancing the retrieval
phase of RAG systems. QB-RAG addresses the inherent semantic misalignment
between user queries and knowledge bases by pre-generating a comprehensive set
of potential questions directly from the content. We then leverage efficient
vector search to map incoming online queries to these pre-computed questions,
facilitating a more accurate and aligned retrieval process. While QB-RAG
requires a significant offline computational investment for question
generation, this trade-off is strategically advantageous. Unlike conventional
methods that rely on online LLM calls for query rewriting or enhancement, QB-
RAG shifts this computational burden offline. This distinction is crucial for
minimizing latency in real-time applications, as online LLM invocations can
significantly impact the user experience. Furthermore, QB-RAG benefits from
the efficiency of highly optimized retrieval algorithms, even when handling
large question sets, ensuring rapid content retrieval for online queries. We
have provided a theoretical motivations for QB-RAG and we have also suggested
potential avenues for further improving QB-RAG, such as expanding the question
set for greater coverage or leveraging matrix completion techniques to infer a
complete answerability matrix.

To assess QB-RAG’s effectiveness, we designed two distinct test sets: the
Rephrase set, where test questions are semantically similar to those in our
pre-generated question base, and the Out-of-Distribution set, where questions
are intentionally crafted to be dissimilar. These test sets represent cases
designed to evaluate QB-RAG’s robustness under varying conditions, including a
potential distribution shift. Our empirical results demonstrate that QB-RAG
consistently outperforms benchmark methods across both test sets. While QB-RAG
significantly improves exact recovery and relevance of retrieved documents,
its impact extends beyond these initial retrieval metrics. Downstream answer
generation using QB-RAG-retrieved content scores higher on key quality
metrics, including faithfulness, relevancy, and adherence to pre-defined
guidelines. Crucially, answers generated using QB-RAG are more frequently
grounded on our trusted source of content, which is key to deliver reliable,
up to date information to patient queries.

Our sensitivity analysis shows that effectiveness of our approach is highly
tied to coverage of extracted questions. Beyond increasing the raw number of
questions, we also suggest favoring a diverse set of questions during the
generation phase. This can be done potentially with prompt engineering,
supervised fine-tuning or RLHF frameworks. Metrics like BERTScore (Zhang et
al., 2019) and Word Mover’s Distance (Kusner et al., 2015) can be used to
evaluate the diversity of generated questions from the RAG knowledge base.
Finally, an LLM could also be potentially used as an auto-evaluator for
measuring diversity of generated questions. We note that retrieval frameworks
are highly optimized even for large question bases, thus, we don’t expect
quantity of generated questions as a bottleneck for many real-world
applications.

Despite its novelty and effectiveness, our work has several limitations that
require further study. First, our evaluation focused on a curated healthcare
content base consisting of well-crafted, concise pieces. Assessing QB-RAG’s
performance on larger, more diverse datasets, including those from other
domains and containing less structured or longer-form content, remains an
important area for future research.

Another limitation stems from our reliance on LLM-based evaluation metrics in
the current study. While LLMs have shown promise in assessing answer quality
according to predefined guidelines, the real-world deployment of QB-RAG in
healthcare settings necessitates thorough expert review. Direct patient
interactions demand the highest level of scrutiny to ensure the accuracy,
safety, and reliability of generated information. Our automated evaluation
serves as a valuable first step, but expert validation remains paramount.

Finally, it’s crucial to acknowledge a broader challenge inherent to all RAG
systems, particularly in fast-changing domains like healthcare: the quality of
any RAG system’s output is fundamentally dependent on the quality and recency
of its underlying knowledge base. In healthcare, where medical consensus and
best practices are constantly evolving, content needs to be regularly reviewed
and updated to reflect the latest advancements. This underscores the
importance of developing robust mechanisms for ongoing knowledge base
maintenance and ensuring alignment with current clinical guidelines,
regardless of the specific retrieval methods employed.

Despite these limitations, our work demonstrates the potential of QB-RAG for
enhancing retrieval accuracy and answer groundedness in RAG systems, paving
the way for more reliable and trustworthy LLM applications in healthcare.
Beyond direct patient interaction, QB-RAG could also be valuable for
augmenting coaching and provider resources within digital health platforms.
For instance, QB-RAG could be used to generate draft answers that healthcare
professionals can then review and refine, ensuring both efficiency and
accuracy. Ultimately, real-world deployment of QB-RAG should be carefully
considered, taking into account specific use cases, potential risks, and the
need for appropriate expert oversight.

## References

  * Alowais et al. (2023) Shuroug A Alowais, Sahar S Alghamdi, Nada Alsuhebany, Tariq Alqahtani, Abdulrahman I Alshaya, Sumaya N Almohareb, Atheer Aldairem, Mohammed Alrashed, Khalid Bin Saleh, Hisham A Badreldin, et al.  Revolutionizing healthcare: the role of artificial intelligence in clinical practice.  _BMC medical education_ , 23(1):689, 2023.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Clusmann et al. (2023) Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al.  The future landscape of large language models in medicine.  _Communications medicine_ , 3(1):141, 2023.
  * Gao et al. (2022) Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan.  Precise zero-shot dense retrieval without relevance labels.  _arXiv preprint arXiv:2212.10496_ , 2022.
  * Gupta et al. (2024) Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al.  Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture.  _arXiv preprint arXiv:2401.08406_ , 2024.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ , 2020.
  * Kim and Min (2024) Jaewoong Kim and Moohong Min.  From rag to qa-rag: Integrating generative ai for pharmaceutical regulatory compliance process.  _arXiv preprint arXiv:2402.01717_ , 2024.
  * Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger.  From word embeddings to document distances.  In _International conference on machine learning_ , pages 957–966. PMLR, 2015.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_ , 2023.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Lin et al. (2023) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al.  Ra-dit: Retrieval-augmented dual instruction tuning.  _arXiv preprint arXiv:2310.01352_ , 2023.
  * Ma et al. (2023) Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan.  Query rewriting for retrieval-augmented large language models.  _arXiv preprint arXiv:2305.14283_ , 2023.
  * Majithia et al. (2020) A. R. Majithia, C. M. Kusiak, A. Armento Lee, F. R. Colangelo, R. J. Romanelli, S. Robertson, D. P. Miller, D. M. Erani, J. E. Layne, R. F. Dixon, and H. Zisser.  Glycemic outcomes in adults with type 2 diabetes participating in a continuous glucose monitor-driven virtual diabetes clinic: Prospective trial.  _J Med Internet Res_ , 22(8):e21778, 2020.  doi: 10.2196/21778.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin.  Multi-stage document ranking with bert.  _arXiv preprint arXiv:1910.14424_ , 2019.
  * Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_ , 2023.
  * OpenAI (2024) OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2024.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Ovadia et al. (2024) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha.  Fine-tuning or retrieval? comparing knowledge injection in llms, 2024.
  * Peng et al. (2023) Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al.  A study of generative large language model for medical research and healthcare.  _arXiv preprint arXiv:2305.13523_ , 2023.
  * Shao et al. (2023) Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen.  Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.  _arXiv preprint arXiv:2305.15294_ , 2023.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_ , 2023.
  * Singhal et al. (2022) Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.  Large language models encode clinical knowledge, 2022.
  * Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_ , 2023.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting.  Large language models in medicine.  _Nature medicine_ , 29(8):1930–1940, 2023.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Tu et al. (2024) Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S Corrado, Yossi Matias, Alan Karthikesalingam, and Vivek Natarajan.  Towards conversational diagnostic ai, 2024.
  * Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei.  Query2doc: Query expansion with large language models.  _arXiv preprint arXiv:2303.07678_ , 2023.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners, 2022.
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof.  C-pack: Packaged resources to advance general chinese embedding.  _arXiv preprint arXiv:2309.07597_ , 2023.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_ , 2019.