# Improving Retrieval Augmented Language Model with Self-Reasoning

Yuan Xia, Jingbo Zhou, Zhenhui Shi, Jun Chen, Haifeng Huang
Baidu Inc., China.
<EMAIL_ADDRESS>
###### Abstract

The Retrieval-Augmented Language Model (RALM) has shown remarkable performance
on knowledge-intensive tasks by incorporating external knowledge during
inference, which mitigates the factual hallucinations inherited in large
language models (LLMs). Despite these advancements, challenges persist in the
implementation of RALMs, particularly concerning their reliability and
traceability. To be specific, the irrelevant document retrieval may result in
unhelpful response generation or even deteriorate the performance of LLMs,
while the lack of proper citations in generated outputs complicates efforts to
verify the trustworthiness of the models. To this end, we propose a novel
self-reasoning framework aimed at improving the reliability and traceability
of RALMs, whose core idea is to leverage reasoning trajectories generated by
the LLM itself. The framework involves constructing self-reason trajectories
with three processes: a relevance-aware process, an evidence-aware selective
process, and a trajectory analysis process. We have evaluated our framework
across four public datasets (two short-form QA datasets, one long-form QA
dataset, and one fact verification dataset) to demonstrate the superiority of
our method, which can outperform existing state-of-art models and can achieve
comparable performance with GPT-4, while only using 2,000 training samples.

Improving Retrieval Augmented Language Model with Self-Reasoning

Yuan Xia, Jingbo Zhou, Zhenhui Shi, Jun Chen, Haifeng Huang Baidu Inc., China.
<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: An example of how self-reasoning framework generates reasoning
trajectories with a relevant-aware process, an evidence-aware selective
process, and a trajectory analysis process.

The Retrieval-Augmented Language Model (RALM) has become a critical
enhancement for Large Language Models (LLMs) by integrating external knowledge
during inference. Despite their advanced capabilities in language
understanding and generation Brown et al. (2020); Touvron et al. (2023), LLMs
are prone to producing hallucinated and inaccurate content, particularly in
knowledge-intensive tasks Ji et al. (2023). Augmenting LLMs with relevant
information obtained from external sources like Wikipedia and search engines
has proven effective in reducing these inaccuracies Guu et al. (2020); Lewis
et al. (2020); Borgeaud et al. (2022); Izacard et al. (2022); Asai et al.
(2024). This method has proven effective in mitigating the factual
hallucinations inherent in LLMs Kwiatkowski et al. (2019); Petroni et al.
(2021); Ram et al. (2023).

Nevertheless, there are still limitations associated with RALMs, particularly
in terms of reliability and traceability. First, the reliability of the
retrieved information remains a significant concern. Previous studies have
shown that noisy retrieval can adversely affect the performance of an LLM
Menick et al. (2022); Li et al. (2023), as irrelevant data can lead to
misguided responses and disturb the model’s ability to leverage its intrinsic
knowledge effectively. Second, the interpretability and traceability of
outputs generated by RALMs need to be improved. While RALMs incorporate
retrieved documents during both training and inference phases, they may fail
to cite these documents explicitly,

thus complicating the process of tracing and verifying the claims made by
LLMs. To improve the retrieval robustness, recent studies have explored
incorporating natural language inference (NLI) models Honovich et al. (2022)
and document summarization models during inference Yoran et al. (2023); Xu et
al. (2023a). However, the effectiveness of these external NLI and
summarization models largely affects the overall performance of RALMs. The
training and optimization of these auxiliary models require additional costs.
Consequently, determining the most appropriate training and selection methods
for these NLI and summarization models presents a further challenge in the
application of such approaches.

To address the above limitations, we propose a novel end-to-end self-reasoning
framework to improve the performance of RALMs. Our intuition is that the
explicit self-reasoning trajectory crafted by LLMs can improve the retrieval
robustness and accuracy of question answering. During the pre-training phase,
an LLM primarily focuses on knowledge acquisition, yet it does not learn to
reason from retrieved documents to generate answers. To address this, a
feasible approach is to incorporate reasoning trajectories into a post-
training phase. Such an approach could potentially teach the model to reason
and distinguish relevant and irrelevant documents, improving its response
accuracy to queries. An example of how our self-reasoning framework generates
reasoning trajectories is illustrated in Figure 1. In contrast, as shown in
the middle part of Figure 2, the conventional RALM methods gather all
documents in a non-selective manner, leading to the distraction of the LLM by
irrelevant content and consequently resulting in the generation of erroneous
answers.

Our framework comprises three self-reasoning processes: 1) a Relevance-Aware
Process (RAP), which instructs the LLM to judge the relevance between the
retrieved documents and the question, 2) an Evidence-Aware Selective Process
(EAP), which directs the LLM to choose and cite relevant documents, and then
automatically select snippets of key sentences as evidence from the cited
documents, 3) a Trajectory Analysis Process (TAP), which requires the LLM to
generate a concise analysis based on all gathered self-reasoning trajectories
generated by previous two processes and subsequently provide the final
inferred answer. Furthermore, we propose a gradual training method by
employing stage-wise masking strategies to enhance the performance of our
framework.

We summarize our contributions as follows:

  * •

We propose a novel end-to-end self-reasoning framework that improves the
robustness of RALMs by leveraging reasoning trajectories generated by the LLM
itself, without the need for external models or tools.

  * •

We carefully design three processes to enhance the interpretability and
traceability of RALMs by requiring LLMs to explicitly generate snippets and
citations from documents, and further explain the reason why cited documents
can help answer the question.

  * •

We evaluate our framework on four public datasets (two short-form QA, one
long-form QA, and one fact verification), demonstrating that our method
surpasses existing state-of-art models in performance, achieving this with
only using 2,000 training samples.

Figure 2: An illustration of the self-reasoning framework for improving the
RALMs. The upper is the basic LLMs which answer the question by inherent
knowledge. The middle is the standard retrieval augmented LMs, which use
retrieved documents to help answer the question. The bottom is our self-
reasoning framework which uses self-generated reason trajectories to output
answers.

##  2 Related Work

###  2.1 Retrieval-augmented LMs

Many studies have investigated augmenting the performance of LLMs with
externally retrieved information Izacard et al. (2022); Guu et al. (2020);
Borgeaud et al. (2022) and some of them pre-train language models with
retrieved passages. For works focusing on RALMs with citations, Menick et al.
(2022); Nakano et al. (2021) instruct or train an LLM to answer questions with
retrieved documents while providing citations. Gao et al. (2023b) proposes an
end-to-end system to retrieve supporting evidence and generate answers with
citations, while only focusing on prompting without updating their model
weights. Other works instruct or fine-tune LLMs to use external tools to
retrieve dynamically Schick et al. (2023); Yao et al. (2023); Jiang et al.
(2023), which offers an adaptive method of when and what to search. Gao et al.
(2023a) improves the attribution and factuality of language models by taking
outputs of LLMs and applying a post-process retrieve-and-edit approach.

###  2.2 Robustness for RALMs

To improve the robustness of RALMs, Yoran et al. (2023) use a natural language
inference model to filter out irrelevant documents before RALMs, and Xu et al.
(2023a) use a memorization model to filter out or compress retrieved documents
before using them to prompt an LLM. The concurrent work Yu et al. (2023)
generates summaries of documents as notes before RALMs,  and Baek et al.
(2023) uses a separate small language model as a verifier to detect and
correct errors in LLMs during retrieval for generating factually correct
outputs. Different from the above works, ours identifies key sentences and
cites relevant documents through an end-to-end framework that eliminates
irrelevant ones, without dependence on external inference models.

The most similar work to ours is presented by Asai et al. (2024), who
developed a method that teaches models to retrieve information using designed
reflection tokens. However, this approach needs to train extra critic models
and generator models to predict the reflection tokens, which requires tens of
thousands of extra training samples. In contrast, our method does not rely on
special tokens. Instead, we construct reasoning trajectories and then use them
to directly enhance the performance of LLMs, offering a more efficient and
scalable solution. More related works on LLMs for reasoning are discussed in
the Appendix $\S$A.1.

##  3 Preliminary

We formally define the problem of retrieval augmented generation with self-
reasoning. Given a query $q$ and a corpus of documents $\mathcal{D}$, an LLM-
generated answer with $m$ statements and $n$ tokens can be defined as
$y=(s_{1},s_{2},\cdots,s_{m})=(w_{1},w_{2},\cdots,w_{n})$, where $s_{i}$ is
the $i$-th statement and $w_{j}$ is the $j$-th token in the generated answer.
In addition, for long-form QA settings, each statement $s_{i}$ should cite a
list of documents $C_{i}=\\{c^{(1)}_{i},c^{(2)}_{i},...\\}$, where
$c^{(k)}_{i}\in\mathcal{D}$. In our work, we train an LLM (e.g. LLaMA2) to
first generate reasoning trajectories $\tau$ through self-reasoning and then
to generate answers ${y}^{*}$ (including long-form answers and short-form
answers) on condition of $\tau$. The model output is
$y={\rm{concat}}(\tau,y^{*})$, which is the concatenation of $\tau$ and
$y^{*}$. Note that the generations of $\tau$ and $y^{*}$ are done in a single
pass within our self-reasoning framework.

##  4 Method

In this section, we provide a detailed implementation of the self-reasoning
process and the method to construct reasoning trajectories.

In this work, we propose a novel framework which involves three processes: 1)
a Relevance-Aware Process (Sec $\S$4.1), 2) an Evidence-Aware Selective
Process (Sec $\S$4.2), and 3) a Trajectory Analysis Process (Sec $\S$4.3). An
illustration of our self-reasoning framework for improving the RALMs is shown
in Figure 2.  In Sec 4.4, we demonstrate the process of data generation and
quality control, and Sec 4.5 presents the details of model training.

###  4.1 Relevance-Aware Process

In this work, we choose the DPR Karpukhin et al. (2020) and the Contriever
Izacard et al. (2021) as the default retriever $R$ to recall the top-$k$
relevant documents. When presented with a question and a set of documents,
people can determine whether the question is relevant to the retrieved
documents. Therefore, we first instruct the model to judge the relevance
between the retrieved documents $\mathcal{D}$ and the given question $q$. We
further request the model to explicitly generate reasons explaining why given
documents are identified as relevant. The output should include two fields as
relevant and relevant reason, as depicted in Figure 2. Noting that if all of
the retrieved documents are irrelevant, the model should provide an answer
based on the internal knowledge acquired during its pre-training phase. We
define the self-reasoning trajectories generated by RAP as $\tau_{r}$.

###  4.2 Evidence-Aware Selective Process

When required to answer the question, people generally will first identify the
crucial sentences from the provided documents, and then cite or highlight them
as key points. This process of citing the document facilitates reading
comprehension and can serve as a technique for combining multiple short
answers to address various aspects. While people may carry out this selective
process and citation instantaneously, LLMs need to formulate the self-
reasoning trajectories explicitly.

In our work, we require the LLM to explicitly state the reason why the
selected sentence is supportive and plausible in answering the question. We
define the selected sentence as evidence in our paper. Specifically, after
retrieving the top-$k$ documents, the self-reasoning method for Evidence-Aware
Selective Process can be formulated as follows: First, we instruct the LLM to
choose relevant documents and automatically select snippets of key sentences
for the selected documents. Then, we request the LLM to output the reason why
the selected snippets can answer the question. The intermediate output is a
list containing multiple contents, each content should include two fields, as
cite content and reason for cite, which is illustrated in Figure 2. We define
the self-reasoning trajectories generated by EAP as $\tau_{e}$.

###  4.3 Trajectory Analysis Process

Finally, we consolidate all the self-reasoning trajectories ($\tau_{r}$ and
$\tau_{e}$) in the previous processes together to form a chain of reasoning
snippets, thereby enhancing the overall performance of the retrieval
augmentation generation. Specifically, we ask the LLM to analyze the reasoning
trajectories within itself and ultimately to output a concise analysis and a
short answer. We instruct the LLM to output content with two fields as
analysis and answer, which is shown in Figure 2. We define the self-reasoning
trajectories generated by TAP as $\tau_{a}$. In this work, the analysis output
is defined as a long-form answer, and the answer output is defined as a short-
short answer. In Section 5.2, we further explored the performance of long-form
and short-form QA settings.

###  4.4 Data Generation and Quality Control

##### Training Data Generation.

For the Relevance-Aware Process data generation, as manually labeling the
relevant and irrelevant documents is label-intensive, we request the GPT-4
OpenAI (2023) to generate answers as ground truth. Specifically, we instruct
the GPT-4 to generate labels regarding irrelevant fields, and further to
output the reasons why the given documents cannot answer the question. We
concatenate the given question and the retrieved documents as positive
samples. For negative samples, we randomly select a different question from
the training set and retrieve the top-$k$ documents related to it. These
documents are then concatenated with the initial question to form negative
samples. To avoid order bias in the training data, we shuffle the order of the
documents.

For the EAP and TAP data generation, manually annotating the citation of the
document and writing the self-reasoning process for each question is not
feasible in practice. Therefore, we follow a similar process to RAP, we first
instruct the GPT-4 to generate a snippet of selected documents and
subsequently output the reasoning process as trajectories. The method to
construct the EAP training data is the same as RAP except that the
instructions for the GPT-4 are different. The details of the instructions are
shown in Appendix $\S$A.2.

##### Data Quality Control.

For training data generation, correct and comprehensive reasoning trajectories
are very important. When training an LLM, the quality of the training samples
is more important than the quantity Zhou et al. (2023). As we cannot guarantee
the correctness of self-reasoning trajectories and citations by the GPT-4, we
develop two efficient methods to control the quality of data generation: 1)
The first method is to use the off-the-shelf tools 111Tools are available at
https://github.com/princeton-nlp/ALCE/tree/main in Gao et al. (2023b) to
automatically verify the performance of data generation for document
citations. We calculate the citation precision and recall score for each
training sample and filter out scores lower than our pre-defined thresholds
$\delta_{p}$ and $\delta_{r}$, for citation precision and recall,
respectively. 2) Second, though the validation of self-reasoning trajectories
and citations generated by GPT-4 is challenging, verifying the correctness of
the final answer is straightforward. Therefore, we filter out the trajectories
that lead to the incorrect answers and only keep the correct ones. We totally
generate 10,000 training samples by GPT-4, after the filtering strategy by
quality control, we finally keep 2,000 training samples with high quality.

###  4.5 Model Training

We train our self-reasoning generation model $\phi$ by our constructed corpus
which is augmented with self-reasoning trajectories $\tau$ using the standard
language modeling objective, maximizing likelihood:

$\max_{\mathcal{\phi}}\mathbb{E}_{(q,\tau,y)\sim\mathcal{D}_{sr}}\log
p_{\mathcal{\phi}}(y\mid\tau,q)p_{\mathcal{\phi}}({\tau\mid q})$ (1)

where $\tau=\tau_{r}\oplus\tau_{e}\oplus\tau_{a}$ are the self-reasoning
trajectories, $\oplus$ is the concatenation operator,
$\tau_{r},\tau_{e},\tau_{a}$ are trajectories generated by above three
processes respectively. $q$ is the provided question, and $y$ is the model
output, including the intermediate reason trajectories and the final answer.
$\mathcal{D}_{sr}$ is the training corpus augmented with self-reasoning
trajectories.

During training, we observed that it is more challenging to ensure the
correctness of an LLM with 13B parameters when generating long reasoning
trajectories than short ones. We hypothesize that an LLM’s effective reasoning
length is limited and exceeding this limit might lead to error accumulation
during the inference stage. Therefore, we propose a gradual training method by
employing stage-wise masking strategies to gradually learn to generate long
trajectories.

Specifically, we propose a stage-wise training process while we train the LLM
stage by stage. In the first stage, we mask the trajectories produced by the
next two stages (EAP and TAP) and train the model with a learning rate
$r_{a}$. Then in the second stage, we only mask the trajectories generated by
TAP and train the model with a learning rate $r_{b}$. Finally, we concatenate
the reasoning trajectories from all stages and put them into a self-reasoning
LLM for end-to-end training with a learning rate $r_{c}$. Hyper-parameters for
training are described in Appendix $\S$A.4.

##  5 Experiments

Models | NaturalQuestion | PopQA | FEVER | ASQA
---|---|---|---|---
(acc) | (acc) | (acc) | (em-recall) | (precision) | (recall)
Baselines without retrieval
LLaMA27B | 19.2 | 18.4 | 23.2 | 10.2 | - | -
LLaMA213B | 24.0 | 22.6 | 25.3 | 15.3 | - | -
LLaMA27B-chat | 20.2 | 21.5 | 26.5 | 16.3 | - | -
LLaMA213B-chat | 23.2 | 25.9 | 28.4 | 18.3 | - | -
Baselines with retrieval
LLaMA27B | 27.8 | 47.8 | 39.8 | 28.5 | 13.6 | 9.59
LLaMA213B | 34.0 | 48.1 | 35.2 | 26.8 | 21.8 | 16.3
LLaMA27B-chat | 27.4 | 52.9 | 43.4 | 25.3 | 34.5 | 33.2
LLaMA213B-chat | 32.7 | 53.5 | 53.4 | 26.4 | 39.4 | 38.4
Vicuna7B Chiang et al. (2023) | 28.0 | 55.2 | 62.4 | 24.3 | 45.7 | 40.8
Vicuna13B Chiang et al. (2023) | 35.4 | 56.1 | 60.6 | 27.3 | 51.3 | 50.2
LLaMA2-FT7B | 36.8 | 54.4 | 67.5 | 28.5 | 47.2 | 45.4
ReAct Yao et al. (2023) | - | - | 64.6 | - | - | -
RECOMP Xu et al. (2023a) | 38.4 | - | - | - | - | -
Self-RAG7B Asai et al. (2024) | 37.2 | 54.9 | 70.2 | 30.0 | 66.9 | 67.8
Self-RAG13B Asai et al. (2024) | 38.8 | 55.8 | 72.1 | 31.7 | 70.3 | 71.3
Self-Reasoning7B | 38.0 | 54.2 | 78.6 | 33.9 | 66.3 | 70.8
Self-Reasoning13B | 41.4 | 57.3 | 83.9 | 35.2 | 71.2 | 72.3
GPT-4 | 46.6 | 62.5 | 87.7 | 41.3 | 75.6 | 68.5
Table 1: Performance comparisons with different baseline models on two short-
form QA datasets, a long-form QA dataset, and a fact verification dataset. The
numbers with bold black represent the best results excluding GPT-4.  The
results are averaged over five runs, and presented with standard variance
values omitted (all $\leq 2\%$).

###  5.1 Datasets and Settings

To demonstrate the effectiveness of our proposed self-reasoning framework, we
conduct an extensive experimental evaluation on two short-form QA datasets
(NaturalQuestion Kwiatkowski et al. (2019) and PopQA Mallen et al. (2023)),
one long-form QA dataset (ASQA Stelmakh et al. (2022)), and one fact
verification dataset (FEVER Thorne et al. (2018)). Detailed descriptions of
the datasets can be found in Appendix $\S$A.3. We explore off-the-shelf
retrievers. We use the DPR Karpukhin et al. (2020) and the Contriever-MS MARCO
Izacard et al. (2021) to retrieve the top five documents from Wikipedia.

By default, we use the DPR as a retriever for the NQ, as the DPR has been
fine-tuned on the high-quality NQ data. On the PopQA, where question and
answer pairs are created based on Wikipedia in 2022, therefore, for the PopQA,
we use the December 2020 preprocessed Wikipedia corpus provided by Izacard et
al. (2022) and use the Contriever as a retriever. For the ASQA dataset, we use
GTR Ni et al. (2022) as a retrieval that corresponds to the experimental
settings in Gao et al. (2023b). More settigns can be found in Appendix
$\S$A.4.

###  5.2 Evaluation Metrics

We use different evaluation metrics for short-form QA, long-form QA, and fact
verification tasks.

##### Short-form QA metrics.

We report accuracy for short-form QA tasks, which is based on whether ground-
truth answers are included in the model predictions instead of strictly
requiring exact matching, following Mallen et al. (2023); Schick et al.
(2023).

##### Long-form QA metrics.

For long-form QA tasks, we report the EM recall as a correctness metric, and
the citation recall and the citation precision for citation quality, which are
the same as the metrics in Gao et al. (2023b).

##### Fact verification metrics.

For the fact verification task, we report the accuracy as a metric, which is a
three-class classification accuracy, following Thorne et al. (2018).

###  5.3 Baseline Models

##### Baseline models without retrieval.

We evaluate strong open-source pre-trained LLMs as baseline models. For basic
LLMs, we test LLaMA2-7B, LLaMA2-13B Touvron et al. (2023) and its instruction-
tuned chat version LLaMA2-Chat-7B, LLaMA2-Chat-13B.

#####  Baseline models with retrieval.

To assess the performance of the retrieval augmented LMs, we use retrievers
described in Section 5.1.  First, we  benchmark the models using the LLaMA2
and the Vicuna Chiang et al. (2023) series models for baselines. Additionally,
for a fair comparison, we also include LLaMA2-FT, where LLaMA2 is fine-tuned
on all the training samples generated by GPT-4 except the self-reasoning
trajectories. To establish strong baselines, we compare our method against
RECOMP Xu et al. (2023a), ReAct Yao et al. (2023), and Self-RAG Asai et al.
(2024), all of which are trained with extra GPT-4 generated samples or
external tools. We also compare our framework with GPT-4 OpenAI (2023). We
include categorical comparisons with the baseline models in the Appendix
$\S$A.5.

Figure 3: Noise robustness experiment results on three different datasets: (a)
On the left is the NQ dataset, (b) in the middle is the PopQA dataset, (c) on
the right is the FEVER dataset.  The Self-RAG and Vicuna are 13B parameter
size models.

###  5.4 Main Results

Table 1 shows the performance comparisons with different methods on the four
public datasets. For short-form QA evaluations, the performance of LLMs with
augmented retrieval is consistently better than that of basic ones, affirming
the effectiveness of the augmented approach. Notably, under the same order of
magnitude parameters, our self-reasoning framework outperforms most of the
strong baseline LLMs.  Specifically, compared to the Self-RAG, our framework
is an end-to-end system trained with only 2,000 self-reasoning trajectory
samples. In contrast, the Self-RAG requires training additional critic LMs to
predict reflection tokens using an additional 46,000 instances generated by
GPT-4. This efficiency not only simplifies the training process but also
significantly reduces resource consumption.

In the context of long-form QA evaluations, for the metrics of EM recall, it
needs to comprehend multiple documents and merge answers. The EAP and TAP are
specifically designed for multi-document reading comprehension, enabling our
performance to surpass other baselines. In terms of citation evaluation
metrics, our self-reasoning framework  can achieve better results than GPT-4
in ASQA citation recall metrics (72.3 vs. 68.5).

This is largely due to the reasoning trajectories generated in the EAP, which
can enhance the recall and precision of citation evaluation, leading to more
interpretable generations.

For fact verification evaluations, we observed that self-seasoning is
dominantly superior to all baseline models. Our method achieves a much higher
accuracy rate than the Self-RAG model (83.9 vs. 72.1). The RAP in our
framework is designed to judge the relevance between the retrieved documents
and the question, which leads to the notable enhancement in accuracy for this
fact verification task.

To clearly demonstrate the practical applications and benefits of our self-
reasoning framework, we provide a case study for a more in-depth analysis in
the Appendix $\S$A.6, which illustrates how the framework operates in real-
world scenarios.

Models | NQ | PopQA | FEVER
---|---|---|---
|  (acc) |  (acc) |  (acc)
Origin |  41.4 |  57.3 |  83.9
w/o (RAP) |  39.9 |  54.3 |  72.2
w/o (EAP) |  37.2 |  53.2 |  78.4
w/o (TAP) |  38.2 |  53.4 |  81.2
w/o (GL) |  39.5 |  55.3 |  81.2
w/o (QC) |  37.7 |  54.2 |  80.8
Table 2: The ablation study on two short-form QA datasets and a fact
verification dataset with 13B parameter size models. In the table, the origin
represents our self-reasoning model enhanced with self-generated trajectories.

##  6 Analysis

###  6.1 Ablation Study

We conduct an ablation study on two short-form QA datasets and a fact
verification dataset to analyze the individual contributions of each process
within our proposed self-reasoning framework. We further explore the
effectiveness of the gradual learning (GL) method and the quality control (QC)
of data generation (a detailed analysis described in Appendix $\S$A.7). The
main ablation study results are shown in Table 2.

#### Effectiveness of Relevant-Aware Process

First, we evaluate the effect of the Relevant-Aware Process (RAP). The removal
of the RAP causes the overall performance to drop in two short-form QA
datasets and a fact verification dataset, suggesting that preliminary
consideration of the relevance between questions and retrieved documents can
help improve performance. We notice that the performance declines most
significantly in the FEVER dataset. Detecting irrelevant documents is critical
in the fact-verification task. Our model will immediately output NotEnoughInfo
if it detects that all documents are irrelevant.

#### Effectiveness of Evidence-Aware Process

Then we evaluate the effect of the Evidence-Aware Selective Process (EAP).
Removing the EAP causes the overall performance of the average accuracy to
decline from 60.9 to 56.3 in three short-form QA datasets. This reduction
indicates that snippets of key sentences and document citations generated
through self-reasoning are instrumental in boosting accuracy.

#### Effectiveness of Trajectory Analysis Process

Finally, we evaluate the effect of the Trajectory Analysis Process (TAP). When
excluding the TAP, we can observe a performance decline on all three datasets,
demonstrating that self-analysis based on two previous processes generated
trajectories can also improve the performance of LLMs. Note that the analysis
content generated by TAP is indispensable for the long-form QA evaluation.

###  6.2 Retrieval Robustness Analysis

As retrievers are not perfect and past work has shown that noisy retrieval can
have negative effects on the performance of LLMs Petroni et al. (2020); Li et
al. (2023). In this section, we design two kinds of settings to validate the
robustness of RALMs. In the first setting, we test whether the order of the
retrieved documents will affect the performance of the RALMs. Specifically,
after retrieving the top-$k$ documents using retrievals (such as the DPR) with
a descending relevance score, we randomly shuffle the order of the retrieved
documents and then input them to an LLM. In the second setting, we test how
noisy documents impact the performance of LLMs. When retrieving the top-$k$
documents from the given question, we randomly replace 50% of the retrieved
documents with other documents sampled from a different question in the
dataset.

Figure 3 shows the noise robustness experiment results on three datasets. Our
self-reasoning framework consistently outperforms the  Self-RAG and Vicuna
models. We observe that random shuffling of retrieved documents has a minimal
impact on the performance of RALMs. If the provided documents are supportive,
it is trivial for a RALM to determine the correct answer. However, when
presented with noisy documents, all models experience a decline in
performance. The performance drop for our self-reasoning framework is
relatively minimal, which demonstrates that our proposed method is robust even
when dealing with noisy documents.

Figure 4: Human citation quality evaluation vs. automatic citation evaluation
on the long-form ASQA dataset.

###  6.3 Citation analysis

As the automatic evaluation by the NLI model cannot detect partially supported
citations, we discuss the analysis of citations with human evaluation in this
section. Similarly to Liu et al. (2023), we conduct a human evaluation on two
dimensions: 1) citation recall: annotators are given a statement and all
documents that the statement refers to and are asked to judge whether the
documents fully support the given statement; 2) citation precision: given a
statement and one of its citations, annotators are asked to validate whether
the citation fully supports, partially supports or does not support the
statement. Each citation gets a precision score of 1 if the output sentence
has a citation recall of 1 and this citation at least partially supports it.
Details of human annotation can be found in Appendix $\S$A.8.

As shown in Figure 4, we observe that the relative rankings by human
evaluation align well with those from the automatic evaluation, and the human
evaluation often yields a closely higher score when compared with the
automatic evaluation.

##  7 Conclusion

RALMs can effectively enhance the performance of LLMs in handling knowledge-
intensive tasks. Despite their effectiveness, notable concerns about their
reliability and traceability persist. To address these limitations, we propose
a novel self-reasoning framework to improve the performance of RALMs by using
reasoning trajectories generated by the LLM itself. It is comprised of a
relevance-aware process, an evidence-aware selective process, and a trajectory
analysis process. We conduct extensive experiments on four public datasets to
demonstrate the superiority of our framework over existing state-of-the-art
models.

##  8 Limitations

In this work, we mainly focus on improving the performance of RALMs with a
self-reasoning framework on the task of open domain question answering and
fact verification. Although we believe our framework can involve the
distribution of real-world user questions, as we have evaluated in four public
datasets, we acknowledge that we have not explored more challenging scenarios,
such as multi-hop reasoning, code generation, and arithmetic reasoning. In
future work, more challenging reasoning tasks, such as arithmetic reasoning,
should be explored for the self-reasoning framework. We believe that our
framework can effectively mitigate factual hallucinations in LLMs and improve
the robustness of RALMs. However, there is still a risk that our method might
generate hallucinations.

## References

  * Asai et al. (2024) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024.  Self-RAG: Learning to retrieve, generate, and critique through self-reflection.  In _The Twelfth International Conference on Learning Representations_.
  * Baek et al. (2023) Jinheon Baek, Soyeong Jeong, Minki Kang, Jong Park, and Sung Hwang. 2023.  Knowledge-augmented language model verification.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1720–1736, Singapore. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Gao et al. (2023a) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a.  RARR: Researching and revising what language models say, using language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 16477–16508, Toronto, Canada. Association for Computational Linguistics.
  * Gao et al. (2023b) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b.  Enabling large language models to generate text with citations.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6465–6488, Singapore. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: retrieval-augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  True: Re-evaluating factual consistency evaluation.  _arXiv preprint arXiv:2204.04991_.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.  _arXiv preprint arXiv:2112.09118_.
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Jiang et al. (2023) Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Active retrieval augmented generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7969–7992, Singapore. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2023) Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023.  Large language models with controllable working memory.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1774–1793, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2023) Nelson Liu, Tianyi Zhang, and Percy Liang. 2023.  Evaluating verifiability in generative search engines.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 7001–7025, Singapore. Association for Computational Linguistics.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5783–5797, Online. Association for Computational Linguistics.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9844–9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * OpenAI (2023) R OpenAI. 2023.  Gpt-4 technical report. arxiv 2303.08774.  _View in Article_ , 2:13.
  * Pan et al. (2024) Zhenyu Pan, Haozheng Luo, Manling Li, and Han Liu. 2024.  Chain-of-action: Faithful and multimodal question answering through large language models.  _arXiv preprint arXiv:2403.17359_.
  * Petroni et al. (2020) Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020.  How context affects language models’ factual predictions.  _arXiv preprint arXiv:2005.04611_.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021.  KILT: a benchmark for knowledge intensive language tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2523–2544, Online. Association for Computational Linguistics.
  * Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023.  Measuring and narrowing the compositionality gap in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5687–5711, Singapore. Association for Computational Linguistics.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 11:1316–1331.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022.  ASQA: Factoid questions meet long-form answers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _ArXiv_ , abs/2203.11171.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Xu et al. (2023a) Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023a.  Recomp: Improving retrieval-augmented lms with compression and selective augmentation.  _arXiv preprint arXiv:2310.04408_.
  * Xu et al. (2023b) Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-seng Chua. 2023b.  Search-in-the-chain: Towards the accurate, credible and traceable content generation for complex knowledge-intensive tasks.  _arXiv preprint arXiv:2304.14732_.
  * Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023.  ReAct: Synergizing reasoning and acting in language models.  In _International Conference on Learning Representations (ICLR)_.
  * Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023.  Making retrieval-augmented language models robust to irrelevant context.  _arXiv preprint arXiv:2310.01558_.
  * Yu et al. (2023) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023.  Chain-of-note: Enhancing robustness in retrieval-augmented language models.  _arXiv preprint arXiv:2311.09210_.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.
  * Zhou et al. (2022) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _ArXiv_ , abs/2205.10625.

##  Appendix A Appendix

Models | End-to-End | External module | Train data for LLM
---|---|---|---
(train) | (inference) | (data)
Self-Reasoning (Ours) | Y | N | N | No Need | 2K
Self-RAG Asai et al. (2024) | N | N | N | No Need | 145K (Generator)/ 46K (Critic)
ReAct Yao et al. (2023) | N | N | Y | No Need | No Need
RECOMP Xu et al. (2023a) | N | Y | Y | 152K | No Need
Table 3: Categorical comparisons with strong baseline models. External module
(train) and External module (data) refer to whether the external module needs
to be trained and the number of samples required, respectively. External
module (inference) indicates whether the external module is needed during the
inference stage. Train data for LLM indicates the number of training samples
needed to train with LLMs.

###  A.1 More Related Work of LMs for Reasoning

One of the most well-known methods of using LLMs for reasoning is the Chain-
of-Thought (CoT) Wei et al. (2022), which demonstrates the capability of LLMs
to create their thinking process for problem-solving. Zhou et al. (2022)
proposes a least-to-most prompting for solving complex tasks. Wang et al.
(2022) introduces a method to reason with self-consistency. Press et al.
(2023) proposes a method to further improve the chain of thought by reasoning
explicitly instead of implicitly.

Recent works have extended beyond the internal reasoning ability of LLMs to
include interactions with external tools (e.g., search engines or retrievers)
for solving complex tasks. The ReAct Yao et al. (2023) presents an iterative
paradigm to combine reasoning and acting with LLMs for tackling language
reasoning and decision-making tasks. Xu et al. (2023b) introduces a framework
to enable information retrieval and LLMs to interact with each other
effectively with chain-of-query decomposition. Pan et al. (2024) proposes a
novel framework named Chain-of-Action (CoA), which integrates a reasoning
retrieval method to decompose complex questions into chains of configurable
actions.

Different from the above works, which are mostly based on relatively large
LLMs (e.g., ChatGPT), our proposed method focuses on enhancing smaller LLMs
(e.g., LLaMA2) using only a limited number of samples to achieve high
robustness and interpretability through single-step interaction.

###  A.2 Instructions

The instructions for GPT-4 to generate self-reasoning trajectories are shown
in Figure 5 (the short-form and long-form QA tasks) and Figure 6 (the fact
verification task). The words in the orange font are key fields that need to
be generated.

###  A.3 Datasets Description

We conducted an extensive experimental evaluation of two short-form QA
datasets, one long-form QA dataset, and a fact verification dataset.

NaturalQuestion (NQ) Kwiatkowski et al. (2019) contains real user questions
issued to the Google search and answers found from Wikipedia by the
annotators. NQ is created to train and evaluate automated question answering
systems.

PopQA Mallen et al. (2023) is a large-scale open-domain question answering
dataset, consisting of entity-centric QA pairs. Each question is made by
converting a knowledge triplet retrieved from Wikidata using a template. In
this work, we use PopQA to evaluate performance in long-tail settings.

ASQA Stelmakh et al. (2022) is a long-form factoid dataset, and most questions
can be answered by Wikipedia. Each question originates from AmbigQA Min et al.
(2020) and represents an ambiguous query that requires multiple short answers
to cover various aspects. The dataset provides a long-form answer that
contains all short answers.

FEVER Thorne et al. (2018) is a fact verification dataset that contains claims
generated by rewriting sentences extracted from Wikipedia and subsequently
verified without knowledge of the sentence from which they were derived. The
claims are classified as Supported, Refuted, or NotEnoughInfo.

###  A.4 Experiment settings

##### Training settings.

During gradual learning, we fine-tune the LLaMA-2 Touvron et al. (2023) model
with our self-reasoning framework for 3 epochs with a batch size set to 32,
leveraging the DeepSpeed library Rasley et al. (2020) and the ZeRO optimizer,
and we use parameter partitioning ZeRO stage 3 with float16 precision. The
learning rate $r_{a}$ for the first stage is set to 5e-5, the learning rate
$r_{b}$ for the second stage is set to 3e-5, and the learning rate $r_{c}$ for
the final stage is set to 1e-5. Our self-reasoning 13B model is trained on the
NVIDIA Tesla 8 $\times$ V100 32GB GPU for 4 hours, while the 7B model is
trained for 2 hours.

##### Inference settings.

We use the vLLM 222Codes are available at https://github.com/vllm-project/vllm
framework Kwon et al. (2023) to accelerate the inference speed during
inference. The codes follow the Apache-2.0 license agreement. We use greedy
decoding in all experiments to ensure deterministic generations. We test the
temperature within a range of $\\{0.2,0.4,0.6,0.8,1.0\\}$, finally we set the
temperature to 0.2, as we observed lower temperature results in better
performance in the open-domain question answering task. The maximum generation
length is set to 2048 for our model. All baseline models are tested with zero-
shot settings for short-form QA datasets, and with one-shot settings for the
long-form QA and fact verification datasets.

##### Other settings.

For the document retrieval, we retrieve the top-$k$ relevant documents, and
the $k$ is set to 5.  We use the DPR and the Contriever in short-form QA
settings. For long-form QA, we use GTR as a retrieval and evaluate it using
one-shot to instruct the model to generate citations. For the data generation
quality control setting, the threshold for citation recall is set to 0.8, and
the threshold of citation precision is set to 0.8.

###  A.5 Categorical Comparisons

We differentiate our method from existing strong baseline models by
categorizing and comparing it across six dimensions, as presented in Table 3.
As illustrated in the table, our method can stand out in several key aspects.

First, our self-reasoning method is the only end-to-end framework among
existing methods that can improve performance without relying on external
models or tools. Second, our method eliminates the need for external modules
during both the training and inference phases. In practical applications, our
framework does not need to call multiple tools or modules. Third, our
framework requires a significantly smaller dataset for training the LLM
compared to other methods, needing only 2,000 samples with self-reasoning
trajectories. This efficiency in training drastically lowers the resources and
time needed, making our method both cost-effective and scalable for practical
applications.

###  A.6 Case Study

In our case study, as illustrated in Figure 7, we compare the responses
generated by the raw LLM, the standard RALM (e.g. LLaMA2 with retrievals), and
our self-reasoning method. The challenge involves reconciling information from
multiple retrieved documents to provide a correct answer as the retrieved
documents contained noisy data.

The response from the raw LLM (e.g., LLaMA2) suggested that the film was made
in 2000, based on its inherited knowledge. However, this answer is incorrect
and is a hallucination generated by the LLM. The standard RALM approach
yielded 1989 as the production date. This answer was based on unrelated
details from the retrieved documents, showing a lack of context-specific
understanding and robustness for noisy retrieved documents.

Our self-reasoning framework provided a comprehensive approach by assessing
the relevance and context of retrieved documents. First, in the relevance-
aware process, the documents were identified as relevant based on their
content regarding the production dates and events surrounding the film.
Second, in the evidence-aware selective process, the model retrieved the first
documents, which highlighted the original start date as January 2002, with
filming commencing in February 2002 (highlighted in green in the figure). This
information was crucial in establishing the timeline for the film’s
production. The model can also understand of the difference between the
production date and the release date in the third retrieved document
(highlighted in red in the figure). In the trajectory analysis process, the
correct timeline was deduced by piecing together self-generated trajectories,
leading to the conclusion that the film Catch Me If You Can was indeed
produced in 2002. As the case illustrated above, by leveraging relevant
documents and focusing on contextual evidence, our self-reasoning framework
can achieve a precise and well-supported answer, highlighting its utility and
robustness in complex information retrieval tasks.

###  A.7 More Analysis on Ablation Study

#### Effectiveness of Gradual Learning

Further, we validate the effect of gradual learning. Rather than training an
LLM with a stage-by-stage approach, we initially concatenate the reasoning
trajectories from all stages and put them into the LLM for end-to-end
training. As shown in Table 2, the performance decline can be observed in
three datasets, suggesting that gradual learning can help improve performance.

#### Effectiveness of Quality Control

The effect of quality control on data generation is also evaluated in our
work. Instead of using the filtered high-quality training samples, we randomly
sampled 2,000 unfiltered training samples generated by GPT-4. As shown in
Table 2, the substitution of unfiltered training data leads to the degradation
of the model results.

###  A.8 Human Evaluation

We randomly sample 100 examples from the ASQA dataset and annotate the outputs
of selected models. Each sample is then assigned to two people for annotation.
Each annotator is required to verify the citation recall and citation
precision acorrding to our provided scheme. The annotation scheme is inspired
by Gao et al. (2023b) as follows:

##### Citation Recall.

The annotators are shown the question $q$, the statement $s_{i}$, and all of
its citations $C_{i}$, and they assess if the set of citations fully support
the statement (recall=1) or if they do not support all the claims (recall=0).
We calculate the overall recall score for the model by averaging the recall
scores of all statements.

##### Citation Precision

The annotators are shown the question $q$ and a statement $s_{i}$ and one of
its citation ${c^{(k)}_{i}}\in C_{i}$. We ask the annotator if the citation
fully supports, partially supports, or does not support the generated claims
in $s_{i}$. Citation ${c^{(k)}_{i}}$ has a citation precision of 1 if $s_{i}$
has a recall of 1, and ${c^{(k)}_{i}}$ fully or partially supports $s_{i}$.
Finally, we calculate the overall precision score for the model by averaging
the precision scores of all statements.

Figure 5: The instructions for the GPT-4 to generate the self-reasoning
trajectories for short-form and long-form QA tasks. Figure 6: The instructions
for the GPT-4 to generate the self-reasoning trajectories for the fact
verification task. Figure 7: A Case Study. We present the self-reasoning
trajectories generated by our framework during inference and demonstrate how
they can logically generate the correct answer.