# Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generation

Yige Shen
Xi’an Jiaotong University
<EMAIL_ADDRESS>&Hao Jiang
Xi’an Jiaotong University
<EMAIL_ADDRESS>Hua Qu
Xi’an Jiaotong University
<EMAIL_ADDRESS>&Jihong Zhao
Xi’an Jiaotong University
<EMAIL_ADDRESS>
###### Abstract

Despite their impressive capabilities, large language models (LLMs) often face
challenges such as temporal misalignment and generating hallucinatory content.
Enhancing LLMs with retrieval mechanisms to fetch relevant information from
external sources offers a promising solution. Inspired by the proverb "Think
twice before you act," we propose a dual-angle evaluated retrieval-augmented
generation framework Think-then-Act. Unlike previous approaches that
indiscriminately rewrite queries or perform retrieval regardless of necessity,
or generate temporary responses before deciding on additional retrieval, which
increases model generation costs, our framework employs a two-phase process:
(i) assessing the input query for clarity and completeness to determine if
rewriting is necessary; and (ii) evaluating the model’s capability to answer
the query and deciding if additional retrieval is needed. Experimental results
on five datasets show that the Think-then-Act framework significantly improves
performance. Our framework demonstrates notable improvements in accuracy and
efficiency compared to existing baselines and performs well in both English
and non-English contexts. Ablation studies validate the optimal model
confidence threshold, highlighting the resource optimization benefits of our
approach.

Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generation

Yige Shen Xi’an Jiaotong University<EMAIL_ADDRESS>Hao Jiang Xi’an
Jiaotong University<EMAIL_ADDRESS>

Hua Qu Xi’an Jiaotong University<EMAIL_ADDRESS>Jihong Zhao Xi’an
Jiaotong University<EMAIL_ADDRESS>

Figure 1: Think-then-Act: (i) assessing the input query for clarity and
completeness to determine if rewriting is necessary; (ii) evaluating the
model’s capability to answer the query and deciding if additional retrieval is
needed.

##  1 Introduction

Large language models (LLMs) have become a cornerstone of natural language
processing (NLP) systems due to their impressive capabilities in understanding
and generating human language (Brown et al., 2020; Ouyang et al., 2022;
OpenAI, 2023). Despite their success, LLMs often suffer from temporal
misalignment Röttger and Pierrehumbert (2021); Luu et al. (2022)or generating
hallucinatory content Ji et al. (2023); Shi et al. (2023); Bang et al. (2023).
This impacts the dependability of LLMs and limits their broader practical use,
as the alignment between LLM outputs and real-world information still requires
further validation. Augmenting LLMs with retrieval mechanisms to fetch
relevant information from external sources has emerged as a promising approach
to mitigate these issues Khandelwal et al. (2019); Izacard et al. (2023).

Retrieval-augmented language models (LMs) typically operate using a retrieve-
and-generate framework. This process begins by retrieving relevant documents
based on the user’s input. Subsequently, the model generates a comprehensive
response that is conditioned on the information contained within these
retrieved documents. This approach leverages the synergy between information
retrieval and natural language generation, enhancing the model’s ability to
provide accurate and contextually relevant answers. Chen et al. (2017); Guu et
al. (2020); Lewis et al. (2021); Izacard and Grave (2021); Sachan et al.
(2021); Lee et al. (2022); Jiang et al. (2022); Izacard et al. (2023); Nakano
et al. (2022); Qian et al. (2023); Lazaridou et al. (2022); Shi et al. (2023).

Standard RAG methods often involve a single retrieval step, which can be
insufficient for complex problems requiring multi-step reasoning. Yoran et al.
(2024). To address these limitations, various retrieval strategies such as
Iterative Retrieval Shao et al. (2023), Recursive Retrieval Trivedi et al.
(2023); Kim et al. (2023), and Adaptive Retrieval Jiang et al. (2023); Asai et
al. (2023); Yang et al. (2023); Schick et al. (2023); Zhang (2023) have been
proposed. Among these, adaptive retrieval refines the RAG framework by
enabling LLMs to actively determine the optimal moments and content for
retrieval, thereby enhancing the efficiency and relevance of the sourced
information. For example, Flare automates temporal retrieval by monitoring the
confidence levels during the generation process, such as the probability of
generated terms Jiang et al. (2023). When this probability falls below a
certain threshold, the retrieval system is activated to gather relevant
information, thereby optimizing the retrieval cycle.

However, another significant challenge with naive RAG is its reliance on the
user’s original query for retrieval. Formulating precise and clear queries is
difficult, leading to suboptimal retrieval effectiveness. Moreover, language
complexity and ambiguity further complicate the process, as models may
struggle with specialized vocabulary or ambiguous abbreviations. To enhance
retrieval effectiveness, query optimization strategies such as query expansion
and query transformation have been developed. Query expansion enriches the
content of the query by breaking down complex questions into simpler sub-
queries or creating multiple parallel queries Zhou et al. (2023); Dhuliawala
et al. (2023). Query transformation involves rewriting or rephrasing the
original query to improve retrieval effectiveness, using techniques like
prompt engineering and hypothetical document generation Ma et al. (2023); Peng
et al. (2024); Gao et al. (2022); Zheng et al. (2024). These query
optimization strategies are crucial for improving the effectiveness of RAG
systems, ensuring they provide accurate and contextually appropriate
responses.

While these existing methods are effective in many applications, they tend to
focus on either query rewriting or retriever adaptation. Even when both
aspects are considered, they are often addressed implicitly during the
generation process. Moreover, in adaptive retrieval methods, the LM typically
generates a response first and then decides whether additional retrieval is
necessary based on the generated output. For instance, Flare automates
temporal retrieval by evaluating the confidence in the generated terms Jiang
et al. (2023).

Given the robust semantic understanding capabilities of large language models
(LLMs), we propose a hypothesis: can we assess the necessity of document
retrieval before generating a response? This concept is inspired by the
behavior of students during open-book exams. Faced with a question, students
first understand the question, then evaluate their ability to answer it. If
they can, they respond directly; if not, they consult their textbooks to
gather the necessary information before crafting their final response. This
two-step approach ensures that answers are both accurate and comprehensive.
Applying this strategy to LLMs could potentially reduce the costs associated
with calling APIs of black-box models, while maintaining or even enhancing
response accuracy and relevance.

Building on this concept, this paper introduces Think-then-Act, an accurate
and efficient framework for retrieval augmentation, as illustrated in Figure
1. This framework incorporates a dual-phase evaluation and response process:
(i) assessing the input query to determine if it is clear and complete and if
it needs rewriting; (ii) evaluating the language model’s capability to answer
the query and whether additional information retrieval is necessary.

To validate the effectiveness of our proposed framework, we examine the
performance of Think-then-Act with gpt-3.5-turbo across five diverse datasets:
HotPotQA Yang et al. (2018), 2WikiMultihopQA Ho et al. (2020), StrategyQA Geva
et al. (2021), FEVER Thorne et al. (2018), and a custom-built Chinese Poetry
dataset. These datasets are chosen to comprehensively test various aspects of
our approach, including multi-hop reasoning, commonsense reasoning, fact-
checking, and domain-specific question answering. Our experimental results
demonstrate that the Think-then-Act framework significantly improves
retrieval-augmented generation’s performance, achieving higher accuracy and
efficiency compared to existing baselines. Notably, the framework shows robust
performance in both English and non-English contexts, highlighting its
versatility and potential for broader applications.

##  2 Related Work

Our framework involves two modules of RAG: (i) query optimization within the
context of RAG; and (ii) adaptive retrieval within the augmentation process of
RAG.

###  2.1 Query Optimization

A key issue with Naive RAG is its dependence on the user’s initial query (Gao
et al., 2024), often resulting in ineffective retrieval due to challenges in
crafting clear questions and managing intricate or ambiguous language. Query
transformation is an effective method for optimizing initial queries, which
focuses on retrieving information using a modified query instead of the user’s
original query.

Some studies use prompt engineering to enable LLM to generate a query based on
the original one for subsequent retrieval (Jagerman et al., 2023). Gao et al.
(2022) generates hypothetical documents, which are presumed answers to the
initial query. This approach emphasizes the similarity of embeddings between
these generated answers rather than focusing on the similarity of embeddings
related to the original problem or query. Zheng et al. (2024) using the Step-
back Prompting method abstracts the initial query to formulate a broader,
high-level conceptual question (step-back question). In addition to using LLM
for rewriting, Ma et al. (2023) also specifically trained a smaller model to
handle query rewriting tasks.

These methods enhance retrieval effectiveness; however, they assume that the
input query always requires rewriting. Our approach introduces an evaluation
step before rewriting, ensuring that the query is only modified if it is
deemed incomplete or ambiguous.

###  2.2 Adaptive Retrieval

To improve factual accuracy, language models often rely on external knowledge
via retrieval augmentation (Lewis et al., 2021). Conventional retrieval-
augmented generation (RAG) methods use a single retrieval step followed by
generation, which can be insufficient for complex, multi-step reasoning tasks.
Adaptive retrieval techniques optimize this process by allowing models to
dynamically decide when and what to retrieve, enhancing both efficiency and
relevance.

One strategy is to add retrieval capabilities through the fine-tuning of a
white-box generation model. Nakano et al. (2022) uses a reinforcement learning
framework to train the GPT-3 model to autonomously use a search engine during
text generation. It employs specific tokens to perform tasks such as making
search queries, reviewing search results, and adding references, thus
enhancing GPT-3’s abilities with the help of external search engines. Asai et
al. (2023) trained a flexible language model (LM) that can dynamically
retrieve passages as required. This model uses special tokens, called
’reflection tokens,’ classified into two types: ’retrieve’ and ’critic,’ to
generate and review both the retrieved passages and its own outputs. By using
these reflection tokens, the LM can be directed during the inference phase,
allowing it to adapt its behavior to suit various task needs. Additionally,
some researchers use prompt engineering methods. Graph-Toolformer (Schick et
al., 2023), for instance, separates the retrieval process into specific
stages, where LLMs actively use retrievers, utilize Self-Ask techniques, and
apply few-shot prompts to start search queries. Others (Jiang et al., 2023)
generates a preliminary answer first, then, based on whether the probability
of the generated terms falls below a certain threshold, decides if additional
information is needed before regenerating the response based on the initial
result.

The generate-then-retrieve approach, while effective, is inefficient for
queries that definitely need retrieval, as it introduces an extra generation
step. We propose an approach where the model’s capabilities are evaluated
prior to generation, which achieves a balance between precision and efficiency
in situations where absolute accuracy is not required.

##  3 Methodology

We present a dual-angle evaluated retrieval-augmented generation framework
Think-then-Act. This approach enhances both the query assessment and model
capability evaluation processes. Figure 1 provides an overview. This section
first introduces the query assessment and rewriting process in Section 3.1,
followed by the model capability check and information retrieval in Section
3.2.

###  3.1 Initial Query Assessment

####  3.1.1 Evaluation

Accurate responses require clear and precise questions. Therefore, our first
step involves evaluating the input query to determine whether it is clear and
complete, incomplete, or ambiguous. Leveraging the inherent semantic
understanding capabilities of large language models, we avoid the need for
additional models for this evaluation. Instead, we use a prompting method that
enables the model to self-assess the clarity and completeness of the input
query, and the model categorizes the query as CLEAR AND COMPLETE, INCOMPLETE,
or AMBIGUOUS.

####  3.1.2 Rewriting

If the evaluation categorizes the query as INCOMPLETE or AMBIGUOUS, the query
requires rewriting. Utilizing the powerful generation capabilities of large
language models, we employ a prompting method that enables the model to
generate the revised queries itself.

INCOMPELTE The model generates a more complete version of the query by filling
in any missing information, ensuring it is clear and comprehensive.

AMBIGUOUS The model resolves ambiguity by breaking down the query into
multiple, straightforward sub-queries, each addressing a specific aspect of
the original query. Formally, the overall process of initial query $q_{init}$
assessment and rewriting, resulting in the model’s final input $q_{final}$,
can be summarized as follows:

$q_{\text{final}}=\left\\{\begin{array}[]{lr}q_{\text{init}}&\quad\text{if}\;\mbox{CLEAR
AND COMPLETE}\\\ q_{\text{comp}}&\quad\text{if}\;\mbox{INCOMPLETE}\\\
q_{\text{group}}=&\\{q^{1},q^{2},\ldots,q^{k}\\}\quad\text{if}\;\mbox{AMBIGUOUS}\end{array}\right.$
(1)

###  3.2 Model Capability Check

After completing the initial query assessment and obtaining the final input
$q_{final}$, the next step involves evaluating the LM’s capability to answer
$q_{final}$. We propose two methods for this evaluation:

Direct Decision: In this straightforward approach, the LM directly outputs
either RETRIEVAL or NO RETRIEVAL. This binary decision indicates whether the
LM needs additional information to answer the query effectively.

Confidence Score: This method involves the LM generating a confidence score,
denoted as $\beta$, which represents its confidence level in answering the
question. By comparing this score to a predefined threshold $\beta^{\prime}$,
we can dynamically decide whether retrieval is necessary.
$\beta<\beta^{\prime}$, indicating that the model lacks sufficient confidence
to answer the query on its own, so retrieval is required.
$\beta>=\beta^{\prime}$, suggesting that the model is confident in its ability
to provide an accurate response without additional information, so retrieval
is not needed. $\beta^{\prime}\in[0,1]$. When $\beta^{\prime}=0$ it means that
retrieval is never performed. When $\beta^{\prime}=1$ it means that retrieval
is performed for every $q_{final}$.

####  3.2.1 Information Retrieval

If the model determines that additional information is necessary, we proceed
with the information retrieval step. Search engines possess features that
large language models (LLMs) lack, such as the ability to be easily and
quickly updated (Kasai et al., 2024). We use the Google search engine and
Wikipedia-API(wiki) as the retriever to obtain relevant documents $D$, 2
example in Table 1. Detailed settings for the retrieval process are described
in Section 4.2.

$y_{\text{output}}=\left\\{\begin{array}[]{ll}LM(\>q_{\text{final}}\>)&\text{if
}\>\beta\geq\beta^{\prime}\\\ LM(\>[\>D,\>q_{\text{final}}\>]\>)&\text{if
}\>\beta<\beta^{\prime}\end{array}\right.$ (2)

Question: Is popular science used to peer review papers?
---
The question is: ambiguous.
Sub-Question: What is popular science?
Probability of correct answer is: 0.5.
Sub-Answer: Popular science is a simplified version of scientific work.
Sub-Question: What types of documents does peer review use to verify papers?
Probability of correct answer is: 0.6.
Sub-Answer: Peer review uses detailed scientific information to verify papers.
So the final answer is: False.
Question: Does a lapidary work with items that are studied by geologists?
The question is: ambiguous.
Sub-Question: What are the materials a lapidary works with?
Probability of correct answer is: 0.4.
Sub-Answer: Quartz is a popular material for lapidary because it is relatively
easy to work with and comes in various colours and patterns. As it is ….
Sub-Question: What do geologists study?
Probability of correct answer is: 0.5.
Sub-Answer: Lapidarists work with stone, minerals and gemstones.
So the final answer is: True.

Table 1: Think-then-Act on StrategyQA: 2 samples

##  4 Experimental Setup

###  4.1 Task Settings

To thoroughly evaluate the capabilities of the Think-then-Act framework, we
selected tasks and datasets designed to comprehensively test various aspects
of our approach. Specifically, we chose three sub-tasks: Multihop QA,
Commonsense Reasoning, Fact Checking and Domain QA.

####  4.1.1 MultihopQA

Two multihop QA datasets are used for evaluation. (i) HotPotQA Yang et al.,
2018: consists of complex questions that require multi-hop reasoning, where
the answer to a question requires synthesizing information from multiple
documents. For our evaluation, we use the full test set, ensuring a
comprehensive assessment of our framework capabilitues of multihop reasoning.
(ii) 2WikiMultihopQA Ho et al., 2020 is a multi-hop question-answering dataset
that exploits the structured format in Wikidata and uses logical rules to
create questions. By evaluating on this dataset, we aim to test the
proficiency of our framework in handling structured information and executing
logical inference.

####  4.1.2 Commonsense Reasoning

Commonsense reasoning requires a blend of world knowledge and commonsense
understanding to generate accurate answers. For this purpose, we utilize the
StrategyQA dataset Geva et al. (2021), which consists of crowd sourced yes/no
questions, such as “Would a pear sink in water?”. The final answers are
extracted and matched against the gold standard answers using exact match to
evaluate the performance of our framework in commonsense reasoning tasks.

####  4.1.3 Fact Checking

We also employ the FEVER dataset Thorne et al. (2018) for fact verification
tasks. This dataset categorizes claims as "SUPPORTS", "REFUTES", or "NOT
ENOUGH INFO" based on evidence paragraphs extracted from Wikipedia. To ensure
a challenging evaluation, we sample a balanced set of instances where GPT-3’s
chain-of-thought (CoT) method makes both correct and incorrect predictions.
This approach allows us to rigorously test the model’s ability to verify facts
using evidence-based reasoning.

####  4.1.4 Domain QA

For the domain-specific question answering task, we utilize a custom-built
dataset focused on Chinese Poetry. This dataset was developed in response to
recurring issues with existing QA models, such as ChatGPT, which often
incorrectly match poetry verses with their titles and authors. Our analysis
revealed two primary reasons for these errors: firstly, the models may possess
accurate parametric knowledge but still generate incorrect answers; secondly,
they may lack the requisite information for obscure poetry verses.
Consequently, this dataset is ideally suited to evaluate the effectiveness of
our framework. Additionally, this allows us to test the effectiveness of our
framework in the Chinese language context, extending its applicability beyond
just English.

Our custom dataset comprises 9,791 poetry verses from 60 different poets,
providing a comprehensive basis for testing. This dataset enables us to
thoroughly assess the ability of our framework to handle both common and
obscure queries in the domain of classical Chinese poetry.

Figure 2: Overall results using the EM metric.Think-then-Act parameter
$\beta^{\prime}=0.5$.

###  4.2 Retriever Details

In the information retrieval step, we use two systems to obtain relevant
information:

Wikipedia-API(wiki): For the final query $q_{final}$, we search through
Wikipedia and select the top sentences from the relevant Wikipedia pages. This
approach leverages the structured and comprehensive nature of Wikipedia to
provide accurate and detailed information.

Google Search Engine: For queries that can be directly answered, such as
"Where is the capital of China?", Google searches often present direct
"$answer\;boxes$". We utilize these explicit answers for straightforward
questions. For more complex queries, Google provides "$organic\;results$" as
the main search output.

For wiki and the second case of Google, we select the top 3 most similar to
the query ranked by the pre-trained Sentence BERT modelReimers and Gurevych
(2019) as context.

###  4.3 Baseline

To provide a comprehensive evaluation of our framework, we compare it against
the following baselines. (i) Standard Prediction (Standard): This baseline
involves directly predicting the label based on the input, utilizing the same
number of in-context learning examples as our framework. (ii) Original Chain-
of-Thought (CoT)Wei et al. (2023): This approach predicts the label after
generating an explanatory chain-of-thought. It helps in understanding the
model’s reasoning process and its impact on the final prediction. (iii)
Retrieve-then-Read: This is the standard retrieval-augmented method where
retrieved documents are concatenated with the question to form the input. This
baseline allows us to measure the performance gains from our dual-focus
approach compared to traditional retrieval-augmented methods.

Datasets | HotPotQA | 2Wiki. | StrategyQA | FEVER | ChinesePoetry
---|---|---|---|---|---
Metrics | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1
Standard | 42.7 | 51.3 | 36.1 | 54.5 | 40.5 | 57.2 | 37.8 | 50.4 | 36.8 | 39.3
CoT | 47.9 | 59.7 | 40.3 | 60.2 | 44.8 | 60.1 | 42.1 | 57.4 | 40.0 | 49.2
Retrieve-then-Read | 52.3 | 66.4 | 42.7 | 69.3 | 37.5 | 50.7 | 40.3 | 59.2 | 69.9 | 76.2
Think-then-Act(ours) | 56.9 | 65.8 | 52.6 | 69.7 | 62.9 | 71.2 | 53.9 | 55.7 | 68.6 | 70.1
Table 2:  Comparison between Think-then-Act ($\beta^{\prime}=0.5$) and
baselines on all datastes using EM & F1.

##  5 Experimental Results

We first report the overall results across the selected tasks and datasets,
comparing the performance of the Think-then-Act framework with all the
baselines introduced in Section 4.3. We then conduct ablation experiments to
study the efficacy of various design choices within our method. This
structured analysis allows us to thoroughly evaluate the strengths and areas
for improvement in our approach.

###  5.1 Comparison with Baselines

Figure 2 displays the performance comparison of the Think-then-Act framework
against the baselines across various tasks and datasets. Our framework
generally outperforms the baselines, indicating its superior capability in
enhancing retrieval-augmented generation.

Figure 3: Comparison of Think-then-Act and Retrieval-then-
Read($\beta^{\prime}=0.5$) on the Chinese Poetry dataset: generation
accuracy(blue) and retrieval ratio(red).

Comparisons on StrategyQA: The most notable improvement is observed in
StrategyQA, shown in Table 3. The Chain of Thought(CoT)method (Wei et al.,
2023), which involves deeper question analysis, outperforms direct retrieval
methods. This is directly related to the characteristics of the StrategyQA
dataset. For commonsense reasoning tasks, deeply analyzing and understanding
the question is more crucial than acquiring additional information, few
example in Table 1. This trend is similarly observed in the FEVER dataset,
where accurate fact verification benefits more from a thorough understanding
of the query rather than from additional data.

Datasets | StrategyQA | FEVER
---|---|---
Metrics | EM | F1 | EM | F1
Retrieve-then-Read | 37.5 | 50.7 | 40.3 | 59.2
Think-then-Act(ours) | 62.9 | 71.2 | 53.9 | 55.7

Table 3: Comparison between Think-then-Act ($\beta^{\prime}=0.5$) and
Retrieve-then-Read on all StrategyQA and FEVER

Comparisons on ChinesePoetry: On our custom ChinesePoetry dataset, our method
performs comparably to the Retrieve-then-Read baseline. This can be attributed
to the clarity and completeness of the questions in this dataset, where
additional information retrieval significantly enhances accuracy. However,
unlike the baseline method that retrieves information for all queries, our
approach first assesses the model’s capability before deciding whether
retrieval is necessary. As shown in Figure 3, our method retrieves information
for only 36.8% of the questions, achieving the same effectiveness as
retrieving for 100% of the queries. This selective retrieval significantly
reduces computational costs.

We report all metrics for the every baselines in Table 2, highlight the
performance metrics (EM and F1 scores) for different methods across various
datasets. Our Think-then-Act framework consistently demonstrates superior
performance, particularly in tasks requiring complex reasoning and fact
verification. Notably, it achieves the highest EM scores in HotPotQA (56.9),
2WikiMultihopQA (52.6), StrategyQA (62.9), and FEVER (53.9), showcasing its
robustness and adaptability. The framework’s comparable performance in the
ChinesePoetry dataset (EM: 68.6) against the Retrieve-then-Read baseline (EM:
69.9) further illustrates its efficiency in handling domain-specific tasks
with reduced computational overhead.

Figure 4: the Exact Match (EM) scores across various datasets with different
$\beta^{\prime}$ values Figure 5: Comparison of Think-then-
Act($\beta^{\prime}=0.5$, LM postprocessing) and FLARE(LM preprocessing) on
the Chinese Poetry dataset: generation accuracy(blue) and retrieval
ratio(red).

###  5.2 Ablation Study

Our framework Think-then-Act, primarily consists of two components: query
assessment and model capability check. Unlike traditional approaches, we do
not omit these parts separately to demonstrate their effectiveness, as
previous studies have already established their importance. Instead, our
ablation study focuses on two main experiments to validate the design choices
and their impact on performance.

Impact of Different $\boldsymbol{\beta^{\prime}}$: As mentioned in Section
3.2, we examine how varying the threshold $\beta^{\prime}$ for model
confidence affects the results. This helps us understand the optimal threshold
for balancing retrieval necessity and model confidence.

Figure 4 displays the Exact Match (EM) scores across various datasets with
different $\beta^{\prime}$ values. As observed, the performance improves
significantly when $\beta^{\prime}$ is increased from $0.0$ to $0.5$. Beyond
$0.5$, the improvement plateaus, indicating diminishing returns. Therefore, we
set $\beta^{\prime}$ to $0.5$ for optimal performance, balancing the trade-off
between retrieval and self-reliance of the model.

Preprocess and Postprocess: We compare our framework with a method inspired by
FLARE (Jiang et al., 2023), where the LM generates a temporary next sentence
and checks the token probabilities before deciding on retrieval. We refer to
this as the LM preprocess approach. In contrast, our framework first assesses
whether retrieval is needed and then generates the response, which we term as
the LM postprocess approach.

Figure 5 illustrates the comparison between LM preprocessing and LM
postprocessing. The results show that our postprocessing approach achieves a
comparable EM score (68.6%) to the preprocessing approach (69.9%), but with a
significantly lower retrieval ratio (36.8% vs. 77.3%). This indicates that our
method is more efficient, reducing the number of retrievals required while
maintaining similar performance. Consequently, this leads to faster processing
times and enhanced resource efficiency.

##  6 Conclusion

This paper presents the Think-then-Act framework, enhancing retrieval-
augmented generation by combining query transformation and model capability
assessment. Our approach evaluates query clarity and model confidence,
triggering retrieval only when necessary, improving accuracy, and optimizing
resources. Experiments on datasets including MultihopQA, Commonsense
Reasoning, FEVER, and a custom Chinese poetry dataset show significant
improvements over baselines. The framework proves effective in both English
and non-English contexts. Ablation studies confirmed the optimal model
confidence threshold and highlighted efficiency gains from our approach. The
Think-then-Act framework offers a robust solution for enhancing retrieval-
augmented generation, paving the way for more accurate and efficient LLM
applications. Future work will refine query assessment and extend the
framework to additional languages and domains.

##  7 Limitations

While the Think-then-Act framework has demonstrated promising results, several
limitations need to be addressed in future research. Firstly, our study
exclusively utilized black-box models, such as GPT-3.5, which necessitate API
calls for each interaction. This approach incurs significant costs and poses
potential security risks due to the transmission of data over external
servers. Inspired by (Asai et al., 2023), future work could focus on fine-
tuning a white-box model based on our framework. This would enable local
deployment, thereby reducing costs and enhancing data security by processing
all information internally. Secondly, although we conducted experiments on
five distinct datasets, each experiment was isolated to a single type of
dataset. This approach does not fully capture the versatility and robustness
of our framework across mixed-type scenarios. Future research should
investigate the performance of our framework on more diverse and comprehensive
datasets that incorporate various types of questions and contexts within a
single dataset. This would provide a more rigorous validation of the
framework’s effectiveness and adaptability in real-world applications.

## References

  * Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023.  Self-rag: Learning to retrieve, generate, and critique through self-reflection.  _Preprint_ , arXiv:2310.11511.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–718, Nusa Dua, Bali. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.  Chain-of-verification reduces hallucination in large language models.  _Preprint_ , arXiv:2309.11495.
  * Gao et al. (2022) Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022.  Precise zero-shot dense retrieval without relevance labels.  _Preprint_ , arXiv:2212.10496.
  * Gao et al. (2024) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024.  Retrieval-augmented generation for large language models: A survey.  _Preprint_ , arXiv:2312.10997.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _Preprint_ , arXiv:2002.08909.
  * Ho et al. (2020) Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020.  Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6609–6625, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Izacard et al. (2023) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023.  Atlas: Few-shot learning with retrieval augmented language models.  _Journal of Machine Learning Research_ , 24(251):1–43.
  * Jagerman et al. (2023) Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023.  Query expansion by prompting large language models.  _Preprint_ , arXiv:2305.03653.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Jiang et al. (2022) Zhengbao Jiang, Luyu Gao, Jun Araki, Haibo Ding, Zhiruo Wang, Jamie Callan, and Graham Neubig. 2022.  Retrieval as attention: End-to-end learning of retrieval and reading within a single transformer.  _Preprint_ , arXiv:2212.02027.
  * Jiang et al. (2023) Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023.  Active retrieval augmented generation.  _Preprint_ , arXiv:2305.06983.
  * Kasai et al. (2024) Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. 2024.  Realtime qa: What’s the answer right now?  _Advances in Neural Information Processing Systems_ , 36.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019.  Generalization through memorization: Nearest neighbor language models.  _arXiv preprint arXiv:1911.00172_.
  * Kim et al. (2023) Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuk Park, and Jaewoo Kang. 2023.  Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models.  _Preprint_ , arXiv:2310.14696.
  * Lazaridou et al. (2022) Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022.  Internet-augmented language models through few-shot prompting for open-domain question answering.  _Preprint_ , arXiv:2203.05115.
  * Lee et al. (2022) Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher D. Manning, and Kyoung-Gu Woo. 2022.  You only need one model for open-domain question answering.  _Preprint_ , arXiv:2112.07381.
  * Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Preprint_ , arXiv:2005.11401.
  * Luu et al. (2022) Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022.  Time waits for no one! analysis and challenges of temporal misalignment.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5944–5958, Seattle, United States. Association for Computational Linguistics.
  * Ma et al. (2023) Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023.  Query rewriting for retrieval-augmented large language models.  _Preprint_ , arXiv:2305.14283.
  * Nakano et al. (2022) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022.  Webgpt: Browser-assisted question-answering with human feedback.  _Preprint_ , arXiv:2112.09332.
  * OpenAI (2023) R OpenAI. 2023.  Gpt-4 technical report. arxiv 2303.08774.  _View in Article_ , 2(5).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744.
  * Peng et al. (2024) Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024.  Large language model based long-tail query rewriting in taobao search.  _Preprint_ , arXiv:2311.03758.
  * Qian et al. (2023) Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  Webbrain: Learning to generate factually correct articles for queries by grounding on large web corpus.  _Preprint_ , arXiv:2304.04358.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _Preprint_ , arXiv:1908.10084.
  * Röttger and Pierrehumbert (2021) Paul Röttger and Janet Pierrehumbert. 2021.  Temporal adaptation of BERT and performance on downstream document classification: Insights from social media.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2400–2412, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sachan et al. (2021) Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. 2021.  End-to-end training of multi-document reader and retriever for open-domain question answering.  _Preprint_ , arXiv:2106.05346.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _Preprint_ , arXiv:2302.04761.
  * Shao et al. (2023) Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023.  Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy.  _Preprint_ , arXiv:2305.15294.
  * Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023.  Replug: Retrieval-augmented black-box language models.  _arXiv preprint arXiv:2301.12652_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Trivedi et al. (2023) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023.  Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.  _Preprint_ , arXiv:2212.10509.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.  _Preprint_ , arXiv:2201.11903.
  * Yang et al. (2023) Hui Yang, Sifu Yue, and Yunzhong He. 2023.  Auto-gpt for online decision making: Benchmarks and additional opinions.  _Preprint_ , arXiv:2306.02224.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yoran et al. (2024) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024.  Making retrieval-augmented language models robust to irrelevant context.  _Preprint_ , arXiv:2310.01558.
  * Zhang (2023) Jiawei Zhang. 2023.  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt.  _Preprint_ , arXiv:2304.11116.
  * Zheng et al. (2024) Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024.  Take a step back: Evoking reasoning via abstraction in large language models.  _Preprint_ , arXiv:2310.06117.
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023.  Least-to-most prompting enables complex reasoning in large language models.  _Preprint_ , arXiv:2205.10625.

##  Appendix A Appendix

Question: Could Carl Friedrich Gauss speak to someone 100 miles away?
---
The question is: ambiguous.
Sub-Question: What device allows people to speak to each other even if they
are 100 miles apart?
Probability of correct answer is: 0.8.
Sub-Answer: Carl Friedrich Gauss was born in 1777.
Sub-Question: When was #1 invented?
Probability of correct answer is: 0.8.
Sub-Answer: Speaking to someone 100 miles away requires a telephone.
Sub-Question: When did Carl Friedrich Gauss die?
Probability of correct answer is: 0.8.
Sub-Answer: The telephone was invented in 1876.
So the final answer is: False.
Question: Did Jackson 5 members exceed number in The Osmonds?
The question is: ambiguous.
Sub-Question: How many members did The Jackson 5 have?
Probability of correct answer is: 0.3.
Sub-Answer: The group was founded in 1964 in Gary, Indiana, and originally
consisted of brothers Jackie, Tito, Jermaine, Marlon and Michael. They were
managed by their ….
Sub-Question: How many members did The Osmonds have?
Probability of correct answer is: 0.3.
Sub-Answer: The Osmonds then and now · Donny Osmond · Marie Osmond · Jay
Osmond · Merrill Osmond. Get info without leaving the page. · Wayne Osmond ·
Alan Osmond.
So the final answer is: False.
Question: Is popular science used to peer review papers?
The question is: ambiguous.
Sub-Question: What is popular science?
Probability of correct answer is: 0.5.
Sub-Answer: Popular science is a simplified version of scientific work.
Sub-Question: What types of documents does peer review use to verify papers?
Probability of correct answer is: 0.6.
Sub-Answer: Peer review uses detailed scientific information to verify papers.
So the final answer is: False.
Question: Does a lapidary work with items that are studied by geologists?
The question is: ambiguous.
Sub-Question: What are the materials a lapidary works with?
Probability of correct answer is: 0.4.
Sub-Answer: Quartz is a popular material for lapidary because it is relatively
easy to work with and comes in various colours and patterns. As it is ….
Sub-Question: What do geologists study?
Probability of correct answer is: 0.5.
Sub-Answer: Lapidarists work with stone, minerals and gemstones.
So the final answer is: True.
Question: Could Carl Friedrich Gauss speak to someone 100 miles away?
The question is: ambiguous.
Sub-Question: What device allows people to speak to each other even if they
are 100 miles apart?
Probability of correct answer is: 0.1.
Sub-Answer: Start with a handheld. I like Yaesus.
Sub-Question: When was #1 invented?
Probability of correct answer is: 0.4.
Sub-Answer: 1 (one, unit, unity) is a number representing a single or the only
entity. 1 is also a numerical digit and represents a single unit of counting
or ….
Sub-Question: When did Carl Friedrich Gauss die?
Probability of correct answer is: 0.3.
Sub-Answer: Carl Friedrich Gauss (born April 30, 1777, Brunswick
[Germany]—died February 23, 1855, · Gauss was the only child of poor parents.
· Gauss’s first ….
So the final answer is: False.
Question: Can you listen to the entire iTunes song catalog in one year?
The question is: ambiguous.
Sub-Question: How many songs are on iTunes?
Probability of correct answer is: 0.8.
Sub-Answer: iTunes has around 43 million songs as of 2017.
Sub-Question: What is the average song length?
Probability of correct answer is: 0.9.
Sub-Answer: The average length of a song is 3 minutes.
Sub-Question: What is #1 multiplies by #2?
Probability of correct answer is: 0.6.
Sub-Answer: There are 525,600 minutes in a year.
So the final answer is: False.
Question: Can you listen to the entire iTunes song catalog in one year?
The question is: ambiguous.
Sub-Question: How many songs are on iTunes?
Probability of correct answer is: 0.0.
Sub-Answer: If you go to the "Songs" window and reenable the status bar by
selecting "View" then "Show Status Bar" you will get the total count at the ….
Sub-Question: What is the average song length?
Probability of correct answer is: 0.2.
Sub-Answer: Nowadays, songs average around 3:15/3:30 which shows a decrease in
length by up to 60 seconds. On top of that, we’re constantly seeing way more
….
Sub-Question: What is #1 multiplies by #2?
Probability of correct answer is: 0.7.
Sub-Answer: 1/2.
So the final answer is: False.

Table 4: Think-then-Act on StrategyQA: 10 samples