# Large Language Models Can Self-Correct with Minimal Effort

Zhenyu Wu1,2, Qingkai Zeng2, Zhihan Zhang2, Zhaoxuan Tan2, Chao Shen1, Meng
Jiang2
1Xi’an Jiaotong University, 2University of Notre Dame
{zwu23, qzeng, zzhang23, ztan3<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author

###### Abstract

Intrinsic self-correct was a method that instructed large language models
(LLMs) to verify and correct their responses without external feedback.
Unfortunately, the study concluded that the LLMs could not self-correct
reasoning yet. We find that a simple yet effective verification method can
unleash inherent capabilities of the LLMs. That is to mask a key condition in
the question, add the current response to construct a verification question,
and predict the condition to verify the response. The condition can be an
entity in an open-domain question or a numeric value in a math question, which
requires minimal effort (via prompting) to identify. We propose an iterative
verify-then-correct framework to progressively identify and correct (probably)
false responses, named ProCo. We conduct experiments on three reasoning tasks.
On average, ProCo, with GPT-3.5-Turbo-1106 as the backend LLM, yields $+6.8$
exact match on four open-domain question answering datasets, $+14.1$ accuracy
on three arithmetic reasoning datasets, and $+9.6$ accuracy on a commonsense
reasoning dataset, compared to Self-Correct.

Large Language Models Can Self-Correct with Minimal Effort

Zhenyu Wu1,2, Qingkai Zeng2, Zhihan Zhang2, Zhaoxuan Tan2, Chao Shen1††thanks:
Corresponding author, Meng Jiang2 1Xi’an Jiaotong University, 2University of
Notre Dame {zwu23, qzeng, zzhang23, ztan3<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Reasoning is a cognitive process that uses evidence, arguments, and logic to
arrive at conclusions or judgements (Huang and Chang, 2023). People have been
exploiting and improving the reasoning ability of large language models
(LLMs). Wei et al. proposed chain-of-thought (CoT) prompting and yielded
promising results on several reasoning tasks, such as arithmetic reasoning
(Kojima et al., 2022; Zhou et al., 2023), commonsense reasoning (Wei et al.,
2022; Zhang et al., 2023; Wang et al., 2023b), and open-domain question
answering (Wang et al., 2023a), using only a few or no reasoning exemplars.
CoT guides LLMs to generate intermediate reasoning steps instead of generating
the final answer directly, which helps the LLMs simulate the human-like
reasoning process.

a Self-Correct (Kim et al., 2023) prompts an LLM to verify the correctness of
its generated responses and uses this verification as feedback to refine the
responses. Huang et al. pointed out that this method was ineffective as LLMs
could not properly identify and correct mistakes in their output without
external feedback.

b ProCo performs three steps: (1) Initialization: Use CoT method to generate
an initial answer. (2) Verification: Mask the key condition in the question
and use the previous generated answer as a new condition to construct the
verification question. Solve the verification question to get the verified
answer and check if the verified answer and the key condition are equivalent.
If they are equivalent, the previous generated answer is adopted as the final
answer, otherwise add it to the set of potentially incorrect answers. (3)
Correction: Use the set of potentially incorrect answers as feedback to
correct previous generated answer. By cycle executing step (2) and step (3),
the performance of LLMs on various complex reasoning tasks is progressively
enhanced.

Figure 1: The proposed ProCo method helps LLMs identify incorrect answers and
progressively correct them.

Although CoT enables LLMs to handle some complex reasoning examples, it
remains vulnerable to the negative impact of individual errors in each step.
Specifically, even a minor error in one step can alter the trajectory of the
entire reasoning process, ultimately leading to an incorrect conclusion. To
address this issue, Dhuliawala et al.; Kim et al. have explored the
verification and correction on the responses. For example, as shown in Figure
1a, for a given question and its initial LLM-generated answer, Self-Correct
(Kim et al., 2023) first instructs the LLM to criticize its generated answer
using the hint: “ _Review previous answer and find mistakes_ ”. Then, Self-
Correct instructs the LLM to refine initial answers based on the critique.

Method NQ CSQA AQuA CoT $40.3$ $72.9$ $51.3$ Self-Correct $40.1$ $65.9$ $48.7$
ProCo (Ours) $\mathbf{48.0}$ $\mathbf{75.5}$ $\mathbf{65.2}$

Table 1: Performance comparison of different prompting methods using
GPT-3.5-Turbo as backend LLM.

However, recent studies (Huang et al., 2024; Gou et al., 2024) have cast doubt
on the intrinsic self-correction capability of LLMs. Their research indicates
that _without external feedback_ , such as ground truth to verify the
correctness of previous responses, LLMs struggle to correct their prior
outputs. Since LLMs could not properly judge the correctness of their prior
responses, the refined response might be even worse than the initial response.

To unleash inherent capabilities of LLMs to detect and rectify incorrect
responses without external feedback, we introduce _substitute verification_.
Let us look at a specific example. Given an open-domain question _“Who plays
Skylar on Lab Rats: Elite Force?”_ , we first prompt an LLM to generate an
initial answer for the question, e.g., _“Paris Berelc”_. Next, we identify a
key condition in the question that is relevant to the problem-solving process,
such as _“Skylar”_. By masking the key condition in the question and adding
the initial answer as a new condition, we can obtain a verification question:
_“Who plays X on Lab Rats: Elite Force? Suppose the answer is Paris Berelc.
What is the value of unknown variable X?”_. We use the LLM to solve the
verification question, and we get that X is _“Skylar Storm”_. By verifying
whether _“Skylar Storm”_ is equivalent to _“Skylar”_ , we can predict that the
initial answer is likely correct.

Based on substitute verification, we propose a simple yet effective prompting
method Progressive Correction (ProCo). Figure 1 illustrates the difference
between the Self-Correct and ProCo methods. Compared with Self-Correct, our
proposed ProCo, highlighting two primary distinctions:

(1) Verification Method. To improve verification accuracy, we propose the
substitute verification method. Specifically, ProCo first identifies key
conditions that are relevant to the problem-solving process. It then masks one
of the key conditions in the question and takes the generated answer as a new
condition to construct the verification question. Finally, ProCo solves the
verification question and gets the verified answer. If the verified answer and
the key condition are equivalent, it indicates that the generated answer is
likely to be correct.

(2) Correction Method. ProCo employs the substitute verification method to
accurately verify the correctness of the LLM-generated answer. If an answer is
deemed incorrect, ProCo adds it to the set of potentially incorrect answers,
which then serves as feedback to guide LLMs in correcting previous mistakes
with the hint: “ _the answer is likely not in_ {set of potentially incorrect
answers}”. By iteratively executing verification and correction, ProCo
effectively prevents the repetition of previous mistakes, thereby
progressively improving the quality of responses.

We conducted evaluations of ProCo using a variety of LLMs, including
GPT-3.5-Turbo-1106, GPT-4-0125-Preview, and the open-source Mixtral-8x7B.
These evaluations spanned three distinct tasks: arithmetic reasoning,
commonsense reasoning, and open-domain question answering. The experimental
results reveal that ProCo consistently outperforms existing methods. As shown
in Table 1, ProCo achieves a $7.9$ exact match (EM) improvement on the NQ
dataset, a $16.5$ absolute increase on the AQuA dataset, and a $9.6$ absolute
improvement on the CSQA dataset compared to the Self-Correct method.

In summary, our main contribution include:

  * •

Based on our research, we have determined that LLMs are capable of intrinsic
self-correction, provided that the prompt design is carefully structured
within a framework focused on verification and correctness.

  * •

We introduce a novel prompting method, ProCo, which utilizes an iterative
verify-then-correct framework. ProCo progressively refines responses by
identifying key conditions and formulating verified questions specific to
these conditions.

  * •

We conduct extensive experiments across three different complex reasoning
tasks and demonstrate that ProCo achieves significant improvements in both
black-box and open-source LLMs.

##  2 Related Work

Self-Correct (Kim et al., 2023) methods, which aim to enhance the quality of
LLM responses by providing feedback on initial attempts (Kim et al., 2023;
Madaan et al., 2023; Chen et al., 2024), have demonstrated effectiveness in
various reasoning tasks. These tasks include arithmetic reasoning (Madaan et
al., 2023; Welleck et al., 2023), open-domain question answering (Dhuliawala
et al., 2023; Yu et al., 2023b), commonsense reasoning (Kim et al., 2023), and
others (Chen et al., 2024; Le et al., 2022). Self-Correct methods vary in the
source and format of feedback, and the process of verifying the correctness of
LLM output.

##### Source and Format of Feedback

Interscript (Tandon et al., 2021) corrected the initial output of the LLM by
integrating natural language feedback from humans. Due to the high cost of
human feedback, various approaches have employed scalar reward functions as an
alternative. For instance, Rainer (Liu et al., 2022) used reinforcement
learning to generate contextual relevant knowledge in response to queries.
Self-Correction (Welleck et al., 2023) trained a corrector to iteratively
correct imperfect outputs. Other sources, such as compilers (Chen et al.,
2024) or search engines (Yu et al., 2023b) can provide domain-specific
feedback. Recent research used LLMs to generate feedback. Self-Correct (Kim et
al., 2023) and Self-Refine (Madaan et al., 2023) utilized LLMs to verify and
refine their initial outputs. However, Huang et al. questioned the intrinsic
self-correcting capability of LLMs, indicating that without external feedback,
LLMs struggle to correct their previous responses. To unleash inherent
capabilities of LLMs to detect and rectify incorrect responses without
external feedback, we introduce _substitute verification_. By providing
natural language feedback based on verification results, we can steer LLMs
away from incorrect answers, thus enhancing their performance in various
reasoning tasks.

##### Verify Correctness of LLM Output

Several studies trained or fine-tuned verification models to check the
correctness of the answer. Cobbe et al. fine-tuned GPT-3 as a scorer to
calculate solution-level verification score and choose the highest-scoring
answer as the final answer. Li et al. fine-tuned deberta-v3-large (He et al.,
2021) to predict the probability that the generated reasoning path leads to
the correct answer. These methods require a significant amount of human
annotations. To eliminate human labor, Peng et al. proposed using an external
database to identify incorrect knowledge in LLM outputs. These methods are
typically restricted to checking certain types of errors. To address these
limitations, Miao et al. used the LLM to individually verify the correctness
of each step in the arithmetic reasoning path based on the preceding steps.
Dhuliawala et al. used manual-crafted demonstrations as context to prompt the
LLM to check the correctness of its output. All of these methods only verify
the correctness of LLM outputs and select the verified answer as the final
answer. In contrast, our method iterates a verify-then-correct process to
progressively identify and correct incorrect answers.

Figure 2: Key conditions in complex reasoning tasks play a crucial role in the
problem-solving process. These conditions can take various forms: a numeric
value in arithmetic questions, an entity in open-domain questions, or a
concept in commonsense questions.

##  3 Preliminaries

Given a question $Q=[\mathcal{C},q]$, consisting of $n$ conditions
$\mathcal{C}=\\{c_{i}\\}_{i=1}^{n}$ and a question sentence $q$, we instruct
the LLM to generate the final answer $\hat{a}$. Condition $c_{i}$ can be a
numeric value in arithmetic questions, an entity in open-domain questions, or
a concept in commonsense questions, as illustrated in Figure 2. Among these
conditions, key conditions $c^{(\text{k})}\in\mathcal{C}$ are crucial to the
problem-solving process and are utilized in the substitute verification
process. We introduce two innovative approaches for identifying these key
conditions.

##### Similarity-based Key Condition Identification

Numerical values are critical in arithmetic questions; thus, we select
numerical values relevant to the problem-solving process as key conditions.
Specifically, for a math word problem $Q$, following (Wu et al., 2024), we
split it into $n$ conditional sentences $\\{c_{i}^{(\text{s})}\\}_{i=1}^{n}$
and a question sentence $q$. We then use a pre-trained language model, such as
SimCSE (Gao et al., 2021), to encode the conditional sentences and the
question sentence into vector representations, resulting in
$\\{\mathbf{c}_{i}^{(\text{s})}\\}_{i=1}^{n}$ and $\mathbf{q}$, which are
$d$-dimensional vectors. Subsequently, we calculate the cosine similarity
between $\mathbf{c}_{i}^{(\text{s})}$ and $\mathbf{q}$:

$s_{i}=\frac{\mathbf{c}_{i}^{(\text{s})\top}\mathbf{q}}{{\|\mathbf{c}_{i}^{(\text{s})}\|}\cdot{\|\mathbf{q}\|}}$
(1)

The conditional sentence with the highest cosine similarity to $q$ is
selected, and regular expressions are used to identify the numerical value in
the conditional sentence as the key condition $c^{(\text{k})}$.

##### Zero-shot Key Condition Identification

For open-domain or commonsense questions, we instruct LLMs to identify
entities or concepts relevant to the problem-solving process as key
conditions. For instance, given an open-domain question $Q$, we construct a
key condition identification prompt: “ _Given the question below, the task is
to identify a set of entities within the question and then select the one that
is most relevant to the problem-solving process. $Q$_”. We then input this
prompt into an LLM to receive the key condition $c^{(\text{k})}$.

##  4 Proposed Approach

###  4.1 Overview

In this section, we present the overall pipeline of the proposed Progressive
Correction (ProCo) prompting method which consists of three steps. Figure 1b
illustrates the ProCo method. Initially, ProCo prompts the LLM to generate an
answer in response to a given question (Sec. 4.2). Subsequently, to enhance
the preliminary answer, ProCo identifies a key condition and generates a
corresponding verification question-answer pair based on that condition (Sec.
4.3). The final answer is refined by verifying the question-answer pair,
ensuring the answer’s consistency and accuracy (Sec. 4.4). The full prompts
used in the experiments can be found in Appendix A.3.

###  4.2 Generate Initial Answer

Given a question $Q=[\mathcal{C},q]$, comprising $n$ conditions
$\mathcal{C}=\\{c_{i}\\}_{i=1}^{n}$ and a question sentence $q$. We use one of
the existing prompting methods to generate an initial answer $a_{0}$ for the
question $Q$, e.g., CoT (Kojima et al., 2022), GenRead (Yu et al., 2023a), RAG
(Khattab et al., 2023), and so on. By default, we use the CoT (Kojima et al.,
2022) prompting method to generate an initial answer.

###  4.3 Iterative Verify-then-Correct Process

We propose a novel iterative verify-then-correct method that first initializes
the set of potentially incorrect answers as an empty set
$\mathcal{P}_{0}=\varnothing$ and identifies the key condition
$c^{(\text{k})}\in\mathcal{C}$ within the question $Q$ (Sec. 3). The method
then progressively corrects the LLM-generated answer across $K$ iterations by
cyclically conducting verification and correction phases. Here we use the
$i$-th iteration as an example to illustrate the verify-then-correct process.

##### Verification Phase

The verification phase uses substitute verification method to verify the
correctness of the previous generated answer $a_{i-1}$. This phase encompasses
several substeps.

Initially, the key condition $c^{(\text{k})}$ within the question $Q$ is
replaced with a specific token X, resulting in a masked question.
Subsequently, a verification question $Q^{(\text{v})}_{i}$ is formulated by
appending the sentence “ _Suppose the answer is $a_{i-1}$. What is the value
of unknown variable X?_” to the masked question. The LLM is then instructed to
solve the verification question $Q^{(\text{v})}_{i}$ and produce the
corresponding answer $a^{(\text{v})}_{i}$. Finally, different strategies are
proposed to verify the correctness of $a_{i-1}$.

Match-based Verification. For arithmetic questions, if $a^{(\text{v})}_{i}$ is
equal to $c^{(\text{k})}$, it indicates that the previous answer $a_{i-1}$ is
most likely correct.

Proposition-based Verification. For open-domain or commonsense questions, we
propose a proposition-based verification method to verify the correctness of
the previously generated answer $a_{i-1}$. The intuition behind this is that
the question $Q^{(\text{v})}_{i}$ may have multiple valid answers, and
directly checking if $a^{(\text{v})}_{i}$ exactly matches $c^{(\text{k})}$
could result in misclassifying a correct answer as incorrect. Specifically, we
construct an answer verification prompt: “ _Determine the correctness of the
proposition: If the answer to question_ $Q^{(\text{v})}_{i}$ _is_
$c^{(\text{k})}$_, then X could also be_ $a^{(\text{v})}_{i}$”. We input this
prompt into an LLM and receive a judgment about the proposition’s correctness.
If the proposition is verified as correct, it indicates that the previously
generated answer $a_{i-1}$ is likely correct, and we select $a_{i-1}$ as the
final answer $\hat{a}$ and exit the loop. Otherwise, we add $a_{i-1}$ to the
set of potentially incorrect answers $\mathcal{P}_{i-1}$ to obtain the updated
set $\mathcal{P}_{i}$.

##### Correction Phase

During the correction phase, we use the set of potentially incorrect answers
$\mathcal{P}_{i}=\\{a_{0},\cdots,a_{i-1}\\}$ as feedback to generate a
corrected answer $a_{i}$. For a given question $Q$ and the set
$\mathcal{P}_{i}$, we append the phrase “ _the answer is likely not
$\mathcal{P}_{i}$_” to the question. This instructs the large language model
to re-answer the question while avoiding repeating previous mistakes.

###  4.4 Final Answer Determination

The process of verify-then-correct can be iterated until specific stopping
conditions are met. This process terminates under three situations: First, if
the answer $a_{i-1}$ is verified to be likely correct, it is selected as the
final answer. Second, if the corrected answer $a_{i}$ matches the previously
generated answer $a_{i-1}$, then $a_{i}$ is chosen as the final answer.
Lastly, if the iteration count surpasses the maximum allowed iterations $K$,
the last LLM-generated answer $a_{K}$ is adopted as the final answer.

##  5 Experiments

Method (GPT-3.5-Turbo-1106) Open-domain Question Answering Commonsense
Reasoning NQ TriviaQA WebQ HotpotQA CSQA EM F1 EM F1 EM F1 EM F1 Accuracy
_*Using LLMs to generate problem-related documents_ GenRead $42.2$ $49.4$
$70.8$ $74.8$ $41.3$ $48.5$ $38.0$ $43.2$ $67.3$ GenRead + ProCo $48.3$ $55.6$
$78.4$ $\mathbf{82.4}$ $46.7$ $53.9$ $\mathbf{47.0}$ $\mathbf{51.0}$
$\mathbf{76.4}$ _*Using search engines to retrieve problem-related documents_
RAG $45.3$ $52.4$ $72.7$ $76.4$ $40.1$ $46.9$ $37.0$ $41.1$ $65.9$ RAG + ProCo
$\mathbf{48.5}$ $\mathbf{56.0}$ $78.4$ $82.1$ $45.2$ $52.5$ $39.0$ $44.2$
$74.2$ _*Direct question answering without external documents_ CoT $40.3$
$46.4$ $69.2$ $72.2$ $38.2$ $44.6$ $28.0$ $31.2$ $72.9$ Self-Correct $40.1$
$47.1$ $71.3$ $74.1$ $39.2$ $45.7$ $29.0$ $32.4$ $65.9$ CoVe $43.4$ $48.9$
$76.4$ $79.4$ $43.1$ $49.0$ $31.0$ $35.2$ $73.1$ ProCo $48.0$ $54.8$
$\mathbf{78.7}$ $82.1$ $\mathbf{47.0}$ $\mathbf{57.0}$ $33.0$ $36.2$ $75.5$

Table 2: Results on NQ, TriviaQA, WebQ, HotpotQA, and CSQA datasets using
GPT-3.5-Turbo-1106 as the backend LLM. GenRead + ProCo indicates that using
the GenRead method generates an initial answer and progressively correcting
the initial answer using our proposed ProCo method. GenRead + ProCo
significantly outperform the original GenRead method. The best performance for
each dataset is shown in bold.

Method (Mixtral-8x7B) Open-domain Question Answering Commonsense Reasoning NQ
TriviaQA WebQ HotpotQA CSQA EM F1 EM F1 EM F1 EM F1 Accuracy _*Using LLMs to
generate problem-related documents_ GenRead $46.7$ $52.0$ $69.0$ $72.4$ $51.1$
$56.5$ $36.0$ $39.7$ $64.3$ GenRead + ProCo $48.5$ $53.7$ $72.3$ $75.8$ $52.0$
$57.5$ $38.0$ $42.3$ $70.4$ _*Using search engines to retrieve problem-related
documents_ RAG $48.8$ $54.6$ $75.3$ $78.5$ $46.3$ $52.1$ $37.0$ $40.2$ $66.3$
RAG + ProCo $\mathbf{51.6}$ $\mathbf{57.1}$ $\mathbf{79.6}$ $\mathbf{83.0}$
$50.3$ $56.3$ $\mathbf{41.0}$ $\mathbf{43.7}$ $71.8$ _*Direct question
answering without external documents_ CoT $42.6$ $48.2$ $66.7$ $70.3$ $46.6$
$51.9$ $29.0$ $34.4$ $68.4$ Self-Correct $44.8$ $50.5$ $71.3$ $74.8$ $47.5$
$51.9$ $32.0$ $36.2$ $49.8$ CoVe $47.6$ $53.0$ $73.2$ $76.4$ $53.4$ $58.2$
$33.0$ $36.9$ $70.8$ ProCo $50.7$ $53.6$ $74.5$ $76.6$ $\mathbf{55.1}$
$\mathbf{59.2}$ $35.0$ $41.3$ $\mathbf{72.7}$

Table 3: Results on NQ, TriviaQA, WebQ, HotpotQA, and CSQA datasets using
Mixtral-8x7B as the backend LLM.

###  5.1 Experimental Setup

##### Datasets.

We evaluate ProCo on three complex reasoning tasks: arithmetic reasoning
(GSM8K (Cobbe et al., 2021b), AQuA (Ling et al., 2017), and MATH (Hendrycks et
al., 2021)); open-domain question answering (NQ (Kwiatkowski et al., 2019),
TriviaQA (Joshi et al., 2017), WebQ (Berant et al., 2013), and HotpotQA (Yang
et al., 2018)); and commonsense reasoning (CSQA (Talmor et al., 2019)).
Detailed information about these datasets is available in Appendix A.1.

##### Baselines.

To verify the effectiveness of our method, we compare ProCo with three
principal baseline categories: (1) Using LLMs to generate problem-related
documents: GenRead (Yu et al., 2023a). (2) Using search engines to retrieve
problem-related documents: RAG (Khattab et al., 2023). (3) Direct question
answering without external documents: CoT (Kojima et al., 2022), CoVe
(Dhuliawala et al., 2023), and Self-Correct (Kim et al., 2023). We employ all
methods as baselines for open-domain question answering and commonsense
reasoning tasks. For arithmetic reasoning, where external documents are
unnecessary, CoT and Self-Correct serve as baselines. These baseline methods
can be integrated into ProCo. For example, we can use the GenRead (Yu et al.,
2023a) to generate an initial answer for a given question and use ProCo to
progressively correct the initial answer (i.e., GenRead + ProCo). Details of
all baselines are provided in Appendix A.2.

##### Evaluation Metrics.

In the open-domain question answering, we use exact match (EM) score and F1
score to evaluate model performance (Zhu et al., 2021). For the EM score, an
answer is considered correct if and only if its normalized form (Yu et al.,
2023a) has a match in the acceptable answer list. Similar to EM score, the F1
score treats the prediction and ground truth as bags of tokens, and computes
the average overlap between them. For other complex reasoning tasks, we use
accuracy as the evaluation metric.

##### Implementation.

We evaluate ProCo across three LLMs of different scales: GPT-3.5-Turbo-1106
and GPT-4-0125-Preview, which are the most widely used LLMs with public
available APIs111https://platform.openai.com/docs/models. Additionally, we
include Mixtral-8x7B222https://github.com/mistralai/mistral-src (Jiang et al.,
2024), an open source LLM with 47 billion parameters. For baselines that use
external documents, such as GenRead (Yu et al., 2023a) and RAG (Khattab et
al., 2023), we set the number of documents $M$ to $5$. When incorporating
these methods with ProCo, we set the number of document $M$ to $1$, meaning we
generate or retrieve one document and generate an answer based on it. In our
experiments, we set the temperature parameter to $0.7$.

Figure 3: Performance comparison of GenRead and GenRead + ProCo. We can
observe that GenRead + ProCo consistently outperforms GenRead across all
datasets, while GenRead + ProCo consumes much fewer tokens than GenRead. Note:
GenRead indicates generating $5$ problem-related documents and generating an
answer based on these documents. GenRead + ProCo indicates generating one
problem-related document and generating an initial answer based on that
document, and then progressively correcting the initial answer. Figure 4:
Analysis of changes in answers after three rounds of corrections. Correct
$\rightarrow$ Incorrect: A correct answer is changed to an incorrect one.
Incorrect $\rightarrow$ Correct: An incorrect answer is revised to a correct
one. Self-Correct is more likely to modify correct answers to an incorrect
ones, rather than fixing erroneous predictions. ProCo can properly judge the
correctness of the answers, and correct wrong answers.

Method (GPT-3.5-Turbo-1106) Arithmetic Reasoning GSM8K AQuA MATH CoT $78.6$
$51.3$ $37.9$ Self-Correct $75.1$ $48.7$ $27.6$ ProCo $\mathbf{87.1}$
$\mathbf{65.2}$ $\mathbf{41.5}$

Table 4: Accuracy on arithmetic reasoning tasks.

###  5.2 Experimental Results

##### Overall performance on open-domain question answering and commonsense
reasoning tasks.

Table 2 and Table 3 demonstrate that incorporating our proposed ProCo method
with baseline methods significantly enhances the problem-solving performance
across five datasets. This enhancement persists regardless of the backend LLM
utilized. Specifically, when applied to GPT-3.5-Turbo-1106, using the GenRead
method to generate an initial answer and then correcting it using ProCo (i.e.,
GenRead + ProCo), the EM score increases by $+6.1$ on NQ, $+7.6$ on TriviaQA,
$+5.4$ on WebQ, $+9.0$ on HotpotQA, and accuracy improves by $+9.1$ on CSQA.
Compared to the Self-Correct method, ProCo demonstrates improved performance
across NQ, TriviaQA, WebQ, HotpotQA, and CSQA datasets, achieving gains of
$+7.9$, $+7.4$, $+7.8$, $+4.0$, and $+9.6$, respectively. These results
indicate that ProCo significantly enhances the ability of LLMs to solve
complex reasoning tasks without relying on external feedback.

Compared to the competitive few-shot baseline, CoVe, the performance of ProCo
remains impressive. When applied to GPT-3.5-Turbo-1106, ProCo enhances the
average EM score by $+3.2$ across four open-domain question answering datasets
compared to CoVe. This result indicates that iteratively correcting previous
mistakes is a more effective strategy than selecting verified answers from
initially generated answers. Additional experimental results are shown in
Appendix A.4.

##### Overall performance on arithmetic reasoning tasks.

As shown in Table 4, ProCo consistently outperforms the baseline methods
across all arithmetic reasoning tasks. Specifically, ProCo improves accuracy
by an average of $+14.1$ compared to the Self-Correct prompting method.

##### Efficiency and effectiveness of ProCo.

In GenRead, following (Yu et al., 2023a), we prompt an LLM to generate $5$
problem-related documents and incorporate them to produce the final answer. In
contrast, GenRead + ProCo prompts an LLM to generate a single problem-related
document, generates an initial answer based on that document, and
progressively corrects the initial answer. Figure 3 shows that GenRead + ProCo
outperforms GenRead in both EM and F1 scores across various open-domain
question answering datasets. Furthermore, GenRead + ProCo consumes much fewer
tokens than GenRead, indicating its superior conciseness and efficiency in
solving open-domain questions. Upon further examination, we find that the
multiple problem-related documents generated by GenRead often contain
excessive irrelevant or redundant information, which can lead to incorrect
answers. In contrast, GenRead + ProCo generates a single document during the
initial step, thus avoiding the issue of information redundancy. Additionally,
GenRead + ProCo enhances answer accuracy by verifying the correctness of
previously generated answers, identifying mistakes, and correcting them in the
subsequent problem-solving process.

##### Underlying mechanisms of ProCo success.

Figure 4 shows the results of the answer changes after three rounds of
correction using GPT-3.5-Turbo-1106 as the backend LLM. For GSM8K, ProCo
modifies a correct answer to an incorrect one in $2.5\%$ of cases and revises
an incorrect answer to a correct one in $8.2\%$ of cases. Conversely, Self-
Correct modifies a correct answer to an incorrect one in $9.1\%$ of cases and
revises an incorrect answer to a correct one in $7.6\%$ of cases. These
findings indicate that ProCo is more likely to revise an incorrect answer to a
correct one than to modify a correct answer to an incorrect one, thereby
enhancing the performance of LLMs in arithmetic reasoning tasks. Additionally,
we compare the accuracy of different prompting methods in verifying the
correctness of LLM-generated answers. ProCo proved to be more accurate in
verifying the correctness of generated answers than Self-Correct.

###  5.3 Ablation Studies

Figure 5: EM score at different number of iterations. Figure 6: Performance
comparison of CoT, ProCo, and CoT with self-consistency (i.e., CoT + SC).
Compared to CoT + SC, ProCo not only exhibits higher accuracy, but also
consumes fewer tokens.

Method (GPT-3.5-Turbo-1106) Dataset TriviaQA CSQA Equivalent $\mathbf{82.4}$
$\mathbf{93.7}$ Match $40.2$ $29.7$ Similarity $69.2$ $65.9$

Table 5: Accuracy comparison of different answer verification methods.
“Equivalent” determines the correctness of an LLM-generated answer by checking
whether the answer to the verification question and the key condition are
equivalent. “Match” determines the correctness of an LLM-generated answer by
checking whether the answer to the verification question exactly matches the
key condition. “Similarity” indicates that the correctness of an LLM-generated
answer is determined by evaluating the semantic similarity between the answer
to the verification question and the key condition.

##### Significance of the iterative process.

Figure 5 shows that the EM score of ProCo improves with an increasing number
of iterations. In contrast, the Self-Correct shows minimal improvement, and
sometimes even a decrease in the EM score with additional iterations. Notably,
for the WebQ dataset, ProCo achieves a significant EM score increase of $8.8$
after three iterations, whereas the Self-Correct method attains only $1.0$ EM
score increase.

##### Can we just use the exact match method during the verification phase?

In ProCo, we consider an LLM-generated answer to be correct as the answer to
the verification question and the key condition are equivalent; we denote this
method as “Equivalent”. To evaluate our answer verification approach, we also
consider: (1) “Match”, where an LLM-generated answer is deemed correct if it
exactly matches the key condition, and (2) “Similarity”, where an LLM-
generated answer is considered correct if it is semantically similar to the
key condition. As shown in Table 5, verifying whether the answer to the
verification question is equivalent to the key condition can accurately assess
the correctness of LLM-generated answers.

##### Can we just generate multiple outputs without correction?

Self-consistency (SC) (Wang et al., 2023c) involves solving a problem $N$
times and using a majority vote strategy to determine the most consistent
answer as the final answer. We evaluate the performance of CoT with self-
consistency (i.e., CoT + SC) on complex reasoning datasets. For fair
comparison, we set $N$ to $3$. As shown in Figure 6, ProCo surpasses CoT + SC
in accuracy while also consuming fewer tokens. The enhanced performance of
ProCo can be attributed to its ability to progressively identify and correct
potentially incorrect answers. In contrast, CoT + SC merely solves the problem
multiple times; this repeated independent process may lead to the same
mistakes, rendering the frequently answer still incorrect.

##  6 Conclusion

In this study, we present a novel zero-shot prompting method for solving
complex reasoning tasks. We name it progressive correction (ProCo), which
first prompts an LLM to generate an initial response, then iterates a verify-
then-correct process to progressively identify and correct (probably) false
responses. Extensive experiments on eight complex reasoning datasets
demonstrate the effectiveness and efficiency of our proposed method.

## References

  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington. Association for Computational Linguistics.
  * Chen et al. (2024) Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024.  Teaching large language models to self-debug.  In _The Twelfth International Conference on Learning Representations_.
  * Cobbe et al. (2021a) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021a.  Training verifiers to solve math word problems.
  * Cobbe et al. (2021b) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021b.  Training verifiers to solve math word problems.  _CoRR_ , abs/2110.14168.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.  Chain-of-verification reduces hallucination in large language models.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gou et al. (2024) Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024.  CRITIC: Large language models can self-correct with tool-interactive critiquing.  In _The Twelfth International Conference on Learning Representations_.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Towards reasoning in large language models: A survey.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2024) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024.  Large language models cannot self-correct reasoning yet.  In _The Twelfth International Conference on Learning Representations_.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Khattab et al. (2023) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2023.  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp.
  * Kim et al. (2023) Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023.  Language models can solve computer tasks.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 39648–39677. Curran Associates.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Le et al. (2022) Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. 2022.  CodeRL: Mastering code generation through pretrained models and deep reinforcement learning.  In _Advances in Neural Information Processing Systems_.
  * Li et al. (2023) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023.  Making language models better reasoners with step-aware verifier.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5315–5333, Toronto, Canada. Association for Computational Linguistics.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 158–167, Vancouver, Canada. Association for Computational Linguistics.
  * Liu et al. (2022) Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022.  Rainier: Reinforced knowledge introspector for commonsense question answering.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8938–8958, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 46534–46594. Curran Associates, Inc.
  * Miao et al. (2024) Ning Miao, Yee Whye Teh, and Tom Rainforth. 2024.  Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning.  In _The Twelfth International Conference on Learning Representations_.
  * Peng et al. (2023) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023.  Check your facts and try again: Improving large language models with external knowledge and automated feedback.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tandon et al. (2021) Niket Tandon, Aman Madaan, Peter Clark, Keisuke Sakaguchi, and Yiming Yang. 2021.  Interscript: A dataset for interactive learning of scripts through error feedback.
  * Wang et al. (2023a) Jinyuan Wang, Junlong Li, and Hai Zhao. 2023a.  Self-prompted chain-of-thought on large language models for open-domain multi-hop reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 2717–2731, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023b) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2609–2634, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2023c) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023c.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 24824–24837. Curran Associates.
  * Welleck et al. (2023) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023.  Generating sequences by learning to self-correct.  In _The Eleventh International Conference on Learning Representations_.
  * Wu et al. (2024) Zhenyu Wu, Chao Shen, and Meng Jiang. 2024.  Instructing large language models to identify and ignore irrelevant conditions.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yu et al. (2023a) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023a.  Generate rather than retrieve: Large language models are strong context generators.  In _The Eleventh International Conference on Learning Representations_.
  * Yu et al. (2023b) Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. 2023b.  Improving language models via plug-and-play retrieval feedback.
  * Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023.  Automatic chain of thought prompting in large language models.  In _The Eleventh International Conference on Learning Representations_.
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.

##  Appendix A Appendix

###  A.1 Datasets

We evaluate ProCo on three complex reasoning tasks: arithmetic reasoning
(GSM8K (Cobbe et al., 2021b), AQuA (Ling et al., 2017), and MATH (Hendrycks et
al., 2021)); open-domain question answering (NQ (Kwiatkowski et al., 2019),
TriviaQA (Joshi et al., 2017), WebQ (Berant et al., 2013), and HotpotQA (Yang
et al., 2018)); and commonsense reasoning (CSQA (Talmor et al., 2019)). All of
these datasets are accessible under the MIT License. Below, we provide brief
descriptions of the datasets used:

  * •

GSM8K (Cobbe et al., 2021b) consists of high quality grade school math word
problems created by human problem writers. These problems require $2$ to $8$
steps to solve, and solutions primarily involve performing a sequence of
elementary calculations using basic arithmetic operations to reach the final
answer.

  * •

AQuA (Ling et al., 2017) contains multiple-choice math questions that cover a
broad range of topics and difficulty levels.

  * •

MATH (Hendrycks et al., 2021) is a challenging datasets consisting of 12k
problems across seven categories, testing models’ advanced math and science
reasoning. The problems in this dataset are very hard as they come from
mathematics competitions written in LaTeX.

  * •

NQ (Kwiatkowski et al., 2019) were collected from real Google search queries
and the answers are one or multiple spans in Wikipedia articles identified by
human annotators.

  * •

TriviaQA (Joshi et al., 2017) includes trivia questions with answers
originally scraped from trivia and quiz-league websites.

  * •

WebQ (Berant et al., 2013) consists of questions selected using Google Suggest
API, where the answers are entities in Freebase.

  * •

HotpotQA (Yang et al., 2018) contains 113k multi-hop questions in natural
language. The questions are collected by crowdsourcing based on Wikipedia
articles with human annotated supporting evidence and answers.

  * •

CSQA (Talmor et al., 2019) offers a collection of multiple-choice questions
testing commonsense reasoning. We use the development set for our evaluation.

###  A.2 Baselines

To verify the effectiveness of our method, we compare ProCo with three
principal baseline categories: (1) Using LLMs to generate problem-related
documents: GenRead (Yu et al., 2023a) first prompts an LLM to generate $M$
contextual documents based on a given question and then reads these documents
to produce the final answer. (2) Using search engines to retrieve problem-
related documents: RAG (Khattab et al., 2023) first retrieves $M$ relevant
documents from Bing search333https://www.microsoft.com/en-us/bing/apis/ based
on a given question and then prompts an LLM to read the retrieved documents to
produce the final answer. (3) Direct question answering without external
documents: CoT (Kojima et al., 2022) appends “ _Let’s think step by step_ ” to
the given question, instructing the LLM to generate a reasoning path leading
to the final answer. CoVe (Dhuliawala et al., 2023) first answers the given
question, generates a list of verification questions based on the initial
answer, answers each of these verification questions, and finally produces the
final answer based on the verification results. Self-Correct (Kim et al.,
2023) instructs an LLM to critique and refine its initial response. We use all
methods as baselines for open-domain question answering and commonsense
reasoning tasks. For arithmetic reasoning, where external documents are
unnecessary, CoT and Self-Correct serve as baselines. These baseline methods
can be integrated into ProCo. For example, we can use the GenRead (Yu et al.,
2023a) method to generate an initial answer for a given question and use our
proposed ProCo method to progressively correct the initial answer (i.e.,
GenRead + ProCo).

###  A.3 Full Prompts in Experiments

####  A.3.1 Arithmetic Reasoning

Given an arithmetic question $Q$, we use the CoT prompting method to generate
an initial answer. Specifically, we first construct a reasoning generation
prompt: “Q: $Q$. A: Let’s think step by step.” as shown in Prompt
subsubsection A.3.1. We then feed the above prompt to the LLM, which
subsequently generates a reasoning path. To extract the answer from the
reasoning path, we append an answer extraction instruction, creating the
numerical answer extraction prompt: “Q: $Q$. A: {reasoning path} The answer
(arabic numerals) is:” as shown in Prompt subsubsection A.3.1.

Prompt A.1: Initial Answer Generation Q: $Q$ A: Let’s think step by step.

Prompt A.2: Numerical Answer Extraction Q: $Q$ A: {reasoning path} The answer
(arabic numerals) is:

We use the substitute verification method to verify the correctness of the
previous generated answer. Specifically, we first identify the key condition
within the question (Sec. 3). By replacing the key condition with a specific
token X, we create a masked question. We then append the sentence, “Suppose
the answer is {previous generated answer}. What is the value of unknown
variable X?” to the masked question to formulate the verification question, as
shown in Prompt subsubsection A.3.1.

Prompt A.3: Verification Question Construction {masked question} Suppose the
answer is {previous generated answer}. What is the value of unknown variable
X?

Using Prompt subsubsection A.3.1 and Prompt subsubsection A.3.1, we can obtain
the numerical answer for the verification question. By checking if the
numerical answer for the verification question is equal to the key condition,
we can assess the correctness of the previous generated answer. If the
previous generated answer is deemed incorrect, we add it to the set of
potentially incorrect answers; otherwise, we select it as the final answer.
For incorrect answers, we can use the Prompt subsubsection A.3.1 to correct
them.

Prompt A.4: Incorrect Answers Correction Q: $Q$ (the answer is likely not in
{set of potentially incorrect answers}) A: Let’s think step by step.

####  A.3.2 Open-domain Question Answering

Given an open-domain question $Q$, we use the Prompt subsubsection A.3.1 to
instruct the LLM to generate a reasoning path. To extract the answer from this
reasoning path, we add an answer extraction instruction, resulting in the
following entity answer extraction prompt: “Answer the following question with
just one entity. Q: $Q$. A: {reasoning path} The answer is:” as shown in
Prompt subsubsection A.3.2.

Prompt A.5: Initial Answer Generation Answer the following question with just
one entity. Q: $Q$ A: {reasoning path} The answer is:

We use the substitute verification method to verify the correctness of the
previous generated answer. Specifically, we first use the Prompt subsubsection
A.3.2 to identify the key condition within the question. By replacing the key
condition with a specific token X, we create a masked question. We then append
the sentence, “Suppose the answer is {previous generated answer}. What is the
value of unknown variable X?” to the masked question to formulate the
verification question, as shown in Prompt subsubsection A.3.1.

Prompt A.6: Key Condition Identification Given the question below, the task is
to identify a set of entities within the question and then select the one that
is most relevant to the problem-solving process. $Q$

Using Prompt subsubsection A.3.1 and Prompt subsubsection A.3.2, we can obtain
the answer for the verification question. By checking if the answer for the
verification question and the key condition are equivalent, we can assess the
correctness of the previous generated answer.

Prompt A.7: Equivalence Check Determine the correctness of the proposition: If
the answer to question {verification question} is {key condition}, then X
could also be {answer for the verification question}

If the previous generated answer is deemed incorrect, we add it to the set of
potentially incorrect answers; otherwise, we select it as the final answer.
For incorrect answers, we can use the Prompt subsubsection A.3.1 to correct
them.

Method GSM8K CSQA HotpotQA CoT $95.5$ $82.0$ $49.0$ Self-Correct $91.5$ $79.5$
$49.0$ ProCo $\mathbf{97.6}$ $\mathbf{86.7}$ $\mathbf{61.0}$

Table 6: Results of GPT-4-0125-Preview on reasoning benchmarks with different
prompting methods.

###  A.4 Additional Experimental Results

##### How does LLM selection affect ProCo?

Table 6 demonstrates that ProCo performs better with more advanced large
language models. Specifically, on the GSM8K dataset, the GPT-4-0125-Preview
model exhibits a $+10.5$ increase in accuracy compared to the
GPT-3.5-Turbo-1106 model. Likewise, on the CSQA dataset, the
GPT-4-0125-Preview model achieves a $+14.0$ improvement in accuracy over the
GPT-3.5-Turbo-1106 model.

Question |  Where does patience is a virtue come from?
---|---
Gold Answer |  Psychomachia
CoT |  The phrase “patience is a virtue” is an old proverb that has been passed down through generations. Its exact origin is not known, but it is believed to have originated from ancient wisdom and moral teachings. ✗
Self-Correct |  The phrase “patience is a virtue” comes from the English poet William Langland’s work Piers Plowman in the 14th century. ✗
CoVe |  “Patience is a virtue” is a well-known proverb that has been in use since at least the 14th century. It is often attributed to the Latin writer Publilius Syrus. ✗
GenRead |  Poem Piers Plowman written by William Langland in the 14th century. ✗
RAG |  The first known recording of the expression in English is in the poem Piers Plowman by William Langland, written around the year 1360. Langland’s poem includes the line “Patience is a high virtue”. (https://www.bookbrowse.com/expressions/detail/index.cfm/expression_number/416/patience-is-a-virtue) ✗
RAG + ProCo |  The idea behind the expression “patience is a virtue” dates back to the 5th century. It can be found in the epic poem Psychomachia. (https://writingexplained.org/idiom-dictionary/patience-is-a-virtue) ✓

Table 7: Case study of answers generated by different methods. Final answer is
highlighted with yellow color.

##### Can we just use the exact match method during the verification phase?

Since verification questions can have multiple valid answers, directly
checking if the LLM-generated response exactly matches the key condition might
misclassify correct answers as incorrect. Consider the following example:
Given an open-domain question _“Who wrote the treasure of the sierra madre?”_
, we first prompt an LLM to generate an initial answer, e.g., _“B. Traven”_.
Next, we identify a key condition in the question relevant to the problem-
solving process, such as _“the treasure of the sierra madre”_. By masking the
key condition, we create a verification question: _“Who wrote X? Suppose the
answer is B. Traven. What is the value of unknown variable X?”_. Using the LLM
to solve the verification question, we receive the response _“The Death
Ship”_. If we directly check whether _“The Death Ship”_ matches _“the treasure
of the sierra madre”_ , we find they do not match, leading us to incorrectly
judge the answer _“B. Traven”_ as wrong. However, all books written by B.
Traven are correct answers to the verification question. Thus, exact matching
is insufficient for verification. Based on this observation, we propose
proposition-based verification. Specifically, we construct an answer
verification prompt: _“Determine the correctness of the proposition: If the
answer to question “Who wrote X? Suppose the answer is B. Traven. What is the
value of unknown variable X?” is “the treasure of the sierra madre”, then X
could also be “The Death Ship””_. We input this prompt into an LLM and receive
a judgement about the proposition’s correctness, e.g., _“The proposition is
correct, since both works were written by the same author.”_. This approach
allows the LLM to properly analyze whether _“The Death Ship”_ and _“the
treasure of the sierra madre”_ are both correct answers for the verification
question, thus accurately determining the correctness of LLM-generated
answers.

###  A.5 Limitations

The scope of this study was limited to solve complex reasoning tasks in
English; tasks in non-English languages are not part of our training or test
data. As a result, the method might not perform satisfactorily for non-English
tasks. Further investigation into solving multilingual complex reasoning tasks
is left for future work.

###  A.6 Case Study

Table 7 shows that, with the exception of RAG + ProCo, all other methods fail
to provide the correct answer to the given problem. CoT generates an incorrect
answer, unable to determine the origin of the phrase “Patience is a virtue”.
Self-Correct, CoVe, GenRead, and RAG erroneously assert that the phrase
“Patience is a virtue” originated in the 14th century. In contrast, RAG +
ProCo accurately identifies the first appearance of the phrase “Patience is a
virtue” in the 5th century, and correctly cites the source.

###  A.7 Sample Predictions for Complex Reasoning Datasets

Question |  Jean is two years older than Mark. Two years ago Mark was 5 years older than half Jan’s age. If Jan is 30 how old is Jean?
---|---
Gold Answer |  $\frac{30-2}{2}+5+2+2=23$
Key Condition |  $5$
Initial Answer |  Two years ago, Mark was 5 years older than half of Jan’s age. Half of Jan’s age two years ago would be 10 (30/2 - 2). Therefore, Mark’s age two years ago would be 15 (10 + 5). We know that Jean is two years older than Mark. Therefore, Jean’s age two years ago would be 17 (15 + 2). Jean’s current age would be 19 (17 + 2). ✗
Verification Question |  Jean is two years older than Mark. Two years ago Mark was X years older than half Jan’s age. If Jan is 30 how old is Jean? Suppose the answer is $\mathbf{19}$. What is the value of unknown variable X?
Answer to the Verification Question |  Two years ago Mark’s age = Jean’s age – 2 Half of Jan’s age = 15 Mark’s age two years ago = Jean’s age – 2 = 15 – X Therefore, X = Jean’s age – 15 – 2 = 19 – 15 – 2 = 2. Answer: X = 2.
Verification Result |  Since $2$ is not equal to the key condition, the answer $19$ is likely incorrect.
Question (with Hint) |  Jean is two years older than Mark. Two years ago Mark was 5 years older than half Jan’s age. If Jan is 30 how old is Jean? (The answer is likely not 19)
Corrected Answer |  Two years ago, Mark was 5 years older than half of Jan’s age. Jan was 28. Half of Jan’s age two years ago would be 14. Mark was 5 years older than 14 two years ago, so he was 19 two years ago. Jean is two years older than Mark, so Jean was 21 two years ago. Now, two years have passed since then, so Jean is 23 today. ✓
Verification Question |  Jean is two years older than Mark. Two years ago Mark was X years older than half Jan’s age. If Jan is 30 how old is Jean? Suppose the answer is $\mathbf{23}$. What is the value of unknown variable X?
Answer to the Verification Question |  First, we’re given that Jean is two years older than Mark. So if Mark is $M$ years old, Jean is $M+2$ years old. Two years ago, Mark was $M-2$ years old and Jan’s age was $30-2=28$ years. We’re also given that two years ago Mark was $X$ years older than half of Jan’s age. So, $M-2=X+\frac{1}{2}(28)$. Now, let’s solve for $M$ using the information given. $M-2=X+14$ $M=X+16$ Now we know Mark’s age in terms of $X$. Since Jean is two years older than Mark, Jean’s age is $X+18$. If the answer is 23 for Jean’s age, then: $X+18=23$ $X=5$ So, the value of the unknown variable X is 5.
Verification Result |  Since $5$ is equal to the key condition, the answer $23$ is likely correct.
Final Answer |  $23$ ✓

Table 8: Example output obtained by ProCo on the GSM8K dataset.

Question |  In a class of 50 students, 28 participate in MATHCOUNTS, 21 participate in science club, and 6 students participate in neither. How many students participate in both MATHCOUNTS and science club?
---|---
Gold Answer |  $5$
Key Condition |  $28$
Initial Answer |  Step 1: Find the total number of students who participate in either MATHCOUNTS or science club. Total students participating = 28 + 21 - 6 Total students participating = 43 Step 2: Calculate the number of students who participate in both MATHCOUNTS and science club. Number of students in both = 28 + 21 - 43 Number of students in both = 6 So, $\boxed{6}$ students participate in both MATHCOUNTS and science club. ✗
Verification Question |  First, we can calculate the number of students participating in either MATHCOUNTS or the science club or both: $50-6=44$. Next, we use the principle of inclusion and exclusion to set up our equation. The principle of inclusion and exclusion for two sets states: $|A\cup B|=|A|+|B|-|A\cap B|$ Here, $|A\cup B|$ represents the total number of students participating in either MATHCOUNTS or science club or both, which we have found to be $44$. $|A|$ represents the number of students in MATHCOUNTS ($X$), $|B|$ represents the number of students in science club ($21$), and $|A\cap B|$ represents the number of students in both clubs ($6$). Substituting these values into the equation, we get: $44=X+21-6$. Thus, the value of $X$ is $\boxed{29}$.
Verification Result |  Since $29$ is not equal to the key condition, the answer $6$ is likely incorrect.
Question (with Hint) |  In a class of 50 students, 28 participate in MATHCOUNTS, 21 participate in science club, and 6 students participate in neither. How many students participate in both MATHCOUNTS and science club? (The answer is likely not 6)
Corrected Answer |  \- Let $M$ represent the set of students participating in MATHCOUNTS. \- Let $S$ represent the set of students participating in the science club. \- Let $n$ represent the total number of students in the class, which is 50. First, we calculate the number of students participating in either MATHCOUNTS or the science club or both: $|M\cup S|=n-|N|=50-6=44$ Using the principle of inclusion-exclusion for the union of two sets: $|M\cup S|=|M|+|S|-|M\cap S|$ Substitute the known values into the equation: $44=28+21-|M\cap S|$ $|M\cap S|=5$ Thus, the number of students who participate in both MATHCOUNTS and the science club is: $None$ ✓

Table 9: Example output obtained by ProCo on the MATH dataset.

Question |  A man spends Rs. 3500 per month and saves 12 1/2% of his income. His monthly income is ? Answer Choices: (A) Rs. 4400 (B) Rs. 4270 (C) Rs. 4000 (D) Rs. 3937.50 (E) None of these
---|---
Gold Answer |  C
Key Condition |  $3500$
Initial Answer |  First, we need to calculate the amount saved each month. To do this, we can calculate 12 1/2% of Rs. 3500. 12 1/2% of Rs. 3500 = (12.5/100) * 3500 = 0.125 * 3500 = Rs. 437.50 So, the man saves Rs. 437.50 each month. To find his monthly income, we can use the formula: Income = Expenditure + Savings Income = Rs. 3500 + Rs. 437.50 Income = Rs. 3937.50 Therefore, his monthly income is Rs. 3937.50, which corresponds to answer choice (D). ✗
Verification Question |  A man spends Rs. X per month and saves 12 1/2% of his income. His monthly income is ? Answer Choices: (A) Rs. 4400 (B) Rs. 4270 (C) Rs. 4000 (D) Rs. 3937.50 (E) None of these Suppose the answer is (D) Rs. 3937.50. What is the value of unknown variable X?
Answer to the Verification Question |  1\. Convert the percentage saved into a decimal: 12.5% can be expressed as: $12.5\%=\frac{12.5}{100}=0.125$ 2\. Determine the amount saved: Since he saves 12.5% of his income, the amount saved each month is: $0.125\times 3937.50=492.1875$ 3\. Calculate the amount spent (X): The amount he spends is the remaining part of his income after saving. Therefore, the amount spent $X$ is: $X=\text{Income}-\text{Amount saved}=3445.3125$ The value of the unknown variable $X$ (the amount spent per month) is: $\boxed{3445.3125}$
Verification Result |  Since $3445.3125$ is not equal to the key condition, the answer (D) Rs. 3937.50 is likely incorrect.
Question (with Hint) |  A man spends Rs. 3500 per month and saves 12 1/2% of his income. His monthly income is ? Answer Choices: (A) Rs. 4400 (B) Rs. 4270 (C) Rs. 4000 (D) Rs. 3937.50 (E) None of these (The answer is likely not (D) Rs. 3937.50)
Corrected Answer |  Let $I$ be the man’s monthly income. Since he saves 12.5% of his income, he spends the remaining percentage of his income. The percentage of income spent is: $100\%-12.5\%=87.5\%$ $\text{Income spent}=0.875\times I$ We know that the amount spent is Rs. 3500. Thus, we set up the equation: $0.875\times I=3500$ To find $I$, we solve the equation for $I$: $I=\frac{3500}{0.875}=4000$ The man’s monthly income is Rs. 4000. The correct answer is: (C) Rs. 4000 ✓

Table 10: Example output obtained by ProCo on the AQuA dataset.

Question |  Right to property according to the constitution of india is a?
---|---
Gold Answer |  constitutional right
Key Condition |  property
Initial Answer |  The right to property in the Constitution of India has undergone significant changes over the years. Prior to the 44th Amendment Act in 1978, the right to property was considered a fundamental right under Part III of the Constitution. However, this amendment removed the right to property from the list of Fundamental Rights and converted it into a simple legal right (https://byjus.com/question-answer/consider-the-following-statements-regarding-right-to-property-1-right-to-property-is-legal-right/) ✗
Verification Question |  Right to X according to the constitution of india is a? Suppose the answer is simple legal right. What is the value of unknown variable X?
Answer to the Verification Question |  the right to legal representation.
Verification Result |  Since the right to legal representation and the property are not equivalent, the answer simple legal right is likely incorrect.
Question (with Hint) |  Right to property according to the constitution of india is a? (The answer is likely not simple legal right)
Corrected Answer |  The right to property in the Constitution of India has evolved from being a fundamental right to a legal and constitutional right, with provisions for legal protection and acquisition for public welfare. (https://testbook.com/question-answer/what-is-the-position-of-the-right-to-property-in-i–616294b1fb5e57239764c589) ✓
Verification Question |  Right to X according to the constitution of india is a? Suppose the answer is constitutional right. What is the value of unknown variable X?
Answer to the Verification Question |  In the context of the Constitution of India, the unknown variable X refers to a specific right that is classified as a "constitutional right." Given this information, one possible value for X is "property." To elaborate, the right to property was originally a fundamental right under Article 31 of the Indian Constitution. However, it was later amended by the 44th Amendment Act of 1978, and the right to property was removed from the list of fundamental rights and reclassified as a constitutional right under Article 300A. Therefore, the value of the unknown variable X is "property."
Verification Result |  Since the property and the property are equivalent, the answer constitutional right is likely correct.
Final Answer |  constitutional right ✓

Table 11: Example output obtained by RAG + ProCo on the NQ dataset.

Question |  Who was the defending champion when Martina Navratilova first won Wimbledon singles?
---|---
Gold Answer |  Virginia Wade
Key Condition |  Martina Navratilova
Initial Answer |  When Martina Navratilova first won the Wimbledon singles title, the defending champion was Chris Evert (https://en.wikipedia.org/wiki/1977_Wimbledon_Championships). ✗
Verification Question |  Who was the defending champion when X first won Wimbledon singles? Suppose the answer is Chris Evert. What is the value of unknown variable X?
Answer to the Verification Question |  Given that Chris Evert is the answer, X must be the name of a person who won Wimbledon singles. Conclusion: X refers to a person who won Wimbledon singles, but without more information, the specific name cannot be determined.
Verification Result |  Since unknown and Martina Navratilova are not equivalent, the answer Chris Evert is likely incorrect.
Question (with Hint) |  Who was the defending champion when Martina Navratilova first won Wimbledon singles? (The answer is likely not Chris Evert)
Corrected Answer |  The defending champion when Martina Navratilova first won the Wimbledon singles title in 1978 was Virginia Wade (https://en.wikipedia.org/wiki/1978_Wimbledon_Championships_–_Women’s_singles). ✓
Verification Question |  Who was the defending champion when X first won Wimbledon singles? Suppose the answer is Virginia Wade. What is the value of unknown variable X?
Answer to the Verification Question |  Martina Navratilova won her first Wimbledon singles title in 1978. Virginia Wade was the Wimbledon singles champion in 1977. So, if we are to match the scenario where X first won Wimbledon when Virginia Wade was the defending champion: In 1978, the defending champion was Virginia Wade (from her 1977 win). Martina Navratilova won Wimbledon singles in 1978. Therefore, ( X ) is indeed Martina Navratilova. Thus, the value of the unknown variable ( X ) is Martina Navratilova.
Verification Result |  Since Martina Navratilova and Martina Navratilova are equivalent, the answer Virginia Wade is likely correct.
Final Answer |  Virginia Wade ✓

Table 12: Example output obtained by RAG + ProCo on the TriviaQA dataset.

Question |  What highschool did harper lee go to?
---|---
Gold Answer |  Monroe County High School
Key Condition |  highschool
Initial Answer |  Harper Lee attended several educational institutions during her academic journey. According to the information available, she attended Huntingdon College from 1944 to 1945. After that, she studied law at the University of Alabama from 1945 to 1949. Additionally, she spent a year at Oxford University (https://www.tokillamockingbird.com/harper-lee). Based on the available information, it can be concluded that Harper Lee did not attend high school. She pursued higher education after completing her grade school education. Conclusion Harper Lee did not attend high school. ✗
Verification Question |  What X did harper lee go to? Suppose the answer is Harper Lee did not attend high school. What is the value of unknown variable X?
Answer to the Verification Question |  If the statement "Harper Lee did not attend high school" is true, then the value of the unknown variable X, representing the high school she attended, would be "none." This means Harper Lee did not go to any high school.
Verification Result |  Since none and highschool are not equivalent, the answer Harper Lee did not attend high school is likely incorrect.
Question (with Hint) |  What highschool did harper lee go to? (The answer is likely not Harper Lee did not attend high school)
Corrected Answer |  Nelle Harper Lee, the acclaimed author of "To Kill a Mockingbird," attended monroe county high school in her hometown of Monroeville, Alabama. She was born on April 28, 1926, and grew up in a rural southern Alabama town, where she attended both elementary school and high school. Her father, Amasa Coleman Lee, was an attorney who served in the state legislature in Alabama, and her childhood experiences were similar to those of her young narrator in "To Kill a Mockingbird," Scout Finch (https://www.chipublib.org/harper-lee-biography/) ✓
Verification Question |  What X did harper lee go to? Suppose the answer is monroe county high school. What is the value of unknown variable X?
Answer to the Verification Question |  If the answer is "Monroe County High School," the value of the unknown variable X is "high school."
Verification Result |  Since high school and highschool are equivalent, the answer monroe county high school is likely correct.
Final Answer |  monroe county high school ✓

Table 13: Example output obtained by RAG + ProCo on the WebQ dataset.

Question |  What could go on top of wood? Answer Choices: (A) lumberyard, (B) synagogue, (C) floor, (D) carpet, (E) hardware store
---|---
Gold Answer |  D
Key Condition |  wood
Initial Answer |  To determine what could go on top of wood, let’s analyze the answer choices provided: (A) Lumberyard: A lumberyard is a facility where wood and lumber are stored and sold. It is not something that goes on top of wood. (B) Synagogue: A synagogue is a place of worship for Jewish people. It is not something that goes on top of wood. (C) Floor: Yes, a floor can go on top of wood. Wood is commonly used as a material for flooring, providing a natural and warm aesthetic. Hardwood floors, for example, are made of solid wood and are installed on top of a subfloor. (D) Carpet: Yes, carpet can go on top of wood. Carpet is often installed on top of a wooden subfloor, providing a soft and comfortable surface. (E) Hardware store: A hardware store is a retail store that sells tools, building materials, and other hardware items. It is not something that goes on top of wood. Therefore, the correct answer is (C) floor. ✗
Verification Question |  What could go on top of X? Suppose the answer is floor. What is the value of unknown variable X?
Answer to the Verification Question |  The question asks what could go on top of X, and the answer is "floor." Based on this, we can infer that X refers to a "ceiling." The ceiling is the upper surface of a room and is what goes on top of the floor. Therefore, X refers to ceiling.
Verification Result |  Since wood and ceiling are not equivalent, the answer (C) floor is likely incorrect.
Question (with Hint) |  What could go on top of wood? Answer Choices: (A) lumberyard, (B) synagogue, (C) floor, (D) carpet, (E) hardware store (The answer is likely not (C) floor)
Corrected Answer |  The answer is likely not in [’C’]. Based on the provided search results, the item that could go on top of wood is (D) carpet. ✓
Verification Question |  What could go on top of X? Suppose the answer is carpet. What is the value of unknown variable X?
Answer to the Verification Question |  If the answer to "What could go on top of X?" is "carpet," then X is likely the floor or flooring. Therefore, the value of the unknown variable X is floor.
Verification Result |  Yes, the proposition is correct. If the answer to the question "What could go on top of X? Suppose the answer is carpet" is "wood", then X could also be "floor". Here’s the reasoning: 1\. The question asks what could go on top of X, and the given answer is "carpet". 2\. This implies that "carpet" can be placed on top of X. 3\. The follow-up statement tells us that the answer to this is "wood". Therefore, wood is X. 4\. Carpet is typically placed on top of a floor surface. 5\. Wood is a common type of flooring. Thus, if wood is the answer to X, and considering the context of flooring, X being "floor" is a reasonable interpretation because a carpet is often placed on a wood floor. Therefore, the unknown variable X, which in this context is the surface on which carpet can be placed, could be "floor". Since floor and wood are equivalent, the answer (D) carpet is likely correct.
Final Answer |  D ✓

Table 14: Example output obtained by ProCo on the CSQA dataset.

Question |  The trophy given to the winner of the college football game between University of Idaho Vandals and University of Montana Grizzlies is derived from which European culture?
---|---
Gold Answer |  German
Key Condition |  European culture
Initial Answer |  The University of Idaho Vandals and the University of Montana Grizzlies are both located in the United States. However, the trophy’s origin may be derived from a European culture due to the historical influence of European traditions on American sports and collegiate traditions. ✗
Verification Question |  The trophy given to the winner of the college football game between University of Idaho Vandals and University of Montana Grizzlies is derived from which X ? Suppose the answer is European culture. What is the value of unknown variable X?
Answer to the Verification Question |  Determine which aspect of European culture matches the question about the trophy. Given that the trophy is derived from European culture, X must be a specific aspect or tradition within European culture. X is "Medieval armor and weaponry".
Verification Result |  Since Medieval armor and weaponry and European culture are not equivalent, the answer European culture is likely incorrect.
Question (with Hint) |  The trophy given to the winner of the college football game between University of Idaho Vandals and University of Montana Grizzlies is derived from which European culture? (The answer is likely not European culture)
Corrected Answer |  First, it’s important to note that the University of Idaho Vandals and the University of Montana Grizzlies are both located in the United States, so the trophy is likely derived from American culture rather than European culture. The trophy for this game is called the Little Brown Stein, and it is a tradition that dates back to 1938. The name "Stein" suggests a German influence, as "stein" is a German word for "stone" or "mug." Therefore, the trophy is likely derived from German culture. ✓
Verification Question |  The trophy given to the winner of the college football game between University of Idaho Vandals and University of Montana Grizzlies is derived from which X ? Suppose the answer is German culture. What is the value of unknown variable X?
Answer to the Verification Question |  If the trophy given to the winner of the college football game between the University of Idaho Vandals and the University of Montana Grizzlies is derived from German culture, the unknown variable X could be "German traditions."
Verification Result |  Given the information provided, if the answer to the question is indeed "European culture", then X could indeed be "German traditions". This is because German traditions could fall under the umbrella of European culture. German traditions are a subset of European culture, so if the answer is broadened from "European culture" to a more specific subset like "German traditions", it would still be consistent with the broader category. Therefore, the proposition appears to be correct. Since European culture and German traditions are equivalent, the answer German culture is likely correct.
Final Answer |  German culture ✓

Table 15: Example output obtained by ProCo on the HotpotQA dataset.


|  Ekram Hossain (F’15) is a Professor and Associate Head (Graduate Studies)
in the Department of Electrical and Computer Engineering at University of
Manitoba, Canada (http://home.cc.umanitoba.ca/$\sim$hossaina). He is a Member
(Class of 2016) of the College of the Royal Society of Canada, a Fellow of the
Canadian Academy of Engineering, and a Fellow of the Engineering Institute of
Canada. Dr. Hossain’s current research interests include design, analysis, and
optimization of wireless networks with emphasis on beyond 5G cellular
networks. He was elevated to an IEEE Fellow “for contributions to spectrum
management and resource allocation in cognitive and cellular radio networks”.
He received the 2017 IEEE ComSoc TCGCC (Technical Committee on Green
Communications & Computing) Distinguished Technical Achievement Recognition
Award “for outstanding technical leadership and achievement in green wireless
communications and networking”. He was listed as a Clarivate Analytics Highly
Cited Researcher in Computer Science in 2017, 2018, 2019, and 2020. Currently
he serves as the Editor-in-Chief of IEEE Press (2018-2021) and the Director of
Magazines (2020-2021) for the IEEE Communications Society. Previously he
served as the Editor-in-Chief for the IEEE Communications Surveys and
Tutorials (2012–2016).
---|---
|  Hai Jiang (SM’15) received the B.Sc. and M.Sc. degrees in electronics
engineering from Peking University, Beijing, China, in 1995 and 1998,
respectively, and the Ph.D. degree in electrical engineering from the
University of Waterloo, Waterloo, ON, Canada, in 2006. Since 2007, he has been
a Faculty Member with the University of Alberta, Edmonton, AB, Canada, where
he is currently a Professor with the Department of Electrical and Computer
Engineering. His research interests include radio resource management,
cognitive radio networking, mobile edge computing, and cooperative
communications.
---|---