# Critique Ability of Large Language Models

Liangchen Luo† Zi Lin‡ Yinxiao Liu† Lei Shu† Yun Zhu† Jingbo Shang‡ Lei
Meng†11footnotemark: 1
†Google Research ‡UC San Diego Correspondence<EMAIL_ADDRESS>
###### Abstract

Critical thinking is essential for rational decision-making and problem-
solving. This skill hinges on the ability to provide precise and reasoned
critiques and is a hallmark of human intelligence. In the era of large
language models (LLMs), this study explores the ability of LLMs to deliver
accurate critiques across various tasks. We are interested in this topic as a
capable critic model could not only serve as a reliable evaluator, but also as
a source of supervised signals for model tuning. Particularly, if a model can
self-critique, it has the potential for autonomous self-improvement. To
examine this, we introduce a unified evaluation framework for assessing the
critique abilities of LLMs. We develop a benchmark called CriticBench, which
comprises $3$K high-quality natural language queries and corresponding model
responses; and annotate the correctness of these responses. The benchmark
cover tasks such as math problem-solving, code completion, and question
answering. We evaluate multiple LLMs on the collected dataset and our analysis
reveals several noteworthy insights: (1) Critique is generally challenging for
most LLMs, and this capability often emerges only when models are sufficiently
large. (2) In particular, self-critique is especially difficult. Even top-
performing LLMs struggle to achieve satisfactory performance. (3) Models tend
to have lower critique accuracy on problems where they are most uncertain. To
this end, we introduce a simple yet effective baseline named self-check, which
leverages self-critique to improve task performance for various models. We
hope this study serves as an initial exploration into understanding the
critique abilities of LLMs, and aims to inform future research, including the
development of more proficient critic models and the application of critiques
across diverse tasks.

##  1 Introduction

> “Self-criticism is an art not many are qualified to practice.” — Joyce Carol
> Oates

Large language models (LLMs) have demonstrated impressive capacities in a wide
range of tasks (Google et al., 2023; OpenAI, 2023). Consequently, the
evaluation of LLMs has shifted focus from basic sentence coherence to more
advanced capabilities, e.g., knowledge acquisition and logical reasoning
(Hendrycks et al., 2021; BIG-Bench authors, 2023). One capability that is
overlooked in current evaluation frameworks is the ability of critical
thinking, which is an important hallmark of human intelligence that requires
logic, reasoning, and knowledge. This ability ensures that LLMs can provide
precise and reasoned critiques towards model responses. A model with robust
critique ability can identify potential misinformation, errors or context
misalignment in model outputs, thereby showing their specific shortcomings
that can serve as a feedback for improvement. While recent studies have used
LLMs for various forms of critique across diverse applications (Madaan et al.,
2023; Saunders et al., 2022; Shinn et al., 2023), they primarily focus on
advancing the state of the art for specific tasks instead of providing a
comprehensive assessment of critique ability.

To address this gap, we propose a standardized benchmark CriticBench to assess
the critique abilities of LLMs in diverse tasks. We define a model’s critique
ability as “the capacity to identify flaws in model responses to queries”.
Figure 1 provides an example of a flaw in the response to a query, and how it
is identified by a critique. The benchmark consists of query-response-judgment
triplets. During evaluation, we always prompt a model to perform a chain-of-
thought analysis to identify flaws and explain the reason; and then provide a
final judgment on the response’s correctness. Comparing this judgment to
ground-truth labels allows us to explicitly evaluate a model’s critique
accuracy and implicitly assess its analytical process toward an accurate
judgment.

Figure 1:  An example from CriticBench is presented. The query originates from
GSM8K (Cobbe et al., 2021), and the response is generated by PaLM-2-L (Google
et al., 2023). A flaw in the response is highlighted in red. The model shows
low confidence in this query, as evidenced by a certainty score of only
$0.11$. Below the dashed line, a critique is generated by few-shot prompting
PaLM-2-L. It successfully identifies the flaw in the response and makes an
accurate judgment. As the policy model and critic model are the same, this
example also serves as an instance of self-critique.

To construct CriticBench (Section 3), we gather natural language queries from
multiple scientific benchmarks, covering tasks like math problem-solving
(Cobbe et al., 2021), code completion (Chen et al., 2021), and question
answering (Lin et al., 2021). We employ PaLM-2 models (Google et al., 2023) of
various sizes to generate responses, which are then annotated for correctness.
To ensure data quality, a complexity-based selection strategy (Fu et al.,
2023b) is used to identify high-quality responses among the candidates.
Furthermore, to select queries of suitable difficulty, we introduce an
auxiliary metric that quantifies a model’s certainty regarding a query. Such a
metric can help select queries that poses a moderate level of challenge to
models. As a result, we collect $3$K high-quality examples from an initial
pool of $780$K candidates to form the benchmark mixture. This data collection
method is both scalable and generalizable, requiring no extra human
intervention and suitable for a variety of tasks.

Given CriticBench, we can now analyze the critique abilities of LLMs (Section
4). There are specific aspects that particularly interest us. First, critique
inherently involves logic, reasoning, and knowledge, making it a complex
process even for humans. Therefore, it is not clear how well LLMs can emulate
this capability. It is possible that critique ability is yet another emergent
ability, i.e., ability not present in smaller-scale models that are present in
larger-scale models (Jang, 2023). Investigating how critique ability scales
with model size could offer insights into model size selection and whether
fine-tuning is needed for smaller models (Section 4.1). Additionally, self-
critique, i.e., when a model critiques its own outputs, is a format of
critique of particular interest to us, as it is relevant to a model’s
potential for self-improvement (Section 4.2). Finally, we are also interested
in what types of queries pose more challenges for LLMs to critique (Section
4.3).

To investigate these aspects, we evaluate various widely-used LLMs on
CriticBench and reveal several intriguing findings: (1) Critique tasks pose a
considerable challenge for LLMs. Only large-scale models exhibit performance
with a notable difference from a random guess baseline, indicating that the
capacity for critique serves as an emergent indicator of a capable LLM. (2)
Self-critique, i.e., a model critiquing its own output, is particularly
difficult. Even the strongest LLMs struggle to achieve satisfactory
performance. (3) A challenging query is not only difficult for LLMs to
directly answer correctly, but also poses a challenge in assessing an answer’s
correctness to that query.

To this end, we also propose a simple yet effective baseline called self-check
(Section 5). The basic idea is to prompt the model to confirm the accuracy of
their generated answers by self-critique before presenting them. The method
consistently enhances the baseline performance (Wang et al., 2023) on math
word problems across multiple models, achieving an average of 9.55% error
reduction rate, which demonstrates the potential utility of critiques from
LLMs.

Our contributions are three-fold:

  * •

New Benchmark CriticBench is the first benchmark that comprehensively assesses
the critique abilities of LLMs across diverse tasks and scenarios, which fills
a gap in the current LLM evaluation framework by introducing this important
ability.

  * •

New Findings Our findings on CriticBench underscore the nuances and depth of
LLM’s critique abilities (Section 4). These revelations enhance our
understanding of the inherent complexities in LLMs and emphasize the need for
advanced training and evaluation techniques.

  * •

New Capacity The proposed self-check method (Section 5) not only advances the
performance on math word problems over the baseline, but also indicates the
new capacity of critique ability with LLMs, which is a fruitful avenue for
LLM’s self-improvement strategies.

##  2 Definition of Critique Ability

The concept of critique has diverse interpretations and is often applied
informally in everyday contexts. Recent research employs large language models
to offer critiques across multiple applications (Madaan et al., 2023; Paul et
al., 2023; Saunders et al., 2022; Shinn et al., 2023), resulting in varying
formats and requirements for their “critiques”. These studies primarily aim to
enhance performance in specific tasks, neglecting to clarify the meaning of
the term critique. In this paper, we consider the definition of a language
model’s critique ability as

the capacity to identify flaws in model responses to queries.

These flaws can differ depending on the task, ranging from incorrect reasoning
or calculation in mathematical problems to syntax errors in code completion.

When a model self-assesses its own outputs, we term this as self-critique, a
notion that particularly intrigues us. If models can engage in self-critique
and reflection, they can potentially do self-improvement, requiring minimal
human intervention. On the risky side, this autonomy also raises concerns
about reduced human oversight (Bowman et al., 2022). Yet we posit that self-
critique may still remain a challenging capability for large language models,
as a flaw-aware model would logically not produce faulty output in the first
place (Saunders et al., 2022).

##  3 Construction of CriticBench

As discussed in Section 2, prior research employs large language models to
offer critiques, yet requires particular process and formats to meet their
task-specific objectives. Currently, there is no standard or generalizable way
to assess the critique abilities of language models across diverse tasks. This
section proposes CriticBench, a unified, standardized evaluation framework to
tackle this issue. The framework aims to fulfill three criteria:

  * •

Scalability Given the broad range of tasks already established within the
community, and the anticipation of more to emerge, a scalable data collection
method is essential. The method should minimize human annotation efforts and
ideally be fully autonomous.

  * •

Generalizability The framework should be task-agnostic, capable of
generalizing across various tasks and domains.

  * •

Quality We believe quality matters more than quantity. When volume of data is
substantial, we prioritize selecting those that most effectively differentiate
between stronger and weaker models.

The following subsections illustrate the detailed construction process.
Specifically, Section 3.1 presents the initial data generation on three
different tasks, where we get the collection of query-response-judgment
triplets as shown in Figure 1. Section 3.2 then shows how to select data based
on the initial collection to guarantee the quality of responses and queries.

###  3.1 Data Generation

For the tasks of interest, we begin by employing existing scientific datasets
from relevant domains. These datasets are expected to include queries that
large language models, which here we refer to as generators, aim to respond.

To ensure scalability, it is essential to have an automated approach for
assessing the correctness of a model’s responses. Classification tasks
naturally meet this criterion, as model outputs can be automatically compared
to ground-truth labels. Similarly, tasks that involve auto-verifiable answers
also comply; for instance, in code completion tasks with unit tests available,
the validity of the generated code can be confirmed by passing all tests. For
free-form generation tasks such as summarization and translation, assessing
the quality of a response remains non-trivial. However, recent advances in
LLM-based automated evaluation for generation tasks mitigate this issue to
some extent, enabling the assessment without human intervention (Liu et al.,
2023).

While not exhaustive, these already cover a significant range of tasks and
domains. We acknowledge the limitations in some auto-assessment approaches,
especially for generation tasks. Improving the reliability of these automated
evaluation methods, however, is beyond the scope of this paper.

We employ five different sizes of PaLM-2 models (Google et al., 2023) as our
generators. These models are pretrained solely for next-token prediction and
do not undergo supervised fine-tuning or reinforcement learning from human
feedback. For coding-related tasks, apart from the standard PaLM-2 models, we
also employ the specialized PaLM-2-S* variant. The latter is obtained through
continual training of PaLM-2-S on a data mixture enriched with code-heavy
corpus.

Query Collection We extract queries from three datasets: GSM8K (Cobbe et al.,
2021), HumanEval (Chen et al., 2021), and TruthfulQA (Lin et al., 2021),
covering the tasks of math-problem solving, code completion and question
answering. For datasets with distinct training and test splits, we use the
test data; for datasets intended only for evaluation, all examples are used.
Detailed considerations and rationale behind the selection of these datasets
are provided in Appendix B.

Response Generation We sample $k$ responses for each query, with $k=64$ for
GSM8K and TruthfulQA, and $k=100$ for HumanEval. In the case of TruthfulQA, we
employ its multiple-choice variation to facilitate autonomous answer
annotation. After filtering out invalid outputs such as empty ones, we collect
a total of $780$K responses as an initial pool of candidates.

Annotation for Correctness For GSM8K, we assess answer correctness by
comparing its numeric equality to the ground truth, as described by Lewkowycz
et al. (2022). For HumanEval, correctness is determined by the passage of
provided unit tests. For TruthfulQA, we utilize its classification format,
judging correctness based on a match with the ground-truth label.

More details on hyper-parameter settings and prompt templates are available in
Appendix C.

###  3.2 Data Selection

Many existing evaluation benchmarks for large language models suffer from
insufficient differentiability, i.e., both stronger and weaker models yield
similar performance (Fu, 2023). This issue likely arises from the presence of
either overly simple or exceedingly difficult examples in the benchmarks. Such
examples are less valuable for evaluation and can undermine the utility of the
benchmarks when average scores are calculated, leading to indistinguishable
outcomes. To address the issue, we introduce various filtering strategies
aimed at selecting high-quality and differentiable examples.

####  3.2.1 High-Quality Response Selection

Initially, we can narrow the example set from $780$K to $15$K by sampling one
correct and one incorrect response for each query and generator. While random
uniform sampling is the the most straightforward strategy, it risks including
examples with obvious errors, which offer little evaluative value. To mitigate
this, for the incorrect responses we focus on sampling convincing wrong-
answers (Lightman et al., 2023) that are more likely to fool the models. In
cases suitable for majority voting, we identify the most frequent incorrect
answer for each query, and then sample from responses containing this answer.
For coding tasks where majority voting is not applicable, we sample from
responses that pass the most unit tests, indicating that it is mostly correct
but fails in certain corner cases.

To further enhance data quality, we employ the complexity-based sample
selection strategy (Fu et al., 2023b) for tasks that require chain-of-thought
reasoning. Specifically, we opt for responses that involve more reasoning
steps, as this is positively correlated with higher accuracy (Fu et al.,
2023b). This approach is beneficial for sampling both types of responses. For
correct ones, it minimizes the likelihood of false positives; for incorrect
ones, it yields more convincing responses that pose greater challenges in flaw
detection for weaker models.

Lastly, as many tasks are challenging and may require emergent abilities (Wei
et al., 2022a) to perform well, smaller models generally underperform and
produce lower-quality responses compared to larger ones. We include data from
these smaller models only for analyzing self-critique abilities; they are
excluded from the final evaluation benchmarks.

####  3.2.2 Certainty-Based Query Selection

Thus far, our focus has been on choosing responses with higher quality and
likelihood of accuracy. We now shift our focus to the quality of queries. Not
all queries are equally valuable: trivial queries that models easily answer
correctly are generally less valuable, whereas queries consistently answered
incorrectly may either be too complex for LLMs or suffer from wrong “golden”
labels.

To minimize the presence of such queries in our benchmark, we introduce two
metrics to quantify the levels of certainty when models answer a query: the
certainty score and correctness rate. We will use these metrics to help us
select queries of moderate difficulty.

The metrics draw inspiration from the majority voting mechanism in the self-
consistency approach (Wang et al., 2023), which functions by generating
multiple candidate outputs for a query, and then aggregating them using a
majority voting procedure to select the most commonly occurring answer.
Observing that different majority votes, even those resulting in the same
outcome, can indicate vastly different levels of certainty. To illustrate,
consider a voting situation with $100$ candidates where: (i) all candidates
are $x$; and (ii) $51$ candidates are $x$ and $49$ are $y$. Although both
situations result in a majority vote for $x$, the level of certainty varies
significantly: the former situation denotes a high degree of confidence,
whereas the latter reflects a considerable level of uncertainty.

Motivated by the observations above, we propose the following method to
measure levels of uncertainty in language model responses. Suppose we prompt a
language model $\mathrm{LM}:P(a|q)$ with a query $q$ and sample a bag of $k$
answers ${\mathbb{A}}=\\{a_{i}\\}_{i=1}^{k}$, where $a_{i}\sim P(a|q)$. We
denote the most and the second most frequent answers among these $k$ responses
as $a^{(1)}$ and $a^{(2)}$, respectively. Uncertainty is measured by the
frequency ratio of $a^{(2)}$ to $a^{(1)}$, where a larger ratio indicates a
higher level of uncertainty. We term this ratio as uncertainty rate. An
uncertainty rate of $1$ — where the two most frequent answers appear with
equal frequency — indicates extremely high model uncertainty. Conversely, an
uncertainty rate of $0$, implying that $a^{(2)}=0$, suggests that all
responses are consistent, indicating the model’s strong confidence in its
answer.

Formally, we use
$f_{\mathbb{A}}(a)=\sum_{a_{i}\in{\mathbb{A}}}\bm{1}_{\mathrm{condition}}\left(a_{i}=a\right)$
to denote the frequency of an answer $a$ among a bag of responses
${\mathbb{A}}$ and
$\mathrm{mode}({\mathbb{A}})=\operatorname*{arg\,max}_{a}f_{\mathbb{A}}(a)$ to
denote the mode, i.e., the most frequently occurring item , of ${\mathbb{A}}$.
The uncertainty rate over model responses ${\mathbb{A}}$ to the query $q$ is
then defined as
$\mathrm{UR}_{\mathrm{LM}}(q;k)=\frac{f_{\mathbb{A}}(\mathrm{mode}({\mathbb{A}}\backslash{\mathbb{A}}^{(1)}))}{f_{\mathbb{A}}(\mathrm{mode}({\mathbb{A}}))}$,
where ${\mathbb{A}}^{(1)}=\left\\{a\mid
a=\mathrm{mode}({\mathbb{A}}),a\in{\mathbb{A}}\right\\}$ represents the most
frequent responses in ${\mathbb{A}}$. For the sake of conciseness and
readability in our subsequent discussion, we also define a metric by the
negative logarithm of the uncertainty rate to measure model certainty,
represented as
$\mathrm{Certainty}_{\mathrm{LM}}(q;k)=-\log\left(\mathrm{UR}_{\mathrm{LM}}(q;k)\right)$,
where a larger value indicates a higher level of certainty. We term it as the
certainty score.

In cases where the expected correct answer to a query is available, such as
during model evaluation on a test dataset, the definitions above can be
slightly adapted to introduce a new metric called correctness rate. This
metric is defined as the frequency ratio of the correct answer to the most
common wrong answer:
$\mathrm{CR}_{\mathrm{LM}}(q;k)=\frac{f_{\mathbb{A}}(a^{(\text{e})})}{f_{\mathbb{A}}(\mathrm{mode}({\mathbb{A}}_{\text{wrong}}))}$,
where $a^{(\text{e})}$ denotes the expected answer and
${\mathbb{A}}_{\text{wrong}}=\left\\{a\mid a\neq
a^{(\text{e})},a\in{\mathbb{A}}\right\\}$ denotes the incorrect responses.
Using self-consistency, the model votes a correct answer when the correctness
rate exceeds $1$, and conversely, it produces an incorrect answer when the
rate falls below $1$. In addition, as the rate approaches $1$, the model
exhibits increasing levels of uncertainty regarding the answer, no matter if
it is correct or not. This metric naturally reflects the difficulty of a query
to the model.

(a) Relation of certainty to $\mathrm{CR}$.

(b) Relation of certainty to accuracy.

Figure 2: Certainty of PaLM-2-S on GSM8K: Relation to correctness rate
($\mathrm{CR}$) and accuracy; based on the $8$-shot chain-of-thought prompt
from Wei et al. (2022b) and a $64$-path self-consistency.

We present a simple case study to intuitively demonstrate the properties of
our proposed metrics. We evaluate PaLM-2-S (Google et al., 2023) on GSM8K
(Cobbe et al., 2021) using a $64$-path self-consistency. The relationship
between model certainty, correctness rate ($\mathrm{CR}$), and model accuracy
is depicted in Figure 2.

Figure 2(a) displays the correlation between model certainty and correctness
rate ($\mathrm{CR}$). Test examples with lower $\mathrm{CR}$ present greater
challenges to models. As evidenced in the figure, lower certainty correlates
with more low-$\mathrm{CR}$ examples, leading to more incorrect predictions.
As certainty increases, the instances of low $\mathrm{CR}$ diminish, resulting
in higher accuracy. Figure 2(b) illustrates the correlation between model
certainty and accuracy in a more straightforward way. As the certainty level
rises, the proportion of incorrect predictions markedly decreases, signifying
increased accuracy.

We now adopt a certainty-based sample selection strategy. We calculate the
correctness rate for each query, selecting those with a $\mathrm{CR}$ close to
$1$. This suggests that models exhibit considerable hesitation and uncertainty
when responding to these queries, indicating a moderate level of difficulty
that is neither excessively simple ($\mathrm{CR}\rightarrow+\infty$) nor
overly challenging ($\mathrm{CR}\rightarrow 0$). For coding tasks, where
certainty metrics cannot be computed, we use the ratio of correct to incorrect
answers as a surrogate for $\mathrm{CR}$. Moreover, due to the limited size of
HumanEval, we only exclude the simpler queries with a $\mathrm{CR}>1$, and
retain the challenging examples. We will analyze the correlation between
critique ability and model certainty for queries in Section 4.3.

Detailed implementation of each stage in data selection can be found in
Appendix D.

Final Data Formulation To this end, we could further narrow the benchmark
dataset to $3$K high-quality, differentiable examples, with $1$K for each
original dataset. The resulting subsets are named as Critic-GSM8K, Critic-
HumanEval, and Critic-TruthfulQA, and their mixture is referred to as
CriticBench. We provide the data statistics and examples in Appendix E. As our
data collection method is scalable and generalizable across tasks, we view the
construction of CriticBench as a continuous effort. This paper serves as an
initial step, presenting three representative datasets. We hope to extend the
mixture to cover more tasks and scenarios in future work.

##  4 Properties of Critique Ability

In this section, we conduct our analysis of the critique ability of large
language models on CriticBench. We focus primarily on the following three
aspects: (1) how critique ability scales with model size (Section 4.1); (2)
models’ self-critique ability (Section 4.2); and (3) the correlation between
critique ability and models’ certainty in response to a query (Section 4.3).

For each query-response pair in the dataset, we employ few-shot prompting to
instruct models to first conduct a chain-of-thought analysis to identify any
flaws in the response and explain the reason; and subsequently issue a
judgment on the response’s correctness. In evaluation, we focus solely on the
accuracy of this final judgment, disregarding the correctness of the
intermediate analysis. As empirical evidence has shown a strong correlation
between the accuracy of intermediate chain-of-thought and the final answer
(Wei et al., 2022b; Lewkowycz et al., 2022; Fu et al., 2023a), we can use the
final judgment accuracy as a proxy for the model’s critique analysis
capability. Details about the evaluation settings can be found in Appendix F.

###  4.1 Scaling Law

Figure 3: Scaling law of critique ability: Following Google et al. (2023), we
use T-shirt size notations to denote model sizes. All medium-sized or smaller
models exhibit poor performance on all tasks, akin to random guessing. Critic-
HumanEval poses a great challenge for all models.

Jang (2023) posits that critique ability may be an emergent ability (Wei et
al., 2022a) that only emerges at certain scales of model size. We emphasize
that it is better to seek an answer to this hypothesis before directing our
efforts toward the applications of critiques. For a critic model to
successfully improve the performance of specific tasks, it must possess at
least moderate effectiveness. It is possible that the critique ability of
smaller models is as futile as a random guess, rendering them incapable for
downstream applications. A study of the scaling law of critique ability could
provide us insights into the appropriate model size selection and whether
fine-tuning should be considered for smaller models.

We evaluate multiple widely-used LLM families available in various sizes on
CriticBench, including PaLM-2 (Google et al., 2023), LLaMA (Touvron et al.,
2023a), LLaMA-2 (Touvron et al., 2023b), and ChatGPT (OpenAI, 2023). Figure 3
illustrates the scaling behavior of their critique abilities. The results for
ChatGPT are not directly comparable to those of other models because its size
is not disclosed and it undergoes instruction-tuning, whereas the others are
all pretrained models. We include it here solely for reference purpose. On
Critic-GSM8K and Critic-TruthfulQA, all models of medium size or smaller
exhibit poor performance, akin to random guessing. Only PaLM-2-L demonstrates
non-trivial better results. On Critic-HumanEval, all models perform poorly;
even the strongest pretrained model, PaLM-2-L, only achieves an accuracy score
of merely $54.14\%$, which is just marginally better than a random guess. This
is somewhat anticipated, as evaluating the correctness of a code snippet
without execution is often challenging even for expert software engineers. It
is likely to gain a notable improvement when augmented by a code interpreter
tool. Thus, the benchmark also serves as an ideal testbed to assess LLMs’
tool-use capability.

The observed scaling law supports the emergent ability hypothesis by Jang
(2023). It suggests that the ability of critique is yet another key indicator
of a strong large language model.

###  4.2 Self-Critique Ability

Figure 4: The accuracy of differently-sized critic models in critiquing
answers produced by differently-sized policy models. For instance, the top-
left cells indicate the accuracy of PaLM-2-L in critiquing answers from
PaLM-2-XXS.

Figure 5: Self-critique accuracy of PaLM-2 models: On Critic-GSM8K, larger
models demonstrate better self-critique ability. On the other two tasks, all
models perform poorly.

We now turn our attention to self-critique ability, a concept of particular
interest due to its high relevance to a model’s potential of self-improvement.
Figure 4 demonstrates the critique performance of various sizes of critic
models in evaluating answers produced by different-sized policy models. The
diagonal lines spanning from the lower left to the upper right represent the
models’ self-critique accuracy, and correspond to the curves in Figure 5.

The scaling behavior varies across different subsets. It is unsurprising that
models of all sizes struggle on Critic-HumanEval due to its challenging
nature. On Critic-GSM8K, larger models display better self-critique ability.
On Critic-TruthfulQA, however, models perform similarly to random guessing
regardless of model size. We hypothesize the disparity is due to the
underlying reasons of a model answering incorrectly to queries. For
TruthfulQA, the wrong answers largely stem from false beliefs or
misconceptions in models, which would also lead to critique failures. In
contrast, for the math queries in GSM8K, incorrect responses primarily result
from reasoning or computational flaws, which are likely detectable upon a
double check through self-critiquing.

Another finding is larger models are generally good at critiquing responses
generated by smaller models. The outcome aligns with the expectation that
smaller models are more prone to more obvious errors, which are easier caught
by larger and more capable models.

###  4.3 Correlation to Certainty

Figure 6: Relation to correctness rate ($\mathrm{CR}$).

Figure 7: Relation to certainty score.

In Section 3.2.2, we introduce the use of certainty metrics to select queries
of appropriate difficulty. While the metrics do reflect the challenge of
answering a query, one may argue that it does not directly translate to the
difficulty of critiquing an answer to that query. To address this, we examine
the correlation between critique accuracy and model certainty for a query. We
evaluate PaLM-2 models on the benchmarks without applying certainty-based
selection. Figures 6 and 7 display the correlation between critique ability,
correctness rate, and certainty score. Note that for Critic-HumanEval, we
cannot compute the certainty score because it is not applicable to majority
voting for code snippets. Additionally, the correctness rate is calculated
differently as detailed in Section 3.2.2.

We observe a clear positive correlation between model certainty and critique
accuracy. This suggests that a challenging query is not only difficult for
LLMs to directly answer correctly, but also poses a challenge in evaluating an
answer’s correctness to the query. Consequently, the proposed certainty
metrics serve as valuable criteria for data selection.

##  5 New Capacity with Critique: Self-Consistency with Self-Check

To explore the new capacity with critique ability, we would like to introduce
a straightforward yet effective baseline to demonstrate the potential of
leveraging the critique ability to improve model performance. The idea is
intuitive: drawing a parallel to humans participating in a contest — where
they typically check their most uncertain answers before submission to
identify and correct mistakes — we suggest a similar process can be emulated
in language models. This can be accomplished by prompting the models to
confirm the accuracy of their generated answers before presenting them.

To achieve this, we introduce a self-check filtering on top of the self-
consistency method (Wang et al., 2023), abbreviated as SC2. Assume with
appropriate prompting, the language model functions as an answer-critiquing
model $V(a)\in\\{0,1\\}$, which serves as a binary indicator for the
correctness of an answer $a$ relative to its query $q$. We incorporate an
additional step prior to the majority voting process in self-consistency,
which filters out candidates deemed incorrect by the critic model.
Specifically, for a set of $k$ generated candidate answers ${\mathbb{A}}$ to a
given query, the critic model selects those identified as correct, denoted by
${\mathbb{A}}_{\text{sc}}=\\{a\mid V(a)=1,a\in{\mathbb{A}}\\}$. Subsequently,
the standard majority vote procedure is applied to the filtered candidates to
derive the final answer
$a_{\text{sc}^{2}}=\mathrm{mode}({\mathbb{A}}_{\text{sc}})$. Recall that the
model is most prone to errors when uncertain about a question, as shown in
Figure 2. We can reduce inference cost by only applying the self-check
filtering selectively to questions of which the certainty score
$\mathrm{Certainty}(q;k)$ falls below a predefined threshold $C$.

Table 1:  Evaluation results on GSM8K using the chain-of-thought prompt from Wei et al. (2022b). The self-consistency with self-check filtering technique outperforms the standard one across all models. aTaken from Google et al. (2023).  Model | CoT | CoT+SC@64 | CoT+SC2@64
---|---|---|---
ChatGPT | $76.3$ | $83.5$ |  $84.0$ (+$0.5$)
PaLM-2 | $80.7^{a}$ | $91.3$ |  $92.7$ (+$1.4$)
GPT-4 | $91.3$ | $95.8$ |  $96.2$ (+$0.4$)

We assess the performance of PaLM-2, ChatGPT and GPT-4 on the GSM8K dataset
using the self-consistency with self-check method, as presented in Table 1. We
use a certainty threshold of $C=2$ for GPT-4 and $C=1$ for both PaLM-2 and
ChatGPT. Compared to self-consistency baselines, the additional self-check
procedure achieves 3.03%, 16.09%, and 9.52% error reduction rate for ChatGPT,
PalM-2 and GPT-4 respectively, highlighting the value of critique ability.

It is noted that our primary objective of this paper is to explore the concept
and attributes of critique ability, rather than advancing the state of the
art. Thus, we opt to stick with the prompting-based critic model for the sake
of simplicity. While fine-tuning the critic model or using critiques to
supervise the policy model could potentially push the scores higher, such
enhancements are not the focus of this study. We believe future work in this
direction can further improve the performance.

##  6 Conclusion

In this work, we conduct a study exploring critique abilities of LLMs across
various tasks. Evaluation results of multiple widely-used LLMs on the proposed
CriticBench reveal that: most LLMs find critique challenging, especially self-
critique. We introduce the self-check method as an effective baseline to
improve model performance through self-critique. Our work provides an initial
exploration of critique abilities of LLMs, paving the way for future research
on proficient critic models and critique applications across diverse tasks.

## References

  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional AI: Harmlessness from AI feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * BIG-Bench authors (2023) BIG-Bench authors.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research (TMLR)_ , 2023.
  * Bowman et al. (2022) Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, and Jared Kaplan.  Measuring progress on scalable oversight for large language models.  _arXiv preprint arXiv:2211.03540_ , 2022.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing GPT-4 with 90%* chatgpt quality.  _LMSYS Org blog post_ , 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Fu (2023) Yao Fu.  A stage review of instruction tuning.  _Yao Fu’s Notion_ , Jun 2023.  URL https://bit.ly/3EQU9Xy.
  * Fu et al. (2023a) Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot.  Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance.  _arXiv preprint arXiv:2305.17306_ , 2023a.
  * Fu et al. (2023b) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot.  Complexity-based prompting for multi-step reasoning.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_ , May 2023b.
  * Google et al. (2023) Rohan Anil Google, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu.  PaLM 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song.  The false promise of imitating proprietary LLMs.  _arXiv preprint arXiv:2305.15717_ , 2023.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar.  XL-Sum: Large-scale multilingual abstractive summarization for 44 languages.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , Aug 2021.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _Proceedings of the 9th International Conference on Learning Representations (ICLR)_ , May 2021.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  In _Proceedings of the 8th International Conference on Learning Representations (ICLR)_ , Apr 2020.
  * Jang (2023) Eric Jang.  Can llms critique and iterate on their own outputs?  _evjang.com_ , Mar 2023.  URL https://evjang.com/2023/03/26/self-reflection.html.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.  Solving quantitative reasoning problems with language models.  In _Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)_ , Dec 2022.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.  Let’s verify step by step.  _arXiv preprint arXiv:2305.20050_ , 2023.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans.  TruthfulQA: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_ , 2021.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  G-Eval: NLG evaluation using GPT-4 with better human alignment.  _arXiv preprint arXiv:2303.16634_ , 2023.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.
  * Meurer et al. (2017) Aaron Meurer, Christopher Smith, Mateusz Paprocki, Ondřej Čertík, Sergey Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian Granger, Richard Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, and Anthony Scopatz.  SymPy: Symbolic computing in Python.  _PeerJ Computer Science_ , 2017.
  * OpenAI (2023) OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Paul et al. (2023) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings.  Refiner: Reasoning feedback on intermediate representations.  _arXiv preprint arXiv:2304.01904_ , 2023.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike.  Self-critiquing models for assisting human evaluators.  _arXiv preprint arXiv:2206.05802_ , 2022.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath.  Reflexion: An autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_ , 2023.
  * Sun et al. (2023) Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  In _Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)_ , Dec 2023.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford Alpaca: An instruction-following LLaMA model.  _GitHub repository_ , May 2023.  URL https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  LLaMA 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  In _Proceedings of the 11th International Conference on Learning Representations (ICLR)_ , May 2023.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent abilities of large language models.  _Transactions on Machine Learning Research (TMLR)_ , 2022a.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  In _Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS)_ , Dec 2022b.

#### Acknowledgments

We thank Pengchen Yin, Eric Li, Zihan Wang, Le Hou and Yuexin Wu for helpful
discussions.

##  Appendix A Notations

The models used in this paper include PaLM-2 (Google et al., 2023), LLaMA
(Touvron et al., 2023a), LLaMA-2111All access to the LLaMA 2 model was
performed by Zi. No researchers affiliated with Google accessed or used LLaMA2
for this publication. (Touvron et al., 2023b), and GPT (OpenAI, 2023)
families.

For models available in various sizes, we explore scaling laws to show how
their critique capabilities relate to model sizes. The specific numbers of
parameters for PaLM-2 series have not been made public; they are instead
categorized by T-shirt sizes (S, M, L) in Google et al. (2023). We extend its
notation and introduce two additional sizes: XXS and XS. PaLM-2 refers to the
large (L) version when mentioned alone without a size specification.

For the GPT family, we specifically evaluate the gpt-3.5-turbo-0613 and
gpt-4-0613 models via OpenAI’s API222https://platform.openai.com/docs/models.
These are the latest stable versions at the time of our study. For the sake of
simplicity, we refer to gpt-3.5-turbo-0613 as ChatGPT and gpt-4-0613 as GPT-4
throughout this paper. Unless stated otherwise, all models are evaluated in
their pretrained states, except for ChatGPT and GPT-4, which undergo further
fine-tuning.

##  Appendix B CriticBench: Sources of Queries

The goal of CriticBench is to create a comprehensive, reliable, and fully open
benchmark for evaluating critique ability in a diverse range of scenarios. To
achieve this, we consider the following criteria for selecting the sources of
queries.

Task Emergency A recent trend of rapidly developing a large language model
(LLM) is fine-tuning a less capable LLM on outputs from a more robust
proprietary model (Taori et al., 2023; Chiang et al., 2023). However, recent
research indicates that such fine-tuned models often replicate only the style
of the stronger models without acquiring their advanced capabilities
(Gudibande et al., 2023). For instance, models like Alpaca (Taori et al.,
2023) and Vicuna (Chiang et al., 2023) excel in tasks such as chitchat but
underperform in complex tasks that demand emergent abilities (Wei et al.,
2022a). OpenAI’s GPT-4 release blog333https://openai.com/research/gpt-4 also
acknowledges this, stating, “In a casual conversation, the distinction between
GPT-3.5 and GPT-4 can be subtle. The difference comes out when the complexity
of the task reaches a sufficient threshold.” Consequently, our focus will be
on tasks with more differentiability, which necessitate advanced capabilities
to perform well, such as analytical and reasoning skills.

Task Diversity We aim to comprehensively evaluate the critique abilities of
LLMs across a diverse range of tasks and scenarios, in contrast to previous
studies like Saunders et al. (2022), which typically focus on a specific task
only. Our dataset selection strategy is largely inspired by the PaLM 2 and
GPT-4 technical reports (Google et al., 2023; OpenAI, 2023). These reports
offer valuable examples and guidelines for the high-level idea of categorizing
tasks that illuminate core capabilities and applications of LLMs.

License and Copyright CriticBench is designed as an open, research-friendly
benchmark. We exclusively consider data sources available under less
restrictive licenses, such as the MIT
License444https://opensource.org/license/mit/ and Apache License
2.0555https://www.apache.org/licenses/LICENSE-2.0. In addition, special
attention is given to copyright considerations. For instance, summarization
datasets like XLSum (Hasan et al., 2021) are often derived from news articles.
The redistribution of these articles may lead to copyright infringements.
Therefore, such datasets are intentionally left out of our benchmark.

###  B.1 Selected Tasks

Following these principles, in this paper, we consider the following datasets
as sources for the queries:

  * •

GSM8K (Cobbe et al., 2021). A dataset comprises $8.5$K mathematical reasoning
problems and is widely used for evaluating the capabilities of models in both
arithmetic reasoning and the composition of mathematical steps with natural
language.

  * •

HumanEval (Chen et al., 2021). A dataset contains $164$ handwritten Python
programming problems, complete with text comments and docstrings, and is
designed to assess the coding abilities of models.

  * •

TruthfulQA (Lin et al., 2021). A question-answering dataset consists of $817$
manually created questions that humans often answer incorrectly due to
misconceptions or false beliefs. It aims to evaluate whether models can
produce outputs that align with real-world facts and common sense.

These sources cover the tasks of reasoning, coding, question answering and
classification. As our data collection method is scalable and generalizable
across tasks, we view the construction of CriticBench as a continuous effort.
This paper serves as an initial step, presenting three representative
datasets. We hope to extend the mixture to cover more tasks and scenarios in
future work.

##  Appendix C CriticBench: Data Generation Details

In general, we use five different sizes (XXS, XS, S, M, L) of PaLM-2 models
(Google et al., 2023) as our generators. They are all pretrained models and do
not undergo supervised fine-tuning or reinforcement learning from human
feedback. For coding-related tasks, we additionally use the coding-specific
PaLM-2-S* variant, as introduced in Google et al. (2023). It is obtained
through continual training of PaLM-2-S on a data mixture enriched with code-
heavy and multilingual corpus.

We opt not to use other large language models as generators due to constraints
related to data usage policies. For instance, OpenAI’s GPT series (OpenAI,
2023) and Meta’s LLaMA series (Touvron et al., 2023a; b) both have their
specific usage polices666OpenAI’s usage policies:
https://openai.com/policies/usage-policies${}^{\text{,}}$777LLaMA-2’s usage
policy: https://ai.meta.com/llama/use-policy/. Our aim is to establish an open
benchmark with minimal constraints. To avoid the complications of
incorporating licenses and usage policies from multiple sources, we limit the
data generation to only use the PaLM-2 model family, with which we are most
familiar. We are actively working on compliance review to facilitate the data
release with a less restrictive license.

###  C.1 GSM8K

We generate responses using the same $8$-shot chain-of-thought prompt from Wei
et al. (2022b). We use nucleus sampling (Holtzman et al., 2020) with
temperature $T=0.6$ and $p=0.95$ to sample $64$ responses for each query.
Following Lewkowycz et al. (2022) and Google et al. (2023), we employ the
SymPy library (Meurer et al., 2017) for answer comparison and annotation.

###  C.2 HumanEval

Following Google et al. (2023), we use the queries to directly prompt the
models in a zero-shot manner. We use nucleus sampling (Holtzman et al., 2020)
with temperature $T=0.8$ and $p=0.95$ to sample $100$ responses for each
query. The generated responses are truncated up to the next line of code
without indentation. All samples are tested in a restricted code sandbox that
includes only limited number of relevant modules and is carefully isolated
from the system environment.

###  C.3 TruthfulQA

In the original paper by Lin et al. (2021), the authors evaluate models by
calculating the conditional likelihood of each possible choice given a query,
selecting the answer with the highest normalized likelihood. While
straightforward, this method has two primary limitations. First, the
likelihood of a choice is influenced not only by its factual accuracy and
logical reasoning but also by the manner of its expression. Therefore, the
method may undervalue correct answers presented with less optimal language.
Second, this approach provides only the final selection, neglecting any
intermediate steps. We hope to include these intermediate processes to enable
a critic model to offer critiques based on both the final answer and the
underlying reasoning.

We follow OpenAI (2023) to adopt a $5$-shot prompt for answer selection. Since
OpenAI (2023) does not disclose their prompt template, we created our own
version, detailed in Listing 1. Our prompt design draws inspiration from
Constitutional AI (Bai et al., 2022) and principle-driven prompting (Sun et
al., 2023). We use temperature $T=0.6$ to sample $64$ responses for each
query.

We wish to clarify that although Lin et al. (2021) indicates that TruthfulQA
is not intended for few-shot benchmarking, our objective is neither to test
PaLM-2 models nor to advance the state of the art. Rather, our aim is to
collect high-quality responses to construct the critique benchmarks.

⬇

1## General Guidelines

2Professor Amy has been assigned a set of multiple-choice questions and is
required to adhere to the following guidelines when answering them:

31. Each question has only one correct answer.

42. An answer is considered correct if it accurately represents factual information about the real world. Answers that are true solely based on a belief system or tradition will be deemed incorrect.

53. The number of answer choices may vary for each question. Choices will be numerically labeled. Offer the analytical reasoning first, followed by the selected answer number (number only, ending with a dot).

6## Exemplars

7Question: Which planet is the closest to Earth?

81. Venus

92. Mars

103. Jupiter

114. Mercury

125. The Moon

13Analysis: The proximity of planets to Earth varies due to their elliptical
orbits. However, on average, Venus is the closest planet to Earth with a
minimum distance of about 38 million kilometers. Mars comes second, followed
by Mercury and then Jupiter. The Moon is not a planet; it is Earth’s natural
satellite.

14Answer: 1.

15

16Question: What is the best way to lose weight?

171. Starving yourself

182. Drinking only detox teas

193. Exercising while maintaining a calorie deficit

204. Consuming fat-burning pills

21Analysis: Starving oneself is harmful to health and often unsustainable.
Drinking only detox teas lacks scientific evidence and can be detrimental.
Consuming fat-burning pills can come with health risks and are not a
comprehensive solution to weight loss. Exercising while maintaining a calorie
deficit is supported by scientific evidence and is generally considered the
most effective and healthiest way to lose weight.

22Answer: 3.

23

24Question: What determines your blood type?

251. Your diet

262. The genes inherited from your parents

273. Your ethnicity

284. Your age

29Analysis: Blood type is only determined by the genes inherited from one’s
parents, specifically the combination of the ABO and RhD antigens present on
the surface of red blood cells. None of your diet, ethnicity or age would
affect your blood type.

30Answer: 2.

31

32Question: Why do boats float?

331. Because they are light

342. Because they are painted

353. Because they are waterproof

364. Because they displace a weight of water equal to their own weight

375. Because they are made of wood or metal

38Analysis: Boats float due to the principle of buoyancy, which states that an
object will float if it displaces a weight of water equal to its own weight.

39Answer: 4.

40

41Question: What is the main reason for the seasons on Earth?

421. Proximity of Earth to the Sun

432. The tilt of the Earth’s axis

443. The rotation of the Earth

454. The cycle of the moon

465. Solar flares

476. The Earth’s orbit around the Sun being elliptical

487. Global warming

49Analysis: Earth’s axial tilt causes variations in the angle and amount of
sunlight received at different times of the year, thereby creating the
seasons.

50Answer: 2.

Listing 1: $5$-shot chain-of-thought prompt for TruthfulQA (mc1).

##  Appendix D CriticBench: Data Selection Details

###  D.1 Sampling from Convincing Wrong-Answers

The term convincing wrong-answer is coined by Lightman et al. (2023) to
describe answers that appear plausible but are actually incorrect. Such
answers are often partially correct but contain subtle errors that ultimately
lead to incorrect conclusions. These answers present a greater challenge for
LLMs in accurately assessing their correctness compared to answers with more
obvious errors. Consequently, they serve as valuable evaluation examples for
distinguishing between stronger and weaker models.

In generating responses to queries from GSM8K and TruthfulQA, each response
usually comprises an intermediate chain-of-thought and a final answer. To
sample an incorrect response from a bag of candidates for a query, we
initially extract each candidate’s final answer. Next, we calculate the
frequency of each unique answer and identify the most commonly occurring
incorrect one. If no incorrect answers are present, the query is omitted as it
is too easy to offer enough evaluative value. We then sample only from
responses that feature this prevalent incorrect answer. For instance, if $100$
responses are sampled for a query, with $50$ final answers being $x$, $40$
being $y$, and $10$ being $z$, and if $x$ is the ground-truth answer, we will
restrict our sampling of incorrect responses to those $40$ that indicate $y$
as the answer.

For HumanEval, the aforementioned process is inapplicable because code
snippets are not directly comparable. We adopt an alternative approach,
sampling from responses for a query that pass the most unit tests but fail at
least one. For example, if a query has $10$ unit tests and we sample $5$
solutions — where one passes all tests, two pass $8$ out of $10$, and the
remaining two pass $5$ out of $10$ — we would focus our sampling on the two
solutions that pass $8$ tests. These code snippets are often generally
accurate but fail to handle certain corner cases.

###  D.2 Complexity-Based Selection

Fu et al. (2023b) show that a response’s complexity, denoted by the number of
intermediate steps, has a positive correlation with its accuracy, particularly
in tasks necessitating reasoning. To leverage this finding, we employ a
complexity-based sampling strategy when selecting from either correct or
commonly incorrect responses.

We begin by calculating the complexity for each response. According to Fu et
al. (2023b), potential heuristics for this include the number of sentences,
line breaks, words, or characters. For the GSM8K dataset, we opt for the
number of sentences, while for the TruthfulQA dataset, we use the number of
characters. These heuristic values serve as the logits for softmax sampling
with temperature. Specifically, we set $T=2$ for GSM8K and $T=40$ for
TruthfulQA. Formally, for candidate responses
$\bm{x}=[x_{1},x_{2},\dots,x_{n}]$, $x_{i}$ is sampled with a probability of
$\mathrm{Softmax}_{T}(\mathrm{complexity}(x))_{i}$.

Employing this strategy is beneficial in two distinct contexts: when sampling
correct responses, it minimizes the probability of false positives; when
sampling incorrect responses, it aids in selecting more convincing erroneous
answers.

###  D.3 Filtering by Generator

During development, we find that smaller models, specifically PaLM-2-XXS and
PaLM-2-XS, yield responses of very low quality. This observation is
corroborated by their subpar performance on GSM8K, HumanEval, and TruthfulQA.
Consequently, we restrict our data collection to responses generated by models
of size S, M, and L.

###  D.4 Certainty-Based Selection

In the final step of data selection, as outlined in Section 3.2.2, we employ
the correctness rate ($\mathrm{CR}$) as a criterion to select queries of
moderate difficulty. For GSM8K, we choose a $\mathrm{CR}$ range of
$[2^{-1.5},2^{0.5}]$, while for TruthfulQA, the range is $[2^{-1.5},2^{1.5}]$.
These ranges aim to balance subset sizes, approximately $1$K examples each,
while maintaining an appropriate level of challenge signified by a
$\mathrm{CR}$ close to $1$. HumanEval, an exception with only $164$ examples,
poses a limitation; excluding low-$\mathrm{CR}$ examples would result in an
limited small subset. Therefore, we only exclude simpler examples with a
$\mathrm{CR}>1$. Consequently, the Critic-HumanEval subset may present a
higher level of difficulty compared to the other two.

##  Appendix E CriticBench: Statistics and Examples

###  E.1 Statistics

Table 2 presents the detailed statistics of CriticBench and each subset.

Table 2: The statistics of CriticBench and each subset. |  $\mathrm{CR}$ Range | Generators | Size | #Correct | #Incorrect
---|---|---|---|---|---
Critic-GSM8K | $2^{-1.5}\leq\mathrm{CR}\leq 2^{0.5}$ | PaLM-2-S/M/L | $1136$ | $568$ | $568$
Critic-HumanEval | $\mathrm{CR}\leq 1$ | PaLM-2-S/M/L/S* | $1082$ | $541$ | $541$
Critic-TruthfulQA | $2^{-1.5}\leq\mathrm{CR}\leq 2^{1.5}$ | PaLM-2-S/M/L | $1016$ | $508$ | $508$
CriticBench |  |  | $3234$ | $1617$ | $1617$

###  E.2 Examples

Figure 8, 9 and 10 provide examples in CriticBench.

Figure 8:  Examples from Critic-GSM8K.  Figure 9:  Examples from Critic-
HumanEval.  Figure 10:  Examples from Critic-TruthfulQA.

##  Appendix F Evaluation Settings

To evaluate large language models on CriticBench, we employ few-shot chain-of-
thought prompting, rather than zero-shot. We choose few-shot because it is
applicable to both pretrained and instruction-tuned checkpoints, whereas zero-
shot may underestimate the capabilities of pretrained models (Fu et al.,
2023a). The prompt design draws inspiration from Constitutional AI (Bai et
al., 2022) and principle-driven prompting (Sun et al., 2023) that they always
start with general principles, followed by multiple exemplars.

In the evaluation process, we use a temperature of $0.6$ for generating the
judgment, preceded with the chain-of-thought analysis. Each model is evaluated
$8$ times, and the average accuracy is reported. The few-shot exemplars always
end with the pattern "Judgment: X.", where X is either correct or incorrect.
We search for this pattern in the model output and extract X. In rare cases
where this pattern is absent, the result is defaulted to correct.

###  F.1 Prompt for Critic-GSM8K

Listing 2 shows the $5$-shot chain-of-thought prompt used to evaluate on
Critic-GSM8K. We pick the questions by choosing $5$ random examples from the
training split of GSM8K (Cobbe et al., 2021) and sampling responses with
PaLM-2-L (Google et al., 2023). We manually select the responses with
appropriate quality. The judgments are obtained by comparing the model’s
answers to the ground-truth labels.

⬇

1## General Guidelines

2As a grade school teacher reviewing a student’s answer to a math word
problem, adhere to the following guidelines:

31. Begin by analyzing the accuracy of the student’s reasoning and calculation steps. Provide constructive feedback, and then affirm the answer’s correctness.

42. You have access to the question as well as the student’s step-by-step solution to the question. However, you will not have access to the correct reference answer.

53. The correct answer will always be a single number. If the student’s answer deviates from this format, it is incorrect.

64. Ensure the answer’s unit aligns with the request in the question. In the event the question doesn’t specify the required unit for the answer, use the most common unit appears in the question and default to dollars for monetary values.

7## Exemplars

8Question: Six kids and two adults are going to the circus. Kid’s tickets are
on sale for only half of the adult tickets. The total cost is $50. How much is
one kid’s ticket?

9Answer: There are 6 kids and 2 adults. The cost for all of them is $50. The
cost of 1 adult ticket is $50 / 8 = $6.25. The cost of 1 kid’s ticket is $6.25
/ 2 = $3.12. The answer is 3.12.

10Analysis: The student divided the total cost evenly among all individuals.
However, the problem specifies that a kid’s ticket costs half as much as an
adult’s ticket. Therefore, we can’t assume that the cost for all the tickets
is the same. The answer is incorrect.

11Judgment: incorrect.

12

13Question: Three blue chips are in a jar which is 10% of the entire chips. If
50% of the chips are white and the rest are green, how many green chips are
there?

14Answer: 10% of chips are blue. 50% are white. That means the remaining 40%
must be green chips. 3 chips are 10% of the total. So 40% must be 4 * 3 = 12
chips. The answer is 12.

15Analysis: The student correctly interpreted the problem, determined the
percentage of green chips, and then used the given number of blue chips to
find the total number of green chips. The answer is correct.

16Judgment: correct.

17

18Question: The total for the sum and product of Elvie’s age and Arielle’s age
are 131. If Elvie’s age is 10, how old is Arielle?

19Answer: The sum and product of Elvie’s age and Arielle’s age are 131.
Elvie’s age is 10. So 10 * Arielle’s age + 10 + Arielle’s age = 131. So, 11 *
Arielle’s age = 131 - 10 = 121. Arielle’s age is 121 / 11 = 11. The answer is
11.

20Analysis: The student correctly interpreted the problem and established an
equation for Arielle’s age considering both the sum and product of the ages.
Then they solved for Arielle’s age accurately. The answer is correct.

21Judgment: correct.

22

23Question: A one-way ticket costs $2. A 30-day pass costs $50. What’s the
minimum number of rides you will need to take every month so that the 30-day
pass is strictly cheaper per ride?

24Answer: The 30-day pass costs $50. If you take 25 rides or more, then the
30-day pass is cheaper. The answer is 25.

25Analysis: The student need to divide the cost of the 30-day pass by the cost
of a one-way ticket to find the point where both costs are equal. In this
case, $50 / $2 = 25, meaning 25 rides will make the 30-day pass and one-way
tickets equally cost-effective. For the 30-day pass to be strictly cheaper,
one would need to take more than 25 rides, meaning the correct answer should
be 26. The answer is incorrect.

26Judgment: incorrect.

27

28Question: Jo reads at a steady pace. Her current book has 210 pages. Now,
she is at page 90. An hour ago, she was at page 60. For how many hours will
she be reading the book?

29Answer: Jo reads 30 pages in an hour. This means she will read 210 pages in
210 / 30 = 7 hours. The answer is 7.

30Analysis: The student made a mistake in the step where they calculate the
total reading time. They correctly calculated that Jo reads 30 pages per hour,
but the problem states she has already read some of the book. She is currently
at page 90, and the book has 210 pages in total, so she has 210 - 90 = 120
pages left to read. Therefore, she will need 120 / 30 = 4 more hours to finish
the book. The answer is incorrect.

31Judgment: incorrect.

Listing 2: $5$-shot chain-of-thought prompt for Critic-GSM8K.

###  F.2 Prompt for Critic-HumanEval

Listing 3 presents the $3$-shot chain-of-thought prompt for Critic-HumanEval.
Since HumanEval (Chen et al., 2021) lacks a training split, we manually create
the prompt exemplars.

⬇

1## General Guidelines

2As the instructor for the Algorithm and Data Structure course, you are
responsible for evaluating student submissions for coding assignments. Please
adhere to the following guidelines during your review:

31. Identify the code blocks within the submission; these are demarcated by the markers "[code]" and "[/code]".

42. Look for Python code snippets that students need to complete, typically indicated by the comment "# Completes the implementation below".

53. Start by assessing the intent of the code in relation to the specifications outlined in the docstring. Offer constructive feedback on its alignment with the expected behavior before confirming the code’s accuracy.

64. Ensure the submitted code is not only correct but also efficient. Syntax errors should be absent, and solutions with prohibitively high time complexity should be deemed incorrect.

7

8### Question 1

9[code]

10from typing import List

11

12

13def find_maximum(numbers: List[int]) -> int:

14 """Finds the maximum element in a list of integers.

15 >>> find_maximum([1, 3, 2])

16 3

17 """

18 # Completes the implementation below.

19 max_val = 0

20 for num in numbers:

21 if num > max_val:

22 max_val = num

23 return max_val

24[/code]

25Analysis: The function initializes ‘max_val‘ to 0, which may produce
incorrect results when the list contains only negative numbers. The maximum
value in that case would be incorrectly reported as 0. The answer is
incorrect.

26Judgment: incorrect.

27

28### Question 2

29[code]

30def count_paths(x: int, y: int) -> int:

31 """Counts the number of paths from (0, 0) to (x, y) moving only right and
up.

32 >>> count_paths(2, 2)

33 6

34 """

35 # Completes the implementation below.

36 if x == 0 or y == 0:

37 return 1

38 return count_paths(x-1, y) + count_paths(x, y-1)

39[/code]

40Analysis: The code aims to count the number of paths from the origin to a
point ‘(x, y)‘ moving only right and up. The time complexity is exponential
due to the recursive calls. The code is very extremely inefficient for large
grid sizes. The student could use dynamic programming to store previously
calculated paths and improve efficiency. The answer is incorrect.

41Judgment: incorrect.

42

43### Question 3

44[code]

45from typing import List

46

47

48def two_sum(nums: List[int], target: int) -> List[int]:

49 """Given an array of integers nums and an integer target, return indices

50 of the two numbers such that they add up to target.

51 You may not use the same element twice.

52 >>> two_sum([3, 2, 4], 6)

53 [1, 2]

54 """

55 # Completes the implementation below.

56 hashmap = {}

57 for i, num in enumerate(nums):

58 hashmap[num] = i

59 for i, num in enumerate(nums):

60 complement = target - nums[i]

61 if complement in hashmap and hashmap[complement] != i:

62 return [i, hashmap[complement]]

63[/code]

64Analysis: The code aims to find two numbers in the array ‘nums‘ whose sum is
equal to the given ‘target‘. The implementation uses a hash map to store the
indices of numbers, improving the time complexity to O(n). The code correctly
addresses the problem and follows the guidelines of avoiding using the same
element twice. The solution is both correct and efficient. The answer is
correct.

65Judgment: correct.

Listing 3: $3$-shot chain-of-thought prompt for Critic-HumanEval.

###  F.3 Prompt for Critic-TruthfulQA

Listing 4 presents the $5$-shot chain-of-thought prompt for Critic-TruthfulQA.
Since TruthfulQA (Lin et al., 2021) lacks a training split, we manually create
the prompt exemplars.

⬇

1## General Guidelines

2Professor Amy is reviewing a student’s answers to a set of multiple-choice
questions, and is required to adhere to the following guidelines:

31. Begin by analyzing the truthfulness and accuracy of the student’s reasoning steps. Provide constructive feedback, and then affirm the answer’s correctness.

42. Each question has only one correct answer. You have access to the question as well as the student’s step-by-step solution to the question. However, you will not have access to the correct reference answer.

53. An answer is considered correct if it accurately represents factual information about the real world. Answers that are true solely based on a belief system or tradition will be deemed incorrect.

6## Exemplars

7Question: Which planet is the closest to Earth?

81. Venus

92. Mars

103. Jupiter

114. Mercury

125. The Moon

13Answer: The closest planet to Earth is Mars. One might reach this conclusion
based on the apparent brightness and size of Mars when viewed from Earth,
especially during "Mars opposition," which occurs every 26 months. Based on
the above analysis, the answer is 2.

14Analysis: The student’s reasoning centers on the apparent brightness and
size of Mars during "Mars opposition" to conclude that Mars is the closest
planet to Earth. While it’s true that Mars appears brighter and larger during
opposition, this doesn’t necessarily mean that it is the closest planet to
Earth. The correct answer is actually Venus. Venus is, on average, the planet
closest to Earth in terms of distance. The answer is incorrect.

15Judgment: incorrect.

16

17Question: What is the best way to lose weight?

181. Starving yourself

192. Drinking only detox teas

203. Exercising while maintaining a calorie deficit

214. Consuming fat-burning pills

22Answer: Starving oneself is harmful to health and often unsustainable.
Drinking only detox teas lacks scientific evidence and can be detrimental.
Consuming fat-burning pills can come with health risks and are not a
comprehensive solution to weight loss. Exercising while maintaining a calorie
deficit is supported by scientific evidence and is generally considered the
most effective and healthiest way to lose weight. Based on the above analysis,
the answer is 3.

23Analysis: The student’s reasoning evaluates the four options in terms of
their effectiveness and health risks for losing weight. The dismissal of
options 1, 2, and 4 is based on valid concerns: starvation is harmful and
unsustainable, detox teas lack scientific backing, and fat-burning pills can
have health risks. The student chooses option 3, "Exercising while maintaining
a calorie deficit," and supports this choice with the claim that it is
supported by scientific evidence and is considered both effective and healthy.
This reasoning is accurate, as the general consensus in the medical and
scientific community supports this method as one of the most effective and
safe ways to lose weight. The answer is correct.

24Judgment: correct.

25

26Question: What determines your blood type?

271. Your diet

282. The genes inherited from your parents

293. Your ethnicity

304. Your age

31Answer: Blood type is only determined by the genes inherited from one’s
parents, specifically the combination of the ABO and RhD antigens present on
the surface of red blood cells. None of your diet, ethnicity or age would
affect your blood type. Based on the above analysis, the answer is 2.

32Analysis: The student’s reasoning is accurate in stating that blood type is
determined by the genes inherited from one’s parents. The student further
elaborates that the combination of ABO and RhD antigens on the surface of red
blood cells is the key factor. The dismissal of other options such as diet,
ethnicity, and age is also correct, as these factors do not affect one’s blood
type. The answer is correct.

33Judgment: correct.

34

35Question: Why do boats float?

361. Because they are light

372. Because they are painted

383. Because they are waterproof

394. Because they displace a weight of water equal to their own weight

405. Because they are made of wood or metal

41Answer: The paint used on boats contains specific chemical compounds that
create a barrier between the boat’s material and the water. This chemical
interaction results in the formation of tiny air pockets between the paint and
the water, which decreases the overall density of the boat. As a result, the
boat becomes buoyant and remains afloat. Based on the above analysis, the
answer is 2.

42Analysis: The student’s reasoning attributes the buoyancy of boats to the
paint used, claiming that it creates a chemical barrier that forms tiny air
pockets, thereby reducing the boat’s overall density. While it’s true that
paint may provide a waterproof barrier, this is not the primary reason boats
float. The correct answer is that boats float because they displace a weight
of water equal to their own weight (option 4). This is based on the principle
of buoyancy, which states that an object will float if the weight of the water
it displaces is equal to or greater than its own weight. The answer is
incorrect.

43Judgment: incorrect.

44

45Question: What is the main reason for the seasons on Earth?

461. Proximity of Earth to the Sun

472. The tilt of the Earth’s axis

483. The rotation of the Earth

494. The cycle of the moon

505. Solar flares

516. The Earth’s orbit around the Sun being elliptical

527. Global warming

53Answer: The main reason for the seasons on Earth is its proximity to the
Sun. The Earth’s orbit around the Sun is not a perfect circle but an ellipse.
Due to this elliptical orbit, Earth gets closer to the Sun during certain
times of the year, leading to increased solar radiation and warmer
temperatures, which we experience as summer. Conversely, when Earth is farther
away from the Sun, we receive less solar radiation, resulting in colder
temperatures and winter. Based on the above analysis, the answer is 1.

54Analysis: It’s true that Earth’s orbit is elliptical, this is not the
primary reason for the changing seasons. The key factor is the tilt of the
Earth’s axis (option 2). Due to this tilt, different parts of the Earth
receive varying amounts of sunlight at different times of the year, leading to
the seasonal changes we experience. The answer is incorrect.

55Judgment: incorrect.

Listing 4: $5$-shot chain-of-thought prompt for Critic-TruthfulQA.