#  Easy-to-Hard Generalization:
Scalable Alignment Beyond Human Supervision

Zhiqing Sun  Longhui Yu  Yikang Shen  Weiyang Liu  Yiming Yang  Sean Welleck
Chuang Gan

###### Abstract

Current AI alignment methodologies rely on human-provided demonstrations or
judgments, and the learned capabilities of AI systems would be upper-bounded
by human capabilities as a result. This raises a challenging research
question: How can we keep improving the systems when their capabilities have
surpassed the levels of humans? This paper answers this question in the
context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via
learning from human annotations on easier tasks (e.g., level 1-3 MATH
problems), which we term as easy-to-hard generalization. Our key insight is
that an evaluator (reward model) trained on supervisions for easier tasks can
be effectively used for scoring candidate solutions of harder tasks and hence
facilitating easy-to-hard generalization over different levels of tasks. Based
on this insight, we propose a novel approach to scalable alignment, which
firstly trains the process-supervised reward models on easy problems (e.g.,
level 1-3), and then uses them to evaluate the performance of policy models on
hard problems. We show that such easy-to-hard generalization from evaluators
can enable easy-to-hard generalizations in generators either through re-
ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL
model achieves an accuracy of 34.0% on MATH500, despite only using human
supervision on easy problems. Our approach suggests a promising path toward AI
systems that advance beyond the frontier of human supervision.

Machine Learning, ICML

##  1 Introduction

Rapid advancements in large language models (LLMs) indicate that in the near
future, highly sophisticated AI systems could surpass human capabilities in
certain areas, significantly enhancing our understanding of various fields and
our capabilities in solving harder problems beyond the levels we can currently
solve (OpenAI, 2022, 2023b). Since the current AI alignment methods mostly
rely on either supervised fine-tuning (SFT) with human-provided demonstrations
(Sanh et al., 2021; Wei et al., 2021; Chung et al., 2022) or reinforcement
learning from human feedback (RLHF) (Ziegler et al., 2019; Stiennon et al.,
2020; Ouyang et al., 2022), their capabilities would be inherently limited as
humans cannot always provide helpful demonstrations or supervision on the hard
tasks beyond their expertise (Sharma et al., 2023).

Figure 1:  We first train the evaluator with process supervision to enable
easy-to-hard evaluation, and then use it to facilitate easy-to-hard generation
via re-ranking or RL.  Figure 2:  Illustration of different alignment
scenarios: traditional alignment relies on human demonstrations or judgements
(Ouyang et al., 2022); scalable alignment (Bowman et al., 2022) assumes that
humans cannot reliably supervise smarter-than-human models; weak-to-strong
generalization(Burns et al., 2023) focuses on using weak models with
unreliable labels to supervise strong models; Our proposed easier-to-general
generalization focuses on the transfer of rewarding policies from weak models
to harder tasks.

In order to build future AI systems for tackling complex challenges, such as
advancing scientific knowledge, it is crucial to develop new approaches for
scalable oversight challenge, i.e., to supervise the AI systems that can
potentially outperform humans in most skills relevant to the task at hand
(Bowman et al., 2022). The key question is:

  * •

Can we limit human supervision to easier tasks, yet enable the model to excel
in performing harder tasks?

We refer to this scenario as Easy-to-Hard Generalization (Schwarzschild et
al., 2021; Zhou et al., 2022a; Burns et al., 2023; Hase et al., 2024). This
setting requires no human supervision on the harder tasks, which differs from
existing work that either enhances humans’ ability to verify the outputs of AI
systems (Wu et al., 2021; Saunders et al., 2022; Bowman et al., 2022; Rein et
al., 2023) or enables weak-to-strong generalization via a teacher that only
offers unreliable or noisy supervision (Burns et al., 2023).

The most basic form of easy-to-hard generalization can be achieved by training
the policy models (i.e., generator) using supervised fine-tuning (SFT) or in-
context learning (ICL) on easy tasks (Radford et al., 2018; Brown et al.,
2020), and expect this will unlock the ability to perform well on hard tasks.
However, it has been observed that SFT or ICL training of generators on easy
tasks often fails to generalize to hard tasks (Swayamdipta et al., 2020; Fu et
al., 2022; Zhou et al., 2022a). We hypothesize and show that methods beyond
these can enable stronger degrees of easy-to-hard generalization. Our
intuition is guided by the observation that evaluation is easier than
generation (Karp, 1975; Naor, 1996), so an evaluator may offer a degree of
easy-to-hard generalization that is useful for improving a generator. If that
is true, we can first train a verifier on easy tasks, then make use of its
generalization ability to supervise the generator on hard tasks.

Complex tasks can often be broken down into smaller steps (Zhou et al., 2022a)
and verified by validating the individual steps – a strategy that is commonly
employed in solving mathematical problems (Uesato et al., 2022; Lightman et
al., 2023; Trinh et al., 2024). Inspired by this, we train process-supervised
reward models (PRMs) (Uesato et al., 2022; Yu et al., 2023a; Wang et al.,
2023; Lightman et al., 2023) as our easy-to-hard evaluators. The training
dataset is often comprised of a set of labeled easy tasks, each with a
question and a high-quality solution111We assume that human supervision is of
high quality on the easy tasks in general., paired with a set of unlabeled
hard tasks that are represented only by their questions. This simulates the
practical setting of having numerous problems with known solutions, as well as
significant unresolved challenges, such as the Millennium Prize Problems
(Carlson et al., 2006), which present challenging open problems. The pivotal
aspect of easy-to-hard generalization thus lies in how we effectively leverage
the capabilities of easier-level models in solving harder problems.

Our investigation includes to training policy and process reward models on the
easy (i.e., level 1-3) portion of the PRM800K (Lightman et al., 2023) dataset,
and comparing the performance of majority voting with the policy model only
and weighted majority voting with the policy model and PRMs (Process-
supervised Reward Models). We also introduce the Outcome & Process Reward
Model (OPRM), which harnesses the complementary strengths of outcome reward
models (ORMs) and process reward models (PRMs): judging if each step in
reasoning is correct (like PRMs do) and deciding if the final answer is right
(like ORMs do). Our findings reveal a marked performance improvement with the
inclusion of PRMs (or OPRMs), especially on the hard (i.e., level 4-5) portion
of the MATH500 test set. This improvement indicates that easier-level
evaluators can maintain their effectiveness on harder tasks. We have similar
observations in our experiments on the MetaMath dataset (Yu et al., 2023b) and
the Math-Shepherd dataset (Wang et al., 2023).

We further investigate the use of the easy-to-hard evaluator as a reward model
in reinforcement learning, where the process-based evaluator provides
targeted, step-by-step guidance in solving hard problems. We have an
intriguing finding that training with human supervision only on the easy tasks
(i.e., training with Level 1-3 problems and answers) can outperform both SFT
and Final-Answer RL training on the full dataset (Level 1-5). This finding
underscores the potential of using easy-to-hard evaluation to improve easy-to-
hard generators, particularly when dealing with varied levels of task
complexity.

##  2 Related Work

###  2.1 Scalable Oversight

While present-day models operate within the scope of human assessment, future,
more advanced models may engage in tasks that are beyond human evaluation
capabilities. This raises a concern that such models might prioritize
objectives other than maintaining accuracy (Andreas 2022; Perez et al. 2022;
Sharma et al. 2023; Wei et al. 2023). To address this, a branch of research
develops techniques to enhance the human capacity to supervise such models,
such as via using AI to evaluate the work of other AIs (Amodei et al., 2016;
Leike et al., 2018; Saunders et al., 2022; Bowman et al., 2022). Our setting
differs from enhancing human oversight; instead, we focus on enabling models
to excel in hard tasks where human supervision may not be available. This also
differs from weak-to-strong generalization (Burns et al., 2023), where human
supervision may be available, but not reliable, on hard tasks. However, our
framework aligns with the “sandwiching” concept proposed for measuring
progress in scalable oversight, which involves domain experts evaluating the
outputs of AI-assisted non-experts (Cotra, 2021; Bowman et al., 2022; Rein et
al., 2023).

###  2.2 Rationale-Augmented (Mathematical) Reasoning

Ling et al. (2017) pioneer the work of solving math word problems by
generating step-by-step solutions before the final answer. Cobbe et al.
(2021a) extend this work by constructing a much larger dataset to finetune a
pre-trained large language model to solve math word problems, and a outcome-
supervised verifier to rank candidate solutions. Wei et al. (2022) demonstrate
that the reasoning ability of a language model can be elicited through the use
of prefixed rationales. Subsequent research (Kojima et al., 2022; Yao et al.,
2022; Lewkowycz et al., 2022; Zhou et al., 2022b; Yao et al., 2023) in tasks
requiring human-level reasoning skills has also highlighted the efficacy of
rationale augmentation.

Among all the reasoning tasks, we select mathematical reasoning to evaluate
easy-to-hard generalization ability, given that mathematical reasoning serves
as a valuable assessment for complex reasoning abilities and features a clear
delineation of difficulty levels. Recent research efforts focus on prompt
design (Wei et al., 2022; Zhou et al., 2022a; Fu et al., 2022; Zhang et al.,
2022; Zheng et al., 2023) to elicit the intrinsic reasoning capabilities of
models, or data engineering for fine-tuning (Luo et al., 2023; Yuan et al.,
2023; Yue et al., 2023; Yu et al., 2023b; Gou et al., 2023; Liu et al., 2023;
An et al., 2023; Azerbayev et al., 2023), which draws on experts to provide
high-quality training datasets. Our work is categorized as fine-tuning based
work. However, unlike previous work, our focus lies in exploring how to
generalize to more challenging mathematical problems when only provided with
easy mathematical data.

###  2.3 Compositional & Easy-to-Hard Generalization

Compositional generalization is a fundamental aspect of how language works
(Chomsky, 1981). It refers to the ability to understand and utilize novel
combinations based on the understanding of basic concepts and a limited number
of their combinations (Fodor & Lepore, 2002). Recently, least-to-most
prompting (Zhou et al., 2022a; Drozdov et al., 2022) teaches language models
how to solve a complex problem by reducing it to a series of easier sub-
problems, achieving easy-to-hard generalization on semantic parsing tasks like
SCAN (Lake & Baroni, 2018) and CFQ (Keysers et al., 2019) with perfect
generalization accuracy. In addition, least-to-most prompting has also been
successful in mathematical reasoning tasks, specifically in datasets like
GSM8K (Cobbe et al., 2021a) and DROP (Dua et al., 2019), by teaching language
models to solve problems more difficult than those seen in the prompts. This
success not only underscores the capacity of language models to effectively
break down complex tasks into simpler sub-tasks Perez et al. (2020), but also
demonstrates their generalization capability in solving these sub-problems.

Besides the decompose-and-solve paradigm, past work has also evaluated easy-
to-hard generalization by training easy-to-hard generators on easy tasks using
supervised finetune-tuning (SFT) or in-context learning (ICL) (Radford et al.,
2018; Brown et al., 2020). Nevertheless, Swayamdipta et al. (2020) showed that
the BERT model performs poorly on common-sense reasoning when only trained on
easy data. Fu et al. (2022) showed similar results for ICL on reasoning tasks
like GSM8K (Cobbe et al., 2021b). In concurrent work, Hase et al. (2024)
evaluate the performance of easy-to-hard generators on more datasets and
models, and find that ICL or SFT on easy tasks is a strong baseline for
multiple-choice tasks like ARC (Clark et al., 2018) and MMLU (Hendrycks et
al., 2020). In contrast, we evaluate the easy-to-hard generation performance
on the more challenging MATH dataset (Hendrycks et al., 2021), and show that
easy-to-hard evaluation can improve a generator’s easy-to-hard generalization
beyond ICL and SFT. Iterative machine teaching (Liu et al., 2017) gives
theoretical justification to show that training classifiers from easy to hard
examples yield better generalization.

###  2.4 Process Reward Models

For some multi-step complex reasoning tasks, such as generating highly complex
code, it may be challenging for humans to fully grasp the outputs produced by
an advanced AI system. In such scenarios, process-supervised reward models
(PRMs) present a promising solution (Uesato et al., 2022; Lightman et al.,
2023). These models operate by supervising each step in the reasoning or
generation process, rather than focusing solely on the end result. They are
particularly effective in tasks where the reasoning process itself is as
important as the final outcome (Hendrycks et al., 2021; Jimenez et al., 2023).

Uesato et al. (2022) find that process-supervised reward models (PRMs) achieve
better performance than outcome-supervised reward models (ORMs) when re-
ranking sampled solutions from the policy model, but their performance is
similar during reinforcement learning (RL) via expert iteration (Silver et
al., 2017; Anthony et al., 2017; Polu & Sutskever, 2020; Zelikman et al.,
2022; Gulcehre et al., 2023; Singh et al., 2023). Lightman et al. (2023)
compare ORMs and PRMs with a more capable base model (OpenAI, 2023a) and
significantly more human-labeled process feedback on the more challenging MATH
dataset, and also find that PRMs significantly outperform ORMs in the
reranking setting. In contrast to these works, which only study the
effectivenss of PRM in an independent and identically distributed (IID)
domain, we study the utilization of PRMs in the easy-to-hard generalization
scenario, and show that easy-to-hard evaluators instantiated by PRMs can
enable easy-to-hard generation of policy models.

Figure 3: The overview diagram of our methods: the different components of
modeling and training and how they are interconnected.

##  3 Methodology

We study the easy-to-hard generalization problem: how can we enable
capabilities beyond human supervision? Specifically, we explore the efficacy
and scalability of various easy-to-hard methodologies on competition-level
mathematical problem-solving problems (MATH; Hendrycks et al. 2021). This
dataset is suitable for our study since it explicitly categorizes problems
across five difficulty levels. We consider levels 1-3 as “easy” tasks,
encompassing both the problems and their respective solution demonstrations,
along with the correct answers. Conversely, levels 4-5, characterized by their
more complex nature, are treated as “hard” tasks and are represented solely by
their questions. The MATH dataset’s difficulty distribution roughly follows a
$1:2:2:3:3$ ratio across levels 1 to 5. So our division maintains a balanced
number of easy and hard tasks.

The remainder of the paper aims to answer following research questions:

RQ1: How do generators generalize from easy to hard?

RQ2: How do evaluators generalize from easy to hard?

RQ3: If evaluators generalize better than generators, how can we take
advantage of this to enable stronger easy-to-hard generalization in
generators?

###  3.1 Setup

Dataset MATH (Hendrycks et al., 2021) is a dataset of 12,500 challenging
competition mathematics problems, where 7,500 of them are training problems
and 5,000 are originally used for testing. Following Lightman et al. (2023);
Wang et al. (2023), we use the identical subset of 500 representative problems
(i.e., MATH500) as our test set, uniformly sample another 500 problems for
validation, and leave the rest 4,000 MATH test split problems combined with
the original 7,500 MATH training split problems as our training set.

Simulated Human Demonstrations While the original MATH dataset provides full
step-by-step solutions, these solutions typically skip many chain-of-thought
steps (Wei et al., 2022), which can be hard for language models to directly
imitate222Hendrycks et al. (2021) found that having models generate MATH-style
step-by-step solutions before producing an answer actually decreased
accuracy.. Instead, we consider filtered PRM800K (Lightman et al., 2023) and
MetaMATH (Yu et al., 2023b) as our SFT training data: the former is generated
by a Minerva-style base GPT-4 model using few-shot prompting after filtering
the correct answers (Lewkowycz et al., 2022; OpenAI, 2023a), while the latter
is generated by ChatGPT (OpenAI, 2022). PRM800K comes with human annotated
process labels, while for MetaMath, we use Math-Shepherd as the corresponding
process labels (Wang et al., 2023).

###  3.2 Generators

For a given dataset (e.g., a variant of MATH), we consider the following
generator models:

Full & Hard ICL Full in-context learning (ICL) is a base model prompted with
exemplars sampled from all difficulty levels, or only from the level 5 (Fu et
al., 2022).

Easy-to-Hard ICL This model is prompted with exemplars from easy problems.
This baseline evaluates the degree to which a model can solve problems more
difficult than those seen in the prompts (Zhou et al., 2022a).

Full SFT As prior work suggests that finetuning should outperform prompting
alone (Stiennon et al., 2020; Perez et al., 2021; Ouyang et al., 2022), the
full supervised fine-tuning (SFT) model is typically considered as a ceiling
that a model can achieve on a type of task (Burns et al., 2023; Hase et al.,
2024).

Easy-to-Hard SFT This generator model is trained only on the easy tasks. Prior
work suggests that it can generalize to hard tasks but with some degeneration
in performance (Swayamdipta et al., 2020).

The generator models are evaluated in greedy decoding and self-consistency
(also known as majority voting) settings (Wang et al., 2022).

###  3.3 Evaluators

Similarly, we consider the following evaluator models that can be trained
either on the easy tasks only, or on the full dataset. Notably, unlike final-
answer rewards, reward models trained on easy tasks can be applied to evaluate
solutions to hard problems.

Final-Answer Reward is a symbolic reward that provides a binary reward based
on the accuracy of the model’s final answer. The matching is performed after
normalization333https://github.com/openai/prm800k/blob/main/prm800k/grading/grader.py.

Outcome Reward Model (ORM) is trained on the Final-Answer rewards. Following
Cobbe et al. (2021a); Uesato et al. (2022); Lightman et al. (2023), we train
the reward head to predict on every token whether the solution is correct, in
a similar sense to a value model (Yu et al., 2023a). At inference time, we use
the ORM’s prediction at the final token as the reward of the solution.

Process Reward Model (PRM) is trained to predict whether each step (delimited
by newlines) in the chain-of-thought reasoning path is correct. The labels are
usually labeled by humans (Uesato et al., 2022; Lightman et al., 2023) or
estimated with rollouts (Silver et al., 2016; Wang et al., 2023).

Outcome & Process Reward Model (OPRM) Building on the distinct advantages of
ORMs and PRMs, we introduce the Outcome & Process Reward Model (OPRM), which
harnesses the complementary strengths of both. OPRM is trained on the mixed
data of ORMs and PRMs. Specifically, OPRM is designed to evaluate the
correctness of each intermediate reasoning step, akin to PRMs, while also
assessing the overall solution’s accuracy at the final answer stage, mirroring
the functionality of ORMs.

###  3.4 Optimizing Generators Against Evaluators

Finally, given a generator model (i.e., policy model) and a evaluator model
(i.e., reward model; RM), we optimize the generator against the evaluator
using either re-ranking or reinforcement learning.

Best-of-$n$ (BoN), also known as rejection sampling, is a reranking approach
that sample multiple solutions from the generator and selects one with the
highest RM score.

Weighted Voting is similar to majority voting or self-consistency (Wang et
al., 2022), but weights each solution according to its RM score (Uesato et
al., 2022).

Reinforcement Learning (RL) We consider three online/offline RL variants,
Reinforced Self-Training (ReST) (Gulcehre et al., 2023; Singh et al., 2023),
Direct Policy Optimization (DPO) (Rafailov et al., 2023), and Proximal Policy
Optimization (PPO) (Schulman et al., 2017). Due to the space limit, please
find their detailed description in Appendix A.

###  3.5 Evaluation Metrics

In this study, we have chosen not to establish terms analogous to the weak-to-
strong performance gap recovery (PGR) as discussed in Burns et al. (2023) or
the easy-to-hard supervision gap recovery (SGR) highlighted by Hase et al.
(2024). This decision is based on our observations that sometimes, models
trained exclusively on simpler tasks—particularly when employing RL
training—can outperform those trained across the entire spectrum of problem
difficulties. Therefore, we mainly focus on the absolute and relative
performance of generators (optionally optimized by the evaluator) on the
MATH500 test set (Lightman et al., 2023).

###  3.6 Implementation Details

Base Language Model Llemma is a large language model for mathematics
(Azerbayev et al., 2023), which is continue pre-trained from Code Llama
(Roziere et al., 2023) / LlaMA-2 (Touvron et al., 2023). We use both 7b and
34b variants in our experiments.

SFT / RL / Reward Model We fine-tune all models in full fine-tuning with
frozen input-output embedding layers and normalization layers. RMs are
initialized from the base model, and have an added scalar head to output the
reward. In PPO training, we initialize the value model from the reward model.

Hyper-parameters Due to the space limit, our training hyper-parameters can be
found in Appendix. B.

Table 1:  Easy-to-hard generalization of generators. We compare generator performance under various decoding settings. PRM800K and MetaMath indicate the SFT training data and ICL exemplars. Evaluations are performed on the same MATH500 test set.  |  | PRM800K | MetaMath
---|---|---|---
|  | Greedy | Maj@16 | Maj@256 | Greedy | Maj@16 | Maj@256
Llemma-7b | Full ICL | 12.8 | 15.6 | 20.8 | 16.4 | 18.4 | 25.6
Hard ICL | 12.6 | 18.0 | 27.0 | 16.6 | 19.0 | 27.0
Easy-to-Hard ICL | 14.0 | 17.6 | 24.4 | 14.2 | 17.4 | 26.8
Full SFT | 20.6 | 32.0 | 36.2 | 31.4 | 40.2 | 41.6
Easy-to-Hard SFT | 19.8 | 31.6 | 36.0 | 30.0 | 38.6 | 42.4
Llemma-34b | Full ICL | 18.6 | 23.6 | 36.0 | 20.6 | 28.8 | 39.2
Hard ICL | 15.8 | 21.4 | 34.2 | 21.8 | 26.4 | 38.6
Easy-to-Hard ICL | 18.2 | 25.2 | 36.8 | 19.8 | 26.8 | 37.2
Full SFT | 25.6 | 41.8 | 46.4 | 35.4 | 44.2 | 45.6
Easy-to-Hard SFT | 24.8 | 40.8 | 46.0 | 32.2 | 42.6 | 43.4

Figure 4: Easy-to-hard generalization of 7b (upper) and 34b (lower)
evaluators. Both SFTs and RMs are trained on the easy data. We found that PRMs
trained on easy tasks can significantly improve the re-ranking (i.e., weighted
voting) performance on hard tasks.

Figure 5: Easy-to-hard generalization of evaluators applied to generators of
different sizes. We evaluated 7b generator + 34b evaluator (upper) and 34b
generator + 7b evaluator (lower). Both SFTs and RMs are trained on the easy
data.

##  4 Main Results

Table 2:  Comparing reinforcement learning (RL) approaches for easy-to-hard generalization. All methods are of 7b size and evaluated with greedy decoding. † indicates the model is trained with additional final-answer labels on hard tasks (similar to Singh et al. (2023)), which is not strictly a easy-to-hard generalization setup.  | RL Data | Reward | Accuracy
---|---|---|---
| Final-Answer | Process RM | Easy (level 1-3) | Hard (level 4-5) | All
(SFT / PRM trained on level 1-3 of PRM800K) |
SFT |  |  |  | 28.2 | 12.2 | 19.8
ReST-EM | Easy | Easy | $\times$ | 33.2 | 12.6 | 22.4
ReST-EM | Hard | Hard | $\times$ | 31.9 | 8.0 | 19.4
ReST-EM† | All | All | $\times$ | 35.7 | 8.8 | 21.6
Iterative DPO | Easy | Easy | $\surd$ | 42.0 | 12.2 | 26.4
Iterative DPO† | All | All | $\surd$ | 38.2 | 11.5 | 24.2
PPO | Easy | Easy | $\times$ | 42.0 | 14.1 | 27.4
PPO | Hard | Hard | $\times$ | 34.0 | 9.2 | 21.0
PPO† | All | All | $\times$ | 42.0 | 10.7 | 25.6
PPO | All | Easy | $\surd$ | 45.4 | 14.9 | 29.4
(SFT / PRM trained on level 1-5 of MetaMath / Math-Shepherd) |
Llemma-based SFT SoTA (Ours) | 51.7 | 13.7 | 31.4
Previous RL SoTA (Wang et al., 2023) | - | - | 33.0
(SFT / PRM trained on level 1-3 of MetaMath / Math-Shepherd) |
SFT |  |  |  | 44.1 | 14.9 | 28.8
ReST-EM | Easy | Easy | $\times$ | 50.4 | 14.5 | 31.6
Iterative DPO | Easy | Easy | $\surd$ | 53.8 | 16.0 | 34.0
Iterative DPO† | All | All | $\surd$ | 47.9 | 12.2 | 29.2
PPO | Easy | Easy | $\times$ | 50.8 | 15.3 | 32.2
PPO† | All | All | $\times$ | 50.8 | 13.4 | 31.2
PPO | All | Easy | $\surd$ | 53.8 | 16.0 | 34.0

###  4.1 Easy-to-Hard Generalization of Generators

In Table 1, we compare the easy-to-hard generalization performance of the
generators under various decoding settings:

##### Supervised Fine-Tuning (SFT) outperforms In-Context Learning (ICL):

This is consistent with prior work (Stiennon et al., 2020; Ouyang et al.,
2022; Uesato et al., 2022). We also find that the performance of ICL has
larger variance than SFT with respect to data ordering (or random seeds)
(Dodge et al., 2020; Zhao et al., 2021).

##### SFT data quality impacts easy-to-hard generalization:

PRM800K data is generated by a base (unaligned) GPT-4 model through few-shot
prompting and is thus of lower quality than well-aligned ChatGPT-generated
MetaMATH data. We find that only MetaMath-trained models have certain easy-to-
hard gaps (e.g., 16.6 v.s. 14.2 in MetaMath-7b-ICL), while such gaps in
PRM800K-trained models are very small (less than 1%), or even inverted in the
ICL setting. We hypothesize that low-quality SFT data may only teach the model
the format of the task (Sanh et al., 2021; Wei et al., 2021; Wang et al.,
2022), while high-quality (imitation) SFT data can teach the model the
principles of solving the task (Sun et al., 2023b; Gudibande et al., 2023).
Nevertheless, the strongest performance is achieved by full SFT on the high-
quality MetaMath data (35.4), showing an unignorable difference, with a gap of
up to 3.2, compared to its easy-to-hard SFT counterpart (32.2).

###  4.2 Easy-to-Hard Generalization of Evaluators

The primary metric we use to assess the effectiveness of our process reward
model is not the average accuracy of verifying each step in a solution but
rather the overall performance achieved through re-ranking methods (See
discussion in Sec. 3.5). We first use re-ranking to evaluate the easy-to-hard
generalization performance of evaluators.

####  4.2.1 Re-ranking

We consider two re-ranking strategies: Best-of-$n$ (or rejection sampling) and
Weighted Voting. In our easy-to-hard generalization setting, both SFT models
and Reward Models (RMs) are trained on easier tasks (levels 1-3), but
evaluated on all difficulty levels (1-5). We compare the performance between
majority voting (SFT only) and re-ranking (SFT + OPRM) on the PRM800K dataset
in Figure 4-5, and the performance of different reward models (PRMs, ORMs, &
OPRMs) on the PRM800K dataset in Figure 8-9. Specifically, we use min as the
reward aggregation function for best-of-$n$ and prod for weighted voting444See
more detailed analysis of reward aggregation functions in Appendix. G.. The
figures illustrate the performance of different decoding strategies or reward
models under the same number of sampled solutions per problem. We have the
following findings:

##### OPRMs outperforms ORMs and PRMs

This confirms our hypothesis that Process Reward Models (PRMs) and Outcome
Reward Models (ORMs) capture different aspects of task-solving processes. By
integrating the strengths of both PRMs and ORMs, Outcome & Process Reward
Models (OPRMs) demonstrate superior performance. However, follow-up
experiments conducted on the MetaMath/Math-Shepherd datasets do not
demonstrate significant improvements from incorporating additional ORM
training examples. This lack of enhancement may be attributed to the fact that
Math-Shepherd is already generated from final-answer reward generation. This
suggests that there remains a substantial difference between process rewards
labeled by humans (e.g., PRM800K) and those generated automatically (e.g.,
Math-Shepherd).

##### Weighted voting outshines Best-of-$n$

This finding diverges from past research where minimal performance differences
were observed between weighted voting and Best-of-$n$ (Lightman et al., 2023;
Uesato et al., 2022). Our hypothesis is that this discrepancy arises from our
specific experiment, which involves training a less powerful base model
(Llemma; Azerbayev et al. 2023) on more difficult tasks (MATH; Hendrycks et
al. 2021). This setup might diminish the effectiveness of the reward model,
potentially leading to an over-optimization of rewards (Gao et al., 2023).
Given these insights, weighted voting is preferred as the primary re-ranking
method for further discussions. Nevertheless, Best-of-$n$ still achieves
competitive performance to majority voting when producing only one full
solution. In Figure 5, we also find that the 34b evaluator can significantly
improve the 7b generator, while the 7b evaluator can still improve the
performance of the 34b generator.

##### Greater effectiveness of re-ranking on harder tasks:

Weighted voting not only consistently surpasses majority voting but also shows
a more pronounced advantage on harder tasks. This observation leads to the
conclusion that evaluators demonstrate better easy-to-hard generalization
capabilities in comparison to generators. This motivates us to explore RL
approaches that optimize the generator against the evaluator to further
improve the performance of easy-to-hard generation.

####  4.2.2 Reinforcement Learning (RL)

Given the conclusion above, an important question arises: how can evaluators
once again assist generators in achieving enhanced easy-to-hard generalization
capabilities? We further investigate the enhancement of policy models through
RL, utilizing easy-to-hard evaluators as reward models. Similar to re-ranking,
SFT and PRM are only trained on easy data. For a fair comparison between
PRM800K and MetaMath, we only use vanilla PRMs in the RL training. All the RL
methods use the validation accuracy for selecting the best checkpoint555This
includes stopping iterations in ReST-EM and iterative DPO, and stopping online
steps in PPO.. Our comparison spans offline (ReST & DPO) and online (PPO) RL
algorithms under various training conditions:

Easy Questions & Easy Final Answers: The SFT model samples from easy questions
and receives the corresponding Final-Answer and optional PRM rewards.

All Questions & Easy Final Answers: This assumes access to a range of easy and
hard problems for RL training, with rewards for hard tasks solely provided by
the easy-to-hard evaluator.

All Questions & All Final Answers: This setting uses all data with the
corresponding final answers, which is similar to Singh et al. (2023), but not
strictly a easy-to-hard generalization setup.

Based on the results reported in Table 2, we have the following findings:

##### DPO and PPO excel over ReST:

Among the RL algorithms trained on the PRM800K dataset, PPO emerges as the
most effective, significantly surpassing both ReST and DPO. On the MetaMATH
dataset, PPO and DPO achieve top performance, while ReST shows only marginal
improvements over the SFT baseline. The comparative analysis between DPO and
PPO across the PRM800K and MetaMATH datasets indicates that while DPO’s
efficacy is on par with PPO given a high-quality SFT model as initialization,
PPO’s effectiveness is less contingent on the quality of the underlying SFT
model (Ouyang et al., 2022; Rafailov et al., 2023).

##### PRM rewards are more beneficial than Final-Answer rewards for hard
tasks:

Notably, models trained with PRM rewards with human supervision on the easy
tasks (achieving a top performance of 34.0) outperform the previous state-of-
the-art model trained across all task levels (33.0). This highlights the
effectiveness of leveraging easy-to-hard evaluations to improve generator
performance across varying task difficulties.

##  5 Conclusion

Our study advances the field of AI alignment by demonstrating the potential of
easy-to-hard generalization, where models trained on simpler tasks can be
guided to solve more complex problems without direct human supervision on
these harder tasks. Through the use of process-supervised reward models (PRMs)
for evaluating and enhancing policy models, we show that evaluators can
significantly facilitate this form of generalization, outperforming
traditional training methods. Our findings highlight the effectiveness of re-
ranking strategies and reinforcement learning (RL) in leveraging evaluators
for substantial performance gains on difficult tasks. This approach presents a
promising direction for developing AI systems capable of surpassing human
problem-solving capabilities, suggesting a scalable alignment method that
could enable AI to independently advance knowledge in complex domains.

## References

  * Amodei et al. (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D.  Concrete problems in ai safety.  _arXiv preprint arXiv:1606.06565_ , 2016.
  * An et al. (2023) An, S., Ma, Z., Lin, Z., Zheng, N., Lou, J.-G., and Chen, W.  Learning from mistakes makes llm better reasoner.  _arXiv preprint arXiv:2310.20689_ , 2023.
  * Andreas (2022) Andreas, J.  Language models as agent models.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 5769–5779, 2022.
  * Anthony et al. (2017) Anthony, T., Tian, Z., and Barber, D.  Thinking fast and slow with deep learning and tree search.  _Advances in neural information processing systems_ , 30, 2017.
  * Azar et al. (2023) Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R.  A general theoretical paradigm to understand learning from human preferences.  _arXiv preprint arXiv:2310.12036_ , 2023.
  * Azerbayev et al. (2023) Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S.  Llemma: An open language model for mathematics.  _arXiv preprint arXiv:2310.10631_ , 2023.
  * Bai et al. (2022a) Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J.  Constitutional ai: Harmlessness from ai feedback, 2022b.
  * Bowman et al. (2022) Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukosuite, K., Askell, A., Jones, A., Chen, A., et al.  Measuring progress on scalable oversight for large language models.  _arXiv preprint arXiv:2211.03540_ , 2022.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ , 33:1877–1901, 2020.
  * Burns et al. (2023) Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al.  Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.  _arXiv preprint arXiv:2312.09390_ , 2023.
  * Carlson et al. (2006) Carlson, J. A., Jaffe, A., and Wiles, A.  _The millennium prize problems_.  American Mathematical Soc., 2006.
  * Chomsky (1981) Chomsky, N.  On the representation of form and function.  1981\.
  * Chung et al. (2022) Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Clark et al. (2018) Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_ , 2018.
  * Cobbe et al. (2021a) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021a.
  * Cobbe et al. (2021b) Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021b.
  * Cotra (2021) Cotra, A.  The case for aligning narrowly superhuman models.  In _AI Alignment Forum_ , 2021.
  * Dodge et al. (2020) Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2002.06305_ , 2020.
  * Drozdov et al. (2022) Drozdov, A., Schärli, N., Akyürek, E., Scales, N., Song, X., Chen, X., Bousquet, O., and Zhou, D.  Compositional semantic parsing with large language models.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Dua et al. (2019) Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 2368–2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1246.  URL https://aclanthology.org/N19-1246.
  * Dubois et al. (2023) Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv preprint arXiv:2305.14387_ , 2023.
  * Fodor & Lepore (2002) Fodor, J. A. and Lepore, E.  _The compositionality papers_.  Oxford University Press, 2002.
  * Fu et al. (2022) Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T.  Complexity-based prompting for multi-step reasoning.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Gao et al. (2023) Gao, L., Schulman, J., and Hilton, J.  Scaling laws for reward model overoptimization.  In _International Conference on Machine Learning_ , pp. 10835–10866. PMLR, 2023.
  * Gou et al. (2023) Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., et al.  Tora: A tool-integrated reasoning agent for mathematical problem solving.  _arXiv preprint arXiv:2309.17452_ , 2023.
  * Gudibande et al. (2023) Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D.  The false promise of imitating proprietary llms.  _arXiv preprint arXiv:2305.15717_ , 2023.
  * Gulcehre et al. (2023) Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al.  Reinforced self-training (rest) for language modeling.  _arXiv preprint arXiv:2308.08998_ , 2023.
  * Hase et al. (2024) Hase, P., Bansal, M., Clark, P., and Wiegreffe, S.  The unreasonable effectiveness of easy training data for hard tasks.  _arXiv preprint arXiv:2401.06751_ , 2024.
  * Hendrycks et al. (2020) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_ , 2020.
  * Hendrycks et al. (2021) Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J.  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_ , 2021.
  * Jimenez et al. (2023) Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K.  Swe-bench: Can language models resolve real-world github issues?  _arXiv preprint arXiv:2310.06770_ , 2023.
  * Karp (1975) Karp, R. M.  On the computational complexity of combinatorial problems.  _Networks_ , 5(1):45–68, 1975.
  * Keysers et al. (2019) Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, L., Tihon, T., et al.  Measuring compositional generalization: A comprehensive method on realistic data.  In _International Conference on Learning Representations_ , 2019.
  * Kojima et al. (2022) Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ , 2022.
  * Lake & Baroni (2018) Lake, B. and Baroni, M.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International conference on machine learning_ , pp. 2873–2882. PMLR, 2018.
  * Leike et al. (2018) Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S.  Scalable agent alignment via reward modeling: a research direction.  _arXiv preprint arXiv:1811.07871_ , 2018.
  * Lewkowycz et al. (2022) Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al.  Solving quantitative reasoning problems with language models.  _arXiv preprint arXiv:2206.14858_ , 2022.
  * Lightman et al. (2023) Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K.  Let’s verify step by step.  _arXiv preprint arXiv:2305.20050_ , 2023.
  * Ling et al. (2017) Ling, W., Yogatama, D., Dyer, C., and Blunsom, P.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 158–167, 2017.
  * Liu et al. (2023) Liu, B., Bubeck, S., Eldan, R., Kulkarni, J., Li, Y., Nguyen, A., Ward, R., and Zhang, Y.  Tinygsm: achieving¿ 80% on gsm8k with small language models.  _arXiv preprint arXiv:2312.09241_ , 2023.
  * Liu et al. (2017) Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., Rehg, J. M., and Song, L.  Iterative machine teaching.  In _International Conference on Machine Learning_ , pp. 2149–2158. PMLR, 2017.
  * Luo et al. (2023) Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _arXiv preprint arXiv:2308.09583_ , 2023.
  * Munos et al. (2023) Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al.  Nash learning from human feedback.  _arXiv preprint arXiv:2312.00886_ , 2023.
  * Naor (1996) Naor, M.  Evaluation may be easier than generation.  In _Proceedings of the twenty-eighth annual ACM symposium on Theory of computing_ , pp. 74–83, 1996.
  * OpenAI (2022) OpenAI.  OpenAI: Introducing ChatGPT, 2022.  URL https://openai.com/blog/chatgpt.
  * OpenAI (2023a) OpenAI.  Gpt-4 technical report, 2023a.
  * OpenAI (2023b) OpenAI.  OpenAI: GPT-4, 2023b.  URL https://openai.com/research/gpt-4.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Perez et al. (2020) Perez, E., Lewis, P., Yih, W.-t., Cho, K., and Kiela, D.  Unsupervised question decomposition for question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 8864–8880, 2020.
  * Perez et al. (2021) Perez, E., Kiela, D., and Cho, K.  True few-shot learning with language models.  _Advances in neural information processing systems_ , 34:11054–11070, 2021.
  * Perez et al. (2022) Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al.  Discovering language model behaviors with model-written evaluations.  _arXiv preprint arXiv:2212.09251_ , 2022.
  * Polu & Sutskever (2020) Polu, S. and Sutskever, I.  Generative language modeling for automated theorem proving.  _arXiv preprint arXiv:2009.03393_ , 2020.
  * Radford et al. (2018) Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I.  Improving language understanding by generative pre-training.  2018\.
  * Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Rein et al. (2023) Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R.  Gpqa: A graduate-level google-proof q&a benchmark.  _arXiv preprint arXiv:2311.12022_ , 2023.
  * Roziere et al. (2023) Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ , 2023.
  * Sanh et al. (2021) Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., et al.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_ , 2021.
  * Saunders et al. (2022) Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J.  Self-critiquing models for assisting human evaluators.  _arXiv preprint arXiv:2206.05802_ , 2022.
  * Schulman et al. (2015) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ , 2015.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Schwarzschild et al. (2021) Schwarzschild, A., Borgnia, E., Gupta, A., Huang, F., Vishkin, U., Goldblum, M., and Goldstein, T.  Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks.  _Advances in Neural Information Processing Systems_ , 34:6695–6706, 2021.
  * Sharma et al. (2023) Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., et al.  Towards understanding sycophancy in language models.  _arXiv preprint arXiv:2310.13548_ , 2023.
  * Silver et al. (2016) Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, 2016.
  * Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Singh et al. (2023) Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al.  Beyond human data: Scaling self-training for problem-solving with language models.  _arXiv preprint arXiv:2312.06585_ , 2023.
  * Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Sun et al. (2023a) Sun, Z., Shen, Y., Zhang, H., Zhou, Q., Chen, Z., Cox, D., Yang, Y., and Gan, C.  Salmon: Self-alignment with principle-following reward models.  _arXiv preprint arXiv:2310.05910_ , 2023a.
  * Sun et al. (2023b) Sun, Z., Shen, Y., Zhou, Q., Zhang, H., Chen, Z., Cox, D., Yang, Y., and Gan, C.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  _arXiv preprint arXiv:2305.03047_ , 2023b.
  * Swayamdipta et al. (2020) Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., and Choi, Y.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 9275–9293, 2020.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Trinh et al. (2024) Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T.  Solving olympiad geometry without human demonstrations.  _Nature_ , 625(7995):476–482, 2024.
  * Uesato et al. (2022) Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I.  Solving math word problems with process- and outcome-based feedback.  _arXiv preprint arXiv:2211.14275_ , 2022.
  * Wang et al. (2023) Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z.  Math-shepherd: Verify and reinforce llms step-by-step without human annotations.  _CoRR, abs/2312.08935_ , 2023.
  * Wang et al. (2022) Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_ , 2022.
  * Wang et al. (2024) Wang, Z., Li, Y., Wu, Y., Luo, L., Hou, L., Yu, H., and Shang, J.  Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision.  _arXiv preprint arXiv:2402.02658_ , 2024.
  * Wei et al. (2021) Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2021.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D.  Chain-of-thought prompting elicits reasoning in large language models.  _NeurIPS_ , 2022.
  * Wei et al. (2023) Wei, J., Huang, D., Lu, Y., Zhou, D., and Le, Q. V.  Simple synthetic data reduces sycophancy in large language models.  _arXiv preprint arXiv:2308.03958_ , 2023.
  * Wu et al. (2021) Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., and Christiano, P.  Recursively summarizing books with human feedback.  _arXiv preprint arXiv:2109.10862_ , 2021.
  * Xu et al. (2023) Xu, J., Lee, A., Sukhbaatar, S., and Weston, J.  Some things are more cringe than others: Preference optimization with the pairwise cringe loss.  _arXiv preprint arXiv:2312.16682_ , 2023.
  * Yao et al. (2022) Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022.
  * Yao et al. (2023) Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_ , 2023.
  * Yu et al. (2023a) Yu, F., Gao, A., and Wang, B.  Outcome-supervised verifiers for planning in mathematical reasoning.  _arXiv preprint arXiv:2311.09724_ , 2023a.
  * Yu et al. (2023b) Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W.  Metamath: Bootstrap your own mathematical questions for large language models.  _arXiv preprint arXiv:2309.12284_ , 2023b.
  * Yuan et al. (2023) Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C.  Scaling relationship on learning mathematical reasoning with large language models.  _arXiv preprint arXiv:2308.01825_ , 2023.
  * Yue et al. (2023) Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W.  Mammoth: Building math generalist models through hybrid instruction tuning.  _arXiv preprint arXiv:2309.05653_ , 2023.
  * Zelikman et al. (2022) Zelikman, E., Wu, Y., Mu, J., and Goodman, N.  Star: Bootstrapping reasoning with reasoning.  _Advances in Neural Information Processing Systems_ , 35:15476–15488, 2022.
  * Zhang et al. (2022) Zhang, Z., Zhang, A., Li, M., and Smola, A.  Automatic chain of thought prompting in large language models.  _arXiv preprint arXiv:2210.03493_ , 2022.
  * Zhao et al. (2022) Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M., and Liu, P. J.  Calibrating sequence likelihood improves conditional language generation.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhao et al. (2023) Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.
  * Zhao et al. (2021) Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pp. 12697–12706. PMLR, 2021.
  * Zheng et al. (2023) Zheng, C., Liu, Z., Xie, E., Li, Z., and Li, Y.  Progressive-hint prompting improves reasoning in large language models.  _arXiv preprint arXiv:2304.09797_ , 2023.
  * Zhou et al. (2022a) Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., et al.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_ , 2022a.
  * Zhou et al. (2022b) Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_ , 2022b.
  * Ziegler et al. (2019) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

##  Appendix A Reinforcement Learning Algorithms

Reinforced Self-Training (ReST) is an offline RL algorithm, which alternates
between generating samples from the policy, which are then used to improve the
LLM policy with RM-weighted SFT (Gulcehre et al., 2023; Singh et al., 2023).
Its variants include expert iteration (Anthony et al., 2017) and rejection
sampling fine-tuning (Touvron et al., 2023; Yuan et al., 2023).

Direct Policy Optimization (DPO) is a class of offline RL algorithms (Rafailov
et al., 2023) that consider both positive and negative gradient updates. It
fine-tunes the policy model on a preference dataset consisting of paired
positive and negative samples. The variants include NLHF (Munos et al., 2023),
IPO (Azar et al., 2023), and SLiC (Zhao et al., 2022, 2023). Recent work shows
that iteratively applying DPO leads to improved performance (Xu et al., 2023).

Proximal Policy Optimization (PPO) is an online RL algorithm which samples
from the policy during fine-tuning (Schulman et al., 2017). It is widely used
in RLHF (Stiennon et al., 2020; Bai et al., 2022a; Ouyang et al., 2022) and
RLAIF (Bai et al., 2022b; Sun et al., 2023a).

##  Appendix B Hyper-parameters

###  B.1 Supervised Fine-Tuning & Reward Modeling

For the PRM800K dataset (Lightman et al., 2023), the SFT model is trained
using steps that are labeled as correct. For the MetaMath dataset (Yu et al.,
2023b), given that the original dataset can contain upwards of ten solutions
for the same question, potentially leading to over-fitting, we implement a
filtering process. This process ensures that, during any given epoch, no more
than three solutions per question are retained, thereby mitigating the risk of
over-fitting.

The PRMs are trained on the corresponding released dataset (Lightman et al.,
2023; Wang et al., 2023). For generating solutions to train ORMs, we sample 32
solutions for each question from the language model using top-K sampling with
K=20 and temperature of 0.7. We also ensure that the ratio between positive
and negative samples for each question is between 1:3 to 3:1.

See Table 3 for a list of training hyper-parameters used in the training jobs.
We use full fine-tuning for all SFT/RM training.

Table 3:  Hyper-parameters in our SFT/RM training jobs  |  | PRM800K |  | MetaMath
---|---|---|---|---
|  | SFT | PRM | ORM | OPRM |  | SFT | PRM
Llemma-7b | Learning Rate | 2e-5 | 2e-5 | 2e-5 | 2e-5 |  | 8e-6 | 2e-5
Epochs | 3 | 2 | 2 | 2 |  | 3 | 2
Batch Size | 128 | 128 | 128 | 128 |  | 128 | 128
Max Seq Len | 768 | 768 | 1024 | 1024 |  | 1024 | 768
Dtype | BF16 | BF16 | BF16 | BF16 |  | FP32 | BF16
Llemma-34b | Learning Rate | 1e-5 | 1e-5 | 1e-5 | 1e-5 |  | 5e-6 | -
Epochs | 3 | 2 | 2 | 2 |  | 3 | -
Batch Size | 128 | 128 | 128 | 128 |  | 128 | -
Max Seq Len | 768 | 768 | 1024 | 1024 |  | 768 | -
Dtype | BF16 | BF16 | BF16 | BF16 |  | FP32 | -

###  B.2 Re-Ranking

For majority voting, weighted voting, and best-of-$n$, we sample from the
language model using top-K sampling with K=20 and temperature of 0.7. At test
time, we use the ORM’s prediction at the final token as the overall score for
the solution, and use the PRM’s prediction at each intermediate step (denoted
by the new line symbol) and the final token as the process reward scores.

###  B.3 Reinforcement Learning

We use full fine-tuning during the RL stage.

##### ReST-EM

Following Singh et al. (2023), we sample 32 solutions for each question from
the language model using top-K sampling with K=40. We also used a cut-off
threshold of 10 for the maximum number of solutions per problem (Zelikman et
al., 2022; Singh et al., 2023). We performed iterative ReST training for two
epochs, and observed performance degeneration starting from the third epoch.
For PRM800K, we used a temperature of 1.0, while for MetaMath, we used a
temperature of 1.2. The rest training hyper-parameters are the same as in SFT
training.

##### Iterative DPO

We sample 8 solutions for each question from the language model using top-K
sampling with K=20 and temperature of 1.0. We use the process reward model to
assign a score between 0 and 1 to each solution, and use final-answer reward
to assign an additional 0/1 score to each solution. A preference training pair
is constructed only when the score difference between positive and negative
solutions is greater than 1.0. We used a cut-off threshold of 3 for the
maximum number of preference pairs per problem.

For all DPO training (Rafailov et al., 2023), we used a learning rate of
$2\times 10^{-6}$, a batch size of 64, and a DPO training epoch of 1. We set
$\beta=0.1$ for all DPO experiments, and performed at most 5 DPO iterations
(i.e., sampling new solutions and performing one DPO epoch).

##### PPO

We follow Dubois et al. (2023) on the implementation of the PPO algorithm,
which is a variant of (Ouyang et al., 2022)666https://github.com/openai/lm-
human-preferences. Specifically, we normalize the advantage across the entire
batch of rollouts obtained for each PPO step and initialize the value model
from the reward model.

We clipped the gradient by its Euclidean norm at a limit of $1$. Our training
spanned $500$ PPO steps on the RL data (MATH questions except MATH500 and our
500 validation questions). For generalized advantage estimation (GAE; Schulman
et al. (2015)), both $\lambda$ and $\gamma$ were set at 1.

For PRM800K, we used a batch size of 512 for each PPO step. This comprised 8
epochs of gradient steps, each having 64 rollouts. We applied a peak learning
rate of $2\times 10^{-5}$ with cosine decay. We opted for a constant KL
regularizer coefficient of $0.01$, and a sampling temperature of $0.7$.

For MetaMath/Math-Shepherd, we used a batch size of 512 for each PPO step.
This comprised 2 epochs of gradient steps, each having 256 rollouts. We
applied a peak learning rate of $5\times 10^{-6}$ with cosine decay. We opted
for a constant KL regularizer coefficient of $0.002$, and a sampling
temperature of $1.2$.

##  Appendix C Re-ranking Results on MetaMath

Similar to Sec. 4.2.1, we assess the effectiveness of process reward models on
the MetaMath/Math-Shepherd dataset (Yu et al., 2023b; Wang et al., 2023). From
Figure 6, we can see that PRMs are also more effective on harder tasks when
trained on MetaMath/Math-Shepherd.

Figure 6: Easy-to-hard generalization of evaluators on MetaMath. Both SFTs and
RMs are trained on the easy data.

##  Appendix D Qualitative Comparison

Question |  The value of $y$ varies inversely as $\sqrt{x}$ and when $x=24$, $y=15$. What is $x$ when $y=3$?
---|---
MetaMath-SFT |  If $y$ varies inversely as $\sqrt{x}$, then $y=\frac{k}{\sqrt{x}}$ for some constant $k$. Plugging in the given values, we have $15=\frac{k}{\sqrt{24}}$, or $k=\frac{15\sqrt{24}}{24}=\frac{15\sqrt{6}}{4}$. Now we can find $x$ when $y=3$ by plugging in these values into the equation $y=\frac{k}{\sqrt{x}}$. We have $3=\frac{\frac{15\sqrt{6}}{4}}{\sqrt{x}}$, or $3=\frac{15\sqrt{6}}{4\sqrt{x}}$. Cross-multiplying, we get $3\cdot 4\sqrt{x}=15\sqrt{6}$, or $12\sqrt{x}=15\sqrt{6}$. Dividing both sides by $12$, we find $\sqrt{x}=\frac{15\sqrt{6}}{12\sqrt{6}}=\frac{5}{2}$. Squaring both sides, we get $x=\left(\frac{5}{2}\right)^{2}=\boxed{6.25}$.
MetaMath-PPO |  If $y$ varies inversely as $\sqrt{x}$, then we can write the equation as $y=\frac{k}{\sqrt{x}}$ for some constant $k$. We are given that when $x=24$, $y=15$, so we can substitute these values into the equation to solve for $k$: $15=\frac{k}{\sqrt{24}}$ Multiplying both sides by $\sqrt{24}$ gives $15\sqrt{24}=k$ Simplifying the left side gives $15\cdot 2\sqrt{6}=k$, which further simplifies to $30\sqrt{6}=k$ Now we can use this value of $k$ in the equation to find $x$ when $y=3$: $3=\frac{30\sqrt{6}}{\sqrt{x}}$ Multiplying both sides by $\sqrt{x}$ gives $3\sqrt{x}=30\sqrt{6}$ Dividing both sides by $3$ gives $\sqrt{x}=10\sqrt{6}$ Squaring both sides gives $x=(10\sqrt{6})^{2}=100(6)=\boxed{600}$
Table 4: Qualitative comparison between MetaMath-7b-SFT and MetaMath-7b-PPO,
as well as the first error detected in MetaMath-SFT greedy decoding by Math-
Shepherd. Examples are cherry-picked for better illustration.

##  Appendix E Performance of DPO Iterations

Figure 7: Test performance as a function of DPO iterations.

##  Appendix F Re-ranking Performance Analysis of PRMs, ORMs & OPRMs

We compare the re-ranking performance of Process Reward Models (PRMs), Outcome
Reward Models (ORMs), and our proposed Outcome & Process Reward Models
(OPRMs). Figure 8 shows the results on 7b models and Figure 9 is on 34b
models. We find that in our setting of Llemma (Azerbayev et al., 2023) \+ MATH
(Hendrycks et al., 2021), PRMs and ORMs perform similarly, with PRMs slightly
outperforming ORMs on hard tasks. But the OPRMs that trained on the mixed data
of PRMs and ORMs significantly outperforms both of them.

Figure 8: Comparing process reward models (PRMs, left), outcome reward models
(ORMs, middle), and outcome & process reward models (OPRMs, right) on 7b
models trained on the PRM800K dataset. Both SFTs and RMs are trained on the
easy data.

Figure 9: Comparing process reward models (PRMs, left), outcome reward models
(ORMs, middle), and outcome & process reward models (OPRMs, right) on 34b
models trained on the PRM800K dataset. Both SFTs and RMs are trained on the
easy data.

##  Appendix G Analysis of Aggregation Functions in PRMs & OPRMs

We explored different methods to consolidate step-wise prediction scores into
a single score value, a process we describe as employing an aggregation
function, during the use of the evaluator. Lightman et al. (2023) report
comparable performance when using min (minimum) and prod (product) as the
aggregation function to reduce multiple scores into a single solution-level
score. Note that when training PRMs on PRM800K (Lightman et al., 2023), we
have already considered neutral steps to be positive as training labels.

Following Wang et al. (2024), given $\\{p_{1},p_{2},\dots,p_{n}\\}$ as a list
of predicted correctness probability of each step (including the final
answer), we considered the following aggregation functions:

$\mathtt{min}=\min\\{p_{1},p_{2},\dots,p_{n}\\}$ (1)
$\mathtt{max}=\max\\{p_{1},p_{2},\dots,p_{n}\\}$ (2)
$\mathtt{prod}=\prod_{i}p_{i}$ (3) $\mathtt{mean}=\frac{\sum_{i}p_{i}}{n}$ (4)
$\mathtt{mean\\_logit}=\sigma\left(\frac{\sum_{i}\log\frac{p_{i}}{1-p_{i}}}{n}\right)$
(5)
$\mathtt{mean\\_odd}=\mathrm{ReLU}\left(\frac{\sum_{i}\frac{p_{i}}{1-p_{i}}}{n}\right)$
(6) $\mathtt{last}=p_{n}$ (7)

In Figure 10-12, we perform analysis of aggregation functions on PRM800K and
Math-Shepherd (from MetaMath) datasets with weighted voting and best-of-$n$
decoding and PRMs or OPRMs. In general, we find prod works universally well in
weighted voting and min works well in best-of-$n$. So we adopt these two
strategies in our main experiments.

One interesting finding is that for reward models trained on the human
annotated process reward (e.g., PRM800K), the last strategy does not perform
very well, but last works much better on OPRMs and pseudo PRMs (e.g., Math-
Shepherd). This could partially explain why OPRMs does not further improve the
performance on the Math-Shepherd dataset.

Figure 10: Analysis of aggregation functions in process reward models (PRMs)
on the PRM800K dataset with Weighted Voting and Best-of-N. Both SFTs and RMs
are trained on the easy data.

Figure 11: Analysis of aggregation functions in outcome & process reward
models (OPRMs) on the PRM800K dataset with Weighted Voting and Best-of-N. Both
SFTs and RMs are trained on the easy data.

Figure 12: Analysis of aggregation functions in psuedo process reward models
(PRMs) on the Math-Shepherd (from MetaMath) dataset with Weighted Voting and
Best-of-N. Both SFTs and RMs are trained on the easy data.

##  Appendix H Few-Shot Prompt in In-Context Learning

We sample with temperature T = 0.9 for ICL-related experiments. We list our
few-shot sample for the In-context learning experiments in Table 1.

4-shot example for PRM800K Full ICL:

⬇

Problem:

In right triangle $PQR$, we have $\\\angle Q = \\\angle R$ and $PR =
6\\\sqrt{2}$. What is the area of $\\\triangle PQR$?

Solution:

I see that this is an isosceles right triangle, since $\\\angle Q = \\\angle R
= 45^\\\circ$.\n\nThat means that $PQ = PR = 6\\\sqrt{2}$.\n\nTo find the area
of a triangle, I need to multiply the base and the height and divide by
2.\n\nIn this case, the base and the height are both $6\\\sqrt{2}$, so the
area is $(6\\\sqrt{2})(6\\\sqrt{2})/2$.\n\nI can simplify this expression by
using the property that $\\\sqrt{a}\\\sqrt{b} = \\\sqrt{ab}$.\n\nSo, the area
is $(\\\sqrt{2}\\\sqrt{2})(6\\\cdot 6)/2 = 2\\\cdot 36/2 = 36$.

Final Answer: The final answer is $36$. I hope it is correct.

Problem:

The length of the longer side of rectangle $R$ is $10$ percent more than the
length of a side of square $S.$ The length of the shorter side of rectangle
$R$ is $10$ percent less than the length of a side of square $S.$ What is the
ratio of the area of rectangle $R$ to the area of square $S?$ Express your
answer as a common fraction.

Solution:

Let the side of the square be $s$.\n\nThen the longer side of the rectangle is
$1.1s$.\n\nAnd the shorter side of the rectangle is $0.9s$.\n\nThe area of the
square is $s^2$.\n\nThe area of the rectangle is $1.1s * 0.9s =
0.99s^2$.\n\nSo the ratio of the area of rectangle $R$ to the area of square
$S$ is $\\\frac{0.99s^2}{s^2} = 0.99$.\n\nTherefore, the answer is
$\\\boxed{\\\frac{99}{100}}$.

Final Answer: The final answer is $\\\frac{99}{100}$. I hope it is correct.

Problem:

Compute $\\\sin(-60^\\\circ)$.

Solution:

I know the sine function is periodic with period $360^\\\circ$, so
$\\\sin(-60^\\\circ) = \\\sin(-60^\\\circ + 360^\\\circ) =
\\\sin(300^\\\circ)$.\n\nI also know that the sine function is symmetrical
about the origin, so $\\\sin(-x) = -\\\sin(x)$ for any angle
$x$.\n\nTherefore, $\\\sin(-60^\\\circ) = -\\\sin(60^\\\circ)$.\n\nTo find
$\\\sin(60^\\\circ)$, I can use the special right triangle with angles
$30^\\\circ$, $60^\\\circ$, and $90^\\\circ$ and sides in the ratio
$1:\\\sqrt{3}:2$.\n\nThe sine of an angle is the ratio of the opposite side to
the hypotenuse, so $\\\sin(60^\\\circ) =
\\\frac{\\\sqrt{3}}{2}$.\n\nTherefore, $\\\sin(-60^\\\circ) =
-\\\frac{\\\sqrt{3}}{2}$.

Final Answer: The final answer is $-\\\frac{\\\sqrt{3}}{2}$. I hope it is
correct.

Problem:

Simplify $\\\sin (x - y) \\\cos y + \\\cos (x - y) \\\sin y.$

Solution:

’Ok, so we have $\\\sin (x - y) \\\cos y + \\\cos (x - y) \\\sin y$\n\nNotice
that $\\\sin (x - y) \\\cos y + \\\cos (x - y) \\\sin y$ is in the form of
$\\\sin(\\\alpha)\\\cos(\\\beta)+\\\cos(\\\alpha)\\\sin(\\\beta)$.\n\nYes, and
we know that
$\\\sin(\\\alpha)\\\cos(\\\beta)+\\\cos(\\\alpha)\\\sin(\\\beta)=\\\sin(\\\alpha+\\\beta)$\n\nSo,
$\\\sin (x - y) \\\cos y + \\\cos (x - y) \\\sin
y=\\\sin((x-y)+y)$\n\n$\\\sin((x-y)+y)=\\\sin(x)$

Final Answer: The final answer is $\\\sinx$. I hope it is correct.

4-shot example for PRM800K Hard ICL:

⬇

Problem:

In right triangle $PQR$, we have $\\\angle Q = \\\angle R$ and $PR =
6\\\sqrt{2}$. What is the area of $\\\triangle PQR$?

Solution:

I see that this is an isosceles right triangle, since $\\\angle Q = \\\angle R
= 45^\\\circ$.\n\nThat means that $PQ = PR = 6\\\sqrt{2}$.\n\nTo find the area
of a triangle, I need to multiply the base and the height and divide by
2.\n\nIn this case, the base and the height are both $6\\\sqrt{2}$, so the
area is $(6\\\sqrt{2})(6\\\sqrt{2})/2$.\n\nI can simplify this expression by
using the property that $\\\sqrt{a}\\\sqrt{b} = \\\sqrt{ab}$.\n\nSo, the area
is $(\\\sqrt{2}\\\sqrt{2})(6\\\cdot 6)/2 = 2\\\cdot 36/2 = 36$.

Final Answer: The final answer is $36$. I hope it is correct.

Problem:

The length of the longer side of rectangle $R$ is $10$ percent more than the
length of a side of square $S.$ The length of the shorter side of rectangle
$R$ is $10$ percent less than the length of a side of square $S.$ What is the
ratio of the area of rectangle $R$ to the area of square $S?$ Express your
answer as a common fraction.

Solution:

Let the side of the square be $s$.\n\nThen the longer side of the rectangle is
$1.1s$.\n\nAnd the shorter side of the rectangle is $0.9s$.\n\nThe area of the
square is $s^2$.\n\nThe area of the rectangle is $1.1s * 0.9s =
0.99s^2$.\n\nSo the ratio of the area of rectangle $R$ to the area of square
$S$ is $\\\frac{0.99s^2}{s^2} = 0.99$.\n\nTherefore, the answer is
$\\\boxed{\\\frac{99}{100}}$.

Final Answer: The final answer is $\\\frac{99}{100}$. I hope it is correct.

Problem:

Suppose that $y^3$ varies inversely with $\\\sqrt[3]{z}$. If $y=2$ when $z=1$,
find the value of $z$ when $y=4$. Express your answer in simplest fractional
form.

Solution:

I know that inverse variation means that the product of the two quantities is
constant, so I can write an equation of the form $y^3\\\cdot\\\sqrt[3]{z}=k$,
where $k$ is some constant.\n\nTo find $k$, I can plug in the given values of
$y$ and $z$: $2^3\\\cdot\\\sqrt[3]{1}=k$, which simplifies to $8=k$.\n\nNow I
can use this equation to find $z$ when $y=4$: $4^3\\\cdot\\\sqrt[3]{z}=8$,
which implies that $\\\sqrt[3]{z}=\\\frac{8}{64}=\\\frac{1}{8}$.\n\nTo get rid
of the cube root, I can cube both sides:
$z=\\\left(\\\frac{1}{8}\\\right)^3=\\\frac{1}{512}$.

Final Answer: The final answer is $\\\frac{1}{512}$. I hope it is correct.

Problem:

Let $d$ be a positive number such that when $109$ is divided by $d$, the
remainder is $4.$ Compute the sum of all possible two-digit values of $d$.

Solution:

This problem involves finding the divisors of a given number, as well as using
the concept of remainders.\n\nOne way to approach this is to write the
division as a quotient and a remainder, like this: $109 = qd + 4$, where $q$
is the quotient.\n\nThen, I can rearrange this equation to get $105 = qd$,
which means that $d$ is a divisor of $105$.\n\nNow, I need to find all the
two-digit divisors of $105$.\n\nI can use prime factorization to help me do
this.\n\nI notice that $105 = 3 \\\times 5 \\\times 7$, which are all prime
numbers.\n\nTherefore, any divisor of $105$ must be a product of some
combination of these three factors.\n\nThe possible products are $1, 3, 5, 7,
15, 21, 35, 105$.\n\nHowever, not all of these are two-digit numbers. Only
$15, 21, 35$ are.\n\nThese are the only possible values of $d$ that satisfy
the given condition.\n\nTo find their sum, I just add them up: $15 + 21 + 35 =
71$.

Final Answer: The final answer is $71$. I hope it is correct.

4-shot example for PRM800K Easy-to-Hard ICL:

⬇

Problem:

How many 4-letter words with at least one consonant can be constructed from
the letters $A$, $B$, $C$, $D$, and $E$? (Note that $B$, $C$, and $D$ are
consonants, any word is valid, not just English language words, and letters
may be used more than once.)

Solution:

To count the number of 4-letter words with at least one consonant, I can use
the complement principle and subtract the number of 4-letter words with no
consonants from the total number of 4-letter words.\n\nThe total number of
4-letter words is simply $5^4$, since each letter has 5 choices and the order
matters.\n\nThe number of 4-letter words with no consonants is just the number
of ways to choose 4 letters from the 2 vowels $A$ and $E$, which is
$2^4$.\n\nTherefore, the number of 4-letter words with at least one consonant
is $5^4 - 2^4$. 625 - 16 = 609

Final Answer: The final answer is $609$. I hope it is correct.

Problem:

Compute the integer $k > 2$ for which\n\\\\[\\\log_{10} (k - 2)! + \\\log_{10}
(k - 1)! + 2 = 2 \\\log_{10} k!.\\\\]

Solution:

I recognize that this equation involves logarithms of factorials, which are
products of consecutive integers.\n\nI also know that logarithms have some
useful properties, such as $\\\log_{10} a + \\\log_{10} b = \\\log_{10} (ab)$
and $c \\\log_{10} d = \\\log_{10} d^c$.\n\nUsing these properties, I can
simplify the equation as follows: \\\\[\\\log_{10} (k - 2)! + \\\log_{10} (k -
1)! + 2 = 2 \\\log_{10} k! \\\implies \\\log_{10} \\\left[ (k - 2)! (k - 1)!
100 \\\right] = \\\log_{10} (k!)^2.\\\\]\n\nSince the bases of the logarithms
are equal, I can conclude that the arguments must also be equal, i.e., \\\\[(k
- 2)! (k - 1)! 100 = (k!)^2.\\\\]\n\nNow I have a simpler equation to solve
for $k$.\n\nI notice that the left-hand side has a factor of $(k - 2)!$, which
is also a factor of $(k - 1)!$ and $(k!)^2$.\n\nTherefore, I can divide both
sides by $(k - 2)!$ to get \\\\[(k - 1)! 100 = k! (k - 1)!\\\\]\n\nSimplifying
further, I get \\\\[100 = k!.\\\\]\n\nThis means that $k$ is the smallest
integer such that its factorial is greater than or equal to 100.\n\nI can try
some values of $k$ to find the right one.\n\nIf $k = 3$, then $k! = 3! = 6 <
100$.\n\nIf $k = 4$, then $k! = 4! = 24 < 100$.\n\nIf $k = 5$, then $k! = 5! =
120 > 100$.\n\nTherefore, $k = 5$ is the smallest integer that satisfies the
equation.

Final Answer: The final answer is $5$. I hope it is correct.

Problem:

What is the largest perfect square factor of 1512?

Solution:

I know that 1512 is divisible by 2, so I can try to factor out powers of 2
until I get an odd number.\n\nDividing by 2 once, I get 756.\n\nDividing by 2
again, I get 378.\n\nDividing by 2 one more time, I get 189.\n\nNow 189 is
odd, so I can’t divide by 2 anymore.\n\nI notice that 189 is divisible by 3,
so I can try to factor out powers of 3.\n\nDividing by 3 once, I get
63.\n\nDividing by 3 again, I get 21.\n\nDividing by 3 one more time, I get
7.\n\nNow 7 is prime, so I can’t factor it anymore.\n\nI have factored 1512 as
2^3 * 3^3 * 7.\n\nTo find the largest perfect square factor, I need to use the
highest even powers of each prime factor.\n\nThe highest even power of 2 that
I have is 2^2.\n\nThe highest even power of 3 that I have is 3^2.\n\nThe
highest even power of 7 that I have is 7^0, which is just 1.\n\nSo the largest
perfect square factor is 2^2 * 3^2 * 7^0, which is 36.

Final Answer: The final answer is $36$. I hope it is correct.

Problem:

If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to?

Solution:

I notice that the coefficients of $p$ and $q$ are reversed in the two
equations, so I wonder if I can use elimination to get rid of one of
them.\n\nI try multiplying the first equation by 4 and the second equation by
3, and then subtracting them to eliminate $p$.\n\nI get
$12p+16q-12p-9q=32-39$, which simplifies to $7q=-7$.\n\nDividing both sides by
7, I get $q=-1$.

Final Answer: The final answer is $-1$. I hope it is correct.

4-shot example for MetaMath Full ICL:

⬇

Problem:

In right triangle $PQR$, we have $\\\angle Q = \\\angle R$ and $PR =
6\\\sqrt{2}$. What is the area of $\\\triangle PQR$?

Solution:

Since $\\\angle Q = \\\angle R$, we have $\\\triangle PQR$ is an isosceles
right triangle.\nTherefore, $PQ = PR = 6\\\sqrt{2}$.\nThe area of $\\\triangle
PQR$ is $\\\frac{1}{2} \\\cdot PQ \\\cdot PR = \\\frac{1}{2} \\\cdot
6\\\sqrt{2} \\\cdot 6\\\sqrt{2} = \\\boxed{36}$.

Final Answer: The final answer is $36$. I hope it is correct.

Problem:

The length of the longer side of rectangle $R$ is $10$ percent more than the
length of a side of square $S.$ The length of the shorter side of rectangle
$R$ is $10$ percent less than the length of a side of square $S.$ What is the
ratio of the area of rectangle $R$ to the area of square $S?$ Express your
answer as a common fraction.

Solution:

Let the side length of square $S$ be $x$.\nThen the longer side of rectangle
$R$ is $1.1x$ and the shorter side is $0.9x$.\nThe area of square $S$ is $x^2$
and the area of rectangle $R$ is $(1.1x)(0.9x) = 0.99x^2$.\nTherefore, the
ratio of the area of rectangle $R$ to the area of square $S$ is
$\\\frac{0.99x^2}{x^2} = \\\boxed{\\\frac{99}{100}}$.

Final Answer: The final answer is $\\\frac{99}{100}$. I hope it is correct.

Problem:

Compute the integer $k > 2$ for which\n\\\\[\\\log_{10} (k - 2)! + \\\log_{10}
(k - 1)! + 2 = 2 \\\log_{10} k!.\\\\]

Solution:

Using the properties of logarithms, we can rewrite the equation
as\n\\\\[\\\log_{10} ((k-2)! \\\cdot (k-1)!) + 2 = \\\log_{10}
(k!)^2.\\\\]\nThen, using the fact that $\\\log_{a} b + \\\log_{a} c =
\\\log_{a} (bc)$, we have\n\\\\[\\\log_{10} ((k-2)! \\\cdot (k-1)! \\\cdot
100) = \\\log_{10} (k!)^2.\\\\]\nFinally, using the fact that $\\\log_{a} b =
\\\log_{a} c$ if and only if $b = c$, we have\n\\\\[(k-2)! \\\cdot (k-1)!
\\\cdot 100 = (k!)^2.\\\\]\nSimplifying, we have\n\\\\[(k-2)! \\\cdot (k-1)!
\\\cdot 100 = (k \\\cdot (k-1) \\\cdot (k-2))!.\\\\]\nCancelling out the
common factors, we have\n\\\\[100 = k \\\cdot (k-1) \\\cdot
(k-2).\\\\]\nFactoring, we find that $100 = 5 \\\cdot 5 \\\cdot 4 = 2^2
\\\cdot 5^2$.\nSince $k > 2$, we see that $k = \\\boxed{5}$.

Final Answer: The final answer is $5$. I hope it is correct.

Problem:

If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to?

Solution:

We can solve this system of equations using the method of
elimination.\nMultiplying the first equation by 3 and the second equation by 4
gives us:\n$9p+12q=24$\n$16p+12q=52$\nWe can subtract the first equation from
the second equation to eliminate the q
term:\n$16p+12q-(9p+12q)=52-24$\n$7p=28$\nDividing both sides by 7 gives
us:\n$p=4$\nSubstituting this value of p into either of the original equations
gives us:\n$4(4)+3q=13$\n$16+3q=13$\n$3q=-3$\n$q=\\\boxed{-1}$

Final Answer: The final answer is $-1$. I hope it is correct.

4-shot example for MetaMath Hard ICL:

⬇

Problem:

In right triangle $PQR$, we have $\\\angle Q = \\\angle R$ and $PR =
6\\\sqrt{2}$. What is the area of $\\\triangle PQR$?

Solution:

Since $\\\angle Q = \\\angle R$, we have $\\\triangle PQR$ is an isosceles
right triangle.\nTherefore, $PQ = PR = 6\\\sqrt{2}$.\nThe area of $\\\triangle
PQR$ is $\\\frac{1}{2} \\\cdot PQ \\\cdot PR = \\\frac{1}{2} \\\cdot
6\\\sqrt{2} \\\cdot 6\\\sqrt{2} = \\\boxed{36}$.

Final Answer: The final answer is $36$. I hope it is correct.

Problem:

The length of the longer side of rectangle $R$ is $10$ percent more than the
length of a side of square $S.$ The length of the shorter side of rectangle
$R$ is $10$ percent less than the length of a side of square $S.$ What is the
ratio of the area of rectangle $R$ to the area of square $S?$ Express your
answer as a common fraction.

Solution:

Let the side length of square $S$ be $x$.\nThen the longer side of rectangle
$R$ is $1.1x$ and the shorter side is $0.9x$.\nThe area of square $S$ is $x^2$
and the area of rectangle $R$ is $(1.1x)(0.9x) = 0.99x^2$.\nTherefore, the
ratio of the area of rectangle $R$ to the area of square $S$ is
$\\\frac{0.99x^2}{x^2} = \\\boxed{\\\frac{99}{100}}$.

Final Answer: The final answer is $\\\frac{99}{100}$. I hope it is correct.

Problem:

Suppose that $y^3$ varies inversely with $\\\sqrt[3]{z}$. If $y=2$ when $z=1$,
find the value of $z$ when $y=4$. Express your answer in simplest fractional
form.

Solution:

Since $y^3$ varies inversely with $\\\sqrt[3]{z}$, we can write the equation
as $y^3\\\sqrt[3]{z}=k$, where $k$ is a constant.\nWe are given that $y=2$
when $z=1$, so we can substitute these values into the equation to solve for
$k$:\n$2^3\\\sqrt[3]{1}=k$\n$8\\\cdot 1=k$\n$k=8$\nNow we can use this value
of $k$ to find $z$ when
$y=4$:\n$4^3\\\sqrt[3]{z}=8$\n$64\\\sqrt[3]{z}=8$\n$\\\sqrt[3]{z}=\\\frac{8}{64}=\\\frac{1}{8}$\n$z=\\\left(\\\frac{1}{8}\\\right)^3=\\\frac{1}{512}$\nSo
when $y=4$, $z=\\\boxed{\\\frac{1}{512}}$.

Final Answer: The final answer is $\\\frac{1}{512}$. I hope it is correct.

Problem:

Let $d$ be a positive number such that when $109$ is divided by $d$, the
remainder is $4.$ Compute the sum of all possible two-digit values of $d$.

Solution:

If $109$ is divided by $d$ and the remainder is $4$, then $d$ divides
$109-4=105$.\nThe prime factorization of $105$ is $3\\\cdot5\\\cdot7$.\nSince
$d$ must be a factor of $105$, the possible values of $d$ are $1, 3, 5, 7, 15,
21, 35, 105$.\nOut of these, only the two-digit values are $15, 21, 35$.\nThe
sum of these values is $15+21+35=\\\boxed{71}$.

Final Answer: The final answer is $71$. I hope it is correct.

4-shot example for MetaMath Easy-to-Hard ICL:

⬇

Problem:

How many 4-letter words with at least one consonant can be constructed from
the letters $A$, $B$, $C$, $D$, and $E$? (Note that $B$, $C$, and $D$ are
consonants, any word is valid, not just English language words, and letters
may be used more than once.)

Solution:

There are a total of $5^4 = 625$ possible 4-letter words that can be
constructed from the given letters.\nTo count the number of words with no
consonants, we can use the principle of complementary counting.\nSince there
are 2 vowels ($A$ and $E$) and 3 consonants ($B$, $C$, and $D$), there are
$2^4 = 16$ words with only vowels.\nTherefore, the number of words with at
least one consonant is $625 - 16 = \\\boxed{609}$.

Final Answer: The final answer is $609$. I hope it is correct.

Problem:

Compute the integer $k > 2$ for which\n\\\\[\\\log_{10} (k - 2)! + \\\log_{10}
(k - 1)! + 2 = 2 \\\log_{10} k!.\\\\]

Solution:

Using the properties of logarithms, we can rewrite the equation
as\n\\\\[\\\log_{10} ((k-2)! \\\cdot (k-1)!) + 2 = \\\log_{10}
(k!)^2.\\\\]\nThen, using the fact that $\\\log_{a} b + \\\log_{a} c =
\\\log_{a} (bc)$, we have\n\\\\[\\\log_{10} ((k-2)! \\\cdot (k-1)! \\\cdot
100) = \\\log_{10} (k!)^2.\\\\]\nFinally, using the fact that $\\\log_{a} b =
\\\log_{a} c$ if and only if $b = c$, we have\n\\\\[(k-2)! \\\cdot (k-1)!
\\\cdot 100 = (k!)^2.\\\\]\nSimplifying, we have\n\\\\[(k-2)! \\\cdot (k-1)!
\\\cdot 100 = (k \\\cdot (k-1) \\\cdot (k-2))!.\\\\]\nCancelling out the
common factors, we have\n\\\\[100 = k \\\cdot (k-1) \\\cdot
(k-2).\\\\]\nFactoring, we find that $100 = 5 \\\cdot 5 \\\cdot 4 = 2^2
\\\cdot 5^2$.\nSince $k > 2$, we see that $k = \\\boxed{5}$.

Final Answer: The final answer is $5$. I hope it is correct.

Problem:

If $n$ is $1$ less than a multiple of $50$, then what is the remainder when
$n^2+2n+3$ is divided by $50$?

Solution:

Since $n$ is $1$ less than a multiple of $50$, we can write $n=50k-1$ for some
integer $k$.\nThen \\\\[n^2+2n+3=(50k-1)^2+2(50k-1)+3=2500k^2-96k+2,\\\\] so
the remainder when $n^2+2n+3$ is divided by $50$ is $\\\boxed{2}$.

Final Answer: The final answer is $2$. I hope it is correct.

Problem:

If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to?

Solution:

We can solve this system of equations using the method of
elimination.\nMultiplying the first equation by 3 and the second equation by 4
gives us:\n$9p+12q=24$\n$16p+12q=52$\nWe can subtract the first equation from
the second equation to eliminate the q
term:\n$16p+12q-(9p+12q)=52-24$\n$7p=28$\nDividing both sides by 7 gives
us:\n$p=4$\nSubstituting this value of p into either of the original equations
gives us:\n$4(4)+3q=13$\n$16+3q=13$\n$3q=-3$\n$q=\\\boxed{-1}$

Final Answer: The final answer is $-1$. I hope it is correct.