# PORT: Preference Optimization on Reasoning Traces

Salem Lahlou∗ Abdalgadar Abubaker∗ Hakim Hacid
<EMAIL_ADDRESS>
Technology Innovation Institute, Abu Dhabi

###### Abstract

Preference optimization methods have been successfully applied to improve not
only the alignment of large language models (LLMs) with human values, but also
specific natural language tasks such as summarization and stylistic
continuations. This paper proposes using preference optimization methods on
Chain-of-Thought steps in order to improve the reasoning performances of
language models. While the chosen answers are obtained from datasets that
include reasoning traces, we propose two complementary schemes for generating
rejected answers: digit corruption, and weak LLM prompting. Our approach leads
to increased accuracy on the GSM8K, AQuA-RAT, and ARC benchmarks for
Falcon2-11B and Mistral-7B. For example, the approach can lead to up to a
relative $8.47\%$ increase in accuracy on the GSM8K benchmark without any
extra annotations. This work suggests that spending resources on creating more
datasets of reasoning traces would further boost LLM performances on informal
reasoning tasks.111Our code is publicly available at
https://github.com/abdalgader-a/port.

PORT: Preference Optimization on Reasoning Traces

Salem Lahlou∗ and Abdalgadar Abubaker∗ and Hakim Hacid
<EMAIL_ADDRESS>Technology Innovation Institute, Abu Dhabi

**footnotetext: denotes equal contribution

##  1 Introduction

In recent years, Large Language Models (LLMs) have been pivotal in
democratizing Artificial Intelligence (AI), given their ease of use and
impressive abilities in a broad spectrum of tasks. While they have
significantly contributed to the striking progress of AI, their success has
heavily relied on scaling-up to ever-larger models and datasets. Nonetheless,
scaling has not proved sufficient for achieving satisfying results on tasks
involving reasoning. Reasoning has been a central theme in the history of AI,
defining goal posts that push the limits of intelligence. The term “reasoning”
is often used to refer to informal reasoning, that “relies on intuition,
experience, and common sense to draw conclusions and solve problems”(Huang and
Chang, 2022). The limits of scale in eliciting reasoning abilities has been
confirmed by analyses in Rae et al. (2021); Bommasani et al. (2021); Cobbe et
al. (2021), amongst others. One reason multi-step reasoning still poses a
challenge to LLMs is that the next-word prediction objective used to train
them does not explicitly encourage step-by-step reasoning.

Chain-of-thought prompting (CoT; Wei et al., 2022b), an augmented prompting
strategy, has been shown to improve LLM performances on reasoning tasks, by
guiding them to generate sequences of intermediate steps. It should be
unsurprising however that solely prompting a language model to “think step by
step”, whether alongside a handful of correct rationales (Wei et al., 2022b)
or not (Kojima et al., 2022), does not necessarily elicit actual system-2-like
(Stanovich et al., 2000; Kahneman, 2003) reasoning abilities, but at best only
mimics humans’ thought processes. Despite claims of the type “LLMs are decent
zero-shot reasoners” (Kojima et al., 2022), the emergent ability of reasoning
appears consistently for very large models ($>100B$ parameters) only (Wei et
al., 2022a).

Figure 1: Illustration of the creation process of a preference dataset with
two complementary approaches to generate rejected answers. The preference
dataset is used to fine-tune a reference model using a Direct Preference
Optimization (DPO) or one of its variants, after a supervised fine-tuning
(SFT) step.

A major limitation of CoT prompting is its reliance on very large models (Wei
et al., 2022b; Kojima et al., 2022). Ho et al. (2022) propose to bypass this
limitation by generating rationales from very large teacher models and using
them to fine-tune smaller student models. In the same line of work, Uesato et
al. (2022) perform a comprehensive comparison between outcome-based
supeversied fine-tuning (SFT), which supervises the final result, and process-
based SFT, which supervises the reasoning process, and find that process-based
supervision significantly helps language models in mathematical reasoning
tasks. However, solely relying on high-quality rationales is costly as it
requires humans or very large language models to generate the reasoning paths.
Furthermore, as evidenced by Ni et al. (2023), SFT alone tends to make the
language model overfit on the rationales seen during training, thus assigning
low probabilities to alternative but correct reasoning paths, and as shown in
Hong et al. (2024), SFT can still lead the language model to assign high-
probabilities to undesired sequences.

A notable advancement in the development of LLMs is a refinement step that
elicits more favorable behaviors. This refinement step is usually performed to
align AI systems with human values (Gabriel, 2020; Ji et al., 2023;
Klingefjord et al., 2024). The simplest refinement strategy requires a set of
demonstrations, or human-made prompt-response examples, and fine-tunes a model
on the dataset using supervised fine-tuning. Preference-based approaches on
the other hand, rely on datasets of comparisons of potential model outputs.
They include reinforcement learning from human feedback (RLHF; Christiano et
al., 2017; Ouyang et al., 2022; Bai et al., 2022), where a reward model is
learned and then optimized using the language model as a policy with
reinforcement learning algorithms such as the proximal policy optimization
algorithm (PPO; Schulman et al., 2017). More recently, methods that bypass
both the need of explicitly modeling the reward function and the need for
online interactions as in RLHF, have become increasingly popular. These
include Direct Preference Optimization (DPO; Rafailov et al., 2024), Identity
Preference Optimization (IPO; Azar et al., 2023), Sequence Likelihood
Calibration with Human Feedback (SLIC; Zhao et al., 2023), and the prospect
theory-based Kahneman-Tversky Optimization (KTO; Ethayarajh et al., 2024).
Preference optimization techniques have been utilized to improve specific
tasks such as summarization and stylistic continuations (Ziegler et al., 2019;
Stiennon et al., 2020), but to the best of our knowledge, have never been used
to tackle reasoning tasks.

In this paper, we propose to apply preference optimization techniques to
chain-of-thought reasoning. More specifically, we propose two complementary
schemes of constructing preference pairs from datasets that include valid
reasoning paths, such as the GSM8K (Cobbe et al., 2021) and AQuA (Ling et al.,
2017). Using Falcon2-11B (Almazrouei et al., 2023) as our base model, we show
that our simplest approach, that relies on corrupting digits in order to
create wrong reasoning steps, can lead to up to $8.47\%$ relative increase in
performances on the GSM8K benchmark, and $18.73\%$ on the AQuA benchmark. We
validate the robustness of our approach by obtaining favorable results using
Mistral-7B (Jiang et al., 2023) as a base model. Additionally, we show the
benefits of the second scheme, that consists of prompting weak LLMs, in
improving the results on ARC (Clark et al., 2018), a non-mathematical
reasoning benchmark. Furthermore, we compare various preference optimization
schemes, and find that DPO leads to better results than its variants. Finally,
we compare the two schemes and various mixtures thereof. Our approach,
exemplified by two schemes which requires no external data as illustrated in
Figure 1, suggests that constructing high-quality chain-of-thought datasets
that span a wide range of domains could be beneficial for improving the
emergent reasoning abilities of language models..

##  2 PORT: Preference optimization on reasoning traces

###  2.1 Problem setup

Starting from a finite set of tokens ${\mathcal{V}}$, called hereafter the
vocabulary, an autoregressive language model can be seen as a collection of
probability distributions $p_{\mathrm{LM}}$ over ${\mathcal{V}}$ conditioned
on elements of
${\mathcal{V}}^{\leq\tau}:=\bigcup_{t=1}^{\tau}{\mathcal{V}}^{t}$, i.e.
sequences of up to $\tau$ tokens. We assume the existence of an end-of-
sentence (EOS) token in ${\mathcal{V}}$, denoted $\mathrm{EOS}$, that can
represent a full stop or a line-break for example.
To generate text, a pre-trained language model prompted with an input
$q\in{\mathcal{V}}^{\leq\tau}$ is queried autoregressively and samples tokens
$s_{i}\in{\mathcal{V}}$, where $s_{i}\sim p_{\mathrm{LM}}(.\mid qs_{1}\dots
s_{i-1})$. The generation process stops at the first index $k$ for which
$s_{k}=\mathrm{EOS}$222or until the prompt $qs_{1}\dots s_{k}$ exceeds $\tau$
tokens, but we disregard this case by assuming a very large context window..
Here, $qs_{1}\dots s_{i-1}$ refers to the concatenation of the tokens
$q,s_{1},\dots,s_{i-1}$.
When interacting with language models, and more specifically in CoT reasoning,
we are interested in generating sentences $z$, i.e. sequences from
${\mathcal{V}}^{\leq T}$ that end with the $\mathrm{EOS}$ token, rather than
an arbitrary amount of tokens. When prompted with a sentence $x$ or a sequence
of sentences $xz^{1}z^{2}\dots z^{k-1}$, the language model can therefore
autoregressively generate a new sentence $z^{k}\sim p_{\mathrm{LM}}(.\mid
xz^{1}z^{2}\dots z^{k-1})$.
Given a question $x$ (e.g., a math problem), we define a chain-of-thought as a
sequence of $n$ sentences $z^{1},\dots,z^{n}$, where $z^{n}$ is the final
answer. Assuming the existence of a binary function $(x,z)\mapsto\eta(x,z)$
that assesses the correctness of the sentence $z$ to the question $x$, our
goal is to tune a pre-trained model $p_{\mathrm{LM}}$ to generate a chain
$z^{1},\dots,z^{n}$ from a question $x$ such that $\eta(x,z^{n})=1$.

###  2.2 Proposed approach

Our approach first requires access to a dataset of reasoning traces, called a
CoT dataset,
${\mathcal{D}}_{\mathrm{train}}=\\{(x_{i},z_{i}^{1},\dots,z_{i}^{n_{i}})\\}_{i=1}^{N}$,
where each training example includes a question $x_{i}$, and a reasoning trace
(or rationale) comprised of $n_{i}$ sentences,
$z_{i}^{1},\dots,z_{i}^{n_{i}}$, of which the last element, $z_{i}^{n_{i}}$ is
a valid answer to the question $x_{i}$, i.e., $\eta(x_{i},z_{i}^{n_{i}})=1$.
Naturally, the number of steps $n_{i}$ needed to reach the answer to $x_{i}$
depends on the question itself.
Such datasets ${\mathcal{D}}_{\mathrm{train}}$ are generally human-made.
Examples of publicly available reasoning datasets include the arithmetic
datasets GSM8K (Cobbe et al., 2021), AQuA-RAT (Ling et al., 2017), MAWPS
(Koncel-Kedziorski et al., 2016) as well as the commonsense reasoning datasets
StrategyQA (Geva et al., 2021), Creak (Onoe et al., 2021), e-SNLI (Camburu et
al., 2018), ECQA (Aggarwal et al., 2021), QASC (Khot et al., 2019), QED (Lamm
et al., 2021), Sen-Making (Wang et al., 2019).

##### SFT data:

From such a dataset, we can construct a dataset
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$ of prompt-response pairs,
where each example $(x_{i},z_{i}^{1},\dots,z_{i}^{n_{i}})$ contributes $n_{i}$
pairs:
$(x_{i},z_{i}^{1}),(x_{i}z_{i}^{1},z_{i}^{2}),\dots,(x_{i}z_{i}^{1}\dots
z_{i}^{n_{i}-1},z_{i}^{n_{i}})$. Such a dataset can be used for supervised
fine-tuning, during which the parameters ${\bm{\theta}}$ of the base language
model $p_{\mathrm{LM}}$ are updated to minimize the SFT loss:

$\displaystyle\leavevmode\resizebox{408.68454pt}{}{${\mathcal{L}}_{\mathrm{SFT}}({\bm{\theta}})=-\frac{1}{N_{\mathrm{SFT}}}\sum_{i=1}^{N}\sum_{k=1}^{n_{i}}p_{\bm{\theta}}(z_{i}^{k}\mid
x_{i}z_{i}^{1:k-1})$},$ (1)

where $N_{\mathrm{SFT}}=\sum_{i=1}^{N}n_{i}$ =
|${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$|. In principle, if the data
is representative of the target task and if the model generalizes well, the
SFT phase should increase the likelihood of valid reasoning steps. Put
differently, because each $z_{i}^{k}$ in the training dataset is a step
towards a valid answer $z_{i}^{n_{i}}$, then after supervised-fine-tuning, on
similar examples, the model should encourage the sentences that unroll the
reasoning and help discover a valid answer to the initial question.

##### Preference data:

From ${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$, we also construct a
preference dataset ${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$, comprised
of triplets of the form (prompt, chosen, rejected). The prompt and the chosen
answers are obtained directly from
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$, but for each prompt (which is
actually either a question or a concatenation of a question and a certain
number of initial reasoning steps), we need an invalid reasoning step.
Naturally, an arbitrary sequence of tokens would be invalid, but it will
provide no useful signal to the model if it is fine-tuned with RLHF or with
preference optimization methods such as DPO using such a preference a dataset.
Ideally, the rejected answers should be almost correct reasoning steps, or
contain errors that either a language model or a human are expected to make.
Naturally, the rejected answers can be obtained using human annotators
explicitly asked to generate wrong but close-enough answers. In this work
however, we investigate two simple and complementary ways of defining such a
dataset:

  * •

LLM generation: For each pair $(x_{i}z_{i}^{1:k-1},z_{i}^{k})$ from
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$, we prompt a smaller language
model (hereafter also referred to as weak LLM) with $x_{i}z_{i}^{1:k-1}$ and
use the response to define the corresponding rejected answer $\hat{z}^{k}$. By
incorporating the resulting triplet in the preference dataset, we naturally
incentivize the base model to avoid errors of the type made by the weak LLM.
This process can be used to generate multiple rejected answers $\hat{z}^{k}$
per prompt $x_{i}z_{i}^{1:k-1}$.

  * •

Digit corruption: In datasets that involve mathematical reasoning, most
reasoning steps $z^{k}$ include digits. Without modifying any non-digit
character of $z^{k}$, we replace each digit with one from $0$ to $9$ with
equal probability. Similarly, this approach can be used to generate multiple
rejected answers $\tilde{z}^{k}$ per prompt $x_{i}z_{i}^{1:k-1}$.

An illustration of this dataset creation process is provided in Table 4 of
Appendix A.
After an SFT phase where $p_{\mathrm{LM}}$ is fine-tuned into
$p_{\mathrm{SFT}}$ using ${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$ by
minimizing the loss in 1, any preference optimization method can be used on
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$. For instance, DPO (Rafailov
et al., 2024) fine-tunes $p_{\mathrm{SFT}}$ into a model $p_{\bm{\theta}}$
that minimizes the following loss:

$\displaystyle{\mathcal{L}}_{\mathrm{DPO}}({\bm{\theta}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim{\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}}\left[l(x,y_{w},y_{l};{\bm{\theta}})\right],$
(2)

where

$\displaystyle l(x,y_{w},y_{l};{\bm{\theta}})=\log$
$\displaystyle\sigma\left(\beta\log\frac{p_{\bm{\theta}}(y_{w}\mid
x)}{p_{\mathrm{SFT}}(y_{w}\mid x)}-\right.$
$\displaystyle\left.\beta\log\frac{p_{\bm{\theta}}(y_{l}\mid
x)}{p_{\mathrm{SFT}}(y_{l}\mid x)}\right).$ (3)

Here $\sigma$ is the sigmoid function and $\beta$ is a scaling hyperparameter.
As an alternative to using two separate SFT and preference optimization
phases, ORPO Hong et al. (2024), a more recent approach, combines both steps
and replaces the loss in 3 with:

$\displaystyle l(x,$ $\displaystyle
y_{w},y_{l};{\bm{\theta}})=-\lambda\log\sigma\left(\log\frac{p_{\bm{\theta}}(y_{w}\mid
x)}{1-p_{\bm{\theta}}(y_{w}\mid x)}-\right.$
$\displaystyle\left.\log\frac{p_{\bm{\theta}}(y_{l}\mid
x)}{1-p_{\bm{\theta}}(y_{l}\mid x)}\right)-\log p(y_{w}\mid x),$ (4)

where $\lambda$ is a weighing hyperparameter.

##  3 Experiments

The goal of the experiments presented in this section are threefold. First, we
empirically investigate the proposed approach. Then, we compare the two
schemes above for constructing rejected answers. Additionally, we investigate
the effect of the preference optimization method by comparing ORPO to DPO and
some of its variants.

##### Evaluation:

To assess the approach, we need to evaluate the models on informal reasoning
tasks, for which chains of thoughts can help reach the valid answers. Our main
evaluation task is the GSM8K test dataset (Cobbe et al., 2021), that contains
$1319$ high quality grade school math word problems. For evaluation, we also
use the two following datasets:

  * •

A harder math word problem: the Algebra Question Answering with Rationales
dataset333https://github.com/google-deepmind/AQuA (AQuA; Ling et al., 2017),
which includes approximately $100,000$ algebraic word problems, each presented
with a rationale leading to one of five multiple-choice options (A to E). We
use the accompanying test set of $254$ examples for evaluation.

  * •

A non-mathematical dataset: The AI2’s Reasoning
Challenge444https://allenai.org/data/arc (ARC; Clark et al., 2018), that
covers multiple science subjects. The questions are split into Easy and
Challenge sets. Questions in the Challenge set cannot be solved with retrieval
or co-occurence methods. Each question admits one valid answer amongst a set
of typically four options. There is no ground-truth chain-of-thought reasoning
provided for this dataset. We use the test set of the ARC-Challenge set, that
consists of $1172$ examples, for evaluation.

More specifically, we use the Language Model Evaluation Harness (Gao et al.,
2023) to calculate the accuracy (between $0$ and $1$) of the tested models. To
elicit the desired CoT behavior, we add few-shot examples from the train set
that contain rationales to each question to be evaluated, extract from the
generated text the proposed answer, and compare it to the ground truth. We
report our results below as percentages. We use 5-shot examples for GSM8K and
AQuA, obtained from their corresponding train sets. For ARC-Challenge, we use
25-shot examples, but given that no rationale is provided in the train set, we
use GPT-4 (et al., 2023) to construct plausible rationales, and filter them
out manually. The used prompt is provided in Appendix B.

##### Base model:

As a base model, we use the newly released pre-trained
Falcon2-11B555https://huggingface.co/tiiuae/falcon-11B (Almazrouei et al.,
2023) for all our experiments, except in Section 3.5, where we confirm that
our method is agnostic to the base model by using Mistral-7B (Jiang et al.,
2023).

##### Training data:

As previously mentioned, our method requires using a CoT dataset. For all our
experiments, except in Section 3.6, we use the GSM8K train dataset Cobbe et
al. (2021), that consists of $7473$ examples, given that it contains solution
steps, in order to construct the SFT and preference datasets
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$ and
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$, as explained in Section 2.2.
Example tuples $(x_{i},z_{i}^{1},\dots,z_{i}^{n_{i}})$ from this dataset are
provided in Appendix A. It is important to note that throughout the training
and evaluation process, we only use the training set, without any extra data
or human annotation.

###  3.1 Supervised fine-tuning

From the GSM8K training dataset, we construct the SFT dataset
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$ as described in Section 2.2.
The train set of GSM8K consists of $7473$ examples, with an average of $4.57$
reasoning step per example, leading to to an SFT dataset of $34197$ examples.
We then fine-tune the based model on this dataset using low-rank adaptation
(LoRA; Hu et al., 2021) for efficient parameter updates, processing each
example 3 times. The learning rate used in $1.4\times 10^{-5}$, and the batch
size is 16. For LoRA, we use rank 64 matrices and a scaling parameter
$\alpha=16$. It is noteworthy that GSM8K examples contain calculation
annotations (between <<>>, as shown in the examples provided in Appendix A).
These annotations can be used to call external tools (e.g., python scripts or
calculators) to perform calculations, rather than asking the LLM to perform
the calculation. While we made no such usage of external tools, we tried both
keeping and removing the annotations from the text before SFT, and found no
significant difference in terms of performance. We thus decided to process the
dataset without annotations.
Details about the choice of the hyperparameters are provided in Appendix C.

##### Results:

In addition to testing the fine-tuned model on the GSM8K’s test set, we assess
SFT’s out-of-distribution generalization, both on the harder math word problem
AQuA, and on the non-mathematical task ARC-Challenge. We report the accuracies
in Table 1. As expected, and as confirmed by other studies (Uesato et al.,
2022), fine-tuning the model on the reasoning steps helps improve the
performances on questions requiring reasoning that come from the same
distribution. Unsurprisingly however, the performances on AQuA and ARC-
Challenge drop after the SFT stage, confirming the overfitting issues of SFT,
and their limited generalization to unseen examples (Ni et al., 2023). This is
confirmed by Table 5 in Appendix D, where we reduce the number of training
epochs (on GSM8K) and observe better performances on AQuA. In the next
subsections, we investigate whether preference optimization algorithms can
lead to even further performance boosts on the three evaluation tasks.

Model | GSM8K | AQuA | ARC
---|---|---|---
Base model | $54.66$ | $31.50$ | $76.11$
SFT | $55.43$ | $30.71$ | $75.60$
DPO (ours) |  $\mathbf{58.91}$+4.25 |  $\mathbf{35.04}$+4.54 | $76.02$
Table 1: Accuracy (in percentage) of the base, SFT, and DPO models on the
three considered tasks. The base model Falcon2-11B is trained on datasets
obtained from the GSM8K training set. The rejected answers for DPO are
obtained using digit corruption, as explained in Section 3.2

###  3.2 Preference optimization with digit corruption

From ${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$, we construct a
preference dataset ${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$ using
digit corruption as explained in Section 2.2. Given the stochasticity of the
digit corruption approach, we ensure that the rejected answers are indeed
invalid, by repeatedly generating reasoning steps until they differ from the
ground truth reasoning steps. For reasoning steps that do not include digits,
we simply do not include them in the preference dataset.

We fine-tune the SFT model on the obtained preference dataset using DPO with a
scale factor $\beta=0.2$, with the same LoRA configuration as SFT. We use the
AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of $8\times
10^{-6}$ along with a linear schedule for the learning rate, as explained in
Appendix C.

##### Results:

We report the accuracies post DPO tuning in Table 1. The significant
performance increase in GSM8K (a relative $7.77\%$) shows how merely
corrupting digits to create bad reasoning steps improves the mathematical
reasoning abilities of Falcon2-11B. Our approach helps boost performances on
the AQuA task, with a relative increase of $14.41\%$, even without using any
example from the AQuA train set during training. We note however, that there
is no benefit on the ARC-Challenge task. We suspect that it is because it does
not require the same type of skills as GSM8K and AQuA. In Section 3.3, we
investigate whether other schemes could boost ARC performances.

####  3.2.1 Variants of DPO

While DPO has emerged as the go-to method for preference optimization, several
variants (Azar et al., 2023; Ethayarajh et al., 2024) claim to address some of
its shortcomings: overfitting, inefficient learning, memory utilization. A
recent attempt at benchmarking these approaches (Saeidi et al., 2024) has
shown that DPO still outperforms its variants on a variety of tasks. In this
section, we make use of our constructed preference dataset to further compare
DPO to its variants. Unlike Saeidi et al. (2024), we also consider ORPO (Hong
et al., 2024) which combines both SFT and preference optimization.

##### Results:

We report the accuracies on the GSM8K test dataset in Figure 2. We find that
the variants of DPO do not lead to improved performances, even with extensive
hyperparameter tuning for each method separately (Appendix C). This confirms
the recent observations from Saeidi et al. (2024)

Figure 2: Comparison of alternatives to DPO. The y-axis represents the
accuracy on the GSM8K test dataset.

###  3.3 Preference optimization using weak LLMs

Unlike Section 3.2, when construct
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$ using weak LLM generation, as
described in Section 2.2, there are a few parameters to take into account:
which weak LLM to use? how to prompt said LLM? how to post-process the
resulting sequences?
We first consider instruct version of the Gemma model (Team et al., 2024),
Gemma-2B-it, to generate answers. We use the prompt provided in Appendix B. We
then filter out the responses that do not start with “Next step: ”, and simply
do not create the corresponding triplet in the preference dataset. The
generation stops at the first line-break or full stop. We also consider the
larger Llama-7B (Touvron et al., 2023) and its chat version, to assess the
effect of the weak LLM size.
When using a weak LLM to generate rejected answers, it is not unlikely that
the LLM outputs valid reasoning steps, in which case, including the resulting
triplet in the preference dataset might hurt generalization of the resulting
model. We experimented with the robust version of DPO (Chowdhury et al.,
2024), which accounts for the ambiguity in the preferences, but that did not
result in improved performances. We therefore consider to corrupt the digits
of the generated sequences similar to the digit corruption scheme alone. In
Figure 7 of Appendix D, we study the effect of post-generation digit
corruption, and find that digit corruption is essential for downstream tasks.
We also compare using the chat version of Llama-7B with the prompt template of
Appendix B to using its base version with few-shot examples only, and find
that using the base version yields to better performances.
Lastly, we consider an iterative approach, where we Falcon-11B fine-tuned with
DPO as described in Section 3.2 as a weak LLM. We report in Table 2 the
accuracies on the three tasks, using the three weak LLMs with post-generation
digit corruption. While the weak LLM scheme does not perform as well as the
digit corruption scheme alone on the math world problems, it is noteworthy
that with Llama-7B and the iterative approach, the performance on ARC-
Challenge improves over the base model.

Weak LLM | GSM8K | AQuA | ARC
---|---|---|---
Gemma-2B-it | $53.68$ | $29.92$ | $75.94$
Llama-7B | $56.10$ | $30.71$ | $77.05$
iterative | $55.65$ | $33.46$ | $76.28$
Table 2: Accuracy (in percentage) of the base, SFT, and DPO models on the
three considered tasks. iterative corresponds to Falcon-11B fine-tuned with
DPO as per Section 3.2. To construct the preference dataset, we generate
sequences with the LLMs, then we corrupt the digits as described in Section
2.2.

###  3.4 Increasing the size of the preference dataset

A natural question at this point is to consider the effect of the size of the
preference dataset on the resulting model fine-tuned with DPO. Given that our
proposed approach allows us to generate arbitrarily many wrong reasoning steps
per valid reasoning step (e.g., we can corrupt the digits in many ways, and
prompt weak LLMs multiple tims), we can construct preference datasets with
triplets (prompt, chosen, rejected) that contain redundant (prompt, chosen)
pairs, with different rejected answers. We thus consider using three rejected
answers for the digit corruption, Gemma-2B-it, and Llama-7B experiments. We
also consider fine-tuning on a dataset consisting of both digit corrupted
answers and Llama-7B-generated answers (themselves digit corrupted), and a
dataset containing three times as many rejected answers.
We report in Figure 3 the results of the different schemes on the three tasks
at hand. Strikingly, simply tripling the number of rejected answers for the
digit corruption scheme leads to a an accuracy of $59.29\%$ on the GSM8K task,
which represents a relative increase of $8.47\%$ over the base performances.
Additionally, mixing different sources of rejected answers (Llama-7B and digit
corruption), can lead to increased performances on ARC, indicating that
diversifying the sources might help with generalization to other tasks.

Figure 3: Effect of increasing the preference dataset size. The answers
generated by the weak LLMs undergo digit corruption as well.

###  3.5 Using Mistral as a base model

To further test our approach, we perform some of the experiments above using
Mistral-7B (Jiang et al., 2023) as a base model. We report the results in
Figure 4, and find that all approaches lead to better performances on the
GSM8K benchmark than the base model, which scores $38.51\%$. With this
experiment, we confirm that digit corruption is the strongest of the two
schemes we proposed.

Figure 4: GSM8K accuracy. Mistral-7B as base model. Comparison of different schemes. Model | GSM8K | AQuA | ARC
---|---|---|---
Base model | $54.66$ | $31.50$ | $76.11$
SFT on AQUA | $54.89$ | $31.50$ | $75.68$
DPO - digit corr. | $57.70$ |  $\mathbf{37.40}$+5.90 | $76.88$
DPO - Llama-7B | $55.57$ | $33.86$ | $76.71$
Table 3: Accuracy of the base, SFT, and DPO models (with different schemes) on
the considered tasks. The datasets used for training are obtained from the
AQuA training set.

###  3.6 Using AQuA as source dataset

So far, we have used the GSM8K training data to construct the SFT and
preference datasets. Naturally, given the proximity of the corresponding
questions in the test set, fine-tuning on such datasets would mostly help
boost the performances on the GSM8K benchmark. In this section, we consider
using the AQuA training set to create
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$ and
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$. For SFT and DPO, we fine-
tune with the same hyper-pararameters as the ones we found best for the
experiments on GSM8K. We report the results in Table 3, using Falcon2-11B as a
base model, and confirm that using the AQuA training set is more helpful for
the AQuA benchmark ($18.73\%$ relative increase) than using the GSM8K training
set.

##  4 Related work

##### What is reasoning?

Reasoning can be thought of as the process of logically and systematically
analyzing information, drawing on evidence and past experiences to form
conclusions or make decisions (McHugh and Way, 2018). Using the taxonomy of
Huang and Chang (2022), reasoning can be either deductive (a conclusion is
drawn based on the truth of the premises), inductive (a conclusion is drawn
based on observations or evidence), or abductive (a conclusion is drawn based
on the best explanation for a given set of observations). Bronkhorst et al.
(2020) also make the distinction between formal reasoning, akin to what is
used in mathematics, in which a fixed set of rules is followed, and informal
reasoning that is less structured, and is akin to what is used in everyday
life.

##### Reasoning in LLMs:

Mathematics, science, and code benchmarks (Austin et al., 2021; Hendrycks et
al., 2021; Liang et al., 2023; Clark et al., 2018) are becoming increasingly
popular to study the emergent reasoning abilities of language models trained
on next token prediction. Chain-of-Thought prompting (Wei et al., 2022b) and
related techniques such as Tree-of-Thought (Yao et al., 2024) and Graph-of-
Thought (Besta et al., 2024) have shown to improve language model performances
on reasoning tasks, simply by prompting them to generate intermediate
computations required for solving the problems. It is not clear however
whether the improved performances brought about by chain-of-thought prompting
are due specifically to human-like task decomposition, or more generally to
the increased computation that additional tokens allow (Pfau et al., 2024). An
orthogonal direction for boosting language model performances on reasoning
tasks is reasoning-enhanced training. For example, Lewkowycz et al. (2022);
Taylor et al. (2022); Chen et al. (2021) show that training or fine-tuning
LLMs on datasets containing scientific, math, or code data helps improve
downstream performances on reasoning tasks. Another line of work (Zelikman et
al., 2022; Huang et al., 2022; Gulcehre et al., 2023; Yuan et al., 2023; Singh
et al., 2023; Hosseini et al., 2024) consists of using LLMs to self-improve
their reasoning abilities via bootstrapping, where rationales generated by the
model that lead to the correct answer are further used to fine-tune the model
itself. Aligned with this direction, and more closely related to our work, Ni
et al. (2023) propose to use intermediate steps as supervision signal. Another
popular set of approaches make use of verifiers, that classify or score
reasoning traces (Cobbe et al., 2021; Uesato et al., 2022). As an example of
such approaches, GRACE (Khalifa et al., 2023) trains a discriminator with a
contrastive loss over correct and incorrect steps, and uses it when generating
answers to questions requiring reasoning, to score next-step candidates based
on their correctness.

##### Preference optimization:

To make the most out of a preference dataset, Reinforcement learning with
human feedback commonly applies the Bradley-Terry model (Bradley and Terry,
1952) to train a reward model that scores instances, and use it to fine-tune
the language model to maximize the score of the reward model for the prefered
responses using algorithms such as PPO (Schulman et al., 2017). More recently,
advances in offline methods such as DPO (Rafailov et al., 2024) and its
variants (Azar et al., 2023; Zhao et al., 2023; Cai et al., 2023; Ethayarajh
et al., 2024) that directly align the language models without the need for an
explicit reward function, have proven successful in practice. These methods
however require an SFT phase to achieve convergence to desired results
(Rafailov et al., 2024; Tunstall et al., 2023). ORPO (Hong et al., 2024) on
the other hand, bypasses the need for the multi-stage process, and uses a loss
that combines both supervised fine-tuning and preference optimization. In our
work, we use these methods as part of the pipeline and compare them
thoroughly.

##  5 Conclusion

We considered the question of using preference optimization to boost reasoning
abilities of language models. More specifically, we proposed two different
schemes for constructing preference datasets of reasoning steps from datasets
that include valid reasonign traces. We showed that by using DPO on these
datasets, we are able to improve the reasoning abilities of Falcon2-11B and
Mistral-7B, even on tasks unseen during training. We also compared DPO to
several of its variants. Our work suggests that constructing high-quality
reasoning traces datasets can boost general informal reasoning abilities.

## Limitations

We considered two schemes for wrong reasoning step generation in this work:
digit corruption, and LLM generation. There are several other ways that could
be considered. For instance, it could be beneficial to consider shuffling the
order of the reasoning steps, or prompting an LLM to slightly tweak the
ground-truth reasoning steps until they become wrong. We leave the study of
other schemes to future work, along with scaling to models over 11B.
Additionally, when using the weak LLM scheme, there is an overhead incurred
when creating the dataset. Finally, our work has focused on mathematical
reasoning, and future work should explore using other sources of reasoning
data. Perhaps mixing between different sources of data, as suggested by recent
work from Chung et al. (2024), could lead to improved abilities in out-of-
distribution reasoning benchmarks.

## References

  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for commonsenseqa: New dataset and models.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  The falcon series of open language models.  _arXiv preprint arXiv: 2311.16867_.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_.
  * Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023.  A general theoretical paradigm to understand learning from human preferences.  _arXiv preprint arXiv:2310.12036_.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Besta et al. (2024) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024.  Graph of thoughts: Solving elaborate problems with large language models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 17682–17690.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv: 2108.07258_.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Bronkhorst et al. (2020) Hugo Bronkhorst, Gerrit Roorda, Cor Suhre, and Martin Goedhart. 2020.  Logical reasoning in formal and everyday reasoning tasks.  _International Journal of Science and Mathematics Education_ , 18:1673–1694.
  * Cai et al. (2023) Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, and Guannan Zhang. 2023.  Ulma: Unified language model alignment with demonstration and point-wise human preference.  _arXiv preprint arXiv:2312.02554_.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-snli: Natural language inference with natural language explanations.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv: 2107.03374_.
  * Chowdhury et al. (2024) Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. 2024.  Provably robust dpo: Aligning language models with noisy feedback.  _arXiv preprint arXiv: 2403.00409_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Chung et al. (2024) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024.  Scaling instruction-finetuned language models.  _Journal of Machine Learning Research_ , 25(70):1–53.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv: 1803.05457_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * et al. (2023) OpenAI et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv: 2303.08774_.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024.  Kto: Model alignment as prospect theoretic optimization.  _arXiv preprint arXiv: 2402.01306_.
  * Gabriel (2020) Iason Gabriel. 2020.  Artificial intelligence, values, and alignment.  _Minds and machines_ , 30(3):411–437.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, D. Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023.  Reinforced self-training (rest) for language modeling.  _arXiv preprint arXiv:2308.08998_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the MATH dataset.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Ho et al. (2022) Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022.  Large language models are reasoning teachers.  _Annual Meeting of the Association for Computational Linguistics_.
  * Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. 2024.  Orpo: Monolithic preference optimization without reference model.  _arXiv preprint arXiv: 2403.07691_.
  * Hosseini et al. (2024) Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. 2024.  V-star: Training verifiers for self-taught reasoners.  _arXiv preprint arXiv:2402.06457_.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv: 2106.09685_.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _arXiv preprint arXiv:2210.11610_.
  * Huang and Chang (2022) Jie Huang and Kevin Chen-Chuan Chang. 2022.  Towards reasoning in large language models: A survey.  _arXiv preprint arXiv:2212.10403_.
  * Ji et al. (2023) Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. 2023.  Ai alignment: A comprehensive survey.  _arXiv preprint arXiv:2310.19852_.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _arXiv preprint arXiv: 2310.06825_.
  * Kahneman (2003) Daniel Kahneman. 2003.  Maps of bounded rationality: Psychology for behavioral economics.  _American economic review_ , 93(5):1449–1475.
  * Khalifa et al. (2023) Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Ho Hin Lee, and Lu Wang. 2023.  Grace: Discriminator-guided chain-of-thought reasoning.  _Conference on Empirical Methods in Natural Language Processing_.
  * Khot et al. (2019) Tushar Khot, Peter Clark, Michal Guerquin, Peter Alexander Jansen, and Ashish Sabharwal. 2019.  Qasc: A dataset for question answering via sentence composition.  _AAAI Conference on Artificial Intelligence_.
  * Klingefjord et al. (2024) Oliver Klingefjord, Ryan Lowe, and Joe Edelman. 2024.  What are human values, and how do we align ai to them? _arXiv preprint arXiv: 2404.10636_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Koncel-Kedziorski et al. (2016) Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016.  MAWPS: A math word problem repository.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1152–1157, San Diego, California. Association for Computational Linguistics.
  * Lamm et al. (2021) Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021.  Qed: A framework and dataset for explanations in question answering.  _Transactions of the Association for computational Linguistics_ , 9:790–806.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models.  _Advances in Neural Information Processing Systems_ , 35:3843–3857.
  * Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023.  Holistic evaluation of language models.  _Transactions on Machine Learning Research_.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 158–167, Vancouver, Canada. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * McHugh and Way (2018) Conor McHugh and Jonathan Way. 2018.  What is reasoning?  _Mind_ , 127(505):167–196.
  * Ni et al. (2023) Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. 2023.  Learning math reasoning from self-sampled correct and partially-correct solutions.  In _The Eleventh International Conference on Learning Representations_.
  * Onoe et al. (2021) Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. 2021.  Creak: A dataset for commonsense reasoning over entity knowledge.  _NeurIPS Datasets and Benchmarks_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Pfau et al. (2024) Jacob Pfau, William Merrill, and Samuel R. Bowman. 2024.  Let’s think dot by dot: Hidden computation in transformer language models.  _arXiv preprint arXiv: 2404.15758_.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Saeidi et al. (2024) Amir Saeidi, Shivanshu Verma, and Chitta Baral. 2024.  Insights into alignment: Evaluating dpo and its variants across multiple tasks.  _arXiv preprint arXiv: 2404.14723_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Singh et al. (2023) Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. 2023.  Beyond human data: Scaling self-training for problem-solving with language models.  _arXiv preprint arXiv: 2312.06585_.
  * Stanovich et al. (2000) Keith E Stanovich, Richard F West, and JE Alder. 2000.  Individual differences in reasoning: Implications for the rationality debate?-open peer commentary-three fallacies.  _Behavioral and Brain Sciences_ , 23(5):665–665.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020.  Learning to summarize from human feedback.  _Neural Information Processing Systems_.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022.  Galactica: A large language model for science.  _arXiv preprint arXiv: 2211.09085_.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Tieleman (2012) Tijmen Tieleman. 2012.  Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ , 4(2):26.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv: 2307.09288_.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_.
  * Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022.  Solving math word problems with process- and outcome-based feedback.  _arXiv preprint arXiv: 2211.14275_.
  * Wang et al. (2019) Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019.  Does it make sense? and why? a pilot study for sense making and explanation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4020–4026, Florence, Italy. Association for Computational Linguistics.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a.  Emergent abilities of large language models.  _Trans. Mach. Learn. Res._ , 2022.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024.  Tree of thoughts: Deliberate problem solving with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023.  Scaling relationship on learning mathematical reasoning with large language models.  _arXiv preprint arXiv:2308.01825_.
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022.  STar: Bootstrapping reasoning with reasoning.  In _Advances in Neural Information Processing Systems_.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv: 1909.08593_.

##  Appendix A Dataset examples

We provide in the following box two examples from the training dataset of
GSM8K, the main source of data used in our experiments. We also provide in
Table 4 the preference triplets obtained from one example from the GSM8K (a
third one). The ground truth rationale for this particular example contains
three sentences, and thus contributes three examples to the preference
dataset.

Examples from the GSM8K training dataset Question: John puts $25 in his piggy
bank every month for 2 years to save up for a vacation. He had to spend $400
from his piggy bank savings last week to repair his car. How many dollars are
left in his piggy bank? Rationale: He saved money for 2 years, which is equal
to 12 x 2 = <<12*2=24>>24 months. The amount of money he saved is $25*24 =
$<<25*24=600>>600\. But he spent some money so there is $600 - $400 =
<<600-400=200>>200 left. #### 200. Question: Five coaster vans are used to
transport students for their field trip. Each van carries 28 students, 60 of
which are boys. How many are girls? Rationale: There are a total of 5 vans x
28 students = <<5*28=140>>140 students. If 60 are boys, then 140 - 60 =
<<140-60=80>>80 of these students are girls. #### 80

##  Appendix B Used prompts

In the following box, we provide the prompt used to generate plausible
rationales for the 25 few shot examples of ARC-Challenge, using GPT-4.

GPT-4 prompt for ARC rationale generation You are expert grade-school science
teacher. Given the following question, provide justification for the answer.
Question: question. Answer Choices: options. Answer: … The Answer is answer
letter. You need to add a two to three sentences rationale before “The answer
is answer letter”, justifying the correct answer.

Next, we provide the template used to prompt the weak LLMs to generate
rejected answers to create the preference dataset for Section 3.3.

Prompt template for weak LLM generation You are an obedient assistant. Your
task is to reason about the following question. Write only the next step of
the reasoning chain. Your answer should include exactly one following
reasoning step and has to be exactly one sentence long! The answer should
start with "Next step: ". Here are two examples:

    Question: {prompt_example_1}
    Next step: {first_step_example_1}
    Next step: {second_step_example_1}
    Next step: {third_step_example_1}
    Next step: {fourth_step_example_1}
    Next step: {final_answer_example_1}

    Question: {prompt_example_2}
    Next step: {first_step_example_2}
    Next step: {second_step_example_2}
    Next step: {third_step_example_2}
    Next step: {fourth_step_example_2}
    Next step: {fifth_step_example_2}
    Next step: {sixth_step_example_2}
    Next step: {final_answer_example_2}

    Question: {prompt}
    Next step: {first_step_ground_truth}
    ...

prompt |  chosen |  rejected(1) |  rejected(2)
---|---|---|---
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? |  Natalia sold 48/2 = 24 clips in May. |  Natalia sold 32/4 = 19 clips in May. |  Natalia sold 48 clips in April, and half as many clips in May, which is 24 clips
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Natalia sold 48/2 = 24 clips in May. |  Natalia sold 48+24 = 72 clips altogether in April and May. |  Natalia sold 25+98 = 12 clips altogether in April and May. |  Natalia sold 24 clips in April.
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Natalia sold 48/2 = 24 clips in May. Natalia sold 48+24 = 72 clips altogether in April and May. |  The solution to the problem is 72. |  The solution to the problem is 13. |  Natalia sold 24 clips in April, so she sold 24 clips in May.
Table 4: Example of preference dataset obtained from the question “Natalia
sold clips to 48 of her friends in April, and then she sold half as many clips
in May. How many clips did Natalia sell altogether in April and May?”, and the
corresponding rationale “Natalia sold $48/2=24$ clips in May. Natalia sold
$48+24=72$ clips altogether in April and May. The solution to the problem is
$72$.” This example is obtained from the GSM8K dataset (Cobbe et al., 2021).
The chosen column represents steps from ground-truth rationale, rejected (1)
are examples obtained by digit corruption, and rejected (2) are examples
obtained by prompting the Llama-2-7B-chat model (Touvron et al., 2023)

##  Appendix C Training details

##### Number of epochs used for SFT:

An important hyperparameter when doing supervised fine-tuning on small
datasets is the number of times each example is processed. We optimized this
hyperparameter independently using the smaller Falcon-7B (Almazrouei et al.,
2023) as a base model, with the GSM8K accuracy as a metric, trying values
$\\{1,2,3,4,5\\}$. We ended up using 3 epochs for all subsequent SFT
experiments (using Falcon2-11B as a base model).

##### SFT learning rate:

Similarly, using the GSM8K accuracy as a metric, and with a random search of a
handful of learning rates in the range $[10^{-8},10^{-4}]$, we ended up using
the learning rate of $1.4\times 10^{-5}$.

##### SFT batch size:

It is commonly agreed upon that larger batch sizes are more desirable when
fine-tuning language models. We used a batch size of 16 as it was the largest
that did not lead to memory issues on the GPUs we used.

##### LoRA parameters:

Given the computational cost of fine-tuning LLMs, we chose not to tune the
hyperparameters of LoRA (Hu et al., 2021), and resorted to using the popular
values of $\alpha=16$ and $rank=64$.

##### Optimizer:

For DPO, we used a linear schedule for the learning rate, and first jointly
optimized the maximal learning rate and number of warm-up steps for the linear
schedule, using RMSProp (Tieleman, 2012) as an optimizer. Optimizing this
couple of hyperparameters was done on Falcon-7B using a preference dataset
with digit corrupted rejected answers only, with the GSM8K accuracy as a
metric. After settling on $10$ warm-up steps, we tuned the maximal learning
rate and the optimizer (choosing between RMSProp and AdamW (Loshchilov and
Hutter, 2019)) on the same (model, task, metric) triplet. We ended up using
the AdamW optimizer with a maximal learning rate of $8\times 10^{-6}$.

##### Further DPO hyperparameters:

We further optimized the learning rate, as well as the number of training
epochs and the value of the $\beta$ hyperparameter on the preference dataset
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$ constructed from GSM8K, and
using digit-corrupted Llama-7B (Touvron et al., 2023), as explained in Section
3.3, to generate wrong answers. Using the resulting model’s performance on the
GSM8K task, with the evaluation protocol described in Section 3, we report the
results of our hyperparameter sweep in Figure 5. This led to a universal
choice of $\beta=0.2$, learning rate of $8\times 10^{-6}$, and number of
epochs equal to $1$ for all DPO experiments with Falcon2-11B.

Figure 5: DPO hyperparameter search. The y axis corresponds to the accuracy on
the test set of GSM8K.

##### Hyperparameters for DPO variants:

Similar to DPO, the KTO (Ethayarajh et al., 2024) loss requires the
specification of a hyperparameter $\beta$ that controls how far the fine-tuned
model drifts from the SFT model. IPO (Azar et al., 2023) needs a
regularization parameter $\tau$, for which the inverse $\tau^{-1}$ is usually
denoted by $\beta$ as well. For both methods, the value of $\beta$ is critical
and needs to be carefully tuned. We report in Figure 6 the results of our
hyperparameter search. We consider the preference dataset
${\mathcal{D}}_{\mathrm{train}}^{\mathrm{pref}}$ constructed from GSM8K, and
using digit corruption, as explained in Section 3.2, to generate wrong
answers.
For ORPO (Hong et al., 2024), an important parameter is the weighing
hyperparameter $\lambda$ in Section 2.2, that specifies the relative
importance of the negative log-likelihood of the chosen answer with respect to
the odds ratio part of the loss. We tried the values in the set
$\\{0.001,0.005,0.01,0.1,0.2,0.3\\}$ along with learning rates from the set
$\\{10^{-8},8\times 10^{-8},8\times 10^{-6}\\}$, and found that $\beta=0.001$
and $10^{-8}$ as a learning rate lead to the best results, which is what we
report in Figure 2.

Figure 6: DPO variants hyperparameter search. The y axis corresponds to the
accuracy on the test set of GSM8K. The learning rate $8\times 10^{-6}$ and
number of epochs ($1$) used are the same as DPO.

##  Appendix D Additional Results

In Table 5, we study how the number of times each example from the GSM8K
training dataset is visited during training affects the downstream performance
on the related but different AQuA evaluation task. The table shows that
reducing the training time could help dampen the overfitting issues of SFT.

| 1 epoch | 3 epochs
---|---|---
AQuA accuracy | $33.46$ | $30.71$
Table 5: Accuracy (in percentage) on the AQuA test dataset of Falcon2-11B
fine-tuned on ${\mathcal{D}}_{\mathrm{train}}^{\mathrm{SFT}}$ obtained from
the GSM8K train dataset, as explained in Section 3.1. Comparison of the effect
of number of epochs.

When using a weak LLM to generate rejected answers, it is not unlikely that
the LLM outputs valid reasoning steps, in which case, including the resulting
triplet in the preference dataset might hurt generalization of the resulting
model. We therefore consider to corrupt the digits of the generated sequences
similar to the digit corruption scheme alone. In Figure 7, we study the effect
of post-generation digit corruption, and find that digit corruption is
essential for downstream tasks. We also compare using the chat version of
Llama-7B with the prompt template of Appendix B to using its base version, and
find that using the base version yields to better performances.

Figure 7: DPO with weak LLM generation for rejected answers. Comparison of
different versions of Llama-7B. The y axis corresponds to the accuracy on the
test set of GSM8K. The learning rate use is $8\times 10^{-6}$ and number of
epochs is $1$.

###  D.1 Ablations

We hypothesized that fine-tuning a language model to predict the next
reasoning step only should help improve performances on reasoning benchmarks.
Our results in the main paper confirm this hypothesis. However, it is natural
to wonder whether using multiple reasoning steps to predict could be
beneficial. More specifically, for SFT, we compare our approach (which
requires fine-tuning on $(xz^{1:k-1},z^{k})$ pairs) to fine-tuning on
$(xz^{1:k-1},z^{k:n})$ pairs. Similarly, for DPO, we compare our approach
(that requires fine-tuning on $(xz^{1:k-1},z^{k},\tilde{z}^{k})$ triplets) to
fine-tuning on $(xz^{1:k-1},z^{k:n},\tilde{z}^{k}z^{k+1:n})$ triplets. Using
Falcon2-11B as a base model, and GSM8K as a data source and for evaluation, we
found that with this change, the performance drops from $55.43$ to $54.81$ for
SFT, and from $58.30$ to $57.01$ for DPO with digit corruption.
We also tested replacing the inputs $xz^{1:k-1}$ with $uxz^{1:k-1}$, where $u$
is a sequence corresponding to 3-shot examples, and found that while the SFT
performance slightly increases to $55.95$, the DPO performance significantly
drops to $50.11$.