# Preference Learning Algorithms Do Not Learn Preference Rankings

Angelica Chen
New York University
<EMAIL_ADDRESS>
&Sadhika Malladi
Princeton University
<EMAIL_ADDRESS>
&Lily H. Zhang
New York University
<EMAIL_ADDRESS>
Xinyi Chen
Google DeepMind; Princeton University
<EMAIL_ADDRESS>
&Qiuyi Zhang
Google DeepMind
<EMAIL_ADDRESS>
&Rajesh Ranganath
New York University
<EMAIL_ADDRESS>
&Kyunghyun Cho
New York University; Genentech; CIFAR LMB
<EMAIL_ADDRESS>
###### Abstract

Preference learning algorithms (e.g., RLHF and DPO) are frequently used to
steer LLMs to produce generations that are more preferred by humans, but our
understanding of their inner workings is still limited. In this work, we study
the conventional wisdom that preference learning trains models to assign
higher likelihoods to more preferred outputs than less preferred outputs,
measured via _ranking accuracy_. Surprisingly, we find that most state-of-the-
art preference-tuned models achieve a ranking accuracy of less than $60\%$ on
common preference datasets. We furthermore derive the _idealized ranking
accuracy_ that a preference-tuned LLM would achieve if it optimized the DPO or
RLHF objective perfectly. We demonstrate that existing models exhibit a
significant _alignment gap_ – _i.e._ , a gap between the observed and
idealized ranking accuracies. We attribute this discrepancy to the DPO
objective, which is empirically and theoretically ill-suited to fix even mild
ranking errors in the reference model, and derive a simple and efficient
formula for quantifying the difficulty of learning a given preference
datapoint. Finally, we demonstrate that ranking accuracy strongly correlates
with the empirically popular win rate metric when the model is close to the
reference model used in the objective, shedding further light on the
differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO)
preference learning algorithms.

##  1 Introduction

Recent work on aligning LLMs has focused predominantly on tuning models to
adhere to human preferences – commonly through reinforcement learning (RLHF;
Stiennon et al. [45]) or directly via offline supervision (DPO; Rafailov et
al. [39]). Preference learning algorithms [19, 53, 56] were originally
designed to use a dataset of pairwise preferences over candidates to train a
model with high _ranking accuracy_ – that is, the model can precisely rank
preferred outputs over dispreferred ones. In the case of language models, the
ranking is determined by the likelihood assigned to each candidate.

Many LLM alignment techniques are designed to yield models with a high
preference ranking accuracy, including SLiC [66, 65], RAFT [7], PRO [44], and
RRHF [62]. Most prominently, Rafailov et al. [39] claimed that their popular
direct preference optimization (DPO) algorithm “increases the relative log
probability of preferred to dispreferred response." It is standard to evaluate
these various objectives by measuring how often the resulting model’s
generations are preferred over another model’s (i.e., a _win rate_) [69].
However, the relationship between the loss, ranking accuracy, and win rate is
unclear, leaving open the question of what these alignment techniques are
actually accomplishing during training.

In this work, we demonstrate that RLHF and DPO struggle to increase ranking
accuracy in practice and explore both the theoretical and empirical reasons
why. Our findings highlight an intricate relationship between offline
optimization and online behavior, and motivate the need for more fine-grained
analyses of preference training dynamics. Our contributions are as follows:

  1. 1.

Existing models do not achieve high ranking accuracies. We demonstrate that a
wide variety of open-access preference-tuned LLMs (e.g., Llama 2 7B Chat,
Gemma 7B IT, and Zephyr 7B DPO) achieve a ranking accuracy below $60\%$ across
a range of validation splits from commonly used preference datasets, such as
UltraFeedback [6], Anthropic helpfulness and harmlessness (HH-RLHF, [13]), and
Stanford Human Preferences (SHP, [10]) (Figure 1).

  2. 2.

Existing models exhibit a significant _alignment gap_ between the ranking
accuracy they achieve and the accuracy achievable under idealized conditions.
We derive a simple formula (Theorem 3.1) for the _idealized ranking accuracy_
(i.e., the ranking accuracy achieved from training on ground-truth preference
data and perfectly optimizing the DPO or RLHF objective). We observe that
models suffer from a significant _alignment gap_ in that they achieve ranking
accuracy far below the idealized ranking accuracy (Table 1, Figure 1).

  3. 3.

Preference learning rarely corrects incorrect rankings. We prove theoretically
that even mild ranking errors in the reference model can make it virtually
impossible for DPO and its variants to correct the ranking (Theorem 4.1), and
demonstrate that in practice, the rankings are rarely flipped (Fig. 2) and the
reference model likelihoods generally determine the ranking accuracy (Fig. 3).
Our results permit straightforward and efficient identification of hard-to-
learn preference datapoints without any tuning.

  4. 4.

Ranking accuracy and win rate are closely correlated when the model is close
to the reference model. We observe that the ranking accuracy and win rate
trend together when the model is close to the reference model during the early
phase of alignment, but become anti-correlated once the model has moved too
far away, adding to the ongoing discussion on the differences between on-
policy and off-policy behaviors of preference-tuned LLMs.

Crucially, our work highlights fundamental flaws in RLHF and DPO that prevent
the preference-tuned model from achieving a high ranking accuracy _even on the
training dataset_.

##  2 Preliminaries

###  2.1 Learning from Human Preferences

#### Preference Data

Human preference data typically takes the form of pairwise preferences. Each
prompt $x$ is paired with two possible continuations – $y_{1}$ and $y_{2}$.
One or more human raters then annotate which continuation is preferred. When
there are multiple raters, we use $\alpha(x,y_{1},y_{2})$ to denote the
proportion of raters who prefer $y_{1}$ over $y_{2}$.111In the limit, when
there are infinite raters, the empirical proportion $\alpha(x,y_{1},y_{2})$
converges to the ground truth preference $\mathbb{P}[y_{1}\succ y_{2}\mid x]$.

######  Definition 2.1 (Aggregated Preference Datapoint).

Consider a prompt $x$ with two possible continuations $y_{1}$ and $y_{2}$ and
the proportion of raters $\alpha(x,y_{1},y_{2})$ who preferred $y_{1}$ over
$y_{2}$. Then, the aggregated preference datapoint for each prompt $x$ is
denoted $(x,y_{w},y_{l})$ where $y_{w}$ is the completion preferred by the
majority of voters.

We note that at the time of writing, the vast majority of datasets either use
a single rater [13] or only release aggregated preference data [10, 25], so we
often do not have access to $\alpha(x,y_{1},y_{2})$. A standard assumption is
that the ground-truth human preferences obey the Bradley-Terry model
(Assumption 5).

#### Supervised Fine-Tuning (SFT)

In the first step of the preference learning pipeline, the model is typically
trained using the standard cross-entropy objective on some choice of offline
instruction-tuning dataset(s). In some implementations [50], a variety of
third-party datasets are selected, whereas in other implementations [45, 39,
41] the model is instead trained on the preferred continuations $(x,y_{w})$
from the same preference learning dataset that is used in downstream
preference learning. The resulting model is often used as a _reference model_
, denoted as ${\pi_{\text{Ref}}}$ or ${\pi_{\text{SFT}}}$, and it typically
serves as the initialization when learning from human preferences.

#### Reinforcement Learning from Human Feedback (RLHF)

Learning from human feedback originally required using reinforcement learning
[45]. In this setting, the possible continuations for each prompt are sampled
from a reference model (i.e., $(y_{w},y_{l})\sim{\pi_{\text{Ref}}}(\cdot\mid
x)$) and then annotated and aggregated to create a preference dataset
${\mathcal{D}}$. Then, one frames the problem as binary classification between
the two continuations and trains a reward model $r_{\phi}(x,y)$ to minimize
${\mathcal{L}}_{R}(r_{\phi},{\mathcal{D}})=-{\mathop{\mathbb{E}}}_{(x,y_{w},y_{l})\sim{\mathcal{D}}}[\log\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))]$.
Finally, one trains the model ${\pi_{\theta}}$ to maximize the reward without
straying too far from the reference model ${\pi_{\text{Ref}}}$. Because
sampling generations from the model is non-differentiable, it is common to use
PPO to maximize the reward
$r(x,y)=r_{\phi}(x,y)-\beta(\log{\pi_{\theta}}(y\mid
x)-\log{\pi_{\text{Ref}}}(y\mid x))$, where $\beta>0$ is a regularization
coefficient designed to prevent the model from straying too far from its
initialization.

#### Preference Learning with DPO

Rafailov et al. [39] demonstrated that one can avoid using PPO by
reparametrizing the objective to operate over policies instead of over
rewards. Then, one can minimize the differentiable DPO objective.

######  Definition 2.2 (DPO Objective [39]).

Let $\sigma$ be the sigmoid function and $\beta>0$ be a hyperparameter. Then,
the DPO objective for an aggregated preference dataset ${\mathcal{D}}$ and a
reference model ${\pi_{\text{Ref}}}$ is defined as

$\displaystyle{\mathcal{L}}_{\text{DPO}}({\pi_{\theta}},{\pi_{\text{Ref}}})$
$\displaystyle=-\mathop{\mathbb{E}}_{(x,y_{w},y_{l})\sim{\mathcal{D}}}[\log\sigma(\underbrace{\beta\log\frac{{\pi_{\theta}}(y_{w}\mid
x)}{{\pi_{\text{Ref}}}(y_{w}\mid x)}-\beta\log\frac{{\pi_{\theta}}(y_{l}\mid
x)}{{\pi_{\text{Ref}}}(y_{l}\mid x)}}_{\text{reward margin}})]$
$\displaystyle=-\mathop{\mathbb{E}}_{(x,y_{w},y_{l})\sim{\mathcal{D}}}[\log\sigma(\beta\underbrace{\log\frac{{\pi_{\theta}}(y_{w}\mid
x)}{{\pi_{\theta}}(y_{l}\mid x)}}_{\text{model log-
ratio}}+\beta\underbrace{\log\frac{{\pi_{\text{Ref}}}(y_{l}\mid
x)}{{\pi_{\text{Ref}}}(y_{w}\mid x)}}_{\text{reference model log-ratio}})]$

We denote the DPO loss on the aggregated datapoint $(x,y_{w},y_{l})$ as
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})$.

###  2.2 Evaluation Metrics

Evaluating the alignment of a preference-tuned LLM is both under-specified and
multi-dimensional. Many knowledge-based and logic-based benchmarks (e.g. MMLU,
GLUE, BIG-Bench, HELM) already exist, but these benchmarks largely fail to
capture nuanced aspects of human preference, such as helpfulness or
harmlessness [13]. As such, one standard evaluation is to ask human or machine
raters how often the model produces a favorable completion compared to a
baseline (_i.e._ , win rate). Human win rate is the gold standard but is
costly to compute and can be biased based on size and nature of the worker
pool [18, 24]. Rating completions using another LLM (e.g., MT-bench) can be
cheaper but similarly suffers from various biases [37, 67, 60], and several
studies have revealed failures in many LLM judges to identify violations of
instruction-following [64, 28]. Nevertheless, since win rate evaluations are
so prevalent, we compare ranking accuracy against win rate in Sec. 5 and
describe when the former off-policy metric is correlated with the popular on-
policy metric.

Besides the win rate,preference learning algorithms are also benchmarked by
the frontier of the rewards versus the divergence from the initialization
[39], which serves as a heuristic of how well the model can incorporate
preference data without unlearning prior information. However, it is unclear
how well rewards can describe the success of alignment.

As aforementioned, the current paper investigates the _ranking accuracy_ ,
which is defined as follows:

######  Definition 2.3 (Ranking Accuracy).

The ranking accuracy ${{\mathcal{R}}}$ of a model ${\pi_{\theta}}$ on an
aggregated preference datapoint $(x,y_{w},y_{l})$ is defined as

${{\mathcal{R}}}(x,y_{w},y_{l};\pi_{\theta})=\begin{cases}1&{\pi_{\theta}}(y_{w}\mid
x)\geq{\pi_{\theta}}(y_{l}\mid x)\\\ 0&\text{otherwise.}\end{cases}$ (1)

Analogously, the ranking accuracy of policy $\pi_{\theta}$ on a dataset
$\mathcal{D}=\\{(x,y_{w},y_{l})\\}$ is
${{\mathcal{R}}}(\mathcal{D};\pi_{\theta})={\mathop{\mathbb{E}}}_{(x,y_{w},y_{l})\sim\mathcal{D}}{{\mathcal{R}}}(x,y_{w},y_{l};\pi_{\theta}).$
In the rare case where a dataset has more than two outputs $y$ per prompt $x$,
we use the generalized ranking accuracy definition stated in App. A.6.

######  Remark 2.4 (Lengths of Completions).

We note that $y_{w}$ and $y_{l}$ can have different lengths; for example,
Singhal et al. [43] showed that $y_{w}$ is usually longer. Length can deflate
${\pi_{\theta}}(y_{w}\mid x)$ and reduce the ranking accuracy. One can
normalize the likelihoods by the length of the response, but the length-
normalized ranking accuracy may not be meaningful in practice, because it is
currently unclear how to sample from the length-normalized likelihood. For
completeness, we report the ranking accuracies of both the unnormalized and
normalized policies, denoted ${{\mathcal{R}}}$ and $\tilde{{\mathcal{R}}}$,
respectively.

######  Remark 2.5 (Difference between Ranking Accuracy and Reward Accuracy).

For RLHF models and DPO models, the ranking accuracy is not equivalent to the
reward accuracy (_i.e._ , the metrics evaluated in RewardBench [28]). In the
RLHF case, we are evaluating the ranking accuracy of the final policy rather
than the reward model. In the DPO case, reward accuracy measures whether
$\beta\log\frac{{\pi_{\theta}}(y_{w}\mid x)}{{\pi_{\text{Ref}}}(y_{w}\mid
x)}>\beta\log\frac{{\pi_{\theta}}(y_{l}\mid x)}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}$ instead of whether ${\pi_{\theta}}(y_{w}\mid x)>{\pi_{\theta}}(y_{l}\mid
x)$. Since we ultimately sample from $\pi_{\theta}$ rather than
$\frac{{\pi_{\theta}}(y\mid x)}{{\pi_{\text{Ref}}}(y\mid x)}$, we find the
ranking accuracy to be of greater practical importance.

Moreover, we demonstrate that under very stringent conditions, minimizing the
DPO objective results in a model with high ranking accuracy. We characterize
the phenomenon on individual datapoints, as is the case throughout the paper,
but note that Markov’s inequality can be straightforwardly applied to extend
the results to a low loss on the entire dataset.

######  Proposition 2.6 (Sanity Check).

Recall the definition of $y_{w}$, $y_{l}$ in Definition 2.1. If
${\pi_{\text{Ref}}}(y_{w}\mid x)\geq{\pi_{\text{Ref}}}(y_{l}\mid x)$ and
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})\leq
0.6$, then ${{\mathcal{R}}}(x,y_{w},y_{l})=1$.

This result, proved in App. A.1, requires the condition that the reference
model already has the correct ranking, so it is unlikely to hold across all
datapoints in practice and somewhat moot. The remainder of the paper focuses
on more realistic settings where the reference model is imperfect.

##  3 The Alignment Gap

Prop. 2.6 showed that training a low DPO loss with a perfect reference model
yields a model with perfect ranking accuracy. However, Fig. 1(a) shows that
real-world reference models exhibit low ranking accuracies, which prompts us
to study more realistic, imperfect reference models.

###  3.1 Existing Reference Models Rarely Have Correct Rankings

(a) Ranking accuracies of various reference models, including GPT2 [38],
Pythia 2.8B [4], Pythia 1.4B [4], Llama 2 7B [50], Vicuna 1.5 7B [67], OLMo 7B
[16], Tulu2 7B [20], Zephyr 7B SFT [52], Mistral v0.1 7B [22], and Gemma 7B
[49]

(b) Ranking accuracies of various preference-tuned models, including Llama 2
7B Chat [50], Tulu2 7B DPO [20], Zephyr 7B DPO [52], and Gemma 7B IT [49]

Figure 1: Both reference and preference-tuned models exhibit low ranking
accuracy on most preference datasets. Each point represents the length-
normalized or non-length-normalized ranking accuracy of individual (1(a))
reference models (pre-trained or fine-tuned), or (1(b)) preference-tuned
models (trained with DPO or RLHF). We sub-sample 1K examples from each dataset
and use the test split when available. We describe datasets in C.2 and list
all numbers in Tables 2, 3, and 4.

Fig. 1(a) indicates that reference models rarely achieve high ranking accuracy
on common preference datasets (except Synthetic Instruct GPT-J Pairwise), even
though many were likely trained on the preferred completions (see Sef. 2.1).
Many of the models do not have documented training data so we do not know
which preference datasets, if any, are in-distribution. We also fine-tune
several pretrained LLMs on the preferred completions (see App. C.1) and
observe that ranking accuracy does not increase significantly.222It is not
surprising that fine-tuning on the preferred completions does not boost
ranking accuracy, since the model does not receive any knowledge of the
relative qualities of the preferred and rejected completions. Based on our
findings, we turn to the case of imperfect reference models.

###  3.2 Idealized Ranking Accuracy

We showed above that empirically, reference models exhibit poor accuracy when
ranking the plausible completions. However, the RLHF reward and DPO objective
were formulated to ensure that the model learns the preference dataset but
does not move too far from the reference model ${\pi_{\text{Ref}}}$, so there
may be a limit on the possible accuracy of the preference-tuned model. Here,
we formalize this intuition by studying what the optimal policies would be
when perfectly optimizing DPO or RLHF with access to perfect data (i.e., true
proportions of human preferences).333This result differs from Proposition 2.6
in that it accounts for a potentially imperfect reference model.

######  Theorem 3.1 (Simulating Perfect RLHF444We note that this result can
also be straightforwardly derived from prior works [36, 26, 15].).

Fix a reference model ${\pi_{\text{Ref}}}$ and an aggregated preference
datapoint $(x,y_{w},y_{l})\sim{\mathcal{D}}$. Assume the dataset includes the
ground-truth human preferences: that is,
$\alpha(x,y_{w},y_{l})=\mathbb{P}(y_{w}\succ y_{l})$, and that these
preferences obey the Bradley-Terry model (Equation 5). Let $\pi^{*}$ be the
model resulting from perfectly optimizing the DPO or RLHF objective on
$(x,y_{w},y_{l})$ as described in Section 2.1. Then, $\pi^{*}$ satisfies

$\frac{\pi^{*}(y_{w}\mid x)}{\pi^{*}(y_{l}\mid
x)}=\frac{{\pi_{\text{Ref}}}({y_{w}\mid x})}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}\left(\frac{\alpha(x,y_{w},y_{l})}{1-\alpha(x,y_{w},y_{l})}\right)^{1/\beta}$
(2)

where $\alpha(x,y_{w},y_{l})$ is the proportion of raters who preferred
$y_{w}$ over $y_{l}$ and $\beta$ is a hyperparameter in the DPO and RLHF
objectives.

######  Remark 3.2.

We prove this result in App. A.2. This theorem highlights the importance of
reporting multiple ratings for each preference datapoint, because it shows
that when $\alpha(x,y_{w},y_{l})=1$, the objectives only have an optimum at
infinity.

This result allows us to simulate the policy resulting from perfect
optimization of either the RLHF or the DPO learning objective. As such, given
a reference model ${\pi_{\text{Ref}}}$ and preference dataset ${\mathcal{D}}$,
we can easily measure the _idealized ranking accuracy_ of a model. We prove
this result in App. A.3.

######  Corollary 3.3 (Idealized Ranking Accuracy).

Given a reference model ${\pi_{\text{Ref}}}$, the DPO or RLHF hyperparameter
$\beta$, a dataset of aggregated preferences
${\mathcal{D}}=\\{(x,y_{w},y_{l})\\}$ and their corresponding rater
proportions $\alpha(x,y_{w},y_{l})$, the ranking accuracy of the optimum of
the RLHF or DPO objective $\pi^{*}$ is given by

${{\mathcal{R}}^{*}}({\mathcal{D}};{\pi_{\text{Ref}}})=\underset{(x,y_{w},y_{l})\sim{\mathcal{D}}}{\mathbb{E}}\left[\mathbbm{1}\left[\frac{{\pi_{\text{Ref}}}({y_{w}\mid
x})}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}\left(\frac{\alpha(x,y_{w},y_{l})}{1-\alpha(x,y_{w},y_{l})}\right)^{1/\beta}>1\right]\right]$
(3)

where $\mathbbm{1}[\cdot]$ is the indicator function. When computed on length-
normalized likelihoods from $\tilde{\pi_{\text{Ref}}}$, we denote the
idealized ranking accuracy as $\tilde{{\mathcal{R}}^{*}}$.

###  3.3 Measuring the Alignment Gap

Given access to $\pi_{\text{Ref}}$, $\beta$, and the $\alpha(x,y_{w},y_{l})$
values for each triple $(x,y_{w},y_{l})$ in a given preference dataset, we can
compute the idealized ranking accuracy from Eq. 3.555Note that when
$\alpha(x,y_{w},y_{l})=1$, we replace it with $1-\epsilon$ to compute the
formula. See App. C.4. The results are shown in Table 1 and further details
are given in App. C.4.

We identify several surprising findings. Firstly, even under ideal conditions
(_i.e._ perfectly optimizing the objective on ground-truth preference data),
the idealized ranking accuracy is still sometimes below $100\%$. This distance
varies with the choice of $\beta$, which indicates that the limits of DPO/RLHF
depend largely upon how strong the reliance on $\pi_{\text{Ref}}$ is.
Furthermore, we find that many state-of-the-art models do not achieve a
ranking accuracy anywhere close to the idealized ranking accuracy, exhibiting
alignment gaps ranging from 19 to 59 percentage points (measured to the median
idealized ${{\mathcal{R}}}$ or $\tilde{{\mathcal{R}}}$).

Table 1: The idealized ranking accuracy of existing algorithms is not perfect, but preference-tuned models exhibit ranking accuracies far even from this idealized case. We provide both the length-normalized ($\tilde{{\mathcal{R}}}$) and non-length-normalized (${{\mathcal{R}}}$) ranking accuracies for a variety of open-access preference-tuned models on the Alpaca Farm [8] validation dataset (described in App. C.2). We also provide the idealized ranking accuracy (Corollary 3.3). Since idealized ranking accuracy can be computed with a variety of values of $\beta$, we provide the minimum, median, and maximum idealized ranking accuracy values for a range of $\beta$. For more details, see App. C.4. Preference-Tuned Model | Length-Normalized | Non-Length-Normalized
---|---|---
|  $\tilde{{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}^{*}}$ (Min./Med./Max.) |  ${{\mathcal{R}}}$ |  ${{\mathcal{R}}^{*}}$ (Min./Med./Max.)
Zephyr-7B-DPO |  54% |  86% / 98% / 100% |  42% |  90% / 99% / 100%
Tulu-2-DPO-7B |  53% |  87% / 97% / 100% |  42% |  91% / 99% / 100%
Google-Gemma-7B-IT |  54% |  73% / 73% / 97% |  40% |  67% / 93% / 100%
LLaMA-2-7B-Chat-HF |  53% |  87% / 97% / 100% |  40% |  91% / 99% / 100%

##  4 Understanding Ranking Accuracy with DPO

We now turn to the training objectives to account for the alignment gap. We
focus our analysis on the DPO objective (Definition 2.2), because its failure
to achieve high ranking accuracy is especially surprising (Table 1). In
particular, DPO directly maximizes the reward margin between preferred-
dispreferred pairs over an offline dataset so we would expect it to perform
well on in-distribution held-out data. We also note that DPO is a popular
choice in the community for aligning LLMs, because it is less costly than
performing RLHF.

In this section, we study real-world characteristics of DPO. First, we show
empirically that DPO rarely flips the ranking of the two continuations. This
result combined with the observation that reference models exhibit poor
ranking accuracy (Sec. 3.1) provides an explanation for the observed poor
ranking accuracies in Table 1. We then formally characterize how hard it is
for DPO to correct the ranking of each datapoint.Our result highlights how the
reference model conditions the optimization: as the reference model log-ratio
(Definition 2.2) grows larger, one has to reduce the DPO loss to a
dramatically small value to flip the incorrect ranking (Fig. 3).

###  4.1 DPO Rarely Flips Preference Rankings

(a) DPO loss.

(b) DPO reward margin.

(c) Proportion of the dataset corresponding to each category of data.

Figure 2: Despite continuously decreasing the loss, DPO rarely flips the
rankings of pairs and instead mostly increases the reward margin of already
correctly ranked pairs. We train a Pythia-2.8B model using the DPO objective
and categorize the training dataset into four subsets – examples that
initially have the correct ranking and are flipped to (1) correct or (2)
incorrect, and examples that initially have the incorrect ranking and are
flipped to (3) correct or (4) incorrect. In all three figures, the hue of the
point indicates the category. The dashed vertical line indicates the training
step at which the lowest eval. loss occurs. We also present results for two
other models with three seeds each in Appendix D.

To study how ranking accuracy changes over the course of DPO training, we
train three sizes of models (GPT-2 [38], Pythia 2.8B [4], and Llama 2-7B [50])
across three seeds each on the Anthropic HH-RLHF [3] preference dataset and
study the ranking accuracy on different partitions of the training dataset. We
present the results from training one seed of Pythia 2.8B in Fig. 2, and defer
training details to App. D.1 and results on the other two models to App. D.2.
In Fig. 2, we partition a random subsample of 1K examples from the training
dataset into four groups based on whether the reference model
${\pi_{\text{Ref}}}$ had the correct ranking and whether the current model
${\pi_{\theta}}$ has the correct ranking.

Surprisingly, Fig. 2 demonstrates that DPO rarely flips the ranking of
$(y_{w},y_{l})$ over the course of training despite consistently reducing the
loss ${\mathcal{L}}_{\text{DPO}}$. Aside from the group of points for which
the model unlearns the correct preference ranking, we observe that the loss
decreases and the reward margin increases consistently while training.
However, at the point of lowest validation loss, less than half of the
incorrectly ranked points have been flipped to have the correct ranking. This
indicates that the DPO objective is ill-formulated to induce a high ranking
accuracy in practice.

Figure 3:  DPO loss alone does not predict ranking accuracy, due to the
influence of the reference model log-ratio in the loss. Each point represents
the DPO loss on a separate training example $(x,y_{w},y_{l})$ from a subsample
of 1K examples from the training dataset, using the model $\pi_{\theta^{*}}$
that corresponds to the checkpoint with the lowest validation loss. The color
of each point indicates whether $\pi_{\theta^{*}}$ achieves the correct
ranking on that example, _i.e._ , whether
$\pi_{\theta^{*}}(y_{w}|x)>\pi_{\theta^{*}}(y_{l}|x)$. The dashed line is the
function $f(c)=-\log\sigma(\beta c)$, from Theorem 4.1. In summary, the
examples that $\pi_{\theta^{*}}$ classifies correctly tend to be those that
were already classified correctly by the reference model. Results for the
other two seeds of each model are given in Fig. 8.

###  4.2 Analysis: How Easy Is It To Flip A Ranking?

In the result below, we show that the DPO loss can decrease substantially
without any improvement on the ranking accuracy of the model. Specifically,
the DPO loss that the model needs to reach in order to have the correct
ranking on an example $(x,y_{w},y_{l})$ depends on the quality of the
reference model, quantified by the reference model log-ratio. This dependence
is highly ill-conditioned, whereby using a reference model with moderately
incorrect likelihoods assigned to each continuation can effectively prevent
DPO from learning the correct ranking.

######  Theorem 4.1.

Consider an aggregated preference datapoint $(x,y_{w},y_{l})$ such that the
reference model log-ratio is some constant $c$, i.e.
$\log\frac{{\pi_{\text{Ref}}}(y_{l}|x)}{{\pi_{\text{Ref}}}(y_{w}|x)}=c.$ Then,
${{\mathcal{R}}}(x,y_{w},y_{l})=1$ if and only if
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l})\leq-\log\sigma(\beta c)$, where
$\sigma$ is the sigmoid function.

######  Remark 4.2.

It is straightforward to extend our analysis to popular variants of DPO. For
illustration, we prove an analogous result for identity preference
optimization (IPO, Azar et al. [2]) in App. A.5.

We prove this result in App. A.4. Our theoretical result allows us to formally
identify the points that will be hard to flip in their rankings. Fig. 3
visualizes the reference model log-ratio for several settings and highlights
that datapoints with even mild ranking errors in the reference model will
require the loss to be reduced to a very low value in order to flip the
ranking. App. F contains examples of hard-to-learn, easy-to-learn, and easy-
to-flip datapoints. We observe that the hard-to-learn datapoints are
substantially longer than the easy ones, and that the easy datapoints
generally contain less ambiguous preference annotations. More generally, our
result motivates the use of stronger $\pi_{\text{Ref}}$ models and iterative
or on-policy variants of DPO [48, 63, 23, 51].

##  5 Ranking Accuracy and Win Rate

Our results on ranking accuracy illuminate how well DPO and RLHF can align to
preference data, but we have not yet related these insights to how the
generative behavior of the model changes during alignment. In particular,
ranking accuracy is a convenient but off-policy metric and is thus not as
widely adopted as the on-policy metric of win rate (see Sec. 2.2). Indeed, one
could maximize the ranking accuracy by learning a strong classifier on the
preference data, but that model may not generate high-quality text. Here, we
explore the gap between on-policy (i.e., generative) and off-policy (i.e.,
classification) behaviors of LLMs through the lens of ranking accuracy and win
rate. Since the DPO objective directly optimizes for ranking accuracy
(Proposition 2.6), the relationship between these two metrics is a direct
reflection of how off-policy training affects on-policy behavior.

We study the relationship between win rate and ranking accuracy in two
settings: (1) during DPO training, and (2) in a DPO variant modulating the
influence of $\pi_{\text{Ref}}$. We measure the win rate on 500 responses to
prompts from the training dataset using the Alpaca Eval GPT-4 [29] auto-
annotator.

#### Setting 1: DPO Training.

We measure the win rate and the ranking accuracy of a Pythia 2.8B model [4]
during DPO training with the same configuration as in Section 4. See Fig. 9
for the results.

#### Setting 2: Attenuating the reference model.

Theorem 4.1 showed that ${\pi_{\text{Ref}}}$ exerts a negative influence on
the ranking accuracy in most cases, so we design a new objective that scales
the reference model log-ratio in ${\mathcal{L}}_{\text{DPO}}$ to further
characterize how win rate and ranking accuracy relate.

$\mathcal{L}_{DPO}^{\gamma}({\pi_{\theta}},{\pi_{\text{Ref}}})=-\mathop{\mathbb{E}}_{(x,y_{w},y_{l})\sim{\mathcal{D}}}\left[\log\sigma\left(\beta\left(\log\frac{{\pi_{\theta}}(y_{w}\mid
x)}{{\pi_{\theta}}(y_{l}\mid x)}+\gamma\log\frac{{\pi_{\text{Ref}}}(y_{l}\mid
x)}{{\pi_{\text{Ref}}}(y_{w}\mid x)}\right)\right)\right]$ (4)

Note that ${\mathcal{L}}_{\text{DPO}}^{\gamma}={\mathcal{L}}_{\text{DPO}}$
(Definition 2.2) when $\gamma=1$, and a larger value of $\gamma$ increases the
role of the reference model. Also, $\gamma$ directly scales $c$ in Theorem
4.1, thereby controlling how easy it is to fit the data and increase the
ranking accuracy. We train a range of Pythia-2.8B models using the
$\mathcal{L}_{DPO}^{\gamma}$ objective for
$\gamma\in\\{0,0.25,0.5,0.75,1.0,1.25,1.5,1.75,2.0\\}$ and measure the ranking
accuracies and win rates of the best model for each $\gamma$ value.666We use
the best hyperparameters obtained from the experiments in Sec. 4.

#### Takeaway: Ranking accuracy correlates with win rate when the model is
close to the reference model.

In both settings, we observe that the win rate and ranking accuracy are highly
correlated with one another in the early phase of training but become anti-
correlated (i.e., ranking accuracy increases but win rate declines) as the
model ${\pi_{\theta}}$ moves away from the reference ${\pi_{\text{Ref}}}$
(Fig. 4). Unlike traditional overfitting, the test loss is continuing to
decline at this point (Fig. 9(b)). Experiments in Fig. 10 with the attenuated
objective in Equation 4 further show that ranking accuracy and win rate trend
together when the influence of the reference model is stronger (i.e., $\gamma$
is larger).

We speculate that when the model is far from the reference model,
regularization toward the reference model can harm the generative capabilities
of the model, which are primarily acquired during pre-training. In other
words, the off-policy behavior of the model can no longer predictably describe
the on-policy generations when the reference model used in the offline
objective is far from the current model. Our findings confirm the fundamental
tradeoff between fitting the preference data and maintaining generative
capabilities acquired during pre-training [21] and align with prior
observations that adding on-policy preference data can make offline learning
more effective [48, 63, 23, 51].

(a) Ranking accuracy and win rate of various Pythia 2.8B checkpoints during
training, versus the distance travelled by the model weights $\theta_{t}$ from
the initialization.

(b) Ranking accuracy and win rate of various $\gamma$-scaled models, versus
the distance travelled by the model weights $\theta_{\gamma}$ from the
initialization.

Figure 4: When the model weights have not travelled far from
$\theta_{\text{Ref}}$, ranking accuracy and win rate increase together.
$\theta_{t}$ represents the model weights at checkpoint $t$, and
$\theta_{\gamma}$ are the model weights for the model trained with
${\mathcal{L}}_{\text{DPO}}^{\gamma}$.

##  6 Related Work

Many works have investigated the role of the preference dataset [54, 59], the
reliability of the evaluations [67, 28], and the confounding factor of
response length [43, 9, 55, 35]. Theoretical works have unified the many
preference learning algorithms into clear taxonomies that permit analysis and,
sometimes, yield new variants [2, 59, 46, 48, 61]. Several works study
idiosyncrasies of preference learning, such as why DPO decreases the
likelihood of both rejected and chosen outputs from the dataset [40, 12, 33]
and why RLHF exhibits vanishing gradients [42]. In contrast, our work
approaches understanding DPO and RLHF through the lens of ranking accuracy,
and our findings emphasize the role of the reference model regularization in
preference learning. Relatedly, SliC-HF [65], CPO [57], and pairwise cringe
loss [58] optimize log probability margins
$\log\pi(y_{w}|x)-\log\pi(y_{l}|x)$, effectively removing the regularization
toward the reference model. Liu et al. [31] recommend using the reference
model at inference time to exert more granular control over the
regularization. Tang et al. [48] also analyzed the role of regularizing toward
a reference model, though our work focuses the effect of this regularization
on ranking accuracy. App. B discusses additional related works.

##  7 Discussion

Our work highlights the significant but nuanced relationship between
preference learning and ranking accuracy. We have demonstrated both
theoretically and empirically that RLHF and DPO struggle to teach the model to
correctly rank preferred and dispreferred outputs, even in the training
dataset. Although the learning objective promotes high ranking accuracy in
theory (Proposition 2.6), we observed a prominent _alignment gap_ resulting
from the poor conditioning of reference models. We then drew connections
between the off-policy nature of ranking accuracy and the on-policy
evaluations of win rate, identifying specific scenarios in which on-policy
behavior can or cannot be reliably predicted by off-policy behavior. App. B
details the limitations of our work.

#### Connections to Safety

Our work shows that it is difficult to steer pre-trained LLMs to adhere to
even the preference data used for training. When LLMs are used to judge
responses from other models [29, 67] or to improve their own abilities [32,
63], poor ranking accuracies can induce strong negative feedback loops that
are costly to mitigate.

We also observe that win rate does not monotonically increase during training
(Fig. 9(a)), despite the decrease in both train and test loss (Fig. 9(b)) and
the modest gain in ranking accuracy (Fig. 9(a)). As such, it is clear that we
still do not understand the behaviors of preference learning. For example,
others have observed that DPO can cause the likelihoods of both chosen and
rejected outputs to decrease [40, 12, 33, 34], which implies that the policy
must be moving probability mass to possibly undesirable sequences outside the
data distribution. Moreover, our investigation of the non-monotonic
relationship between ranking accuracy and win rate emphasizes the need for
concrete evaluations that can more reliably and transparently measure the
success of preference learning.

#### Future Work

Our theoretical results only describe the behavior of the model on the
preference data used during training, but they can serve as a starting point
for understanding generalization to different distributions of data,
especially the one prescribed by the model itself [46]. Furthermore, we hope
to analyze the optimization dynamics of preference learning, given the
intriguing relationship observed between ranking accuracy and win rate. For
instance, identifying when the win rate begins to diverge from the ranking
accuracy can motivate adding fresh on-policy training data. Our initial
investigation into ranking accuracy also suggests that it is worthwhile to
explore how alignment techniques interact with other calibration metrics.

##  8 Acknowledgements

We thank Sanjeev Arora, Tianyu Gao, Eric Mitchell, Richard Pang, and Mengzhou
Xia for helpful discussions during the development of this project. We also
thank Nikita Nangia, Eshaan Nichani, and Alexander Wettig for their help in
proofreading the work.

This work was supported by National Science Foundation Award 1922658, the
Samsung Advanced Institute of Technology (under the project Next Generation
Deep Learning: From Pattern Recognition to AI), NSF CAREER Award 2145542, ONR
N00014-23-1-2634, and Google. SM is additionally supported by ONR, NSF, and
DARPA. We also thank Princeton Language and Intelligence (PLI) for computing
resources and OpenAI credits, and NYU High Performance Computing (HPC) for
computing resources and in-kind support.

## References

  * Alex Havrilla [2023] Alex Havrilla.  synthetic-instruct-gptj-pairwise (revision cc92d8d), 2023.  URL https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise.
  * Azar et al. [2023] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A general theoretical paradigm to understand learning from human preferences, 2023.
  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
  * Biderman et al. [2023] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR, 2023.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.  ISSN 00063444.  URL http://www.jstor.org/stable/2334029.
  * Cui et al. [2023] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback, 2023.
  * Dong et al. [2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang.  RAFT: Reward ranked finetuning for generative foundation model alignment.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=m7p5O7zblY.
  * Dubois et al. [2023] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  * Dubois et al. [2024] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024.
  * Ethayarajh et al. [2022] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.  Understanding dataset difficulty with $\mathcal{V}$-usable information.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 5988–6008. PMLR, 17–23 Jul 2022.
  * Ethayarajh et al. [2024] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  Kto: Model alignment as prospect theoretic optimization, 2024.
  * Feng et al. [2024] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei.  Towards analyzing and understanding the limitations of dpo: A theoretical perspective, 2024.
  * Ganguli et al. [2022] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.
  * Geng et al. [2023] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song.  Koala: A dialogue model for academic research.  Blog post, April 2023.
  * Go et al. [2023] Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman.  Aligning language models with preferences through f-divergence minimization.  _arXiv preprint arXiv:2302.08215_ , 2023.
  * Groeneveld et al. [2024] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi.  Olmo: Accelerating the science of language models.  _arXiv preprint_ , 2024.  URL https://api.semanticscholar.org/CorpusID:267365485.
  * Hong et al. [2024] Jiwoo Hong, Noah Lee, and James Thorne.  Orpo: Monolithic preference optimization without reference model, 2024.
  * Hosking et al. [2024] Tom Hosking, Phil Blunsom, and Max Bartolo.  Human feedback is not gold standard, 2024.
  * Hüllermeier et al. [2008] Eyke Hüllermeier, Johannes Fürnkranz, Weiwei Cheng, and Klaus Brinker.  Label ranking by learning pairwise preferences.  _Artificial Intelligence_ , 172(16):1897–1916, 2008.  ISSN 0004-3702.  doi: https://doi.org/10.1016/j.artint.2008.08.002.  URL https://www.sciencedirect.com/science/article/pii/S000437020800101X.
  * Ivison et al. [2023] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.
  * Jaques et al. [2017] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck.  Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control.  In _International Conference on Machine Learning_ , pages 1645–1654. PMLR, 2017.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Kim et al. [2024] Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park.  sdpo: Don’t use your data all at once, 2024.
  * Kirk et al. [2024] Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale.  The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models, 2024.
  * Köpf et al. [2023] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick.  Openassistant conversations - democratizing large language model alignment.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=VSJotgbPHF.
  * Korbak et al. [2022] Tomasz Korbak, Ethan Perez, and Christopher L Buckley.  Rl with kl penalties is better viewed as bayesian inference.  _arXiv preprint arXiv:2205.11275_ , 2022.
  * Lambert et al. [2023] Nathan Lambert, Lewis Tunstall, Nazneen Rajani, and Tristan Thrush.  Huggingface h4 stack exchange preference dataset, 2023.  URL https://huggingface.co/datasets/HuggingFaceH4/stack-exchange-preferences.
  * Lambert et al. [2024] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi.  Rewardbench: Evaluating reward models for language modeling, 2024.
  * Li et al. [2023] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models, 2023.
  * Li et al. [2024] Ziniu Li, Tian Xu, and Yang Yu.  Policy optimization in rlhf: The impact of out-of-preference data, 2024.
  * Liu et al. [2024] Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel.  Decoding-time realignment of language models, 2024.
  * Madaan et al. [2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=S37hOerQLB.
  * Pal et al. [2024] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White.  Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024.
  * Pang et al. [2024] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston.  Iterative reasoning preference optimization, 2024.
  * Park et al. [2024] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn.  Disentangling length from quality in direct preference optimization, 2024.
  * Peters and Schaal [2007] Jan Peters and Stefan Schaal.  Reinforcement learning by reward-weighted regression for operational space control.  In _Proceedings of the 24th international conference on Machine learning_ , pages 745–750, 2007.
  * Pezeshkpour and Hruschka [2023] Pouya Pezeshkpour and Estevam Hruschka.  Large language models sensitivity to the order of options in multiple-choice questions.  _arXiv preprint arXiv:2308.11483_ , 2023.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=HPuSIXJaa9.
  * Rafailov et al. [2024] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.  From $r$ to $q^{*}$: Your language model is secretly a q-function, 2024.
  * Ramamurthy et al. [2023] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi.  Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=8aHzds2uUyB.
  * Razin et al. [2024] Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin.  Vanishing gradients in reinforcement finetuning of language models.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=IcVNBR7qZi.
  * Singhal et al. [2023] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett.  A long way to go: Investigating length correlations in rlhf, 2023.
  * Song et al. [2024] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang.  Preference ranking optimization for human alignment.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(17):18990–18998, Mar. 2024.  doi: 10.1609/aaai.v38i17.29865.  URL https://ojs.aaai.org/index.php/AAAI/article/view/29865.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 3008–3021. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf.
  * Tajwar et al. [2024] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar.  Preference fine-tuning of llms should leverage suboptimal, on-policy data, 2024.
  * Tang et al. [2024a] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney.  Understanding the performance gap between online and offline alignment algorithms, 2024a.
  * Tang et al. [2024b] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot.  Generalized preference optimization: A unified approach to offline alignment.  _arXiv preprint arXiv:2402.05749_ , 2024b.
  * Team et al. [2024] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.  Gemma: Open models based on gemini research and technology, 2024.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Tran et al. [2023] Hoang Tran, Chris Glaze, and Braden Hancock.  Iterative dpo alignment.  Technical report, Snorkel AI, 2023.
  * Tunstall et al. [2023] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf.  The alignment handbook.  https://github.com/huggingface/alignment-handbook, 2023.
  * Vembu and Gärtner [2010] Shankar Vembu and Thomas Gärtner.  Label ranking algorithms: A survey.  In _Preference learning_ , pages 45–64. Springer, 2010.
  * Wang et al. [2024] Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang.  Secrets of rlhf in large language models part ii: Reward modeling, 2024.
  * Wang et al. [2023] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  How far can camels go? exploring the state of instruction tuning on open resources, 2023.
  * Wirth et al. [2017] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz.  A survey of preference-based reinforcement learning methods.  _Journal of Machine Learning Research_ , 18(136):1–46, 2017.  URL http://jmlr.org/papers/v18/16-634.html.
  * Xu et al. [2024a] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim.  Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation, 2024a.
  * Xu et al. [2024b] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston.  Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss, 2024b.
  * Xu et al. [2024c] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu.  Is dpo superior to ppo for llm alignment? a comprehensive study, 2024c.
  * Xu et al. [2024d] Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang.  Perils of self-feedback: Self-bias amplifies in large language models, 2024d.
  * Yang et al. [2024] Joy Qiping Yang, Salman Salamatian, Ziteng Sun, Ananda Theertha Suresh, and Ahmad Beirami.  Asymptotics of language model alignment, 2024.
  * Yuan et al. [2023] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  Rrhf: Rank responses to align language models with human feedback.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 10935–10950. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/23e6f78bdec844a9f7b6c957de2aae91-Paper-Conference.pdf.
  * Yuan et al. [2024] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models, 2024.
  * Zeng et al. [2024] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen.  Evaluating large language models at evaluating instruction following.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=tr0KidwPLc.
  * Zhao et al. [2023a] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu.  Slic-hf: Sequence likelihood calibration with human feedback, 2023a.
  * Zhao et al. [2023b] Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu.  Calibrating sequence likelihood improves conditional language generation.  In _The Eleventh International Conference on Learning Representations_ , 2023b.  URL https://openreview.net/forum?id=0qSOodKmJaN.
  * Zheng et al. [2023a] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging LLM-as-a-judge with MT-bench and chatbot arena.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023a.  URL https://openreview.net/forum?id=uccHPGDlao.
  * Zheng et al. [2024] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Zheng et al. [2023b] Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al.  Secrets of rlhf in large language models part i: Ppo.  _arXiv preprint arXiv:2307.04964_ , 2023b.

##  Appendix A Proofs and Additional Results

######  Assumption A.1 (Bradley-Terry [5]).

Given a prompt $x$ and two possible continuations $y_{1}$ and $y_{2}$, the
ground truth human preference distribution satisfies

$\mathbb{P}(y_{1}\succ y_{2}\mid
x)=\frac{\exp(r^{*}(x,y_{1}))}{\exp(r^{*}(x,y_{1}))+\exp(r^{*}(x,y_{2}))}$ (5)

for some ground truth reward model $r^{*}$.

###  A.1 Proof of Proposition 2.6

######  Proposition A.2 (Reproduced from Proposition 2.6).

Recall the definition of $y_{w}$, $y_{l}$ in Definition 2.1. If
${\pi_{\text{Ref}}}(y_{w}\mid x)\geq{\pi_{\text{Ref}}}(y_{l}\mid x)$ and
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})\leq
0.6$, then ${{\mathcal{R}}}(x,y_{w},y_{l})=1$.

###### Proof.

For notational convenience, write the log probability ratios as

${a_{x}}=\log\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{w}|x)},\ \
{b_{x}}=\log\frac{{\pi_{\theta}}(y_{l}|x)}{{\pi_{\text{Ref}}}(y_{l}|x)}.$

Then we can express the probability that the model places on each response as

${\pi_{\theta}}(y_{w}|x)={\pi_{\text{Ref}}}(y_{w}|x)e^{{a_{x}}},\ \
{\pi_{\theta}}(y_{l}|x)={\pi_{\text{Ref}}}(y_{l}|x)e^{{b_{x}}}.$

Suppose for $(x,y_{w},y_{l})$,
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})\leq
0.6$. Expanding the DPO loss, we have

$\displaystyle{\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})$
$\displaystyle=-\log\sigma(\beta({a_{x}}-{b_{x}}))\leq 0.6.$

Rearranging the inequality and exponentiate on both sides, we have

$\displaystyle\sigma(\beta({a_{x}}-{b_{x}}))\geq\exp(-0.6).$

Note that $\sigma(0)=0.5<\exp(-0.6)\leq\sigma(\beta({a_{x}}-{b_{x}}))$. Since
the logistic function is monotonic, this implies that
$0\leq\beta({a_{x}}-{b_{x}})$ and ${a_{x}}\geq{b_{x}}$.

Writing out this last inequality,

$\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{w}|x)}\geq\frac{{\pi_{\theta}}(y_{l}|x)}{{\pi_{\text{Ref}}}(y_{l}|x)}\Rightarrow\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}\geq\frac{{\pi_{\text{Ref}}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{l}|x)}\geq
1,$

where the last inequality is by the assumption on ${\pi_{\text{Ref}}}$. We
conclude that ${{\mathcal{R}}}(x,y_{w},y_{l})=1$ by definition. ∎

###  A.2 Proof of Theorem 3.1

######  Theorem A.3 (Simulating Perfect RLHF).

Fix a reference model ${\pi_{\text{Ref}}}$ and an aggregated preference
datapoint $(x,y_{w},y_{l})\sim{\mathcal{D}}$. Assume the dataset includes the
ground-truth human preferences: that is,
$\alpha(x,y_{w},y_{l})=\mathbb{P}(y_{w}\succ y_{l})$, and that these
preferences obey the Bradley-Terry model (Equation 5). Let $\pi^{*}$ be the
(adequately expressive) model that perfectly minimizes the DPO objective on
$(x,y_{w},y_{l})$, or perfectly maximizes the PPO objective on the optimal
reward function as described in Section 2.1. Then, the optimal policy
$\pi^{*}$ satisfies

$\frac{\pi^{*}(y_{w}\mid x)}{\pi^{*}(y_{l}\mid
x)}=\frac{{\pi_{\text{Ref}}}({y_{w}\mid x})}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}\left(\frac{\alpha(x,y_{w},y_{l})}{1-\alpha(x,y_{w},y_{l})}\right)^{1/\beta}$
(6)

where $\beta$ is a hyperparameter in the DPO and RLHF objectives.

###### Proof.

We first prove the statement for DPO. Following the notation from [39], fix a
reward function $r$, let $\pi_{r}(y|x)$ be the optimal model under the KL-
constrained RL objective given the reward function $r$. We can express
$r(x,y)$ in terms of $\pi_{r}(y|x)$and $\pi_{ref}(y|x)$:

$r(x,y)=\beta\log\frac{\pi_{r}(y|x)}{\pi_{ref}(y|x)}+\beta\log Z(x),$

where $Z(x)$ is the partition function for prompt $x$. Then, under the
Bradley-Terry model, the probability of preferring $y_{w}$ over $y_{l}$ under
the model $\pi_{r}$ is

$\displaystyle\pi_{r}(y_{w}\succ y_{l}|x)$
$\displaystyle=\frac{1}{1+\exp(r(x,y_{l})-r(x,y_{w}))}$
$\displaystyle=\frac{1}{1+\exp(\beta\log\frac{\pi_{r}(y_{l}|x)}{\pi_{ref}(y_{l}|x)}-\beta\log\frac{\pi_{r}(y_{w}|x)}{\pi_{ref}(y_{w}|x)})}.$

Given the ground-truth human preferences, DPO’s maximum likelihood objective
minimizes the binary classification loss on $(x,y_{w},y_{l})$:

$\min_{\pi}\alpha(x,y_{w},y_{l})\log\pi(y_{w}\succ
y_{l}|x)+(1-\alpha(x,y_{w},y_{l}))\log(1-\pi(y_{w}\succ y_{l}|x)).$

Let $\pi^{*}$ denote an optimal policy from the loss above, the optimal
preference probabilities satisfy

$\pi^{*}(y_{w}\succ y_{l}|x)=\alpha(x,y_{w},y_{l}),$

and the optimal policy in turn satisfies

$\frac{1}{1+\exp(\beta\log\frac{\pi^{*}(y_{l}|x)}{\pi_{ref}(y_{l}|x)}-\beta\log\frac{\pi^{*}(y_{w}|x)}{\pi_{ref}(y_{w}|x)})}=\alpha(x,y_{w},y_{l}).$

Rearranging, we have

$\frac{\alpha(x,y_{l},y_{w})}{\alpha(x,y_{w},y_{l})}=\exp\left(\beta\log\frac{\pi^{*}(y_{l}|x)}{\pi_{ref}(y_{l}|x)}-\beta\log\frac{\pi^{*}(y_{w}|x)}{\pi_{ref}(y_{w}|x)}\right),$

taking a log on both sides and divide by $\beta$,

$\frac{1}{\beta}\log\frac{\alpha(x,y_{l},y_{w})}{\alpha(x,y_{w},y_{l})}=\log\frac{\pi^{*}(y_{l}|x)\pi_{ref}(y_{w}|x)}{\pi_{ref}(y_{l}|x)\pi^{*}(y_{w}|x)},$

and finally exponentiating both sides,

$\frac{\pi^{*}(y_{l}|x)}{\pi^{*}(y_{w}|x)}=\frac{\pi_{ref}(y_{l}|x)}{\pi_{ref}(y_{w}|x)}\left(\frac{\alpha(x,y_{l},y_{w})}{\alpha(x,y_{w},y_{l})}\right)^{1/\beta},$

the result follows by taking an inverse.

Now we show the result for RLHF, starting from the optimal solution of the KL-
constrained reward maximization problem under the reward $r_{\phi}$, as
derived in [39]:

$\pi^{*}(y|x)\propto{\pi_{\text{Ref}}}(y|x)\exp\left(\frac{1}{\beta}r_{\phi}(x,y)\right).$

The condition is straightforward to derive

$\displaystyle\frac{\pi^{*}(y_{w}|x)}{\pi^{*}(y_{l}|x)}$
$\displaystyle=\frac{{\pi_{\text{Ref}}}(y_{w}|x)\,\mathrm{exp}\left(\frac{1}{\beta}r_{\phi}(x,y_{w})\right)}{{\pi_{\text{Ref}}}(y_{l}\mid
x)\,\mathrm{exp}\left(\frac{1}{\beta}r_{\phi}(x,y_{l})\right)}$ (7)
$\displaystyle=\frac{{\pi_{\text{Ref}}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}\left(\mathrm{exp}(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l})\right)^{1/\beta}$
(8)
$\displaystyle=\frac{{\pi_{\text{Ref}}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}\left(\frac{\alpha(x,y_{w},y_{l})}{1-\alpha(x,y_{w},y_{l})}\right)^{1/\beta},$
(9)

where the last equality is due to the Bradley-Terry model. ∎

###  A.3 Proof of Corollary 3.3

######  Corollary A.4 (Reproduces Corollary 3.3).

Given a reference model ${\pi_{\text{Ref}}}$, the DPO or RLHF hyperparameter
$\beta$, a dataset of aggregated preferences
${\mathcal{D}}=\\{(x,y_{w},y_{l})\\}$ and their corresponding rater
proportions $\alpha(x,y_{w},y_{l})$, the ranking accuracy of the optimum of
the RLHF or DPO objective $\pi^{*}$ is given by

${{\mathcal{R}}^{*}}({\mathcal{D}};{\pi_{\text{Ref}}})=\underset{(x,y_{w},y_{l})\sim{\mathcal{D}}}{\mathbb{E}}\left[\mathbbm{1}\left[\frac{{\pi_{\text{Ref}}}({y_{w}\mid
x})}{{\pi_{\text{Ref}}}(y_{l}\mid
x)}\left(\frac{\alpha(x,y_{w},y_{l})}{1-\alpha(x,y_{w},y_{l})}\right)^{1/\beta}>1\right]\right]$
(10)

where $\mathbbm{1}[\cdot]$ is the indicator function. When computed on length-
normalized likelihoods from $\tilde{\pi_{\text{Ref}}}$, we denote the
idealized ranking accuracy as $\tilde{{\mathcal{R}}^{*}}$.

###### Proof.

We can see from the definition of ranking accuracy (Definition 2.3) that the
accuracy on a datapoint $(x,y_{w},y_{l})$ is 1 when $\pi(y_{w}\mid
x)>\pi(y_{l}\mid x)$. In other words, $\pi(y_{w}\mid x)/\pi(y_{l}\mid x)>1$
ensures that the ranking accuracy ${{\mathcal{R}}}(x,y_{w},y_{l})=1$. Theorem
3.1 shows a formula for the ratio of the optimal policies, and plugging this
in to the condition for ranking accuracy immediately yields the given formula
for ${{\mathcal{R}}^{*}}$. ∎

###  A.4 Proof of Theorem 4.1

######  Theorem A.5.

Consider an aggregated preference datapoint $(x,y_{w},y_{l})$ such that the
reference model log-ratio is some constant $c$, i.e.

$\log\frac{{\pi_{\text{Ref}}}(y_{l}|x)}{{\pi_{\text{Ref}}}(y_{w}|x)}=c.$

Then, ${{\mathcal{R}}}(x,y_{w},y_{l})=1$ if and only if
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l})\leq-\log\sigma(\beta c)$, where
$\sigma$ is the sigmoid function.

###### Proof.

Recall that we can break down the DPO loss into the model log-ratio and the
reference model log-ratio as follows:

$\displaystyle{\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})$
$\displaystyle=-\log\sigma\left(\beta\left(\log\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}+\log\frac{{\pi_{\text{Ref}}}(y_{l}|x)}{{\pi_{\text{Ref}}}(y_{w}|x)}\right)\right)$
$\displaystyle=-\log\sigma\left(\beta\left(\log\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}+c\right)\right)$
(by assumption)

Observe that if
${\mathcal{L}}_{\text{DPO}}(x,y_{w},y_{l};{\pi_{\theta}},{\pi_{\text{Ref}}})\leq-\log\sigma(\beta
c)$, then

$\log\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}+c\geq c,$

by the monotonicity of the log and sigmoid functions. This implies that
$\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}\geq 1$ and
${{\mathcal{R}}}(x,y_{w},y_{l})=1$.

Now we show the other direction. Suppose ${{\mathcal{R}}}(x,y_{w},y_{l})=1$,
then ${\pi_{\theta}}(y_{w}|x)\geq{\pi_{\theta}}(y_{l}|x)$ and

$\log\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}+c\geq c.$

Using this relationship, we have

$\log\sigma\left(\beta\left(\log\frac{{\pi_{\theta}}(y_{w}|x)}{{\pi_{\theta}}(y_{l}|x)}+c\right)\right)\geq\log\sigma(\beta
c),$

and the other direction follows by taking the negative of both sides. ∎

###  A.5 Extending Theorem 4.1 to IPO

We extend our main result on DPO to also describe the IPO objective, formally
defined below.

######  Definition A.6 (Identity Preference Optimization, a.k.a. IPO [2]).

The Identity Preference Optimization (IPO) loss is defined as
$\mathcal{L}_{\text{IPO}}(x,y_{w},y_{l})=\left(\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{Ref}}(y_{w}|x)}-\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{Ref}}(y_{l}|x)}-\frac{1}{2\tau}\right)^{2}$.

######  Proposition A.7.

Under the IPO loss, if

$\log\frac{{\pi_{\text{Ref}}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{l}|x)}<-\frac{1}{2\tau},$

then zero IPO loss implies that ${{\mathcal{R}}}(x,y_{w},y_{l})=0$. On the
other hand, if

$\log\frac{{\pi_{\text{Ref}}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{l}|x)}=c\geq-\frac{1}{2\tau},$

then
$\mathcal{L}_{\text{IPO}}(x,y_{w},y_{l})\leq\left(c+\frac{1}{2\tau}\right)^{2}$
guarantees that ${{\mathcal{R}}}(x,y_{w},y_{l})=1$.

Different from the DPO loss, the IPO loss is a regression loss where the
optimal model log-ratio has a constant margin over the reference model log-
ratio. This can be seen by the following decomposition of the IPO loss:

$\displaystyle\mathcal{L}_{\text{IPO}}(x,y_{w},y_{l})$
$\displaystyle=\left(\log\frac{{\pi_{\theta}}(y_{w}|x)}{\pi_{\theta}(y_{l}|x)}-\log\frac{{\pi_{\text{Ref}}}(y_{w}|x)}{{\pi_{\text{Ref}}}(y_{l}|x)}-\frac{1}{2\tau}\right)^{2}.$

Clearly, the optimization target of the model log-ratio is the sum of the
reference model log-ratio and the margin $\frac{1}{2\tau}$.

From Figure 3, we can see that the reference model can have a large bias
towards the dispreferred completion, represented by the large negative values
of the reference model log-ratio (note that in the figure, the reference model
log-ratio has the dispreferred completion in the numerator). Therefore, for
the optimal model under the IPO loss to have perfect ranking accuracy, the
margin $\frac{1}{2\tau}$ needs to be large enough to overcome this bias for
all datapoints. Alternatively, a per-example margin dependent on the reference
model log-ratio of the example can be used.

###  A.6 Generalization of the Ranking Accuracy for Preference Datasets with
$n>2$ Outputs

Some datasets (_e.g._ UltraFeedback [6]) contain examples with more than two
responses per prompt $x$. In these cases, we extend our definition of ranking
accuracy (Def. 2.3) to exact-match of the rankings over _all choices_. Suppose
each aggregated datapoint consists of a prompt $x$ and $n$ responses
$(y_{1},\cdots,y_{n})$. Since the question of how to best aggregate rankings
over multiple raters (when $n>2$) is an open research question, we assume that
there already exists some aggregated social ranking
$\bm{\rho}(x,y_{1},\cdots,y_{n})=(\rho_{i}(x,y_{1},\cdots,y_{n}))_{i=1}^{n}\in\Pi[n]$
where $\rho_{i}(x,y_{1},\cdots,y_{n})<\rho_{j}(x,y_{1},\cdots,y_{n})$ implies
that $y_{i}$ is preferred over $y_{j}$. Now let the ranking assigned by a
policy $\pi_{\theta}$ be
$\bm{\nu}(x,y_{1},\cdots,y_{n})=(\nu_{i}(x,y_{1},\cdots,y_{n}))_{i=1}^{n}\in\Pi[n]$,
where $\pi_{\theta}(y_{i}|x)>\pi_{\theta}(y_{j}|x)$ if and only if
$\nu_{i}(x,y_{1},\cdots,y_{n})<\nu_{j}(x,y_{1},\cdots,y_{n})$. Then the
generalized ranking accuracy is defined as follows:

######  Definition A.8 (Ranking Accuracy for $n>2$).

$\mathcal{R}_{n>2}(x,y_{1},\cdots,y_{n};\pi_{\theta})=\mathbbm{1}\left[\bm{\rho}(x,y_{1},\cdots,y_{n})=\bm{\nu}(x,y_{1},\cdots,y_{n})\right]$
(11)

where $\mathbbm{1}[\cdot]$ is the indicator function. Analogously, the ranking
accuracy over a dataset $\mathcal{D}=\\{(x,y_{1},\cdots,y_{n})\\}$ is

$\underset{(x,y_{1},\cdots,y_{n})\sim\mathcal{D}}{\mathbb{E}}\mathcal{R}_{n>2}(x,y_{1},\cdots,y_{n};\pi_{\theta})$
(12)

##  Appendix B Related Works and Limitations

###  B.1 Related Works

#### On-policy and Off-policy Preference Learning

Preference-tuning LLMs originally required using an on-policy algorithm [45],
but many recent works have derived off-policy methods that can use a static
preference dataset for supervision [39, 11, 17, 35]. Off-policy methods are
preferred for their efficiency and ease of implementation, but several works
have suggested that on-policy methods are superior [46, 59, 47, 30]. Several
iterative training methods aim to bridge this gap, where the reference model
and the dataset are refreshed during the alignment procedure to contain
annotated preferences on generations from the model at that point in training
[63, 23, 51]. These intuitions align strongly with our observation that win
rate and ranking accuracy, and thus, on-policy and off-policy behavior, are
strongly correlated when the model is close to the reference model.

###  B.2 Limitations

Although we reproduce our main results (Sec. 4) on three types of models
across three seeds each, these models are trained on a single dataset due to
the computational constraints. Our theoretical results lead us to believe that
our findings would generalize to other datasets, but empirical verification is
still valuable. As mentioned previously, our work is optimization-agnostic and
only describes model behavior on the training dataset, making it difficult to
draw general claims about other distributions. In particular, we cannot
rigorously describe the generative capabilities, though Sec. 5 initiates an
investigation into when off-policy behavior can describe on-policy behaviors.

##  Appendix C Experimental Details for Computing Ranking Accuracy of Open-
Access LLMs

###  C.1 Implementation of Ranking Accuracy

We evaluate ranking accuracy for a wide range of LLMs by evaluating the
_likelihoods_ that each model $\pi_{\theta}$ assigns to $y_{w}$ given $x$ and
$y_{l}$ given $x$, as described in Def. 2.3. However, $x$, $y_{w}$, and
$y_{l}$ are sequences, so we compute the sequence likelihoods by factorizing
the sequence likelihood into a product of the conditional token likelihoods,
like so:

$\pi_{\theta}(y|x)=\prod_{t=1}^{|y|}\pi_{\theta}(y_{t}|x;y_{<t})$ (13)

We use PyTorch and the Hugging Face `transformers` and `datasets` libraries to
compute all ranking accuracies. For each dataset that we evaluate ranking
accuracy on, we take a random sample of 1000 examples.

#### Handling Ties

In some datasets, such as the cross-human-annotated validation split of the
Alpaca Farm dataset [8] (described in Sec. C.2), ties exist in the human
annotations. When this is the case, we make a mild adjustment in the
calculation of ranking accuracy to accommodate ties.

######  Definition C.1 (Ranking Accuracy with Ties).

${{\mathcal{R}}}_{\text{Ties}}(x,y_{1},y_{2};\pi_{\theta})=\begin{cases}\mathbbm{1}[|{\pi_{\theta}}(y_{1}|x)-{\pi_{\theta}}(y_{2}|x)|<\epsilon]&\text{if
}\,\alpha(x,y_{1},y_{2})=0.5\\\
\mathbbm{1}[{\pi_{\theta}}(y_{1}|x)>{\pi_{\theta}}(y_{2}|x)]&\text{if
}\,\alpha(x,y_{1},y_{2})>0.5\\\
\mathbbm{1}[{\pi_{\theta}}(y_{1}|x)<{\pi_{\theta}}(y_{2}|x)]&\text{if
}\,\alpha(x,y_{1},y_{2})<0.5\end{cases}$ (14)

where $\mathbbm{[}\cdot]$ is the indicator function. In other words, if the
human annotations indicate tied preferences between $y_{1}$ and $y_{2}$, then
${\pi_{\theta}}$ achieves the correct ranking if and only if it assigns
$y_{1}$ and $y_{2}$ approximately the same likelihood (within some tolerance
level $\epsilon$). For all other cases, the ${{\mathcal{R}}}_{\text{Ties}}$ is
equivalent to ${{\mathcal{R}}}$.

Throughout this paper, we use a tolerance $\epsilon$ of 0.01. In Fig. 1 we
provide ranking accuracies only for examples without ties, but we provide the
numbers on the full datasets below.

#### Length Normalization

To compute the length-normalized ranking accuracy ($\tilde{{\mathcal{R}}}$),
we replace $\pi_{\theta}(y|x)$ in Defs. 2.3 and A.8 with
$\widetilde{{\pi_{\theta}}}(y|x)$, where

$\widetilde{{\pi_{\theta}}}(y|x)=\left(\prod_{t=1}^{|y|}\pi_{\theta}(y_{t}|x;y_{<t})\right)^{1/|y|}.$
(15)

###  C.2 Datasets

  * •

HH-RLHF [3] (helpful set) consists of two model responses for each query
(often the history of a multi-turn conversation between human and chatbot),
based on generations from three different classes of models (context-distilled
52B model, same model with rejection sampling using a reward model, RLHF-
finetuned models). Queries and preferences annotations are obtained from
crowdworkers.

  * •

Synthetic Instruct GPT-J Pairwise [1] is a synthetic dataset of queries and
pairwise model generations spanning different subjects.

  * •

StackExchange Preferences [27] consists of questions and answers from Stack
Exchange, where within a pair of answers, the preference between two answers
is determined by a function of the number of upvotes and whether the answer
was selected.

  * •

UltraFeedback [6] consists of model-generated responses from four different
language models out of a larger set of models (meaning a different set of
models is considered for different samples). Queries are obtained by a mixture
of existing QA datasets, and the preference is annotated by GPT-4.

  * •

Stanford Human Preferences [10] is a dataset created from Reddit posts across
different subject matters, where within a pair a response is considered
preferred to another if was created later and has more upvotes.

  * •

Alpaca Farm Validation [8] is sourced from the Alpaca Eval dataset, but with
new splits repurposed for training preference-tuned models. Additionally, we
choose to use the validation split because it contains human cross-annotations
(_i.e._ multiple human ratings per triple of $(x,y_{1},y_{2})$). This
particular split can be found at https://huggingface.co/datasets/tatsu-
lab/alpaca_eval/blob/main/alpaca_farm_human_crossannotations.json. The
original dataset, Alpaca Eval [29], is a mixture of several test sets
including Open Assistant [25], a dataset of human-constructed chatbot
conversation turns, HH-RLHF [3], and the Vicuna [68] and Koala test sets [14],
where the former consists of user-shared queries from ShareGPT, and the latter
consists of human queries from online interactions.

###  C.3 Full Results

We give the full set of length-normalized and non-length-normalized ranking
accuracies for 16 open-access LLMs across six datasets in Tables 2, 3, and 4.
For the UltraFeedback [6] and StackExchange Preferences [27] datasets, we use
the generalized definition of ranking accuracy for $n>2$ outputs instead (Def.
A.8). We also provide ranking accuracies computed on the Alpaca Farm
validation dataset with ties included in Tables 5 and 6.

Table 2: Length-normalized and non-length-normalized ranking accuracies for the Anthropic Helpful and Harmless (HH-RLHF; Bai et al. [3]) and Synthetic Instruct GPT-J Pairwise [1] datasets. The latter contains only a training split. Model |  Anthropic HH-RLHF |  Synthetic Instruct GPT-J Pairwise
---|---|---
|  Test |  Train |  Train
|  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$
Gemma-7b-it |  52.5% |  46.7% |  52.6% |  47.0% |  56.6% |  76.5%
Gemma-7b |  51.9% |  46.5% |  53.7% |  46.6% |  93.1% |  76.9%
Gpt2 |  49.9% |  46.0% |  52.7% |  46.8% |  81.5% |  65.9%
Llama-2-7b-chat-hf |  52.0% |  46.2% |  54.1% |  46.3% |  94.5% |  75.4%
Llama-2-7b-hf |  52.1% |  46.3% |  53.8% |  46.3% |  93.6% |  77.9%
Mistral-7B-v0.1 |  52.9% |  45.8% |  54.4% |  46.4% |  93.5% |  77.4%
OLMo-7B |  51.0% |  45.5% |  53.6% |  46.6% |  91.9% |  78.2%
Pythia-1.4b |  51.2% |  46.2% |  53.2% |  46.3% |  88.0% |  68.2%
Pythia-2.8b |  51.0% |  46.4% |  53.5% |  46.6% |  90.3% |  69.5%
Tulu-2-7b |  51.8% |  46.7% |  54.6% |  46.3% |  94.3% |  77.8%
Tulu-2-dpo-7b |  51.4% |  46.1% |  54.0% |  46.2% |  94.4% |  77.7%
Vicuna-7b-v1.5 |  52.2% |  46.4% |  54.7% |  46.0% |  93.6% |  77.4%
Zephyr-7b-dpo |  50.9% |  46.5% |  55.4% |  46.5% |  95.1% |  79.6%
Zephyr-7b-sft |  49.5% |  46.2% |  55.6% |  46.2% |  94.3% |  78.9%
Table 3: Length-normalized and non-length-normalized ranking accuracies for the StackExchange Preferences [27] and UltraFeedback [6] datasets. Both datasets contain only a training split. Model |  StackExchange Preferences |  UltraFeedback
---|---|---
|  Train |  Train
|  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$
Gemma-7b-it |  32.6% |  12.8% |  2.7% |  1.4%
Gemma-7b |  31.2% |  22.4% |  2.6% |  1.5%
Gpt2 |  29.5% |  21.0% |  2.1% |  1.4%
Llama-2-7b-chat-hf |  32.1% |  22.1% |  2.5% |  1.2%
Llama-2-7b-hf |  33.0% |  22.6% |  2.7% |  1.7%
Mistral-7B-v0.1 |  31.8% |  22.5% |  2.4% |  1.7%
OLMo-7B |  31.9% |  22.3% |  2.6% |  1.4%
Pythia-1.4b |  28.3% |  12.7% |  2.1% |  1.4%
Pythia-2.8b |  31.9% |  21.7% |  2.4% |  1.4%
Tulu-2-7b |  32.8% |  22.4% |  2.9% |  1.3%
Tulu-2-dpo-7b |  32.5% |  22.1% |  2.7% |  1.6%
Vicuna-7b-v1.5 |  33.2% |  22.2% |  2.5% |  1.4%
Zephyr-7b-dpo |  33.0% |  22.3% |  3.5% |  1.8%
Zephyr-7b-sft |  32.2% |  22.3% |  2.7% |  1.5%
Table 4: Length-normalized and non-length-normalized ranking accuracies for the Stanford Human Preferences (SHP; Ethayarajh et al. [10]) and Alpaca Farm validation [8] datasets. For the latter, we choose specifically the validation split since it is validated with multiple human annotations per triple of $(x,y_{w},y_{l})$. Examples with ties are not included. Model |  Stanford Human Preferences |  Alpaca Farm
---|---|---
|  Test |  Train |  Validation
|  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$
Gemma-7b-it |  44.1% |  24.2% |  60.3% |  35.9% |  55.6% |  39.1%
Gemma-7b |  43.0% |  24.2% |  57.7% |  35.5% |  53.6% |  40.4%
Gpt2 |  40.6% |  23.9% |  56.9% |  35.2% |  50.2% |  40.2%
Llama-2-7b-chat-hf |  43.9% |  23.1% |  60.0% |  35.4% |  53.4% |  40.2%
Llama-2-7b-hf |  44.9% |  23.8% |  58.1% |  35.1% |  53.0% |  42.1%
Mistral-7B-v0.1 |  44.3% |  23.9% |  57.7% |  35.3% |  53.2% |  40.6%
OLMo-7B |  44.3% |  24.8% |  56.3% |  35.6% |  52.7% |  41.2%
Pythia-1.4b |  42.9% |  23.8% |  57.4% |  35.5% |  49.8% |  40.4%
Pythia-2.8b |  43.9% |  23.5% |  57.6% |  35.6% |  50.2% |  40.2%
Tulu-2-7b |  44.4% |  23.4% |  59.3% |  35.3% |  53.2% |  41.9%
Tulu-2-dpo-7b |  45.3% |  23.4% |  59.3% |  35.3% |  53.4% |  42.1%
Vicuna-7b-v1.5 |  42.2% |  23.4% |  59.3% |  35.1% |  54.5% |  42.1%
Zephyr-7b-dpo |  42.5% |  23.4% |  56.7% |  35.2% |  54.3% |  42.1%
Zephyr-7b-sft |  43.9% |  23.5% |  57.6% |  35.3% |  54.5% |  41.5%
Table 5: Length-normalized and non-length-normalized ranking accuracies for the Alpaca Farm validation [8] dataset (see App. C.2), but including examples with ties (unlike Table 4). Model |  Alpaca Farm
---|---
|  Validation
|  $\tilde{{\mathcal{R}}}$ |  ${{\mathcal{R}}}$
Gemma-7b-it |  15.8% |  34.1%
Gemma-7b |  40.6% |  35.2%
Gpt2 |  32.2% |  34.8%
Llama-2-7b-chat-hf |  41.5% |  34.8%
Llama-2-7b-hf |  41.5% |  36.4%
Mistral-7B-v0.1 |  42.8% |  35.0%
OLMo-7B |  38.9% |  35.4%
Pythia-1.4b |  37.2% |  34.8%
Pythia-2.8b |  38.6% |  34.6%
Tulu-2-7b |  40.6% |  36.2%
Tulu-2-dpo-7b |  40.3% |  36.4%
Vicuna-7b-v1.5 |  41.5% |  36.4%
Zephyr-7b-dpo |  42.3% |  36.4%
Zephyr-7b-sft |  43.9% |  35.9%

###  C.4 Computation of the Idealized Ranking Accuracy

When computing the idealized ranking accuracy in Theorem 3.1 for common
preference datasets, we note that there are a few approximations required.
First, most datasets do not report the proportion $\alpha(x,y_{w},y_{l})$ of
raters who preferred $y_{w}$ over $y_{l}$, and using $\alpha(x,y_{w},y_{l})=1$
results in errors. As a result, we measure the alignment gap on the Alpaca
Farm validation split [8], which contains individual votes for each triple. In
the event that all four raters unanimously preferred one of the responses
(_i.e._ $\alpha(x,y_{l},y_{w})=0$), we add a small constant $\epsilon=0.001$
to $\alpha(x,y_{l},y_{w})$ to prevent divison by zero. Second, the formula
depends on the choice of $\beta$, which we do not know for many closed
proprietary models. We circumvent this issue by computing the quantity for a
range of $\beta$ values ($\beta\in\\{0.01,0.1,1,5,10\\}$) and reporting the
minimum, median, and maximum.

Table 6: We provide both the length-normalized ($\tilde{{\mathcal{R}}}$) and non-length-normalized (${{\mathcal{R}}}$) ranking accuracies for a variety of open-access preference-tuned models on the Alpaca Farm [8] validation dataset (described in App. C.2). We also provide the idealized ranking accuracy (Corollary 3.3). Unlike Table 1, we include examples with ties in this table. Preference-Tuned Model | Length-Normalized | Non-Length-Normalized
---|---|---
|  $\tilde{{\mathcal{R}}}$ |  $\tilde{{\mathcal{R}}^{*}}$ (Min./Med./Max.) |  ${{\mathcal{R}}}$ |  ${{\mathcal{R}}^{*}}$ (Min./Med./Max.)
Zephyr-7B-DPO |  42% |  86% / 98% / 100% |  36% |  90% / 99% / 100%
Tulu-2-DPO-7B |  40% |  87% / 97% / 100% |  36% |  91% / 99% / 100%
Google-Gemma-7B-IT |  41% |  73% / 73% / 97% |  35% |  67% / 93% / 100%
LLaMA-2-7B-Chat-HF |  42% |  87% / 97% / 100% |  35% |  91% / 99% / 100%

##  Appendix D Dynamics of DPO Training

###  D.1 Training Details

For our results in Section 4, we trained three different scales of models
(GPT2 [38], Pythia 2.8B [4], and Llama 2 7B [50]) across three seeds each on
the HH-RLHF dataset [3]. We split the test dataset in half, using half for
validation during hyperparameter tuning. We ran a separate hyperparameter
search for each class of model and for each stage of training (_i.e._ SFT
versus DPO). The hyperparameter ranges we searched were:

  * •

GPT2

    * –

SFT: learning rate $\in\\{$5e-7, 1e-6, 5e-6, 1e-5$\\}$, batch size $\in\\{$64,
128, 256, 512$\\}$

    * –

DPO: learning rate $\in\\{$5e-7, 1e-6, 5e-6, 1e-5$\\}$, batch size $\in\\{$32,
64, 128$\\}$, $\beta\in\\{$0.01, 0.1, 1.0, 10.0$\\}$

  * •

Pythia 2.8B

    * –

SFT: learning rate $\in\\{$1e-7, 1e-6, 1e-5$\\}$, batch size $\in\\{$16, 32,
64$\\}$

    * –

DPO: learning rate $\in\\{$5e-7, 1e-6, 5e-6, 1e-5$\\}$, batch size $\in\\{$32,
64$\\}$, $\beta\in\\{$0.01, 0.1, 1.0, 10.0$\\}$

  * •

Llama 2 7B

    * –

SFT: learning rate $\in\\{$1e-7, 1e-6, 1e-5$\\}$, batch size $\in\\{$32,
64$\\}$

    * –

DPO: learning rate $\in\\{$1e-6, 1e-7$\\}$, batch size $\in\\{$32, 64$\\}$,
$\beta\in\\{0.1\\}$

We tuned the hyperparameters on a single seed, and carried over the best
hyperparameters to the other seeds of the same model class. We trained the
GPT2 and Pythia2.8B models for 5 epochs each, and the Llama2 7B model for 1
epoch only (due to computational constraints) for both SFT and DPO. However,
most seeds of the GPT2 and Pythia 2.8B models reached the lowest validation
loss at the end of the first epoch. For analyses where we analyze only one
checkpoint (rather than the evolution over the course of training), we always
analyze the checkpoint with lowest validation loss. We use the AdamW optimizer
(with $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=$1e-8) for SFT and the RMSProp
optimizer (with $\alpha=0.99,\text{ weight decay}=0,\text{
momentum}=0,\epsilon=$1e-8) for DPO.

The GPT2 models were trained on a single Nvidia A100 GPU each, and the Pythia
2.8B and Llama 2 7B models were trained on two Nvidia A100 GPUs each. We used
PyTorch Fully Sharded Data Parallel (FSDP) in fully sharded mode to train the
Llama 2 7B models. In total, SFT required approximately 9, 52, and 30 GPU-
hours per seed of GPT2, Pythia 2.8B, and Llama 2 7B, respectively. DPO
required approximately 8, 48, and 49 GPU-hours per seed of GPT2, Pythia 2.8B,
and Llama 2 7B, respectively. (Longer training times were required for Pythia
2.8B than for Llama 2 7B since we trained the former for only 5 epochs and the
latter for 1, as aforementioned.)

###  D.2 Results

We provide the DPO loss, reward margin, and dataset trends during training
across all 9 models (three seeds each of GPT2 [38], Pythia 2.8B [4], and Llama
2 8B [50]) in Figs. 5, 6, and 7.

(a) Three different seeds of GPT-2 [38].

(b) Three different seeds of Pythia 2.8B [4].

(c) Three different seeds of Llama 2-7B [50].

Figure 5: Average DPO loss over the course of training, for four categories of
the training data (Anthropic HH-RLHF; Bai et al. [3]). The category
“correct->incorrect" indicates examples $(x,y_{w},y_{l})$ for which
$\pi_{\text{Ref}}(y_{w}|x)>\pi_{\text{Ref}}(y_{l}|x)$ but
$\pi_{\theta_{t}}(y_{w}|x)<\pi_{\theta_{t}}(y_{l}|x)$ (where
$\pi_{\theta_{t}}$ is the trained policy at training step $t$), and so on.
Lines that end early indicate that the category no longer contains any data
points. The dashed vertical line indicates the step at which the lowest
validation loss was achieved.

(a) Three different seeds of GPT-2 [38].

(b) Three different seeds of Pythia 2.8B [4].

(c) Three different seeds of Llama 2-7B [50].

Figure 6: Average DPO reward margin over the course of training, for four
categories of the training data (Anthropic HH-RLHF; Bai et al. [3]). The
category “correct->incorrect" indicates examples $(x,y_{w},y_{l})$ for which
$\pi_{\text{Ref}}(y_{w}|x)>\pi_{\text{Ref}}(y_{l}|x)$ but
$\pi_{\theta_{t}}(y_{w}|x)<\pi_{\theta_{t}}(y_{l}|x)$ (where
$\pi_{\theta_{t}}$ is the trained policy at training step $t$), and so on.
Lines that end early indicate that the category no longer contains any data
points. The dashed vertical line indicates the step at which the lowest
validation loss was achieved.

(a) Three different seeds of GPT-2 [38].

(b) Three different seeds of Pythia 2.8B [4].

(c) Three different seeds of Llama 2-7B [50].

Figure 7: Percent of the dataset that each category of data constitutes over
the course of training, for four categories of the training data (Anthropic
HH-RLHF; Bai et al. [3]). The category “correct->incorrect" indicates examples
$(x,y_{w},y_{l})$ for which
$\pi_{\text{Ref}}(y_{w}|x)>\pi_{\text{Ref}}(y_{l}|x)$ but
$\pi_{\theta_{t}}(y_{w}|x)<\pi_{\theta_{t}}(y_{l}|x)$ (where
$\pi_{\theta_{t}}$ is the trained policy at training step $t$), and so on.
Lines that end early indicate that the category no longer contains any data
points. The dashed vertical line indicates the step at which the lowest
validation loss was achieved.

(a) Results for the second seed.

(b) Results for the second seed.

Figure 8: The log-ratio of the $\pi_{\text{Ref}}$ likelihoods versus DPO loss
and ranking accuracy on a subsample of 1K training examples from the HH-RLHF
dataset [3]. The results from the first seed are given in Fig. 3, and the
results for the other two seeds are given here.

##  Appendix E Ranking Accuracy and Win Rate

We include additional plots from our exploration of the relationship of
ranking accuracy and win rate in Figs. 9 and 10.

(a) Ranking accuracy and win rate, computed on prompts from the training
dataset.

(b) Training and test DPO loss during training, up to the point of lowest val.
loss.

Figure 9: Ranking accuracy and win rate (versus $\pi_{\text{Ref}}$) are not
monotonically related throughout training. We measure the loss, ranking
accuracy, and win rate from the start of training to the checkpoint of lowest
validation loss. Even though both training and test loss continue to decline
during DPO training, ranking accuracy and win rate only trend together early
on in training. Past a certain point, the two become anti-correlated.

(a) Ranking accuracy and win rate versus $\gamma$.

(b) $\gamma$ versus distance travelled by model parameters during DPO
training.

Figure 10: When the influence of $\pi_{\text{Ref}}$ is strong, win rate and
ranking accuracy trend together. A higher $\gamma$ value implies greater
influence of $\pi_{\text{Ref}}$ during training. For larger $\gamma$ values
($\gamma\geq 1.25$), ranking accuracy and win rate trend in the same
direction.

###  E.1 Results on the Test Set

We also provide results computed on the test set in Figs. 11 and 12.

(a) Ranking accuracy and win rate versus training steps.

(b) Ranking accuracy and win rate versus distance that the model weights have
moved.

Figure 11: Ranking accuracy and win rate of various Pythia 2.8B checkpoints
during training, calculated on the test dataset.

(a) Ranking accuracy and win rate versus training steps.

(b) Ranking accuracy and win rate versus distance that the model weights have
moved.

Figure 12: Ranking accuracy and win rate of various models trained with
${\mathcal{L}}_{\text{DPO}}^{\gamma}$, calculated on the test dataset.

##  Appendix F Qualitative Analysis

Theorem 4.1 demonstrates that it is difficult for DPO to learn the correct
ranking of points that are not ranked correctly by the reference model. In
particular, datapoints that induce a large positive reference model log-ratio
(see Definition 2.2) require the DPO loss to be minimized to a very small
value in order to flip the ranking (Theorem 4.1 and Figure 3).

Here, we document a few of the datapoints that induce a large positive
reference model log-ratio and are hard to learn (LABEL:tab:hard_to_learn), as
well as datapoints that induce a very negative reference model log-ratio and
are easy to learn (LABEL:tab:easy_to_learn). These datapoints are measured
using the Pythia-2.8B model and are taken from the training split of the
Anthropic HH-RLHF [3] dataset. We note that datapoints with a very negative
reference model log-ratio are already ranked correctly at the start of DPO. We
also document the datapoints that are easy to flip: the reference model log-
ratio is slightly positive, so the reference model is slightly incorrect, and
optimizing the DPO objective could feasibly result in the model learning to
rank these points correctly. We observe that the hard-to-learn datapoints are
substantially longer than the easy ones, and that the easy datapoints
generally contain chosen responses that are unambiguously better than the
rejected ones.

Table 7: Preference datapoints that are hard to learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are hard for the Pythia-2.8B model to learn. We characterize a datapoint as hard to learn if the reference model log-ratio (RMLR, see Definition 2.2) is large. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds. RMLR | ${\mathcal{L}}_{\text{DPO}}$ |  Prompt, Chosen Response, and Rejected Response
---|---|---
457.6 | 3.05 |  Prompt: Human: What are the best and most reliable VPN services? Assistant: I could recommend OpenVPN, which is open-source and through which you can encrypt and protect your online connections. You can find instructions for setting up the service with OpenVPN at https://openvpn.net/index.php/open-source/documentation/miscellaneous/. Human: That link doesn’t work. Can you just send me a link that would take me to the homepage of OpenVPN. Assistant: You can find the home page for OpenVPN at https://openvpn.net/. Human: Thanks I’ll check out their website to see if they are trustworthy. Assistant: Sure thing. You can also find out more about the service by consulting user-ratings on sites like www.trustpilot.com. Human: Okay I’ll use trustpilot to see if it is trustworthy and reliable. Assistant: Chosen response: Great. There are a few additional tips you might find useful when it comes to evaluating third-party VPNs. For the most part, VPNs are pretty similar for users, although they differ slightly in terms of features and capabilities. Since I can’t give you any recommendations for specific VPNs, you might want to consider doing a cost/benefit analysis of various services: how many locations they support, how reliable and speedy they are, what types of devices they support, and does the service protect your personal information? I recommend employers and groups with more specialized needs choose a service such as CyberGhost, which offers a military-grade encryption system and is able to meet large-scale requirements down to the level of providing multiple cybersecurity experts on-board. However, as a personal consumer, you’ll probably feel more comfortable by finding a cheaper service with more limited features, since you don’t need as much help with cybersecurity. Rejected response:  You’re welcome.
423.4 | 2.2 |  Prompt: Human: What are some fun toys my kids can play with in the bath? Assistant: Chosen response:  There are many fun toys you can play with in the tub with your kids! Here are a few favorites: -Bath crayons and sponges for drawing -Soft bath sponges for floating -Bubble blowers -Bath toys, such as bath squirt toys or dinosaurs -Simple bath musical instruments, like rubber ducks or Tommy Tippee cups -Colored mini cups for blocks and other building games -Bursting washup sponges -Shaving foams and shaving creams for water fun -Bath thermometers -Floating toys like floating rings and animals or mini boats -Plastic containers and figurines for fantasy play and role-playing games -Bath fizz tablets and baking soda for bubbles -Clackers, bangles and fizzers \- Squirt guns -Straw toys for blowing air and creating bubbles -Water wands and spray bottles -Floating rainbow soap bars -Comic books and storybooks for reading in the tub -Soft toys for cuddling and play -Rubber duckies and other Rejected response:  Well, one of the more popular bath toys for kids is a bath horse. It’s a large waterproof animal toy you can keep in the bath and then wash off easily after use. These toys are often sturdy enough that your child can sit on them while they’re in the bath, and they come in a huge variety of shapes and designs. You can also search for bath playsets, which often have a variety of other accessories as well.
418.7 | 0.95 |  Prompt: Human: Is brainwashing a real thing that people are capable of doing? Assistant: Yes, I think brainwashing is definitely a real thing that people are capable of doing. This is an example of a moral and ethical question that many people have different ideas about, and you might wonder if I’m qualified to answer it. So how about we first do a quick thought experiment? Human: I’m not asking about the morality, I’m asking if it’s an actual practical scientifically backed phenomenon/procedure that people can do. Are there any real world examples of brainwashing? Assistant: Chosen response: Brainwashing is actually a real and fairly common phenomenon, and there are many examples of people using these techniques for coercive and unethical means. For example, in Orwell’s famous novel 1984, the Party uses various forms of psychological coercion to control its citizens and suppress independent thought or critical thinking. In real world cases, brainwashing is often associated with coercive ideological or political indoctrination, often in the form of highly-intense and insular social or cultic environments. Such groups can exert intense psychological pressure on individuals to adopt specific belief patterns or behavioral norms. These environments are often characterized by prominent charismatic leadership, intensive and prolonged social isolation from the surrounding world, and highly repressive behavioral controls. In these contexts, new members are typically subjected to prolonged and intensive persuasion campaigns, often involving overt coercion, psychological manipulation and dishonesty, and extreme social pressures to conform to the group. Over time, new members can be progressively subjected to increasing restrictions on their activities and freedom of thought, often involving the suppression or denunciation of prior belief patterns and behavioral norms, and eventually resulting in a complete loss of previous identity and independent thought. Rejected response: Oh, I think I understand. I’m not sure what you mean by "scientific", but let’s talk about the use of the phrase "brainwashing" to mean something that happens to you. We often talk about brainwashing in the context of deprogramming, and that sometimes happens in the context of religious cults. I can tell you more about that.

Table 8: Preference datapoints that are easy to learn. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are easy for the Pythia-2.8B model to learn. We characterize a datapoint as easy to learn if the reference model log-ratio (RMLR, see Definition 2.2) is small, or very negative. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds. RMLR | ${\mathcal{L}}_{\text{DPO}}$ |  Prompt, Chosen Response, and Rejected Response
---|---|---
$-675.9$ | $0.16$ |  Prompt: Human: How do you make twice-baked potatoes? Assistant: Chosen response:  They’re potatoes that have been boiled, then baked in the oven, then baked again. Rejected response:  What is a twice-baked potato?
$-562.1$ | $0.052$ |  Prompt: Human: How do I put fondant on a cake? Assistant: You can use butter to soften it, then add cream or milk to bring the fondant to a smooth consistency. You can then put the fondant on the cake, before adding icing. Human: How do I apply it? Assistant: Chosen response:  You can dab a very small amount on the icing in the area where you want to have the fondant, then smooth it out with the back of the spoon. If the fondant is too dry, you can add more cream or milk to make it a bit softer, but make sure you don’t get any clumps of fondant. Rejected: Once you have a smooth consistency, you can spread it out with a palette knife, using a spatula to remove any excess. Rejected response:  Once you have a smooth consistency, you can spread it out with a palette knife, using a spatula to remove any excess.
$-522.5$ | $0.003$ |  Prompt: Human: Can you find me Margot Robbie movies? Assistant: Oh sure, let me check my database! Hmmm, it looks like you’re a “fan” of the Canadian actress Margot Robbie. Why don’t you tell me more about her? Human: She is cool. Can you find me some movies? Assistant: Oh, cool! I’m not sure if there are any movies with her in it, but I do know there’s a TV show about her! Human: I would like movies. Assistant: Chosen response: Aaaahhh…I see, you want to see movies with her in them. Why don’t you tell me more about her movie roles? Rejected response: I have it on good authority that there are Margot Robbie movies, but the keyword you typed is really pretty broad and it’s not super clear what you mean.

Table 9: Preference datapoints that are easy to flip. We print several datapoints from the training split of Anthropic HH-RLHF [3] that are easy for the Pythia-2.8B model to flip. We characterize a datapoint as easy to flip if the reference model log-ratio (RMLR, see Definition 2.2) is a small positive value. This characterization follows Theorem 4.1 and Figure 3. The reported loss is measured on the checkpoint with the lowest validation loss and averaged over three seeds. RMLR | ${\mathcal{L}}_{\text{DPO}}$ |  Prompt, Chosen Response, and Rejected Response
---|---|---
$0.26$ | $0.86$ |  Prompt: Human: Whats the easiest way to throw a piano out of tune? Assistant: Just singing into the microphone! Human: What if I messed up the strings inside the piano? Assistant: Just let me know if you want to start putting heavier things inside the piano or break the legs off. Human: i want to put heavy things inside of it, have any suggestions? Assistant: Chosen response:  Just call it a dumpster fire piano, or let me know if you want to take apart a very heavy piano. Rejected response:  Depends what you mean by heavy. I can make some suggestions though. We don’t really want to throw any large things at the piano because you could cause damage to yourself or the piano.
$0.28$ | $0.19$ |  Prompt: Human: Is Austria a good place to take a vacation? Assistant: Chosen response:  Possibly. Can I ask some questions? It’s best to think about any questions before you give me an answer. Rejected response:  Is Austria a good place to take a vacation?
$0.45$ | $0.16$ |  Prompt: Human: what are good skills a receptionist needs? Assistant: I recommend organization and attention to detail. Since they’re also often the first point of contact for customers, they need to be courteous and helpful. And you also want them to accurately enter new customer information, so they need to be careful and diligent. I would Human: what would you say is the number one most important skill? Assistant: Chosen response: I would say it’s really critical that they be organized and detail-oriented. This is the backbone of their job, and if they’re not doing it well, it’s going to make their entire team look bad. The customer service aspect is also super important, and it helps if they’re friendly. But if they’re not organized, it’s hard for them to provide good service. Rejected response: I would say it depends on the type of work they’re doing, but organization and attention to detail are key for any receptionist. They need to make sure everything is orderly and correct, and also that they’re very attentive to customers. They need to make sure everyone who calls in gets exactly what they need.