# New Desiderata for Direct Preference Optimization

Xiangkun Hu Tong He David Wipf
Amazon Web Services
<EMAIL_ADDRESS>

###### Abstract

Large language models in the past have typically relied on some form of
reinforcement learning with human feedback (RLHF) to better align model
responses with human preferences. However, because of oft-observed
instabilities when implementing these RLHF pipelines, various
reparameterization techniques have recently been introduced to sidestep the
need for separately learning an RL reward model. Instead, directly fine-tuning
for human preferences is achieved via the minimization of a single closed-form
training objective, a process originally referred to as direct preference
optimization (DPO) and followed by several notable descendants. Although
effective in certain real-world settings, we introduce new evaluation criteria
that serve to highlight unresolved shortcomings in the ability of existing DPO
methods to interpolate between a pre-trained reference model and empirical
measures of human preferences, as well as unavoidable trade-offs in how low-
and high-quality responses are regularized and constraints are handled. Our
insights then motivate an alternative DPO-like loss that provably mitigates
these limitations. Empirical results serve to corroborate notable aspects of
our analyses111A preliminary version of this work appears in the ICML 2024
Workshop on Models of Human Feedback for AI Alignment:
https://openreview.net/pdf?id=Fgf0iAOb22.

##  1 Introduction

Although pre-trained large language models (LLMs) often display remarkable
capabilities [9, 11, 26, 44], it is well-established that they are prone to
responding in ways that may be at odds with human preferences for rationale
discourse [5, 17]. To this end, after an initial supervised fine-tuning phase
that produces a reference model or policy $\pi_{\tiny\mbox{ref}}(y|x)$, it is
now commonplace to apply reinforcement learning with human feedback (RLHF) to
further refine the LLM responses $y$ to input prompts $x$ [46, 38, 4, 27].
This multi-step process involves first learning a reward model that reflects
human inclinations culled from labeled preference data, and then subsequently
training a new policy that balances reward maximization with proximity to
$\pi_{\tiny\mbox{ref}}(y|x)$.

Because RLHF introduces additional complexity, computational overhead, and
entry points for instability, clever reparameterization techniques have
recently been proposed that sidestep the need for separately learning a reward
model altogether. Instead, increased alignment with human preferences is
achieved via the minimization of a single closed-form training objective, a
process originally referred to as direct preference optimization (DPO) [33]
followed by several notable descendants and generalizations [3, 40, 41, 45].
These alternatives dramatically economize model development; however, with
recency comes the potential that the consequences of less obvious properties
of DPO-based objectives may still be under-explored. It is along these lines
that our attention herein lies, with the end goal of quantifying and steering
model behavior in transparently beneficial directions.

After introducing basic concepts and the details of existing preference
optimization models in Section 2, the remainder of the paper devoted to our
technical contributions can be distilled as follows:

  * •

We introduce new evaluation desiderata that comport with intuition regarding
how a preference model ideally should behave, and yet (somewhat surprisingly)
are provably not satisfied by a broad class of existing DPO-based approaches.
In particular, we show that because of uniform regularization effects, the
minimizers of commonly-used preference optimization objectives like DPO are at
times unable to preserve performance in regions where the reference model is
strong while simultaneously improving upon the reference model elsewhere
(Section 3.1). Moreover, we also elucidate limitations in the ability to
interpolate between ideal endpoints as model trade-off parameters are varied
(Section 3.2).

  * •

We prove that once inevitable learning constraints are introduced (explicitly
or implicitly, e.g., early-stopping, weight decay, etc.), the core
reparameterizations that underpin certain DPO models no longer strictly hold
(Section 3.3). This motivates alternative justifications based solely on
properties of the final loss functions involved (Appendices C and D).

  * •

Based on the above, we introduce a new preference optimization loss called
$\ell_{\tiny\mbox{TYPO}}$ that, by design, satisfies our evaluation desiderata
while avoiding any dependency on constraint-dependent reparameterizations
(Section 4). Properties of this loss relative to its precursors are also
corroborated using Monte-Carlo simulations (Section 5 and Appendix A.2).

##  2 Background

We adopt $x\sim{\mathcal{D}}_{x}$ to denote an input prompt $x$ drawn from
some distribution ${\mathcal{D}}_{x}$. From here, conditioned on such prompts
we may then generate responses $y$ using a pre-trained reference language
model/policy $\pi_{\tiny\mbox{ref}}(y|x)$. Moreover, given a pair of such
responses $y_{1}\neq y_{2}$, we adopt the binary indicator variable
$z=\mathbb{I}[y_{1}\succ y_{2}|y_{1},y_{2},x]$ to convey that $y_{1}$ is
preferred over $y_{2}$ by a human evaluator when $z=1$, or else $z=0$ if
instead $y_{2}\succ y_{1}$. Given a population of such evaluators, we express
the ground-truth human preference distribution as
$p^{*}(z|y_{1},y_{2},x)=p^{*}(y_{1}\succ y_{2}|y_{1},y_{2},x)$. And finally,
we define a set of human labeled tuples drawn from a training distribution
${\mathcal{D}}_{tr}$ as

$\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}~{}~{}\equiv~{}~{}\\{z,y_{1},y_{2},x\\}\sim{\mathcal{D}}_{tr}~{}~{}\equiv~{}~{}z\sim
p^{*}(z|y_{1},y_{2},x),~{}\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x},$
(1)

where $y_{w}\succ y_{l}$ (subscripts here stand for ‘win’ and ‘lose’).222We
generally assume that $y_{1}\neq y_{2}$; however, the $y_{1}=y_{2}$ case can
nonetheless be handled by simply assigning $p^{*}(z|y,y,x)=1/2$, inclusion of
which does not effect the analysis that follows. In particular, such cases
merely introduce an irrelevant constant into the human preference loss
functions under consideration. In other words, each training tuple is
generated by drawing $x$ from ${\mathcal{D}}_{x}$, $y_{1}\neq y_{2}$ from the
reference policy $\pi_{\tiny\mbox{ref}}$, and finally $z$ is produced by human
labelers that operate according to $p^{*}$. Note that per convention in prior
work and ease of presentation, we will often abbreviate the preference
distribution notation as $p^{*}(y_{1}\succ y_{2}|y_{1},y_{2},x)\equiv
p^{*}(y_{1}\succ y_{2}|x)$ when the context is sufficiently clear.

###  2.1 Reinforcement Learning with Human Feedback (RLHF)

#### Reward Function Estimation:

Given two candidate responses $y_{1}\neq y_{2}$ sampled using prompt $x$, the
Bradley-Terry (BT) model [8] for human preferences stipulates that

$p^{*}(y_{1}\succ
y_{2}|x)=\frac{\exp[r^{*}(y_{1},x)]}{\exp[r^{*}(y_{1},x)]+\exp[r^{*}(y_{2},x)]}=\sigma\big{[}r^{*}(y_{1},x)-r^{*}(y_{2},x)\big{]},$
(2)

where $r^{*}(y,x)$ is a so-called latent reward model and $\sigma$ is the
logistic function. Because $r^{*}(y,x)$ is unobservable, it is not possible to
directly compute $p^{*}(y_{1}\succ y_{2}|x)$; however, we can train an
approximation $p_{\phi}(y_{1}\succ y_{2}|x)$ (equivalent to
$p_{\phi}(y_{1}\succ y_{2}|y_{1},y_{2},x)$ as before) defined by a
parameterized proxy reward $r_{\phi}(y,x)$. Specifically, we can minimize the
loss

$\ell_{\tiny\mbox{BT}}(r_{\phi}):=\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\Big{[}-\log
p_{\phi}(y_{w}\succ
y_{l}|x)\Big{]}=\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\Big{[}-\log\sigma\big{[}r_{\phi}(y_{w},x)-r_{\phi}(y_{l},x)\big{]}\Big{]}.$
(3)

The optimized reward
$\hat{r}_{\phi}(y,x):=\arg\min_{r_{\phi}}\ell_{\tiny\mbox{BT}}(r_{\phi})\approx
r^{*}(y,x)$ can then be applied to fine-tuning the pre-trained reference model
$\pi_{\tiny\mbox{ref}}(y|x)$ as described next.

#### RL Fine-Tuning with Estimated Reward Function:

The goal here is to improve upon a given $\pi_{\tiny\mbox{ref}}(y|x)$ using a
separate trainable model $\pi_{\theta}(y|x)$, the high-level desiderata being:
(i) Maximize the previously-estimated reward function $\hat{r}_{\phi}(y,x)$
when following $\pi_{\theta}(y|x)$, while (ii) Minimizing some measure of
distance between $\pi_{\theta}(y|x)$ and $\pi_{\tiny\mbox{ref}}(y|x)$ to avoid
overfitting merely to preference rewards. These objectives typically
materialize through the minimization of

$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},\hat{r}_{\phi},\lambda\right)~{}~{}:=~{}~{}\mathbb{E}_{y\sim\pi_{\theta}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}-\hat{r}_{\phi}(y,x)\Big{]}+\lambda~{}\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}\pi_{\theta}(y|x)||\pi_{\tiny\mbox{ref}}(y|x)\big{]}\Big{]},$
(4)

where $\lambda>0$ is a trade-off parameter. Although not differentiable,
starting from an initialization such as $\pi_{\theta}=\pi_{\tiny\mbox{ref}}$,
the loss
$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},\hat{r}_{\phi},\lambda\right)$
can be optimized over $\pi_{\theta}$ using various forms of RL [37, 34]

###  2.2 Direct Preference Optimization (DPO)

Consider now the reward-dependent RLHF loss $\ell_{\tiny\mbox{RLHF}}$ from (4)
defined w.r.t. and arbitrary reward function $r(y,x)$. DPO [33] is based on
the observation that, provided $\pi_{\theta}$ is sufficiently flexible such
that we may treat it as an arbitrary function for optimization
purposes,333This is a key assumption with non-trivial consequences; Section
3.3 will explore this issue in further detail. the minimum of
$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r,\lambda\right)$
w.r.t. $\pi_{\theta}$ can be directly computed as

$\pi_{r}(y|x)~{}:=~{}\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r,\lambda\right)=\frac{1}{Z(x)}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r(y,x)\right],$
(5)

where
$Z(x):=\sum_{y}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r(y,x)\right]$
is the partition function ensuring that $\pi_{r}(y|x)$ forms a proper
distribution [31, 32]. From here, assuming $\pi_{\tiny\mbox{ref}}(y|x)>0$, we
can rearrange (5) to equivalently establish that

$r(y,x)=\lambda\log\frac{\pi_{r}(y|x)}{\pi_{\tiny\mbox{ref}}(y|x)}+\lambda\log
Z(x).$ (6)

Because thus far $r$ has remained unspecified, it naturally follows that these
policy/reward relationships hold even for the ground-truth reward $r^{*}$ and
the associated optimal policy
$\pi^{**}(y|x):=\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)$.
Hence instead of approximating $r^{*}(y,x)$ with $r_{\phi}(y,x)$ as in (2), we
may equivalently approximate $\pi^{**}(y|x)$ with some $\pi_{\theta}(y|x)$
leading to the DPO loss
$\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)~{}:=$

$\ell_{\tiny\mbox{BT}}\left(\lambda\log\frac{\pi_{\theta}(y|x)}{\pi_{\tiny\mbox{ref}}(y|x)}\right)=\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[-\log\sigma\left(\lambda\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}-\lambda\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right)\right],$
(7)

noting that the partition function $Z(x)$ conveniently cancels out and can be
excluded from further consideration. It is now possible to directly optimize
(7) over $\pi_{\theta}$ using SGD without the need for any challenging RLHF
procedure. The basic intuition here is that the parameterized policy
$\pi_{\theta}$ induces an implicit reward
$\lambda\log\left[\pi_{\theta}(y|x)\pi_{\tiny\mbox{ref}}^{-1}(y|x)\right]$
that is being optimized via the original BT preference model. Moreover this
equivalence is exact assuming data distributed as in (1).

###  2.3 Identity Preference Optimization (IPO)

Similar to DPO, the identity preference optimization (IPO) formulation [3]
avoids both a 2-step learning process and cumbersome, potentially unstable RL
training. To accomplish this, IPO is predicated on minimizing the original
RLHF loss from (4) but with an alternative reward function. Specifically, the
motivating IPO objective is to minimize
$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r_{\tiny\mbox{IPO}},\lambda\right)$,
where

$r_{\tiny\mbox{IPO}}(y,x):=\mathbb{E}_{y^{\prime}\sim\pi_{\tiny\mbox{ref}}(y|x)}\big{[}p^{*}(y\succ
y^{\prime}|x,y,y^{\prime})\big{]},$ (8)

over $\pi_{\theta}$.444Note that in principle the distribution used to draw
samples $y^{\prime}$ in defining $r_{\tiny\mbox{IPO}}$ need not be set to
$\pi_{\tiny\mbox{ref}}$; however, in practice $\pi_{\tiny\mbox{ref}}$ is a
typical choice, which we adopt throughout for simplicity. Because of the
special structure of this particular reward function, it turns out that it is
possible to minimize
$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r_{\tiny\mbox{IPO}},\lambda\right)$
over $\pi_{\theta}$ without RL. In brief, this is accomplished by first noting
that for any pair of responses $y_{1}\neq y_{2}$ the existence of an optimal
IPO policy, denoted $\pi_{\tiny\mbox{IPO}}$, evaluated at these responses can
be computed as a function of the reward $r_{\tiny\mbox{IPO}}$ using (5).
Combining $y_{1}$ and $y_{2}$ dependent terms, after a few algebraic
manipulations this then leads to the equivalence relation

$\log\left[\frac{\pi_{\tiny\mbox{IPO}}(y_{1}|x)\pi_{\tiny\mbox{ref}}(y_{2}|x)}{\pi_{\tiny\mbox{IPO}}(y_{2}|x)\pi_{\tiny\mbox{ref}}(y_{1}|x)}\right]=\frac{1}{\lambda}\big{[}r_{\tiny\mbox{IPO}}(y_{1},x)-r_{\tiny\mbox{IPO}}(y_{2},x)\big{]}.$
(9)

However, unlike DPO where an analogous expression is inverted to create an
implicit reward for integration within the BT model, IPO instead attempts to
approximate this equivalence relation by replacing the unknown
$\pi_{\tiny\mbox{IPO}}(y|x)$ with some $\pi_{\theta}(y|x)$. Although
technically $r_{\tiny\mbox{IPO}}$ is also unknown, given samples
$\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}$, it is nicely shown in
[3] that
$\ell_{\tiny\mbox{IPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)~{}~{}:=$

$\displaystyle\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}}\left[\left(\log\left[\frac{\pi_{\theta}(y_{1}|x)\pi_{\tiny\mbox{ref}}(y_{2}|x)}{\pi_{\theta}(y_{2}|x)\pi_{\tiny\mbox{ref}}(y_{1}|x)}\right]-\frac{1}{\lambda}\big{[}r_{\tiny\mbox{IPO}}(y_{1},x)-r_{\tiny\mbox{IPO}}(y_{2},x)\big{]}\right)^{2}\right]$
$\displaystyle=~{}~{}~{}\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[\left(\log\left[\frac{\pi_{\theta}(y_{w}|x)\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\theta}(y_{l}|x)\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]-\frac{1}{2\lambda}\right)^{2}\right]$
(10)

provided ${\mathcal{D}}_{tr}$ follows from (1). Note that this closed-form
consistency is a direct consequence of how $r_{\tiny\mbox{IPO}}$ is defined in
(8) and will not generally hold for other choices of the reward function.
Regardless, it is straightforward to minimize
$\ell_{\tiny\mbox{IPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ in its
present form via SGD as with DPO.

###  2.4 Flexible Quasi-Convex Generalizations

From the expressions above, it is clear that both DPO and IPO reduce to
functions of
$\log\left[\frac{\pi_{\theta}(y_{w}|x)\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\theta}(y_{l}|x)\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]$
and a tunable hyperparameter $\lambda$. As such, it is natural to consider
extensions to broader choices in the form

$\ell_{\tiny\mbox{QPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\psi,\mu,\lambda):=\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}~{}\psi\left(\mu\left[\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]-\mu\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right],\lambda\right),$
(11)

where $\mu:\mathbb{R}^{+}\rightarrow\mathbb{R}$ is a monotonically increasing
function (which generalizes the logarithm), and the function
$\psi:\mathbb{R}\times\mathbb{R}^{+}\rightarrow\mathbb{R}$ influences the
overall loss shape. We stipulate that $\psi$ is a differentiable quasi-convex
function [20]; hence the chosen loss notation $\ell_{\tiny\mbox{QPO}}$ for
quasi-convex preference optimization. By definition of quasi-convexity, $\psi$
monotonically increases to the right or left away from the minimum.

These specifications cover DPO and IPO as representative special cases, and
include essentially all reasonable choices for a loss within this family,
e.g., it is nonsensical to include multi-modal losses. The generalized
preference optimization (GPO) [40] and f-DPO [41] frameworks are also special
cases of QPO as defined herein. With GPO, $\mu$ is a logarithm and $\psi$ is
chosen as an arbitrary convex function (such as used by SLiC [45]). Meanwhile
f-DPO involves $\psi(\cdot,\lambda)=-\log\sigma[\lambda(\cdot)]$ analogous to
DPO but with $\mu=f^{\prime}$, where $f^{\prime}$ denotes the derivative of an
$f$-divergence [36]; given that $f$ must be convex, its derivative will
necessarily be monotonically increasing. In this way, the RLHF objective from
(4) is still optimized via $f$-DPO, but with an $f$-divergence replacing the
KL term.

While overall quite general, we will nonetheless later demonstrate that any
loss in the form of (11) will unavoidably be saddled with certain limitations.
See also Appendix B for additional context w.r.t. very recent and/or
concurrent DPO enhancements that lie outside the scope of our present work.

##  3 Comparative Analysis of Existing Approaches

We now turn to comparative analysis of existing approaches, which all have
ties relating back to the BT preference model. Throughout this section we say
that a policy $\pi^{*}$ is BT-optimal at prompt $x$ if $p^{*}(y_{1}\succ
y_{2}|x)$ implies that $\pi^{*}(y_{1}|x)>\pi^{*}(y_{2}|x)$ for all response
pairs $\\{y_{1},y_{2}\\}$ with nonzero probability (as determined by the
reference policy generating the preference data). Appendix F.1 introduces how
$\pi^{*}$ can be formed.

###  3.1 Selective Preservation of Optimal Policies

Consider the following plausible scenario, variations of which are likely to
occur (at least in varying degrees) with real-world data. Suppose the support
of prompts generated by ${\mathcal{D}}_{x}$ partitions as $d_{x}^{good}\cup
d_{x}^{bad}$, with $d_{x}^{good}\cap d_{x}^{bad}=\emptyset$. Furthermore,
assume we have access to a reference policy $\pi_{\tiny\mbox{ref}}$ such that
$\pi_{\tiny\mbox{ref}}=\pi^{*}$ for $x\in d_{x}^{good}$ and
$\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]\gg 0$ for $x\in d_{x}^{bad}$,
where $\mbox{dist}[\cdot,\cdot]$ is an arbitrary distance measure. In other
words, when evaluated w.r.t. a policy $\pi^{*}$ that proportionally reflects
human preferences, $\pi_{\tiny\mbox{ref}}$ performs ideally on a subset of
prompts but not on others.

This dichotomy provides a useful lens for examining certain loss function
properties. In particular, we would like any policy that minimizes a candidate
loss to preserve $\pi_{\tiny\mbox{ref}}$ for prompts $x\in d_{x}^{good}$,
while pushing away from $\pi_{\tiny\mbox{ref}}$ towards $\pi^{*}$ for prompts
$x\in d_{x}^{bad}$. However, because of uniform regularization effects
intrinsic to the QPO loss, it is not actually possible to achieve even this
modest objective.

###### Theorem 1

(Informal version) Given the prompt partitioning, reference policy, and
optimal policy described above, define
$\hat{\pi}_{\theta}^{\tiny\mbox{QPO}}:=\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{QPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\psi,\lambda)$
for any fixed selection of $(\psi,\lambda)$. Then under relatively mild
assumptions on the labeled responses in ${\mathcal{D}}_{tr}$, if
$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{QPO}},~{}\pi^{*}]<\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]$
for $x\in d_{x}^{bad}$, then
$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{QPO}},~{}\pi^{*}]>0$ for $x\in
d_{x}^{good}$.

The proof and formal version are provided in Appendix E.1, while Figure
1(left) below provides an illustration. The somewhat unexpected implication
here is that if we minimize any possible QPO loss in the form of (11) and
improve the policy quality in areas where $\pi_{\tiny\mbox{ref}}$ performs
poorly w.r.t. $\pi^{*}$, then it must also be the case that performance
becomes worse in areas where $\pi_{\tiny\mbox{ref}}$ was originally optimal.
This phenomena represents an unavoidable trade-off when we restrict ourselves
to using a QPO loss, of which DPO and IPO (as well as GPO and $f$-DPO) are
special cases inheriting the same limitation. The core issue here is that QPO
losses unselectively apply the same regularization, starting from the same
initialization point, to both good and bad cases relative to $\pi^{*}$.

###  3.2 Interpolation Capabilities

As the underlying goal shared by all approaches is to balance proximity to a
reference policy $\pi_{\tiny\mbox{ref}}$ with respect for the human preference
model $p^{*}$, a non-negative trade-off parameter $\lambda\in[a,b]$ that
allows for interpolating between these competing objectives is inevitable,
where $a\in\mathbb{R}$ and $b\in\mathbb{R}$ are lower and upper bounds
respectively.555Depending on the method, if $a=0$ or $b=\infty$ we may replace
the $\lambda$ range with an open set. In this section we examine more closely
the nature of loss function minimizers as $\lambda$ is varied, zooming in on
their behavior in the limit as $\lambda\rightarrow a$ and $\lambda\rightarrow
b$. To this end, we first introduce the following definitions :

###### Definition 1

We say that an arbitrary preference optimization loss
$\ell(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ satisfies the strong
interpolation criteria (SIC) if the following conditions hold:

  1. 1.

$\lim_{\lambda\rightarrow
a}\arg\min_{\pi_{\theta}}\ell(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)=\pi^{*}$;

  2. 2.

$\lim_{\lambda\rightarrow
b}\arg\min_{\pi_{\theta}}\ell(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)=\pi_{\tiny\mbox{ref}}$;

  3. 3.

For all other $\lambda\in(a,b)$, the optimal policy interpolates between the
above two extremes.

###### Definition 2

For any prompt $x$ and response $y$ define666See Appendix F.1 for the
derivation of the right-hand equality in (12).

$\pi^{\delta}(y|x)~{}~{}:=~{}~{}\arg\max_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\big{[}r^{*}(y,x)\big{]}~{}~{}=~{}~{}\left\\{\begin{array}[]{cc}1&\mbox{if}~{}~{}y=\arg\max_{y^{\prime}}\pi^{*}(y|x)\\\
0&\mbox{otherwise.}\end{array}\right.$ (12)

In this way, $\pi^{\delta}(y|x)$ assigns probability one to the mode of
$\pi^{*}(y|x)$, i.e., akin to a delta function with no generation diversity.
We then say that a loss $\ell(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$
satisfies the weak interpolation criteria (WIC) analogously to the SIC, only
for the lower bound we instead require $\lim_{\lambda\rightarrow
a}\arg\min_{\pi_{\theta}}\ell(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)=\pi^{\delta}$.

In summary, the only difference between these interpolation criteria is their
limiting behavior w.r.t. the lower bounding $\lambda$; for the SIC we approach
the BT-optimal policy, while for the WIC we approach a degenerate policy with
all probability mass restricted to the mode of the BT-optimal policy. We
remark that both the SIC and WIC cannot be simultaneously satisfied unless
$\pi^{*}$ itself is a degenerate delta function. We now explore how these
distinctions are reflected in the behavior of DPO and IPO loss minimizers,
with Figure 1(middle) illustrating the basic concepts.

###### Proposition 1

Assume preference data distributed according to ${\mathcal{D}}_{tr}$ from (1),
and that $p^{*}(y_{1}\succ y_{2}|x)\in(0,1)$ for all responses with
$\pi_{\tiny\mbox{ref}}(y|x)>0$. Then the DPO loss from (7) satisfies the WIC
(but not the SIC).

In terms of practical applicability of this result, there exists one important
caveat: the empirical distribution of a finite set of labeled preference data
need not actually satisfy the conditions of Proposition 1. For example,
suppose for each prompt $x\in{\mathcal{D}}_{x}$ we collect only two responses
$\\{y_{1},y_{2}\\}$ along with a single preference label $z$, which together
produce the tuple $\\{y_{w},y_{l},x\\}$. In this scenario, which reflects
certain publicly-available human preference datasets [4, 18], the empirical
distribution of preferences will be $p^{*}(y_{w}\succ y_{l}|x)=1\notin(0,1)$
for all $x\in{\mathcal{D}}_{x}$. Notably, Proposition 1 will not hold, and in
particular, it can be easily shown that minimizers of any valid $f$-DPO loss
will be completely independent of $\pi_{\tiny\mbox{ref}}$ for all
$\lambda\in(0,\infty)$; in other words, no interpolation occurs at all; see
Appendix F.2 for the derivation. A similar observation specific to DPO (but
not $f$-DPO) can be found in [1]. The fact that DPO-based solutions may still
reflect $\pi_{\tiny\mbox{ref}}$ in practice, and more-so as $\lambda$
increases, relates to implicit constraints and subtle regularization effects
as discussed further in Section 3.3 and Appendix C.

###### Proposition 2

Assume preference data distributed according to ${\mathcal{D}}_{tr}$ from (1).
Then the IPO loss from (2.3) satisfies the WIC (but not the SIC).

Comparing Proposition 2 with Proposition 1, we observe that IPO maintains its
ability to interpolate under broader conditions than DPO, particularly in the
empirical sampling regime involving binary probability values. That being
said, neither DPO nor IPO satisfy the SIC, which motivates consideration of
alternative losses that do, at least if our priority is to actually achieve
the SIC (which of course may depend on the application scenario). For this
purpose, it turns out that selections beyond the family of QPO objectives
(which includes DPO, $f$-DPO, and IPO) are necessary per the following:

###### Theorem 2

Assume preference data distributed according to ${\mathcal{D}}_{tr}$ from (1).
Then no possible QPO loss from (11) will satisfy the SIC.

Section 4 will consider objectives outside of the QPO family which circumvent
this limitation.

###  3.3 Impact of Optimization Constraints

Originally in [33], and later supported by follow-up analysis [3], it has been
shown that minimizing the DPO loss
$\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ is
effectively the same as minimizing the RLHF loss
$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)$
with optimal reward model $r^{*}$. But there is a pivotal assumption
underlying this association which previous analysis has not rigorously
accounted for. Specifically, the key equalities that facilitate the DPO and
IPO reparameterizations, namely (6) and (9) (and the analogous for $f$-DPO),
are all predicated on the solution of an uncononstrained optimization problem
over an arbitrary policy $\pi_{\theta}$.

However, when actually training models in real-world settings, constraints
will always exist, whether implicitly or explicitly. Such constraints stem
from any number of factors including the model architecture/capacity
limitations, early stopping, weight decay, drop-out regularization, machine
precision, and so on. Hence in reality we are never exactly minimizing some
preference loss $\ell\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda\right)$
over any possible $\pi_{\theta}$ (as assumed by DPO, IPO, and $f$-DPO
derivations). Instead, we must consider properties of the constrained problem
$\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi}}\ell\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda\right)$,
where ${\mathcal{S}}_{\pi}$ is a constraint set. For example, if we restrict
training to a single epoch with a fixed learning rate, then
${\mathcal{S}}_{\pi}$ can be viewed as the set of all points reachable within
a limited number of SGD updates.

###### Theorem 3

Let ${\mathcal{S}}_{\pi}$ denote a constraint set on the learnable policy
$\pi_{\theta}$. Then we can have that

$\arg\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)~{}\neq~{}\arg\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi}}\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda).$
(13)

As can be observed by the proof in Appendix E.5, the difference between the
two is akin to the difference between applying a constraint to a trainable
policy with respect to either the forward or backward KL divergence, which are
generally quite distinct [7]; see also Figure 1(right). There are several
important consequences of this result worth considering:

  * •

As discussed in Section 3.2, the DPO-based losses can have degenerate
unconstrained minimizers that completely ignore $\pi_{\tiny\mbox{ref}}$ on
certain real-world datasets; therefore counter-measures like early stopping
are imposed that effectively introduce a ${\mathcal{S}}_{\pi}$ that
dramatically alters the estimated policy. But in doing so, the inequality from
(13) is introduced and so we can no longer say that DPO provides an optimal
implicit reward for the original RLHF problem, i.e., the original connection
is now ambiguous.

  * •

As such, the value of DPO in practice (and indeed it often does work well)
cannot be unreservedly attributed to its original affiliation with an optimal
RLHF solution, and instead, should be evaluated based on properties of
$\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi}}\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$.
See Appendix C for one step in this direction.

  * •

To further illustrate the above points, in Appendix D we rederive the DPO loss
from scratch based solely on a Gaussian estimation perspective that is
completely unrelated to RLHF. But of course we do not actually believe that
binary human preference data are really Gaussian. Instead, this exercise
serves to highlight that what matters are properties of the underlying loss
when deployed in practice, not necessarily the assumptions made in deriving
the loss in the first place.

  * •

Other losses based on unconstrained RLHF-based reparameterizations in the
$f$-DPO and IPO families may be similarly influenced by the inevitable
introduction of policy constraints.

Figure 1: Desiderata visualizations, including added context w.r.t. our
proposed TYPO approach.

##  4 New Objectives for Human Preference Optimization

Motivated by the analysis in Section 3 and illustrated in Figure 1, we next
examine alternative objective functions adhering to the following desiderata:

  1. 1.

Perservation: Capable of selectively preserving an optimal policy in ideal
regimes, while simultaneously improving the policy in regions of poor
performance (from Section 3.1);

  2. 2.

Interpolation: Smoothly interpolates between the BT-optimal policy and the
reference policy, i.e., it achieves the SIC (from Section 3.2);

  3. 3.

Constraints: Independent of any derivation or required
equivalence/reparameterization that no longer holds upon the introduction of
constraints (from Section 3.3).

We label the our new objective $\ell_{\tiny\mbox{TYPO}}$ to highlight the
potential ability to “tame your preference optimization” (and “lower typos”)
by explicitly targeting these desiderata.

###  4.1 TYPO Objective Function

Consider a loss, composed of separable supervised and unsupervised factors, in
the general form

$\displaystyle\ell_{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)~{}~{}:=~{}~{}\ell_{\tiny\mbox{sup}}(\pi_{\theta})~{}+~{}\lambda\ell_{\tiny\mbox{unsup}}(\pi_{\theta},\pi_{\tiny\mbox{ref}})~{}~{}=$
(14)
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\Big{[}d_{\tiny\mbox{sup}}\big{[}\pi_{\theta}(y_{w}|x),\pi_{\theta}(y_{l}|x)\big{]}\Big{]}~{}+~{}\lambda\mathbb{E}_{y\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}d_{\tiny\mbox{unsup}}\big{[}\pi_{\theta}(y|x),\pi_{\tiny\mbox{ref}}(y|x),\big{]}\Big{]},$

where $d_{\tiny\mbox{sup}}$ serves as a supervised penalty over labeled
training tuples $(x,y_{w},y_{l})$ while $d_{\tiny\mbox{unsup}}$ represents an
additional regularization term independent of labeled preferences. We remark
that objectives in the form of (14) are natural candidates for SGD given that
all sampling is independent of $\theta$, unlike the typical regularized loss
adopted by RLHF, which requires samples from $\pi_{\theta}(y|x)$.

#### Supervised Term:

After first defining

$p_{\theta}(z|y_{1},y_{2},x):=\left\\{\begin{array}[]{cc}\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\theta}(y_{1}|x)+\pi_{\theta}(y_{2}|x)}&\mbox{if
}z=1\\\
\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\theta}(y_{1}|x)+\pi_{\theta}(y_{2}|x)}&\mbox{if
}z=0\end{array}\right.$ (15)

we then consider the supervised term

$\displaystyle\ell_{\tiny\mbox{sup}}(\pi_{\theta})$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}p^{*}(z|y_{1},y_{2},x)||p_{\theta}(z|y_{1},y_{2},x)\big{]}\Big{]}$
(16) $\displaystyle\equiv$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\left[\log\left(1+\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}\right)\right].$

Please see Appendix F.3 for the derivation of this equivalence. Importantly
here, because the KL-divergence is minimized iff
$p^{*}(z|y_{1},y_{2},x)=p_{\theta}(z|y_{1},y_{2},x)$, unlike an arbitrary
reward, the optimal solution to $\ell_{\tiny\mbox{sup}}(\pi_{\theta})$ will
necessarily recover the BT-optimal distribution as will be analyzed below.

#### Unsupervised Term:

For the unsupervised term in (14) we simply adopt

$\ell_{\tiny\mbox{unsup}}(\pi_{\theta},\pi_{\tiny\mbox{ref}})=\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}\pi_{\tiny\mbox{ref}}(y|x)||\pi_{\theta}(y|x)\big{]}\Big{]}\equiv-\mathbb{E}_{y\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}\log\pi_{\theta}(y|x)\Big{]},$
(17)

ignoring terms independent of $\pi_{\theta}$. Like (16), this expression also
does not require sampling from $\pi_{\theta}$. That being said, (17) can
exploit out-of-preference data (meaning unlabeled responses), and prior work
[25] has argued for the merits of using such data in broader RLHF contexts.
(It may also be reasonable to consider switching
$\ell_{\tiny\mbox{unsup}}(\pi_{\theta},\pi_{\tiny\mbox{ref}})$ to a reverse-KL
term and optimize with REINFORCE per general observations from [1]; however,
we do not pursue this direction further here.)

###  4.2 $\ell_{\tiny\mbox{TYPO}}$ Properties

Notable attributes of
$\ell{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ w.r.t.
the three desiderata from above are as follows:

###### Proposition 3

Under the same setup as Theorem 1, let
$\hat{\pi}_{\theta}^{\tiny\mbox{TYPO}}:=\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$,
instantiated using (16) and (17). Then
$\hat{\pi}_{\theta}^{\tiny\mbox{TYPO}}=\pi^{*}$ for all $x\in d_{x}^{good}$
including in cases where
$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{TYPO}},~{}\pi^{*}]<\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]$
for $x\in d_{x}^{bad}$.

Per this result, minimizers of
$\ell_{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ are
capable of preserving $\pi_{\tiny\mbox{ref}}$ in regions $d_{x}^{good}$ where
performance is strong relative to $\pi^{*}$, while concurrently improving
performance in other areas where it is not. Figure 1(left) visualizes this
unique TYPO capability.

###### Proposition 4

The loss $\ell{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$,
when instantiated using (16) and (17), satisfies the SIC.

Figure 1(middle) contrasts this property with the WIC achieved by prior
methods. We also remark that none of the derivations used to motivate
$\ell{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$ rely on
unconstrained optimization to form a reparameterized objective function as
with DPO, $f$-DPO, and IPO. As such, the inevitable introduction of such
constraints in practice does not compromise the TYPO origin story. In other
words, since TYPO is not based on any implicit association with RLHF in the
first place, adding constraints that might otherwise compromise such an
association pose no issue.

##  5 Empirical Validation

Although more of an analysis-driven contribution, our core insights from
Sections 3 and 4 can nonetheless benefit from empirical corroboration. To this
end, we first present a series of experiments adapted from [3] to highlight
aspects of TYPO behavior vis-à-vis our proposed desiderata. As the most
relevant published points of reference, we contrast with DPO, IPO, and
$f$-DPO; for the latter we choose the Jensen–Shannon divergence, which next to
the reverse-KL implicitly assumed by DPO, performed well in prior experiments
[41]. Later we test using the Anthropic Helpfulness and Harmlessness (HH)
real-world preference dataset [4, 18]. For space considerations, some
experiment details, including hyperparameters and training setups, are
deferred to Appendix A.

#### Interpolation Tests:

As in [3] we consider the bandit setting with a discrete space of three
responses/actions ${\mathcal{Y}}=\\{y_{a},y_{b},y_{c}\\}$ and create a dataset
of labeled pairs as
$\big{\\{}\\{y_{a},y_{b}\\},\\{y_{b},y_{c}\\},\\{y_{a},y_{c}\\}\big{\\}}$,
i.e., a total ordering consistent with the BT model. Preferences are assigned
via $p(y_{1}\succ y_{2})$ computed using (55) with $\pi^{*}(y_{a})=0.6$,
$\pi^{*}(y_{b})=0.3$, and $\pi^{*}(y_{c})=0.1$. Furthermore, again following
[3] we form our trainable policy as
$\pi_{\theta}(y_{i})=\mbox{softmax}[\theta_{i}]$ with
$\theta\in\mathbb{R}^{3}$ optimized using Adam over each different preference
loss. Results using a small $\lambda=10^{-5}$ are shown in Figure 2, where we
observe that TYPO closely converges to the BT-optimal solution, while DPO and
IPO converge to $\pi^{\delta}$ (the mode of $\pi^{*}$) consistent with
Propositions 1 (DPO), 2 (IPO), and 4 (TYPO), as well as Theorem 2 which
applies to $f$-DPO. Additional interpolation results traversing different
$\lambda$ towards the upper limit are presented in Appendix A.2.

Figure 2: Support for Sections 3.2 and 4.2 interpolation analysis. Dashed
lines represent BT-optimal preference probabilities $\pi^{*}$, while solid
lines are model learning curves for $\lambda=10^{-5}$ (small). Only TYPO
converges to $\pi^{*}$, others converge to $\pi^{\delta}$.

#### Preservation Tests:

We next modify the setting from above to include two input prompts
$\\{x_{g},x_{b}\\}$ chosen such that $x_{g}\in d_{x}^{good}$ and $x_{b}\in
d_{x}^{bad}$ sampled with equal probability. We then specify the corresponding
response space
${\mathcal{Y}}(x_{g})=\\{y_{ga},y_{gb},y_{gc}\\};~{}~{}~{}{\mathcal{Y}}(x_{b})=\\{y_{ba},y_{bb},y_{bc}\\}$
and prompt-dependent probabilities (see Appendix A.1). For the reference
policy we set $\pi_{\tiny\mbox{ref}}(y|x_{g})=\pi^{*}(y|x_{g})$ and
$\pi_{\tiny\mbox{ref}}(y|x_{b})\neq\pi^{*}(y|x_{b})$. We generate pair-wise
preference data as before, only now with prompt-dependent responses. Results
shown in Figure 3(left & middle) are in direct accordance with Theorem 1 and
Proposition 3, whereby TYPO is the only approach that preserves a strong
policy with prompt $x_{g}\in d_{x}^{good}$ while at the same time improving
performance relative to $\pi_{\tiny\mbox{ref}}$ for $x_{b}\in d_{x}^{bad}$
over all $\lambda$.

Figure 3: Preservation tests varying $\lambda$ (left and middle plots); unlike
TYPO, existing approaches are unable to both retain negligible error on the
good cases while improving performance (over the dashed line representing the
reference model) on the bad cases. Constraint test varying $\alpha$ and
plotting
$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{DPO}},~{}\hat{\pi}_{\theta}^{\tiny\mbox{RLHF}}]$
(right plot); DPO is no longer equivalent to RLHF with an optimal reward once
an additional constraint/regularization factor is introduced.

#### Constraint Tests:

We probe the extent to which learning constraints can interfere with the
equivalence between DPO and RLHF implemented with an optimal reward function.
To this end, we adopted the same data generation setup as in the interpolation
experiments from above. We then train policies to separately minimize the
right- and left-hand sides of (13), but with one key modification: we added an
identical penalty function $\alpha\|\pi_{\theta}\|_{2}^{2}$ to both models to
instantiate weight decay (a typical form of constraint used in practice),
where $\alpha\geq 0$ is a tunable hyperparameter. Figure 3(right) plots the
distance ($y$-axis) between learned policies from RLHF and DPO as $\alpha$ is
varied. Consistent with the original DPO derivations and analysis from [33],
we observe negligible error when $\alpha=0$ given that unconstrained DPO is
explicitly designed to mimic RLHF with an optimal reward $r^{*}$. However, in
accordance with our Theorem 3, as $\alpha>0$ increases, the distance between
RLHF and DPO grows considerably, and their relationship is no longer clear-
cut.

Figure 4: Real-world example.

#### Testing on Anthropic HH Preference Data:

Finally, to explore TYPO capabilities in a real-world scenario, we train a
Pythia 2.8B model [6] on the Anthropic Helpfulness and Harmlessness (HH)
preference dataset [4, 18] as previously used in [33]. Following their
settings, we first execute supervised fine-tuning (SFT) on the Pythia model
using $y_{w}$ values as the target response. We then use this SFT model as
$\pi_{\tiny\mbox{ref}}$ for training DPO, IPO and TYPO. Given that alignment
results (our focus) from [41] already show that reverse KL (i.e., DPO) works
best among $f$-divergences, we do not compare with other $f$-DPO selections
here. We use GPT-4 to evaluate the win rate of the generated responses from
each model against the chosen $y_{w}$ on the test set for single turn
dialogues. We emphasize that our comparisons cover both helpfulness and
harmlessness (see Appendix A.3), whereas the original DPO paper [33] only
tests the former.

##  6 Conclusions

In this work we have proposed multiple desiderata that existing methodology
for human preference optimization does not satisfy and yet our proposed TYPO
approach does.

## References

  * [1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker.  Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.  arXiv preprint arXiv:2402.14740, 2024.
  * [2] Afra Amini, Tim Vieira, and Ryan Cotterell.  Direct preference optimization with an offset.  arXiv preprint arXiv:2402.10571, 2024.
  * [3] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.  A general theoretical paradigm to understand learning from human preferences.  In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024.
  * [4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862, 2022.
  * [5] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  arXiv preprint arXiv:2212.08073, 2022.
  * [6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.
  * [7] C.M. Bishop.  Pattern recognition and machine learning.  Springer, New York, 2006.
  * [8] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  Biometrika, 39(3/4):324–345, 1952.
  * [9] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.  Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
  * [10] Emmanuel J Candes, Michael B Wakin, and Stephen P Boyd.  Enhancing sparsity by reweighted l1 minimization.  Journal of Fourier analysis and applications, 14:877–905, 2008\.
  * [11] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al.  A survey on evaluation of large language models.  ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024.
  * [12] Rick Chartrand and Wotao Yin.  Iteratively reweighted algorithms for compressive sensing.  International Conference on Accoustics, Speech, and Signal Processing, 2008.
  * [13] Yichen Chen, Dongdong Ge, Mengdi Wang, Zizhuo Wang, Yinyu Ye, and Hao Yin.  Strong NP-hardness for sparse optimization with concave penalty functions.  In International Confernece on Machine Learning, 2017.
  * [14] Bin Dai, Chen Zhu, Baining Guo, and David Wipf.  Compressing neural networks using the variational information bottleneck.  In International Conference on Machine Learning, pages 1135–1144. PMLR, 2018.
  * [15] Jianqing Fan and Runze Li.  Variable selection via nonconcave penalized likelihood and its oracle properties.  JASTA, 96(456):1348–1360, 2001.
  * [16] Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei.  Towards analyzing and understanding the limitations of dpo: A theoretical perspective.  arXiv preprint arXiv:2404.04626, 2024.
  * [17] Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed.  Bias and fairness in large language models: A survey.  arXiv preprint arXiv:2309.00770, 2023.
  * [18] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  arXiv preprint arXiv:2209.07858, 2022.
  * [19] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov.  Learn your reference model for real good alignment.  arXiv preprint arXiv:2404.09656, 2024.
  * [20] Harvey Greenberg and William Pierskalla.  A review of quasi-convex functions.  Operations research, 19(7):1553–1570, 1971.
  * [21] Jiwoo Hong, Noah Lee, and James Thorne.  Orpo: Monolithic preference optimization without reference model.  arXiv preprint arXiv:2403.07691, 2024.
  * [22] Shawn Im and Yixuan Li.  Understanding the learning dynamics of alignment with human feedback.  arXiv preprint arXiv:2403.18742, 2024.
  * [23] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [24] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.  Efficient memory management for large language model serving with pagedattention.  In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
  * [25] Ziniu Li, Tian Xu, and Yang Yu.  Policy optimization in rlhf: The impact of out-of-preference data.  arXiv preprint arXiv:2312.10584v2, 2024.
  * [26] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2024.
  * [27] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in neural information processing systems, 35:27730–27744, 2022.
  * [28] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White.  Smaug: Fixing failure modes of preference optimisation with dpo-positive.  arXiv preprint arXiv:2402.13228, 2024.
  * [29] Jason Palmer.  Relative convexity.  UC San Diego Technical Report, 2003.
  * [30] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn.  Disentangling length from quality in direct preference optimization.  arXiv preprint arXiv:2403.19159, 2024.
  * [31] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  arXiv preprint arXiv:1910.00177, 2019.
  * [32] Jan Peters and Stefan Schaal.  Reinforcement learning by reward-weighted regression for operational space control.  In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007.
  * [33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  Advances in Neural Information Processing Systems, 36, 2024.
  * [34] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi.  Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.  arXiv preprint arXiv:2210.01241, 2022.
  * [35] B.D. Rao, K. Engan, S. F. Cotter, J. Palmer, and K. Kreutz-Delgado.  Subset selection in noise based on diversity measure minimization.  IEEE Trans. Signal Processing, 51(3):760–770, March 2003.
  * [36] Paul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya O Tolstikhin.  Practical and consistent estimation of f-divergences.  Advances in Neural Information Processing Systems, 32, 2019.
  * [37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [38] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback, 2020.  URL https://arxiv. org/abs, 2009.
  * [39] Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar.  Preference fine-tuning of llms should leverage suboptimal, on-policy data.  arXiv preprint arXiv:2404.14367, 2024.
  * [40] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot.  Generalized preference optimization: A unified approach to offline alignment.  arXiv preprint arXiv:2402.05749, 2024.
  * [41] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen.  Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints.  International Conference on Learning Representations, 2024.
  * [42] David Wipf and Srikantan Nagarajan.  Iterative reweighted $\ell_{1}$ and $\ell_{2}$ methods for finding sparse solutions.  Journal of Selected Topics in Signal Processing (Special Issue on Compressive Sensing), 4(2), 2010.
  * [43] David Wipf and Haichao Zhang.  Revisiting Bayesian blind deconvolution.  Journal of Machine Learning Research (JMLR), 2014.
  * [44] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * [45] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  SLiC-HF: Sequence likelihood calibration with human feedback.  arXiv preprint arXiv:2305.10425, 2023.
  * [46] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  arXiv preprint arXiv:1909.08593, 2019.

##  Appendix A Additional Experimental Details and Results

This section describes experiment details/settings and additional results.

###  A.1 Details of the Tests with Synthetic Data

  * •

For the tests of interpolation, preservation and constraints, we train the
models with Adam optimizer [23] and clip the gradients via a max norm of 10.
And we run the experiments of the tests on a single A10 GPU. Unless otherwise
mentioned, we use batch size of 1.

  * •

For the interpolation tests, we use batch size of 20 and choose
$\pi_{\tiny\mbox{ref}}(y_{a})=0.4$, $\pi_{\tiny\mbox{ref}}(y_{b})=0.4$, and
$\pi_{\tiny\mbox{ref}}(y_{c})=0.2$. We use learning rate of $1e-3$ for DPO,
IPO and $f$-DPO and $5e-4$ for TYPO; we train DPO, IPO and TYPO for 1,000
epochs and $f$-DPO for 3,000 epochs as it converges slower.

  * •

For the preservation test, we choose

$\displaystyle{\mathcal{Y}}(x_{g})=\\{y_{ga},y_{gb},y_{gc}\\};~{}~{}~{}{\mathcal{Y}}(x_{b})=\\{y_{ba},y_{bb},y_{bc}\\}$
$\displaystyle\pi^{*}(y_{ga}|x_{g})=0.6;~{}~{}\pi^{*}(y_{gb}|x_{g})=0.3;~{}~{}\pi^{*}(y_{gc}|x_{g})=0.1;$
(18)
$\displaystyle\pi^{*}(y_{ba}|x_{b})=0.4;~{}~{}\pi^{*}(y_{bb}|x_{b})=0.2;~{}~{}\pi^{*}(y_{bc}|x_{b})=0.4.$

And for the reference model we select
$\pi_{\tiny\mbox{ref}}(y_{ba}|x_{b})=0.6$,
$\pi_{\tiny\mbox{ref}}(y_{bb}|x_{b})=0.2$ and
$\pi_{\tiny\mbox{ref}}(y_{bc}|x_{b})=0.2$. We randomly sample examples for
good and bad prompts respectively. The model parameters are
$\theta\in\mathbb{R}^{2\times 3}$ and we set the values of $x_{g}$ and $x_{b}$
as vectors of $[1,0]$ and $[0,1]$.

  * •

In the constraint test, we use the same setting and data as the interpolation
test. We use $\beta=0.1$ for both RLHF and DPO and train them for 100 epochs
for all the values of $\alpha$.

###  A.2 Additional Results with Synthetic Data

We conduct additional experiments for the interpolation test by varying
$\lambda$ from very small to very large values as shown in Figure 5 and Figure
6.

Figure 5: Converged probability distributions of $\pi_{\theta}(y)$ for DPO,
IPO, $f$-DPO and TYPO with large $\lambda$. All methods stabilize around
$\pi_{\tiny\mbox{ref}}$ as expected. Figure 6: Interpolation of converged
probability distributions $\pi_{\theta}(y)$ for DPO, IPO and TYPO across
varying $\lambda$. As $\lambda$ becomes small, only TYPO converges to the BT-
optimal policy $\pi^{*}$. The others converge to the mode of the optimal
policy consistent with expectations. Meanwhile, as $\lambda$ grows all methods
converge to $\pi_{\tiny\mbox{ref}}$.

###  A.3 Details of Experiments on Anthropic HH Dataset

We train the SFT model with 2 epochs and 1 epoch for all the other models with
a learning rate of $1e-6$ and batch size of 40. We set $\beta=0.1$ for DPO,
$\tau=0.1$ for IPO and $\lambda=0.05$ for TYPO. We evaluate the win rate on
the single turn dialogues in the test set with GPT-4 using modified version
used in the DPO paper to cover harmlessness examples as shown in Figure 7. All
the experiments are conducted in a 8$\times$A100 40G GPU instance.

For the training of TYPO, we first sample responses from the reference model,
i.e. the SFT model, for the unsupervised term. We apply vLLM [24] to randomly
sample responses from the Anthropic HH dataset by setting temperature=1,
top_k=60, top_p=0.8, max_tokens=256 and repetition_penalty=1.1. During the
training, we use one sampled response for each prompt in the unsupervised
term.

Figure 7: Prompt used for evaluate win rate of the generated responses against
the chosen responses for single turn dialogues on the test set of Anthropic HH
dataset.

##  Appendix B Extended Related Work

There has been a flurry of interesting recent work on DPO-related topics, with
numerous papers appearing on arXiv not long before the NeurIPS deadline. In
this section we call attention to several notable examples that propose
modifications of the original DPO paradigm, or else provide relevant analysis
of its properties. We believe these efforts to be complementary to our
contribution, as well as the existing DPO-like extensions by others discussed
in the main body of paper.

#### Algorithmic Enhancements to DPO:

There exist multiple DPO extensions that involving supplementing the original
loss from (7) with additional penalty factors targeting potential failure
modes. For example, based on the observation that DPO may exhibit a decrease
in accuracy when applied to preference data with small edit distances between
responses, the Smaug framework [28] augments the DPO loss with an additional
factor designed to maintain high log-likelihoods in such cases. Meanwhile,
sensitivity to response lengths are investigated in [30], where as a counter-
measure, the DPO loss is supplemented with a penalty on length differences
between winning and losing responses. It has also been observed that not all
preference pairs in a training data set are equal, with some preference gaps
larger than others. As a mitigation strategy for this discrepancy, the ODPO
approach [2] introduces a preference offset term during model training. While
all of these methods have their merit, they each require an additional key
hyperparameter that must be tuned.

Somewhat differently, the ORPO algorithm [21] proposes an alternative to DPO
that combines an odds ratio-based penalty with a conventional negative log-
likelihood SFT (i.e., supervised fine-tuning) loss. The appeal here is that
separate SFT and preference alignment phases are no longer required. Another
deviation from DPO is proposed in [19], whereby the reference policy itself is
no longer fixed, but iteratively updated during training.

#### Analysis of DPO:

Topics addressed by recent work include analysis of DPO learning dynamics
[22], the impact of out-of-preference data on estimation errors [25], and the
disproportionate rates with which the DPO loss gradients favor reducing the
probability of dispreferred responses relative to increasing the probability
of desired responses [16]. Broader consideration of preference optimization
spanning various DPO-based and RLHF-based approaches is presented in [39]

##  Appendix C DPO Loss Induces Noise Adaptive Regularization

Using several straightforward algebraic manipulations, the DPO loss from (7)
can be modified as

$\displaystyle\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[-\log\sigma\left(\lambda\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}-\lambda\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right)\right]$
(19) $\displaystyle\equiv$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[\log\left(\left[\frac{\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]^{\lambda}+\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}\right]^{\lambda}\right)\right],$

excluding constants independent of $\pi_{\theta}$. This expression represents
an expectation over a regularization factor in the form $\log(\gamma+u)$,
where $\gamma$ corresponding to
$\left[\frac{\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]^{\lambda}$
is fixed, and $u$ corresponding to
$\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}\right]^{\lambda}$
is the variable of interest to be optimized. We will now examine several
notable properties of $\log(\gamma+u)$ that serve to elucidate
underappreciated DPO regularization characteristics. For this purpose, we
first introduce the following definition from [29]:

###### Definition 3

Let $f$ be a strictly increasing differentiable function on the interval
$[a,b]$. Then the differentiable function $g$ is concave relative to $f$ on
$[a,b]$ iff

$g(u_{2})\leq
g(u_{1})+\frac{g^{\prime}(u_{1})}{f^{\prime}(u_{1})}\left[f(u_{2})-f(u_{1})\right],$
(20)

where $g^{\prime}$ and $f^{\prime}$ denote the respective derivatives.

Intuitively, this definition indicates that if $g$ is concave relative to $f$,
it has greater curvature at any evaluation point $u$ once normalizing (via an
affine transformation of $f$ or $g$) such that $g(u)=f(u)$ and
$g^{\prime}(u)=f^{\prime}(u)$. Equipped with this definition, we then point
out the following observations linking DPO with prior work on robust
estimation in the presence of noise:

  * •

$\log(\gamma+u)$ is a concave non-decreasing function of $u\in[0,\infty)$,
which represents a well-known characteristic of sparsity-favoring penalty
factors commonly used in robust estimation [12, 13, 15, 35].777Most prior work
involves parameters that can be negative, which can be accommodated by simply
replacing $u$ with $|u|$. Such penalties introduce a steep gradient around
zero, but then flatten away from zero to avoid incurring significant
additional loss (as would occur, for example, with a common quadratic loss).

  * •

For any $\gamma_{1}<\gamma_{2}$, $\log(\gamma_{1}+u)$ is concave relative to
$\log(\gamma_{2}+u)$ per Definition 3. Figure 8 illustrates this phenomena by
contrasting with two extremes producing the convex $\ell_{1}$ norm and the
non-convex $\ell_{0}$ norm.

  * •

Prior work [10, 42] has investigated general optimization problems of the form

$\min_{\\{u_{i}\\}\in{\mathcal{S}}_{u}}\sum_{i}\log(\gamma+|u_{i}|),$ (21)

sometimes generalized to
$\min_{\\{u_{i}\\}\in{\mathcal{S}}_{u}}\sum_{i}f(|u_{i}|,\gamma)$ over a
concave, non-decreasing function $f$ of $|u_{i}|$, where $S_{u}$ is some
constraint set.888In some applications the constraint set may be replaced by
an additional regularization factor, and there is often an equivalency between
the two. Moreover, $\gamma$ reflects a noise parameter or an analogous measure
of uncertainty, with relative concavity dictated by $\gamma$ as above. In
these contexts, it has been argued that adjusting the curvature of the
regularization factor based on noise levels can provide additional robustness
to bad local minima and high noise regimes [10, 14, 43]. The basic intuition
here is that when noise is high, a more convex shape is preferable, while when
the noise is low, a more concave alternative may be appropriate.

  * •

Regarding DPO, it is natural to treat
$\left[\frac{\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]^{\lambda}$
as an analogous noise factor, given that whenever this ratio is large, it
implies that our reference policy is poor. Hence, once we introduce a
constraint ${\mathcal{S}}_{\pi}$ on $\pi_{\theta}$ (as will always occur in
practice; see Section 3.3), solving

$\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi}}\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$
(22)

can be viewed as a special case of (21), involving a robust regularization
factor with noise-adaptive curvature.

Figure 8: Visualization of different penalty factors associated with the DPO
loss. When $\gamma\rightarrow 0$,
$\log(\gamma+|u|)\rightarrow\log|u|=\lim_{p\rightarrow
0}\frac{1}{p}[|u|^{p}-1]\propto\mathbb{I}[u\neq 0]$ mimicking an $\ell_{0}$
norm (red curve) w.r.t. relative concavity (if $u\geq 0$ as with DPO, can
remove absolute value, but we nonetheless include the general case here.). In
contrast, $\lim_{\gamma\rightarrow\infty}\gamma\log(\gamma+|u|)=|u|$
reflecting the relative concavity of the convex $\ell_{1}$ norm (green curve).
Note that in both limiting cases, affine transformations do not impact
relative concavity. For a fixed $\gamma$ value, the relative concavity of
$\log(\gamma+|u|)$ lies within these two extremes.

##  Appendix D DPO from a Naive Gaussian Estimation Perspective

Any preference probability given by the BT model in (2) can be equivalently
re-expressed as

$p^{*}(y_{1}\succ
y_{2}|x)=\mu\left[\frac{\pi^{*}(y_{2}|x)}{\pi^{*}(y_{1}|x)}\right],$ (23)

where $\pi^{*}(y|x)$ is a conditional probability of $y$ given $x$ (i.e., the
BT-optimal policy introduced in Section 3) and
$\mu:\mathbb{R}\rightarrow[0,1]$ is a monotonically increasing function. While
we may optionally choose $\mu$ to exactly reproduce the BT model, it is of
course reasonable to consider other monotonically increasing choices to
explore the additional generality of (23) (and indeed we will exploit one such
alternative choice below).

Given a trainable policy $\pi_{\theta}$ we can always minimize the negative
log-likelihood
$-\log\mu\left[\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\theta}(y_{1}|x)}\right]$
averaged over preference samples $\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}$
to approximate $p^{*}(y_{1}\succ y_{2}|x)$; however, this procedure would be
completely independent of any regularization effects of a reference policy
$\pi_{\tiny\mbox{ref}}$. We now examine how to introduce the reference policy
by relying only on a simple Gaussian model with trainable variances, rather
than any association with RLHF or implicit reward modeling. The end result is
an independent re-derivation of DPO using basic Gaussian assumptions.

For convenience, we first define the functions $\xi_{\theta}$ and
$\xi_{\tiny\mbox{ref}}$ as

$\xi_{\theta}(y_{1},y_{2},x):=\mu\left[\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\theta}(y_{1}|x)}\right],~{}~{}~{}\xi_{\tiny\mbox{ref}}(y_{1},y_{2},x):=\mu\left[\frac{\pi_{\tiny\mbox{ref}}(y_{2}|x)}{\pi_{\tiny\mbox{ref}}(y_{1}|x)}\right].$
(24)

Now suppose we assume the naive joint distribution given by

$p\left(\left[\begin{array}[]{c}\xi_{\theta}(y_{1},y_{2},x)\\\
\xi_{\tiny\mbox{ref}}(y_{1},y_{2},x)\end{array}\right]\right)={\mathcal{N}}\left(\left.\left[\begin{array}[]{c}\xi_{\theta}(y_{1},y_{2},x)\\\
\xi_{\tiny\mbox{ref}}(y_{1},y_{2},x)\end{array}\right]\right|0,\gamma(y_{1},y_{2},x)I\right),$
(25)

where ${\mathcal{N}}(\cdot|0,\Sigma)$ denotes a 2D, zero-mean Gaussian with
covariance $\Sigma\in\mathbb{R}^{2\times 2}$, and
$\gamma(y_{1},y_{2},x)\in\mathbb{R}^{+}$ is a variance parameter that depends
on the tuple $\\{y_{1},y_{2},x\\}$. Since each $\gamma(y_{1},y_{2},x)$ is
unknown, we can group them together with $\pi_{\theta}$ and estimate all
unknowns jointly. In the context of labeled human preference data drawn from
${\mathcal{D}}_{tr}$, this involves minimizing

$\min_{\pi_{\theta}\in{\mathcal{S}}_{\pi},~{}\\{\gamma(y_{w},y_{l},x)>0\\}}\left\\{\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}-\log{\mathcal{N}}\left(\left[\begin{array}[]{c}\xi_{\theta}(y_{w},y_{l},x)\\\
\xi_{\tiny\mbox{ref}}(y_{w},y_{l},x)\end{array}\right]~{}\Big{|}~{}0,\gamma(y_{w},y_{l},x)I\right)\right\\},$
(26)

where $I$ is a $2\times 2$ identity matrix and ${\mathcal{S}}_{\pi}$ is any
constraint set on $\pi_{\theta}$ as introduced in Section 3.3. The intuition
here is that, although $\gamma(y_{w},y_{l},x)$ is unknown, sharing this
parameter across both $\xi_{\theta}$ and $\xi_{\tiny\mbox{ref}}$ and
estimating jointly will induce a reference policy-dependent regularization
effect.

And indeed, this simple Gaussian model exactly reproduces DPO. More
concretely, the stated equivalence follows from the fact that, for an
arbitrary vector $v$ we have that

$\arg\min_{\gamma>0}-\log{\mathcal{N}}(v|0,\gamma
I)\equiv\arg\min_{\gamma>0}\left[\frac{v^{\top}v}{\gamma}+\log|\gamma
I|\right]=\frac{1}{2}v^{\top}v.$ (27)

And therefore, we have

$\min_{\gamma>0}-\log{\mathcal{N}}(v|0,\gamma I)~{}\equiv~{}\log(v^{\top}v)$
(28)

excluding irrelevant constants. Returning to (26), if we first optimize over
$\gamma(y_{w},y_{l},x)$ for each tuple, we obtain the loss factor

$\log\left[\xi_{\tiny\mbox{ref}}(y_{w},y_{l},x)^{2}+\xi_{\theta}(y_{w},y_{l},x)^{2}\right]~{}=~{}\log\left[\mu\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}\right]^{2}+\mu\left[\frac{\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]^{2}\right].$
(29)

From here, by choosing $\mu(\cdot)=(\cdot)^{\frac{\lambda}{2}}$ we can modify
(29) as

$\displaystyle\log\left[\frac{\pi_{\theta}(y_{l}|x)^{\lambda}}{\pi_{\theta}(y_{w}|x)^{\lambda}}+\frac{\pi_{\tiny\mbox{ref}}(y_{l}|x)^{\lambda}}{\pi_{\tiny\mbox{ref}}(y_{w}|x)^{\lambda}}\right]$
$\displaystyle=$
$\displaystyle\log\left[1+\left(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right)^{\lambda}\left(\frac{\pi_{\tiny\mbox{ref}}(y_{w}|x)}{\pi_{\theta}(y_{w}|x)}\right)^{\lambda}\right]+C$
(30) $\displaystyle=$
$\displaystyle-\log\sigma\left(\lambda\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}-\lambda\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right),$

ignoring the irrelevant constant $C$ which is independent of $\pi_{\theta}$.
Hence we have recovered the DPO loss for each tuple $\\{y_{w},y_{l},x\\}$ and
once the requisite expectation is reintroduced, we exactly recover the full
DPO loss from (7).

##  Appendix E Technical Proofs

###  E.1 Proof of Theorem 1

###### Definition 4

We define labeled human preference data $\bar{{\mathcal{D}}}_{tr}$ as some
${\mathcal{D}}_{tr}$, as introduced via (1), satisfying the following
additional properties:

  1. 1.

The prompts drawn from $\bar{{\mathcal{D}}}_{tr}$ are split between two
disjoint support partitions $d_{x}^{good}$ and $d_{x}^{bad}$, i.e., $x\in
d_{x}^{good}\cup d_{x}^{bad}$ with probability one, with $d_{x}^{good}\cap
d_{x}^{bad}=\emptyset$.

  2. 2.

For each prompt $x\in d_{x}^{good}\cup d_{x}^{bad}$ within
$\bar{{\mathcal{D}}}_{tr}$, the preference distribution filling out
$\bar{{\mathcal{D}}}_{tr}$ maintains support over a single (prompt-dependent)
response pair $\\{y_{1},y_{2}\\}$.

  3. 3.

Pair-wise preferences are dictated by a ground-truth BT model satisfying
$p^{*}(y_{1}\succ y_{2}|x)\in(0,1)$ for all $x\in d_{x}^{good}\cup
d_{x}^{bad}$.

Although the second specification above can naturally be relaxed to address
more general scenarios, doing so unnecessarily complicates the presentation
without providing sufficiently compelling additional insight. Additionally,
for convenience below we adopt $\mbox{dist}[\cdot,\cdot]$ to indicate an
arbitrary distance measure.

#### Theorem 1

(Restated formal version) Assume preference data $\bar{{\mathcal{D}}}_{tr}$
that satisfies Definition 4. Furthermore, assume a reference policy
$\pi_{\tiny\mbox{ref}}$ such that $\pi_{\tiny\mbox{ref}}=\pi^{*}$ for $x\in
d_{x}^{good}$ and $\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]>0$ for $x\in
d_{x}^{bad}$, where $\pi^{*}$ is a BT-optimal policy. It follows that for any
selection of $(\psi,\mu,\lambda)$, if

$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{QPO}},~{}\pi^{*}]~{}~{}<~{}~{}\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]~{}~{}\mbox{for}~{}~{}x\in
d_{x}^{bad},$ (31)

then

$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{QPO}},~{}\pi^{*}]~{}~{}>~{}~{}0~{}~{}\mbox{for}~{}~{}x\in
d_{x}^{good},$ (32)

where
$\hat{\pi}_{\theta}^{\tiny\mbox{QPO}}:=\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{QPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\psi,\mu,\lambda)$.

The proof proceeds as follows. With some abuse/imprecision of notation, we
first define

$u(y_{1},y_{2},x):=\mu\left[\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\tiny\mbox{ref}}(y_{1}|x)}\right]-\mu\left[\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\tiny\mbox{ref}}(y_{2}|x)}\right].$
(33)

Next, per the assumptions of the theorem statement and Definition 4, we have
that the QPO loss decouples as

$\displaystyle\ell_{\tiny\mbox{QPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\psi,\mu,\lambda)$
$\displaystyle=~{}~{}\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim\bar{{\mathcal{D}}}_{tr}}~{}\psi\left(\mu\left[\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]-\mu\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right],\lambda\right)$
(34)
$\displaystyle=~{}~{}\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{(}p^{*}(y_{1}\succ
y_{2}|x)\psi\Big{[}u(y_{1},y_{2},x),\lambda\Big{]}+p^{*}(y_{2}\succ
y_{1}|x)\psi\Big{[}u(y_{2},y_{1},x),\lambda\Big{]}\Big{)}$
$\displaystyle=~{}~{}\mathbb{E}_{x\sim d_{x}^{good}}\Big{[}p^{*}(y_{1}\succ
y_{2}|x)\psi[u(y_{1},y_{2},x),\lambda]+p^{*}(y_{2}\succ
y_{1}|x)\psi[-u(y_{1},y_{2},x),\lambda]\Big{]}$ $\displaystyle\hskip
28.45274pt+~{}~{}\mathbb{E}_{x\sim d_{x}^{bad}}\Big{[}p^{*}(y_{1}\succ
y_{2}|x)\psi[u(y_{1},y_{2},x),\lambda]+p^{*}(y_{2}\succ
y_{1}|x)\psi[-u(y_{1},y_{2},x),\lambda]\Big{]}.$

Now consider a single prompt $x^{bad}$ drawn from $d_{x}^{bad}$. In order to
reduce $\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]$, it must be the case
that $\pi_{\theta}(y|x^{bad})\neq\pi_{\tiny\mbox{ref}}(y|x^{bad})$, which then
implies that $u(y_{1},y_{2},x^{bad})\neq 0$. To achieve this,
$(\psi,\mu,\lambda)$ must be chosen such that

$\arg\min_{u(y_{1},y_{2},x^{bad})}\Big{[}p^{*}(y_{1}\succ
y_{2}|x^{\prime})\psi[u(y_{1},y_{2},x^{bad}),\lambda]+p^{*}(y_{2}\succ
y_{1}|x^{bad})\psi[-u(y_{1},y_{2},x^{bad}),\lambda]\Big{]}\neq 0.$ (35)

However, to simultaneously maintain
$\pi_{\theta}(y|x^{good})=\pi_{\tiny\mbox{ref}}(y|x^{good})=\pi^{*}(y|x^{good})$
for some prompt $x^{good}$ drawn from $d_{x}^{good}$, it must also be true,
for the same fixed $(\psi,\mu,\lambda)$ tuple, that

$\arg\min_{u(y_{1},y_{2},x^{good})}\Big{[}p^{*}(y_{1}\succ
y_{2}|x^{\prime})\psi[u(y_{1},y_{2},x^{good}),\lambda]+p^{*}(y_{2}\succ
y_{1}|x^{good})\psi[-u(y_{1},y_{2},x^{good}),\lambda]\Big{]}=0.$ (36)

But this is a contradiction, as the respective arguments that minimize (35)
and (36) will be identical. Hence if (35) is true then
$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{QPO}},~{}\pi^{*}]~{}~{}>~{}~{}0$
for $x\in d_{x}^{good}$. $\blacksquare$

###  E.2 Proof of Proposition 1

#### DPO lower limit:

Given our assumption that $0<p^{*}(y_{1}\succ y_{2}|x)<1$, it follows that an
optimal finite reward $r^{*}(y,x)\in(-\infty,\infty)$ exists. Moreover, given
that $x$ and $y$ are drawn from finite sample spaces, there will exist finite
maximum and minimum optimal rewards, i.e., $r^{*}(y,x)\in(-B,B)$ for some
$B<\infty$. Furthermore,

$\lim_{\lambda\rightarrow
0}\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)=\arg\max_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\big{[}r^{*}(y,x)\big{]}=\pi^{\delta}(y|x).$
(37)

Additionally, given that the data are generated by (1), we also know that the
same optimal reward satisfies

$r^{*}=\arg\min_{r_{\phi}}\ell_{\tiny\mbox{BT}}\left(r_{\phi}\right),$ (38)

which is independent of $\pi_{\tiny\mbox{ref}}$. However, without constraints
on $\pi_{\theta}$, there also exists a bijection between policy and reward
such that

$\lambda\log\left[\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{BT}}\left(\lambda\log\frac{\pi_{\theta}(y|x)}{\pi_{\tiny\mbox{ref}}(y|x)}\right)\right]-\lambda\log\pi_{\tiny\mbox{ref}}(y|x)~{}~{}=~{}~{}r^{*}.$
(39)

Hence the DPO reparameterization produces the policy given by (5) with
$r=r^{*}$. From this point we then observe that

$\lim_{\lambda\rightarrow
0}\frac{1}{Z(x)}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r^{*}(y,x)\right]=\pi^{\delta}(y|x),$
(40)

noting that for any $\alpha>\beta>0$ we have
$\exp\left[\frac{\alpha}{\lambda}\right]/\exp\left[\frac{\beta}{\lambda}\right]=\exp\left[\frac{(\alpha-\beta)}{\lambda}\right]\rightarrow\infty$
as $\lambda\rightarrow 0$. Hence we have fulfilled the requirements of the
lower limit.

#### DPO upper limit:

The upper limit follows trivially from the fact that for any bounded reward

$\lim_{\lambda\rightarrow\infty}\frac{1}{Z(x)}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r(y,x)\right]=\frac{1}{Z(x)}\pi_{\tiny\mbox{ref}}(y|x)\exp[0]=\pi_{\tiny\mbox{ref}}.$
(41)

$\blacksquare$

###  E.3 Proof of Proposition 2

Establishing the upper and lower limiting values for IPO follows a similar
pattern to the proof of Proposition 2. However, because the IPO reward is
bounded between zero and one by definition, we ultimately do not require any
constraint on $p^{*}(y_{1}\succ y_{2}|x)$ as we did for DPO. $\blacksquare$

###  E.4 Proof of Theorem 2

We first define

$\hat{\rho}:=\arg\min_{\rho}\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim\bar{{\mathcal{D}}}_{tr}}~{}\psi\Big{[}\rho(y_{w},y_{l},x,\pi_{\theta},\pi_{\tiny\mbox{ref}}),\lambda\Big{]}.$
(42)

Now suppose that for a given tuple $\\{y_{w},y_{l},x\\}$ we observe

$\hat{\rho}(y_{w},y_{l},x,\pi_{\theta},\pi_{\tiny\mbox{ref}})=\log\left[\frac{\hat{\pi}_{\theta}(y_{w}|x)\pi_{\tiny\mbox{ref}}(y_{l}|x)}{\hat{\pi}_{\theta}(y_{l}|x)\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]=B(\lambda)$
(43)

for some optimal $\hat{\pi}_{\theta}$ and fixed $\lambda\in(0,\infty)$, where
$B(\lambda)\in(0,\infty)$ is a finite value dependent on $\lambda$ through the
definition of $\psi$. Therefore, we have that

$\frac{\hat{\pi}_{\theta}(y_{w}|x)}{\hat{\pi}_{\theta}(y_{l}|x)}=\exp\left(B(\lambda)+\log\left[\frac{\pi_{\tiny\mbox{ref}}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right]\right).$
(44)

Obviously this ratio will depend on $\pi_{\tiny\mbox{ref}}$ for any fixed
$B(\lambda)$. To satisfy the SIC though, in the limit $\lambda\rightarrow 0$
the optimized policy $\hat{\pi}_{\theta}$ must be independent of
$\pi_{\tiny\mbox{ref}}$ and converge to $\pi^{*}$. However, the only way for
$\hat{\pi}_{\theta}$ to be independent of $\pi_{\tiny\mbox{ref}}$ is if
$\lim_{\lambda\rightarrow 0}B(\lambda)=\pm\infty$. But if so, only the WIC is
achievable, not the SIC. $\blacksquare$

###  E.5 Proof of Theorem 3

Our strategy here is to construct a simplified situation whereby we can
pinpoint emergent differences between RLHF and DPO losses in the presence of
policy constraints. To this end, we assume the following:

  * •

For all $x\sim{\mathcal{D}}_{x}$, there exists two unique responses $y_{1}$
and $y_{2}$ with equal probability of 1/2 under $\pi_{\tiny\mbox{ref}}$;

  * •

Preference data $\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}$ are sampled
according to (1);

  * •

The loss trade-off parameter satisfies $\lambda=1$; and

  * •

$p^{*}(y_{1}\succ y_{2}|x)\in(0,1)$ for all
$\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x)$ and $x\in{\mathcal{D}}_{x}$.

#### RLHF loss processing:

When evaluated with optimal reward model $r^{*}$, we have that

$\displaystyle\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)$
$\displaystyle=$
$\displaystyle\mathbb{E}_{y\sim\pi_{\theta}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}-r^{*}(y,x)\Big{]}+\lambda~{}\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}\pi_{\theta}(y|x)||\pi_{\tiny\mbox{ref}}(y|x)\big{]}\Big{]}$
(45) $\displaystyle\equiv$
$\displaystyle\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}\pi_{\theta}(y|x)||\pi^{**}(y|x)\big{]}\Big{]},$

where

$\pi^{**}(y|x)~{}:=~{}\frac{1}{Z(x)}\pi_{\tiny\mbox{ref}}(y|x)\exp\left[\frac{1}{\lambda}r^{*}(y,x)\right].$
(46)

This stems directly from the analysis in [31, 32]. However, because we are
assuming $\lambda=1$ and $\pi_{\tiny\mbox{ref}}(y|x)$ is constant for any
given $x$, it follows that

$\pi^{**}(y|x)=\frac{\exp\left[r^{*}(y,x)\right]}{\sum_{y}\exp\left[r^{*}(y,x)\right]},$
(47)

where the denominator is independent of $y$. Since the BT-optimal solution
$\pi^{*}$ satisfies

$\frac{\pi^{*}(y_{1}|x)}{\pi^{*}(y_{1}|x)+\pi^{*}(y_{2}|x)}=p^{*}(y_{1}\succ
y_{2}|x)=\frac{\exp\left[r^{*}(y_{1},x)\right]}{\exp\left[r^{*}(y_{1},x)\right]+\exp\left[r^{*}(y_{2},x)\right]},$
(48)

we may conclude that $\pi^{**}=\pi^{*}$, and therefore

$\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)~{}=~{}\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}\pi_{\theta}(y|x)||\pi^{*}(y|x)\big{]}\Big{]}$
(49)

under the stated conditions.

#### DPO loss processing:

When $\lambda=1$ and $\pi_{\tiny\mbox{ref}}(y|x)$ is constant, we have that

$\displaystyle\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}\left[-\log\sigma\left(\lambda\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}-\lambda\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right)\right]$
(50) $\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\left[\log\left(\frac{\pi_{\theta}(y_{w}|x)+\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}\right)\right].$

Next, given the additional data generation assumptions, it follows that
$\pi_{\theta}(y_{w}|x)+\pi_{\theta}(y_{l}|x)=1$, and so the DPO loss can be
further modified as

$\displaystyle\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\left[\log\left(\frac{1}{\pi_{\theta}(y_{w}|x)}\right)\right]$
(51) $\displaystyle=$
$\displaystyle\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\left[p^{*}(z=1|y_{1},y_{2},x)\log\left(\frac{1}{\pi_{\theta}(y_{1}|x)}\right)\right.$
$\displaystyle\hskip
28.45274pt\left.+~{}(p^{*}(z=0|y_{1},y_{2},x)\log\left(\frac{1}{\pi_{\theta}(y_{2}|x)}\right)\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\left[\pi^{*}(y_{1}|x)\log\left(\frac{1}{\pi_{\theta}(y_{1}|x)}\right)\right.$
$\displaystyle\hskip
28.45274pt\left.+~{}\pi^{*}(y_{2}|x)\log\left(\frac{1}{\pi_{\theta}(y_{2}|x)}\right)\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\left[\pi^{*}(y_{1}|x)\log\left(\frac{\pi^{*}(y_{1}|x)}{\pi_{\theta}(y_{1}|x)}\right)\right.$
$\displaystyle\hskip
28.45274pt\left.+~{}\pi^{*}(y_{2}|x)\log\left(\frac{\pi^{*}(y_{2}|x)}{\pi_{\theta}(y_{2}|x)}\right)\right]~{}+C$
$\displaystyle\equiv$
$\displaystyle\mathbb{E}_{x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}\pi^{*}(y|x)||\pi_{\theta}(y|x)\big{]}\Big{]},$

where $C$ is an irrelevant constant. Note that in progressing from the first
to second equality, we can ignore cases where where sampled responses satisfy
$y_{1}=y_{2}$, since these contribute only another irrelevant constant to the
loss. Along with our stated response data assumptions, this allows us to
remove expectation over $\\{y_{1},y_{2}\\}$ without loss of generality.

#### Final step:

From (49) and (51) we observe that the only difference between the RLHF and
DPO losses under the given conditions is whether a forward or backward KL is
used. And of course without any constraints, the minimizing solutions are
equivalent as expected, consistent with the analysis from [33], i.e.,

$\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{RLHF}}\left(\pi_{\theta},\pi_{\tiny\mbox{ref}},r^{*},\lambda\right)~{}=~{}\arg\min_{\pi_{\theta}}\ell_{\tiny\mbox{DPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda).$
(52)

Critically though, this KL equivalence transparently need not still hold once
constraints are introduced, as the forward KL will favor mode covering while
the backward KL will push mode following [7]. $\blacksquare$

###  E.6 Proof of Propositions 3 and 4

These results both follow directly from the original design of
$\ell_{\tiny\mbox{TYPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},\lambda)$.
Regarding Proposition 3, given that $\pi_{\tiny\mbox{ref}}=\pi^{*}$ for all
$x\in d_{x}^{good}$, then for the unsupervised term we have

$\arg\min_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\tiny\mbox{ref}}(y|x),x\in
d_{x}^{good}}\Big{[}\mathbb{KL}\big{[}\pi_{\tiny\mbox{ref}}(y|x)||\pi_{\theta}(y|x)\big{]}\Big{]}~{}~{}=~{}~{}\pi^{*}.$
(53)

And for the supervised term we have

$\arg\min_{\pi_{\theta}}\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}p^{*}(z|y_{1},y_{2},x)||p_{\theta}(z|y_{1},y_{2},x)\big{]}\Big{]}~{}~{}=~{}~{}\pi^{*}.$
(54)

Hence overall, for any $x\in d_{x}^{good}$, $\pi_{\theta}=\pi^{*}$ will be
optimal for any $\lambda$, as this selection independently optimizes the
constituent terms. Moreover, this optimality is independent of optimization
over $x\in d_{x}^{bad}$, which retains the flexibility to achieve solutions
with
$\mbox{dist}[\hat{\pi}_{\theta}^{\tiny\mbox{TYPO}},~{}\pi^{*}]<\mbox{dist}[\pi_{\tiny\mbox{ref}},~{}\pi^{*}]$.
From this Proposition 3 immediately follows.

Additionally, Proposition 4 follows from the same basic line of reasoning. For
completeness, we note that when $\lambda\rightarrow 0$, only the supervised
term will be minimized (which recovers the BT-optimal policy as above), while
when $\lambda\rightarrow\infty$, the unsupervised term will dominate the
optimization (which transparently produces $\pi_{\tiny\mbox{ref}}$).
$\blacksquare$

##  Appendix F Other Derivations

###  F.1 Derivation of (12)

Note that

$\displaystyle p^{*}(y_{1}\succ y_{2}|x)$ $\displaystyle=$
$\displaystyle\frac{\exp[r^{*}(y_{1},x)]}{\exp[r^{*}(y_{1},x)]+\exp[r^{*}(y_{2},x)]}=\frac{\frac{\exp[r^{*}(y_{1},x)]}{Z(x)}}{\frac{\exp[r^{*}(y_{1},x)]}{Z(x)}+\frac{\exp[r^{*}(y_{2},x)]}{Z(x)}}$
(55) $\displaystyle=$
$\displaystyle\frac{\pi^{*}(y_{1}|x)}{\pi^{*}(y_{1}|x)+\pi^{*}(y_{2}|x)},$

where $\pi^{*}(y|x)=\frac{\exp[r^{*}(y_{1},x)]}{Z(x)}$ and
$Z(x):=\sum_{y}\exp[r^{*}(y,x)]$. The policy $\pi^{*}$ so-defined is
necessarily BT-optimal by construction. From here then we have

$\displaystyle\arg\max_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\big{[}r^{*}(y,x)\big{]}$
$\displaystyle=$
$\displaystyle\arg\max_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\big{[}r^{*}(y,x)\big{]}$
(58) $\displaystyle=$
$\displaystyle\arg\max_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\left[\frac{\exp[r^{*}(y_{1},x)]}{Z(x)}\right]$
$\displaystyle=$
$\displaystyle\arg\max_{\pi_{\theta}}\mathbb{E}_{y\sim\pi_{\theta}(y|x)}\big{[}\pi^{*}(y|x)\big{]}$
$\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{cc}1&\mbox{if}~{}~{}y=\arg\max_{y^{\prime}}\pi^{*}(y^{\prime}|x)\\\
0&\mbox{otherwise}\end{array}\right.,$

which is the definition of $\pi^{\delta}$. $\blacksquare$

###  F.2 Additional $f$-DPO Analysis

$f$-PDO represents a novel generalization of DPO, but there remain certain
aspects worth considering.

#### Minima that ignore the reference policy:

Consider general $f$-DPO losses as described in Section 2.4, which as special
cases of QPO are expressible in the form

$\displaystyle\ell_{\tiny\mbox{QPO}}(\pi_{\theta},\pi_{\tiny\mbox{ref}},-\log\sigma[\lambda(\cdot)],f^{\prime},\lambda)=$
(59)
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{\tiny\mbox{tr}}}~{}-\log\sigma\left(\lambda
f^{\prime}\left[\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\tiny\mbox{ref}}(y_{w}|x)}\right]-\lambda
f^{\prime}\left[\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\tiny\mbox{ref}}(y_{l}|x)}\right],\lambda\right).$

In addition to the requirements on $f$ to form an $f$-divergence, to produce a
valid $f$-DPO loss per Theorem 1 from [41] it must be that $f^{\prime}$ is
invertible with $0\notin\mbox{domain of }f^{\prime}$. Therefore the domain of
$f$ will be $(0,\infty)$ and $f^{\prime}(u)\rightarrow-\infty$ as
$u\rightarrow 0$ because of convexity. But if this is the case, upon
inspection of (59) we observe that when $\pi_{\theta}(y_{l}|x)\rightarrow 0$,
then for any fixed $\pi_{\theta}(y_{w}|x)>0$ the input argument to the
logistic function $\sigma(\cdot)=\frac{1}{1+\exp[-(\cdot)]}$ will converge to
infinity, pushing the output to one and subsequently minimizing the
corresponding negative-log factor. And so the global optimum can be achieved
independent of the value of $\pi_{\tiny\mbox{ref}}$. $\blacksquare$

###  F.3 Derivation of (16)

$\displaystyle d_{\tiny\mbox{sup}}(\pi_{\theta},\pi_{\tiny\mbox{ref}})$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{KL}\big{[}p^{*}(z|y_{1},y_{2},x)||p_{\theta}(z|y_{1},y_{2},x)\big{]}\Big{]}$
(60) $\displaystyle=$
$\displaystyle-\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}\mathbb{E}_{z\sim
p^{*}(z|y_{1},y_{2},x)}\log p_{\theta}(z|y_{1},y_{2},x)\Big{]}~{}+~{}C$
$\displaystyle\equiv$
$\displaystyle-\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}p^{*}(z=1|y_{1},y_{2},x)\log
p_{\theta}(z=1|y_{1},y_{2},x)\Big{]}$
$\displaystyle+~{}~{}-\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}p^{*}(z=0|y_{1},y_{2},x)\log
p_{\theta}(z=0|y_{1},y_{2},x)\Big{]},$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\\{y_{1},y_{2}\\}\sim\pi_{\tiny\mbox{ref}}(y|x),x\sim{\mathcal{D}}_{x}}\Big{[}p^{*}(z=1|y_{1},y_{2},x)\log
p_{\theta}(z=1|y_{1},y_{2},x)$ $\displaystyle\hskip
102.43008pt+~{}~{}p^{*}(z=1|y_{2},y_{1},x)\log
p_{\theta}(z=1|y_{2},y_{1},x)\Big{]}$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\Big{[}\log
p_{\theta}(z=1|y_{w},y_{l},x)\Big{]}$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\left[\log\left(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\theta}(y_{w}|x)+\pi_{\theta}(y_{l}|x)}\right)\right],$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\\{y_{w},y_{l},x\\}\sim{\mathcal{D}}_{tr}}\left[\log\left(1+\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\theta}(y_{w}|x)}\right)\right],$

where $C$ is a constant independent of $\theta$. Additionally, the third-to-
last equality stems from the definition of how tuples $\\{y_{w},y_{l},x\\}$
are sampled. In particular, for a given pair $\\{y_{1},y_{2}\\}$, by
definition a proportion $p^{*}(z=1|y_{1},y_{2},x)$ of the time $y_{w}=y_{1}$,
while a proportion $p^{*}(z=0|y_{1},y_{2},x)=p^{*}(z=1|y_{2},y_{1},x)$ of the
time $y_{w}=y_{2}$. Hence

$\displaystyle p^{*}(z=1|y_{1},y_{2},x)\log
p_{\theta}(z=1|y_{1},y_{2},x)~{}+~{}p^{*}(z=1|y_{2},y_{1},x)\log
p_{\theta}(z=1|y_{2},y_{1},x)$ $\displaystyle~{}~{}\equiv~{}~{}\log
p_{\theta}(z=1|y_{w},y_{l},x)$ (61)

when the latter is averaged over the preference distribution. $\blacksquare$

##  Appendix G Limitations

As more of an analysis-driven contribution, our experiments on real-world data
are limited to Figure 4. Moreover, there are promising possibilities raised by
pairing our contribution with prior work in new ways that we have not yet been
explored. One example is the potential use of REINFORCE in conjunction with
modifications to the proposed $\ell_{\tiny\mbox{TYPO}}$ loss.

##  Appendix H Broader Impacts

Aligning the output of LLMs with human preferences has obvious, well-
documented benefits. However, there nonetheless remains the risk that tools
designed to improve LLM responses could be repurposed for nefarious aims. For
example, preference data labels could potentially be modified to train models,
using preference losses such as ours, that intentionally produce toxic
content.