# Human Alignment of Large Language Models through
Online Preference Optimisation

Daniele Calandriello  Daniel Guo  Rémi Munos  Mark Rowland  Yunhao Tang
Bernardo Avila Pires  Pierre Harvey Richemond  Charline Le Lan  Michal Valko
Tianqi Liu  Rishabh Joshi  Zeyu Zheng  Bilal Piot

###### Abstract

Ensuring alignment of language models’ outputs with human preferences is
critical to guarantee a useful, safe, and pleasant user experience. Thus,
human alignment has been extensively studied recently and several methods such
as Reinforcement Learning from Human Feedback (RLHF), Direct Policy
Optimisation (DPO) and Sequence Likelihood Calibration (SLiC) have emerged. In
this paper, our contribution is two-fold. First, we show the equivalence
between two recent alignment methods, namely Identity Policy Optimisation
(IPO) and Nash Mirror Descent (Nash-MD). Second, we introduce a generalisation
of IPO, named IPO-MD, that leverages the regularised sampling approach
proposed by Nash-MD.

This equivalence may seem surprising at first sight, since IPO is an offline
method whereas Nash-MD is an online method using a preference model. However,
this equivalence can be proven when we consider the online version of IPO,
that is when both generations are sampled by the online policy and annotated
by a trained preference model. Optimising the IPO loss with such a stream of
data becomes then equivalent to finding the Nash equilibrium of the preference
model through self-play. Building on this equivalence, we introduce the IPO-MD
algorithm that generates data with a mixture policy (between the online and
reference policy) similarly as the general Nash-MD algorithm. We compare
online-IPO and IPO-MD to different online versions of existing losses on
preference data such as DPO and SLiC on a summarisation task.

Machine Learning, ICML, RLHF, RL, reinforcement learning, policy optimisation

##  1 Introduction

Learning from feedback is a common approach to align the behaviour of
artificial agents with human preferences (Knox & Stone, 2008; Griffith et al.,
2013; Christiano et al., 2017; Warnell et al., 2018). In recent years,
reinforcement learning from human feedback has become a common paradigm for
fine-tuning large language models (Glaese et al., 2022; OpenAI, 2022).

The traditional approach to fine-tuning of large language models from human
preferences is to learn a reward signal under the Bradley-Terry model (Bradley
& Terry, 1952), and then perform reinforcement learning (RL) against this
learnt reward signal (Christiano et al., 2017). Recently, Rafailov et al.
(2023) proposed an alternative model-free approach, direct policy optimisation
(DPO). DPO is mathematically equivalent to the method above, in the sense that
the minimiser of the population loss is identical (Azar et al., 2023), yet DPO
bypasses the learning of the reward signal. Both approaches, however, rely on
the Bradley-Terry model.

Recently, two particular approaches to directly optimise against preference
probabilities themselves, rather than a Bradley-Terry-derived reward function
have been proposed. Identity preference optimisation (Azar et al., 2023, IPO)
is an algorithm that aims to optimise preference probabilities against a fixed
data distribution, and does so with an offline contrastive loss, as with DPO.
By contrast, Nash-MD-PG (Munos et al., 2023) is an algorithm that aims to find
a Nash equilibrium with respect to the preference probabilities, via online
policy gradient updates against a regularised policy. Both algorithms have
appealing properties, though on the face of it are unrelated: one is an
offline contrastive algorithm optimising against a fixed policy, the other is
an online algorithm aiming to find a Nash equilibrium.

In this work we bridge the gap between IPO and Nash-MD-PG, and use this
theoretical bridge to propose a novel class of preference optimisation
algorithms. Specifically, our principal contributions are: first, we identify
several key factors of variation between IPO and Nash-MD-PG, including their
use of offline/online data, contrastivity of their losses, and the nature of
their equilibria. We use this understanding to identify the strengths of these
algorithms, and combine these strengths into new preference optimisation
algorithms. This allows us to propose _Online IPO_ , an online variant of IPO.
In addition, we establish a theoretical connection between Online IPO and
_self-play_ in the regularised two-player preference game used in deriving
Nash-MD-PG. Second, motivated by Online IPO, we propose a preference
optimisation algorithm, aiming to capture the best aspects of both IPO and
Nash-MD-PG: _IPO-MD_ , a version of IPO that interpolates between offline and
online variants by using the lagged data distribution of Nash-MD-PG. Finally,
we provide an experimental suite contrasting these algorithms in several
applications, which provides detailed comparisons between the proposed methods
and several baselines, with notable take-aways for practitioners.

##  2 Background

We begin by introducing the central preference optimisation problem, and
relevant prior work.

###  2.1 Preference optimisation in bandits

We consider a (non-contextual) bandit problem (rather than a sequential
setting), with finite action space $\mathcal{Y}$. This simplifies the notation
considerably, and the ideas presented are straightforwardly extensible to the
contextual/sequential setting where the actions/generations $y$ are
conditioned on a state/prompt $x$; we explain this in further detail in the
descriptions of implementation details in the experiments.

Preferences. A _preference function_
$p:\mathcal{Y}\times\mathcal{Y}\rightarrow[0,1]$ specifies pairwise preference
probabilities between elements of $\mathcal{Y}$. Specifically, given
$y,y^{\prime}\in\mathcal{Y}$, $p(y\succ y^{\prime})$ is the probability that
$y$ is preferred to $y^{\prime}$. We will assume that preference functions
satisfy the symmetry condition $p(y^{\prime}\succ y)=1-p(y\succ y^{\prime})$.

At a high level, the goal of learning in this context is to find a policy in
$\Delta(\mathcal{Y})$ (the set of distributions over $\mathcal{Y}$) that tends
to select actions $y\in\mathcal{Y}$ that are preferred over others. This is of
course not a precise mathematical objective as stated, and there are several
distinct ways in which this can be formalised, which we explore in greater
detail below.

Data model. Actions are sampled from policies
$\mu,\mu^{\prime}\in\Delta(\mathcal{Y})$ as
$Y\sim\mu,Y^{\prime}\sim\mu^{\prime}$ (we write $Y,Y^{\prime}\sim\mu$ when
$\mu=\mu^{\prime}$). Given a preference function $p$, the _preference
distribution_ $\lambda_{p}$ is defined for $y,y^{\prime}\in\mathcal{Y}$ as the
distribution corresponding to the following sampling procedure:

$\lambda_{p}(y,y^{\prime})\mbox{ yields }\begin{cases}(y,y^{\prime})\mbox{
with probability }p(y\succ y^{\prime})\\\ (y^{\prime},y)\mbox{ with
probability }1-p(y\succ y^{\prime})\,.\end{cases}$

We will frequently make use of a collection of samples drawn from the data-
generating policies and the preference distribution, which we denote by
$(y^{+}_{i},y^{-}_{i})_{i=1}^{N}$. In this case we will say the data is
sampled from $(\mu,\mu^{\prime},\lambda_{p})$ (or $(\mu,\lambda_{p})$ when
$\mu=\mu^{\prime}$). In _offline_ settings, the data is typically generated
from a fixed policy $\mu$, whereas in _online_ settings, new data can be
generated from a currently estimated policy $\pi$.

Another important aspect of the data available to the learner is an initial
policy $\pi^{\mathrm{ref}}\in\Delta(\mathcal{Y})$. Typically, this policy
provides acceptable behaviour in some (though not all) aspects of interest in
the problem, and may be used to define a trust region for the policy
optimisation problem. Our central contributions in this paper draw together,
unify, and contrast several different approaches to preference optimisation;
we now recall these below.

###  2.2 RLHF with a Bradley-Terry reward model

The approach to preference optimisation proposed by Christiano et al. (2017)
is split into two steps. First, a reward model
$r:\mathcal{Y}\rightarrow\mathbb{R}$ is fitted via the Bradley-Terry model
(Bradley & Terry, 1952). In more detail, the preference probability $p(y\succ
y^{\prime})$ is approximated by the logistic function

$\displaystyle\sigma(r(y)-r(y^{\prime}))=\frac{e^{r(y)-r(y^{\prime})}}{1+e^{r(y)-r(y^{\prime})}}\,,$

and a reward function $\hat{r}$ is learnt essentially by performing maximum
likelihood in this model, given a collection of observed preferences
$(y^{+}_{i},y^{-}_{i})_{i=1}^{N}$ sampled from $(\mu,\lambda_{p})$,
(approximately) maximising the objective

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\log\left(\sigma(r(y^{+}_{i})-r(y^{-}_{i}))\right)\,.$
(1)

Policy optimisation then proceeds by aiming to maximise the expected reward
$\hat{r}$ under the $\pi$, subject to a KL constraint against the initial
policy:

$\displaystyle\operatorname*{arg\,max}_{\pi\in\Delta(\mathcal{Y})}\
\bigg{[}\mathop{\mathbb{E}}_{Y\sim\pi}[\hat{r}(Y)]-\tau\text{KL}(\pi\;||\;\pi^{\mathrm{ref}})\bigg{]}\,,$
(2)

with $\tau>0$ a temperature parameter controlling the degree of regularisation
towards $\pi^{\mathrm{ref}}$. Note that the solution $\pi^{*}$ to Equation (2)
is available in closed form, as

$\displaystyle\pi^{*}(y)\propto\pi^{\mathrm{ref}}(y)\exp(\tau^{-1}\hat{r}(y))\,.$
(3)

###  2.3 Direct preference optimisation (DPO)

Rafailov et al. (2023) propose _direct policy optimisation_ (DPO) as an
alternative to RLHF as described above, noting that with the closed form in
Equation (3), the learning of a reward function can be completely bypassed,
instead reparametrising the optimal reward in terms of the optimal policy,
substituting into Equation (1), and aiming to maximise the resulting objective
with respect to $\pi$:

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\log\left(\sigma\left(\tau\;\log\left(\frac{\pi(y^{+}_{i})\pi^{\mathrm{ref}}(y^{-}_{i})}{\pi(y^{-}_{i})\pi^{\mathrm{ref}}(y^{+}_{i})}\right)\right)\right)\,.$
(4)

The derivation of DPO implies that, mathematically, it yields the same optimal
policy as the RLHF approach in Section 2.2, which is a regularised optimiser
for the reward model $\hat{r}$.

###  2.4 Sequence Likelihood Calibration (SLiC)

Zhao et al. (2023) propose _sequence likelihood calibration_ (SLiC) as an
alternative to RLHF. Soon after, Liu et al. (2023) refine the SLiC loss by
normalising the policy probabilities with the reference policy probabilities
in order to get a regularised offline loss. In the remaining, we will refer to
the following loss, with a dataset $(y^{+}_{i},y^{-}_{i})_{i=1}^{N}$ sampled
from $(\mu,\lambda_{p})$:

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\max\left(0,1-\tau\log\bigg{(}\frac{\pi(y^{+}_{i})\pi^{\mathrm{ref}}(y^{-}_{i})}{\pi(y^{-}_{i})\pi^{\mathrm{ref}}(y^{+}_{i})}\bigg{)}\right)\,$
(5)

as the SLiC loss. This loss can be interpreted as an hinge-loss variation of
DPO (Liu et al., 2023).

###  2.5 Identity policy optimisation (IPO)

Azar et al. (2023) note that in general, optimisation of the reward model
described in Section 2.2, which is implicitly optimised by DPO, may not always
yield an intuitively good policy for the preference probabilities $p$. They
also note that in practice, removal of the learnt reward function from the
pipeline removes an important source of regularisation in the learning
problem, and as such DPO may learn policies that are under-regularised, and
converge to deterministic actions.

In order to circumvent these two issues, Azar et al. (2023) propose _identity
policy optimisation_ (IPO). The derivation begins from the objective of aiming
to directly optimise preference probabilities (rather than a proxy reward)
against a fixed policy $\mu$:

$\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}Y\sim\pi\\\
Y^{\prime}\sim\mu\end{subarray}}[p(Y\succ
Y^{\prime})]-\tau\text{KL}(\pi\;||\;\pi^{\mathrm{ref}})\,.$ (6)

Similar to Equation (3), the optimal policy for this objective is expressible
directly as

$\displaystyle\pi^{*}(y)\propto\pi^{\mathrm{ref}}(y)\exp(\tau^{-1}\mathbb{E}_{Y^{\prime}\sim\mu}[p(y\succ
Y^{\prime})])\,,$ (7)

which Azar et al. (2023) use to derive the following equivalent offline IPO
loss, with a dataset $(y^{+}_{i},y^{-}_{i})_{i=1}^{N}$ sampled from
$(\mu,\lambda_{p})$:

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}\left(\log\bigg{(}\frac{\pi(y^{+}_{i})\pi^{\mathrm{ref}}(y^{-}_{i})}{\pi(y^{-}_{i})\pi^{\mathrm{ref}}(y^{+}_{i})}\bigg{)}-\tau^{-1}/2\right)^{2}\,.$
(8)

The quadratic aspect of the loss discourages log-probability ratios between
pairs of actions under $\pi$ from deviating too far from those under
$\pi^{\mathrm{ref}}$, which ensures regularisation of $\pi$ against
$\pi^{\mathrm{ref}}$. By contrast, the minimiser of the DPO empirical loss
does not have this property.

###  2.6 Nash-MD-PG

Rather than optimising preference probabilities against an offline dataset
generated from some data-generating policy $\mu$, Munos et al. (2023) propose
instead interpreting the objective in Equation (6) as one player’s objective
in a two-player, constant-sum game. Specifically, two players select policies
$\pi_{1},\pi_{2}\in\Delta(\mathcal{Y})$, with player $i$ receiving payoff

$\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}Y\sim\pi_{i}\\\
Y^{\prime}\sim\pi_{-i}\end{subarray}}[p(Y\succ Y^{\prime})]$ (9)
$\displaystyle\qquad-\tau\text{KL}(\pi_{i}\;||\;\pi^{\mathrm{ref}})+\tau\text{KL}(\pi_{-i}\;||\;\pi^{\mathrm{ref}})\,,$

where $\pi_{-i}$ denotes the policy of the other player. Note that holding
$\pi_{-i}$ fixed at $\mu$ then yields an equivalent objective to Equation (6)
for $\pi_{i}$. The proposal of Munos et al. (2023) is then to find a Nash
equilibrium for this game, motivated by the idea that the policies in the
resulting Nash equilibrium may be more robust, and are not overly specific to
the data-generating distribution $\mu$. On the flip-side there may be some
benefit to regularising the sampling distribution toward the data
distribution, and Nash-MD-PG has a parameter $\beta$ that allows for this
tradeoff, with self-play in one extreme $(\beta=0)$ and sampling from $\mu$ in
the other ($\beta=1$). We denote this algorithm by Nash-MD-PG($\beta$).

The Nash-MD-PG($\beta$) algorithm, motivated by mirror-descent approaches to
saddle-point computation, aims to do so by updating the policy $\pi$ in the
direction of the following policy gradient:

$\displaystyle\nabla\log\pi(y)\left(p(y\succ
y^{\prime})-\tfrac{1}{2}-\tau\log\left(\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\right)\right)\,,$

where importantly $y\sim\pi$, and $y^{\prime}$ is sampled from a _geometric
mixture policy_ $\pi^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$, for some choice
of $\beta\in[0,1]$.

##  3 Comparative discussion of preference optimisation algorithms

The algorithms DPO (Section 2.3), SLiC (Section 2.4), IPO (Section 2.5), and
Nash-MD-PG (Section 2.6) are distinct along a number of axes.

Contrastivity. IPO is a _contrastive_ algorithm, in that it labels a pair
$(Y,Y^{\prime})$ according to the preference function (via $\lambda_{p}$) into
a positive (preferred) and a negative (not preferred) example $(Y^{+},Y^{-})$,
and then updates the policy via gradients flowing through _both samples_ ,
$\pi(Y^{+})$ and $\pi(Y^{-})$. By contrast, Nash-MD-PG is not contrastive;
only the sampled action’s policy probability is directly updated based on the
preference. Contrastive algorithms in general have the potential to be more
data-efficient, making direct use of both samples in each policy update (see
Appendix D for a condition under which a contrastive gradient estimate has
lower variance than its non-contrastive counterpart).

Offline/online data. IPO is an _offline_ algorithm, working with a fixed
dataset, while Nash-MD-PG is an _online_ algorithm that makes use of sampled
actions from both the current estimated policy $\pi$, and a geometric mixture
$\pi^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$. In many settings it may be
desirable to work with static, offline data sets, although if it is feasible
to gather data online, this can be beneficial from the point of view of
effective regularisation. In the offline setting, the data limits our ability
to evaluate the quality of learned policies in terms of preferences, which can
lead to learning bad policies that choose actions outside the data
distribution, but have low empirical loss. Similar to offline RL (Fujimoto et
al., 2019), this can be mitigated through regularisation to a reference policy
$\pi^{\mathrm{ref}}$, possibly related to the sampling distribution $\mu$
(Jaques et al., 2019; Wu et al., 2019). An alternative to regularisation is to
use online data, and to train $\pi$ on data that is close to what $\pi$
generates. Online data may not be available in all settings, but, when it is,
it can be an effective way to improve performance of learned policies.

Equilibria. The IPO loss is a supervised objective. In particular, its data
distribution is fixed, and the optimiser is given by the closed-form policy in
Equation (7), which in particular can be interpreted as a policy that is
preferred over the data-generating distribution $\mu$, regularised towards
$\pi^{\mathrm{ref}}$. By contrast, Nash-MD-PG is a game-theoretic algorithm,
with a loss function whose data distribution and objective change as the
estimated policy change themselves. The stationary points for Nash-MD-PG are
not defined in a closed-form manner against reference and data policies
$\pi^{\mathrm{ref}}$ and $\mu$, but in a self-referential manner.

Regularised Sampling. As discussed in Section 2.6, Nash-MD-PG allows for
sampling from a mixture distribution between $\pi$ and the data-generating
distribution $\mu$, and this can also lead to improved performance versus
sampling from either policy.

It is clear that there are several combinations of the various properties of
previous methods for which no algorithm yet exists, including combinations
that could have advantages over previous work, for example an online
contrastive method. Table 1 gives an overview of existing methods in terms of
what we consider are strengths of these methods, and how the methods
introduced in this paper fit in this context, combining these strengths.

Method | Contrastive | Online | Regularised Sampling
---|---|---|---
RLHF (Christiano et al., 2017) |  | ✓ |
DPO (Rafailov et al., 2023) | ✓ |  |
SLiC (Zhao et al., 2023) | ✓ |  |
IPO (Azar et al., 2023) | ✓ |  |
Self-play (Nash-MD-PG($\beta=0$)) (Munos et al., 2023) |  | ✓ |
Nash-MD-PG (Munos et al., 2023) |  | ✓ | ✓
Online-IPO (Section 4) | ✓ | ✓ |
IPO-MD (Section 5) | ✓ | ✓ | ✓
Table 1: Method comparison in terms of their properties.

##  4 Online IPO

We first aim to bridge the online/offline divide between IPO and Nash-MD-PG,
by proposing a new variant of IPO, _Online IPO_ , which makes use of an
online, shifting data distribution.

###  4.1 Algorithm

To derive an update, we first start with the _population_ loss for IPO, which
is obtained by taking the minibatch loss in Equation (8), and taking an
expectation over the dataset (under i.i.d. sampling from $(\mu,\lambda_{p})$).
This yields the (offline) IPO population loss

$\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}Y,Y^{\prime}\sim{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mu}\\\
Y^{+},Y^{-}\sim\lambda_{p}(Y,Y^{\prime})\end{subarray}}\bigg{[}\left(\log\bigg{(}\frac{\pi(Y^{+})\pi^{\mathrm{ref}}(Y^{-})}{\pi(Y^{-})\pi^{\mathrm{ref}}(Y^{+})}\bigg{)}-\tau^{-1}/2\right)^{2}\bigg{]}\,.$
(10)

The _Online IPO_ population loss is given by replacing the static data
distribution, highlighted in red above, with the data distribution generated
by the current policy, as displayed below

$\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}Y,Y^{\prime}\sim{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\texttt{SG}[\pi]}\\\
Y^{+},Y^{-}\sim\lambda_{p}(Y,Y^{\prime})\end{subarray}}\bigg{[}\left(\log\bigg{(}\frac{\pi(Y^{+})\pi^{\mathrm{ref}}(Y^{-})}{\pi(Y^{-})\pi^{\mathrm{ref}}(Y^{+})}\bigg{)}-\tau^{-1}/2\right)^{2}\bigg{]}\,.$
(11)

Here, $\texttt{SG}[\pi]$ denotes a stop-gradient around $\pi$ in the data
distribution, meaning that although we generate data from $\pi$ to construct
the loss, we do not differentiate through the data-generation process itself.

The population form of the Online IPO loss will be useful in the analysis that
follows. We conclude our description of the approach by noting that the
sample-based Online IPO loss coincides with the Offline IPO111For clarity, we
will refer to the original formulation of IPO by Azar et al. (2023) as
_Offline IPO_ in what follows. loss in Equation (8), with the exception that
the samples $(y^{+}_{i},y^{-}_{i})_{i=1}^{N}$ are drawn from the current
policy $\pi$.

###  4.2 Analysis

Before studying the performance of the newly derived Online IPO loss
empirically, we pause to consider it from a theoretical perspective. In
particular, we aim to understand for which policies this loss is stationary.

By the analysis for (Offline) IPO (Azar et al., 2023) summarised in Section
2.5, the gradient for the Online IPO loss in Equation (11) is zero iff $\pi$
satisfies

$\displaystyle\pi(y)\propto\pi^{\mathrm{ref}}(y)\exp(\tau^{-1}p(y\succ\pi))\,,$
(12)

where we use the shorthand
$p(y\succ\pi)=\mathbb{E}_{Y^{\prime}\sim\pi}[p(y\succ Y^{\prime})]$. This is a
fixed-point condition; note that $\pi$ appears on both sides. This in fact
says that $\pi$ is a best-response against itself in the regularised game
described in the background for Nash-MD-PG in Section 2.6. Hence, if $\pi$
satisfies Equation (12), it must be the Nash equilibrium for this game; we
have shown the following.

######  Proposition 4.1.

The minimiser of the online IPO objective is the Nash equilibrium of the
regularised game described in Equation (9).

This is perhaps a surprising conclusion. Offline IPO is not motivated by game-
theoretic considerations, yet by moving to an online variant, we have obtained
a loss whose stationary point is precisely the Nash equilibrium of the
preference game optimised by Nash-MD-PG. In fact, we can go further, and
deduce a direct equivalence of expected updates between Online IPO, and _self-
play_ in this game. The proof for the following result is given in Appendix C.

######  Proposition 4.2.

The expected gradient of the Online IPO loss in Equation (11) is identical to
the self-play update direction in the game with payoff as in Equation 9.

Here, _self-play_ refers to the algorithm in which a policy is updated using
gradient ascent on its expected payoff in the game described in Equation (9),
against another player using the same policy:

$\displaystyle\nabla_{\pi}\Bigg{(}\mathop{\mathbb{E}}_{Y\sim\pi,Y^{\prime}\sim\texttt{SG}[\pi]}\bigg{[}p(Y\succ
Y^{\prime})-\tau\text{KL}(\pi\;||\;\pi^{\mathrm{ref}})\bigg{]}\Bigg{)}\,.$

Note that in expectation, this corresponds to Nash-MD-PG with $\beta=0$;
however, an important difference is that in Online IPO, updates are
_contrastive_ , which may result in variance reduction of the gradient
estimate.

We have therefore established a close connection between Online IPO and Nash
equilibria for the regularised game.

###  4.3 Online DPO

As the DPO and SLiC losses are similar to IPO, a natural question is whether
online variants of DPO and SLiC are also related to the regularised game given
Equation 9. We explore this question for online DPO in Appendix F. Lemma F.5
gives conditions for the stationary point of the regularised game and online
IPO to be a stationary point of online DPO. However, the conditions seem
difficult to satisfy: For example, Theorem F.6 says that there is no 2-action
problem for which the condition is satisfied, except when preferences are
uniform ($p(1\succ 2)=\frac{1}{2}$). In this sense, apart from the trivial
uniform-preference case, online DPO and online IPO are different objectives
when $|\mathcal{Y}|=2$.

As for stationary points of online DPO, we show that, under the Bradley-Terry
model assumption, the RLHF solution (Equation 3) is a stationary point of
online DPO (Theorem F.7), coinciding with that of offline DPO.

##  5 IPO-MD

Having established a connection between Online IPO and self-play, it is
natural to consider whether we can improve on self-play by using regularised
policies to generate the data, similar to how Nash-MD optimises preferences
against a regularised adversary.

###  5.1 Algorithm

We consider modulating the data distribution used in the IPO loss using the
same geometric mixture ($\pi^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$) between
online and reference policies as in Nash-MD, we arrive at a new family of
algorithms that we call IPO-MD that directly corresponds to the family of
policies for Nash-MD-PG. This leads to the population loss

$\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}Y,Y^{\prime}\sim{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\texttt{SG}[\pi^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}]}\\\
Y^{+},Y^{-}\sim\lambda_{p}(Y,Y^{\prime})\end{subarray}}\bigg{[}\left(\log\bigg{(}\frac{\pi(Y^{+})\pi^{\mathrm{ref}}(Y^{-})}{\pi(Y^{-})\pi^{\mathrm{ref}}(Y^{+})}\bigg{)}-\tau^{-1}/2\right)^{2}\bigg{]}\,.$

where in the data distribution, we write
$\pi^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$ as shorthand for the policy which
is given by normalising the (unnormalised) geometric mixture
$\pi^{1-\beta}(y)(\pi^{\mathrm{ref}})^{\beta}(y)$. For $\beta\in[0,1]$, we
obtain an algorithm that interpolates between these two in a certain sense,
analogous to Nash-MD-PG (Munos et al., 2023); we call this algorithm IPO-
MD($\beta$). When $\beta=0$, we obtain Online IPO, i.e. self-play, and when
$\beta=1$, we get a variation of IPO that tries to improve against a fixed
policy $\pi^{\mathrm{ref}}$. An interesting observation is that if we use a
slightly different mixture policy that mixes with $\mu$,
$\pi^{1-\beta}(\mu)^{\beta}$, then this actually interpolates between Online
IPO and Offline IPO, where for $\beta=1$ we get back the Offline IPO
objective. In practice it is often difficult to get direct access to $\mu$ so
we would not be able to form this geometric mixture. Some example dynamics in
tabular settings are plotted in Appendix E.

###  5.2 Analysis

We now compare Nash-MD-PG with the new algorithm class IPO-MD described above.
First, these two classes of algorithms are practically different. Nash-MD-PG
is on-policy, in that the only gradient contributions appearing in its update
are those corresponding to actions sampled under the current policy. By
contrast, IPO-MD is an off-policy algorithm, since it updates its current
policy based on actions sampled under
$\pi^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$.

By the analysis of Offline IPO (Azar et al., 2023), we have that any fixed
point $\pi^{*}_{\beta}$ of IPO-MD($\beta$) must satisfy

$\displaystyle\pi^{*}_{\beta}(y)\\!\propto\\!\pi^{\mathrm{ref}}(y)\exp\big{(}\tau^{-1}p(y\succ(\pi^{*}_{\beta})^{1-\beta}(\pi^{\mathrm{ref}})^{\beta})\big{)}\,.$
(13)

In other words, $\pi^{*}_{\beta}$ is a best-response against
$(\pi^{*}_{\beta})^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$ in the regularised
game described in Equation (9). But from the description of the Nash-MD-
PG($\beta$) algorithm by Munos et al. (2023), we have that a policy $\pi^{*}$
is stationary under this algorithm’s update iff

$\displaystyle\pi^{*}(y)\propto\pi^{\mathrm{ref}}(y)\exp\big{(}\tau^{-1}p(y\succ(\pi^{*})^{1-\beta}(\pi^{\mathrm{ref}})^{\beta})\big{)}\,.$

Hence, the fixed point of IPO-MD$(\beta)$ coincides with the fixed point of
Nash-MD-PG($\beta$).

Having established the equivalence of the stationary points of IPO-MD($\beta$)
and Nash-MD-PG($\beta$), we now study their gradients more generally in the
following result; see Appendix C for the proof.

######  Proposition 5.1.

The gradients of the algorithms Nash-MD-PG($\beta$) and IPO-MD($\beta$) are,
respectively,

$\displaystyle g_{\mbox{{\tiny Nash-MD-PG}}(\beta)}$ $\displaystyle=$
$\displaystyle-{\mathbb{E}}_{y\sim\pi}\left[g(y)\right]$ $\displaystyle
g_{\mbox{{\tiny IPO-MD}}(\beta)}$ $\displaystyle=$
$\displaystyle-\frac{2}{\tau}{\mathbb{E}}_{y\sim(\pi)^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}}\left[g(y)\right]$

where $g(y)$ is given by

$\displaystyle\nabla\log\pi(y)\left(p(y\succ(\pi)^{1-\beta}(\pi^{\mathrm{ref}})^{\beta})-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\right)\,.$

We recover the result mentioned earlier that when $\beta=0$, IPO-MD($\beta=0$)
(i.e., Online IPO) has a gradient aligned with that of Nash-MD-PG($\beta=0$)
(i.e., Self-Play). Now as soon as $\beta>0$, the gradients of these two
algorithms are different. Interestingly however, as noted above, their fixed
points remain the same.

We can also relate the fixed points themselves back to the original
regularised game given in Equation (9), as described below.

######  Proposition 5.2.

Let $\pi^{*}_{\beta}$ be the fixed-point of IPO-MD($\beta$), satisfying
Equation (13). The policy
$\pi^{\prime}_{\beta}=(\pi^{*}_{\beta})^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$
is the Nash equilibrium for the version of the game in Equation (9) with
regularisation parameter $\tau$ modified to $\tau(1-\beta)^{-1}$.

###### Proof.

Using the property in Equation (13), we have

$\displaystyle\pi^{\prime}_{\beta}(y)$ $\displaystyle\propto$
$\displaystyle(\pi^{\mathrm{ref}}(y))^{1-\beta}\exp(\tau^{-1}(1-\beta)p(y\succ\pi^{\prime}_{\beta}))(\pi^{\mathrm{ref}}(y))^{\beta}$
$\displaystyle\propto$
$\displaystyle\pi^{\mathrm{ref}}(y)\exp(\tau^{-1}(1-\beta)p(y\succ\pi^{\prime}_{\beta}))\,,$

which is the Nash equilibrium condition for the game described, as required. ∎

##  6 Experiments

We present our results on fine-tuning large language models where we compare
our algorithms, online-IPO and IPO-MD, against recent baselines. In this
section we only present the results for the online versions of IPO, DPO and
SLIC to make the comparison against IPO-MD and Nash-MD-PG fair, and drop the
corresponding ”online-” prefix for simplicity. We refer the reader to the
appendix for results concerning the offline versions of those algorithms. Note
that to aid interpretability and reproducibility, we now consider the
contextual bandit case where the actions $y$, also referred as generations,
are conditioned on a prompt $x$.

Setup and algorithms. We perform RLHF-style experiments where we initialise
from a supervised-fine-tuned checkpoint, and then further fine-tune using one
of the following algorithms: RL (regularised policy gradient), IPO, DPO ,
SLiC, Nash-MD and IPO-MD. These algorithms use either a learned reward model
$r_{\phi}$ (RLHF) or a learned preference model $p_{\phi}$ (IPO, DPO, SLiC,
Nash-MD, and IPO-MD). For our RL baseline, similarly to Munos et al. (2023),
we use a regularised policy gradient update:

$\displaystyle\mathop{\mathbb{E}}_{\begin{subarray}{c}x\sim\rho\\\
y\sim\pi_{\theta}(\cdot|x)\end{subarray}}\\!\\!\\!\\!\\!\\!\big{[}\nabla_{\theta}\log\pi_{\theta}(y|x)\\!\left(r_{\phi}(y|x)\\!-\\!\tau\text{KL}(\pi_{\theta}(\cdot|x),\pi^{\mathrm{ref}}\\!(\cdot|x))\right)\\!\big{]},$

where $r_{\phi}(y|x)$ is the reward model’s value for generation $y$ and
context $x$.

Implementation details. The contrastive offline algorithms such as IPO, DPO
and SliC directly optimise the policy $\pi_{\theta}$ by minimising their
respective losses over a pairwise dataset
$(x_{i},y_{i}^{+},y_{i}^{-})_{i=1}^{N}$ . In practice, we sample batches
$(x_{i},y_{i}^{+},y_{i}^{-})_{i=1}^{B}$ of size $B\ll N$ and we minimise the
following loss:

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}\mathcal{L}_{\texttt{ALGO}}(\theta,x_{i},y_{i},y^{\prime}_{i}),$

where:

$\displaystyle\mathcal{L}_{\texttt{IPO}}(\theta,x,y,y^{\prime})\\!$
$\displaystyle=\\!\left(\log\left(\frac{\pi_{\theta}(y|x)\pi^{\mathrm{ref}}(y^{\prime}|x)}{\pi_{\theta}(y^{\prime}|x)\pi^{\mathrm{ref}}(y|x)}\right)-\tau^{-1}/2\right)^{2},$
$\displaystyle\mathcal{L}_{\texttt{DPO}}(\theta,x,y,y^{\prime})\\!$
$\displaystyle=\\!\sigma\left(\tau\log\left(\frac{\pi_{\theta}(y|x)\pi^{\mathrm{ref}}(y^{\prime}|x)}{\pi_{\theta}(y^{\prime}|x)\pi^{\mathrm{ref}}(y|x)}\right)\right),$
$\displaystyle\mathcal{L}_{\texttt{SLiC}}(\theta,x,y,y^{\prime})\\!$
$\displaystyle=\\!\max\left(0,1\\!-\\!\tau\log\left(\frac{\pi_{\theta}(y|x)\pi^{\mathrm{ref}}(y^{\prime}|x)}{\pi_{\theta}(y^{\prime}|x)\pi^{\mathrm{ref}}(y|x)}\right)\right)\\!.$

One important detail concerning IPO is that we use a simplified loss in our
code. One can remark by expanding the square and removing terms that do not
depend on $\theta$ that the IPO loss is equivalent to:

$\displaystyle-\log\left(\frac{\pi_{\theta}(y|x)}{\pi_{\theta}(y^{\prime}|x)}\right)+\tau\left(\log\left(\frac{\pi_{\theta}(y|x)\pi^{\mathrm{ref}}(y^{\prime}|x)}{\pi_{\theta}(y^{\prime}|x)\pi^{\mathrm{ref}}(y|x)}\right)\\!\right)^{2}.$

The contrastive online algorithms such as IPO, DPO and SLiC use a trained
preference model $p_{\phi}$. To train $p_{\phi}$, we use a pairwise dataset
$(x_{i},y_{i}^{+},y_{i}^{-})_{i=1}^{N}$ and follow the same protocol as Munos
et al. (2023). Then, to train the policy $\pi_{\theta}$, for each context
$x_{i}$ of a batch, we sample two completely new generations
$(y_{i},y_{i}^{\prime})\sim\pi_{\theta}$ according to $\pi_{\theta}$ and
compute the preference $p_{i}=p_{\phi}(y_{i}\succ y_{i}^{\prime}|x_{i})$ via
the preference model. Then, for each algorithm, we minimise the respective
following loss:

$\displaystyle\frac{1}{B}\sum_{i=1}^{B}\left(p_{i}\mathcal{L}_{\texttt{ALGO}}(\theta,x_{i},y_{i},y^{\prime}_{i})+(1-p_{i})\mathcal{L}_{\texttt{ALGO}}(\theta,x_{i},y^{\prime}_{i},y_{i})\right).$

Finally, regarding IPO-MD, the only difference with (online)-IPO is how the
generations are sampled. In theory, we should sample from the mixture
$\pi_{\theta}^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$ which is not feasible
(see (Munos et al., 2023)). In practice, we sample from the one-step-at-a-time
mixture $\hat{\pi}_{\beta}$, which consists at step $n$ to compute the mixture
of logits between the online and reference logits:

$\displaystyle\log(\hat{\pi}_{\beta}(.|y_{0:n-1},x))=(1-\beta)\log(\pi_{\theta}(.|y_{0:n-1},x))$
$\displaystyle+\beta\log(\pi^{\mathrm{ref}}(.|y_{0:n-1},x))+C(y_{0:n-1},x),$

where $C(y_{0:n-1},x)$ is a path-dependent constant and sample according to
the softmax of this mixture of logits. This sampling process is not equivalent
to sampling from $\pi_{\theta}^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$ as shown
in (Munos et al., 2023).

$p(y\succ y^{\prime})$ | IPO | IPO-MD | DPO | Nash-MD-PG | SLiC | RL
---|---|---|---|---|---|---
IPO | 0.500 |  0.515 (0.024) |  0.608 (0.038) |  0.621 (0.030) |  0.608 (0.025) |  0.791 (0.012)
IPO-MD |  0.485 (0.024) | 0.500 |  0.600 (0.028) |  0.608 (0.026) |  0.594 (0.020) |  0.778 (0.004)
DPO | 0.392 (0.038) | 0.400 (0.028) | 0.500 | 0.520 (0.041) | 0.493 (0.040) | 0.727 (0.020)
Nash-MD-PG | 0.379 (0.030) | 0.392 (0.026) | 0.480 (0.041) | 0.500 | 0.479 (0.029) | 0.729 (0.020)
SLiC | 0.392 (0.025) | 0.406 (0.020) | 0.507 (0.040) | 0.521 (0.029) | 0.500 | 0.728 (0.010)
RL | 0.209 (0.012) | 0.222 (0.004) | 0.273 (0.020) | 0.271 (0.020) | 0.272 (0.010) | 0.500
Table 2: Side-by-side evaluation for summarisation. Each entry is the average
preference of responses generated by the row method ($y$) over the responses
generated by the column method ($y^{\prime}$). We also evaluated each method
using 3 different seeds, computed a $3\times 3$ comparisons matrix across
seeds and report the standard deviation of this matrix’s entries.

Evaluation tasks and Models. In our experiments, we test all of the algorithms
on an article summarisation task. We use the dataset described by Stiennon et
al. (2020) that has been built from the TL;DR dataset (Völske et al., 2017).
This is a dataset with pairwise preferences between alternate summaries. We
train our preference and reward model on the train set $D_{\texttt{Train}}$,
which contains $92820$ examples. We evaluate reward and preference models on a
test set of high confidence data $D_{\texttt{Test}}$ and use the checkpoints
with the highest evaluation agreement score. To train the policies with online
algorithms, we use prompts of the train set of the XSum dataset (Shashi et
al., 2018).

We use T5X large language models (Roberts et al., 2022) to train our policies,
rewards and preference models. The T5X models we use are auto-regressive
transformers with an encoder-decoder architecture. All the details of the
models architecture and the different sizes are provided in the documentation
(Roberts et al., 2022). For the policy model, we use a _large_ (L) encoder-
decoder model ($770M$ parameters). For the preference and reward models, we
use an _XL_ encoder-decoder model ($3B$ parameters). To train reward and
preference models we use the same losses and protocol as Munos et al. (2023).
For summarisation, we initialise our policy with a T5X-L model and fine-tune
it with supervised learning using the OpenAI dataset described by Stiennon et
al. (2020). We call this supervised fine-tuned model the SFT. All our policies
for summarisation are initialised with this SFT checkpoint.

Our evaluation pipeline is based upon the use of PaLM2 (Anil et al., 2023) as
a judge for side-by-side comparisons. We sample responses for each of the
policies trained by each algorithm from a test set of prompts, and ask PaLM2
to pick which one is better. We use validation and test prompts from the XSum
dataset (Shashi et al., 2018) for evaluation for the summarisation task, which
is the same procedure used by Munos et al. (2023). The evaluation prompt we
use for the side-by-side comparison is:

You are an expert summary rater. Given a piece of text and two of its possible
summaries, output 1 or 2 to indicate which summary is better.

Text - <text>, Summary 1 - <summary1>, Summary 2 - <summary2>

Preferred Summary -

We use cloud Tensor Processing Units (TPUs; Jouppi et al., 2023) in their
version $5e$ for our hardware compute, either in configurations of $2\times 4$
devices for training offline experiments, or $4\times 4$ devices for online
experiments. This setup typically yields speed of around $0.25$ training steps
per second ($24$ hours per $20,000$ steps). We run our experiments with
default parameters $10^{-4}$ for the learning rate, and a default total of
$30,000$ training steps, using a batch size of $32$. The $\tau$ factor is held
constant throughout training, and we do not employ any warmup steps. We use
the AdaFactor (Shazeer & Stern, 2018) optimizer with decay set to $0.8$.

###  6.1 Main results

In this section, we present side-by-side evaluation scores between the
following online algorithms: RL, IPO, DPO, SLiC, IPO-MD and Nash-MD-PG. Table
2 presents the side by side scores for the summarisation task. The checkpoints
we evaluate are the best checkpoints except for RL that we use as a baseline
of comparison to find the best checkpoints. The RL checkpoint is fixed and was
chosen following the protocol of (Munos et al., 2023) after sweeping over $6$
values of $\tau$ ($\\{0.01,0.02,0.05,0.1,0.15,0.2\\}$) and comparing the
performance against the SFT checkpoint after $10k$ learner steps. To find the
best checkpoints for the other algorithms, we evaluate every checkpoint of
each algorithm against the RL checkpoint (over $2000$ prompts sampled from a
validation split) at different learning steps values (we checkpoint every
$2000$ learner steps for a total of $30k$ learner steps), regularisation
parameter $\tau$ (we sweep over $5$ values $\\{0.1,0.5,1.0,5.0,10.0\\}$) and
also $\beta$ for IPO-MD and Nash-MD-PG (we sweep over $2$ values $0.125$ and
$0.25$) and we take the best checkpoint. After finding the best checkpoint for
every algorithm (see App. B.3), we re-run each method for 3 different seeds
using the best hyperparamters. We then perform 9 side-by-side evaluation
(i.e., $3\times 3$ 1vs1 evaluations between each of the 3 seeds for each pair
of methods) using $2000$ prompts from a different validation split for each
comparison. We report mean and standard deviation across these 9 comparisons.

On the summarisation task, looking only at the mean the best algorithm is IPO
as it beats all the other algorithms on a side-by-side comparison. However,
once we take into consideration the standard deviation IPO and IPO-MD’s
performance becomes statistically indistinguishable, with both algorithms
consistently beating all the other algorithms. This shows that those
algorithms are indeed robust and are closer to a Nash optimum than the other
algorithms. Those results are limited to a summarisation task and more
experiments should be conducted to validate these results on a general
conversational agent. However, we do think that summarisation is a good test
bed to showcase the quality of human alignment algorithms because it is a
complex and high-in-demand task.

###  6.2 Ablations and Additional Results on Summarisation

#### Sweep on the regularisation parameter $\tau$:

Figure 1 sweeps the regularisation parameter for IPO and DPO. It’s interesting
to see for small values of regularisation IPO and DPO behave very similarly,
but for larger values the score for IPO decays much faster. This matches the
findings in Azar et al. (2023) that show that IPO has a much stronger
regularisation effect than DPO as $\tau$ gets larger.

Figure 1: Sweep on $\tau$ vs. RL on the _summarisation_ task.

#### Learning steps curve

Figure 2 shows the performance against RL for IPO Online as it trains. We can
see that as the regularisation gets stronger, more training time is required
to reach the best performance.

Figure 2: Training Progress of Online IPO (as a function of number of steps)
on the _summarisation_ task.

##  7 Conclusion

In this paper, we have identified several factors of variation, such as
contrastivity, online/offline and regularised-sampling, between two recently
proposed algorithms for preference optimisation, IPO and Nash-MD-PG. In doing
so, we have introduced two new algorithms, online-IPO and IPO-MD, that combine
different strengths of these existing algorithms, namely the loss function of
IPO with the online sampling and regularised data distribution of Nash-MD.
Theoretical analysis reveals a surprising equivalence at the level of expected
update between online-IPO and self-play in a regularised two-player preference
optimisation game. This important property is not possessed by online-DPO.
Finally, our empirical investigation on a summarisation task also reveals that
IPO-MD and online-IPO are promising approaches to preference optimisation at
scale as they are the most robust algorithms. At the moment, our work is
restricted to model of size $770M$ on a single task, future works will consist
to scale our approach to a full conversational agent using a larger model
($100+$ billions parameter).

Acknowledgements. We are grateful for the collaborative environment at Google
DeepMind. We would like to thank Shantanu Thakoor, Will Dabney, Doina Precup,
Mohammad Gheshlaghi Azar, Olivier Bachem, Sertan Girgin, Matt Hoffman, Nikola
Momchev, Bobak Shahriari, and Piotr Stanczyk.

## References

  * Amodei et al. (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D.  Concrete problems in AI safety.  _arXiv_ , 2016.
  * Anil et al. (2023) Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G. H., Ahn, J., Austin, J., Barham, P., Botha, J., Bradbury, J., Brahma, S., Brooks, K., Catasta, M., Cheng, Y., Cherry, C., Choquette-Choo, C. A., Chowdhery, A., Crepy, C., Dave, S., Dehghani, M., Dev, S., Devlin, J., Díaz, M., Du, N., Dyer, E., Feinberg, V., Feng, F., Fienber, V., Freitag, M., Garcia, X., Gehrmann, S., Gonzalez, L., Gur-Ari, G., Hand, S., Hashemi, H., Hou, L., Howland, J., Hu, A., Hui, J., Hurwitz, J., Isard, M., Ittycheriah, A., Jagielski, M., Jia, W., Kenealy, K., Krikun, M., Kudugunta, S., Lan, C., Lee, K., Lee, B., Li, E., Li, M., Li, W., Li, Y., Li, J., Lim, H., Lin, H., Liu, Z., Liu, F., Maggioni, M., Mahendru, A., Maynez, J., Misra, V., Moussalem, M., Nado, Z., Nham, J., Ni, E., Nystrom, A., Parrish, A., Pellat, M., Polacek, M., Polozov, A., Pope, R., Qiao, S., Reif, E., Richter, B., Riley, P., Ros, A. C., Roy, A., Saeta, B., Samuel, R., Shelby, R., Slone, A., Smilkov, D., So, D. R., Sohn, D., Tokumine, S., Valter, D., Vasudevan, V., Vodrahalli, K., Wang, X., Wang, P., Wang, Z., Wang, T., Wieting, J., Wu, Y., Xu, K., Xu, Y., Xue, L., Yin, P., Yu, J., Zhang, Q., Zheng, S., Zheng, C., Zhou, W., Zhou, D., Petrov, S., and Wu, Y.  PaLM 2 technical report, 2023.
  * Azar et al. (2023) Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R.  A general theoretical paradigm to understand learning from human preferences.  _arXiv_ , 2023.
  * Bai et al. (2022a) Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv_ , 2022a.
  * Bai et al. (2022b) Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukoiūtė, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T. J., Hume, T., Bowman, S., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T. B., and Kaplan, J.  Constitutional AI: Harmlessness from AI feedback.  _arXiv_ , 2022b.
  * Boyd & Vandenberghe (2004) Boyd, S. P. and Vandenberghe, L.  _Convex Optimization_.  Cambridge University Press, 2004.
  * Bradley & Terry (1952) Bradley, R. A. and Terry, M. E.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Casper et al. (2023) Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _arXiv_ , 2023.
  * Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Coste et al. (2023) Coste, T., Anwar, U., Kirk, R., and Krueger, D. S.  Reward model ensembles help mitigate overoptimization.  _arXiv_ , 2023.
  * Dong et al. (2023) Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T.  RAFT: Reward rAnked FineTuning for generative foundation model alignment.  _arXiv_ , 2023.
  * Eisenstein et al. (2023) Eisenstein, J., Nagpal, C., Agarwal, A., Beirami, A., D’Amour, A., Dvijotham, D., Fisch, A., Heller, K., Pfohl, S. R., Ramachandran, D., Shaw, P., and Berant, J.  Helping or herding? Rward model ensembles mitigate but do not eliminate reward hacking.  _arXiv_ , 2023.
  * Fujimoto et al. (2019) Fujimoto, S., Meger, D., and Precup, D.  Off-policy deep reinforcement learning without exploration.  In _Proceedings of the International Conference on Machine Learning_ , 2019.
  * Gao et al. (2022) Gao, L., Schulman, J., and Hilton, J.  Scaling laws for reward model overoptimization.  In _Proceedings of the International Conference on Machine Learning_ , 2022.
  * Glaese et al. (2022) Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L. A., and Irving, G.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv_ , 2022.
  * Griffith et al. (2013) Griffith, S., Subramanian, K., Scholz, J., Isbell, C. L., and Thomaz, A. L.  Policy shaping: Integrating human feedback with reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2013.
  * Hejna et al. (2023) Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D.  Contrastive prefence learning: Learning from human feedback without RL.  _arXiv_ , 2023.
  * Ivison et al. (2023) Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., and Hajishirzi, H.  Camels in a changing climate: Enhancing LM adaptation with Tulu 2\.  _arXiv_ , 2023.
  * Jaques et al. (2019) Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  _arXiv_ , 2019.
  * Jouppi et al. (2023) Jouppi, N. P., Kurian, G., Li, S., Ma, P. C., Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., Young, C., Zhou, X., Zhou, Z., and Patterson, D. A.  TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings.  In _Proceedings of the Annual International Symposium on Computer Architecture_ , 2023.
  * Kirk et al. (2023) Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R.  Understanding the effects of RLHF on LLM generalisation and diversity.  _arXiv_ , 2023.
  * Knox & Stone (2008) Knox, W. B. and Stone, P.  TAMER: Training an agent manually via evaluative reinforcement.  In _Proceedings of the IEEE International Conference on Development and Learning_ , 2008.
  * Lee et al. (2023) Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., and Rastogi, A.  RLAIF: Scaling reinforcement learning from human feedback with AI feedback.  _arXiv_ , 2023.
  * Liu et al. (2023) Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J.  Statistical rejection sampling improves preference optimization.  _arXiv_ , 2023.
  * Mnih et al. (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K.  Asynchronous methods for deep reinforcement learning.  In _Proceedings of the International Conference on Machine Learning_ , 2016.
  * Munos et al. (2023) Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, D., Tang, Y., Geist, M., Mesnard, T., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D. J., Precup, D., and Piot, B.  Nash learning from human feedback.  _arXiv_ , 2023.
  * Nakano et al. (2021) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J.  WebGPT: Browser-assisted question-answering with human feedback.  _arXiv_ , 2021.
  * OpenAI (2022) OpenAI.  Introducing ChatGPT, 2022.  URL https://openai.com/blog/chatgpt.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R.  Training language models to follow instructions with human feedback.  _arXiv_ , 2022.
  * Pan et al. (2022) Pan, A., Bhatia, K., and Steinhardt, J.  The effects of reward misspecification: Mapping and mitigating misaligned models.  _arXiv_ , 2022.
  * Pang et al. (2022) Pang, R. Y., Padmakumar, V., Sellam, T., Parikh, A. P., and He, H.  Reward gaming in conditional text generation.  In _Annual Meeting of the Association for Computational Linguistics_ , 2022.
  * Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C.  Direct preference optimization: Your language model is secretly a reward model.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Ramé et al. (2024) Ramé, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J.  WARM: On the benefits of weight averaged reward models.  _arXiv_ , 2024.
  * Roberts et al. (2022) Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J., Garcia, X., Ni, J., Chen, A., Kenealy, K., Clark, J. H., Lee, S., Garrette, D., Lee-Thorp, J., Raffel, C., Shazeer, N., Ritter, M., Bosma, M., Passos, A., Maitin-Shepard, J., Fiedel, N., Omernick, M., Saeta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Gesmundo, A.  Scaling up models and data with t5x and seqio.  _arXiv_ , 2022.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv_ , 2017.
  * Shashi et al. (2018) Shashi, N., Cohen, S. B., and Mirella, L.  Don’t Give Me the Details, Just the Summary! Topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2018.
  * Shazeer & Stern (2018) Shazeer, N. M. and Stern, M.  Adafactor: Adaptive learning rates with sublinear memory cost.  _arXiv_ , 2018.
  * Shin et al. (2023) Shin, D., Dragan, A. D., and Brown, D. S.  Benchmarks and algorithms for offline preference-based reward learning.  _arXiv_ , 2023.
  * Singhal et al. (2023) Singhal, P., Goyal, T., Xu, J., and Durrett, G.  A long way to go: Investigating length correlations in RLHF.  _arXiv_ , 2023.
  * Skalse et al. (2022) Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger, D.  Defining and characterizing reward gaming.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F.  Learning to summarize with human feedback.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Swamy et al. (2024) Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A.  A minimaximalist approach to reinforcement learning from human feedback.  _arXiv_ , 2024.
  * Tang et al. (2024) Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B.  Generalized preference optimization: A unified approach to offline alignment.  _arXiv preprint arXiv:2402.05749_ , 2024.
  * Touvron et al. (2023) Touvron, H., Martin, L., Stone, K. R., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D. M., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A. S., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I. M., Korenev, A. V., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv_ , 2023.
  * Tunstall et al. (2023) Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Rasul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., Sarrazin, N., Sanseviero, O., Rush, A. M., and Wolf, T.  Zephyr: Direct distillation of LM alignment.  _arXiv_ , 2023.
  * Völske et al. (2017) Völske, M., Potthast, M., Syed, S., and Stein, B.  TL;DR: Mining Reddit to learn automatic summarization.  In _Proceedings of the Workshop on New Frontiers in Summarization_. Association for Computational Linguistics, 2017.
  * Wang et al. (2023) Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y.  Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints.  _arXiv_ , 2023.
  * Warnell et al. (2018) Warnell, G., Waytowich, N., Lawhern, V., and Stone, P.  Deep TAMER: Interactive agent shaping in high-dimensional state spaces.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Wortsman et al. (2022) Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _Proceedings of the International Conference on Machine Learning_ , 2022.
  * Wu et al. (2019) Wu, Y., Tucker, G., and Nachum, O.  Behavior regularized offline reinforcement learning.  _arXiv_ , 2019.
  * Yuan et al. (2024) Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J.  Self-rewarding language models, 2024.
  * Yuan et al. (2023) Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F.  RRHF: Rank responses to align language models with human feedback without tears.  _arXiv_ , 2023.
  * Zhao et al. (2023) Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J.  SLiC-HF: Sequence likelihood calibration with human feedback.  _arXiv_ , 2023.
  * Zhuang & Hadfield-Menell (2020) Zhuang, S. and Hadfield-Menell, D.  Consequences of misaligned AI.  In _Advances in Neural Information Processing Systems_ , 2020.

## APPENDICES

##  Appendix A Related Work

RLHF. Reinforcement learning from human feedback, as introduced in Christiano
et al. (2017) (see also Bai et al. (2022a); Ouyang et al. (2022)) and often
based on proximal policy optimisation (Schulman et al., 2017), is a critical
element of making large language models helpful and aligned with preferences
of human operators. While in itself it typically does not result in improved
benchmark performance (Touvron et al., 2023), RLHF is nonetheless key to
satisfying human-mediated interactions such as dialogue (Nakano et al., 2021;
Ouyang et al., 2022). The complexity of the RLHF procedure (Casper et al.,
2023), which can also be accomplished by multiple reinforcement learning
algorithms such as actor-critic (Mnih et al., 2016; Glaese et al., 2022), has
led to searching for algorithmic alternatives (Dong et al., 2023; Yuan et al.,
2023; Zhao et al., 2023).

Recent developments in policy optimisation. In the special case of an
additional Bradley-Terry model (Bradley & Terry, 1952) assumption for the
human reward model, reinforcement learning has been found redundant; this
allows for casting the problem of RLHF as a supervised one (Rafailov et al.,
2023). Recent developments have focused on scaling the performance of such
direct policy optimisation (DPO) methods (Tunstall et al., 2023; Ivison et
al., 2023), as well as generalising its mathematical formulation (Azar et al.,
2023; Wang et al., 2023; Tang et al., 2024). One of the key issues with direct
policy optimisation - and RLHF in general - resides in their propensity to
game or _hack_ rewards (Amodei et al., 2016; Skalse et al., 2022; Pan et al.,
2022; Pang et al., 2022) and become overoptimised, or under-regularised (Gao
et al., 2022; Singhal et al., 2023; Kirk et al., 2023), which can be mitigated
e.g. by using ensembling techniques (Wortsman et al., 2022; Eisenstein et al.,
2023; Coste et al., 2023; Ramé et al., 2024). Rather than a reinforcement
versus supervised learning dichotomy, it is the distinction between _online_
and _offline_ (Jaques et al., 2019) methods that seems more relevant in
practice, as an online policy’s generations might start deviating
substantially from the original dataset, leading to distribution shifts
(Zhuang & Hadfield-Menell, 2020; Shin et al., 2023). Finally, alignment can
also result from the self-play form of a two-player game, and not just single-
policy optimisation. This perspective, taken in Munos et al. (2023); Swamy et
al. (2024) has the added benefit of encompassing both online and offline
settings, enabling smooth interpolation between them via a hyperparameter. In
a similar vein, DPO has been shown to be able and improve thanks to iterated
successive rounds (Yuan et al., 2024), expanding on known machine-critic
alignment methods such as reinforcement learning from AI feedback (Bai et al.,
2022b; Lee et al., 2023).

##  Appendix B Additional Experimental Results

###  B.1 Regularisation Sweep for Online and Offline

Figure 3 and Figure 4 show a sweep over the regularisation parameter for
Online and Offline DPO and IPO vs. RLHF and SFT respectively. It is
interesting to note that the online versions significantly outperform the
offline versions. This is understandable as this setting favours tremendously
online methods. Indeed, the starting point is an already fine-tuned policy on
summarisation data. Therefore, for online methods the first checkpoint is
already able to sample good summaries which make it very easy to obtain good
rewards/preferences and from there optimise either the reward/preference
model.

Figure 3: Sweep on Regularisation parameter vs. RL on the _summarisation_
task. Figure 4: Sweep on Regularisation parameter vs. SFT on the
_summarisation_ task.

###  B.2 Mixing ratio curve

Here in Figure 5 we show a sweep over the mixing ratio $\beta$ for IPO-MD and
how it affects its win-rate over the RL baseline in summarisation. We draw the
curves for a different learning rate and different learning steps than the
optimal checkpoint of IPO-MD to show that most of the time the mixing ratio
still help improve the performance.

Figure 5: IPO-MD winning rate. The evolving parameter is the mixture ratio
$\beta$ while we fix the learning rate to $3\cdot 10^{-5}$ and the
regularisation parameter to $\tau=1$. We plot this curves for $3$ different
learning steps $12k$, $16k$ and $20k$.

###  B.3 Best hyperparameters found

We report here the optimal $\tau$, learning rate lr and, where applicable,
mixture ratio $\beta$ used to obtain each algorithm’s best performance in
Table 2.

| $\tau$ | lr | $\beta$
---|---|---|---
RL | 0.05 | $10^{-4}$ | N/A
IPO | 1.0 | $10^{-4}$ | N/A
DPO | 5.0 | $10^{-4}$ | N/A
SLiC | 10.0 | $10^{-4}$ | N/A
IPO-MD | 1.0 | $10^{-4}$ | 0.125
Nash-MD-PG | 0.008 | $3\cdot 10^{-5}$ | 0.125

##  Appendix C Proofs

See 4.2

###### Proof.

We calculate expressions for the update directions directly, assuming that the
policy $\pi$ is parametrised via a vector $\phi$.

Online IPO update. The update direction for online IPO is given by the
negative of the derivative:

$\displaystyle\nabla_{\phi}\mathop{\mathbb{E}}_{\begin{subarray}{c}Y,Y^{\prime}\sim{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\texttt{SG}[\pi]}\\\
Y^{+},Y^{-}\sim\lambda_{p}(Y,Y^{\prime})\end{subarray}}\bigg{[}\left(\log\bigg{(}\frac{\pi(Y)\pi^{\mathrm{ref}}(Y^{\prime})}{\pi(Y^{\prime})\pi^{\mathrm{ref}}(Y)}\bigg{)}-\tau^{-1}/2\right)^{2}\bigg{]}$
$\displaystyle\propto$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\nabla_{\phi}\bigg{(}\bigg{(}\log\bigg{(}\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}\bigg{)}-\tau^{-1}/2\bigg{)}^{2}\bigg{)}$
$\displaystyle\propto$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\bigg{(}\log\bigg{(}\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}\bigg{)}-\tau^{-1}/2\bigg{)}\nabla_{\phi}\log(\pi(y)/\pi(y^{\prime}))\,.$

We can simplify the above by first considering the terms with a factor of
$\tau^{-1}$:

$\displaystyle-\tau^{-1}/2\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})(\nabla_{\phi}\log\pi(y)-\nabla_{\phi}\log\pi(y^{\prime}))$
$\displaystyle=$
$\displaystyle-\tau^{-1}/2\left(\sum_{y\in\mathcal{Y}}\pi(y)p(y\succ\pi)\nabla_{\phi}\log\pi(y)-\sum_{y^{\prime}\in\mathcal{Y}}p(\pi\succ
y^{\prime})\pi(y^{\prime})\nabla_{\phi}\log\pi(y^{\prime})\right)$
$\displaystyle=$
$\displaystyle-\tau^{-1}/2\left(\sum_{y\in\mathcal{Y}}\pi(y)p(y\succ\pi)\nabla_{\phi}\log\pi(y)-\sum_{y^{\prime}\in\mathcal{Y}}(1-p(y^{\prime}\succ\pi))\pi(y^{\prime})\nabla_{\phi}\log\pi(y^{\prime})\right)$
$\displaystyle=$
$\displaystyle-\tau^{-1}\sum_{y\in\mathcal{Y}}\pi(y)p(y\succ\pi)\nabla_{\phi}\log\pi(y)\,.$

Now considering the terms not involving $\tau$:

$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\Bigg{(}\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}-\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\Bigg{)}(\nabla_{\phi}\log\pi(y)-\nabla_{\phi}\log\pi(y^{\prime}))$
$\displaystyle=$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y)$
$\displaystyle\qquad+\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle\qquad-\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle\qquad-\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\nabla_{\phi}\log\pi(y)$
$\displaystyle=$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)p(y\succ\pi)\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y)+\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y^{\prime})p(\pi\succ
y^{\prime})\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle\qquad-\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle\qquad-\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})(1-p(y\succ
y^{\prime}))\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle=$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)p(y\succ\pi)\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y)+\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y^{\prime})(1-p(y\succ\pi))\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle\qquad-\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle=$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y^{\prime})\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle\qquad-\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})\log\bigg{(}\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})$
$\displaystyle=$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y^{\prime})\log\bigg{(}\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\bigg{)}\nabla_{\phi}\log\pi(y^{\prime})\,.$

Self-play update. Self-play leads to the update direction given by

$\displaystyle\nabla_{\phi}\Bigg{[}\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\texttt{SG}[\pi(y^{\prime})]p(y\succ
y^{\prime})-\tau\sum_{y\in\mathcal{Y}}\pi(y)\log(\pi(y)/\pi^{\mathrm{ref}}(y))\Bigg{]}$
$\displaystyle=$
$\displaystyle\sum_{y,y^{\prime}\in\mathcal{Y}}\pi(y)\pi(y^{\prime})p(y\succ
y^{\prime})\nabla_{\phi}\log\pi(y)-\tau\sum_{y\in\mathcal{Y}}\pi(y)\log(\pi(y)/\pi^{\mathrm{ref}}(y))\nabla_{\phi}\log\pi(y)$
$\displaystyle=$
$\displaystyle\sum_{y}\pi(y)p(y\succ\pi)\nabla_{\phi}\log\pi(y)-\tau\sum_{y\in\mathcal{Y}}\pi(y)\log(\pi(y)/\pi^{\mathrm{ref}}(y))\nabla_{\phi}\log\pi(y)\,.$

Therefore the expected online IPO update direction is exactly the same as that
of self-play. ∎

See 5.1

###### Proof.

The gradient of Nash-MD-PG($\beta$) is:

$\displaystyle g_{\mbox{{\tiny Nash-MD-PG}}(\beta)}$ $\displaystyle=$
$\displaystyle-{\mathbb{E}}_{y\sim\pi,y^{\prime}\sim\pi^{\prime}}\left[\nabla\log\pi(y)\left(p(y\succ
y^{\prime})-\frac{1}{2}\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\right)\right]$
$\displaystyle=$
$\displaystyle-{\mathbb{E}}_{y\sim\pi}\Big{[}\underbrace{\nabla\log\pi(y)\left(p(y\succ\pi^{\prime})-\frac{1}{2}-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\right)}_{g(y)}\Big{]}$

where we write $\pi^{\prime}=(\pi)^{1-\beta}(\pi^{\mathrm{ref}})^{\beta}$.

Now the gradient of IPO-MD($\beta$) is:

$\displaystyle g_{\mbox{{\tiny IPO-MD}}(\beta)}$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\nabla\left(\rho_{\pi}(y,y^{\prime})-\frac{1}{2\tau}\right)^{2}\right]$
$\displaystyle=$ $\displaystyle
2{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\left(\rho_{\pi}(y,y^{\prime})-\frac{1}{2\tau}\right)\nabla\rho_{\pi}(y,y^{\prime})\right],$

where
$\rho_{\pi}(y,y^{\prime})=\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}-\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}$,
thus
$\nabla\rho_{\pi}(y,y^{\prime})=\nabla\log\pi(y)-\nabla\log\pi(y^{\prime})$.

Using the anti-symmetry of the preference model, i.e., $p(y\succ
y^{\prime})=1-p(y^{\prime}\succ y)$, and combining terms, we have that

$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\nabla\rho_{\pi}(y,y^{\prime})\right]$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{y\sim\pi^{\prime}}\left[p(y\succ\pi^{\prime})\nabla\log\pi(y)\right]-{\mathbb{E}}_{y^{\prime}\sim\pi^{\prime}}\left[p(\pi^{\prime}\succ
y^{\prime})\nabla\log\pi(y^{\prime})\right]$ $\displaystyle=$ $\displaystyle
2{\mathbb{E}}_{y\sim\pi^{\prime}}\left[\nabla\log\pi(y)\left(p(y\succ\pi^{\prime})-\frac{1}{2}\right)\right].$

Similarly:

$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\rho_{\pi}(y,y^{\prime})\nabla\rho_{\pi}(y,y^{\prime})\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\left(\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}-\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\right)[\nabla\log\pi(y)-\nabla\log\pi(y^{\prime})]\right],$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\nabla\log\pi(y)+(1-p(y^{\prime}\succ
y))\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\nabla\log\pi(y^{\prime})\right]$
$\displaystyle-{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[p(y\succ
y^{\prime})\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\nabla\log\pi(y^{\prime})+(1-p(y^{\prime}\succ
y))\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\nabla\log\pi(y)\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\left(\nabla\log\pi(y)-\nabla\log\pi(y^{\prime})\right)\right]$

We deduce that:

$\displaystyle g_{\mbox{{\tiny IPO-MD}}(\beta)}$ $\displaystyle=$
$\displaystyle-\frac{2}{\tau}{\mathbb{E}}_{y\sim\pi^{\prime}}\left[\nabla\log\pi(y)\left(p(y\succ\pi^{\prime})-\frac{1}{2}\right)\right]+2{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\left(\nabla\log\pi(y)-\nabla\log\pi(y^{\prime})\right)\right]$
$\displaystyle=$
$\displaystyle-\frac{2}{\tau}{\mathbb{E}}_{y,y^{\prime}\sim\pi^{\prime}}\left[\nabla\log\pi(y)\left(p(y\succ\pi^{\prime})-\frac{1}{2}\right)-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\left(\nabla\log\pi(y)-\nabla\log\pi(y^{\prime})\right)\right]$
$\displaystyle=$
$\displaystyle-\frac{2}{\tau}{\mathbb{E}}_{y\sim\pi^{\prime}}\Big{[}\underbrace{\nabla\log\pi(y)\left(p(y\succ\pi^{\prime})-\frac{1}{2}-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\right)}_{=g(y)}\Big{]}-2{\mathbb{E}}_{y\sim\pi^{\prime}}\left[\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}\right]\underbrace{{\mathbb{E}}_{y\sim\pi^{\prime}}\left[\nabla\log\pi(y)\right]}_{=0}$

where we used that
${\mathbb{E}}_{y\sim\pi^{\prime}}\left[\nabla\log\pi(y)\right]=\frac{1}{1-\beta}{\mathbb{E}}_{y\sim\pi^{\prime}}\left[\nabla\log\pi^{\prime}(y)\right]=0$,
from the definition of $\pi^{\prime}$.

We deduce that

$\displaystyle g_{\mbox{{\tiny Nash-MD-PG}}(\beta)}=$
$\displaystyle-{\mathbb{E}}_{y\sim\pi}\left[g(y)\right]$ $\displaystyle
g_{\mbox{{\tiny IPO-MD}}(\beta)}=$
$\displaystyle-\frac{2}{\tau}{\mathbb{E}}_{y\sim\pi^{\prime}}\left[g(y)\right].\qed$

##  Appendix D Comparison of the variance of contrastive versus non-
contrastive gradient estimates

Define the following gradient estimates based on non-contrastive vs
contrastive loss functions:

$\displaystyle\hat{g}_{\mbox{{\tiny non-contrastive}}}$ $\displaystyle=$
$\displaystyle-\nabla\log\pi(y)\left(p(y\succ
y^{\prime})-\frac{1}{2}-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}+\tau\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\right)$
$\displaystyle\hat{g}_{\mbox{{\tiny contrastive}}}$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\left(\nabla\log\pi(y)-\nabla\log\pi(y^{\prime})\right)\left(p(y\succ
y^{\prime})-\frac{1}{2}-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}+\tau\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}\right).$

These two estimate resemble those of the algorithms Self-Play (as implemented
by Nash-MD-PG($\beta=0$)), which uses a non-contrastive loss, and online-IPO
(equivalent to IPO-MD($\beta=0$)), which uses a contrastive loss, and we have

$\displaystyle g_{\mbox{{\tiny Self-Play}}}$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\hat{g}_{\mbox{{\tiny
non-contrastive}}}\right]$ $\displaystyle g_{\mbox{{\tiny online-IPO}}}$
$\displaystyle=$
$\displaystyle\frac{2}{\tau}{\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\hat{g}_{\mbox{{\tiny
contrastive}}}\right].$

We know from the previous result that these two estimates have the same
expectation. However their variance may differ. Their respective variance
depends on a non-trivial combinaison of the policy representation and the
specifics of the preference model. We now state a sufficient condition under
which the contrastive gradient estimate has lower variance than its non-
contrastive counterpart.

######  Proposition D.1.

If the policy representation and the preference model are such that we have

${\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\nabla\log\pi(y)\nabla\log\pi(y^{\prime})f(y,y^{\prime})^{2}\right]\geq
0,$ (14)

where $f(y,y^{\prime}):=p(y\succ
y^{\prime})-\frac{1}{2}-\tau\log\frac{\pi(y)}{\pi^{\mathrm{ref}}(y)}+\tau\log\frac{\pi(y^{\prime})}{\pi^{\mathrm{ref}}(y^{\prime})}$,
then the variance of the contrastive gradient estimate is at least as small as
that of the non-contrastive one: $\mbox{Var}(\hat{g}_{\mbox{{\tiny
contrastive}}})\leq\mbox{Var}(\hat{g}_{\mbox{{\tiny non-contrastive}}})$.

###### Proof.

Defining the random variables
$X_{1}(y,y^{\prime}):=-\nabla\log\pi(y)f(y,y^{\prime})$ and
$X_{2}(y,y^{\prime}):=\nabla\log\pi(y^{\prime})f(y,y^{\prime})$, we have

$\displaystyle\hat{g}_{\mbox{{\tiny non-contrastive}}}$ $\displaystyle=$
$\displaystyle X_{1}(y,y^{\prime})$ $\displaystyle\hat{g}_{\mbox{{\tiny
contrastive}}}$ $\displaystyle=$
$\displaystyle\frac{X_{1}(y,y^{\prime})+X_{2}(y,y^{\prime})}{2}.$

Now, let us compare the variance of these estimates when $y$ and $y^{\prime}$
are independently drawn from the same distribution $\pi$. The variance of the
contrastive estimate is

$\mbox{Var}\left(\frac{X_{1}+X_{2}}{2}\right)=\frac{\mbox{Var}(X_{1})+\mbox{Var}(X_{2})+2\mbox{Cov}(X_{1},X_{2})}{4}=\frac{1}{2}\left(\mbox{Var}(X_{1})+\mbox{Cov}(X_{1},X_{2})\right).$

This variance is lower than $\mbox{Var}(X_{1})$ as soon as
$\mbox{Cov}(X_{1},X_{2})$ is negative (this is the principle of antithetic
variates for variance reduction).

Using the property that $f(y,y^{\prime})=-f(y^{\prime},y)$, and writing
$c={\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\nabla\log\pi(y)f(y,y^{\prime})\right]=-{\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\nabla\log\pi(y^{\prime})f(y,y^{\prime})\right]$,
we have:

$\displaystyle\mbox{Cov}(X_{1},X_{2})$ $\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\left(-\nabla\log\pi(y)f(y,y^{\prime})+c\right)\left(\nabla\log\pi(y^{\prime})f(y,y^{\prime})+c\right)\right]$
$\displaystyle=$
$\displaystyle{\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[-\nabla\log\pi(y)\nabla\log\pi(y^{\prime})f(y,y^{\prime})^{2}\right]-c^{2}.$

We deduce that a sufficient condition for the contrastive estimate to have a
lower variance than that of the non-contrastive estimate is:

${\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\nabla\log\pi(y)\nabla\log\pi(y^{\prime})f(y,y^{\prime})^{2}\right]+\left({\mathbb{E}}_{y,y^{\prime}\sim\pi}\left[\nabla\log\pi(y)f(y,y^{\prime})\right]\right)^{2}\geq
0.$

This condition is true as soon as Equation (14) is satisfied. ∎

##  Appendix E Tabular example

To build some intuition for IPO-MD, in Figure 6 we provide a plot of the
trajectories obtained with a variety of values of $\beta$ for the game with
preference probabilities as displayed below, with $\tau=0.1$, and
$\pi^{\mathrm{ref}}$ set to the uniform policy.

$\displaystyle\begin{pmatrix}0.5&0.8&0.1\\\ 0.1&0.5&0.8\\\
0.9&0.1&0.5\end{pmatrix}$

Figure 6: Online, offline, and MD variants of IPO.

##  Appendix F Supplementary Theoretical Study of Online DPO

In this section, we explore whether online DPO is related to the regularised
game given Equation 9, given that Proposition 4.1 shows a similar relationship
for online IPO and the regularised game.

Our strategy is to derive the gradient of the offline DPO objective (Lemma
F.3), and then inspect whether different sampling distributions $\mu$ and
candidate solutions $\pi$ satisfy the KKT conditions (Boyd & Vandenberghe,
2004) for the minimising the objective. We can also explore online DPO, by
inspecting the KKT conditions when the sampling distribution matches the
candidate solution ($\mu=\pi$). For example, in Lemma F.5 we present
conditions for the solution of the regularised game (see Equation 12) to also
be a stationary point of online DPO. Given that the minimiser of the online
online IPO objective is the Nash equilibrium of the regularised game given in
Equation 9 (Proposition 4.1), Lemma F.5 gives us a condition for the online
DPO problem to be equivalent (solution-wise) to the online IPO problem.
However, the condition seems difficult to satisfy: For example, there is no
2-action problem for which the condition is satisfied with the exception when
preferences are uniform ($p(1\succ 2)=\frac{1}{2}$). In this sense, apart from
the trivial uniform-preference case, online DPO and online IPO are different
objectives when $|\mathcal{Y}|=2$.

Regaring stationary points of online DPO, we show that, under the Bradley-
Terry model assumption, the RLHF solution (Equation 3) is a stationary point
of online DPO (Theorem F.7). We note in passing (Remark F.9) that the RLHF
solution is not the only solution of offline DPO when $\mu(y)=0$ for some
$y\in\mathcal{Y}$. In fact, there are infinitely many, with arbitrarily small
probabilities for $y$ such that $\mu(y)>0$. So while we can say that
stationary points of online DPO are also stationary points of offline DPO, a
solution for the offline DPO problem with a given $\mu$ may not be a solution
for corresponding the online DPO problem, because of what happens for $y$ such
that $\mu(y)=0$.

###  F.1 Results and Proofs

We let
$\Delta^{\circ}(\mathcal{Y})\doteq\left\\{p\in\Delta(\mathcal{Y}):p(y)>0,\forall
y\in\mathcal{Y}\right\\}$ be the interior of the simplex.

######  Proposition F.1.

For $t\in\mathbb{R}$, we have

$\frac{d}{dt}\log\sigma(t)=\sigma(-t).$

###### Proof.

We have:

$\displaystyle\frac{d}{dt}\log\sigma(t)$
$\displaystyle=\frac{d}{dt}-\log(1+e^{-t})=-\frac{1}{1+e^{-t}}\cdot\frac{d}{dt}(1+e^{-t})$
$\displaystyle=\frac{e^{-t}}{1+e^{-t}}=\frac{1}{1+e^{t}}=\sigma(-t).$

∎

######  Proposition F.2.

For $t\in\mathbb{R}$, we have

$\sigma(t)+\sigma(-t)=1$

###### Proof.

We have:

$\displaystyle\sigma(t)+\sigma(-t)=\frac{1}{1+e^{-t}}+\frac{1}{1+e^{t}}=\frac{1}{1+e^{t}}+\frac{e^{-t}}{1+e^{-t}}=1.$

∎

######  Lemma F.3.

Assume that $|\mathcal{Y}|<\infty$ and that $p(y\succ
y^{\prime})=1-p(y^{\prime}\succ y)$ for all $y,y^{\prime}\in\mathcal{Y}$. The
offline DPO problem with sampling distribution $\mu$ can be written as

$\sup_{\pi\in\Delta(\mathcal{Y})}J_{\mathrm{DPO}}(\pi),$

where

$\displaystyle J_{\mathrm{DPO}}(\pi)$
$\displaystyle\doteq\sum_{y,y^{\prime}}\mu(y)\mu(y^{\prime})\left(p(y\succ
y^{\prime})\log\sigma\left(\tau\log\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}\right)\right.$
$\displaystyle\phantom{\doteq}+\left.(1-p(y\succ
y^{\prime}))\log\sigma\left(-\tau\log\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}\right)\right).$

Moreover, for $\pi\in\Delta^{\circ}(\mathcal{Y})$,

$\nabla
J_{\mathrm{DPO}}(\pi)_{y}=2\tau\frac{\mu(y)}{\pi(y)}\sum_{y^{\prime}}\mu(y^{\prime})\left(p(y\succ
y^{\prime})-\sigma\left(\tau\log\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}\right)\right).$
(15)

###### Proof.

We will use the shorthands $\mu_{y,y^{\prime}}\equiv\mu(y)\mu(y^{\prime})$,
$\pi_{y}\equiv\pi(y)$ (and so forth) and
$s_{y,y^{\prime}}\doteq\tau\log\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}$.

The offline DPO objective is obtained by taking the expectation of the DPO
loss (Rafailov et al., 2023) with $Y^{+},Y^{-}\sim(\mu,\lambda_{p})$:

$\sum_{y,y^{\prime}}\mu_{y,y^{\prime}}(p(y\succ
y^{\prime})\log\sigma(s_{y,y^{\prime}})+(1-p(y\succ
y^{\prime}))\log\sigma(-s_{y,y^{\prime}}))=J_{\mathrm{DPO}}(\pi).$

Before looking at $\nabla J_{\mathrm{DPO}}(\pi)$, we will make some
simplifying observations. For $y^{\prime\prime}\neq y,y^{\prime}$,

$\frac{d}{d\pi_{y^{\prime\prime}}}s_{y,y^{\prime}}=0.$ (16)

Since $\mu_{y,y^{\prime}}=\mu_{y^{\prime},y}$,
$s_{y,y^{\prime}}=-s_{y^{\prime},y}$ and $p(y\succ
y^{\prime})=1-p(y^{\prime}\succ y)$, we have

$\displaystyle\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})\log\sigma(s_{y,y^{\prime}})+(1-p(y\succ
y^{\prime}))\log\sigma(-s_{y,y^{\prime}})\right)$ (17)
$\displaystyle=\mu_{y^{\prime},y}\left((1-p(y^{\prime}\succ
y))\log\sigma(-s_{y^{\prime},y})+p(y^{\prime}\succ
y)\log\sigma(s_{y^{\prime},y})\right).$

Also note that

$\frac{d}{d\pi_{y}}s_{y,y^{\prime}}=\frac{d}{d\pi_{y}}\left(\tau\log\frac{\pi_{y}\pi^{\mathrm{ref}}(y^{\prime})}{\pi_{y^{\prime}}\pi^{\mathrm{ref}}(y)}\right)=\tau\frac{d}{d\pi_{y}}\log\pi_{y}=\frac{\tau}{\pi_{y}}.$
(18)

Therefore, for all $y\in\mathcal{Y}$

$\displaystyle\frac{d}{d\pi_{y}}J_{\mathrm{DPO}}(\pi)$
$\displaystyle=2\sum_{y^{\prime}}\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})\frac{d}{d\pi_{y}}\log\sigma(s_{y,y^{\prime}})+(1-p(y\succ
y^{\prime}))\frac{d}{d\pi_{y}}\log\sigma(-s_{y,y^{\prime}})\right),$
(Equations 16 and 17)
$\displaystyle=2\sum_{y^{\prime}}\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})\sigma(-s_{y,y^{\prime}})\frac{d}{d\pi_{y}}s_{y,y^{\prime}}-(1-p(y\succ
y^{\prime}))\sigma(s_{y,y^{\prime}})\frac{d}{d\pi_{y}}s_{y,y^{\prime}}\right)$
(Proposition F.1)
$\displaystyle=2\sum_{y^{\prime}}\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})\sigma(-s_{y,y^{\prime}})-(1-p(y\succ
y^{\prime}))\sigma(s_{y,y^{\prime}})\right)\frac{d}{d\pi_{y}}s_{y,y^{\prime}}$
$\displaystyle=2\sum_{y^{\prime}}\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})-\sigma(s_{y,y^{\prime}})\right)\frac{d}{d\pi_{y}}s_{y,y^{\prime}}$
(Proposition F.2)
$\displaystyle=2\sum_{y^{\prime}}\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})-\sigma\left(\tau\log\frac{\pi_{y}\pi^{\mathrm{ref}}_{y^{\prime}}}{\pi_{y^{\prime}}\pi^{\mathrm{ref}}_{y}}\right)\right)\frac{\tau}{\pi_{y}},$
(Equation 18)
$\displaystyle=2\tau\frac{\mu_{y}}{\pi_{y}}\sum_{y^{\prime}}\mu_{y,y^{\prime}}\left(p(y\succ
y^{\prime})-\sigma\left(\tau\log\frac{\pi_{y}\pi^{\mathrm{ref}}_{y^{\prime}}}{\pi_{y^{\prime}}\pi^{\mathrm{ref}}_{y}}\right)\right).$

∎

######  Remark F.4.

We have restricted the statement of Lemma F.3 to the interior of the simplex
to stay consistent with the DPO formulation, however Equation 15 holds for any
function $\exp\circ f$ with $f:\mathcal{Y}\rightarrow\mathbb{R}$. To see this,
it suffices to notice that $J_{\mathrm{DPO}}$ is a function of the ratios
$\frac{\pi_{i}}{\pi_{j}}$, so for any $\alpha>0$
$J_{\mathrm{DPO}}(\pi)=J_{\mathrm{DPO}}(\alpha\cdot\pi)$ (where
$\alpha\cdot\pi\doteq y\mapsto\alpha\cdot\pi(y)$).

######  Lemma F.5.

The Nash equilibrium $\pi^{*}$ of the regularised game given in Equation 9 is
a stationary point of online DPO iff:

$p(y\succ\pi^{*})=\sum_{y^{\prime}}\pi^{*}(y^{\prime})\sigma(p(y\succ\pi^{*})-p(y^{\prime}\succ\pi^{*})).$
(19)

###### Proof.

Let $\pi^{*}$ be the Nash equilibrium of the regularised game given in
Equation 9, the fixed point in Equation 12:

$\pi^{*}(y)\propto\pi^{\mathrm{ref}}(y)\exp\left(\frac{1}{\tau}p(y\succ\pi^{*})\right).$
(20)

$\pi^{*}$ will be a stationary point of online DPO iff $\pi=\pi^{*}$ is a
solution of the offline DPO problem with $\mu=\pi^{*}$. Since
$\pi^{*}\in\Delta^{\circ}(\mathcal{Y})$, we can use Lemma F.3, $\pi^{*}$ is a
solution of the offline DPO problem iff

$\nabla J_{\mathrm{DPO}}(\pi^{*})=0.$

We have for all $y\in\mathcal{Y}$:

$\displaystyle\frac{d}{d\pi_{y}}J_{\mathrm{DPO}}(\pi^{*})$
$\displaystyle=2\tau\frac{\mu(y)}{\pi^{*}(y)}\sum_{y^{\prime}}\mu(y^{\prime})\left(p(y\succ
y^{\prime})-\sigma\left(\tau\log\frac{\pi^{*}(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi^{*}(y^{\prime})\pi^{\mathrm{ref}}(y)}\right)\right)$
(Equation 15)
$\displaystyle=2\tau\sum_{y^{\prime}}\pi^{*}(y^{\prime})\left(p(y\succ
y^{\prime})-\sigma\left(\tau\log\frac{\pi^{*}(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi^{*}(y^{\prime})\pi^{\mathrm{ref}}(y)}\right)\right)$
($\mu=\pi^{*}$)
$\displaystyle=2\tau\sum_{y^{\prime}}\pi^{*}(y^{\prime})\left(p(y\succ
y^{\prime})-\sigma\left(p(y\succ\pi^{*})-p(y^{\prime}\succ\pi^{*})\right)\right)$
(Equation 20)
$\displaystyle=2\tau\left(p(y\succ\pi^{*})-\sum_{y^{\prime}}\pi^{*}(y^{\prime})\sigma\left(p(y\succ\pi^{*})-p(y^{\prime}\succ\pi^{*})\right)\right),$

and the result follows by using the fact that $\tau>0$. ∎

######  Theorem F.6.

No 2-action regularised game given in Equation 9 has a Nash equilibrium that
satisfies Equation 19, except for the regularised games with $p(y_{1}\succ
y_{2})=\frac{1}{2}$.

###### Proof.

Let $\mathcal{Y}=\\{1,2\\}$. For this two-action problem, we can write the
preference matrix as

$P\doteq\left(\begin{array}[]{cc}\frac{1}{2}&1-p\\\
p&\frac{1}{2}\end{array}\right),$

where $P_{yy^{\prime}}=p(y\succ y^{\prime})$.

Let $\alpha$ be such that $\pi^{*}=(\alpha,1-\alpha)^{\top}$. Then

$p(y\succ\pi^{*})=P\pi^{*}=\left(1-\frac{\alpha}{2}-p+\alpha
p,\frac{1}{2}-\frac{\alpha}{2}+\alpha p\right)^{\top},$

and

$p(1\succ\pi^{*})-p(2\succ\pi^{*})=\frac{1}{2}-p.$

Then the difference between both sides of Equation 19 for $y=1$ is:

$\displaystyle
p(1\succ\pi^{*})-\sum_{y^{\prime}}\pi^{*}(y^{\prime})\sigma(p(1\succ\pi^{*})-p(y^{\prime}\succ\pi^{*}))$
$\displaystyle=1-\frac{\alpha}{2}-p+\alpha
p-\alpha\sigma(0)-(1-\alpha)\sigma\left(\frac{1}{2}-p\right)$
$\displaystyle=1-\alpha-p+\alpha p-(1-\alpha)\sigma\left(\frac{1}{2}-p\right)$
$\displaystyle=(1-\alpha)\left(1-p-\sigma\left(\frac{1}{2}-p\right)\right).$

For $y=2$ we get

$\displaystyle
p(2\succ\pi^{*})-\sum_{y^{\prime}}\pi^{*}(y^{\prime})\sigma(p(2\succ\pi^{*})-p(y^{\prime}\succ\pi^{*}))$
$\displaystyle=\alpha
p+\frac{(1-\alpha)}{2}-\alpha\sigma\left(p-\frac{1}{2}\right)-(1-\alpha)\sigma(0)$
$\displaystyle=\alpha p-\alpha\sigma\left(p-\frac{1}{2}\right)$
$\displaystyle=\alpha p-\alpha+\alpha\sigma\left(\frac{1}{2}-p\right)$
$\displaystyle=-\alpha\left(1-p-\sigma\left(\frac{1}{2}-p\right)\right).$

Now, if we let $\varepsilon\doteq\frac{1}{2}-p$, we can see that for
$p<\frac{1}{2}$

$\sigma\left(\frac{1}{2}-p\right)=\sigma(\varepsilon)<\sigma(0)=\frac{1}{2}<\frac{1}{2}+\varepsilon=1-p,$

so (considering the analogous case for $p<\frac{1}{2}$)

$1-p-\sigma\left(\frac{1}{2}-p\right)\begin{cases}>0,&p<\frac{1}{2},\\\
=0,&p=\frac{1}{2},\\\ <0,&p>\frac{1}{2}.\end{cases}$

Therefore, if $p\neq\frac{1}{2}$, we cannot satisfy Equation 19 for $y=1$ and
$y=2$ (note that $\alpha=0$ or $\alpha=1$ satisfy the equation for only one
$y$). ∎

######  Theorem F.7.

Assume that the preferences admit a Bradley-Terry model (Rafailov et al.,
2023), that is, there exists $r:\mathcal{Y}\rightarrow\mathbb{R}$ such that
for all $y,y^{\prime}\in\mathcal{Y}$

$p(y\succ y^{\prime})=\sigma(r(y)-r(y^{\prime})),$

Then

$\pi^{r}(y)\propto\pi^{\mathrm{ref}}(y)\exp\left(\frac{1}{\tau}r(y)\right)$
(21)

is a critical point for offline IPO for any $\mu$, and a stationary point of
online DPO.

###### Proof.

First, assume that $\pi^{\mathrm{ref}}\in\Delta^{\circ}(\mathcal{Y})$ (we will
deal with the general case at the end).

Let us consider the offline DPO problem with sampling distribution $\mu$.
Assuming that the preferences admit a Bradley-Terry model, we know from
Rafailov et al. (2023) that the solution of the offline DPO problem is given
by

$\pi^{r}(y)\propto\pi^{\mathrm{ref}}(y)\exp\left(\frac{1}{\tau}r(y)\right).$
(22)

Indeed, we can verify that for any $\mu$

$\displaystyle\nabla J_{\mathrm{DPO}}(\pi^{r})$
$\displaystyle=2\tau\frac{\mu(y)}{\pi(y)}\sum_{y^{\prime}}\mu(y^{\prime})\left(p(y\succ
y^{\prime})-\sigma\left(\tau\log\frac{\pi(y)\pi^{\mathrm{ref}}(y^{\prime})}{\pi(y^{\prime})\pi^{\mathrm{ref}}(y)}\right)\right)$
($\pi^{r}\in\Delta^{\circ}(\mathcal{Y})$, Equation 15)
$\displaystyle=2\tau\frac{\mu(y)}{\pi^{r}(y)}\sum_{y^{\prime}}\mu(y^{\prime})\left(p(y\succ
y^{\prime})-\sigma\left(r(y)-r(y^{\prime})\right)\right)$ (Equation 21)
$\displaystyle=2\tau\frac{\mu(y)}{\pi(y)}\sum_{y^{\prime}}\mu(y^{\prime})\left(p(y\succ
y^{\prime})-p(y\succ y^{\prime})\right)$ (Bradley-Terry model assumption)
$\displaystyle=0$

It follows that $\pi^{r}$ is an offline DPO solution under any sampling
distribution $\mu$, and if $\mu\in\Delta^{\circ}(\mathcal{Y})$ then $\pi^{r}$
is the only solution. In particular this means that $\pi^{r}$ is also a
stationary point for online DPO (by taking $\mu=\pi^{r}$).

In the case where
$\pi^{\mathrm{ref}}\in\Delta(\mathcal{Y})-\Delta^{\circ}(\mathcal{Y})$, we can
prove the result as follows. Let
$\mathcal{Y}^{\prime}\doteq\\{y:\pi^{\mathrm{ref}}(y)=0\\}$. For
$y\in\mathcal{Y}^{\prime}$, we have $\pi^{r}(y)=0$ by Equation 21. Since
$\pi^{r}$ is the sampling distribution, the gradient of the offline DPO
objective in $\mathcal{Y^{\prime}}$ is zero (and it does not matter that
$\log\pi^{r}(y)$ and $\log\pi^{\mathrm{ref}}(y)$ are undefined since they do
not appear in the objective). Now it suffices to note that $\pi^{r}$
restricted to $\mathcal{Y}-\mathcal{Y}^{\prime}$ is a distribution in
$\Delta^{\circ}(\mathcal{Y}-\mathcal{Y}^{\prime})$, so we can use the previous
case ($\pi^{r}\in\Delta^{\circ}(\mathcal{Y}))$ to show the result, and it
follows that $\pi^{r}$ satisfies the KKT conditions for all $y\in\mathcal{Y}$.
∎

######  Remark F.8.

In the special case of rock-paper-scissors preferences (which do not admit a
Bradley-Terry model) with $\pi^{\mathrm{ref}}$ uniform and $\tau>0$, we can
still satisfy Equation 19, since $\pi^{*}$ is also uniform and
$p(y\succ\pi^{*})=\frac{1}{2}$.

######  Remark F.9.

In offline DPO, the presence of sets of measure zero guarantees the solution
is not unique. Let $\pi^{\mathrm{DPO}}$ be a solution of offline DPO under
sampling distribution $\mu$ (assume it exists). $\pi^{\mathrm{DPO}}$ may not
necessarily be $\pi^{r}$ from (21) (even if we assume that the preferences
admit a Bradley-Terry model).

To see this, assume that there exists $\mathcal{Y}^{\prime}\subset\mathcal{Y}$
such that $\mathcal{Y}^{\prime}\neq\emptyset$ and $\mu(y)=0$ for all
$y\in\mathcal{Y}^{\prime}$. For any $\alpha\in(0,1]$, define
$\pi^{\alpha}\in\Delta^{\circ}(\mathcal{Y})$ by

$\pi^{\alpha}(y)\doteq\begin{cases}\alpha\cdot\pi^{\mathrm{DPO}}(y),&y\notin\mathcal{Y}^{\prime},\\\
\mbox{arbitrary},&y\in\mathcal{Y}^{\prime}.\end{cases}$

By Lemma F.3, $\nabla J_{\mathrm{DPO}}(\pi^{\alpha})=0$, so $\pi^{\alpha}$ is
a minimum of the offline DPO objective. In fact, this holds for any
$\alpha>0$, no matter how small, and this also means that if
$\mu\notin\Delta^{\circ}(\mathcal{Y})$ we can find offline DPO solutions with
arbitrarily small probabilities for sampled $y$ ($y$ such that $\mu(y)>0$).