Ana M. Teixeira

# Charged lepton flavour violation in muons: theory overview

###### Abstract

We briefly overview some theoretical aspects of charged lepton flavour
violation in rare muon transitions and decays. Relying on the effective field
theory approach (model-independent), we discuss the probing power of charged
lepton flavour violation processes regarding the new physics scale, especially
in what concerns muon-electron conversion in nuclei. We then illustrate how
these processes can provide valuable information (allowing to constrain,
falsify - or conversely to learn about the underlying new physics properties)
for a few examples of well-motivated extensions of the Standard Model. We
finally comment on how leptonic CP violation should be carefully considered
upon studies of charged lepton flavour violation in extensions of the Standard
Model featuring heavy sterile fermions.

##  1 Introduction

Flavour physics has played a key role in the shaping and evolution of particle
physics since the early days of the Standard Model (SM). In recent years,
flavours have paved the way to searches for New Physics (NP): from the
discovery of neutrino oscillations, to the long-standing tensions between
theory and observation regarding the anomalous magnetic moment of the muon, to
conflicting data on possible violation of lepton flavour universality in meson
decays. Curiously, many of these tensions are deeply rooted in the lepton
sector. Strictly forbidden in the context of the SM, the violation of lepton
flavour was clearly demonstrated with neutrino oscillation phenomena, and in
principle, once the latter occurs, nothing precludes the violation of charged
lepton flavours. However, minimal extensions of the SM envisaged to
accommodate neutrino oscillation data (via the addition of massive Dirac
neutrinos, and lepton mixings) lead to tiny rates for charged lepton flavour
violating (cLFV) transitions. Thus, any observation of a cLFV process would
offer striking evidence for NP - beyond such minimal SM extensions.

In view of their very appealing properties, muons are ideal objects to search
for cLFV, with numerous muonic transitions as the object of extensive
dedicated searches at low-energies and at colliders. The former include
radiative decays ($\mu\to e\gamma$), three-body decays ($\mu\to 3e$), muonic-
bound state processes, such as Muonium (Mu $=\mu^{+}e^{-}$) oscillations and
decays, or transitions in the presence of muonic atoms (including neutrinoless
$\mu-e$ conversion, among others). At higher energies - at the LHC or at
future lepton colliders -, numerous processes can also lead to a discovery of
cLFV, as is the case of neutral boson decays ($Z,H\to\mu e$), or the study of
high-$p_{T}$ dilepton tails in $pp\to\mu\ell$. The expected future
sensitivities of muon-cLFV dedicated facilities suggest that these processes
might offer a sensitivity to NP mediators up to scales around $10^{5}$ TeV
[1].

Muon cLFV is thus poised to offer evidence for NP, or then further strengthen
the constraints on the scale of the new mediators and on the intensity of the
new interactions with leptons; from a theory perspective, studies of muon cLFV
can be carried out following an effective (model-independent) approach, or
investigating the prospects for cLFV of a given, well-motivated NP model
(especially those which also offer an explanation to the SM observational
issues, including neutrino mass generation and the baryon asymmetry of the
Universe, or a viable dark matter candidate). In what follows, we will
illustrate these approaches.

##  2 Muon cLFV: effective approach to NP

In the effective field theory (EFT) approach, the SM Lagrangian is generalised
to include high-order, non-renormalisable terms that encode the effects of
heavy new states (and their interactions), possibly present at scales far
above the electroweak one, $\Lambda_{\text{NP}}\gg\Lambda_{\text{EW}}$. The
observables can be expressed in terms of the new couplings and scale, and
available bounds are then used to constrain combinations of the latter. In
view of the extensive number of operators that can be at the origin of the
cLFV transitions (dimension 6, and higher), the task is far from simple. In
recent years, numerous analyses have emphasised the relevance of taking into
account higher-order contributions, renormalisation group running (leading to
operator mixing), among other points, also suggesting the importance of
including as many observables and operators as possible, so to more
efficiently constrain the nature of the NP contributions (see, e.g. [2]). A
new parametrisation proposal [3], relying on projections onto a space spanned
by the most relevant operators for muon cLFV transitions,
$\vec{C}=\\{C_{D},C_{S},C_{VR},C_{VL},C_{\text{N-light}},C_{\text{N-heavy}}\\}$,
allowed to infer the sensitivity to NP scales of data (current bounds and
future sensitivities) stemming from searches for radiative and 3-body decays,
and conversion in nuclei. As can be seen from the left panel of Fig. 1,
current bounds from MEG on $\mu\to e\gamma$ and from SINDRUM on neutrinoless
conversion in Gold nuclei allow to draw limits on the scale of cLFV new
physics up to $\mathcal{O}(10^{3}~{}\text{TeV})$, respectively for the cases
of dipole or 4-fermion dominated NP contributions ($\theta_{D}\to 0,\pi$ or
$\theta_{D}\to\pi/2$). In the future, and for both regimes, the expected
sensitivity of searches for $\mu-e$ conversion in Aluminium by both COMET and
Mu2e should allow probing scales $\Lambda_{\text{NP}}\sim 10^{4}$ TeV (or even
$10^{5}$ TeV for an advanced muon facility) [3].

Figure 1: On the left, sensitivity to the NP scale of different cLFV muon
transitions, as a function of the “dipole” dominance parameter $\theta_{D}$:
radiative decays (blue), three-body decays (black) and neutrinoless conversion
in nuclei (red); solid (dotted) lines denote current bounds (future
experimental sensitivity); from [3]. On the right, complementarity of the
information for distinct nuclei ($Z$) regarding the NP at the source of
$\mu-e$ conversion, with respect to that of Aluminium targets; from [4].

The impressive sensitivity associated with $\mu-e$ conversion searches lends
hope to a future observation of these rare transitions. In the event of a
discovery, it becomes important to (re)visit the choice of nuclear target, so
as to infer as much information as possible on the nature of the underlying
NP. Recent studies suggest that Vanadium-51 or Lithium-7 are feasible choices
of targets, which could offer the most complementary (or orthogonal)
information to that of Aluminium targets [4]. The potential complementarity of
several targets is displayed in the right panel of Fig. 1.

A very interesting process, also occurring in muonic atoms, is that of the
flavour and lepton number violation transition (LNV), $\mu^{-}+(A,Z)\to
e^{+}+(A,Z+2)^{(*)}$, in which the final state nucleus might be in an excited
state. This transition could offer a pivotal connection between cLFV and LNV
(by two units), in association with the possible Majorana nature of the NP
mediator, and a link to the mechanism of neutrino mass generation). Already
experimentally searched for, this process is highly non-trivial to tackle from
a theory point of view: not only does it call upon higher-order operators
(contrary to the “usual” cLFV dimension 6 operators) but, as of today, the
nuclear matrix elements remain extremely challenging to compute, rendering any
inference hard to draw.

##  3 Muon cLFV - learning about models of NP

If the SM extension includes sources of cLFV, the model can in principle
account for extensive ranges for the cLFV observables. Depending on the
dominant contributions to given transitions, one can nevertheless try to
identify distinctive patterns for the observables: for example, should one
have photon-penguin dominance in $\mu-e$ conversion, then one can expect a
correlation between the former and radiative (as well as 3-body) decays;
should there be a $Z$ penguin dominance, then it is likely that BR($\mu\to
3e$) and CR($\mu-e$, N) be correlated. The confrontation of ratios of future
measurements of observables with the predictions of a given model might offer
valuable information to falsify a given construction, or to strengthen its
viability. Several examples have been presented in [5]. Here we very briefly
consider a few (recent) examples of how cLFV - especially through the muon
channels - can offer powerful probes of NP models. We consider illustrative
cases associated with mechanisms of neutrino mass generation, grand unified
theories (GUT), or those aiming at explaining current tensions of the SM.

Scotogenic models offer a powerful link between two of the SM’s most pressing
issues - massive neutrinos and the need for a dark matter (DM) candidates.
Furthermore, scotogenic models usually call upon symmetries to stabilise the
DM candidate, which in general prevents tree-level neutrino mass generation
(which then occurs at 1-loop or at higher levels).
Many scotogenic models lead to interesting cLFV signatures: a recent analysis
[6] considered an extension of the SM via 2 Weyl fermions as well as Majorana
fermions (singlets and scalars); such a construction was shown to successfully
explain neutrino oscillation data, put forward a viable DM candidate, and
further saturate the tensions on the muon anomalous magnetic moment. In the
muon-electron sector, cLFV 3-body decays are dominated by photon-penguin
contributions, leading to a strict correlation between BR($\mu\to e\gamma$)
and BR($\mu\to 3e$), as shown in the left panel of Fig. 2; a future
determination of these observables might allow to falsify such a NP scenario.

Figure 2: On the left, correlated behaviour of $\mu\to e\gamma$ and $\mu\to
3e$ decays for a scotogenic NP model, as studied in [6]. On the right, ratios
of cLFV observables (BR($\mu\to 3e$), BR($K_{L}\to\mu e$) and CR($\mu-e$, N))
as a function of the ratio of the NP mediator masses (scalar triplet and LQ
masses) in an SU(5) GUT model; from [7].

Another enlightening example can be found in GUT models, ambitious
constructions which aim at addressing not only the SM observational issues,
but also at overcoming some of its theoretical caveats. Despite their appeal,
many such models rely on extended field contents (which can be present at very
high scales), rendering them hard to probe. However, certain GUT models lead
to interesting cLFV “signatures”, which offer a means to learn about the new
spectrum (among other features). This was explored in [7], in which a type II
seesaw was embedded into an SU(5) non-supersymmetric GUT. Dedicated cLFV
studies revealed peculiar patterns, dictated by the mass ratio of the new
triplet seesaw ($\Delta$) mediator and its SU(5) leptoquark (LQ) partners. As
visible on the right panel of Fig. 2, there is a clear correlation of
BR($\mu\to 3e$), BR($K_{L}\to\mu e$) and CR($\mu-e$, N). A near-future
observation (as well as current bounds) seem to prefer scenarios in which
$M_{\Delta}>M_{\text{LQ}}$. Furthermore, cLFV observations can also hint on
the ordering of the light neutrino spectrum: if BR($K_{L}\to\mu e$)
$>10\,\times\,$ CR($\mu-e$, N), the inverted ordering would be disfavoured.

The previous example also highlights how other cLFV transitions (in addition
to the well-known muon radiative and 3-body decays, and conversion in nuclei)
might be instrumental in probing NP models. These modes include
(semi-)leptonic meson decays, as well as the already mentioned Muonium cLFV
decays. Likewise, the tau sector should also be included, even the more
involved decay modes (e.g. doubly lepton-flavour violating
$\tau\to\mu\bar{e}\mu$ 3-body decays), which can sometimes be at the source of
the most stringent constraints (see, for instance, the study of [8]).

##  4 Leptonic phases and muon cLFV

As mentioned before, “patterns” of cLFV observables have been intensively
explored as means to falsify numerous NP models - including mechanisms of mass
generation. In general, the different fundamental seesaw realisations lead to
well-defined signatures of cLFV ratios, which reflect the nature of the
distinct mediators, and whether the transitions occur at tree-level or at
higher orders. Especially in the case of low-scale seesaw realisations, the
contributions to the distinct cLFV transitions can be sizeable, and thus lead
to important constraints on these SM extensions.

However, most studies rely on CP conserving interactions - but leptonic CP
phases (be them Dirac and/or Majorana) are in general present in SM extensions
aiming at accounting for neutrino masses and lepton mixings, and are a crucial
ingredient to explain the baryon asymmetry of the Universe via leptogenesis.
Recently, the impact of the leptonic CP phases on cLFV observables was
addressed for the generic case of SM extensions via at least 2 Majorana heavy
neutral leptons (HNL), relying on a simplified-model approach to well-
motivated NP models, as for instance the type I seesaw and its variants [9,
10]. In association with the 2 heavy sterile states, the effects due to the
new CPV phases can be at the source of important constructive and destructive
interferences, which strongly alter the theoretical expectations, and the
interpretation of any future data. This is clearly manifest in the left panel
of Fig. 3: for the CP conserving case (blue) there is a visible correlation of
the neutrinoless conversion and 3-body muon decays (a consequence of having
both processes dominated by $Z$-penguin contributions); in particular, and
should HNL be at the source of cLFV phenomena, any future observation of
$\mu\to 3e$ would suggest that $\mu-e$ conversion should also be observed.
Once CP violation is present (green and orange), there is a dramatic loss of
correlation. In view of the potential presence of CPV phases, constraining HNL
extensions from cLFV observables is an exercise that must be carefully
done111Although in general cLFV observables remain privileged probes for low-
scale seesaw realisations relying on additional heavy sterile fermions, lepton
flavour universality violation observables and $\Gamma(Z\to\text{inv.})$ can
be powerful complementary probes to cLFV in the $\mu-e$ sector, as recently
pointed out in [11].. Leptonic CPV phases are also expected to impact cLFV
(neutral) boson decays; at a future FCC-ee, one can look for their presence by
considering CP asymmetries in cLFV $Z$ boson decays. In particular, and as
seen from the right panel of Fig. 3, for both BR($Z\to\mu\tau$) and
BR($\tau\to 3\mu$) within experimental reach, $\mathcal{A}_{CP}(Z\to\mu\tau)$
can be as large as 20%, thus offering a potential 3-fold way to test such
minimal SM extensions via at least 2 heavy sterile states.

Figure 3: On the left, projections for CR($\mu-e$, Al) vs. BR($\mu\to 3e$),
for a minimal (ad-hoc) SM extension via 2 HNL: CP conserving limit (blue) and
the effect of CP violating leptonic phases (orange and green); dotted (dashed)
lines denote current bounds (future sensitivity); from [9]. On the right,
prospects for BR($\tau\to 3\mu$) vs. BR($Z\to\tau\mu$), also for a SM
extension via 2 HNL. The colour code denotes the CP-asymmetry in $Z\to\tau\mu$
decays; dotted (dashed) lines denote current Belle bounds (FCC-ee future
sensitivity); from [10].

##  5 Overview

In view of the very ambitious experimental programme, lepton physics is
expected to provide valuable hints on the underlying New Physics model, which
is clearly needed to address the SM caveats. Charged lepton flavour violating
muon decays and transitions will offer precious help in identifying the NP
model at work, hinting on the scale and nature of the new mediators, reducing
ambiguities, constraining parameter spaces, or even falsifying SM extensions.

The near future will witness a unique experimental progress in muon-cLFV
dedicated searches [12], rendering a possible a discovery of New Physics,
potentially before the observation of new resonance at colliders. Theory
efforts and improved theoretical approaches must keep on par with experimental
developments: one must thus consider as many observables as possible
(including proposals of new ones), explore distinct approaches (effective
theories, and dedicated phenomenological studies of well-motivated NP models),
striving to increase theoretical control at all levels, as well as fully
exploring synergies of observables and flavour sectors.

## Acknowledgments

We acknowledge support from the EU’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement No. 860881 (HIDDeN
network) and from the IN2P3 (CNRS) Master Project, “Flavour probes: lepton
sector and beyond” (16-PH-169).

## References

  * [1] R. K. Ellis et al. arXiv:1910.11775 [hep-ex].
  * [2] A. Crivellin, S. Davidson, G. M. Pruna and A. Signer, JHEP 05 (2017), 117.
  * [3] S. Davidson and B. Echenard, Eur. Phys. J. C 82 (2022) no.9, 836.
  * [4] J. Heeck, R. Szafron and Y. Uesaka, Nucl. Phys. B 980 (2022), 115833.
  * [5] L. Calibbi and G. Signorelli, Riv. Nuovo Cim. 41 (2018) no.2, 71-174.
  * [6] A. Alvarez et al, JHEP 06 (2023), 163.
  * [7] L. Calibbi and X. Gao, Phys. Rev. D 106 (2022) no.9, 095036.
  * [8] J. Kriewald, J. Orloff, E. Pinsard and A. M. Teixeira, Eur. Phys. J. C 82 (2022) no.9, 844.
  * [9] A. Abada, J. Kriewald and A. M. Teixeira, Eur. Phys. J. C 81 (2021) no.11, 1016.
  * [10] A. Abada, J. Kriewald, E. Pinsard, S. Rosauro-Alcaraz and A. M. Teixeira, Eur. Phys. J. C 83 (2023) no.6, 494.
  * [11] A. Abada, J. Kriewald, E. Pinsard, S. Rosauro-Alcaraz and A. M. Teixeira, arXiv:2307.02558 [hep-ph].
  * [12] See many dedicated presentations during this Workshop (and associated proceedings at PoS).
  * [13]


# A density estimation perspective on learning from pairwise human preferences

Vincent Dumoulin22footnotemark: 2, Daniel D. Johnson22footnotemark:
2,33footnotemark: 3, Pablo Samuel Castro22footnotemark: 2,44footnotemark: 4,
Hugo Larochelle22footnotemark: 2,44footnotemark: 4,55footnotemark: 5,
Yann Dauphin22footnotemark: 2
<EMAIL_ADDRESS>
22footnotemark: 2 Google DeepMind
33footnotemark: 3 University of Toronto
44footnotemark: 4 Mila, Université de Montréal
55footnotemark: 5 CIFAR Fellow

###### Abstract

Learning from human feedback (LHF)—and in particular learning from pairwise
preferences—has recently become a crucial ingredient in training large
language models (LLMs), and has been the subject of much research. Most recent
works frame it as a reinforcement learning problem, where a reward function is
learned from pairwise preference data and the LLM is treated as a policy which
is adapted to maximize the rewards, often under additional regularization
constraints. We propose an alternative interpretation which centers on the
generative process for pairwise preferences and treats LHF as a density
estimation problem. We provide theoretical and empirical results showing that
for a family of generative processes defined via preference behavior
distribution equations, training a reward function on pairwise preferences
effectively models an annotator’s implicit preference distribution. Finally,
we discuss and present findings on “annotator misspecification”—failure cases
where wrong modeling assumptions are made about annotator behavior, resulting
in poorly-adapted models—suggesting that approaches that learn from pairwise
human preferences could have trouble learning from a population of annotators
with diverse viewpoints.111A notebook reproducing all experiments in this
paper can be accessed at https://github.com/google-deepmind/pbde.

##  1 Introduction

With the recent surge of interest in large language models (LLMs), learning
from human feedback (LHF) has received a great deal of attention from the
machine learning community. Pretraining of large Transformer architectures
(Vaswani et al., 2017) on web-scale data has been essential to their success,
but by itself the pretraining objective produces unconditional generative
models that are difficult to control. A critical component in training LLMs is
therefore to finetune them to produce “good” responses, which is typically
formalized as aligning their outputs with human preferences.

The current dominant approach to aligning LLMs with human preferences relies
on pairwise comparisons of model outputs, and frames the problem as a
reinforcement learning (RL) problem where the LLM is treated as a policy.
Preference-based learning of reward functions (Sadigh et al., 2017; Biyik &
Sadigh, 2018)—and more specifically preference-based RL or RL from human
preferences (abbreviated as RLHF; Wirth et al., 2017; Christiano et al., 2017;
Ibarz et al., 2018; Brown et al., 2019; 2020; Lee et al., 2021; Shin et al.,
2023)—is used to finetune LLMs from model output pairs which have been ranked
by annotators to reflect how preferable those outputs are (Stiennon et al.,
2020; Ouyang et al., 2022; Bai et al., 2022).

However, interpreting preference-alignment as an RL problem requires a change
of perspective from the original pretraining task. For LLM pretraining, the
loss is derived from a probabilistic modeling formulation of the problem: The
language model is treated as an autoregressive probability distribution, and
next-token prediction is used to maximize the likelihood of human language
samples. For human preference alignment, the language model is treated as a
policy and the loss aims to maximize a (learned) reward which is reflective of
human preferences. Given the probabilistic origins of training LLMs, it is
worth considering whether certain probabilistic modeling tools and techniques
are being overlooked when taking the RL perspective.

We argue that treating human preference alignment as a probabilistic modeling
task forces us to be explicit about our assumptions on the generative process
for annotator preferences and provides important insights into the properties
and failure modes of algorithms that learn from them. Our main contributions
are:

  * •

We show that the standard procedure for training a reward function on pairwise
human preferences can be reinterpreted as performing density estimation on an
annotator’s implicit preference distribution, assuming annotators behave
according to the Luce choice rule (Luce et al., 1963);

  * •

We generalize the Luce choice rule to a family of generative processes for
pairwise preferences defined via preference behavior distribution equations
(PBDEs) and provide global optimality results in the case where the annotator
and model behave according to the same PBDE;

  * •

Finally, we show that an unintentional mismatch in generative processes for
pairwise preferences between the annotator and model (which we term annotator
misspecification) can lead to badly-tuned policies, highlighting the
importance of explicitly stating assumptions on annotator behavior.

##  2 Background

Here we provide some technical background and notation which will be used
throughout the manuscript.

###  2.1 Reinforcement Learning

Reinforcement learning tackles sequential decision-making problems and is
typically formulated as a Markov decision process
$\mathcal{M}=\\{\mathcal{X},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\\}$
using the following definitions: $\mathcal{X}$ is the set of states;
$\mathcal{A}$ is a set of actions (or “one-step decisions”);
$\mathcal{P}:\mathcal{X}\times\mathcal{A}\rightarrow\Delta(\mathcal{X})$
defines the transition dynamics, where $\mathcal{P}(x,a)(x^{\prime})$ is the
probability of transitioning to state $x^{\prime}$ after performing action $a$
from state $x$;
$\mathcal{R}:\mathcal{X}\times\mathcal{A}\rightarrow\mathbb{R}$ is the one-
step reward, where $\mathcal{R}(x,a)$ is the reward received after performing
action $a$ from state $x$; $\gamma\in[0,1)$ is a discount factor that places
more emphasis on rewards received in the short-term.

An agent’s behaviour is formalized via a policy
$\pi:\mathcal{X}\rightarrow\Delta(\mathcal{A})$ which gives a distribution
over possible actions conditioned on a state (e.g., $\pi(x)(a)$ is the
probability given by policy $\pi$ of picking action $a$ when in state $x$).
Given a fixed policy $\pi$, we can express rewards as a function of state:
$r_{\pi}(x):=\mathbb{E}_{a\sim\pi(x)}\mathcal{R}(x,a)$; this notation will
prove useful when considering reward functions over LLM outputs. Every policy
$\pi$ induces a value function given by the expected sum of discounted rewards
when following policy $\pi$:

$V^{\pi}(x)=\sum_{t=0}^{\infty}\bigg{[}\gamma^{t}\mathcal{R}(x_{t},a_{t})\mid
x_{0}=x,a_{t}\sim\pi(x_{t}),x_{t+1}\sim\mathcal{P}(x_{t},a_{t})\bigg{]}.$

It is useful to also consider state-action value functions, which quantify the
value of taking an action $a$ from state $x$, and following the policy $\pi$
afterwards:
$Q^{\pi}(x,a):=\mathcal{R}(x,a)+\gamma\mathbb{E}_{x^{\prime}\sim\mathcal{P}(x,a)}V^{\pi}(x^{\prime})$.
Finally, we denote by $\mu_{\pi}\in\Delta(\mathcal{X})$ the stationary state
distribution induced by a policy $\pi$.

The goal of RL agents is then to find the optimal policy $\pi^{*}$, where
optimality is defined uniformly across the state space (e.g. $V^{\pi^{*}}\geq
V^{\pi}$ for every $\pi$). In most modern RL applications, policies are
expressed by neural networks with parameters $\theta$, and we denote the
respective policy as $\pi_{\theta}$. There are a number of approaches for
tackling this problem (Sutton & Barto, 1998), but for the purposes of this
work we focus on policy gradient methods. Specifically, the policy gradient
Theorem asserts the following relationship:

$\nabla
V^{\pi_{\theta}}\propto\sum_{x\in\mathcal{X}}\mu_{\pi_{\theta}}(x)\sum_{a\in\mathcal{A}}Q^{\pi_{\theta}}(x,a)\nabla\pi_{\theta}(x)(a).$

This result establishes that we can use gradient-based optimization methods to
maximize returns by only requiring gradients of the parameterized policy
$\pi_{\theta}$. Therefore we can consider the output of any neural network
(such as an LLM) as a policy, and we can then update this policy so as to
maximize some reward function of interest. This observation is what underlies
RLHF, which we discuss in the next section.

###  2.2 Reinforcement Learning from Human Feedback (RLHF)

RLHF works in two stages: (1) a reward function is learned from the annotator
preference data; and (2) the LLM is treated as an RL policy and finetuned to
maximize the learned reward function. The two stages are usually alternated in
multiple iterations of a closed loop where new annotator feedback is gathered
between each iteration.

More formally, the annotator ranks pairs of model outputs ${\bm{x}}_{A}$ and
${\bm{x}}_{B}$, and the outcome can be represented with a binary variable $y$
that takes the value $y=1$ if ${\bm{x}}_{A}$ is preferred to ${\bm{x}}_{B}$
(noted ${\bm{x}}_{A}\succ{\bm{x}}_{B}$) and $y=0$ otherwise. In the context of
language modeling, the reward function is usually defined as a parameterized
mapping $r_{\phi}({\bm{x}}_{0},{\bm{x}})$ from prompt ${\bm{x}}_{0}$ and
continuation ${\bm{x}}$ to a scalar reward. To simplify the notation and
treatment, we will omit ${\bm{x}}_{0}$ and consider the unconditional
generation case, with the understanding that the discussion extends trivially
to the conditional generation case. The reward function $r_{\phi}({\bm{x}})$
is trained on a dataset $D$ of annotator preferences:

$\phi\leftarrow\min_{\phi}\quad\mathbb{E}_{{\bm{x}}_{A},{\bm{x}}_{B},y\sim
D}\bigg{[}-y\cdot\log
p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B})-(1-y)\cdot\log(1-p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B}))\bigg{]}$
(1)

where

$p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sigma\Big{(}r_{\phi}({\bm{x}}_{A})-r_{\phi}({\bm{x}}_{B})\Big{)}=\frac{e^{r_{\phi}({\bm{x}}_{A})}}{e^{r_{\phi}({\bm{x}}_{A})}+e^{r_{\phi}({\bm{x}}_{B})}}$
(2)

is the probability that ${\bm{x}}_{A}\succ{\bm{x}}_{B}$ under the Bradley-
Terry model (Bradley & Terry, 1952) for pairwise comparisons. In plain
language, the reward function is trained using a binary cross-entropy loss on
the comparison outcome variable $y$, which incentivizes increasing the reward
margin between the preferred and non-preferred outputs. Then, the LLM (noted
$\pi_{\theta}({\bm{x}})$)222The choice of notation stems from the fact that it
is treated as a policy for the purpose of RLHF. is tuned so that

$\theta\leftarrow\max_{\theta}\quad\mathbb{E}_{\pi_{\theta}({\bm{x}})}\bigg{[}r_{\phi}({\bm{x}})\bigg{]}.$
(3)

Using the well-known identity for policy gradient in RL, this results in a
gradient of the form

$\nabla_{\theta}\mathbb{E}_{\pi_{\theta}({\bm{x}})}\bigg{[}r_{\phi}({\bm{x}})\bigg{]}=\mathbb{E}_{\pi_{\theta}({\bm{x}})}\bigg{[}r_{\phi}({\bm{x}})\nabla_{\theta}\log\pi_{\theta}({\bm{x}})\bigg{]}.$
(4)

In order to prevent “catastrophic forgetting” in the model, it is common to
combine the learned reward with a KL-divergence term between the finetuned LLM
and the pretrained LLM (noted $\pi_{\textnormal{pre}}$)—a practice known as
KL-control—yielding a gradient of the form

$\mathbb{E}_{\pi_{\theta}({\bm{x}})}\bigg{[}\left(r_{\phi}({\bm{x}})-\beta\log\frac{\pi_{\theta}({\bm{x}})}{\pi_{\textnormal{pre}}({\bm{x}})}\right)\nabla_{\theta}\log\pi_{\theta}({\bm{x}})\bigg{]}.$
(5)

Such a constraint is usually handled using policy gradient methods such as
proximal policy optimization (PPO; Schulman et al., 2017).

##  3 Related work

Our work is related to numerous previous works investigating alternatives to
RL to learn from pairwise human preferences.

Liu et al. (2023a) move away from RL and propose a technique called Chain of
Hindsight which turns annotator-ranked pairs of model outputs and their
associated contexts into training sequences of the form “{Context} {Positive
prefix}: {Desirable output} {Negative prefix}: {Undesirable output}”. The
model is finetuned into these chain-of-hindsight sequences and then
conditioned at inference time by appending “{Positive prefix}” to the
annotator prompt.

Korbak et al. (2022) examine the relationship between reward maximization
approaches like RLHF (with or without KL-control) and distribution matching
approaches like distributional policy gradients (DPG). They show that RLHF
with KL-control can also be thought of as a distribution matching approach, as
it is equivalent to minimizing the reverse KL-divergence between
$\pi_{\theta}({\bm{x}})$ and an energy-based model of the form
$p_{z}({\bm{x}})\propto\pi_{\textnormal{pre}}({\bm{x}})e^{r({\bm{x}})/\beta}$.
Our work complements their analysis by characterizing what the optimal reward
function captures about the annotator (their implicit preference distribution)
under various hypothesized generative processes for pairwise preferences.

Yuan et al. (2023) align the model with pairwise human preferences through a
combination of a ranking loss on the log-likelihood margin between preferred
and non-preferred model outputs and a negative log-likelihood loss on the
preferred model output. Zhao et al. (2023) optimize for pairwise preference
alignment using a hinge loss on the log-likelihood margin between preferred
and non-preferred model outputs and a regularization term towards a model
finetuned with supervision. Liu et al. (2023b) extend the loss introduced by
Zhao et al. (2023) by normalizing the log-likelihood margin using
$\pi_{\textnormal{pre}}({\bm{x}})$. However, their primary contribution is a
statistical rejection sampling algorithm called RSO that draws samples from
the optimal policy using samples from $\pi_{\textnormal{pre}}({\bm{x}})$ and a
learned reward function.

Rafailov et al. (2023) show an equivalence between RLHF with KL-control and
binary classification of pairwise comparison outcomes when the reward function
used to make a prediction (Equation 2) is defined as
$r({\bm{x}})=\beta(\log\pi_{\theta}({\bm{x}})-\log\pi_{\textnormal{pre}}({\bm{x}}))$
(called DPO, for direct preference optimization). Azar et al. (2023) frame DPO
as a special case of a broader family of approaches called
$\Psi\textnormal{PO}$ where the maximized quantity is a function $\Psi$ of the
preference probability defined by the Bradley-Terry model. They introduce
another special case called identity preference optimization (IPO) which is
optimized via a quadratic loss instead of a sigmoid binary cross-entropy loss
and is meant to make KL-regularization more effective in the face of
deterministic annotator preferences. Whereas Azar et al. (2023) consider
various transformations of the preference probability for the optimal reward
under the Bradley-Terry model, we systematically use the logit function on the
preference probability and instead examine various transformations of
$\log\pi_{\theta}({\bm{x}})$ to obtain the preference probability itself.

##  4 A probabilistic interpretation of learning from pairwise human
preferences

We now consider a probabilistic interpretation of LHF in the case where the
training signal originates from pairwise preferences. First, we need to
consider the generative process for the preferences and make explicit
assumptions about it.

###  4.1 Reward learning as density estimation under the Luce choice rule

Let $p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})$ be an annotator’s preference
probability whose functional form we will make more explicit shortly and
encapsulates our assumptions on the generative process for pairwise human
preferences. Let $q({\bm{x}}_{A},{\bm{x}}_{B})$ be a proposal distribution
over pairs ${\bm{x}}_{A}$ and ${\bm{x}}_{B}$ to be ranked by the annotator.
One common choice of proposal is to draw independent sample pairs from
$\pi_{\textnormal{pre}}({\bm{x}})$. Using the two distributions we can
decompose the true distribution $D$ of preference data used in the expectation
in Equation 1 into

$D({\bm{x}}_{A},{\bm{x}}_{B},y)=q({\bm{x}}_{A},{\bm{x}}_{B})p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B}),$
(6)

and we can rewrite Equation 1 as

$\phi\leftarrow\min_{\phi}\quad\mathbb{E}_{q({\bm{x}}_{A},{\bm{x}}_{B})}\left[\mathbb{E}_{p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})}\bigg{[}-y\cdot\log
p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B})-(1-y)\cdot\log(1-p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B}))\bigg{]}\right],$
(7)

which is equivalent333Recall that the entropy of $p^{*}(\cdot)$ does not
depend on $\phi$ to

$\phi\leftarrow\min_{\phi}\quad\mathbb{E}_{{\bm{x}}_{A},{\bm{x}}_{B}\sim
q({\bm{x}}_{A},{\bm{x}}_{B})}\bigg{[}D_{\mathrm{KL}}\big{(}p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})\mid\mid
p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B})\big{)}\bigg{]}$ (8)

and is globally minimized when the KL-divergence is zero for all possible
${\bm{x}}_{A}$ and ${\bm{x}}_{B}$ pairs.

The correct assumption to make about the generative process for pairwise human
preferences is essentially an empirical question, but a reasonable hypothesis
is a model of decision processes known as the Luce choice rule (or ratio rule;
Shepard, 1957; Luce et al., 1963). As Sanborn & Griffiths (2007) and Peterson
et al. (2018) explain, a common assumption about the behavior of rational
Bayesian subjects is that they make choices according to some implicit
preference distribution $p^{*}({\bm{x}})$ over outcomes: for any two outcomes
${\bm{x}}_{A}$ and ${\bm{x}}_{B}$, they are assumed to prefer ${\bm{x}}_{A}$
over ${\bm{x}}_{B}$ with probability

$\textnormal{Prob}({\bm{x}}_{A}\succ{\bm{x}}_{B})=\frac{p^{*}({\bm{x}}_{A})}{p^{*}({\bm{x}}_{A})+p^{*}({\bm{x}}_{B})}.$
(9)

This model is also known as the Bradley-Terry model (Bradley & Terry, 1952).
In fact, Sanborn & Griffiths (2007) and Peterson et al. (2018) use this very
assumption to place human subjects at the core of a Monte Carlo Markov Chain
(MCMC) by having them perform the accept/reject step and ultimately drawing
samples from their implicit preference distribution $p^{*}({\bm{x}})$ over
outcomes.

Equation 9 and Equation 2 share the same functional form for a reason: if we
assume the generative process for pairwise human preferences follows the Luce
choice rule, then predicting pairwise comparison outcomes using the same
generative process results in a well-specified model of annotator behavior.

######  Theorem 1.

Let $p^{*}({\bm{x}})$ be a probability distribution with support
${\mathcal{S}}$, and let $q({\bm{x}}_{A},{\bm{x}}_{B})$ be a joint probability
distribution with support ${\mathcal{S}}\times{\mathcal{S}}$. Assume
$q({\bm{x}}_{A},{\bm{x}}_{B})>0$ for all
${\bm{x}}_{A},{\bm{x}}_{B}\in{\mathcal{S}}\times{\mathcal{S}}$. Let
$p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B})$ and
$p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})$ be defined according to Equations 2 and
9, respectively. The loss function

$\mathbb{E}_{{\bm{x}}_{A},{\bm{x}}_{B}\sim
q({\bm{x}}_{A},{\bm{x}}_{B})}\bigg{[}D_{\mathrm{KL}}\big{(}p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})\mid\mid
p_{\phi}(y;{\bm{x}}_{A},{\bm{x}}_{B})\big{)}\bigg{]}$

is globally minimized when

$e^{r_{\phi}({\bm{x}})}\propto
p^{*}({\bm{x}}),\quad\forall{\bm{x}}\in{\mathcal{S}}.$

A proof is provided in Appendix A.1. We can therefore reinterpret the training
of a reward function on pairwise human preferences as the following
probabilistic procedure:

  1. 1.

We assume the generative process for pairwise human preferences follows the
Luce choice rule (Equation 9) and consider annotator preferences as samples
from this process;

  2. 2.

We predict comparison outcomes with the learnable reward function using the
same generative process (Equation 2);

  3. 3.

We use a maximum-likelihood criterion to optimize the reward-derived
preference probability (Equation 8) and backpropagate the binary cross-entropy
loss through Equation 2 and into the reward function itself; and

  4. 4.

Provided the Luce choice rule assumption made in step 1 holds, the resulting
optimal reward function is then equal to the annotator’s implicit log-
preference distribution (up to some additive constant).

To demonstrate this, consider a univariate implicit preference distribution

$p^{*}(x)=\frac{2}{5}\cdot
N_{\textrm{truncated}}(\mu=-2.5,\sigma=0.25)+\frac{3}{5}\cdot
N_{\textrm{truncated}}(\mu=2.5,\sigma=1.0)$ (10)

Figure 1:  Univariate toy experiment hyperparameters. Hyperparameter |  Value
---|---
Architecture |  MLP
Hidden layers |  4 layers of width 64
Activation |  $tanh$
Optimizer |  Adam (Kingma & Ba, 2015)
Optimization steps |  $8192$
Learning rate |  $5\text{\times}{10}^{-4}$
Learning rate schedule |  Cosine decay to 0.0 over $8192$ steps

defined over the $[-10,10]$ interval (Figure 2, dashed blue). We simulate
learning from pairwise preferences by drawing $2^{15}$ observation pairs
${\bm{x}}_{A}$ and ${\bm{x}}_{B}$ uniformly at random in
$[-10,10]\times[-10,10]$ and drawing corresponding comparison outcomes $y$
using Equation 9. We then train a reward function in the form of a four-layer
MLP (refer to Figure 1 for details). Finally, we approximate the normalizing
constant for the distribution
$p_{\phi}({\bm{x}})=e^{r_{\phi}({\bm{x}})}/Z(\phi)$ and visualize
$p_{\phi}({\bm{x}})$ (Figure 2, solid green). We can clearly see that in the
large-data regime (in terms of number of ranked pairs), the reward function
learns to model the implicit preference distribution (see also Figure 8 in the
Appendix).

In other words, if “your language model is secretly a reward model” (Rafailov
et al., 2023), it is perhaps equally fair to say that your reward model
secretly learns the annotator’s implicit preference distribution.444Provided
the annotator’s preferences follow the Luce choice rule. Ultimately, the
reward function is a means to the end of aligning a generative model (for
example, an LLM “policy”) with human preferences, but the above discussion
shows that under the Luce choice rule assumption the optimal reward function
actually learns the implicit preference distribution $p^{*}({\bm{x}})$.

###  4.2 Specifying policies as normalized preference distributions

Figure 2:  Training a reward model on comparison outcomes stemming from a
synthetic implicit preference distribution (Equation 10; dashed blue) recovers
the implicit distribution (solid green).

How, then, should we use our model of human preferences to construct a
generative model that aligns with those preferences? We could use an RL
algorithm to search for a policy that optimizes preferences, but Theorem 1
suggests that this may not be necessary: the “reward model” can already be
interpreted as a probabilistic model of preferred outputs, since it estimates
the implicit preference distribution $p^{*}({\bm{x}})$ that is assumed to
govern the annotator’s behavior. This means that we could in principle simply
use our approximation of the annotator’s implicit preference distribution as a
policy. The only remaining difficulty is that Theorem 1 only specifies the
preference distribution up to a constant.

To address this, suppose we simply define the “reward” $r_{\theta}({\bm{x}})$
as the log-density of a parameterized generative model, i.e.
$r_{\theta}({\bm{x}})=\log\pi_{\theta}({\bm{x}})$.555We purposefully switch
from $r_{\phi}({\bm{x}})$ to $r_{\theta}({\bm{x}})$ to highlight the fact that
the reward function now depends on the LLM’s parameters $\theta$. Since
$\pi_{\theta}({\bm{x}})$ is a normalized probability distribution, attaining
the global minimum of the loss function directly results in
$\pi_{\theta}({\bm{x}})=p^{*}({\bm{x}})$. This removes the need for a separate
reward model, and the original “reward” is now just implicitly encoded in our
learned approximation of the implicit preference distribution.

This idea of bypassing the reward function and directly optimizing the LLM on
pairwise human feedback is explored in recent works like DPO (Rafailov et al.,
2023), IPO (Azar et al., 2023), RRHF (Yuan et al., 2023), SLiC-HF (Zhao et
al., 2023), and RSO (Liu et al., 2023b), which we will discuss in more detail
shortly. We also note the probabilistic procedure described above can be
thought of as an inductive bias in and of itself: if one believes that the
implicit preference distribution is the ideal generative model, then
expressing the reward as the LLM’s log-likelihood is the correct thing to do.
In practice, however, stronger inductive biases are used to regularize the
tuned LLM towards its pretrained parameterization.

###  4.3 Expanding to a broader family of generative processes for pairwise
preferences

Our results show that—assuming the generative process for the annotator’s
pairwise preferences follows the Luce choice rule—fitting a model to those
preferences using the parameterization in Equation 2 recovers the annotator’s
implicit preference distribution $p^{*}({\bm{x}})$. This recasts the standard
reward modeling procedure as a density estimation procedure. However, it is
not necessarily obvious that annotators exactly follow the Luce choice rule,
nor that the implicit preference distribution $p^{*}({\bm{x}})$ we recover is
itself a good “policy”.

Luckily, our density estimation results can be extended to hold for generative
processes beyond the Luce choice rule. We focus on the following family of
equations for describing generative processes which we name preference
behavior distribution equations (PBDEs), noted $\Omega_{p}({\bm{x}})$:

$p(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sigma\Big{(}\Omega_{p}({\bm{x}}_{A})-\Omega_{p}({\bm{x}}_{B})\Big{)},\quad\Omega_{p}({\bm{x}})=f({\bm{x}})\cdot\log
p({\bm{x}})+g(x),\quad f({\bm{x}})>0\quad\forall{\bm{x}}.$ (11)

The Luce choice rule is itself a special case for which $f({\bm{x}})=1$ and
$g({\bm{x}})=0$, and we will cover two other special cases later on.

######  Theorem 2.

Let the generative processes for pairwise preferences for the annotator and
the model be governed by the same PBDE, i.e.,

$\Omega_{p^{*}}({\bm{x}})=f({\bm{x}})\cdot\log
p^{*}({\bm{x}})+g({\bm{x}})\quad\textnormal{and}\quad\Omega_{\pi_{\theta}}({\bm{x}})=f({\bm{x}})\cdot\log\pi_{\theta}({\bm{x}})+g({\bm{x}}).$

Then the global minimizer of

$\mathbb{E}_{q({\bm{x}}_{A},{\bm{x}}_{B})}\left[D_{\mathrm{KL}}\Big{(}p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})\mid\mid
p_{\theta}(y;{\bm{x}}_{A},{\bm{x}}_{B})\Big{)}\right]$

is

$\pi_{\theta}({\bm{x}})=p^{*}({\bm{x}}),\quad\forall{\bm{x}}.$

A proof is provided in Appendix A.1. Theorem 2 provides a global optimality
result for a broad family of generative processes for pairwise preferences
when the annotator and model share the same process, but it can sometimes be
useful to define a different generative process for the model than the one
assumed for the annotator—for instance to influence the properties of the
globally optimal $\pi_{\theta}({\bm{x}})$.

We now show how Direct Preference Optimization (DPO; Rafailov et al., 2023)
can be recast as an instance of this general density estimation procedure. DPO
bypasses the policy optimization step in RLHF and instead directly expresses
the reward modeling step in terms of $\pi_{\theta}({\bm{x}})$. The authors
point out that the KL-controlled policy optimization stage

$\theta\leftarrow\max_{\theta}\quad\mathbb{E}_{\pi_{\theta}({\bm{x}})}\bigg{[}r_{\phi}({\bm{x}})\bigg{]}-\beta
D_{\mathrm{KL}}\big{(}\pi_{\theta}({\bm{x}})\mid\mid\pi_{\textnormal{pre}}({\bm{x}})\big{)}$
(12)

has a closed-form solution

$\pi_{\theta}({\bm{x}})\propto\pi_{\textnormal{pre}}({\bm{x}})\exp\left(\frac{1}{\beta}r_{\phi}({\bm{x}})\right).$
(13)

Solving for $r_{\phi}({\bm{x}})$ and performing the substitution in Equation
1, they obtain a loss function for the reward which expresses the probability
that ${\bm{x}}_{A}\succ{\bm{x}}_{B}$ as

$p_{\theta}(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sigma\left(\beta\log\frac{\pi_{\theta}({\bm{x}}_{A})}{\pi_{\textnormal{pre}}({\bm{x}}_{A})}-\beta\log\frac{\pi_{\theta}({\bm{x}}_{B})}{\pi_{\textnormal{pre}}({\bm{x}}_{B})}\right).$
(14)

This equation effectively defines a generative process for pairwise
preferences for $\pi_{\theta}({\bm{x}})$ that is guided by a PBDE with
$f({\bm{x}})=\beta$ and
$g({\bm{x}})=-\beta\log\pi_{\textnormal{pre}}({\bm{x}})$, i.e.,

$\Omega_{\pi_{\theta}}({\bm{x}})=\beta\log\pi_{\theta}({\bm{x}})-\beta\log\pi_{\textnormal{pre}}({\bm{x}}).$
(15)

Figure 3:  Theorem 2 also holds for DPO if the annotator and model share the
same generative process (Equation 14).

If we assume the preference generative process for the annotator is governed
by the same PBDE—which could be the case if annotator preferences express only
_relative_ improvements that correct for the base fluency of the outputs—then
Theorem 2 holds and the model recovers
$\pi_{\theta}({\bm{x}})=p^{*}({\bm{x}})$ (Figure 3).

Note that different assumptions about the preference generative processes can
lead to a different relationship between $p^{*}$ and $\pi_{\theta}$, even if
those assumptions produce the same observed preference distribution
$p(y;{\bm{x}}_{A},{\bm{x}}_{B})$. For instance, if we instead assume the
annotator behaves according to the Luce choice rule, but still minimize the
DPO objective, the globally optimal $\pi_{\theta}({\bm{x}})$ would become

$\pi_{\theta}({\bm{x}})\propto\pi_{\textnormal{pre}}({\bm{x}})\cdot
p^{*}({\bm{x}})^{\frac{1}{\beta}}.$ (16)

A proof is provided in Appendix A.1. Under this assumption, DPO converges to a
product of experts between a pretrained “prior” policy
$\pi_{\textnormal{pre}}({\bm{x}})$ and the annotator’s temperature-smoothed
implicit preference distribution $p^{*}({\bm{x}})$. This means that whether or
not $\pi_{\theta}({\bm{x}})$ converges to $p^{*}({\bm{x}})$ depends as much on
our assumptions about the preference generating process as it does on the
algorithm we use.

We illustrate this by finetuning a pretrained model
$\pi_{\textnormal{pre}}({\bm{x}})$ with DPO on pairwise preferences acquired
from a synthetic annotator whose generative process for pairwise preferences
obeys the Luce choice rule with an implicit preference distribution defined in
Equation 10 (Figure 4, dashed blue). The model
$\pi_{\textnormal{pre}}({\bm{x}})$ (Figure 4, dotted orange) is an energy-
based model with an energy function expressed as a four-layer MLP (we reuse
hyperparameters detailed in Figure 1):

$\pi_{\textnormal{pre}}({\bm{x}})=\frac{1}{Z_{\textnormal{pre}}}e^{-\textnormal{MLP}_{\textnormal{pre}}({\bm{x}})}.$
(17)

Figure 4:  Under the Luce choice rule assumption for the annotator’s
generative process on pairwise preferences, using DPO to tune a generative
model $\pi_{\textnormal{pre}}({\bm{x}})$ (dotted orange) on preferences
derived from the implicit preference distribution (dashed blue) results in a
mixture of experts model between the initial model $\pi_{\textnormal{pre}}$
and the temperature-smoothed implicit preference distribution, as demonstrated
by the agreement between the empirical (solid green) and theoretical (dash-
dotted red) curves.

We initialize it by regressing on the log-likelihood of
$N_{\textrm{truncated}}(\mu=0.0,\sigma=5.0)$ defined over $[-10,10]$, and we
sweep over $\beta\in\\{1,4,16\\}$ to adapt it with DPO. The adapted model
(Figure 4, solid green) behaves just as predicted by Equation 16 (Figure 4,
dash-dotted orange).

We point out that Equation 16 is almost a weighted geometric mean but is
missing a $1-\frac{1}{\beta}$ exponent on $\pi_{\textnormal{pre}}({\bm{x}})$.
If we were instead to define $f({\bm{x}})=\alpha^{-1}$ and
$g({\bm{x}})=(1-\alpha^{-1})\log\pi_{\textrm{pre}}({\bm{x}})$, the PBDE for
the model would result in a globally optimal $\pi_{\theta}({\bm{x}})$ (again,
assuming the annotator behaves according to the Luce choice rule) of the form

$\pi_{\theta}({\bm{x}})\propto\pi_{\textnormal{pre}}({\bm{x}})^{1-\alpha}\cdot
p^{*}({\bm{x}})^{\alpha}.$ (18)

Refer to Appendix A.1 for a proof and Figure 9 in the Appendix for an
empirical learning behavior visualization in that case.

###  4.4 Alternatives to the binary cross-entropy loss

Other recent works adapt the model directly on pairwise preferences but move
beyond a binary cross-entropy loss formulation. As such, Theorem 2 is not
applicable, and more work is needed to characterize their learning properties
when viewed through the density estimation lens.

Rank Responses to align Human Feedback (RRHF; Yuan et al., 2023) uses a
ranking loss instead of the binary cross-entropy loss and considers annotator
rankings of $k$ observations ${\bm{x}}_{1}$ through ${\bm{x}}_{k}$:

$\mathcal{L}_{\textnormal{rank}}(\theta)=\mathbb{E}_{D({\bm{x}}_{1},\ldots,{\bm{x}}_{k})}\left[\sum_{{\bm{x}}_{i}\succ{\bm{x}}_{j}}-\left(\frac{1}{|{\bm{x}}_{i}|}\log\pi_{\theta}({\bm{x}}_{i})-\frac{1}{|{\bm{x}}_{j}|}\log\pi_{\theta}({\bm{x}}_{j})\right)^{+}\right],$
(19)

where $|{\bm{x}}|$ is the length of the token sequence ${\bm{x}}$. An
additional loss component in the form of negative log-likelihood on the
highest-ranked ${\bm{x}}_{i}$ is also used.

Sequence Likelihood Calibration from Human Feedback (SLiC-HF; Zhao et al.,
2023) considers (among other alternatives) a SLiC-HF-direct variant which
optimizes $\pi_{\theta}({\bm{x}})$ directly on pairwise human preferences. The
loss function is defined as

$\mathcal{L}(\theta)=\mathbb{E}_{D({\bm{x}}_{A},{\bm{x}}_{B},y)}\left[\left(\delta-\left(y\cdot\log\frac{\pi_{\theta}({\bm{x}}_{A})}{\pi_{\theta}({\bm{x}}_{B})}+(1-y)\cdot\log\frac{\pi_{\theta}({\bm{x}}_{B})}{\pi_{\theta}({\bm{x}}_{A})}\right)\right)^{+}\right]-\lambda\cdot\mathbb{E}_{D_{\textnormal{SFT}}}({\bm{x}})\left[\log\pi_{\theta}({\bm{x}})\right].$
(20)

The first expectation represents a hinge loss over the difference in log-
likelihoods between ${\bm{x}}_{A}$ and ${\bm{x}}_{B}$: it encourages the
difference to be positively large in favor of the higher-ranked observation,
up to a margin $\delta$. The second expectation regularizes
$\pi_{\theta}({\bm{x}})$ towards a supervised finetuned (SFT) model trained on
a dataset $D_{\textnormal{SFT}}$ of target domain examples, either using the
supervised finetuning data directly, drawing samples from the supervised
finetuned model, or maximizing the likelihood of preferred observations.

Ignoring the regularization term, the SLiC-HF-direct loss function resembles
Equation 1 but uses a hinge loss on the difference in log-likelihoods between
${\bm{x}}_{A}$ and ${\bm{x}}_{B}$ rather than the sigmoid binary cross-
entropy. We examine the properties of SLiC-HF-direct using our synthetic
density estimation problem in Appendix A.2.

Statistical Rejection Sampling Optimization (RSO; Liu et al., 2023b) samples
from the optimal policy using samples from the pretrained model
$\pi_{\textnormal{pre}}({\bm{x}})$, but Liu et al. (2023b) also investigate an
extension to SLiC-FH which normalizes the model’s log-likelihoods using the
pretrained model $\pi_{\textnormal{pre}}({\bm{x}})$:

$\mathcal{L}(\theta)=\mathbb{E}_{D({\bm{x}}_{A},{\bm{x}}_{B},y)}\left[\left(\delta-\left(y\cdot\log\frac{\pi_{\theta}({\bm{x}}_{A})\pi_{\textnormal{pre}}({\bm{x}}_{B})}{\pi_{\theta}({\bm{x}}_{B})\pi_{\textnormal{pre}}({\bm{x}}_{A})}+(1-y)\cdot\log\frac{\pi_{\theta}({\bm{x}}_{B})\pi_{\textnormal{pre}}({\bm{x}}_{A})}{\pi_{\theta}({\bm{x}}_{A})\pi_{\textnormal{pre}}({\bm{x}}_{B})}\right)\right)^{+}\right].$
(21)

$\Psi$PO with identity mapping (IPO; Azar et al., 2023) derives a learning
rule for a KL-controlled preference probability maximization problem

$\theta\leftarrow\max_{\theta}\quad\mathbb{E}_{\begin{subarray}{c}{\bm{x}}_{A}\sim\pi_{\theta}({\bm{x}})\\\
{\bm{x}}_{B}\sim\mu({\bm{x}})\end{subarray}}\bigg{[}p^{*}(y;{\bm{x}}_{A},{\bm{x}}_{B})\bigg{]}-\tau
D_{\mathrm{KL}}\left(\pi_{\theta}({\bm{x}})\mid\mid\pi_{\textnormal{pre}}({\bm{x}})\right)$
(22)

which has the form

$\mathcal{L}(\theta)=\mathbb{E}_{D({\bm{x}}_{A},{\bm{x}}_{B},y)}\left[\bigg{(}y\cdot\log\frac{\pi_{\theta}({\bm{x}}_{A})\pi_{\textnormal{pre}}({\bm{x}}_{B})}{\pi_{\theta}({\bm{x}}_{B})\pi_{\textnormal{pre}}({\bm{x}}_{A})}+(1-y)\cdot\log\frac{\pi_{\theta}({\bm{x}}_{B})\pi_{\textnormal{pre}}({\bm{x}}_{A})}{\pi_{\theta}({\bm{x}}_{A})\pi_{\textnormal{pre}}({\bm{x}}_{B})}-\frac{\tau^{-1}}{2}\bigg{)}^{2}\right].$
(23)

Practically speaking, the loss is similar to the one investigated in SLiC-FH
(and RSO) but is constructed using a two-sided quadratic expression rather
than a one-sided hinge expression. We examine the properties of IPO using our
synthetic density estimation problem in Appendix A.3.

##  5 Annotator misspecification

In Section 4.3 we touched on cases where the mismatch generative processes for
pairwise preferences between the annotator and the model was used to our
advantage to induce specific properties in the globally optimal model. In this
section, we show that an unwanted mismatch in generative processes (which we
call annotator misspecification) can lead to badly tuned policies.

###  5.1 Annotator misspecification in a toy setting

Figure 5:  $\textnormal{Prob}({\bm{x}}_{A}\succ{\bm{x}}_{B})$ for single-
annotator (Equation 24) and multi-annotator (Equation 25) behavior (left two
plots) as well as models adapted with a misspecified and well-specified
annotator behavior model (right two plots). The large regions of near-0.5
probability in the multi-annotator case (second plot) are caused by strong but
opposing preferences for the two annotators, which cannot be captured by a
single-annotator reward model (third plot).

Consider a scenario where two annotators—Annotator 1 and Annotator 2—are
providing feedback on coffees served at various temperatures. Annotator 1
happens to really like iced coffee, while Annotator 2 prefers hot coffee. In
the experiment, an annotator is selected at random and is served a pair of
coffees at temperatures sampled independently at random. The annotator
expresses a preference which is added to the feedback dataset alongside the
two coffee temperatures, but without noting the annotator’s identity.

We simulate this by modifying the way in which feedback is acquired from
Equation 10: instead of having a single annotator following the Luce choice
rule with an implicit preference distribution which covers two modes, we
assume the distribution of _preferences_ is itself a mixture of two preference
distributions, each of which is obtained from a distinct annotator following
the Luce choice rule with an implicit preference distribution covering one of
the two modes. Mathematically speaking, instead of drawing preferences from

$\textnormal{Prob}({\bm{x}}_{A}\succ{\bm{x}}_{B})=\frac{p^{*}({\bm{x}}_{A})}{p^{*}({\bm{x}}_{A})+p^{*}({\bm{x}}_{B})}=\frac{\sum_{i=1}^{2}w_{i}\cdot
p_{i}^{*}({\bm{x}}_{A})}{\sum_{i=1}^{2}w_{i}\cdot
p_{i}^{*}({\bm{x}}_{A})+\sum_{i=1}^{2}w_{i}\cdot p_{i}^{*}({\bm{x}}_{B})},$
(24)

(where $w_{1}=2/5$,
$p^{*}_{1}=N_{\textnormal{truncated}}(\mu=-2.5,\sigma=0.25)$, $w_{2}=3/5$, and
$p^{*}_{2}=N_{\textnormal{truncated}}(\mu=2.5,\sigma=1.0)$), we now sample
preferences from a mixture of two separate preference distributions

$\textnormal{Prob}({\bm{x}}_{A}\succ{\bm{x}}_{B})=\sum_{i=1}^{2}w_{i}\cdot\textnormal{Prob}_{i}({\bm{x}}_{A}\succ{\bm{x}}_{B})=\sum_{i=1}^{2}w_{i}\cdot\frac{p_{i}^{*}({\bm{x}}_{A})}{p_{i}^{*}({\bm{x}}_{A})+p_{i}^{*}({\bm{x}}_{B})}$
(25)

A visual way to reason about the differences between Equations 24 and 25 is to
plot $\textnormal{Prob}({\bm{x}}_{A}\succ{\bm{x}}_{B})$ in both cases (Figure
5, left half). A cursory examination shows that the two result in very
different comparison outcome statistics.

Figure 6:  If preferences are aggregated across two annotators (dashed blue,
dot-dashed pink) but the model is adapted under a single-annotator assumption,
it fails to capture either annotator’s PDF (solid green).

If we adapt $\pi_{\textnormal{pre}}({\bm{x}})$ on comparison outcomes sampled
from Equation 25 using a Luce choice rule generative process for the model, we
are in a annotator misspecification setting and the adapted model (Figure 6,
solid green) fails to capture regions of high density under either of the two
annotators’ PDFs (Figure 6, dashed blue and dot-dashed pink). In other words,
we learn to serve lukewarm coffee to both annotators despite neither of them
enjoying it. This is also reflected in the comparison outcome statistics
captured by the model (Figure 5, third plot). Addressing annotator
misspecification is an open problem, especially in the absence of annotator
IDs. In Appendix A.4 we present results where we succeed in disentangling the
two annotators’ preferences without annotator IDs (also refer to Figure 5’s
rightmost plot), but applying the same approach proved unsuccesful in the
larger-scale setting which we will discuss shortly.

This failure mode of learning from pairwise preferences is particularly
relevant in a context where LLMs are aligned with human preferences by
crowdsourcing the acquisition of pairwise preferences across multiple human
annotators. It could appear at first glance that the resulting preferences are
inconsistent with one another when in fact annotator identity is a confounding
factor that needs to be taken into account. For instance, consider a setting
where a group of human annotators are asked to rank pairs of model outputs in
terms of fluency. Given equally fluent outputs, some annotators may be biased
towards short sentences while others may be biased towards longer sentences.

###  5.2 Annotator misspecification in a language modeling setting

We now move to the One Billion Words Benchmark (LM1B; Chelba et al., 2013) to
illustrate the consequences of annotator misspecification in a language
modeling setting. Starting from Flax’s LM1B example
code666https://github.com/google/flax/tree/main/examples/lm1b we train three
separate instances of a Transformer-based architecture on distinct subsets of
LM1B according to sequence length: less than 50 tokens (whole), between 1 and
20 tokens inclusively (short), and between 30 and 50 tokens inclusively
(long). We treat the whole model as the pretrained LM to adapt via pairwise
preference feedback and use the short and long models to form synthetic
annotators. Figure 7 (left) shows sequence length distributions for each
model.

The first observation we make is that the Luce choice rule is not a realistic
generative process for pairwise preferences in the case of autoregressive
models applied to language modeling tasks due to “length bias” in likelihood
values. Since the likelihood assigned by the model to a token sequence
${\bm{x}}$ is decomposed into a product of per-token likelihoods in
$[0,1]^{|{\bm{x}}|}$, longer sequences naturally have lower likelihood
magnitudes, which means for instance that the long model assigns higher
likelihoods to short token sequences despite having a near-zero probability of
sampling them. As a consequence, if we derive preferences from the long model
using the Luce choice rule it will often end up preferring shorter sequences
to longer ones. This is especially noticeable when comparing sequence length
distributions (Figure 7, left) to the probability given by the Luce choice
rule (Equation 9) that a synthetic annotator with a long implicit preference
distribution will prefer short sequences over long ones (Table 1): that
probability is around $97\%$. We argue that this is unrealistic: an annotator
should not prefer shorter sequences simply because the total number of
possible short sequences is smaller than the total number of possible long
sequences. (In fact, real-world human feedback datasets tend to show a slight
preference towards _longer_ responses (Singhal et al., 2023).)

A more realistic alternative to the Luce choice rule in this context would be
the length-normalized variant

$\textnormal{Prob}({\bm{x}}_{A}\succ{\bm{x}}_{B})=\frac{p^{*}({\bm{x}}_{A})^{\frac{1}{|{\bm{x}}_{A}|}}}{p^{*}({\bm{x}}_{A})^{\frac{1}{|{\bm{x}}_{A}|}}+p^{*}({\bm{x}}_{B})^{\frac{1}{|{\bm{x}}_{B}|}}}.$
(26)

Figure 7: Sentence length distributions. (Left) Histogram of sequence lengths
for sequences sampled from the whole, short, and long model checkpoints.
(Right) Histogram of sequence lengths after adapting the pretrained model
$\pi_{\textnormal{pre}}({\bm{x}})$ (the whole model checkpoint) on two
distinct preference datasets: the “annotator mixture” dataset (Equation 27)
and the “density mixture” dataset (Equation 28).

The above has the advantage that its outcome is no longer biased by the
inherent differences in likelihood magnitudes between sequences of varying
lengths. This is likely why length-normalized log-likelihoods have been used
by Yuan et al. (2023) in RRHF, for instance. When comparing the outcome
probabilities for short, medium, and long sequences (Table 1), Equation 26
better aligns with the intuitions built from examining Figure 7 (left). Since
Equation 26 is a PBDE with $f({\bm{x}})=|{\bm{x}}|^{-1}$ and $g({\bm{x}})=0$,
we know through Theorem 2 that it is possible to recover a globally optimal
$\pi_{\theta}({\bm{x}})=p^{*}({\bm{x}})$ when both the annotator and the model
preferences are sampled according to that generative process.

We now revisit annotator misspecification when pairwise preferences are
acquired from multiple annotators, assuming annotators and the model behave
according to the length-normalized Luce choice rule. We sample $2^{15}$ pairs
from the whole model and compare the outcome of learning on two preference
datasets:

  * •

The “annotator mixture” dataset is built as a uniform mixture of preferences
sampled independently using the short and long model checkpoints. For each
$({\bm{x}}_{A},{\bm{x}}_{B})$ pair, we first sample uniformly at random which
checkpoint will be used as the implicit preference distribution, and we then
sample a preference using the length-normalized Luce choice rule:

$p(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sum_{\theta_{\textnormal{pre}}\in\\{\theta_{\textnormal{short}},\theta_{\textnormal{long}}\\}}\frac{1}{2}\cdot\sigma\bigg{(}\frac{\log\pi_{\theta_{\textnormal{pre}}}({\bm{x}}_{A})}{|{\bm{x}}_{A}|}-\frac{\log\pi_{\theta_{\textnormal{pre}}}({\bm{x}}_{B})}{|{\bm{x}}_{B}|}\bigg{)}.$
(27)

  * •

The “density mixture” dataset is built following the length-normalized Luce
choice rule with an implicit preference distribution that is a uniform mixture
of the distributions computed by the short and long model checkpoints:

$p(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sigma\bigg{(}\frac{\log\pi_{\textnormal{mix}}({\bm{x}}_{A})}{|{\bm{x}}_{A}|}-\frac{\log\pi_{\textnormal{mix}}({\bm{x}}_{B})}{|{\bm{x}}_{B}|}\bigg{)},\quad\pi_{\textnormal{mix}}({\bm{x}})=\frac{\pi_{\theta_{\textnormal{short}}}({\bm{x}})+\pi_{\theta_{\textnormal{long}}}({\bm{x}})}{2}.$
(28)

In other words, the “annotator mixture” dataset contains a mixture of
preferences from a synthetic annotator who likes short sequences and a
synthetic annotator who likes long sequences, whereas the “density mixture”
dataset contains preferences from a single synthetic annotator who likes both
short and long sequences, but not medium-length sequences. Figure 7 (right)
shows sequence length distributions after adapting on either of the two
preference datasets. We notice that the model adapted on the “annotator
mixture” preference dataset generates medium-length sentences much more
frequently than either the short or long model checkpoints, exactly like in
the “cold and hot coffee” toy example. This is a direct result of annotator
misspecification: pairwise preferences are obtained by querying two synthetic
annotators, whereas the learning rule used for adaptation assumes a generative
process for preferences that involves a single annotator. On the other hand,
when learning from preferences acquired from a synthetic annotator whose
implicit preference distribution is a mixture of the short and long model
checkpoints (“density mixture” dataset), the generative processes for pairwise
preferences are the same across annotator and model, and the model succeeds in
learning to output a bimodal distribution of sequence lengths.

Table 1:  Probability of a synthetic annotator preferring some token sequence over another token sequence as a function of their respective lengths. Token sequences are sampled from the pretrained LM (whole) and are binned into small (S, 20 tokens or less), medium (M, between 21 and 29 tokens), and large (L, 30 tokens or more) sets of sequences. $P(X\succ Y)$ represents an average of $P(x_{A}\succ x_{B})$ over all $x_{A}\in X,x_{B}\in Y$. | $P(S\succ M)$ |  | $P(S\succ L)$ |  | $P(M\succ L)$
---|---|---|---|---|---
Synthetic Annotator | Eqn 9 | Eqn 26 |  | Eqn 9 | Eqn 26 |  | Eqn 9 | Eqn 26
whole | $0.909\,376$ | $0.404\,204$ |  | $0.992\,775$ | $0.362\,137$ |  | $0.917\,535$ | $0.455\,379$
short | $0.990\,599$ | $0.953\,972$ |  | $1.000\,000$ | $0.999\,999$ |  | $0.994\,535$ | $0.978\,312$
long | $0.919\,023$ | $0.248\,861$ |  | $0.972\,909$ | $0.053\,297$ |  | $0.787\,296$ | $0.157\,454$

##  6 Discussion and Limitations

In this work we use a simplified, synthetic experimental framework to
empirically verify the theoretical properties of approaches which learn from
pairwise human feedback. Specifically, we

  1. 1.

define a probability distribution over a random variable with finite support
and use it as the implicit preference distribution for a synthetic annotator
following a fixed preference generating process;

  2. 2.

provide the model with access to the implicit preference distribution only
through a fixed number of queries to the synthetic annotator on observation
pairs which are ranked stochastically using a PBDE; and

  3. 3.

adapt the model on the resulting $({\bm{x}}_{A},{\bm{x}}_{B},y)$ triplets.

In this synthetic setting, we have shown that it is indeed possible to
approximately recover the annotator’s implicit preference distribution. We
have also demonstrated how this procedure fails in the presence of annotator
misspecification. However, beyond annotator misspecification, there are a
number of other ways our synthetic density estimation setting differs from
practical applications of learning from pairwise human feedback.

##### Finite data

We do not account for the labor necessary to acquire large amounts of human
preference data. The theoretical results we present hold in the infinite data
limit and the existence of a global minimizer for the loss function does not
guarantee convergence through gradient descent. The univariate toy experiment
(Figure 2) is carried at the larger end of the data regime spectrum given the
dimensionality of the problem and the number of queries made to the synthetic
annotator, and in that setting we achieve convergence.

Practical applications are high-dimensional (images, language tokens, etc.),
and the number of synthetic annotator queries we make in our LM1B experiment
is comparatively small: the size of the vocabulary ($30\,000$) and the maximum
sequence length (50) result in a set of possible sequences that is
considerably larger than the number of synthetic annotator queries ($2^{15}$).
It is interesting to note, then, that our LM1B experiment (and more broadly
contemporary works on adapting LLMs to pairwise human feedback) are reasonably
effective at steering models from pairwise feedback alone. How this is
possible despite the combinatorial nature of language is an interesting
question for future work. Plausible hypotheses include:

  * •

Natural language conforms to the manifold hypothesis (Cayton, 2005; Narayanan
& Mitter, 2010), and the pretrained model $\pi_{\textnormal{pre}}({\bm{x}})$
is a particularly good proposal distribution to explore the “manifold” of
natural language. The result is that characterizing a distribution over
natural language outputs requires far fewer pairwise comparisons than the
curse of dimensionality would suggest.

  * •

Density estimation using pairwise human feedback is really hard in the data
regimes in which it is deployed, but it is not necessary to fully capture the
implicit preference distribution in order to steer the generative model in the
right direction. Provided careful regularization is applied (which, to our
knowledge, is a requirement in practice), preference datasets are informative
enough.

##### Stationarity

Our experiments assume all observation pairs are sampled at once before
adapting the model. In practice some approaches (like RLHF) use a
nonstationary proposal distribution and alternate between querying for human
preferences using sample pairs drawn from the model and adapting the model on
human preferences. This raises interesting questions on whether stationarity
is preferable, and whether active learning could play a role in reducing the
amount of annotator queries necessary for good performance.

##  7 Conclusion

In this work we present a probabilistic modeling as an alternative to
reinforcement learning for learning from pairwise human preferences. For a
broad family of generative processes guided by what we call preference
behavior distribution equations (PBDEs) we show through theoretical and
empirical results that, so long as the human annotator and the model share the
same PBDE for generating pairwise preferences, the global optimum of the
resulting optimization problem is the annotator’s implicit preference
distribution over observations. We argue that being explicit about the assumed
generative process for an annotator’s preferences is important both to
manipulate the properties of the globally optimal model that learns on those
preferences and to avoid annotator misspecification failure modes. Finally, we
illustrate the practical implications of annotator misspecification in a
language modeling problem where the usual single-annotator assumption fails to
adequately capture two annotators’ conflicting preferences.

#### Broader Impact Statement

Our work investigates learning from pairwise human preferences from a
theoretical standpoint, but it has multiple practical implications. Density
estimation from pairwise feedback is a very difficult problem—arguably much
more so than density estimation from target distribution samples—and most
practical applications operate in the low-data regime. This has the potential
to exacerbate biases in the pairwise preference data and makes it all the more
important to ensure that annotators are representative of a diverse population
of end-users. The insights we provide on annotator misspecification also
highlight the importance of reasoning explicitly about the generative process
for pairwise preferences, as a misspecified process has the potential to steer
the model towards unwanted behavior or erase underrepresented perspectives
altogether. We believe future research on both these aspects of learning from
pairwise human preferences is important in order to achieve inclusive
alignment with human judgement.

#### Author Contributions

Vincent Dumoulin wrote a first draft of the manuscript which co-authors then
added to and helped edit. He implemented and carried out the experiments
(starting from Yann’s code and model checkpoints in the case of LM1B). He
derived the proofs for Theorem 1 (with input from Laurent Dinh) and Theorem 2
(with input from Daniel) and developed the length-normalized Luce choice rule
solution to the “length bias” problem jointly with Yann.

Daniel D. Johnson identified the problem of annotator misspecification with
two annotators and proposed the “family of generative processes” framing to
unify the Luce choice rule, DPO, and the length-normalized Luce choice rule.
He contributed to writing and editing, and helped with some of the proofs. He
explored alternative variants of the LM1B experiment which ended up not making
it into the paper.

Pablo Samuel Castro provided regular input in the research discussions and
helped write and edit the manuscript.

Hugo Larochelle proposed the research direction that eventually led to the
paper’s narrative of framing learning from pairwise human preferences as a
probabilistic modeling problem. He advised on the project and helped revise
the manuscript.

Yann Dauphin helped with experiments and trained the LM1B model checkpoints
used in the submission. He proposed latent variables as a solution to user
misspecification with two annotators (Section A.4). He identified the “length
bias” issue with the Luce choice rule for autoregressive models and he
developed the length-normalized Luce choice rule solution jointly with
Vincent. He helped develop and frame the theoretical results in the paper.

#### Acknowledgements

We would like to thank Laurent Dinh for his help in deriving the proof for
Theorem 1; Jesse Engel for his input in the early stages of the project;
Christian Walder for his input on inducing specific properties in globally
optimal models by manipulating the model’s generative process for preferences;
Ben Poole for his thoughtful feedback in revising the manuscript; and Rishabh
Joshi for clarifying the implementation details of SLiC-HF-direct.

## References

  * Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A general theoretical paradigm to understand learning from human preferences.  _arXiv preprint arXiv:2310.12036_ , 2023.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Biyik & Sadigh (2018) Erdem Biyik and Dorsa Sadigh.  Batch active preference-based learning of reward functions.  In _Proceedings of the Conference on Robot Learning_ , 2018.
  * Bradley & Terry (1952) Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Brown et al. (2019) Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum.  Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations.  In _Proceedings of the International Conference on Machine Learning_ , 2019.
  * Brown et al. (2020) Daniel S Brown, Wonjoon Goo, and Scott Niekum.  Better-than-demonstrator imitation learning via automatically-ranked demonstrations.  In _Proceedings of the Conference on Robot Learning_ , 2020.
  * Cayton (2005) Lawrence Cayton.  Algorithms for manifold learning.  Technical Report CS2008-0923, University of California, 2005.
  * Chelba et al. (2013) Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson.  One billion word benchmark for measuring progress in statistical language modeling.  _arXiv preprint arXiv:1312.3005_ , 2013.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in Neural Information Processing Systems_ , 2017.
  * Ibarz et al. (2018) Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei.  Reward learning from human preferences and demonstrations in atari.  _Advances in Neural Information Processing Systems_ , 2018.
  * Kingma & Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _Proceedings of the International Conference on Learning Representations_ , 2015.
  * Korbak et al. (2022) Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Marc Dymetman.  On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting.  In _Advances in Neural Processing Systems_ , 2022.
  * Lee et al. (2021) Kimin Lee, Laura Smith, and Pieter Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  In _Proceedings of the International Conference on Machine Learning_ , 2021.
  * Liu et al. (2023a) Hao Liu, Carmelo Sferrazza, and Pieter Abbeel.  Chain of hindsight aligns language models with feedback.  _arXiv preprint arXiv:2302.02676_ , 2023a.
  * Liu et al. (2023b) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu.  Statistical rejection sampling improves preference optimization.  _arXiv preprint arXiv:2309.06657_ , 2023b.
  * Luce et al. (1963) R Luce, Robert R Bush, and Eugene Ed Galanter.  _Handbook of mathematical psychology_.  John Wiley, 1963.
  * Narayanan & Mitter (2010) Hariharan Narayanan and Sanjoy Mitter.  Sample complexity of testing the manifold hypothesis.  _Advances in neural information processing systems_ , 23, 2010.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Peterson et al. (2018) Joshua C Peterson, Jordan W Suchow, Krisha Aghi, Alexander Y Ku, and Thomas L Griffiths.  Capturing human category representations by sampling in deep feature spaces.  In _CogSci_ , 2018.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Sadigh et al. (2017) Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia.  Active preference-based learning of reward functions.  Technical report, UC Berkeley, 2017.
  * Sanborn & Griffiths (2007) Adam Sanborn and Thomas Griffiths.  Markov chain monte carlo with people.  In _Advances in Neural Information Processing Systems_ , 2007.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shepard (1957) Roger N Shepard.  Stimulus and response generalization: A stochastic model relating generalization to distance in psychological space.  _Psychometrika_ , 22(4):325–345, 1957.
  * Shin et al. (2023) Daniel Shin, Anca D Dragan, and Daniel S Brown.  Benchmarks and algorithms for offline preference-based reward learning.  _Transactions on Machine Learning Research_ , 2023.
  * Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett.  A long way to go: Investigating length correlations in RLHF.  _arXiv preprint arXiv:2310.03716_ , 2023.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 2020.
  * Sutton & Barto (1998) Richard S. Sutton and Andrew G. Barto.  _Introduction to Reinforcement Learning_.  MIT Press, Cambridge, MA, USA, 1st edition, 1998.  ISBN 0262193981.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Wirth et al. (2017) Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al.  A survey of preference-based reinforcement learning methods.  _Journal of Machine Learning Research_ , 18(136):1–46, 2017.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  RRHF: Rank responses to align language models with human feedback without tears.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.

##  Appendix A Appendix

###  A.1 Theorems and proofs

######  Lemma 1.

Let $q({\bm{x}}_{A},{\bm{x}}_{B})$ be a joint probability distribution with
support ${\mathcal{S}}\times{\mathcal{S}}$ and for which
$q({\bm{x}}_{A},{\bm{x}}_{B})>0$ for all
${\bm{x}}_{A},{\bm{x}}_{B}\in{\mathcal{S}}\times{\mathcal{S}}$. Let
$\chi({\bm{x}})\colon{\mathcal{S}}\rightarrow\mathbb{R}$ and
$\omega({\bm{x}})\colon{\mathcal{S}}\rightarrow\mathbb{R}$ be two injective
functions mapping ${\bm{x}}$ to scalar “scores”. Let

$p_{\chi}(y;{\bm{x}}_{A},{\bm{x}}_{B})=\frac{e^{\chi({\bm{x}}_{A})}}{e^{\chi({\bm{x}}_{A})}+e^{\chi({\bm{x}}_{B})}}=\sigma\Big{(}\chi({\bm{x}}_{A})-\chi({\bm{x}}_{B})\Big{)}\quad\textnormal{and}\quad
p_{\omega}(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sigma\Big{(}\omega({\bm{x}}_{A})-\omega({\bm{x}}_{B})\Big{)}.$

The loss function

$\mathbb{E}_{{\bm{x}}_{A},{\bm{x}}_{B}\sim
q({\bm{x}}_{A},{\bm{x}}_{B})}\bigg{[}D_{\mathrm{KL}}\big{(}p_{\chi}(y;{\bm{x}}_{A},{\bm{x}}_{B})\mid\mid
p_{\omega}(y;{\bm{x}}_{A},{\bm{x}}_{B})\big{)}\bigg{]}.$

is globally minimized when

$\chi({\bm{x}})=\omega({\bm{x}})+C,\quad
C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}}.$

###### Proof.

We know from the properties of the KL-divergence that it is minimized when the
left- and right-hand side represent the same distribution, which means that
the expectation itself has a global minimum at zero when

$\displaystyle\frac{e^{\chi({\bm{x}}_{A})}}{e^{\chi({\bm{x}}_{A})}+e^{\chi({\bm{x}}_{B})}}=\frac{e^{\omega({\bm{x}}_{A})}}{e^{\omega({\bm{x}}_{A})}+e^{\omega({\bm{x}}_{B})}},$
$\displaystyle\forall{\bm{x}}_{A},{\bm{x}}_{B}\in{\mathcal{S}}\times{\mathcal{S}}$
$\displaystyle\Rightarrow$
$\displaystyle\cancel{e^{\chi({\bm{x}}_{A})+\omega({\bm{x}}_{A})}}+e^{\chi({\bm{x}}_{A})+\omega({\bm{x}}_{B})}=\cancel{e^{\chi({\bm{x}}_{A})+\omega({\bm{x}}_{A})}}+e^{\chi({\bm{x}}_{B})+\omega({\bm{x}}_{A})},$
$\displaystyle\forall{\bm{x}}_{A},{\bm{x}}_{B}\in{\mathcal{S}}\times{\mathcal{S}}$
$\displaystyle\Rightarrow$
$\displaystyle\chi({\bm{x}}_{A})=\omega({\bm{x}}_{A})+(\chi({\bm{x}}_{B})-\omega({\bm{x}}_{B})),$
$\displaystyle\forall{\bm{x}}_{A},{\bm{x}}_{B}\in{\mathcal{S}}\times{\mathcal{S}}$

For any given ${\bm{x}}_{A}$, the above equation needs to hold for all
${\bm{x}}_{B}\in{\mathcal{S}}$, which is only possible if the term
$\chi({\bm{x}}_{B})-\omega({\bm{x}}_{B})$ is constant for all
${\bm{x}}_{B}\in{\mathcal{S}}$. We therefore conclude that

$\chi({\bm{x}})=\omega({\bm{x}})+C,\quad
C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}}.$

∎

#### Proof for Theorem 1

###### Proof.

The proof follows directly from applying Lemma 1 with $\chi({\bm{x}})=\log
p^{*}({\bm{x}})$ and $\omega({\bm{x}})=r_{\phi}({\bm{x}})$: at the global
optimum we have that

$\log p^{*}({\bm{x}})=r_{\phi}({\bm{x}})+C,\quad
C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}},$

which means that

$e^{r_{\phi}({\bm{x}})}\propto
p^{*}({\bm{x}})\quad\forall{\bm{x}}\in{\mathcal{S}}.$

∎

#### Proof for Theorem 2

###### Proof.

Using Lemma 1 with $\chi({\bm{x}})=\Omega_{p^{*}}({\bm{x}})$ and
$\omega({\bm{x}})=\Omega_{\pi_{\theta}}({\bm{x}})$, at the global optimum we
have that

$\displaystyle\Omega_{p^{*}}({\bm{x}})=\Omega_{\pi_{\theta}}({\bm{x}})+C,$
$\displaystyle C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}}$
$\displaystyle\Rightarrow$ $\displaystyle f({\bm{x}})\log
p^{*}({\bm{x}})+g({\bm{x}})=f({\bm{x}})\log\pi_{\theta}({\bm{x}})+g({\bm{x}})+C,$
$\displaystyle C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}}$
$\displaystyle\Rightarrow$ $\displaystyle
p^{*}({\bm{x}})=\pi_{\theta}({\bm{x}})e^{\frac{C}{f({\bm{x}})}},$
$\displaystyle C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}}$

Since both $p^{*}({\bm{x}})$ and $\pi_{\theta}({\bm{x}})$ are probability
distributions, summing or integrating both sides results in

$\mathbb{E}_{\pi_{\theta}({\bm{x}})}\left[e^{\frac{C}{f({\bm{x}})}}\right]=1.$

There are three possibilities for $C$: either $C<0$, $C>0$, or $C=0$. Since we
know $f({\bm{x}})>0$ for all ${\bm{x}}$, $C<0$ would result in an expectation
smaller than 1 and $C>0$ would result in an expectation larger than 1. We
conclude that $C=0$ and therefore

$\pi_{\theta}({\bm{x}})=p^{*}({\bm{x}}),\quad\forall{\bm{x}}.$

∎

#### Proof of Equation 16’s global optimality

###### Proof.

We observe that

$p_{\theta}(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sigma\left(\beta\log\frac{\pi_{\theta}({\bm{x}}_{A})}{\pi_{\textnormal{pre}}({\bm{x}}_{A})}-\beta\log\frac{\pi_{\theta}({\bm{x}}_{B})}{\pi_{\textnormal{pre}}({\bm{x}}_{B})}\right).$

Using Lemma 1 with $\chi({\bm{x}})=\log p^{*}({\bm{x}})$ and
$\omega({\bm{x}})=\beta(\log\pi_{\theta}({\bm{x}})-\log\pi_{\textnormal{pre}}({\bm{x}}))$,
at the global optimum we have that

$\log
p^{*}({\bm{x}})=\beta(\log\pi_{\theta}({\bm{x}})-\log\pi_{\textnormal{pre}}({\bm{x}}))+C,\quad
C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}},$

from which we conclude that

$\pi_{\theta}({\bm{x}})\propto\pi_{\textnormal{pre}}({\bm{x}})\cdot
p^{*}({\bm{x}})^{\frac{1}{\beta}}.$

∎

#### Proof of Equation 18’s global optimality

###### Proof.

Using Lemma 1 with $\chi({\bm{x}})=\log p^{*}({\bm{x}})$ and
$\omega({\bm{x}})=\frac{1}{\alpha}(\log\pi_{\theta}({\bm{x}})-\log\pi_{\textnormal{pre}}({\bm{x}}))+\log\pi_{\textrm{pre}}({\bm{x}})$,
at the global optimum we have that

$\log
p^{*}({\bm{x}})=\frac{1}{\alpha}(\log\pi_{\theta}({\bm{x}})-\log\pi_{\textnormal{pre}}({\bm{x}}))+\log\pi_{\textrm{pre}}({\bm{x}})+C,\quad
C\in\mathbb{R},\forall{\bm{x}}\in{\mathcal{S}},$

from which we conclude that

$\pi_{\theta}({\bm{x}})\propto\pi_{\textnormal{pre}}({\bm{x}})^{1-\alpha}\cdot
p^{*}({\bm{x}})^{\alpha}.$

∎

###  A.2 Properties of SLiC-HF in the toy 1D setting

We finetune the pretrained model $\pi_{\textnormal{pre}}({\bm{x}})$ with SLiC-
HF-direct on pairwise preferences acquired from the implicit preference
distribution defined in Equation 10. Since we consider a purely preference-
based setting, we cannot assume access to a finetuning dataset, however we can
still regularize the model towards the pretrained model by sampling from
$\pi_{\textnormal{pre}}({\bm{x}})$ in Equation 20’s second expectation. We
sweep over $\delta\in\\{0.0,0.25,0.5,1.0,2.0\\}$ and $\lambda\in\\{0.0,1.0\\}$
to adapt the pretrained model with SLiC-HF-direct. Figure 10 shows the effect
of $\delta$ and $\lambda$. The $\delta$ hyperparameter has a non-uniform
smoothing effect on $\pi_{\theta}({\bm{x}})$, as it bounds the maximum
difference in log-probabilities allowed between any two observations. The
$\lambda$ hyperparameter has the expected effect of regularizing the adapted
model towards $\pi_{\textnormal{pre}}({\bm{x}})$.

###  A.3 Properties of IPO in the toy 1D setting

We finetune the pretrained model $\pi_{\textnormal{pre}}({\bm{x}})$ with IPO
on pairwise preferences acquired from the implicit preference distribution
defined in Equation 10. We sweep over $\tau\in\\{0.1,0.5,1.0\\}$ to adapt the
pretrained model with IPO. Figure 11 shows the regularizing effect of $\tau$
towards the pretrained model’s density.

###  A.4 Addressing annotator misspecification in a toy setting

To account for annotator identity in the experiment described in Section 5, we
initialize two copies of $\pi_{\textnormal{pre}}({\bm{x}})$ (adding Gaussian
noise with standard deviation $1\text{\times}{10}^{-4}$ to break symmetries)
to create a mixture model

$\pi_{\theta}({\bm{x}})=\sum_{i=1}^{2}\frac{w_{i}}{w_{1}+w_{2}}\pi_{\theta}({\bm{x}})$
(29)

and adjust our annotator behavior model to account for Annotator 1 and
Annotator 2 so that

$p_{\theta}(y;{\bm{x}}_{A},{\bm{x}}_{B})=\sum_{i=1}^{2}\frac{w_{i}}{w_{1}+w_{2}}\cdot\frac{\pi_{\theta}({\bm{x}}_{A})}{\pi_{\theta}({\bm{x}}_{A})+\pi_{\theta}({\bm{x}}_{B})}.$
(30)

We then train as usual with a binary cross-entropy loss on the comparison
outcome variable $y$ (Figure 12). Doing so allows to recover the correct
marginal implicit preference distribution, even in the absence of annotator
IDs in the preference data. This is particularly interesting in contexts where
human annotators “cluster” in their preferences. In the coffee example
presented in section 5, the two annotators may represent entire subpopulations
of annotators.

Figure 8:  A reward model trained on comparison outcomes sampled according to
an implicit preference distribution (Equation 10; dashed blue) converges to
the annotator’s unnormalized preference distribution (solid green), as
indicated by the constant offset between the two curves. Figure 9:  Under the
Luce choice rule for the annotator, using a PBDE for the model with
$f({\bm{x}})=\alpha^{-1}$ and
$g({\bm{x}})=(1-\alpha^{-1})\log\pi_{\textnormal{pre}}({\bm{x}})$ to tune
$\pi_{\textnormal{pre}}({\bm{x}})$ (dotted orange) on pairwise preferences
derived from the implicit preference distribution (dashed blue) results in a
weighted geometric average between the initial model $\pi_{\textnormal{pre}}$
and the implicit preference distribution, as demonstrated by the agreement
between the empirical (solid green) and theoretical (dash-dotted red) curves.

Figure 10:  Using SLiC-HF-direct to tune a generative model
$\pi_{\textnormal{pre}}({\bm{x}})$ (dotted orange) on pairwise preferences
derived from the implicit preference distribution (dashed blue) results in
different adapted models (solid green) depending on the choice of
hyperparameters $\delta$ and $\lambda$. Figure 11:  Using IPO to tune a
generative model $\pi_{\textnormal{pre}}({\bm{x}})$ (dotted orange) on
pairwise preferences derived from the implicit preference distribution (dashed
blue) results in different adapted models (solid green) depending on the
choice of the $\tau$ hyperparameter. Figure 12:  Choosing the correct
annotator behavior model (Equation 30) allows to recover the correct marginal
implicit preference distribution (left) and individual implicit preference
distributions (right), even without annotator IDs in the preference data.