# RLHF and IIA: Perverse Incentives

Wanqiao Xu  Stanford University  Shi Dong  Google DeepMind  Xiuyuan Lu  Google
DeepMind
Grace Lam  Google DeepMind  Zheng Wen  Google DeepMind  Benjamin Van Roy
Stanford University  Google DeepMind

###### Abstract

Existing algorithms for reinforcement learning from human feedback (RLHF) can
incentivize responses at odds with preferences because they are based on
models that assume independence of irrelevant alternatives (IIA). The perverse
incentives induced by IIA give rise to egregious behavior when innovating on
query formats or learning algorithms.

##  1 Introduction

Modern generative AIs ingest trillions of data bytes from the World Wide Web
to produce a large pretrained model. Trained to imitate what is observed, this
model represents an agglomeration of behaviors, some of which are more or less
desirable to mimic. Further training through human interaction, even on fewer
than a hundred thousand bits of data, has proven to greatly enhance usefulness
and safety, enabling the remarkable AIs we have today. This process of
reinforcement learning from human feedback (RLHF) steers AIs toward the more
desirable among behaviors observed during pretraining.

While AIs now routinely generate drawings, music, speech, and computer code,
the text-based chatbot remains an emblematic artifact. To produce a chatbot,
starting with a pretrained language model, a prototypical approach to RLHF [4,
15, 11] progresses through several steps. First, queries, each taking the form
of a prompt and a pair of alternative responses, are presented to human
annotators who each identify their favorite among a pair. The annotated data
is then used to train a reward model to score any response to a given prompt.
Finally, the language model is tuned to align its responses toward those that
earn high reward.

The tremendous impact of RLHF in generative AI has sparked a flurry of
research that aims to understand and improve the process. Some propose
alternative algorithms [12, 16, 18, 6]. Others consider alternative query
formats [19, 14, 17, 12] for which annotators, rather than comparing only a
pair of responses, are asked to choose from a longer list, or to rank-order.
Feedback can also be garnered from interactions with humans in their regular
use of online chatbots. RLHF research is continually growing in importance
with the volume of human feedback data.

With all the effort and resources directed at RLHF, it is worth asking whether
current algorithms rest on firm foundations. Maybe not, as these algorithms
are based on models that assume independence of irrelevant alternatives (IIA),
which does not cohere with human preferences for text content. Interestingly,
this flaw is not pronounced when using the most common approach of fitting a
reward model to pairwise comparison data, then tuning a language model to
optimize reward. However, working with alternative query formats or
algorithmic innovations can induce egregious behavior.

A simple experiment we will present in Section 6 illustrates our point. This
experiment applies a standard reward learning algorithm [4, 15, 11]. We first
consider learning from queries that are each comprised of the prompt

> prompt Did Oppenheimer win a Nobel Prize?

and a pair of responses, one generated by GPT-3.5 and the other by GPT-4. The
former generally produces more concise and the latter more informative
responses. For example, here are representative responses:

> GPT-3.5 response No, Oppenheimer did not win the Nobel Prize. GPT-4 response
> No, Robert Oppenheimer, often called the “father of the atomic bomb” for his
> role in the Manhattan Project, did not win a Nobel Prize.

If a large majority of annotators prefer responses generated by GPT-4, the
learned reward function correctly assigns higher scores to GPT-4 responses in
independent test data.

A variation in which training queries include four rather than two responses
reveals egregious behavior induced by the IIA assumption. In particular,
suppose that each query includes one response generated by GPT-3.5 and three
by GPT-4. Then, even if a large majority of annotators prefer responses
generated by GPT-4, the learned reward function erroneously assigns higher
scores to GPT-3.5 responses in test data.

Aside from innocuous changes to the query format, innovating on RLHF
algorithms can also result in undesirable behavior. To offer an example, we
will consider inclusive learning (IL), an approach that aims to produce a
language model that samples responses according to human choice probabilities
[2, 16]. This is in contrast to the standard approach to RLHF, which targets
responses with the highest human choice probabilities. In Section 4, we will
demonstrate that, even with two responses per query, the IIA assumption exacts
egregious behavior from IL.

The remainder of the paper explains issues and methods more deeply, with a
goal to enhance understanding of flaws in current algorithms. We develop this
understanding through interpreting simulations of didactic models, an
empirical study of data produced by GPT-3.5 and GPT-4, and theoretical results
that corroborate the observed behavior. These theoretical results establish
that such behavior generalizes beyond specific instances. We leave for future
work the design of algorithms that alleviate flaws we identify here.

##  2 Language Models, Messages, and Preference Data

We begin by defining a few basic concepts that we will use when describing and
analyzing algorithms.

###  2.1 Language Models

Let $\mathcal{A}$ be a finite set, which we will refer to as an alphabet. We
refer to elements as tokens. We denote the set of finite token sequences by
$\mathcal{A}^{+}$. A language model is a function $\pi$ that, for each
$x\in\mathcal{A}^{+}$, specifies a probability mass function $\pi(\cdot|x)$
over $\mathcal{A}$. For example, if the alphabet is the set of ASCII
characters, a language model provides a distribution
$\pi(\cdot|(X_{1},X_{2},\ldots,X_{t}))$ conditioned on $t$ characters typed so
far. Sequentially sampling characters
$X_{t+1}\sim\pi(\cdot|(X_{1},\ldots,X_{t}))$ generates text. A language model
is alternately referred to as a policy, as it provides a rule for selecting a
next token based on the preceding sequence.

###  2.2 Messages

There is a set $\mathcal{M}\subseteq\mathcal{A}^{+}$ of token sequences that
we refer to as messages. We interpret $\mathcal{M}$ as the set of complete
individual statements. For example, in a text messaging application, a message
might terminate with either the ASCII enter character or upon reaching a
maximum allowed message length.

A language model generates a random message by sampling sequentially according
to $X_{t+1}\sim\pi(\cdot|(X_{1},\ldots,X_{t}))$, terminating at the first
index $T$ for which $(X_{1},\ldots,X_{T})\in\mathcal{M}$. For any token
sequence $x=(x_{1},\ldots,x_{T})\in\mathcal{A}^{+}$, we use the notation
$x_{t_{1}:t_{2}}$ to denote the subsequence $(x_{t_{1}},\ldots,x_{t_{2}})$.
Hence, we can use the abbreviated form $X_{t+1}\sim\pi(\cdot|X_{1:t})$. We
assume that, for any language model $\pi$, any token sequence generated
sequentially in this manner terminates with probability one. Through this
procedure, a language model $\pi$ samples messages from a distribution
$P_{\pi}$, defined by

$P_{\pi}(x)=\prod_{t=1}^{|x|}\pi(x_{t}|x_{1:t-1}),$ (1)

for $x\in\mathcal{M}$, where $|x|$ denotes the length of sequence $x$. For a
set of messages $\mathcal{S}$, we use $P_{\pi}(\mathcal{S})$ to denote
$\sum_{x\in\mathcal{S}}P_{\pi}(x)$.

###  2.3 Prefixes

We refer to a token sequence $x$ as a prefix if $x_{1:t}\notin\mathcal{M}$ for
all $t=1,\ldots,|x|$. In our aforementioned text messaging example, a prefix
would be any text absent the enter character and of length less than the
maximum allowed. Let $\mathcal{P}$ denote the set of prefixes. While a
language model $\pi$ specifies probabilities $\pi(\cdot|x)$ for all token
sequences $x\in\mathcal{A}^{+}$, only those for which $x\in\mathcal{P}$ are
consequential to message generation.

###  2.4 Preference Data

Each preference datum is a pair $(\mathcal{Y},y)$. The first item is a set
$\mathcal{Y}\subseteq\mathcal{M}$ of alternative messages, while the second is
a choice $y\in\mathcal{Y}$. For example, each message in $\mathcal{Y}$ could
be a poem, with the choice $y$ indicating an individual’s favorite among them.
RLHF algorithms we present use a preference dataset $\mathcal{D}$. Each
element is a preference datum, and the number of alternative messages can vary
across elements. Note that $\mathcal{Y}$ and $\mathcal{D}$ are technically
multisets because elements can repeat. However, we will refer to these
informally as sets.

###  2.5 What About Prompting?

Language models typically generate a message in response to a prompt made up
of previous messages. In that context, each preference datum expresses a
choice between responses to a prompt. Except for our empirical study of
Section 6, we omit prompting from our formulation and analysis, because that
would only complicate the discussion without contributing to insight.
Examples, algorithms, and results we present can easily be extended to treat
prompting.

##  3 Choice Models

Given a set $\mathcal{Y}\subseteq\mathcal{M}$ of alternatives, a choice model
generates a random element $Y\in\mathcal{Y}$. We interpret $Y$ to be the
choice made by a random individual. Choice models we consider are each defined
with respect to a message set $\mathcal{M}$ and a triple $(\mathcal{Z},p,r)$,
consisting of a set $\mathcal{Z}$ of individual types, a type distribution
$p$, and a reward function $r$. We will define these terms and then present
examples. Choices in the first and second examples, but not the third example,
satisfy IIA. After presenting these models, we discuss their relation to IIA
and why human preferences for text violate IIA.

###  3.1 Individual Types

Individuals are distinguished by type. Let $\mathcal{Z}$ denote the set of
types. The type of a random individual, $Z$, is distributed according to a
type distribution $p$. If the set of types $\mathcal{Z}$ is finite, this
distribution assigns a probability $p(z)$ to each $z\in\mathcal{Z}$.

###  3.2 Reward Functions

A reward function $r$ expresses individual preferences between elements of a
message set $\mathcal{M}$. In particular, an individual of type
$z\in\mathcal{Z}$ prefers a message $x$ to $x^{\prime}$ if and only if
$r(x|z)\geq r(x^{\prime}|z)$.

To illustrate with an example, two elements $z,z^{\prime}\in\mathcal{Z}$ could
express detailed characteristics of particular individuals, one of whom
resides in China and the other in the United States. These individuals have
different tastes and needs – a message $x$ that is helpful or interesting to
one may not be to the other. The distinct preferences are expressed by
differing reward assignments $r(x|z)\neq r(x|z^{\prime})$.

###  3.3 Choice Probabilities

A choice model $(\mathcal{Z},p,r)$ expresses how random individuals make
choices. The model posits that, when presented with the set $\mathcal{Y}$ of
alternatives, an individual of type $z\in\mathcal{Z}$ samples their choice $Y$
uniformly from $\operatorname*{arg\,max}_{y\in\mathcal{Y}}r(y|z)$. This
implies choice probabilities

$\mathbb{P}(Y=y|\mathcal{Y})=\mathbb{E}\left[\frac{\mathbbm{1}(y\in\operatorname*{arg\,max}_{x\in\mathcal{Y}}r(x|Z))}{|\operatorname*{arg\,max}_{x\in\mathcal{Y}}r(x|Z)|}\right],$
(2)

where $Z$ is sampled from $p$. In particular, $\mathbb{P}(Y=y|\mathcal{Y})$ is
the probability that a randomly sampled individual presented with
$\mathcal{Y}$ will choose $y\in\mathcal{Y}$.

###  3.4 Example 1: Logit Models

The standard logit model can be expressed as a choice model
$(\mathcal{Z},p,r)$. The set $\mathcal{Z}$ of types is comprised of functions
that map $\mathcal{M}$ to $\Re_{+}$. Hence, the type of a random individual is
expressed by a random function $Z$. For each $x\in\mathcal{M}$, $Z(x)$ is
distributed as an independent standard Gumbel. For some fixed base reward
function $\overline{r}:\mathcal{A}^{+}\rightarrow\Re$, let
$r(x|z)=\overline{r}(x)+z(x)$. Given this reward function and type
distribution, sampling a choice $Y$ as described above implies choice
probabilities governed by the standard logit model with rewards specified by
the base reward function $\overline{r}$ [8]. In particular,
$(\mathcal{Z},p,r)$ gives rise to choice probabilities

$\mathbb{P}(Y=y|\mathcal{Y})=\frac{e^{\overline{r}(y)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{\overline{r}(y^{\prime})}}.$
(3)

Conversely, only the choice model $(\mathcal{Z},p,r)$ that we have specified
or one with a reward function that differs from $r$ by a constant produces
these choice probabilities [9].

As an example, consider a language model $\pi$ that recommends recipes for
dinner. For each recipe $y$, $\overline{r}(y)$ indicates its nominal
desirability – no individual attributes any less reward to $y$. But for a
specific individual of type $z$, a positive perturbation $z(y)$ is added to
reflect their idiosyncratic preferences. Presented with a set $\mathcal{Y}$ of
recipes, the individual selects the recipe $y\in\mathcal{Y}$ that maximizes
$\overline{r}(y)+z(y)$. In this way, $z$ induces variation in how different
individuals make their selection. The Gumbel distribution enjoys elegant
mathematical structure: if, for each recipe $y$, $z(y)$ is an independent
sample from the standard Gumbel then the probability that a random individual
selects $y$ over the remaining recipes in $\mathcal{Y}$ is given by (3).

###  3.5 Example 2: Soft Choice Models

Soft choice models generalize on the logit. Similarly with the choice model
formalism defined in Equation (2) – which we alternately refer to as a hard
choice model – a soft choice model is specified by a triple
$(\mathcal{Z},p,r)$. However, unlike a hard choice model, choice probabilities
are given by

$\mathbb{P}(Y=y|\mathcal{Y})=\mathbb{E}\left[\frac{e^{r(y|Z)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{r(y^{\prime}|Z)}}\right],$
(4)

instead of (2).

A soft choice model $(\mathcal{Z},p,r)$ can be expressed as a hard choice
model $(\mathcal{Z}^{\prime},p^{\prime},r^{\prime})$. By this we mean that
choice probabilities $\mathbb{P}(Y=y|\mathcal{Y})$ implied by the two models
are identical. The type set $\mathcal{Z}^{\prime}$ is the product set
$\mathcal{Z}\times\Re_{+}^{\mathcal{M}}$. The type of a random individual is a
pair $Z^{\prime}=(Z,W)$. The first item $Z$ is sampled from $p$. The second
item comprises of an independent standard Gumbel sample $W(x)$ for each
$x\in\mathcal{M}$. The reward function is given by
$r^{\prime}(x|z^{\prime})=r(x|z)+w(x)$. It is easy to see that this hard
choice model $(\mathcal{Z}^{\prime},p^{\prime},r^{\prime})$ produces the same
choice probabilities (4) as the soft choice model $(\mathcal{Z},p,r)$.

Note that a logit model can be expressed as a soft choice model. Consider a
logit model expressed as a hard choice model $(\mathcal{Z},p,r)$. Define a
soft choice model $(\mathcal{Z}^{\prime},p^{\prime},r^{\prime})$ with a single
type $\mathcal{Z}=\\{0\\}$, a trivial type distribution for which $p(0)=1$,
and a reward function $r^{\prime}(x|0)=r(x|0)$. This soft choice model implies
choice probabilties

$\mathbb{P}(Y=y|\mathcal{Y})=\frac{e^{r^{\prime}(y|0)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{r^{\prime}(y^{\prime}|0)}},$

which are equivalent to the logit choice probabilities (3) if we take the base
reward function to be $\overline{r}(x)=r(x|0)$.

###  3.6 Example 3: Dichotomy Models

Our next model class is crafted to illustrate in a transparent manner perverse
incentives induced by current RLHF algorithms. There are two individual types
$\mathcal{Z}=\\{1,2\\}$, each of which prefers one of two categories of
messages, identified by nonempty sets $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$
for which $\mathcal{M}_{1}\cap\mathcal{M}_{2}=\emptyset$ and
$\mathcal{M}_{1}\cup\mathcal{M}_{2}=\mathcal{M}$. The reward function takes
the form

$r(x|z)=\left\\{\begin{array}[]{ll}1&\text{if }x\in\mathcal{M}_{z}\\\
0&\text{otherwise.}\end{array}\right.$

Note that reward depends on the message $x$ only through its membership in
$\mathcal{M}_{1}$ or $\mathcal{M}_{2}$. In other words, each individual is
indifferent between any two messages of the same type.

The type set $\mathcal{Z}$ and reward function $r$ we have defined, together
with any type distribution $p$, constructs a choice model $(\mathcal{Z},p,r)$.
According to this choice model, when presented with alternatives
$\mathcal{Y}$, the choice $Y$ made by an individual of type $Z$ is sampled
uniformly from $\mathcal{Y}\cap\mathcal{M}_{Z}$. Hence,

$\mathbb{P}(Y=y|\mathcal{Y})=\mathbb{E}\left[\frac{\mathbbm{1}_{\mathcal{M}_{Z}}(y)}{|\mathcal{Y}\cap\mathcal{M}_{Z}|}\right],$
(5)

where $Z$ is sampled from $p$.

A hypothetical sort of homophily serves to illustrate a context where this
sort of model may be relevant. Suppose a language model $\pi$ generates
political commentary, with $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ conveying
conservative versus liberal views. In this hypothetical world, there are two
types of individuals – conservatives and liberals – labeled $1$ and $2$. Each
prefers commentary aligned with their predisposition. There is no reward for
opposing views.

###  3.7 Independence of Irrelevant Alternatives

Consider three messages
$\mathcal{M}=\\{\mathrm{dog},\mathrm{cat},\mathrm{feline}\\}$ that each guess
an individual’s favorite pet. Consider a logit choice model, as described in
Section 3.4, that assigns equal base rewards
$r(\mathrm{dog})=r(\mathrm{cat})=r(\mathrm{feline})$ to indicate that a random
individual is as likely to prefer dogs or cats. Presented with two
alternatives $\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat}\\}$, the logit model
produces choice probabilities
$\mathbb{P}(Y=\mathrm{dog}|\mathcal{Y})=\mathbb{P}(Y=\mathrm{cat}|\mathcal{Y})=1/2$.
On the other hand, with three alternatives
$\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat},\mathrm{feline}\\}$,
$\mathbb{P}(Y=\mathrm{dog}|\mathcal{Y})=\mathbb{P}(Y=\mathrm{cat}|\mathcal{Y})=\mathbb{P}(Y=\mathrm{feline}|\mathcal{Y})=1/3$.
This of course makes no sense, because feline is synonymous to cat, so
presenting it as an alternative ought not reduce the probability that an
individual prefers dogs.

The aforementioned implausible implication of a logit model derives from its
assumption of independence of irrelevant alternatives (IIA). While this notion
was formalized in earlier work [1], the treatment of [7] best suits our usage.
In that treatment, IIA indicates that the ratio of any two choice
probabilities does not vary with the set of alternatives. More formally, for
all messages $x,x^{\prime}\in\mathcal{M}$ and sets $\mathcal{X}$ and
$\mathcal{X}^{\prime}$ such that
$x,x^{\prime}\in\mathcal{X}\cap\mathcal{X}^{\prime}$,

$\frac{\mathbb{P}(Y=x|\mathcal{Y}=\mathcal{X})}{\mathbb{P}(Y=x^{\prime}|\mathcal{Y}=\mathcal{X})}=\frac{\mathbb{P}(Y=x|\mathcal{Y}=\mathcal{X}^{\prime})}{\mathbb{P}(Y=x^{\prime}|\mathcal{Y}=\mathcal{X}^{\prime})}.$
(6)

This property implies that if cat-lovers are indifferent and randomly choose
between cat and feline then the introduction of feline as an alternative
reduces not only the probability that a random individual selects cat but also
dog. In particular, (6) specializes to

$\frac{\mathbb{P}(Y=\mathrm{dog}|\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat}\\})}{\mathbb{P}(Y=\mathrm{cat}|\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat}\\})}=\frac{\mathbb{P}(Y=\mathrm{dog}|\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat},\mathrm{feline}\\})}{\mathbb{P}(Y=\mathrm{cat}|\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat},\mathrm{feline}\\})}.$

Our dichotomy model of Section 3.6 relaxes IIA and can generate more realistic
pet choices, as we will now explain. Take the individual types
$\mathcal{Z}=\\{1,2\\}$ to be dog and cat lovers, with type probabilities
$p_{*}(1)=p_{*}(2)=1/2$, and rewards $r_{1}=r_{2}=1$. Then, presented with two
alternatives $\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat}\\}$, the dichotomy
model produces the same choice probabilities as the logit model:
$\mathbb{P}(Y=\mathrm{dog}|\mathcal{Y})=\mathbb{P}(Y=\mathrm{cat}|\mathcal{Y})=1/2$.
On the other hand, with three alternatives
$\mathcal{Y}=\\{\mathrm{dog},\mathrm{cat},\mathrm{feline}\\}$,
$\mathbb{P}(Y=\mathrm{dog}|\mathcal{Y})=1/2$, while
$\mathbb{P}(Y=\mathrm{cat}|\mathcal{Y})=\mathbb{P}(Y=\mathrm{feline}|\mathcal{Y})=1/4$.
Hence, unlike the logit model, with the dichotomy model the probability that a
random individual chooses dog appropriately remains unchanged.

###  3.8 IIA for Language

Human preferences for text messages do not satisfy IIA. This is because, for
any given message, others can be virtually equivalent. Our preceding example
with three messages – dog, cat, and feline – illustrates this. The
introduction of feline as an alternative should not reduce the probability
that an individual chooses dog.

In spite of this, current RLHF algorithms are based on models that assume IIA.
In particular, preference estimates used in RLHF satisfy IIA even though human
preferences expressed through data do not. These estimates can give rise to
perverse incentives to generate undesirable messages. We will discuss in
Section 5 potentially egregious consequences. But first, we review current
RLHF algorithms.

##  4 RLHF Algorithms

An RLHF algorithm, given a base language model $\overline{\pi}$ and preference
data $\mathcal{D}$, produces a fine-tuned language model $\widehat{\pi}$. In
practice, the base language model is typicaly a pretrained model that has
undergone some supervised fine-tuning. Figure 1 illustrates the RLHF algorithm
interface. In this section, we describe several instances of RLHF algorithms.

Figure 1: RLHF algorithm interface.

###  4.1 Reward Learning Followed by Policy Optimization

A standard approach to RLHF first fits a logit model to preference data and
then tunes a language model to optimize reward [15, 11]. We will refer to this
as RLPO, which stands for reward learning followed by policy optimization.
RLPO uses a reward model $\widehat{r}_{\psi}$, which is parameterized by a
vector $\psi$. This reward model maps $\mathcal{M}$ to $\Re$ and is
interpreted as expressing approximate soft choice probabilities
$\mathbb{P}(Y=y|\mathcal{Y})\approx
e^{\widehat{r}_{\psi}(y)}/\sum_{y^{\prime}\in\mathcal{Y}}e^{\widehat{r}_{\psi}(y^{\prime})}$.
RLPO tunes parameters to minimize cross-entropy loss on preference data:

$\mathcal{L}_{\mathrm{reward}}(r)=-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\ln\left(\frac{e^{r(y)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{r(y^{\prime})}}\right).$
(7)

The reward function $r_{\psi}$ could, for example, be linear in a vector
message embedding $\phi(x)$, taking the form
$\widehat{r}_{\psi}(x)=\psi^{\top}\phi(x)$. Alternatively,
$\widehat{r}_{\psi}$ could be a neural network with parameters $\psi$.

For any $r:\mathcal{M}\rightarrow\Re$ and language model $\pi$, we define a
policy loss function

$\mathcal{L}_{\mathrm{policy}}(\pi|r)=-|\mathcal{D}|\sum_{x\in\mathcal{M}}P_{\pi}(x)r(x)+\beta\mathbf{d}_{\mathrm{KL}}(P_{\pi}\|P_{\overline{\pi}}),$
(8)

for some scalar hyperparameter $\beta>0$. The first term of this loss function
expresses expected reward while the second regularizes toward the base
language model.

A fine-tuned language model is obtained via two steps. The first minimizes
$\mathcal{L}_{\mathrm{reward}}$ to produce a reward function
$\widehat{r}_{\widehat{\psi}}$. This is followed by a policy optimization
step, which, given a language model $\widehat{\pi}_{\theta}$ parameterized by
a vector $\theta$, minimizes
$\mathcal{L}_{\mathrm{policy}}(\cdot|\widehat{r}_{\widehat{\psi}})$ to produce
the fine-tuned language model $\widehat{\pi}_{\widehat{\theta}}$. This
minimization is typically carried out via stochastic gradient descent starting
with $\theta$ initialized to match the base policy
$\widehat{\pi}_{\theta}=\overline{\pi}$.

###  4.2 Direct Preference Optimization

Direct preference optimization (DPO) approximates RLPO while bypassing the
reward modeling step [12]. The single-step process can be derived by first
observing that if $\pi$ minimizes $\mathcal{L}_{\mathrm{policy}}(\cdot|r)$
then

$P_{\pi}(x)=\frac{P_{\overline{\pi}}(x)e^{r(x)|\mathcal{D}|/\beta}}{\sum_{x^{\prime}\in\mathcal{M}}P_{\overline{\pi}}(x^{\prime})e^{r(x^{\prime})|\mathcal{D}|/\beta}},$
(9)

and therefore,

$r(x)=\frac{\beta}{|\mathcal{D}|}\ln\frac{P_{\pi}(x)}{P_{\overline{\pi}}(x)}+\frac{\beta}{|\mathcal{D}|}\ln\sum_{x^{\prime}\in\mathcal{M}}P_{\overline{\pi}}(x^{\prime})e^{r(x^{\prime})|\mathcal{D}|/\beta}.$
(10)

Substituting $r$ in (7) with the right-hand-side of (10) yields a new loss
function

$\mathcal{L}_{\mathrm{DPO}}(\pi)=-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\ln\frac{\left(P_{\pi}(y)/P_{\overline{\pi}}(y)\right)^{\beta/|\mathcal{D}|}}{\sum_{y^{\prime}\in\mathcal{Y}}\left(P_{\pi}(y^{\prime})/P_{\overline{\pi}}(y^{\prime})\right)^{\beta/|\mathcal{D}|}}.$
(11)

For a parameterized language model $\widehat{\pi}_{\theta}$, minimizing
$\mathcal{L}_{\mathrm{DPO}}$ produces a fine-tuned language model
$\widehat{\pi}_{\widehat{\theta}}$.

As we will see, RLPO and DPO both can give rise to egregious behavior when
queries include more than two alternatives. Recent work proposes a family of
RLHF algorithms that unifies and extends RLPO and DPO [3]. Other so-called
$\Psi$PO algorithms suffer in the same manner when queries include more than
two alternatives.

###  4.3 Inclusive Learning

RLPO and DPO maximize reward. This tends toward monolithic language models
that serve a majority of the population. Inclusive learning (IL) offers an
alternative approach to RLHF, aiming to produce inclusive language models that
reflect the diversity of preferences across the population [16].

IL produces a language model $\pi$ that simultaneously serves as a reward
model. In particular, reward is taken to be the log-probability $\ln
P_{\pi}(x)$ assigned to a message. Minimizing cross-entropy loss while
regularizing toward the base policy $\overline{\pi}$ gives rise to an
inclusive loss function

$\mathcal{L}_{\mathrm{IL}}(\pi)=-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\ln\frac{P_{\pi}(y)}{\sum_{y^{\prime}\in\mathcal{Y}}P_{\pi}(y^{\prime})}+\beta\mathbf{d}_{\mathrm{KL}}(P_{\pi}\|P_{\overline{\pi}}).$
(12)

For a parameterized language model $\widehat{\pi}_{\theta}$, minimizing
$\mathcal{L}_{\mathrm{IL}}$ produces a fine-tuned language model
$\widehat{\pi}_{\widehat{\theta}}$. While this IL algorithm was introduced in
[16], it is closely related to other approaches proposed in the literature
[18, 6], which share in the merits and faults of IL that we will discuss.

##  5 Perverse Incentives

We will explain how RLPO, DPO, and IL can fail to produce desirable results
due to the fact that underlying models assume IIA. Our explanations build on
theoretical and computational results. These results assume simple processes
for generation of preference data, which we now describe. In the cases of RLPO
and DPO, failure modes arise only when queries include more than two
alternative messages.

###  5.1 Dichotomy Data

Our simulation and theoretical results assume a particular data generating
process, articulated by the following assumption.

######  Assumption 1 (dichotomy data).

Elements of $\mathcal{D}$ are sampled iid. For each datum
$(\mathcal{Y},Y)\in\mathcal{D}$, the choice $Y$ is sampled according to a
dichotomy model $(\mathcal{Z},p_{*},r_{*})$. For any
$(\mathcal{Y},Y),(\mathcal{Y}^{\prime},Y^{\prime})\in\mathcal{D}$,
$|\mathcal{Y}|=|\mathcal{Y}^{\prime}|$. Each element of each set $\mathcal{Y}$
of alternatives is sampled independently by a language model $\overline{\pi}$.
For any message category $\tau\in\\{1,2\\}$ and messages
$y,y^{\prime}\in\mathcal{M}_{\tau}$,
$P_{\overline{\pi}}(y)=P_{\overline{\pi}}(y^{\prime})$.

Recall that, for the dichotomy model, there are two individual types,
$\mathcal{Z}=\\{1,2\\}$, and $r_{*}(x|z)=\mathbbm{1}_{\mathcal{M}_{z}}(x)$.
Under our assumption, each tuple $(\mathcal{Y},y)$ can be viewed as generated
as follows. First, each element of $\mathcal{Y}$ is generated by sampling
$\tau$ and then a message uniformly from $\mathcal{M}_{\tau}$. The
distribution of $\tau$ is implied by the language model $\overline{\pi}$.
Then, $Z$ is sampled from $p_{*}$ and $Y$ is sampled uniformly from
$\mathcal{Y}\cap\mathcal{M}_{Z}$.

###  5.2 Architectures

Each RLHF algorithm we have described operates by minimizing one or more loss
functions. The argument of each is itself a function. Optimization is carried
out by tuning parameters of an approximation architecture taking the form, for
example, of a neural network. Some of our results pertain to particular simple
architectures chosen to produce transparent analyses specifically for data
satisfying Assumption 1. Our next two assumptions describe these
architectures. The first pertains to the reward function architecture.

######  Assumption 2 (reward architecture).

For each $\psi\in\Re^{2}$, $z\in\mathcal{Z}$, and $x\in\mathcal{M}_{z}$,
$\widehat{r}_{\psi}(x)=\psi_{z}$.

Under this assumption, each reward function $r_{\psi}$ is parameterized by two
scalars – $\psi_{1}$ and $\psi_{2}$ – which express estimates of rewards
enjoyed by individuals who receive their desired type of message.

Our next assumption pertains to the policy architecture.

######  Assumption 3 (policy architecture).

For each $\theta\in\Re^{2}$, $z\in\mathcal{Z}$, and $x\in\mathcal{M}_{z}$,
$P_{\widehat{\pi}_{\theta}}(x)=e^{\theta_{z}}/(|\mathcal{M}_{1}|e^{\theta_{1}}+|\mathcal{M}_{2}|e^{\theta_{2}})$.

Under this assumption, each policy is identified by two scalars – $\theta_{1}$
and $\theta_{2}$. Increasing either $\theta_{1}$ or $\theta_{2}$ increases the
chances of generating messages of the corresponding type.

###  5.3 Simulation Setup

Our simulations are carried out with data and architectures that satisfy
Assumptions 1, 2, and 3. Message sets are of cardinality
$|\mathcal{M}_{1}|=10$ and $|\mathcal{M}_{2}|=100$ unless noted otherwise. The
choice model type distribution and the reward function are given by
$p_{*}(1)=0.6$, $p_{*}(2)=0.4$, and
$r_{*}(x|z)=\mathbbm{1}_{\mathcal{M}_{z}}(x)$. The base language model
$\overline{\pi}$ satisfies $P_{\overline{\pi}}(x)=0.8/|\mathcal{M}_{1}|$ for
$x\in\mathcal{M}_{1}$ and $P_{\overline{\pi}}(x)=0.2/|\mathcal{M}_{2}|$ for
$x\in\mathcal{M}_{2}$. Hence, a dominant fraction $p_{*}(1)>p_{*}(2)$ of the
population is of type $1$, individuals of that type prefer messages in
$\mathcal{M}_{1}$, and the baseline $\overline{\pi}$ tends to generate type
$1$ messages an even larger fraction of the time than $p_{*}(1)$. We use a
regularization penalty coefficient of $\beta=1$. Under these circumstances, it
is surprising that, as we will see, RLHF algorithms can produce language
models that almost always generate messages of type $2$.

###  5.4 RLPO and DPO

RLPO and DPO are designed to produce language models that generate desired
messages. As such, we should expect language models produced by these RLHF
algorithms to gravitate toward messages in $\mathcal{M}_{1}$, which are
preferred by $60\%$ of the population. As can be seen in Figure 2, this is
indeed the case when each choice set $|\mathcal{Y}|$ contains two messages. In
particular, for sufficiently large datasets, language models produced by RLPO
and DPO consistently generate elements of $\mathcal{M}_{1}$. However, with
larger choice sets, the language models consistently generate elements of
$\mathcal{M}_{2}$.111Larger choice sets are common, for example, in
applications where a language model is used to suggest alternative messages
for use by a human agent who is assisting a user. In such contexts, the human
agent selects one of the alternatives or manually crafts a response. The human
agent’s choice serves as feedback that can be used to train the language model
in order to improve subsequent suggestions.

Figure 2: For choice sets $\mathcal{Y}$ containing two messages, as
$\mathcal{D}$ grows, RLPO and DPO produce language models that consistently
generate messages most likely to be preferred. However, with larger choice
sets, less preferred messages are consistently generated. Each plot is
averaged over one hundred independent simulations.

It may seem surprising that RLPO and DPO fail so egregiously when choice sets
include more than a pair of alternatives, and it is natural to wonder whether
this could be due to technical details of our simulation. However, the
following theoretical results establish in greater generality that minimizing
RLPO loss functions can lead to egregious outcomes.

######  Proposition 4 (RLPO failure).

Under Assumptions 1, 2, and 3, for all $|\mathcal{Y}|\geq 3$, if
$p_{*}(1)<F(P_{\overline{\pi}}(\mathcal{M}_{1}))$ with
$F(\zeta)=\frac{\zeta-\zeta^{|\mathcal{Y}|}}{1-\zeta^{|\mathcal{Y}|}-(1-\zeta)^{|\mathcal{Y}|}}$,
then, as $|\mathcal{D}|\rightarrow\infty$,

$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})\stackrel{{\scriptstyle
p.}}{{\longrightarrow}}0\qquad\text{and}\qquad
P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})\stackrel{{\scriptstyle
p.}}{{\longrightarrow}}1,$

where $\widehat{\theta}$ minimizes
$\mathcal{L}_{\mathrm{policy}}(\widehat{\pi}_{\theta}|\widehat{r}_{\widehat{\psi}})$,
and
$\widehat{\psi}\in\operatorname*{arg\,min}_{\psi}\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$
if the loss $\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$ has a
minimizer, and $\widehat{\psi}=0$ otherwise.

Given that DPO is designed to approximate RLPO, one would expect a similar
theoretical result to hold for DPO. The following proposition formally
establishes this.

######  Proposition 5 (DPO failure).

Under Assumptions 1, 2, and 3, for all $|\mathcal{Y}|\geq 3$, if
$p_{*}(1)<F(P_{\overline{\pi}}(\mathcal{M}_{1}))$ with
$F(\zeta)=\frac{\zeta-\zeta^{|\mathcal{Y}|}}{1-\zeta^{|\mathcal{Y}|}-(1-\zeta)^{|\mathcal{Y}|}}$,
then as $|\mathcal{D}|\rightarrow\infty$,

$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})\stackrel{{\scriptstyle
p.}}{{\longrightarrow}}0\qquad\text{and}\qquad
P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})\stackrel{{\scriptstyle
p.}}{{\longrightarrow}}1,$

where
$\widehat{\theta}\in\operatorname*{arg\,min}_{\theta}\mathcal{L}_{\mathrm{DPO}}(\widehat{\pi}_{\theta})$
if the loss $\mathcal{L}_{\mathrm{DPO}}(\widehat{\pi}_{\theta})$ has a
minimizer, and $\widehat{\theta}=0$ otherwise.

To understand what causes these failures, let us consider as a thought
experiment a simplified data generating process where each choice set contains
the same triple $\mathcal{Y}=\\{y_{1},y_{2},y_{3}\\}$, with
$y_{1},y_{2}\in\mathcal{M}_{1}$ and $y_{3}\in\mathcal{M}_{2}$. Since
$p_{*}(1)=0.6$ and $p_{*}(2)=0.4$, choice probabilities generating the
preference data are given by
$\mathbb{P}(Y=1|\mathcal{Y})=\mathbb{P}(Y=2|\mathcal{Y})=0.3$ and
$\mathbb{P}(Y=3|\mathcal{Y})=0.4$. As the dataset grows, minimizing
$\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$ identifies parameters
$\widehat{\psi}$ to match these probabilities, if possible. In particular,

$\frac{e^{\widehat{r}_{\widehat{\psi}}(y_{1})}}{\sum_{i=1}^{3}e^{\widehat{r}_{\widehat{\psi}}(y_{i})}}=\frac{e^{\widehat{r}_{\widehat{\psi}}(y_{2})}}{\sum_{i=1}^{3}e^{\widehat{r}_{\widehat{\psi}}(y_{i})}}=0.3\qquad\text{and}\qquad\frac{e^{\widehat{r}_{\widehat{\psi}}(y_{3})}}{\sum_{i=1}^{3}e^{\widehat{r}_{\widehat{\psi}}(y_{i})}}=0.4.$
(13)

These equations imply that $\widehat{\psi}_{2}-\widehat{\psi}_{1}=\ln(4/3)>0$.
Hence, the estimated reward $\widehat{r}_{\widehat{\psi}}(x)$ is maximized, as
RLPO aims to do, by generating messages $x\in\mathcal{M}_{2}$. DPO is designed
to approximate RLPO and therefore leads to similar behavior.

In our thought experiment, the fact that each choice set had twice as many
more desirable ($\mathcal{M}_{1}$) than the less desirable ($\mathcal{M}_{2}$)
messages biased reward estimates in favor of $\mathcal{M}_{2}$. This tendency
is expressed in the above propositions though the requirement that
$F(P_{\overline{\pi}}(\mathcal{M}_{1}))>p_{*}(1)$, with $F(\cdot)$ defined in
Propositions 4 and 5. The baseline policy $\overline{\pi}$ is used to generate
choice sets, and perhaps surprisingly, the fact that it biases samples toward
desirable messages leads to undesirable outcomes.

The root cause is that $\widehat{r}_{\widehat{\psi}}$ assumes IIA while the
process generating preferences does not. In our thought experiment, for
example, where $\mathbb{P}(Y=1|\mathcal{Y})=\mathbb{P}(Y=2|\mathcal{Y})=0.3$
and $\mathbb{P}(Y=3|\mathcal{Y})=0.4$, if the choice sets were instead to only
contain two alternatives $\mathcal{Y}=\\{y_{2},y_{3}\\}$ of different types,
then we would have $\mathbb{P}(Y=2|\mathcal{Y})=0.6$ and
$\mathbb{P}(Y=3|\mathcal{Y})=0.4$, giving rise to different equations

$\frac{e^{\widehat{r}_{\widehat{\psi}}(y_{2})}}{\sum_{i=2}^{3}e^{\widehat{r}_{\widehat{\psi}}(y_{i})}}=0.6\qquad\text{and}\qquad\frac{e^{\widehat{r}_{\widehat{\psi}}(y_{3})}}{\sum_{i=2}^{3}e^{\widehat{r}_{\widehat{\psi}}(y_{i})}}=0.4.$
(14)

These new equations imply that
$\widehat{\psi}_{2}-\widehat{\psi}_{1}=\ln(2/3)<0$, correctly identifying
$\mathcal{M}_{1}$ as more desirable than $\mathcal{M}_{2}$. The irrelevant
alternative $y_{1}$ is able to distort estimates because the reward model
assumes IIA.

###  5.5 Inclusive Learning

IL is designed to produce language models that reflect the diversity of
preferences across the population [16]. In particular, IL ought to generate
messages in $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$ according to the
probabilities $0.6$ and $0.4$ with which they are preferred by random
individuals. As can be seen in Figure 3, this is indeed the case when
$|\mathcal{Y}|=2$ and $\mathcal{M}_{1}=\mathcal{M}_{2}=10$. However, as the
set $\mathcal{M}_{2}$ of less desired messages grows, the probability
$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})$ of generating a more
desired message vanishes.

Figure 3: Inclusive learning tends to generate a message in the less desired
set $\mathcal{M}_{2}$ as that set grows.

The following proposition formalizes this phenomenon, establishing that IL
fails as $\mathcal{D}$ and $\mathcal{M}_{2}$ grow, not just for the specific
simulated examples, but more generally.

######  Proposition 6 (IL failure).

Under Assumptions 1 and 3, if $|\mathcal{Y}|\geq 2$ and $\widehat{\theta}$
minimizes $\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})$, then for fixed
$\mathcal{M}_{1}$,

$\lim_{|\mathcal{D}|\to\infty}\mathbb{P}\left(\lim_{|\mathcal{M}_{2}|\to\infty}P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})=0\right)=1.$
(15)

To understand what causes these failures, let us consider another thought
experiment. Suppose that each choice set contains two messages
$\mathcal{Y}=\\{y_{1},y_{2}\\}$, with $y_{1}\in\mathcal{M}_{1}$ and
$y_{2}\in\mathcal{M}_{2}$. Choice probabilities generating the preference data
are given by $\mathbb{P}(Y=1|\mathcal{Y})=p_{*}(1)=0.6$ and
$\mathbb{P}(Y=2|\mathcal{Y})=p_{*}(2)=0.4$. As the dataset grows, minimizing
$\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})$ identifies parameters
$\widehat{\theta}$ to match these probabilities, if possible. In particular,

$\frac{P_{\widehat{\pi}_{\widehat{\theta}}}(y_{1})}{P_{\widehat{\pi}_{\widehat{\theta}}}(y_{1})+P_{\widehat{\pi}_{\widehat{\theta}}}(y_{2})}=0.6\qquad\text{and}\qquad\frac{P_{\widehat{\pi}_{\widehat{\theta}}}(y_{2})}{P_{\widehat{\pi}_{\widehat{\theta}}}(y_{1})+P_{\widehat{\pi}_{\widehat{\theta}}}(y_{2})}=0.4.$
(16)

This implies that
$P_{\widehat{\pi}_{\widehat{\theta}}}(y_{1})/P_{\widehat{\pi}_{\widehat{\theta}}}(y_{2})=3/2$.
Since
$P_{\widehat{\pi}_{\widehat{\theta}}}(y_{i})=P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{i})/|\mathcal{M}_{i}|$
for $i\in\\{1,2\\}$, we have

$\frac{3}{2}=\frac{P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})/|\mathcal{M}_{1}|}{P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})/|\mathcal{M}_{2}|}=\frac{|\mathcal{M}_{2}|}{10}\cdot\frac{P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})}{1-P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})}.$
(17)

Hence,
$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})=15/(15+|\mathcal{M}_{2}|)$.
It follows that, as $\mathcal{M}_{2}$ grows,
$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})$ vanishes.

While the number of messages in $\mathcal{M}_{2}$ does not influence how a
human would choose between two messages, (17) implies that it impacts the
probabilities $P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})$ and
$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})$ of generating more or
less liked messages. The more equivalent messages there are in
$\mathcal{M}_{2}$, the more likely the resulting fine-tuned language model is
to produce less-liked messages. This is again due to the fact that the model
underlying IL satisfies IIA while the preference data generating process does
not.

This observation suggests that using IL to fine-tune real language models will
induce a bias toward generating longer messages. This is because the number of
ways to express information scales quickly with the length of message required
to express that information. To see why, suppose there are two ideas that
might be expressed as responses to a prompt, and that the first requires a
about ten words, while the second requires about one thousand words. Suppose
that $60\%$ of the population would prefer the first idea while $40\%$ would
prefer the second. Because there are so many more thousand word than ten word
statements, there are likely to be many more roughly equivalent expressions of
the second idea than the first. Since IL biases generation toward messages
with many equivalent alternatives, the fine-tuned language model would tend to
express the second idea.

##  6 Empirical Study

We next demonstrate that the sort of egregious behavior we have identified
manifests when learning reward models of practical scale from realistic
datasets. In particular, we fit reward models that build on the PaLM 2 [5] XXS
language model to data generated using GPT-3.5 and GPT-4 [10]. Each query
includes reponses generated by GPT-3.5 and GPT-4. We additionally use GPT-4 to
simulate choices made by human annotators.

Our results establish that, when training queries each include a pair of
responses, the reward model correctly learns to assign higher scores to GPT-4
responses. However, when training queries each include four responses, the
reward model erroneously assigns favorable scores to GPT-3.5 responses. It is
striking that a seemingly innocuous change to the training query format gives
rise to such egregious behavior. The results are summarized in Figure 4.
Before discussing them in detail, we will describe the datasets, the reward
model architecture, and the training algorithm.

(a) Test accuracy of a standard reward model trained on preference data with 2
or 4 responses per query. The dashed line indicates a purely random baseline.

(b) Percentage of test data where the reward model assigns a higher score to
GPT-4’s response.

Figure 4: Standard reward model training can lead to egregious outcomes when
training data involves more than two responses per query.

###  6.1 Preference Datasets

For simplicity, we consider only a single prompt. While other prompts lead to
similar results, we arbitrarily choose to present results produced by the
following prompt, based on a popular movie this year:

> prompt Did Oppenheimer win a Nobel Prize?

We sample candidate responses using GPT-3.5 and GPT-4 with the temperature
parameter of each language model set to the default value of one. Here are
representative responses:

> GPT-3.5 response No, Oppenheimer did not win the Nobel Prize. GPT-4 response
> No, Robert Oppenheimer, often called the “father of the atomic bomb” for his
> role in the Manhattan Project, did not win a Nobel Prize.

GPT-3.5 tends to produce concise responses relative to the more informative
ones from GPT-4. From each of these two language models, we sample 100
responses for training and validation, and 100 responses for testing. We
subsample from these responses when constructing each training or validation
query, or each test sample. We produce two training sets that differ in the
number of alternative responses per query. The first training set has 2
responses per query, one from GPT-3.5 and one from GPT-4. The second training
set has 4 responses per query, one from GPT-3.5 and three from GPT-4. We
generate 800 queries for each training set. Additionally, we generate 200
queries for each validation set. These are used in tuning hyperparameters to
optimize validation accuracy. We produce a single test dataset made up of 1000
queries, each with one response from GPT-3.5 and one from GPT-4.

To simulate human annotator choices, we prompt GPT-4 to select its favorite
response among a set. We use two types of prompts, which express preference
for concise and informative responses respectively:

> concise choice prompt Suppose that you are looking for a concise answer to
> the following question. Which response below do you like the best…
> informative choice prompt Suppose that you are trying to learn more about
> the following question. Which response below do you like the best…

We find that prompting for a concise choice typically favors GPT-3.5, while
prompting for an informative choice typically favors GPT-4. When simulating a
human annotator, we sample one of these two types of choice prompts according
to probabilities 0.3 and 0.7, respectively. Using these choice prompts, we
find that regardless of the number of responses per query, our simulated human
annotators select responses from GPT-4 approximately 70% of the time.

###  6.2 Reward Model Architecture and Training Algorithm

Our reward model builds on the PaLM 2 [5] XXS language model. In particular,
given a context formed by concatenating a prompt and a response, we take the
final embedding produced by the base language model and apply a linear layer
to obtain reward. We train the linear layer, as well as the base language
model, to minimize cross-entropy loss on preference data. For each training
dataset, we train the reward model over 150 gradient steps using the Adafactor
optimizer [13] with a learning rate of 1e-4 and batch size of 16. The
hyperparameters are tuned to optimize validation accuracy. We average results
over 3 seeds.

###  6.3 Results

Figure 4(a) plots the test accuracy of the reward model trained on queries
with 2 or 4 responses. When each training query includes only a pair of
responses, the reward model correctly selects the preferred response for
around 70% of test samples. However, with four responses per training query,
the test accuracy drops below 50%. In other words, choice predictions
generated by the reward model fare worse than random guessing. Figure 4(b)
provides more insight into how learned reward models score responses. Since
the majority of simulated annotators favor GPT-4, we would expect that the
learned reward model assigns higher scores to GPT-4 responses. We see in
Figure 4(b) that, when trained on pairwise comparisons, the reward model
almost always assigns higher scores to GPT-4 responses. However, when trained
on queries each with 4 responses, the reward model tends to erroneously assign
higher scores to GPT-3.5 responses.

These results highlight a failing of the standard reward learning approach.
The culprit is the IIA assumption made in reward learning, which is violated
by the data as we will now explain. Intuitively, each simulated human
annotator prefers either concise or informative responses but is relatively
indifferent between alternatives in each of these two categories. Consider a
simulated annotator who, when presented with one concise and one informative
response, selects the former with probability 3/5. Under the IIA assumption,
increasing the number of informative alternatives from one to three would
reduce the probability of choosing the concise response to 1/3. However, if
the annotator is indifferent between the three informative responses, the
probability of choosing the concise response should remain 3/5, regardless of
whether there are one or three informative alternatives.

##  7 Closing Remarks

The essential role of RLHF in the success of generative AI has attracted
tremendous effort and resources to the subject. It is worthwhile to think
carefully about foundations of this methodology to inform algorithmic
innovation and ultimately produce reliable AIs. What we have presented, which
points out how the IIA assumption adopted by models underlying current
algorithms can give rise to egregious behavior, offers one of what will
hopefully be many steps in this direction.

The design of RLHF algorithms that mitigate perverse incentives we have
identified remains a subject for future research. Trade-offs between query
formats and how feedback is processed also deserve further study. For example,
while RLPO can fail when queries present more than two alternatives, one can
heuristically convert a feedback from such a query to multiple pairwise
choices. This could offer more robust results, though there may be other flaws
to this approach.

##  Appendix A Proofs

###  A.1 RLPO

In this subsection, we provide the proof for Proposition 4. We start by
defining two key quantities for messages of category $1$ in $\mathcal{D}$.
First, we let

$\rho_{\mathrm{chosen}}=\frac{1}{|\mathcal{D}|}\sum_{(\mathcal{Y},y)\in\mathcal{D}}\mathbbm{1}\left(y\in\mathcal{M}_{1}\right)$

denote the fraction of data in $\mathcal{D}$ where the chosen message is in
$\mathcal{M}_{1}$. Second, we let

$\rho_{\mathrm{data}}=\frac{1}{|\mathcal{D}|}\sum_{(\mathcal{Y},y)\in\mathcal{D}}\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}|}$

denote the fraction of messages in $\mathcal{D}$ that belong to
$\mathcal{M}_{1}$.

######  Lemma 7.

For all strictly convex functions $f:\Re\to\Re$ such that $f^{\prime}(1)<0$,
if $f(\zeta)=0$ for some $\zeta\in\Re$, then
$\zeta>1-\frac{f(1)}{f^{\prime}(1)}$.

###### Proof.

Since $f$ is strictly convex, $f(\zeta)>f(1)+f^{\prime}(1)(\zeta-1)$. Since
$f^{\prime}(1)<0$, this implies $\zeta-1>-\frac{f(1)}{f^{\prime}(1)}$. ∎

######  Lemma 8.

Under Assumption 2, if $\rho_{\mathrm{data}}>\rho_{\mathrm{chosen}}>0$, then
the minimizer $\widehat{\psi}$ of
$\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$ exists and satisfies

$\widehat{\psi}_{2}-\widehat{\psi}_{1}>\frac{\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}}{1+\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}}.$

###### Proof.

Since $\mathcal{L}_{\mathrm{reward}}$ is invariant under translations of
$\psi$, without loss of generality, set $\psi_{1}=0$.

$\displaystyle\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$
$\displaystyle=-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\ln\frac{e^{\widehat{r}_{\psi}(y)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{\widehat{r}_{\psi}(y^{\prime})}}=-\psi_{2}|\mathcal{D}|(1-\rho_{\mathrm{chosen}})+\sum_{(\mathcal{Y},y)\in\mathcal{D}}\ln\left(|\mathcal{Y}\cap\mathcal{M}_{1}|+|\mathcal{Y}\cap\mathcal{M}_{2}|e^{\psi_{2}}\right).$

Note that this loss function is strictly convex and thus a local minima is
also a global minima. Examining the derivative, we see that
$\widehat{\psi}_{2}$, if exists, satisfies

$\sum_{(\mathcal{Y},y)\in\mathcal{D}}\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}\cap\mathcal{M}_{1}|+|\mathcal{Y}\cap\mathcal{M}_{2}|e^{\widehat{\psi}_{2}}}-|\mathcal{D}|\rho_{\mathrm{chosen}}=0.$

With a change of variable, let $x=e^{\psi_{2}}$ and

$f(x)=\sum_{(\mathcal{Y},y)\in\mathcal{D}}\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}\cap\mathcal{M}_{1}|+|\mathcal{Y}\cap\mathcal{M}_{2}|x}-|\mathcal{D}|\rho_{\mathrm{chosen}}.$

Clearly, $f(x)$ is strictly decreasing on $\Re_{+}$ and $f(x)\to 0$ as
$x\to\infty$. Since $\rho_{\mathrm{chosen}}>0$, there exists a unique
$\zeta>0$ such that $f(\zeta)=0$. Thus, $\widehat{\psi}_{2}=\ln\zeta$ exists.
Further, we note that $f(x)$ is strictly convex and $f^{\prime}(1)<0$. By
Lemma 7,

$\zeta>1-\frac{f(1)}{f^{\prime}(1)}=1+|\mathcal{D}|(\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}})\cdot|\mathcal{Y}|^{2}\left(\sum_{(\mathcal{Y},y)\in\mathcal{D}}|\mathcal{Y}\cap\mathcal{M}_{1}|\cdot|\mathcal{Y}\cap\mathcal{M}_{2}|\right)^{-1}>1+\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}.$
(18)

The result follows from combining Equation (18) and the fact that
$\ln(1+\alpha)\geq\frac{\alpha}{1+\alpha}$ for all $\alpha>-1$.

∎

For all $\eta>0$, define the high probability event

$\mathcal{E}_{\eta}(\mathcal{D})=\left\\{\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}>\eta\right\\}\bigcap\left\\{\rho_{\mathrm{chosen}}>0\right\\}.$
(19)

######  Lemma 9.

Under Assumptions 2 and 3, if there exists $\eta>0$ such that
$\mathbb{P}(\mathcal{E}_{\eta}(\mathcal{D}))\to 1$ as
$|\mathcal{D}|\to\infty$, then as $|\mathcal{D}|\to\infty$,

$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})\stackrel{{\scriptstyle
p.}}{{\longrightarrow}}0\qquad\text{and}\qquad
P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})\stackrel{{\scriptstyle
p.}}{{\longrightarrow}}1,$

where $\widehat{\theta}$ minimizes
$\mathcal{L}_{\mathrm{policy}}(\widehat{\pi}_{\theta}|\widehat{r}_{\widehat{\psi}})$,
and
$\widehat{\psi}\in\operatorname*{arg\,min}_{\psi}\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$
if the loss $\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$ has a
minimizer, and $\widehat{\psi}=0$ otherwise.

###### Proof.

By Lemma 8, under event $\mathcal{E}_{\eta}(\mathcal{D})$,
$\widehat{\psi}=\operatorname*{arg\,min}_{\psi}\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$
and
$\widehat{\psi}_{2}-\widehat{\psi}_{1}>\frac{\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}}{1+\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}}>\frac{\eta}{1+\eta}$.
When $\beta=0$, $\widehat{\pi}_{\theta}$ maximizes reward. Thus, under event
$\mathcal{E}_{\eta}(\mathcal{D})$,
$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})=0$ and
$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})=1$. When $\beta>0$,
$\widehat{\pi}_{\theta}$ satisfies

$P_{\widehat{\pi}_{\theta}}(y)=\frac{P_{\overline{\pi}}(y)e^{\widehat{r}_{\widehat{\psi}}(y)|\mathcal{D}|/\beta}}{\sum_{y^{\prime}\in\mathcal{M}}P_{\overline{\pi}}(y^{\prime})e^{\widehat{r}_{\widehat{\psi}}(y^{\prime})|\mathcal{D}|/\beta}}.$

Under event $\mathcal{E}_{\eta}(\mathcal{D})$,

$\frac{P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{1})}{P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{2})}=\frac{p_{1}}{p_{2}}\cdot
e^{\left(\widehat{\psi}_{1}-\widehat{\psi}_{2}\right)|\mathcal{D}|/\beta}<e^{-\frac{\eta}{1+\eta}\cdot\frac{|\mathcal{D}|}{\beta}}.$

For all $\epsilon>0$, there exists $N>0$ such that
$e^{-\frac{\eta}{1+\eta}\cdot\frac{|\mathcal{D}|}{\beta}}<\epsilon$ for all
$|\mathcal{D}|>N$. In turn, for all $\beta\geq 0$ and $\epsilon>0$, there
exists $N>0$ such that for all $|\mathcal{D}|>N$,

$\mathbb{P}\left(\frac{P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{1})}{P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{2})}\geq\epsilon\right)\leq\mathbb{P}\left(\mathcal{E}_{\eta}(\mathcal{D})^{c}\right).$

The proof follows from the fact that
$\mathbb{P}(\mathcal{E}_{\eta}(\mathcal{D})^{c})\to 0$ as
$|\mathcal{D}|\to\infty$ and
$P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{1})+P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{2})=1$.

∎

Finally, we prove that under the dichotomy data assumption, there exists
$\eta>0$ such that the event $\mathcal{E}_{\eta}(\mathcal{D})$ holds with high
probability. Define

$\mathcal{I}=\big{\\{}(\mathcal{Y},y)\in\mathcal{D}\,|\,\mathcal{Y}\cap\mathcal{M}_{1}\neq\emptyset,\,\mathcal{Y}\cap\mathcal{M}_{2}\neq\emptyset\big{\\}}$

as the subset of $\mathcal{D}$ that contains messages in both categories. The
first lemma below shows that most of the data contain both categories of
messages with high probability.

######  Lemma 10.

Under Assumption 1, if $P_{\overline{\pi}}(\mathcal{M}_{1})\in(0,1)$,
$|\mathcal{Y}|\geq 2$, then there exists a constant
$\gamma=\gamma(|\mathcal{Y}|)\in(0,1)$ such that for all $|\mathcal{D}|>0$,
$\mathbb{P}(|\mathcal{I}|\leq|\mathcal{D}|(1-\gamma))\leq
e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}$.

###### Proof.

Under Assumption 1, for each $(\mathcal{Y},y)\in\mathcal{D}$, the membership
$\tau$ of each message $y^{\prime}\in\mathcal{Y}$ follows
$\mathrm{Bernoulli}(P_{\overline{\pi}}(\mathcal{M}_{1}))$. Let
$s=1-P_{\overline{\pi}}(\mathcal{M}_{1})^{|\mathcal{Y}|}-P_{\overline{\pi}}(\mathcal{M}_{2})^{|\mathcal{Y}|}<1$.
Then $s$ equals the probability of sampling both categories of messages in
each datum, and $|\mathcal{I}|\sim\mathrm{Binomial}(|\mathcal{D}|,s)$. By
Chernoff’s bound,

$\mathbb{P}\left(|\mathcal{I}|\leq s\cdot|\mathcal{D}|s\right)\leq
e^{-\frac{|\mathcal{D}|s(1-s)^{2}}{2}}.$

Letting $\gamma=1-s^{2}\in(0,1)$, we have $s(1-s)^{2}>(1-\gamma)\gamma^{2}/4$.
Therefore,

$\mathbb{P}\left(|\mathcal{I}|\leq|\mathcal{D}|(1-\gamma)\right)\leq
e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}.$ (20)

∎

######  Lemma 11.

Under Assumption 1, for all $|\mathcal{Y}|\geq 3$, if
$p_{*}(1)<F(P_{\overline{\pi}}(\mathcal{M}_{1}))$ with
$F(\zeta)=\frac{\zeta-\zeta^{|\mathcal{Y}|}}{1-\zeta^{|\mathcal{Y}|}-(1-\zeta)^{|\mathcal{Y}|}}$,
then there exists $\eta>0$ such that
$\mathbb{P}(\mathcal{E}_{\eta}(\mathcal{D}))\to 1$ as
$|\mathcal{D}|\to\infty$.

###### Proof.

Let $\delta=F(P_{\overline{\pi}}(\mathcal{M}_{1}))-p_{*}(1)>0$. Under
Assumption 1,
$|\mathcal{Y}\cap\mathcal{M}_{1}|\sim\mathrm{Binomial}(|\mathcal{Y}|,P_{\overline{\pi}}(\mathcal{M}_{1}))$,
giving

$\mathbb{E}\left[\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}|}\mid(\mathcal{Y},y)\in\mathcal{I}\right]=\frac{P_{\overline{\pi}}(\mathcal{M}_{1})-P_{\overline{\pi}}(\mathcal{M}_{1})^{|\mathcal{Y}|}}{1-P_{\overline{\pi}}(\mathcal{M}_{1})^{|\mathcal{Y}|}-P_{\overline{\pi}}(\mathcal{M}_{2})^{|\mathcal{Y}|}}=F(P_{\overline{\pi}}(\mathcal{M}_{1})).$

On the other hand,

$\mathbb{E}\left[\mathbbm{1}(y\in\mathcal{M}_{1})\mid(\mathcal{Y},y)\in\mathcal{I}\right]=p_{*}(1).$

For $(\mathcal{Y},y)\notin\mathcal{I}$, observe that
$\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}|}=\mathbbm{1}(y\in\mathcal{M}_{1})$.
Thus

$\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}|}-\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{1})=\frac{|\mathcal{D}|}{|\mathcal{I}|}(\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}).$
(21)

By Hoeffding’s inequality,

$\displaystyle\mathbb{P}\left(\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}|}-F(P_{\overline{\pi}}(\mathcal{M}_{1}))\leq-\frac{\delta}{2}\mid\mathcal{I}\right)$
$\displaystyle\leq e^{-\frac{|\mathcal{I}|\delta^{2}}{4}}$
$\displaystyle\mathbb{P}\left(\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{1})-p_{*}(1)\geq\frac{\delta}{4}\mid\mathcal{I}\right)$
$\displaystyle\leq e^{-\frac{|\mathcal{I}|\delta^{2}}{8}}$
$\displaystyle\mathbb{P}\left(\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{1})-p_{*}(1)\leq-\frac{p_{*}(1)}{2}\mid\mathcal{I}\right)$
$\displaystyle\leq e^{-\frac{|\mathcal{I}|p_{*}^{2}(1)}{2}}.$

A union bound gives

$\displaystyle\quad\mathbb{P}\left(\left\\{\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{1}|}{|\mathcal{Y}|}-\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{1})>\frac{\delta}{4}\right\\}\bigcap\left\\{\frac{1}{|\mathcal{I}|}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{1})>0\right\\}\mid\mathcal{I}\right)$
$\displaystyle>1-e^{-\frac{|\mathcal{I}|\delta^{2}}{4}}-e^{-\frac{|\mathcal{I}|\delta^{2}}{8}}-e^{-\frac{|\mathcal{I}|p_{*}^{2}(1)}{2}}.$

By Lemma 10, there exists $0<\gamma<1$ such that
$\mathbb{P}(|\mathcal{I}|\leq|\mathcal{D}|(1-\gamma))\leq
e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}$. By Equation (21),

$\displaystyle\quad\mathbb{P}\left(\left\\{\frac{|\mathcal{D}|}{|\mathcal{I}|}(\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}})>\frac{\delta}{4}\right\\}\bigcap\left\\{\frac{|\mathcal{D}|}{|\mathcal{I}|}\rho_{\mathrm{chosen}}>0\right\\}\bigcap\left\\{|\mathcal{I}|>|\mathcal{D}|(1-\gamma))\right\\}\right)$
$\displaystyle=\mathbb{P}\left(\left\\{\frac{|\mathcal{D}|}{|\mathcal{I}|}(\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}})>\frac{\delta}{4}\right\\}\bigcap\left\\{\frac{|\mathcal{D}|}{|\mathcal{I}|}\rho_{\mathrm{chosen}}>0\right\\}\mid|\mathcal{I}|>|\mathcal{D}|(1-\gamma))\right)\mathbb{P}\left(|\mathcal{I}|>|\mathcal{D}|(1-\gamma))\right)$
$\displaystyle>\left(1-e^{-\frac{|\mathcal{D}|(1-\gamma)\delta^{2}}{4}}-e^{-\frac{|\mathcal{D}|(1-\gamma)\delta^{2}}{8}}-e^{-\frac{|\mathcal{D}|(1-\gamma)p_{*}^{2}(1)}{2}}\right)\left(1-e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}\right)$
$\displaystyle>1-e^{-\frac{|\mathcal{D}|(1-\gamma)\delta^{2}}{4}}-e^{-\frac{|\mathcal{D}|(1-\gamma)\delta^{2}}{8}}-e^{-\frac{|\mathcal{D}|(1-\gamma)p_{*}^{2}(1)}{2}}-e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}.$

Let $\eta=\frac{(1-\gamma)\delta}{4}>0$. It follows that

$\displaystyle\mathbb{P}\left(\mathcal{E}_{\eta}(\mathcal{D})\right)$
$\displaystyle=\mathbb{P}\left(\left\\{\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}}>\eta\right\\}\bigcap\left\\{\rho_{\mathrm{chosen}}>0\right\\}\right)$
$\displaystyle\geq\mathbb{P}\left(\left\\{\frac{|\mathcal{D}|}{|\mathcal{I}|}(\rho_{\mathrm{data}}-\rho_{\mathrm{chosen}})>\frac{\delta}{4}\right\\}\bigcap\left\\{\frac{|\mathcal{D}|}{|\mathcal{I}|}\rho_{\mathrm{chosen}}>0\right\\}\bigcap\left\\{|\mathcal{I}|>|\mathcal{D}|(1-\gamma))\right\\}\right)$
$\displaystyle>1-e^{-\frac{|\mathcal{D}|(1-\gamma)\delta^{2}}{4}}-e^{-\frac{|\mathcal{D}|(1-\gamma)\delta^{2}}{8}}-e^{-\frac{|\mathcal{D}|(1-\gamma)p_{*}^{2}(1)}{2}}-e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}.$

The proof follows by taking $|\mathcal{D}|\to\infty$.

∎

We are now ready to prove the following failure case for RLPO.

See 4

###### Proof.

The proof follows from applying Lemmas 9 and 11. ∎

###  A.2 DPO

In Lemma 12, we show that DPO and RLPO produces the same policy, which then
allows us to directly apply the analysis for RLPO to reach the same conclusion
for DPO.

######  Lemma 12.

If there exists a minimizer $\widehat{r}$ to $\mathcal{L}_{\mathrm{reward}}$,
then the optimal policy for $\mathcal{L}_{\mathrm{DPO}}$ and for
$\mathcal{L}_{\mathrm{policy}}(\cdot|\widehat{r})$ are identical.

###### Proof.

Recall that for any given reward function $r$, the optimal policy $\pi_{r}$
for $\mathcal{L}_{\mathrm{policy}}$ satisfies

$P_{\pi_{r}}(x)=\frac{P_{\overline{\pi}}(x)e^{r(x)|\mathcal{D}|/\beta}}{\sum_{x^{\prime}\in\mathcal{M}}P_{\overline{\pi}}(x^{\prime})e^{r(x^{\prime})|\mathcal{D}|/\beta}}.$
(22)

It follows that

$\displaystyle-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\left[\ln\left(\frac{e^{r(y)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{r(y^{\prime})}}\right)\right]=-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\left[\ln\frac{\left(P_{\pi_{r}}(y)/P_{\overline{\pi}}(y)\right)^{\beta/|\mathcal{D}|}}{\sum_{y^{\prime}\in\mathcal{Y}}\left(P_{\pi_{r}}(y^{\prime})/P_{\overline{\pi}}(y^{\prime})\right)^{\beta/|\mathcal{D}|}}\right].$
(23)

Thus the value $\mathcal{L}_{\mathrm{reward}}(r)$ equals the value
$\mathcal{L}_{\mathrm{DPO}}(\pi_{r})$. Note also that $r\mapsto\pi_{r}$ is
surjective.

Suppose $\widehat{r}$ is the minimizer for $\mathcal{L}_{\mathrm{reward}}$.
Then $\pi_{\widehat{r}}$ is the optimal policy obtained by minimizing
$\mathcal{L}_{\mathrm{policy}}(\cdot|\widehat{r})$. If $\pi_{\widehat{r}}$ is
not optimal for $\mathcal{L}_{\mathrm{DPO}}$, then there exists another policy
$\pi^{\prime}$ that obtains a strictly lower loss
$\mathcal{L}_{\mathrm{DPO}}(\pi^{\prime})$. Since $r\mapsto\pi_{r}$ is
surjective, there exists a reward function $r^{\prime}$ such that
$\pi^{\prime}=\pi_{r^{\prime}}$. For example, we can take
$r^{\prime}(x)=\frac{\beta}{|\mathcal{D}|}\ln\frac{P_{\pi^{\prime}}(x)}{P_{\overline{\pi}}(x)}$,
which achieves a lower reward loss
$\mathcal{L}_{\mathrm{reward}}(r^{\prime})=\mathcal{L}_{\mathrm{DPO}}(\pi_{r^{\prime}})<\mathcal{L}_{\mathrm{DPO}}(\pi_{\widehat{r}})=\mathcal{L}_{\mathrm{reward}}(\widehat{r})$,
a contradiction.

Similarly, if $\pi^{*}$ is optimal for $\mathcal{L}_{\mathrm{DPO}}$, the
corresponding reward
$r^{*}(x)=\frac{\beta}{|\mathcal{D}|}\ln\frac{P_{\pi^{*}}(x)}{P_{\overline{\pi}}(x)}$
must be optimal for $\mathcal{L}_{\mathrm{reward}}$. The corresponding optimal
policy $\pi_{r^{*}}$ for $\mathcal{L}_{\mathrm{policy}}(\cdot|r^{*})$ then
satisfies

$P_{\pi_{r^{*}}}(x)\propto
P_{\overline{\pi}}(x)e^{\frac{\beta}{|\mathcal{D}|}\ln\frac{P_{\pi^{*}}(x)}{P_{\overline{\pi}}(x)}|\mathcal{D}|/\beta}=P_{\pi^{*}}(x),$

as desired.

∎

See 5

###### Proof.

By Lemma 12, if $\widehat{\psi}$ is as defined in Proposition 4, then
$\widehat{\psi}=\operatorname*{arg\,min}_{\psi}\mathcal{L}_{\mathrm{reward}}(\widehat{r}_{\psi})$
and we have
$\widehat{\theta}=\operatorname*{arg\,min}_{\theta}\mathcal{L}_{\mathrm{DPO}}(\widehat{\pi}_{\theta})=\operatorname*{arg\,min}_{\theta}\mathcal{L}_{\mathrm{policy}}(\widehat{\pi}_{\theta}|\widehat{r}_{\widehat{\psi}})$.
The proof follows from Proposition 4. ∎

###  A.3 Inclusive Learning

In this subsection, we prove Proposition 6. We start by defining two key sets
that describe messages in $\mathcal{D}$. Recall that in the proof for
Proposition 4, we defined

$\mathcal{I}=\big{\\{}(\mathcal{Y},y)\in\mathcal{D}\,|\,\mathcal{Y}\cap\mathcal{M}_{1}\neq\emptyset,\,\mathcal{Y}\cap\mathcal{M}_{2}\neq\emptyset\big{\\}}.$

To facilitate the exposition, we also define

$\mathcal{I}_{\tau}=\left\\{(\mathcal{Y},y)\in\mathcal{I}\mid
y\in\mathcal{M}_{\tau}\right\\}\quad\text{for }\tau=1,2$

to be the set of data in $\mathcal{I}$ where a message of category $\tau$ is
chosen. We first prove the following lemma that characterizes the optimal
solution for a class of convex loss functions.

######  Lemma 13.

Suppose that for all $m>0$, the loss $\mathcal{L}(\cdot;m):(0,1)\to\Re$ is
convex in $x$, $\mathcal{L}(x_{m};m)=\min_{x}\mathcal{L}(x;m)$, and there
exists $M>0$ such that for all $m>M$,

$\frac{\partial\mathcal{L}(x;m)}{\partial x}<0.$

Then $\lim_{m\to\infty}x_{m}=1$.

###### Proof.

Since $\mathcal{L}(x;m)$ is convex on $(0,1)$, its local minima is also a
global minima. For all $m>M$, $\mathcal{L}$ is strictly decreasing for
$x\in(0,1)$, so $\lim_{m\to\infty}x_{m}=1$. ∎

Then, we define the high probability event

$\mathcal{E}(\mathcal{D})=\left\\{|\mathcal{I}_{1}|>1\right\\}\cap\left\\{|\mathcal{I}_{2}|>\max\\{1+\beta,2\beta\ln\frac{p_{*}(2)}{p_{*}(1)}\\}\right\\}.$

######  Lemma 14.

Under Assumption 3, for all fixed $\mathcal{M}_{1}$ and $\mathcal{D}$ such
that $\mathcal{E}(\mathcal{D})$ holds, if $\widehat{\theta}$ minimizes
$\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})$, then

$P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})\rightarrow
0\qquad\text{and}\qquad
P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{2})\rightarrow 1$

as the size of $\mathcal{M}_{2}$ grows.

###### Proof.

First, we notice that $\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})$ is
strongly convex for $\beta>0$ and
$\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})\to\infty$ as
$\|\theta\|_{2}\to\infty$, thus a minimizer for
$\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})$ exists and is unique. Let
$m=\frac{|\mathcal{M}_{2}|}{|\mathcal{M}_{1}|}$ and
$q=P_{\widehat{\pi}_{\theta}}(\mathcal{M}_{2})$. The loss function can be
written in terms of $m$ and $q$ as

$\mathcal{L}(q,m)=\mathcal{L}_{\mathrm{IL}}(\widehat{\pi}_{\theta})=-\sum_{(\mathcal{Y},y)\in\mathcal{D}}\ln\frac{m(1-q)\cdot\mathbbm{1}_{\mathcal{M}_{1}}(y)+q\cdot\mathbbm{1}_{\mathcal{M}_{2}}(y)}{m(1-q)\cdot|\mathcal{Y}\cap\mathcal{M}_{1}|+q\cdot|\mathcal{Y}\cap\mathcal{M}_{2}|}+\beta\left[(1-q)\ln\frac{1-q}{p_{*}(1)}+q\ln\frac{q}{p_{*}(2)}\right].$
(24)

Taking the partial derivative with respect to $q$, under the event
$\mathcal{E}(\mathcal{D})$,

$\displaystyle\frac{\partial\mathcal{L}(q,m)}{\partial q}$
$\displaystyle=\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{2}|}{m(1-q)^{2}|\mathcal{Y}\cap\mathcal{M}_{1}|+q(1-q)|\mathcal{Y}\cap\mathcal{M}_{2}|}-\sum_{(\mathcal{Y},y)\in\mathcal{I}_{2}}\left(\frac{1}{q}+\frac{1}{1-q}\right)+\beta\left(\ln\frac{q}{1-q}+\ln\frac{p_{*}(1)}{p_{*}(2)}\right)$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{<}}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{2}|}{m(1-q)^{2}|\mathcal{Y}\cap\mathcal{M}_{1}|+q(1-q)|\mathcal{Y}\cap\mathcal{M}_{2}|}-\frac{|\mathcal{I}_{2}|}{q}-\frac{|\mathcal{I}_{2}|}{1-q}+\frac{\beta}{1-q}+\frac{|\mathcal{I}_{2}|}{2}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{<}}\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{2}|}{m(1-q)^{2}|\mathcal{Y}\cap\mathcal{M}_{1}|+q(1-q)|\mathcal{Y}\cap\mathcal{M}_{2}|}-\frac{|\mathcal{I}_{2}|}{2q}-\frac{1}{1-q},$

where $(a)$ follows from
$|\mathcal{I}_{2}|>2\beta\ln\frac{p_{*}(2)}{p_{*}(1)}$ and
$\ln\frac{q}{1-q}<\frac{1}{1-q}$, and $(b)$ follows from
$|\mathcal{I}_{2}|>1+\beta$ and $q<1$. For all $0<q<1$, the first term is
always positive and converges to 0 as $m\to\infty$. Thus, there exists an
$M>0$ such that for all $m>M$,
$\sum_{(\mathcal{Y},y)\in\mathcal{I}}\frac{|\mathcal{Y}\cap\mathcal{M}_{2}|}{m(1-q)^{2}|\mathcal{Y}\cap\mathcal{M}_{1}|+q(1-q)|\mathcal{Y}\cap\mathcal{M}_{2}|}<\frac{|\mathcal{I}_{2}|}{2q}$.
This implies that

$\frac{\partial\mathcal{L}(q,m)}{\partial q}<-\frac{1}{1-q}<0.$

The proof follows from applying Lemma 13.

∎

Finally, we prove that under the dichotomy data assumption, the event
$\mathcal{E}(\mathcal{D})$ holds with high probability.

######  Lemma 15.

Under Assumption 1, if $|\mathcal{Y}|\geq 2$, then for fixed
$\mathcal{M}_{1}$, $\mathbb{P}(\mathcal{E}(\mathcal{D}))\to 1$ as
$|\mathcal{M}_{2}|\to\infty$.

###### Proof.

By Lemma 10, there exists a constant $\gamma\in(0,1)$ such that for all
$|\mathcal{D}|>0$, $\mathbb{P}(|\mathcal{I}|\leq|\mathcal{D}|(1-\gamma))\leq
e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}$. Let
$\epsilon(\beta)=\max\\{\beta+1,2\beta\ln\frac{p_{*}(1)}{p_{*}(2)}\\}$, then
$\mathbb{E}[\mathbbm{1}(y\in\mathcal{M}_{2})\mid(\mathcal{Y},y)\in\mathcal{I}]=p_{*}(2)$.
By Hoeffding’s inequality,

$\displaystyle\mathbb{P}\left(\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{2})-|\mathcal{I}|\cdot
p_{*}(2)\leq\epsilon(\beta)-|\mathcal{I}|\cdot p_{*}(2)\mid\mathcal{I}\right)$
$\displaystyle\leq
e^{-2\frac{(\epsilon(\beta)-|\mathcal{I}|p_{*}(2))^{2}}{|\mathcal{I}|}}<e^{-2|\mathcal{I}|p_{*}^{2}(2)+4\epsilon(\beta)p_{*}(2)}$
$\displaystyle\mathbb{P}\left(\sum_{(\mathcal{Y},y)\in\mathcal{I}}\mathbbm{1}(y\in\mathcal{M}_{1})-|\mathcal{I}|\cdot
p_{*}(1)\leq 1-|\mathcal{I}|\cdot p_{*}(1)\mid\mathcal{I}\right)$
$\displaystyle\leq
e^{-2\frac{(1-|\mathcal{I}|p_{*}(2))^{2}}{|\mathcal{I}|}}<e^{-2|\mathcal{I}|p_{*}^{2}(2)+4p_{*}(2)}.$

Following a similar argument as that in the proof for Lemma 11, we have

$\displaystyle\mathbb{P}(\mathcal{E}(\mathcal{D}))$
$\displaystyle\geq\mathbb{P}\left(\left\\{|\mathcal{I}|>|\mathcal{D}|(1-\gamma)\right\\}\cap\left\\{|\mathcal{I}_{1}|>1\right\\}\cap\left\\{|\mathcal{I}_{2}|>\epsilon(\beta)\right\\}\right)$
$\displaystyle\geq
1-e^{-\frac{|\mathcal{D}|(1-\gamma)\gamma^{2}}{8}}-e^{-2|\mathcal{D}|(1-\gamma)p_{*}^{2}(2)+4\epsilon(\beta)p_{*}(2)}-e^{-2|\mathcal{D}|(1-\gamma)p_{*}^{2}(2)+4p_{*}(2)}.$

The proof follows by taking $|\mathcal{M}_{2}|\to\infty$.

∎

Finally, we prove the following failure case for inclusive learning. See 6

###### Proof.

The proof follows from applying Lemmas 14 and 15, and noticing that
$\mathbb{P}(\lim_{|\mathcal{M}_{2}|\to\infty}P_{\widehat{\pi}_{\widehat{\theta}}}(\mathcal{M}_{1})=0)\geq\mathbb{P}(\mathcal{E}(\mathcal{D}))$.
∎

## References

  * [1] Kenneth J Arrow.  Social choice and individual values.  John Wiley & Sons, 1951.
  * [2] Dilip Arumugam, Shi Dong, and Benjamin Van Roy.  Inclusive Artificial Intelligence.  arXiv preprint arXiv:2212.12633, 2022.
  * [3] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A general theoretical paradigm to understand learning from human preferences.  arXiv preprint arXiv:2310.12036, 2023.
  * [4] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  Advances in Neural Information Processing Systems, 2017.
  * [5] Google, Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu.  Palm 2 technical report, 2023.
  * [6] Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh.  Contrastive preference learning: Learning from human feedback without RL.  arXiv preprint arXiv:2310.13639, 2023.
  * [7] Duncan R Luce.  Individual Choice Behavior.  John Wiley & Sons, 1959.
  * [8] R. Duncan Luce and Patrick Suppes.  Preferences, utility and subjective probability.  In R. Duncan Luce, Robert R. Bush, and Eugene Galanter, editors, Handbook of Mathematical Psychology, chapter 19, pages 249–410. John Wiley & Sons, New York, 1965.
  * [9] Daniel McFadden.  Conditional logit analysis of qualitative choice behavior.  In Paul Zarembka, editor, Frontiers in Econometrics, chapter 4, pages 105–142. Academic Pres, New York, 1974.
  * [10] OpenAI.  GPT-4 Technical Report.  Technical report, OpenAI, 2023.
  * [11] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 2022.
  * [12] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  Advances in Neural Information Processing Systems, 2023.
  * [13] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  CoRR, abs/1804.04235, 2018.
  * [14] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang.  Preference ranking optimization for human alignment.  arXiv preprint arXiv:2306.17492, 2023.
  * [15] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 2020.
  * [16] Wanqiao Xu, Shi Dong, Dilip Arumugam, and Benjamin Van Roy.  Shattering the agent-environment interface for fine-tuning inclusive language models.  arXiv preprint arXiv:2305.11455, 2023.
  * [17] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  RRHF: Rank responses to align language models with human feedback without tears.  Advances in Neural Information Processing Systems, 2023.
  * [18] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu.  SLiC-HF: Sequence likelihood calibration with human feedback.  International Conference on Learning Representations, 2023.
  * [19] Banghua Zhu, Michael I. Jordan, and Jiantao Jiao.  Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.  International Conference on Machine Learning, 2023.