<EMAIL_ADDRESS><EMAIL_ADDRESS>Project Website:
https://understanding-rlhf.github.io/

# Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

Fahim Tajwar  Anikait Singh  Archit Sharma  Stanford University  Rafael
Rafailov  Stanford University  Jeff Schneider  Carnegie Mellon University
Tengyang Xie  UW-Madison  Stefano Ermon  Stanford University  Chelsea Finn
Stanford University  Aviral Kumar  Google DeepMind

###### Abstract

Learning from preference labels plays a crucial role in fine-tuning large
language models. There are several distinct approaches for preference fine-
tuning, including supervised learning, on-policy reinforcement learning (RL),
and contrastive learning. Different methods come with different implementation
tradeoffs and performance differences, and existing empirical findings present
different conclusions, for instance, some results show that online RL is quite
important to attain good fine-tuning results, while others find (offline)
contrastive or even purely supervised methods sufficient. This raises a
natural question: _what kind of approaches are important for fine-tuning with
preference data and why?_ In this paper, we answer this question by performing
a rigorous analysis of a number of fine-tuning techniques on didactic and
full-scale LLM problems. Our main finding is that, in general, approaches that
use on-policy sampling or attempt to push down the likelihood on certain
responses (i.e., employ a “negative gradient”) outperform offline and maximum
likelihood objectives. We conceptualize our insights and unify methods that
use on-policy sampling or negative gradient under a notion of mode-seeking
objectives for categorical distributions. Mode-seeking objectives are able to
alter probability mass on specific bins of a categorical distribution at a
fast rate compared to maximum likelihood, allowing them to relocate masses
across bins more effectively. Our analysis prescribes actionable insights for
preference fine-tuning of LLMs and informs how data should be collected for
maximal improvement.

###  1 Introduction

Pre-training endows a large language model (LLM) with knowledge about the
world. Yet, it does not provide a lever to control responses from these
models, especially when we want these solutions to optimize some task-
dependent success criteria (e.g., align with human preferences, optimize
correctness or compactness). To align LLMs with downstream success criteria,
they are then fine-tuned with downstream objectives after pre-training. In
this paper, we focus on fine-tuning problems that aim to optimize for binary
preferences (from humans or other AI models). A plethora of methods have been
proposed for this sort of fine-tuning, including supervised learning on
filtered responses (Gulcehre et al., 2023), contrastive training (Rafailov et
al., 2023), and on-policy reinforcement learning (RL) (Ouyang et al., 2022) on
a reward function extracted from human preferences.

In theory, while all of these methods aim to discover identical optimal
policies, achieving this in practice would require full data coverage and
infinite computation. These requirements are not met in practice, and hence,
the choice of the loss function and the optimization procedure affects
performance. However, a lack of a clear understanding of different approaches,
coupled with different tradeoffs in implementation, has resulted in
substantial confusion: practitioners are unsure as to: (1) whether RL (Ouyang
et al., 2022) is required at all, or contrastive approaches (Rafailov et al.,
2023; Gheshlaghi Azar et al., 2023), supervised fine-tuning are good enough;
and (2) whether preference data should be collected with models in the loop
(i.e., in an “on-policy” fashion) or not.

Our goal is to provide clarity on these questions by performing a rigorous
study to understand the behavior of existing methods when optimizing for
preferences. Our study operates under assumptions typical in preference fine-
tuning, including the existence of an underlying ground truth reward function
that explains the preference data. We study methods that train an LLM policy
to optimize a surrogate loss given by the expected reward under a model of the
reward function (learned from preference data) penalized by the KL-divergence
between the policy and a reference policy.

Figure 1: Left: an illustration of various fine-tuning techniques. On-policy
sampling gradually shifts policy mass from $\pi_{\mathrm{ref}}$ to $\pi_{i}$,
moving it towards the peak in the reward function indicated by $r^{*}$.
Offline methods that employ a negative gradient push down the likelihood of
bad responses under the learned policy, resulting in a larger deviation of
$\pi_{\text{neg}}$ compared to policies that only maximize some sort of
likelihood, $\pi_{\text{sup}}$. Right: our key takeaways for practitioners:
When the peak of the reward function lies in the less likely regions of
$\pi_{\mathrm{ref}}$, on-policy sampling is generally beneficial. In
conjunction, an explicit negative gradient approach (e.g., via RL objectives
or via contrastive objectives) is beneficial when the preference data is
skewed away from $\pi_{\mathrm{ref}}$ (case 2). When $r^{*}$ already lies in
the high likelihood regions of $\pi_{\mathrm{ref}}$, offline supervised
methods can work well. No on-policy sampling or negative gradients may be
needed.

To answer the above questions, we develop an analysis framework consisting of
didactic bandit problems, synthetic LLM problems, and full-scale LLM problems,
constructed out of AlpacaFarm (Dubois et al., 2024) and UltraFeedback (Cui et
al., 2023). We then study behaviors of different methods given coverage
conditions and geometric relationships in the problem. _Our main observation_
is that algorithms that use on-policy RL in a reward model or attempt to push-
down likelihood on certain responses, i.e., utilize a negative gradient term
as in contrastive objectives tend to outperform other offline supervised
objectives with no on-policy sampling or negative gradient. This is surprising
because both on-policy and offline methods still utilize the same data for
learning. We also find that using on-policy sampling and negative gradients
are especially important when high-reward responses appear in less-likely
regions of the reference policy distribution, and provide benefits
complementary to each other. In particular, we find that supervised objectives
such as Pref-FT and Binary Feed-ME (Dubois et al., 2024) are not able to
effectively move probability mass from low reward responses to high-reward
responses. Sampling on-policy responses for training, contrastive learning, or
employing both on-policy sampling and contrastive training can accomplish
this.

We theoretically show that approaches that use on-policy RL or certain
variants of contrastive training exhibit “mode-seeking” behavior, resulting in
faster accumulation of probability mass on a subset of high-reward responses
during learning. This behavior is in contrast to “mode-covering” supervised
objectives that attempt to increase likelihood on all high-reward responses,
and as a result, are unable to efficiently increase probability mass enough on
one subset of high-reward responses. We then compare the behavior of a
representative mode-seeking objective, the reverse KL-divergence, with the
mode-covering forward KL-divergence to formalize this behavior for categorical
distributions. Conceptually, this ability to commit to a certain subset of
high-reward responses enables algorithms with on-policy sampling (and
optionally, a negative gradient) to perform better than likelihood.

Our work presents several actionable takeaways for practitioners. First, we
tie the performance of various methods to geometric conditions on the problem,
which can inform practitioners which approach to use. Second, we observe a
tradeoff between drawing more on-policy samples and performing more gradient
steps with a different policy training objective. Understanding this tradeoff
is useful for practitioners since on-policy sampling and training present
different computational tradeoffs. Finally, since the performance of fine-
tuning is tied to the data composition, we study the effect of conditions on
the coverage of the preference data, which could inform data collection.

###  2 Related Work

A dominant recipe for fine-tuning LLMs is to run supervised next token
prediction (“supervised fine-tuning”) on a dataset of high-quality responses
to obtain a good policy initialization. This is followed by fine-tuning on a
dataset of human preferences (Casper et al., 2023; Ouyang et al., 2022). This
fine-tuning can use on-policy RL methods such as REINFORCE (Sutton et al.,
1999) or PPO (Schulman et al., 2017) to maximize the predictions of a reward
model obtained from the preference data, regularized with a KL constraint.
Another approach (Dubois et al., 2024) performs supervised fine-tuning on the
filtered set of preferred completions in the preference dataset. A different
family of methods runs supervised learning on preferred responses iteratively
such as ReST (Gulcehre et al., 2023), RWR (Hu et al., 2023), and SuperHF
(Mukobi et al., 2023). Alternatively, methods such as DPO (Rafailov et al.,
2023), IPO (Gheshlaghi Azar et al., 2023), SLiC-HF (Zhao et al., 2023), and
KTO (ContextualAI, 2024) learn directly from human preferences, with no
explicit reward model. Concurrent work also runs DPO iteratively (Yuan et al.,
2024; Chen et al., 2024). These methods come with different tradeoffs
necessitating a study to understand their behaviors.

Prior analysis work. To understand the effect of preference fine-tuning, prior
work attempts to uncover its effect on network parameters for a certain set of
tasks (Jain et al., 2023; Lee et al., 2024). Our analysis is complementary in
that it studies conditions when different algorithms perform well, and is
applicable to any downstream task. Kirk et al. (2023) study the contribution
of RL fine-tuning on generalization to out-of-distribution prompts but this is
complementary to our approach. Gao et al. (2022); Coste et al. (2023);
Eisenstein et al. (2023) study reward over-optimization to better build reward
models, which is complementary to the behavior of the policy optimization
approach. Agarwal et al. (2023) develop a recipe that uses the mode-seeking KL
divergence for knowledge distillation: this prior work is largely centered in
the problem setting of distillation and does not study the optimization
behavior of RL, contrastive, or supervised objectives. Perhaps closely related
to our work is Singhal et al. (2023), which investigates the interplay between
PPO and the composition of preference data, but this analysis is largely
concentrated on studying the length bias of RL fine-tuning rather than
developing insights into the behavior of fine-tuning algorithms. We do design
didactic examples that use rewards dependent on length, but this is solely for
analysis.

Concurrently, Ahmadian et al. (2024) show that REINFORCE may simply be enough
for preference fine-tuning of LLMs and complex policy optimization methods
such as PPO may not be needed. Our conclusions are mostly complementary,
though we do observe that PPO is more robust to sample reuse than REINFORCE.
Concurrently, Sharma et al. (2024) compares contrastive and supervised fine-
tuning on LLM-generated data, but this work does not study the role of
coverage or geometric conditions. Nevertheless their conclusions that various
approaches perform similarly when the peak in the reward function (i.e.,
oracle AI preferences) aligns with the likely regions in the data (i.e.,
responses generated from the same AI model), thus providing evidence to
support our findings.

###  3 Characterizing And Unifying Preference Fine-Tuning Methods

Typical preference fine-tuning methods use a variety of objectives including
RL, maximum likelihood, and contrastive learning. While the huge number of
fine-tuning methods inhibits us from empirically analyzing each of them, in
this section we characterize several existing methods into different families
and subsequently study a representative member from each family.

####  3.1 Preliminaries and Notation

Typically, before training on preference data, a pre-trained model is fine-
tuned on high-quality data from the task of interest via supervised fine-
tuning (SFT), to obtain a “reference” model $\pi_{\mathrm{ref}}$. Then, to
fine-tune $\pi_{\mathrm{ref}}$ with human preferences, usually a preference
dataset
$\mathcal{D}_{\text{pref}}=\\{\mathbf{x}^{(i)},\mathbf{y}_{w}^{(i)},\mathbf{y}_{l}^{(i)}\\}$
is collected, where $\mathbf{x}^{(i)}$ denotes a prompt and
$\mathbf{y}_{w}^{(i)},\mathbf{y}_{l}^{(i)}$ denote preferred and dispreferred
responses, obtained typically by sampling from $\pi_{\mathrm{ref}}$. Given a
preference dataset, most fine-tuning pipelines assume the existence of an
underlying reward function $r^{*}(\mathbf{x},\cdot)$. One popular framework
for this is the Bradley-Terry (BT) model (Bradley and Terry, 1952), assuming
that human preferences can be written as:

$\displaystyle\textstyle
p^{*}(\mathbf{y}_{1}\succ\mathbf{y}_{2}|\mathbf{x})=\frac{e^{r^{*}(\mathbf{x},\mathbf{y}_{1})}}{e^{r^{*}(\mathbf{x},\mathbf{y}_{1})}+e^{r^{*}(\mathbf{x},\mathbf{y}_{2})}}$
(3.1)

Given this reward function $r^{*}$, preference fine-tuning aims to find the
optimum of the reward $r^{*}$. While the ultimate goal of preference fine-
tuning is to find the _unconstrained_ optimum of the reward function, in
practice, we often replace the reward function with a reward model. Since the
reward model is erroneous, we apply KL-constraint to prevent exploitation in
the reward model. To align our results with typical preference fine-tuning
procedures, we will consider such a KL-constrained reward optimization as our
fine-tuning goal:

$\displaystyle\\!\\!\\!\\!\max_{\pi_{\theta}}\leavevmode\nobreak\ $
$\displaystyle\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}},\mathbf{y}\sim\pi_{\theta}(\cdot|x)}[r^{*}(\mathbf{x},\mathbf{y})]-\beta\mathbb{D}_{\text{KL}}[\pi_{\theta}(\cdot|\mathbf{x})||\pi_{\text{ref}}(\cdot|\mathbf{x})]\leavevmode\nobreak\
\leavevmode\nobreak\
\text{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{(Surrogate
fine-tuning goal)}}}$ (3.2)

The regularizer, weighted by $\beta$, controls the deviation of $\pi$ from
$\pi_{\mathrm{ref}}$ under the reverse KL divergence.

Reward model training. In order to fine-tune an LLM policy
$\pi_{\theta}(\mathbf{y}|\mathbf{x})$, Equation 3.1 provides a convenient way
to learn a reward model either explicitly (i.e., by fitting a parametric
reward model $r_{\phi}(\mathbf{x},\mathbf{y})$) or implicitly (i.e., via
direct preference optimization (DPO) Rafailov et al. (2023) or IPO (Gheshlaghi
Azar et al., 2023), that re-purposes the log-likelihood
$\log\pi_{\theta}(\mathbf{y}|\mathbf{x})$ of the policy to represent the
reward $r_{\theta}(\mathbf{x},\mathbf{y})$). Explicit reward models are
trained using the following classification objective:

$\displaystyle\\!\\!\\!\max_{\phi}\leavevmode\nobreak\
\mathbb{E}_{(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\sim\mathcal{D}_{\text{pref}}}\left[\log\sigma\left(r_{\phi}(\mathbf{x},\mathbf{y}_{w})-r_{\phi}(\mathbf{x},\mathbf{y}_{l})\right)\right],$
(3.3)

where $\sigma$ is the logistic function. Contrastive learning objectives
(Rafailov et al., 2023; Gheshlaghi Azar et al., 2023) on the other hand
repurposes $\log\pi_{\theta}(\mathbf{y}|\mathbf{x})$ as the implicit reward
$r_{\theta}(\mathbf{x},\mathbf{y})$:

$\displaystyle
r_{\theta}(\mathbf{x},\mathbf{y})=\beta\left[\log\pi_{\theta}(\mathbf{y}|\mathbf{x})-\log\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})\right].$
(3.4)

####  3.2 Characterizing Fine-Tuning Methods

With a reward model $r_{\phi}(\mathbf{x},\mathbf{y})$, most fine-tuning
approaches attempt to discover the policy
$\pi_{\theta}(\mathbf{y}|\mathbf{x})$ which optimizes Equation 3.2 by using
$r_{\phi}$ as a surrogate for $r^{*}$. Since we cannot empirically investigate
all of these methods, we group them into different categories (summary shown
in Table 1). In particular, we are interested in whether these methods employ:

  1. 1.

_on-policy sampling_ : an explicit sampling of new responses from the policy
(e.g., PPO, REINFORCE) or purely learning from offline data (e.g., RWR, DPO,
IPO)

  2. 2.

_on-policy sample reuse_ : for only those approaches that perform on-policy
sampling, whether the approach makes more than one gradient update on a given
prompt-response $(\mathbf{x},\mathbf{y})$ pair (e.g., exactly 1 update for
REINFORCE, $\geq 1$ for PPO, online RWR)

  3. 3.

_negative gradient_ : whether the approach explicitly minimizes a loss that
attempts to “push-down” likelihood on certain responses by multiplying the
gradient of their likelihood with a negative coefficient (e.g., contrastive
methods such as DPO; RL methods REINFORCE, PPO)

On-policy RL approaches such as PPO (Schulman et al., 2017) and REINFORCE
(Williams, 1992) explicitly sample new responses from the current snapshot of
the learned policy, $\mathbf{y}_{i}\sim\pi_{\theta}(\cdot|\mathbf{x}_{i})$,
score them under the reward model, and perform a policy gradient update on
parameters $\theta$, for example:

$\displaystyle\theta^{\prime}\leftarrow\theta-\eta\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}},\mathbf{y}\sim\pi_{\theta}(\cdot|\mathbf{x})}\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}|\mathbf{x})\cdot\bar{r}_{\phi}(\mathbf{x},\mathbf{y})\right]\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\text{(REINFORCE)},$ (3.5)

is the gradient update employed by REINFORCE, where
$\bar{r}_{\phi}(\mathbf{x},\mathbf{y})$ corresponds to a normalized estimate
of the reward model’s predictions over a batch of samples drawn from the
policy. As we discuss in more detail in Section D.1), using a normalized
reward estimate instead of directly the raw reward value helps reduce the
variance of the policy gradient estimate. High variance gradients slow down
convergence and even sometimes lead to sub-optimal solutions in deep RL (Mei
et al., 2022).

Due to the use of normalized reward estimates, policy gradient approaches
behave distinctly from maximum likelihood supervised learning: a policy
gradient update also updates the parameters $\theta$ in a direction that
attempts to push down likelihood
$\log\pi_{\theta}(\mathbf{y}^{\prime}|\mathbf{x})$ for samples
$\mathbf{y}^{\prime}$ on which normalized reward
$\bar{r}_{\phi}(\mathbf{x},\mathbf{y}^{\prime})<0$. This means that on-policy
RL also has a form of the “negative gradient”.

PPO differs from REINFORCE because it employs _sample reuse_ in addition to
on-policy sampling: unlike REINFORCE which only performs a single gradient
update on a response sampled from the current policy, PPO can utilize a
response for several policy updates. To prevent making updates on overly off-
policy responses, there is a mechanism in place to filter responses by the
magnitude of the importance ratio between the current policy
$\pi_{\theta}(\mathbf{y}|\mathbf{x})$ and the data collection policy.

Finally, we also remark that while on-policy methods do generate new rollouts
from the policy, these responses are still scored by a reward model (and not
the ground truth reward function, i.e., humans). Since reward labels come from
a reward model, on-policy preference fine-tuning approaches are instances of
offline model-based RL (Yu et al., 2021, 2020; Kidambi et al., 2020) methods
that run on-policy rollouts against a learned dynamics and reward model (due
to the single step nature of preference fine-tuning, there is no dynamics
model).

Fine-Tuning Approach | On-Policy Sampling | Sample Reuse | Negative Gradient
---|---|---|---
PPO | $\checkmark$ | $\checkmark$ | $\checkmark$
REINFORCE | $\checkmark$ | $\times$ | $\checkmark$
DPO, IPO, and variants | $\times$ | N/A | $\checkmark$
Pref-FT, Binary FeedMe | $\times$ | N/A | $\times$
offline RWR, offline Best-of-N | $\times$ | N/A | $\times$
ReST, RWR, online Best-of-N | $\checkmark$ | $\checkmark$ | $\times$

Table 1: Grouping various fine-tuning methods along the axes on-policy
sampling, sample reuse, and negative gradient. Since offline methods do not
collect on-policy data, the question of discarding or reusing on-policy
samples is not applicable.

On-policy supervised approaches such as RAFT (Dong et al., 2023), ReST
(Gulcehre et al., 2023), and SuperHF (Mukobi et al., 2023) iteratively
minimize a weighted maximum likelihood loss inspired by Peters and Schaal
(2007); Korbak et al. (2022). For a given prompt $\mathbf{x}_{i}$, these
methods sample $N$ responses from the model:
$\mathbf{y}^{1}_{i},\cdots,\mathbf{y}^{N}_{i}\sim\pi_{\theta}(\cdots|\mathbf{x}_{i})$,
then weight these responses by the exponentiated reward,
$\exp(r_{\phi}(\mathbf{x}_{i},\mathbf{y}_{i}^{j})/\beta)$ as in the case of
reward-weighted regression (RWR) or obtain the subset of $K$ highest rewarding
responses as in the case of ReST or Best-of-N. Finally, these methods train
via supervised next-token prediction on these filtered or weighted responses.
Given a weighting function,
$F(\mathbf{x}_{i},\mathbf{y}_{i}^{j}|\mathbf{y}_{i}^{0\cdots N})$ that maps a
response $\mathbf{y}_{i}^{j}$ for a given prompt $\mathbf{x}_{i}$ to a scalar
value conditioned on other responses $\mathbf{y}_{i}^{k}$ sampled from the
model for the same prompt $\mathbf{x}$, these methods maximize:

$\displaystyle\max_{\theta}\leavevmode\nobreak\
\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}},\mathbf{y}^{0\cdots
N}\sim\pi_{\theta^{\text{old}}}}\left[\log\pi_{\theta}(\mathbf{y}^{i}|\mathbf{x})\cdot
F(\mathbf{x},\mathbf{y}^{i}|\mathbf{y}^{0\cdots N})\right].\\!\\!\\!$

These algorithms employ sample reuse because they operate in a “batched”
online fashion: instead of performing _exactly one_ gradient step on a given
model sample; RWR, ReST, and SuperHF run more gradient updates, after which
new samples are drawn. However, since these methods only maximize likelihood
(i.e., only positive multipliers), there is no negative gradient effect.

Fully offline methods like DPO and IPO run contrastive training on the
preference dataset $\mathcal{D}_{\text{pref}}$ without any on-policy sampling.
These methods train using variants of Equation 3.3 (objective for IPO is shown
Section D.2) combined with Equation 3.4 on responses $\mathbf{y}_{w}$ and
$\mathbf{y}_{l}$ from the preference dataset $\mathcal{D}_{\text{pref}}$.
Despite no on-policy sampling, contrastive loss between winning and losing
responses explicitly attempts to reduce log-likelihood ratio
$\log\left(\frac{\pi_{\theta}(\mathbf{y}|\mathbf{x})}{\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})}\right)$
for $\mathbf{y}_{l}$. Another offline method is Pref-FT (Dubois et al., 2024)
which runs supervised fine-tuning on preferred responses. These methods in
general are akin to offline model-free methods, in that no reward model is
utilized by these methods.

###  4 Research Questions and Analysis Setup

Our goal is to understand the behaviors of various procedures for fine-tuning
language models. As discussed above, typically these methods differ along the
use of on-policy sampling (with additional differences pertaining to sample
reuse) and the presence of a negative gradient. We build a setup to understand
these differences empirically by answering the following questions:

Question 1: When does on-policy sampling improve over offline fine-tuning,
even though on-policy samples are annotated by a reward model, which itself is
learned from offline data? Is sample reuse useful or harmful for on-policy
methods?

Question 2: When does an explicit negative gradient help the discovery of
effective policies compared to maximum likelihood approaches such as
distilling the Best-of-N policy?

Question 3: Does on-policy sampling offer complementary benefits to negative
gradient, resulting in better performance with effective contrastive
approaches (e.g., DPO)?

To gain practically useful and actionable insights, we must answer these
questions in the context of coverage and geometric relations between the
training data, reference policy, and the reward function. These relations
affect the shape of the optimally fine-tuned policy and dictate the dynamics
of various objectives under consideration. We consider specific conditions and
relations that we discuss next.

####  4.1 Coverage Conditions and Geometric Relationships

The dynamics of the KL-constrained surrogate optimization problem (Equation
3.2) depends on the geometric alignment between the ground-truth reward
function $r^{*}$ and the reference policy initialization $\pi_{\mathrm{ref}}$
(see Figure 1). When the surrogate reward model $r_{\phi}$ is learned from the
preference data, the coverage of the preference data used to train this reward
model relative to the reference policy $\pi_{\mathrm{ref}}$ also dictates the
correctness of reward estimates and hence controls the efficacy of the
surrogate fine-tuning optimization. Likewise, the performance of purely
offline methods (e.g., offline best-of-N or contrastive methods such as
offline DPO) that do not use a reward model also depends on the relative
geometric alignment between $r^{*}$ and $\pi_{\mathrm{ref}}$ (i.e., a smaller
alignment would necessitate more deviation from $\pi_{\mathrm{ref}}$) and also
on the relative coverage of preference data (i.e., the lower the coverage, the
harder it is to discover high-reward responses). To understand the efficacy of
various methods, we consider multiple scenarios that differ along these two
factors:

  * •

[C1]: the geometric alignment between the ground-truth reward function $r^{*}$
and the reference $\pi_{\mathrm{ref}}$, that can be measured in terms of any
probabilistic divergence $\mathrm{D}(\pi_{\mathrm{ref}},\exp(r^{*}))$. This
concept is analogous to that of a “concentrability coefficient” (Munos and
Szepesvári, 2008).

  * •

[C2]: the coverage of the preference data used to train the surrogate reward
model $r_{\phi}$ relative to the reference policy $\pi_{\mathrm{ref}}$, that
can be measured in terms of the average density of the responses in the
preference dataset under the reference policy initialization,
$\pi_{\mathrm{ref}}$.

Understanding the behavior of various approaches as a function of these
factors will allow us to better understand the performance of various
approaches on downstream fine-tuning in terms of problem geometry [C1] and
statistical learning considerations [C2].

####  4.2 Tasks and Datasets

We construct a variety of didactic and LLM tasks that allow us to gain
intuition for performance of different methods under various scenarios grouped
along relationships [C1] and [C2].

Figure 2:  The didactic bandit problem which we use for our analysis in this
paper. Reference policy initialization and reward slice for each token (the
total reward is a mean of token-level rewards). The optima of reward functions
$\mathbf{R}_{1}$ and $\mathbf{R}_{2}$ occur in low-density and high-density
regions respectively.

Didactic $N$-d bandit problems. Equation 3.2 poses preference fine-tuning as a
KL-regularized contextual bandit problem over contexts $\mathbf{x}$.
Therefore, we develop a didactic $N$-dimensional contextual bandit problem. We
use a set of tokens of size $V$ of size $100$. The context, $\mathbf{x}$, is a
single discrete token from $V$. A response $\mathbf{a}$ is a sequence of
$N=10$ discrete tokens from $V$. We primarily study the effect of geometric
relationship [C1] and assume that the reward function is known exactly,
therefore not accounting for the data coverage and training of the reward
model. We consider two reward functions that differ in their relative geometry
relative to the reference policy, as shown in Figure 2. Specifically, the
difference lies in how perfectly the optimum of the reward function aligns
with the high-density regions of the reference distribution. The optimum of
the reward function $\mathbf{R}_{1}$ is located in low likelihood regions of
the reference policy, whereas the optimum of $\mathbf{R}_{2}$ is roughly
aligned with the mode of the reference policy. We hypothesize that on-policy
sampling will be crucial to optimize reward function $\mathbf{R}_{1}$, whereas
offline or maximum likelihood methods could be sufficient for the optimization
of $\mathbf{R}_{2}$.

Figure 3: Word length distribution. Above, we show the word length
distribution for the preferred and dispreferred completions of the Left: min
and Right: skew synthetic LLM datasets.

Synthetic LLM fine-tuning problems. Next, we will generalize our intuitions
from bandit problems to the LLM setting. Instead of directly experimenting
with human preferences, we first study two synthetic problems that utilize
hand-crafted reward functions, which can be approximated via reward models.
Access to functional forms of these hand-crafted reward functions will enable
us to track the ground-truth objective throughout training to see if our
insights about various approaches under condition [C1] will hold even when
learning against a reward model. Subsequently, we run this experiment with an
altered skewed preference data distribution (see Figure 3) to understand the
effect of coverage conditions [C2]. We consider two reward functions: (1) one
that minimizes the response length (“Min Length”), analogous to
$\mathbf{R}_{1}$ in the bandit problem, and (2) that attempts to anchor the
response length to a pre-specified target value (“Avg Length”), which lies in
the mode of the target distribution. This second condition exhibits similar
characteristics to $\mathbf{R}_{2}$. The Skew Length scenario skews the
preference data in the Min Length problem scenario.

Full-scale LLM fine-tuning. Finally, we scale up our study to full-scale LLMs,
with real preference data. Recent work (Singhal et al., 2023) shows that
preference labels are usually biased towards much longer responses, indicating
that preference fine-tuning usually admits a geometric relationship where the
mode of the reward function is distinct from the mode of human data (and
hence, any reference policy). For the majority of our experiments, we use
preference datasets from the AlpacaFarm benchmark (Dubois et al., 2024). We
also scale up our experiments to UltraChat (Ding et al., 2023), a $\sim 10$
times larger dataset with responses from many strong LLMs such as GPT 4 and
GPT-3.5.

####  4.3 A Generic Fine-Tuning Algorithm Encapsulating All Axes

To systematically analyze the behavior of fine-tuning methods that differ
along the axes discussed in Section 3.2, in this section, we introduce a
generic algorithm with different hyperparameters associated with each axes.
With a generic algorithm of this sort, we will be able to answer our research
questions by varying each hyperparameter. Our unified practical algorithm is
shown Algorithm 1. While on-policy algorithms perform steps 1 and 2 of on-
policy data collection with a reward model, purely offline methods (e.g., DPO
and RWR) utilize preference data directly.

Algorithm 1 A Unified Fine-Tuning Algorithm

for training iterations do

(1) Sample $B/C$ prompts
$[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{B/C}]$.

(2) Generate dataset $\mathcal{D}$ with $C$ responses for $\frac{B}{C}$
prompts, from the policy

for online
($\mathbf{y}_{i}^{1},\mathbf{y}_{i}^{2},\cdots,\mathbf{y}_{i}^{C}\sim\pi_{\theta}(\cdot|\mathbf{x}_{i})$)
or from an offline dataset
($\mathbf{y}_{i}^{1},\mathbf{y}_{i}^{2},\cdots,\mathbf{y}_{i}^{C}\sim\mathcal{D}_{\text{pref}}$).

(3) If applicable, label the responses
$\mathbf{y}_{i}^{1},\mathbf{y}_{i}^{2},\cdots,\mathbf{y}_{i}^{C}$, with
rewards drawn

from the learned reward model $\widehat{r}_{\phi}(\mathbf{y}|\mathbf{x})$

for $T$ inner iteration steps do

(a) Divide $\mathcal{D}$ into mini-batches
$\mathcal{D}_{1},\ldots,\mathcal{D}_{N}$, each with $M$ prompts-response pairs

for $i=1,\ldots,N$ do

(i) Apply the gradient of the objective
$\mathcal{L}(\theta;\mathcal{D}_{i};\widehat{r}_{\phi})$ prescribed by the
fine-tuning method.

end for

end for

end for

To study the impact of on-policy sampling, we vary the extent to which updates
are made on data from the current policy. We can control this by two means in
Algorithm 1: (1) by varying the total number of samples
$|\mathcal{D}|=\frac{B}{C}\times C=B$ used for a given training iteration
assuming the algorithm performs exactly one pass over all this sampled data
while keeping the minibatch size $M$ fixed, and (2) by varying the number $T$
of gradient steps performed on a given set $\mathcal{D}$ of on-policy samples
(i.e., a larger $T$ leads to more off-policy updates). In other words,
approach (1) will perform more updates using stale data for large values of
$|\mathcal{D}|$; and for small values of $|\mathcal{D}|$, approach (2) will
make more off-policy updates if $T$ is larger. While both approaches enable us
to control how on-policy an algorithm is, approach (1) does not reuse samples
(since $\mathcal{D}$ is large), but approach (2) reuses samples for different
number of gradient updates, controlled directly by $T$. By studying both
approaches for inducing off-policyness, we can isolate the effect of sample
reuse on on-policy methods. We also study offline methods with no on-policy
sampling, such as DPO, and filtered supervised learning on the preferred
response $\mathbf{y}_{w}$ in the dataset to understand the role of the
negative gradient.

###  5 Empirical Analysis Results

In this section, we will present the results of our empirical study to answer
our research questions. To answer each question, we will begin by studying the
didactic bandit problem with the ground-truth reward function, followed by
synthetic and then full-scale LLM fine-tuning problems.

####  5.1 Question 1: The Role of On-Policy Sampling

To understand the role of on-policy sampling, we will investigate if on-policy
sampling can improve performance for several approaches followed by making
conclusions regarding sample reuse.

#####  5.1.1 Takeaway 1: On-Policy Sampling in the Reward Model Improves
Performance

We first study on-policy sampling as a function of the geometric relationship
[C1] in our bandit setting (see Figure 2), with no sampling error. Then, we
will extend our conclusions to the LLM setting.

Figure 4: On-policy sampling on bandit problems. Performance of on-policy
best-of-N as a function of the data sampled in each iteration. Larger batch
sizes result in more off-policy updates. Left: (i) reward vs update step for
$\mathbf{R}_{1}$, (ii) divergence between the policy parameters and data
collection policy during training; Right: (i) reward vs update step for
$\mathbf{R}_{2}$, (ii) KL divergence for $\mathbf{R}_{2}$. Observe the slow
learning speed of more off-policy updates in $\mathbf{R}_{1}$, but less severe
degradation for $\mathbf{R}_{2}$, where peaks in the reference policy and
reward function are more aligned.

Didactic bandit problems. Figure 4 shows that given a fixed amount of total
data budget, _sampling data more frequently from more recent policies_ , but
in smaller batches, results in better performance with both $\mathbf{R}_{1}$
and $\mathbf{R}_{2}$. Doing so, naturally makes the algorithm more on-policy
since each gradient update uses a mini-batch sampled from a more recent
policy. This is also reflected in larger values of divergences between the
sampling policy $\pi_{\text{gen}}$ and the policy $\pi_{\theta}$,
$\mathbb{D}_{\text{KL}}(\pi_{\theta}||\pi_{\text{gen}})$, in Figure 4.
Concretely, larger $B$ results in higher peak values of this divergence during
training indicating further deviation from the data at intermediate times
during training. This means that being more on-policy corresponds to better
performance and faster convergence for best-of-N.

That said, we also note in Figure 4 that the performance degradation with more
off-policy updates is substantially milder for $\mathbf{R}_{2}$, indicating
that when the peak in the reward function lies in the high likely regions of
the reference policy, a higher degree of off-policy updates is tolerable.

[C1] $\downarrow$ $||$ [C2] $\rightarrow$ | high $\mathcal{D}_{\text{pref}}$ and $\pi_{\text{ref}}$ overlap | low $\mathcal{D}_{\text{pref}}$ and $\pi_{\text{ref}}$ overlap
---|---|---
peaks of $r^{*}$ and $\pi_{\text{ref}}$ overlap |  $\checkmark$ Mode Length | $\times$
peaks of $r^{*}$ and $\pi_{\text{ref}}$ disjoint |  $\checkmark$ Min Length |  $\checkmark$ Skew Length
Table 2: Coverage conditions and geometric relations that we study with
synthetic LLM fine-tuning data. The three settings we study differ in terms of
overlap between $\pi_{\text{ref}}$, reward $r^{*}$, and the preference
dataset, $\mathcal{D}_{\text{pref}}$.

Synthetic LLM problems. In this problem setting, we optimize the policy
against a reward model, which is learned from preference data. Per Section
4.2, we construct three scenarios that differ along geometric ([C1]) and
coverage ([C2]) conditions as depicted in Table 2. The peak of the reward in
the Min Length scenario appears in the less likely regions of
$\pi_{\text{ref}}$, whereas the peak of the reward function in the Mode Length
scenario appears in highly likely regions under $\pi_{\text{ref}}$.

Figure 5: On-policy sampling for PPO in the Min Length scenario. This plot
keeps the minibatch size $M$ fixed to 64, but samples more stale data when $B$
is large. Increasing $B$ results in more off-policy updates and consequently
slower convergence to ground-truth reward (i.e., a completion length of $0$).
Left: average completion length (lower the better), and Right: proxy reward vs
gradient steps. Being more on-policy results in better performance. The mini-
batch size $M$ used for gradient updates is kept fixed to avoid confounders
arising from the use of stochastic optimization procedures.

We present our results for one algorithm in detail (in this case, PPO)
(Figures 5, 6 and 7) and then present a summary plot showing that our
conclusions also transfer to other algorithms (such as REINFORCE and RWR)
(Figure 8). Extending insights from the bandit problem, in the Min Length
scenario, we find that being more on-policy (i.e., a smaller $B$) leads to a
lower completion length and hence a higher gold reward, despite potential
inaccuracies in the proxy reward model that PPO is actually optimizing (Figure
5). Akin to our bandit experiments, we also observe that smaller batch sizes
($B=64$ and $B=128$) optimize the proxy reward at a faster rate compared to
$B=192$ and $B=256$. This indicates that with a significant overlap between
the preference data and the reference policy, on-policy sampling still leads
to better performance with fewer updates. We also find similar trends across
on-policy variants of RWR and REINFORCE, where modulo training instabilities,
being more on-policy results in better performance (Figure 8; Min Length).

Figure 6: On-policy sampling for PPO in the Mode Length scenario. In this
case, since the peak in the reward function and the highly likely regions of
the reference policy are close, we find that the degree of on-policyness does
not significantly affect performance. Left: distance to mode i.e., |completion
length - average length in the dataset| (lower the better), Right: proxy
reward vs gradient steps. As optimal policy $\pi^{*}$ and reference policy
$\pi_{\mathrm{ref}}$ are very close to each other in this scenario, we don’t
see any significant performance gains from being on-policy. The mini-batch
size $M$ used for the gradient update is kept fixed.

In the Mode Length scenario, where the preferred response for each preference
pair are those that are closest to the average length in the dataset (203),
varying the degree of on-policy sampling by adjusting the sampling frequency
largely does not affect either the proxy or gold reward for PPO (Figure 6). We
make similar observations for other algorithms: Figure 8; Mode Length:
different degrees of on-policyness perform similarly, except the more on-
policy runs sometimes exhibit instability. This is in agreement with the
results from the bandit setting above: when the peak in the reward function
lies in highly likely regions under the reference policy, on-policy sampling
has minor effect and more off-policy configurations of the algorithm can
perform similarly too.

Figure 7: On-policy sampling for PPO on the Skew Length scenario. Being more
on-policy results in faster convergence and better performance. Left: average
completion length (lower the better), and Right: proxy reward vs gradient
steps. Being more on-policy results in better performance.

Finally, to evaluate the robustness of these findings under more challenging
coverage conditions, we deliberately skew the length distribution in the
preference dataset to make it distinct from the reference policy (called Skew
Length). Concretely, with a 95% probability, we truncate the length of the
response by sampling a length from an exponential distribution, which
naturally leads to a shorter completion length. The remaining 5% of samples
are drawn from the standard SFT policy to simulate the broader coverage for
the preference data. Overall, the resulting data admits a significantly skewed
distribution over response lengths, as visualized in Figure 3. Not only does
the peak in the reward function now appear in less likely regions of the
reference policy, but to succeed, an optimization algorithm must now do the
required heavy lifting to shift the probability mass to the low-density
regions of the response space that maximize reward.

Figure 8: Summary: effect of on-policy sampling on synthetic LLM problems.
Average gold reward over the course of training for RWR, and REINFORCE with
different $B$. For Min Length and Skew Length, generally being more on-policy
(i.e., smaller batch size) leads to a higher gold reward. For Mode Length, all
batch sizes perform close to each other (note that the range of the $y$-axis
is small), with performance differences largely due to instability.

Our detailed results of running PPO in this setting are shown in Figure 7. In
this setting, we still find that more on-policy updates lead to a higher gold
reward with PPO. In addition, we also observe much larger gaps in proxy reward
values attained at any given gradient step compared to the Min Length
scenario, in favor of on-policy sampling. For other algorithms, we also
observe strong and clear trends supporting that on-policy sampling with a
smaller but frequently sampled batch results in better performance as shown in
the summary plot (see Figure 8; Skew Length).

Figure 9: Effect of on-policy sampling on AlpacaFarm with a fixed mini-batch,
but varying batch size $B$, for (Left) on-policy RWR and (Right) REINFORCE.
Increasing $B$ makes updates more off-policy and this results in lower
performance.

Full-scale LLM problems. Finally, we evaluate if our insights transfer to the
full-scale AlpacaFarm setup. We use a Pythia-1.4B model as our reference
policy and generate two responses per prompt. We label the preferred and
dispreferred responses with a gold reward model of human preferences from
AlpacaFarm to construct a preference dataset. Figure 9 shows that our
intuitions from the simple bandit and synthetic LLM experiments transfer to
this real preference learning task, as making updates on only on-policy
samples leads to higher gold reward for both on-policy RWR and REINFORCE.

#####  5.1.2 Takeaway 2: On-Policy Sample Reuse Can Enable Leveraging Off-
Policy Data

In the previous section, exactly one gradient step was taken on a given sample
and we found that making updates on stale data was not helpful due to off-
policy updates. Is there any scenario under which we can still attain good
policy performance despite employing off-policy updates? In this section, we
will answer this question, and show that it might be possible to learn with
off-policy updates for some algorithms if we are allowed to make more than one
update on a given sample. Of course, a substantial amount of sample reuse is
detrimental since it would lead to more off-policy updates, thus leading to
statistical or even propensity overfiting (Swaminathan and Joachims, 2015) for
some methods, but it is reasonable to surmise that some amount of sample reuse
can help. To study sample reuse, we compare methods when $T>1$ gradient steps
can be made on a given sample.

Figure 10: Effect of on-policy sample reuse on bandit problems. Reward vs
gradient steps for a different number of inner iteration steps, $T$, on the
same data batch for RWR. Increasing $T$ controls the number of gradient steps
taken before collecting the new batch of on-policy samples. We observe non-
monotonic performance trends while varying $T$.

We study sample reuse for on-policy RWR in the bandit setting in Figure 10.
While increasing $T$ can slow down convergence in general, we note that using
a larger value of $T$ may be better (e.g., $T=5$ learns faster than $T=2$;
$T=10$ learns faster than $T=7$).

Synthetic LLM problems. We also evaluate the effect of sample reuse on
synthetic LLM problems. In this case, we study two algorithms PPO and on-
policy best-of-N to be able to understand the effect of sample reuse on
multiple algorithms. In contrast to the performance degradation with off-
policy updates induced due to stale samples in PPO, we find that off-policy
updates induced due to sample reuse do not hurt performance (Figure 11; PPO),
with even $T=8$ performing similarly to $T=1$. On the other hand increasing
$T$ from $1$ to $2$, i.e., performing two gradient updates on each sample
improves the golden reward for best-of-N (Figure 11; Best-of-N) within a given
data sampling budget.

Figure 11: Effect of on-policy sample reuse in the Min Length scenario.
Average completion length (i.e., the lower the better) vs gradient steps for a
different number of inner iteration steps, $T$, on the same data batch. A
larger value of $T$ implies that the algorithm is more off-policy. Observe
that some sample reuse can improve sample efficiency (T = 2 outperforms T =
1), but excessive sample reuse can hurt performance. Also note that algorithms
with mechanisms to control off-policy updates such as PPO are suited to
perform better in the off-policy sample reuse setting.

Why do PPO and best-of-N respond differently to sample reuse? We believe that
this is because PPO employs an off-policy correction, and hence, significantly
off-policy samples do not contribute to the gradient, addressing the well-
known challenge of propensity overfitting (Swaminathan and Joachims, 2015).
This is not the case with on-policy best-of-N, where excessive sample reuse
can hurt exploration, because training on old samples with a log-likelihood
loss push the current policy to be close to the stale data-generating policy.
That said, more than one gradient step can still be useful when presented with
a fixed data budget, unless it bottlenecks exploration of high reward regions.

Takeaways for on-policy sampling On-policy sampling generally improves
performance and efficiency, especially in cases when the peak of reward
appears farther from the reference policy, even when the reward model is
learned from the same preference dataset that methods without on-policy
learning also use. In some cases, sample reuse can reduce the dependency on
on-policy sampling of data, but it presents a tradeoff by reducing the
exploration of the response space.

####  5.2 Question 2: The Role of Negative Gradient

To understand the role of negative gradient, we will compare contrastive
algorithms such as DPO and IPO with maximum likelihood methods such as RWR (or
Pref-FT, which attempts to increase the likelihood of the preferred response
only) and best-of-N in a fully offline setting, where no new on-policy samples
are used. We will also aim to understand the mechanisms behind these methods.

#####  5.2.1 Takeaway 1: Negative Gradient Enables Faster Convergence Amongst
Offline Methods

Figure 12: Negative gradients on the didactic bandit problems. Average reward
during training and the KL-reward trade-off for four algorithms in the fully
offline setting: best-of-N (no negative gradient), RWR (no negative gradient),
best-of-N + an explicit negative gradient on dispreferred actions, and IPO
(with negative gradient). Negative gradient helps find a better policy by
aggressively pushing down the likelihood of bad actions, and this leads to
larger KL values.

We begin by comparing a representative set of offline algorithms on the
didactic bandit problem. These methods include those that do not use a
contrastive update on the didactic bandit problem, namely offline supervised
approaches, Best-of-N and offline RWR, and offline IPO (Gheshlaghi Azar et
al., 2023), a representative offline fine-tuning method which uses a
contrastive negative gradient term. We also consider a variant of best-of-N
where we explicitly add a term to the loss function that attempts to minimize
the likelihood of the dispreferred response akin to unlikelihood (Welleck et
al., 2020) (see Section G.2 for more details). In Figure 12, we find that IPO
and best-of-N + negative gradient learn a better policy from an offline
dataset collected from sub-optimal $\pi_{\mathrm{ref}}$, compared to best-of-N
and RWR. IPO achieves a better KL-reward trade-off in $\mathbf{R}_{1}$ (where
high likelihood regions of $\pi_{\mathrm{ref}}$ and the peak in $r^{*}$ are
far away from each other). While best-of-N attains a higher reward when the
reward function is given by $\mathbf{R}_{2}$ (where the peaks in
$\pi_{\mathrm{ref}}$ and $r^{*}$ overlap) compared to $\mathbf{R}_{1}$, it
still underperforms IPO. We suspect that this is because maximizing likelihood
on some responses alone is not enough to steer the learned policy away
meaningfully away from $\pi_{\mathrm{ref}}$ towards the peak in the reward
function, especially when this peak is far away from $\pi_{\text{ref}}$. Best-
of-N + negative gradient significantly outperforms Best-of-N in both scenarios
and closes the performance gap to IPO, which shows that explicitly adding a
loss term to minimize the probability on dispreferred responses can provide a
substantial performance improvement. That said, for reward function
$\mathbf{R}_{2}$, we also observe a smaller gap between the best algorithm
without a negative gradient (i.e., RWR) and offline IPO, indicating that when
the peak in $\pi_{\mathrm{ref}}$ and $r^{*}$ exhibit more overlap, the
performance benefits of contrastive training are smaller. We also investigated
a simpler 1-token bandit problem where we found best-of-N to be better than
IPO for $\mathbf{R}_{2}$. This is possibly due to the much smaller space of
possible tokens and responses, where maximum likelihood methods perform well
enough.

Figure 13:  Negative gradients in synthetic LLM problems. Completion length
(inverse of the true reward) for three offline algorithms. DPO outperforms
Pref-FT and offline RWR in Min Length and the Skew Length settings, where the
peak in $r^{*}$ and $\pi_{\mathrm{ref}}$ are misaligned. For the Mode Length
setting, all of the algorithms perform similarly.

Synthetic LLM problems. Our experiments in the synthetic LLM setting
corroborate this finding. Here we compare Pref-FT with DPO (with negative
gradients). In the Min Length setting, we find in Figure 13 that DPO
significantly outperforms Pref-FT. On the other hand, when the peak in the
ground-truth reward appears in high-likely regions of the reference policy and
the preference data $\mathcal{D}_{\text{pref}}$ covers this region (Mode
Length), we find both approaches to perform similarly. Finally, in the Skew
Length scenario when $\pi_{\mathrm{ref}}$ and $\mathcal{D}_{\text{pref}}$ do
not overlap significantly, but the peak in $r^{*}$ is covered by the
preference dataset $\mathcal{D}_{\text{pref}}$, we also find that DPO is much
more effective in driving the policy further from the reference initialization
and outperforms Pref-FT.

Figure 14: Negative gradients in AlpacaFarm (left) and UltraFeedback (right)
for offline methods. We plot the increase in average gold reward compared to
the reference model for different offline approaches. Algorithms with a
negative gradient such as DPO outperform approaches such as Pref-FT not
utilizing any negative gradient term.

Full-scale LLM fine-tuning. Finally, we compare supervised Pref-FT and
contrastive DPO when fine-tuning on actual preference data. In addition to
AlpacaFarm, we also run experiments using the Ultra-Feedback (Ding et al.,
2023) dataset. For the Ultra-Feedback dataset, we use different models
(GPT-3.5, GPT-4) to generate responses to various prompts. The resulting
dataset has a broader preference dataset distribution than
$\pi_{\mathrm{ref}}$. We utilize a checkpoint of the Mistral7B model obtained
by running supervised next-token prediction on a subset of UltraChat
(comprising of GPT-3.5 responses) as the reference initialization. We use the
UltraRM model with a LLaMA2-13B base architecture as our gold reward model. As
shown in Figure 14, DPO which utilizes a negative gradient shows a much larger
improvement over the reference policy $\pi_{\mathrm{ref}}$compared to methods
that do not utilize a negative gradient (e.g., Pref-FT).

#####  5.2.2 Takeaway 2: Mechanisms Explaining the Behavior of the Negative
Gradient

Having seen that using a negative gradient leads to much better performance,
we next attempt to understand the mechanism behind this better performance. To
do so, we visualize the evolution of the log-likelihoods of the preferred
response and the dispreferred response in a held-out dataset as multiple
gradient steps are taken on an offline preference optimization loss.

Contrastive training increases the gap between the likelihoods of preferred
and dispreferred responses. Perhaps as expected, we find that DPO-style
contrastive training is more effective at increasing the gap between the
likelihoods of preferred and dispreferred responses compared to offline Pref-
FT in several LLM settings: the synthetic LLM settings with Min Length and
Skew Length, and full-scale AlpacaFarm and UltraFeedback settings (Figure 15).
More concretely, note that the margin for Pref-FT largely converges to 0,
whereas offline DPO can enable a larger margin.

Figure 15: Difference in likelihoods of preferred and dispreferred responses.
DPO increases the log probability margin
$\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})$
more compared to non-contrastive methods such as Pref-FT. Figure 16: DPO
implicit reward during training. We observe that with fewer prompts,
contrastive methods can increase the implicit reward,
$r_{\theta}(\mathbf{x},\mathbf{y})=\log\left(\pi_{\theta}(\mathbf{y}|\mathbf{x})\right)-\log\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})$,
of the preferred response while reducing this quantity for the dispreferred
response, however as the number of data points grows, this may not be possible
and the likelihood of both positives and negatives might reduce.

Changes in log likelihoods depend on model capacity, reference initialization,
data size, and composition. The natural next question is if DPO-like
objectives use the probability mass recovered by increasing the reward margin
between $\mathbf{y}_{w}$ and $\mathbf{y}_{l}$ to increase the probability mass
on the preferred responses111Concurrent work (Rafailov et al., 2024) also
studies the induced rewards for DPO and shows that when
$\pi_{\mathrm{ref}}(\cdot|\mathbf{x})$ is exactly equal to the _empirical_
distribution of preferred responses $p(\mathbf{y}_{w}|\mathbf{x})$ in the
dataset, then induced rewards will always decrease. This does not contradict
our findings because this condition is not satisfied in typical fine-tuning
pipelines where _both_ $\mathbf{y}_{w}$ and $\mathbf{y}_{l}$ are sampled from
$\pi_{\mathrm{ref}}$. Furthermore, even if $\pi_{\mathrm{ref}}$ is obtained by
first running supervised Pref-FT only on $\mathbf{y}_{w}$, it is unclear
whether the parametric model representing
$\pi_{\mathrm{ref}}(\cdot|\mathbf{x})$ will induce an identical probability
distribution to the empirical distribution of preferred responses. That said,
it is indeed the case that the likelihood of $\mathbf{y}_{w}$ decreases often
when training with DPO even though the reference policy does not satisfy the
condition highlighted in this concurrent work, implying that this phenomenon
is a result of many factors (data size, similarity of $\mathbf{y}_{w}$ and
$\mathbf{y}_{l}$, capacity). We also show in Section 6 that with appropriate
negatives, likelihoods might not decrease for some contrastive methods.. We
track the induced rewards
$\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\log\pi_{\mathrm{ref}}(\mathbf{y}_{w}|\mathbf{x})$
and
$\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})-\log\pi_{\mathrm{ref}}(\mathbf{y}_{l}|\mathbf{x})$
in expectation over prompts $\mathbf{x}$ on the bandit problem while varying
the size of the preference dataset. Following standard protocols, both
$\mathbf{y}_{l}$ and $\mathbf{y}_{w}$ are sampled from $\pi_{\mathrm{ref}}$.
Observe in Figure 16 that when the dataset size is small relative to the model
capacity, contrastive training via IPO can increase the likelihood of
$\mathbf{y}_{w}$ while reducing the likelihood of $\mathbf{y}_{l}$. However,
as the number of prompts increases, contrastive training counter-intuitively
results in a decreasing value of
$\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\log\pi_{\mathrm{ref}}(\mathbf{y}_{w}|\mathbf{x})$,
although the loss attempts to push up this likelihood term. The recovered
probability mass is instead used to increase the likelihood of other out-of-
distribution responses. Thus, depending upon $\pi_{\mathrm{ref}}$, dataset
size, and composition, contrastive objectives such as DPO extrapolate, and
this extrapolation might produce good or bad responses.

We also observe a similar trend in full-scale LLM experiments in Figure 17: we
observe a decrease in the log-likelihoods of both the preferred and
dispreferred responses throughout training on AlpacaFarm with small 1.4B
Pythia policies. However, using a Mistral7B model to train a policy on the
UltraFeedback dataset results in an increasing value of log-likelihood of
$\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})$ and a decreasing value of
$\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})$ when starting from an SFT model on
the Ultrachat-200K dataset (same setup as Zephyr (Tunstall et al., 2023)). We
believe that these opposite trends are a consequence of the responses in that
the UltraFeedback dataset are more semantically distinct from each other, as
different responses come from models with different capabilities (e.g., a
GPT-4 response is paired with a GPT-3.5 response) such that given enough model
capacity, contrastive training can push up likelihoods of
$\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})$ while pushing down
$\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})$. In contrast, running Pref-FT
increases the likelihoods of both $\mathbf{y}_{w}$ and $\mathbf{y}_{l}$
(Figure 17) despite training only on $\mathbf{y}_{w}$: this observation about
Pref-FT was also noted by concurrent work such as Hong et al. (2024); Pang et
al. (2024).

Figure 17: DPO reward estimates for Pref-FT and DPO on AlpacaFarm and
UltraFeedback. For a Pythia-1.4B model trained on AlpacaFarm, DPO decreases
the implicit reward,
$r_{\theta}(\mathbf{x},\mathbf{y})=\beta\left[\log\pi_{\theta}(\mathbf{y}|\mathbf{x})-\log\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})\right]$,
for both $\mathbf{y}_{w}$ and $\mathbf{y}_{l}$, whereas Pref-FT increases
both. For a Mistral-7B model trained on UltraFeedback, DPO is able to increase
the reward for $\mathbf{y}_{w}$ and decrease the reward for $\mathbf{y}_{l}$,
whereas Pref-FT increases both. In both cases, DPO leads to a higher margin
than Pref-FT.

Takeaways for negative gradients A negative gradient improves over offline
supervised methods when the peak in the reward appears in less likely regions
of $\pi_{\mathrm{ref}}$. It can increase the likelihood of $\mathbf{y}_{w}$
when $\mathbf{y}_{l}$ is sufficiently different from $\mathbf{y}_{w}$, model
capacity is large, and $\pi_{\mathrm{ref}}$ is chosen appropriately. If not,
the margin
$\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})$
will still be larger when a negative gradient is used, but the recovered
probability mass will go into increasing likelihoods of other responses, not
$\mathbf{y}_{w}$.

####  5.3 Question 3: On-Policy Sampling and Negative Gradients are
Complementary

Based on our findings that both on-policy sampling and negative gradients are
independently effective, we now study if combining them would provide any
additional benefits. To understand this, we empirically study a
straightforward on-policy variant of DPO/IPO: instead of utilizing the PPO or
Best-of-N objective on on-policy samples, for each prompt $\mathbf{x}$, we
sample $N$ responses from the policy
$\mathbf{y}_{1},\ldots,\mathbf{y}_{n}\sim\pi_{\theta}(.|\mathbf{x})$, rank
them according to a reward model $r_{\phi}$, and construct preference pairs by
taking the higher reward completion as the preferred one and lower reward
completion as the dispreferred one. This recipe is similar to concurrent works
such as Rosset et al. (2024). Then we calculate the DPO/IPO loss on this
preference dataset and update our model accordingly.

Figure 18: On-policy sampling + negative gradients in bandit setup.
Complimentary benefit of on-policy sampling and negative gradients. Online IPO
(using both on-policy sampling and negative gradients) performs better than
offline IPO (negative gradients but no on-policy sampling) and RWR (on-policy
sampling but no negative gradients).

Performance on bandit and synthetic LLM problems. Figure 18 shows that the on-
policy version of IPO achieves both faster convergence and better performance
compared to the offline version, for both $\mathbf{R}_{1}$ and
$\mathbf{R}_{2}$ in the didactic bandit problem. We also ran on-policy DPO in
synthetic LLM problems we studied and found it to converge significantly
faster and to a better solution than offline DPO, on-policy RL, and on-policy
variants of supervised learning approaches as shown in Figure 19. We also find
that on-policy versions of contrastive approaches exhibit favorable
computational vs wall-clock time tradeoffs compared to purely on-policy RL
methods and even offline contrastive methods that may not find as good
solutions as their on-policy counterparts (see Appendix B).

Figure 19:  On-policy sampling + negative gradients in LLM length experiments.
Complimentary benefit of on-policy sampling and negative gradients on the
synthetic LLM length experiments. On-policy DPO performs the best where
optimal policy and reference policy lies far from each other (min length and
skew length), and all algorithms perform similarly when these two policies are
close (mode length).

Why can on-policy versions of contrastive methods perform better than on-
policy RL? We saw in Section 5.2.1 that offline contrastive training with a
negative gradient was effective at quickly reorganizing probability mass to
high-reward responses covered by the preference data. When combined with on-
policy sampling, this behavior results in faster convergence: for any given
batch of on-policy data, contrastive training with a negative gradient can
quickly reconfigure the policy distribution within the support of the on-
policy data obtained thus far (i.e., it provides a stronger, low-variance
learning signal). Similarly to how best-of-N + negative gradient outperforms
vanilla best-of-N but underperforms DPO in Figure 12, PPO also improves over
RWR without a negative gradient term (in the bandit setting this corresponds
to a better reward-KL tradeoff in Figure 18 and in the synthetic LLM setting
this appears in final performance), but it is still unable to match on-policy
DPO in Figure 19. Note that this does not mean that on-policy DPO would always
outperform PPO, but that it might be a good choice for users to experiment
with on-policy versions of contrastive methods.

Takeaways for on-policy sampling + negative gradient On-policy sampling and
offline negative gradients present complementary benefits, in that the best
offline loss function with negative gradients can be used to train on on-
policy data, improving over on-policy RL or supervised learning. Conceptually,
while sampling responses on policy provides coverage of the response space, an
effective negative gradient loss provides a stronger learning signal given a
set of samples. It can also result in computational benefits.

###  6 Conceptual Unification and Theoretical Analysis

With empirical results showing the benefits of on-policy sampling and negative
gradient for preference fine-tuning of LLMs, in this section, we attempt to
conceptually understand the benefits by building a mental model. In this
section, we will first unify these seemingly distinct notions of on-policy
sampling and negative gradient into a unified notion of mode-seeking
objectives, and contrast them against mode-covering maximum likelihood
objectives. Then, we will contrast the learning dynamics of the reverse KL-
divergence, a representative mode-seeking objective against the mode-seeking
forward KL-divergence (i.e., the supervised learning loss) to intuitively
explain some of our findings.

####  6.1 Seeking Modes Unifies On-Policy Sampling and Negative Gradients

In this section, we will show that the notion of mode-seeking divergences
unifies on-policy sampling and negative gradients for the various objectives
we investigated in the paper. Specifically, we show below that several on-
policy RL methods that we studied optimize the reverse KL-divergence, and are
hence mode-seeking, offline contrastive methods that employ a negative
gradient are also mode-seeking, and finally, supervised weighted maximum
likelihood approaches (e.g., offline Best-of-N, Pref-FT, Binary FeedMe) are
mode-covering. First, we show that on-policy sampling leads to mode-seeking
behavior. To do this, we prove that RL and supervised objectives combined on-
policy sampling optimize the reverse KL divergence, which is known to be mode-
seeking.

######  Lemma 6.1.

On-policy RL and on-policy weighted-likelihood methods optimize a regularized
version of a reverse KL-divergence with respect to the optimal policy and are
hence mode seeking.

A proof for Lemma 6.1 is shown in Section C.1.1. Next, we show that offline
contrastive methods that employ a negative gradient are also mode-seeking.
While these approaches do not optimize the reverse KL-divergence, we can still
show that the probability mass obtained by minimizing density on negative
responses $\mathbf{y}_{l}$ gets disproportionately utilized, far more for
increasing the probability mass on the “mode” (i.e., highest probability
categories under the current policy $\pi_{\theta}$) compared to other
categories. When the offline dataset consists of multiple high-reward
categories, this preference to put more probability mass on the mode of the
current policy results in mode-seeking behavior, compared to increasing
probability mass on all high-reward categories.

######  Lemma 6.2.

Let $\theta_{t}$ denote the parameters of the model at a given iteration $t$.
Consider contrastive approaches that induce a negative gradient under a
functional form shown below:

$\displaystyle\theta_{t+1}\leftarrow\theta_{t}+\eta\leavevmode\nobreak\
\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\cdot
c_{1}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\cdot
c_{2}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\right]\Big{|}_{\theta_{t}},$
(6.1)

where $c_{1}$ and $c_{2}$ are non-negative functions that depend on the reward
value and the associated samples, $\mathbf{y}_{w}$ and $\mathbf{y}_{l}$. In
contrast, weighted maximum likelihood without the negative gradient sets
$c_{2}=0$. Define
$\omega_{t}:=\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})$.
Then, for all models $\theta$ and for all $t$, there always exists an
appropriate dataset of positive and negative samples $\mathcal{D}$, such that:

$\displaystyle\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\omega_{t+1}\right]\Big{|}_{{c_{2}>0}}\geq\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\omega_{t+1}\right]\Big{|}_{{c_{2}=0}}.$
(6.2)

In addition, if the model class $\pi_{\theta}$ and $\mathbf{y}_{l}$ can
jointly realize the following gradient alignment condition (note that for any
$\theta_{t}$, there always exists a $\mathbf{y}_{l}$ that satisfies this
condition):

$\displaystyle\forall\theta\in\left[\theta_{t}\right]_{t},\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right]\leq
0,$

then, we find that the likelihood of positives is larger (and similarly
likelihood of negatives is smaller) when $c_{2}>0$, i.e., when a negative
gradient term is used:

$\displaystyle\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right]\Big{|}_{c_{2}>0,\theta=\theta_{t}}\leavevmode\nobreak\
$
$\displaystyle\geq\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right]\Big{|}_{c_{2}=0,\theta=\theta_{t}}$
(6.3)
$\displaystyle\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right]\Big{|}_{c_{2}>0,\theta=\theta_{t}}\leavevmode\nobreak\
$
$\displaystyle\leq\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right]\Big{|}_{c_{2}=0,\theta=\theta_{t}}$
(6.4)

A proof for Lemma 6.2 is provided in Section C.1.2. This result indicates that
for appropriate negative responses, a contrastive update accelerates the rate
of increase of probability mass on $\mathbf{y}_{w}$, for any model class
$\pi_{\theta}$ and reference initialization $\theta_{0}$, compared to setting
$c_{2}=0$, which offline weighted maximum likelihood. This corresponds to
mode-seeking behavior. The update induced by DPO admits a similar form (see
the discussion after Equation 7 in Rafailov et al. (2023)). This theoretical
result also corroborates our findings in the experiments in Section 5.2.1
regarding the negative gradient term. The gradient of IPO also admits a
similar form (Section C.1.2).

Next, we note that purely offline versions of supervised methods such as RWR,
ReST, and BoN, that only maximize weighted likelihood are mode-covering
because these objectives can be shown to maximize the forward KL-divergence
against the optimal policy (proof in Section C.1.3).

######  Lemma 6.3.

Consider offline supervised methods that maximize weighted log-likelihood:

$\displaystyle\mathcal{L}_{\text{off-
sup}}(\pi_{\theta};\pi_{\mathrm{ref}})=-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\mathrm{ref}}(.|\mathbf{x})}[\log\pi_{\theta}(\mathbf{y}|\mathbf{x})\cdot
F(\mathbf{x},\mathbf{y})]\right]$ (6.5)

where $F(\mathbf{x},\mathbf{y})\geq 0$ is the weight for
$(\mathbf{x},\mathbf{y})$. Furthermore,
$\sum_{\mathbf{y}}F(\mathbf{x},\mathbf{y})>0$ (i.e., for every $\mathbf{x}$,
there exists a response $\mathbf{y}$ with non-zero
$F(\mathbf{x},\mathbf{y})$). Then these methods optimize a forward KL-
divergence.

####  6.2 Case Study: Mode-Seeking Reverse KL vs. Mode-Covering Forward KL

Having seen that mode-seeking and mode-covering divergences can unify on-
policy sampling and negative gradients, in this section, we perform a
theoretical analysis to quantify the behavior of the two representative mode-
seeking and mode-covering objectives: reverse KL (mode-seeking) and forward KL
(mode-covering) objectives on categorical distributions, parameterized via
independent logits. Our goal is to formalize the intuition that a mode-seeking
objective can sharpen the probability mass on only certain high-reward
regions, thereby leading to aggressive reorganization of probability mass.
This helps corroborate our experiments that on-policy sampling in a reward
model and offline negative sampling is still useful to quickly align the
policy with the target distribution.

Notation and setup. For this result, we will study training a categorical
distribution $p(\mathbf{x})$ to match the theoretically optimal fine-tuned
policy, $q(\mathbf{x})$. We assume that
$p(\mathbf{x})\propto\exp(f(\mathbf{x}))$, where each logit $f(\mathbf{x})$ is
an independent parameter. We train $p(\mathbf{x})$ by performing gradient
descent, starting from an initial reference distribution $p_{0}$ on a fine-
tuning loss with gradient descent and a learning rate $\eta$. We denote the
distribution at step $t$ of this gradient descent as $p_{t}$. For this
analysis it would be helpful to explicitly write out the parameter updates at
any iteration $t$, induced by forward and reverse KL.

######  Lemma 6.4.

For any given distribution $p_{t}$, with
$p_{t}(\mathbf{x})=\exp(f_{t}(\mathbf{x}))$, the updates induced by the
forward and reverse KL-divergences within one step of gradient descent with a
learning rate $\eta$ are given by:

Forward KL:
$\displaystyle\log\frac{p^{f}_{t+1}(\mathbf{x})}{p_{t}(\mathbf{x})}=\eta\left(q(\mathbf{x})-p_{t}(\mathbf{x})\right)+\mathbb{Z}.$
(6.6) Reverse KL:
$\displaystyle\log\frac{p^{r}_{t+1}(\mathbf{x})}{p_{t}(\mathbf{x})}=\eta\left(p_{t}(\mathbf{x})\left[\log\frac{q(\mathbf{x})}{p_{t}(\mathbf{x})}+\mathbb{D}_{\text{KL}}(p_{t}(\cdot)||q(\cdot))\right]\right)+\mathbb{Z}^{\prime},$
(6.7)

where $\mathbb{Z}$ and $\mathbb{Z}^{\prime}$ denote constant normalization
factors.

For a proof of Lemma 6.4, see Section C.2. In principle, upon convergence,
both the reverse and forward KL-divergences should find the optimally fine-
tuned distribution, $q(\mathbf{x})$ in this simple setting. But to understand
their behavior in relevant practical situations, we are particularly
interested in understanding their behavior at intermediate points during
training, when either divergence is not minimized to exactly 0. Insights about
intermediate points in training can make useful predictions about practical
problems when early stopping is used to prevent overfitting and the loss is
rarely 0. Thus, our result below attempts to characterize these objectives at
any given iteration $t$:

######  Theorem 6.5.

Let $p^{f}_{t+1}(\mathbf{x})$ be the distribution obtained after one gradient
step, starting from $p_{t}$ using the forward KL divergence. Likewise, let
$p^{r}_{t+1}(\mathbf{x})$ be the distribution obtained using the reverse KL
divergence, from $p_{t}$. Define $\Delta^{f}_{t}$ and $\Delta^{r}_{t}$ as the
difference of log probability ratios across two categories $\mathbf{x}_{1}$
and $\mathbf{x}_{2}$, obtained from the forward and reverse divergences
respectively:

$\displaystyle\Delta_{t}^{f}(\mathbf{x}_{1},\mathbf{x}_{2}):=\log\frac{p^{f}_{t+1}(\mathbf{x}_{1})}{p_{t}(\mathbf{x}_{1})}-\log\frac{p^{f}_{t+1}(\mathbf{x}_{2})}{p_{t}(\mathbf{x}_{2})},$
(6.8)

and $\Delta_{t}^{r}$ is similarly defined. Then we have the following (for
appropriate positive constants $\beta$, $\delta_{1}$, $\delta_{2})$:

  1. 1.

Reverse KL modifies probability mass more aggressively than the forward KL. If
$\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are such that, $\delta_{1}\leq
p_{t}(\mathbf{x}_{1})=p_{t}(\mathbf{x}_{2})\leq 1-\delta_{2}$ (where
$\delta_{1}>0$, $\delta_{2}>0$), but $q(\mathbf{x}_{1})\geq
q(\mathbf{x}_{2})+\beta$, then,
$\Delta^{r}_{t}(\mathbf{x}_{1},\mathbf{x}_{2})>\Delta^{f}_{t}(\mathbf{x}_{1},\mathbf{x}_{2})$.

  2. 2.

Reverse KL increases probability mass only on a subset of categories that
equal target likelihoods. If $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are such
that, $p_{t}(\mathbf{x}_{2})+\beta\leq p_{t}(\mathbf{x}_{1})\leq
1-\delta_{2}$, and $q(\mathbf{x}_{1})=q(\mathbf{x}_{2})>\mathbf{c}_{0}\cdot
p_{t}(\mathbf{x}_{1})$, where $\mathbf{c}_{0}$ is a positive constant $>1$,
then,
$\Delta^{r}_{t}(\mathbf{x}_{1},\mathbf{x}_{2})>\Delta^{f}_{t}(\mathbf{x}_{1},\mathbf{x}_{2})$.

  3. 3.

Reverse KL aggressively reduces probability mass on less-likely categories in
the target distribution. If $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are such
that, $p_{t}(\mathbf{x}_{2})+\beta\leq p_{t}(\mathbf{x}_{1})\leq
1-\delta_{2}$, and $q(\mathbf{x}_{1})=q(\mathbf{x}_{2})<\mathbf{c}_{1}\cdot
p_{t}(\mathbf{x}_{2})$, where $\mathbf{c}_{1}$ is a positive constant $<1$,
then,
$\Delta^{r}_{t}(\mathbf{x}_{1},\mathbf{x}_{2})<\Delta^{f}_{t}(\mathbf{x}_{1},\mathbf{x}_{2})$.

A proof of Theorem 6.5 is shown in Section C.3. Essentially, this theorem
enlists several cases where the forward KL modifies probability mass in
different amounts across various categories, but the reverse KL acts
disproportionately. In particular, case 1 says that the reverse KL exhibits
more disproportionate probability mass changes on categories with equal
likelihood $p_{t}(\mathbf{x})$, due to the logarithmic dependency on the
probability mass $q(\mathbf{x})$ (compared to the linear dependency for the
forward KL). Case 2 says that when the target value $q(\mathbf{x})$ for two
categories is much larger than the probability mass currently assigned to
those categories, then the reverse KL can attempt to preferentially increase
probability mass more in the category with a larger likelihood
$p_{t}(\mathbf{x})$ under certain conditions. Finally, case 3 shows that when
the likelihood of a category is significantly larger than the target
$q(\mathbf{x})$, the reverse KL is more effective at reducing this probability
mass and re-distributing it to other categories within one update step.
Finally, consider another special case, where the difference
$q(\mathbf{x})-p_{t}(\mathbf{x})$ is identical for two categories
$\mathbf{x}_{1}$ and $\mathbf{x}_{2}$. In this case, while the forward KL will
increase log probability ratios for both $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$
equally, i.e., $\Delta^{f}(\mathbf{x}_{1},\mathbf{x}_{2})=0$, the reverse KL
will prioritize the category with a higher $p_{t}(\mathbf{x})$ value. These
results highlight some scenarios under which the reverse KL can more
efficiently re-organize probability mass across categories.

Mode-seeking vs. mode-covering objectives for categorical distributions
Typically the benefits of mode-seeking behavior are more apparent when the
model $p(\mathbf{x})$ is unable to realize the target distribution
$q(\mathbf{x})$ such that minimizing either KL would give rise to different
solutions. Unlike this argument, we show that even when the $p(\mathbf{x})$
can fully represent the target distribution $q(\mathbf{x})$, reverse KL can
quickly re-distribute probability mass to only a subset of the required
categories likely in target distribution, within a few gradient steps.

###  7 Discussion, Conclusion, and Limitations

We attempted to understand which components are particularly important for
fine-tuning language models with preference data. Through extensive
experiments on different fine-tuning problems in both didactic and LLM
settings, we established that on-policy sampling is crucial for good
performance especially when the peak in the ground-truth reward lies in less-
likely regions of the reference policy initialization. That said, in practice,
doing so requires preference datasets with broader coverage than the reference
policy. We also showed that negative gradients can enable faster convergence
and that objectives that induce a negative gradient are complementary to using
on-policy sampling. Finally, we show that the notion of mode-seeking
divergences unifies the notion of on-policy sampling and negative gradient.
Our case study comparing forward and reverse KL divergences demonstrates the
superiority of the reverse KL divergence in re-distributing probability mass
efficiently, supporting our empirical findings pertaining to on-policy
sampling and negative gradients.

While we conceptualize our observations, a limitation is that we don’t derive
rigorous statistical guarantees in this work. As an example, we note that
while the notion of concentrability coefficients (and associated guarantees)
can potentially provide guarantees on on-policy sampling, to the best of our
knowledge the notion of the negative gradient is not fully studied in the
literature. We conjecture that negative gradient can perhaps be formalized
statistically from the lens of providing a lower variance learning signal; it
would be interesting for future work to formalize this. It would also be
interesting to study more recent approaches based on minimax formulations
(e.g., Munos et al. (2023); Yuan et al. (2024); Swamy et al. (2024); Chen et
al. (2024)) in our empirical and conceptual framework. Next, while we consider
the coverage of preference data relative to that of the reference policy in
our study, this is a simplification that does not account for the coverage of
the pre-training distribution which future work can incorporate. Finally, we
remark that our study does not explore the effect of reward model quality,
which tends to also play a central role in LLM fine-tuning. It would be
interesting to extend our analysis to incorporate the role of reward model
quality and parameterization.

### Acknowledgements

We would like to thank Yi Su, Rishabh Agarwal, Zhang-Wei Hong, Young Geng,
Abitha Thankaraj, Yuxiao Qu, So Yeon Min, Yutong He, Kevin Li, Sukjun Hwang,
Khurram Yamin, Charlie Snell, Amrith Setlur, Kaylee Burns, Eric Mitchell, and
others in CMU Russ Lab, CMU Auton Lab, Stanford IRIS Lab, and Stanford Ermon
Group for discussions and feedback. AK thanks Aleksandra Faust, George Tucker,
and Sergey Levine for informative discussions. This research is supported by
computational resources from Google TPU Research Cloud (TRC) and the National
Science Foundation. FT thanks Ruslan Salakhutdinov for insightful suggestions
during this project. AS gratefully acknowledges the support of the NSF
Graduate Research Fellowship Program.

### References

  * Adolphs et al. (2022) Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston.  The CRINGE Loss: Learning what language not to model.  _arXiv e-prints_ , art. arXiv:2211.05826, November 2022.  10.48550/arXiv.2211.05826.
  * Agarwal et al. (2023) Rishabh Agarwal, Nino Vieillard, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem.  Gkd: Generalized knowledge distillation for auto-regressive sequence models.  _arXiv preprint arXiv:2306.13649_ , 2023.
  * Ahmadian et al. (2024) Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker.  Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.  _arXiv preprint arXiv:2402.14740_ , 2024.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.  ISSN 00063444.  URL http://www.jstor.org/stable/2334029.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=bx24KpJ4Eb.  Survey Certification.
  * Chang et al. (2024) Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, and Wen Sun.  Dataset Reset Policy Optimization for RLHF.  _arXiv e-prints_ , art. arXiv:2404.08495, April 2024.  10.48550/arXiv.2404.08495.
  * Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.  _arXiv e-prints_ , art. arXiv:2401.01335, January 2024.  10.48550/arXiv.2401.01335.
  * ContextualAI (2024) ContextualAI.  Human-centered loss functions (halos), 2024.  URL https://github.com/ContextualAI/HALOs.
  * Coste et al. (2023) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger.  Reward Model Ensembles Help Mitigate Overoptimization.  _arXiv e-prints_ , art. arXiv:2310.02743, October 2023.  10.48550/arXiv.2310.02743.
  * Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  UltraFeedback: Boosting Language Models with High-quality Feedback.  _arXiv e-prints_ , art. arXiv:2310.01377, October 2023.  10.48550/arXiv.2310.01377.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  _arXiv preprint arXiv:2305.14233_ , 2023.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang.  RAFT: Reward ranked finetuning for generative foundation model alignment.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=m7p5O7zblY.
  * Dubois et al. (2024) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback, 2024.
  * Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant.  Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking.  _arXiv e-prints_ , art. arXiv:2312.09244, December 2023.  10.48550/arXiv.2312.09244.
  * Ethayarajh et al. (2023) Kawin Ethayarajh, Winnie Xu, Dan Jurafsky, and Douwe Kiela.  Human-aware loss functions (halos).  Technical report, Contextual AI, 2023.  https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf.
  * Gao et al. (2022) Leo Gao, John Schulman, and Jacob Hilton.  Scaling Laws for Reward Model Overoptimization.  _arXiv e-prints_ , art. arXiv:2210.10760, October 2022.  10.48550/arXiv.2210.10760.
  * Gheshlaghi Azar et al. (2023) Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A General Theoretical Paradigm to Understand Learning from Human Preferences.  _arXiv e-prints_ , art. arXiv:2310.12036, October 2023.  10.48550/arXiv.2310.12036.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.  Reinforced self-training (rest) for language modeling.  _arXiv preprint arXiv:2308.08998_ , 2023.
  * Guo et al. (2024) Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel.  Direct Language Model Alignment from Online AI Feedback.  _arXiv e-prints_ , art. arXiv:2402.04792, February 2024.  10.48550/arXiv.2402.04792.
  * Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne.  Orpo: Monolithic preference optimization without reference model, 2024.
  * Hu et al. (2023) Jian Hu, Li Tao, June Yang, and Chandler Zhou.  Aligning Language Models with Offline Learning from Human Feedback.  _arXiv e-prints_ , art. arXiv:2308.12050, August 2023.  10.48550/arXiv.2308.12050.
  * Jain et al. (2023) Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger.  Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks.  _arXiv e-prints_ , art. arXiv:2311.12786, November 2023.  10.48550/arXiv.2311.12786.
  * (24) Andrej Karpathy.  minGPT.  URL https://github.com/karpathy/minGPT.
  * Khaki et al. (2024) Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra.  RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models.  _arXiv e-prints_ , art. arXiv:2402.10038, February 2024.  10.48550/arXiv.2402.10038.
  * Kidambi et al. (2020) Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims.  Morel: Model-based offline reinforcement learning.  _arXiv preprint arXiv:2005.05951_ , 2020.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * Kirk et al. (2023) Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu.  Understanding the Effects of RLHF on LLM Generalisation and Diversity.  _arXiv e-prints_ , art. arXiv:2310.06452, October 2023.  10.48550/arXiv.2310.06452.
  * Korbak et al. (2022) Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Marc Dymetman.  On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting.  _Advances in Neural Information Processing Systems_ , 35:16203–16220, 2022.
  * Lee et al. (2024) Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea.  A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity.  _arXiv e-prints_ , art. arXiv:2401.01967, January 2024.  10.48550/arXiv.2401.01967.
  * Liu et al. (2024) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu.  Statistical rejection sampling improves preference optimization.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=xbjSwwrQOe.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi.  Quark: Controllable text generation with reinforced unlearning.  _Advances in neural information processing systems_ , 35:27591–27609, 2022.
  * Mei et al. (2022) Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, and Dale Schuurmans.  The role of baselines in policy gradient optimization.  _Advances in Neural Information Processing Systems_ , 35:17818–17830, 2022.
  * Mukobi et al. (2023) Gabriel Mukobi, Peter Chatain, Su Fong, Robert Windesheim, Gitta Kutyniok, Kush Bhatia, and Silas Alberti.  SuperHF: Supervised Iterative Learning from Human Feedback.  _arXiv e-prints_ , art. arXiv:2310.16763, October 2023.  10.48550/arXiv.2310.16763.
  * Munos and Szepesvári (2008) Rémi Munos and Csaba Szepesvári.  Finite-time bounds for fitted value iteration.  _Journal of Machine Learning Research_ , 9(5), 2008.
  * Munos et al. (2023) Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot.  Nash Learning from Human Feedback.  _arXiv e-prints_ , art. arXiv:2312.00886, December 2023.  10.48550/arXiv.2312.00886.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.
  * Pang et al. (2024) Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston.  Iterative reasoning preference optimization, 2024.
  * Peters and Schaal (2007) Jan Peters and Stefan Schaal.  Reinforcement learning by reward-weighted regression for operational space control.  In _Proceedings of the 24th International Conference on Machine Learning_ , pages 745–750. ACM, 2007.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Rafailov et al. (2024) Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.  From r to $q^{*}$: Your language model is secretly a q-function.  _arXiv preprint arXiv:2404.12358_ , 2024.
  * Rosset et al. (2024) Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie.  Direct nash optimization: Teaching language models to self-improve with general preferences.  _arXiv preprint arXiv:2404.03715_ , 2024.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal Policy Optimization Algorithms.  _arXiv e-prints_ , art. arXiv:1707.06347, July 2017.  10.48550/arXiv.1707.06347.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sharma et al. (2024) Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar.  A critical evaluation of ai feedback for aligning large language models.  _arXiv preprint arXiv:2402.12366_ , 2024.
  * Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett.  A long way to go: Investigating length correlations in rlhf.  _arXiv preprint arXiv:2310.03716_ , 2023.
  * Sutton et al. (1999) Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In S. Solla, T. Leen, and K. Müller, editors, _Advances in Neural Information Processing Systems_ , volume 12. MIT Press, 1999.  URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf.
  * Swaminathan and Joachims (2015) Adith Swaminathan and Thorsten Joachims.  The self-normalized estimator for counterfactual learning.  In _advances in neural information processing systems_ , pages 3231–3239, 2015.
  * Swamy et al. (2024) Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal.  A minimaximalist approach to reinforcement learning from human feedback.  _arXiv preprint arXiv:2401.04056_ , 2024.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf.  Zephyr: Direct distillation of lm alignment, 2023.
  * von Werra et al. (2020) Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang.  Trl: Transformer reinforcement learning.  https://github.com/huggingface/trl, 2020.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston.  Neural text generation with unlikelihood training.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SJeYe0NtvH.
  * Williams (1992) R. J. Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ , 8(3-4):229–256, May 1992.
  * Xie et al. (2022) Annie Xie, Fahim Tajwar, Archit Sharma, and Chelsea Finn.  When to ask for help: Proactive interventions in autonomous reinforcement learning.  In _Advances in Neural Information Processing Systems_ , volume 35, 2022.
  * Xiong et al. (2023) Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang.  Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint.  _arXiv e-prints_ , art. arXiv:2312.11456, December 2023.  10.48550/arXiv.2312.11456.
  * Xu et al. (2024) Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu.  Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study.  _arXiv e-prints_ , art. arXiv:2404.10719, April 2024.  10.48550/arXiv.2404.10719.
  * Yarats et al. (2021a) Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.  Mastering visual continuous control: Improved data-augmented reinforcement learning.  _arXiv preprint arXiv:2107.09645_ , 2021a.
  * Yarats et al. (2021b) Denis Yarats, Ilya Kostrikov, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In _International Conference on Learning Representations_ , 2021b.  URL https://openreview.net/forum?id=GY6-6sTvGaf.
  * Yu et al. (2020) Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.  Mopo: Model-based offline policy optimization.  _arXiv preprint arXiv:2005.13239_ , 2020.
  * Yu et al. (2021) Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn.  Combo: Conservative offline model-based policy optimization.  _Advances in neural information processing systems_ , 34:28954–28967, 2021.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_ , 2024.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-Rewarding Language Models.  _arXiv e-prints_ , art. arXiv:2401.10020, January 2024.  10.48550/arXiv.2401.10020.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu.  SLiC-HF: Sequence Likelihood Calibration with Human Feedback.  _arXiv e-prints_ , art. arXiv:2305.10425, May 2023.  10.48550/arXiv.2305.10425.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences, 2020.

## Appendices

###  Appendix A Connections to Existing Fine-Tuning Results

Our proposed framework also allows us to explain experiments and evaluations
in several existing LLM fine-tuning results, and as a result, implies several
practical guidelines for LLM practitioners. On the AlpacaFarm benchmark
(Dubois et al., 2024), our results corroborate the gap between conditional
supervised fine-tuning objectives such as binary FeedME and reward
conditioning, and RL or contrastive training methods such as PPO and DPO:
these results are perhaps even more extreme in that these conditional and
weighted supervised fine-tuning objectives are not even able to outperform
regular SFT. Methods that utilize on-policy sampling such as ReST (Gulcehre et
al., 2023) and Quark (Lu et al., 2022) do outperform SFT but still
underperform on-policy RL or on-policy contrastive training. The top-
performing methods on the benchmark are offline DPO, which uses a negative
gradient, and PPO, which leverages on-policy sampling.

Additionally, methods such as self-rewarding language models (Yuan et al.,
2024), RSO (Liu et al., 2024), OAIF (Guo et al., 2024), DR-PO (Chang et al.,
2024), Hybrid-DPO (Xiong et al., 2023), and RS-DPO (Khaki et al., 2024) couple
on-policy sampling or rejection sampling with contrastive training objectives.
These works corroborate our observation regarding the efficacy of on-policy
sampling and negative gradients and how they are complementary. Approaches
such as CRINGE (Adolphs et al., 2022) combine maximum likelihood with a token
level contrastive loss term and show gains over solely utilizing supervised
likelihood, corroborating our insights about negative gradients.

Concurrently to us, Xu et al. (2024) show that on many practical LLM fine-
tuning problems offline DPO underperforms on-policy PPO. While we do not study
the same LLM fine-tuning problems, the insights from this work corroborate our
findings, which in turn extend insights from this work. For instance, this
work observes that DPO can learn to find out-of-distribution responses, which
is consistent with our analysis in Section 5.2.2 that offline DPO training
might increase probability mass on the highly likely regions of
$\pi_{\theta}$, deviating significantly from the distribution of preferred
responses $p(\mathbf{y}_{w}|\mathbf{x})$. To avoid this issue, this work
prescribes an iterated DPO recipe where the reference policy (i.e., the SFT
policy in their setting) is used to iteratively collect new samples for DPO
training. Section 5.3 arrives at a similar conclusion that using on-policy
samples for policy optimization, though we recommend collecting samples from
the current policy and not the reference policy, which might fail to cover
important regions of the space when the peak in the reward function appears
farther away from the high-likely regions of the reference policy.

###  Appendix B Computational vs Wall-Clock Time Tradeoff for Various Methods

| Bandit (R1) | Min Length | Skew Length
---|---|---|---
| Reward ($\uparrow$) | Time | Completion Length ($\downarrow$) | Time | Completion Length ($\downarrow$) | Time
Offline DPO / IPO | 0.82 (0.04) | 1.7 hours | 1.0 (0.0) | 1.3 hours | 11.8 (14.0) | 0.12 hours
On-policy PPO | 0.92 (0.01) | 0.93 hours | 20.5 (25.4) | 4.84 hours | 15.8 (11.1) | 7.26 hours
On-policy RWR | 0.88 (0.01) | 0.12 hours | 65.5 (36.7) | 15.5 hours | 15.8 (9.3) | 15.5 hours
On-policy DPO / IPO | 0.92 (0.01) | 0.12 hours | 1.0 (0.0) | 0.4 hours | 0.0 (0.0) | 0.4 hours

Table 3: Wall-clock time comparisons. Comparison between on-policy and offline
variants of contrastive objectives (DPO/IPO) in terms of reward and wall-clock
time required till convergence of the run. Generally, on-policy contrastive
approaches achieve both superior reward and wall-clock time as opposed to
offline contrastive approaches (offline DPO/IPO) and on-policy RL (PPO, RWR).
Synthetic LLM experiments use a single A40 GPU. Bandit experiments use a
Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz CPU, with 4 threads.

A natural takeaway extending the empirical results from Section 5.3 is that
on-policy variants of contrastive approaches might provide for an better
tradeoff between computation and wall-clock time. We perform a comparison of
wall-clock time needed to run our experiments in Table 3. in particular, we
found that on-policy DPO only requires 0.4 hours to converge, while offline
DPO requires a wall-clock time of 1.3 hours to converge to the same solution
in the Min Length scenario. In the Skew Length scenario, where the learned
policy must deviate from the initial reference policy substantially, we find
that while offline DPO can converge a bit quickly (0.12 hours), it flatlines
at a sub-optimal solution (completion length of 11.8) as compared to on-policy
DPO which takes merely 0.4 hours to reach a more optimal solution. This is far
more time-efficient compared to other on-policy methods such as PPO and RWR
that present a sampling bottleneck.

###  Appendix C More Details on Conceptual Unification and Theoretical
Analysis

####  C.1 Unifying On-Policy Sampling and Negative Gradients via Mode-Seeking
Divergences

Here we provide proofs for the claims in 6.1. We will show that on-policy
methods and offline constrastive methods, both are mode-seeking as opposed to
supervised maximum likelihood approaches, which are mode-covering. This
conceptually explains the differences in their behaviors that we observe in
our experiments.

#####  C.1.1 On-policy Methods Are Mode-Seeking

First, we prove Lemma 6.1, i.e., we want to show that on-policy RL methods and
on-policy versions of weighted supervised learning methods optimize
regularized version of a reverse KL-divergence.

###### Proof.

Both on-policy RL algorithms and on-policy versions of weighted supervised
learning, optimize the following loss function:

$\displaystyle\mathcal{L}_{\text{RL}}(\mathcal{D}_{\text{pref}},\pi_{\theta})=-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(.|\mathbf{x})}[r(\mathbf{x},\mathbf{y})]-\beta\mathbb{D}_{\text{KL}}[\pi_{\theta}(.|\mathbf{x})||\pi_{\mathrm{ref}}(.|\mathbf{x})]]$
(C.1)

Following Appendix A.1 of Rafailov et al. (2023), there exists some policy
$\pi^{*}$ such that we can express the reward function
$r(\mathbf{x},\mathbf{y})$ as follows:

$\displaystyle r(\mathbf{x},\mathbf{y})=\beta\log
Z(\mathbf{x})+\beta\log\left(\frac{\pi^{*}(\mathbf{y}|\mathbf{x})}{\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})}\right)$

where
$Z(\mathbf{x})=\sum_{\mathbf{y}}\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})\exp\left(\frac{r(\mathbf{x},\mathbf{y})}{\beta}\right)$
is the partition function. Combining these two, we get:

$\displaystyle\mathcal{L}_{\text{RL}}(\mathcal{D}_{\text{pref}},\pi_{\theta})={}$
$\displaystyle-\beta\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(.|\mathbf{x})}\left[\log
Z(\mathbf{x})+\log\left(\frac{\pi^{*}(\mathbf{y}|\mathbf{x})}{\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})}\right)\right]-\mathbb{D}_{\text{KL}}[\pi_{\theta}(.|\mathbf{x})||\pi_{\mathrm{ref}}(.|\mathbf{x})]\right]$
$\displaystyle={}$
$\displaystyle-\beta\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(.|\mathbf{x})}\left[\log
Z(\mathbf{x})+\log\left(\frac{\pi^{*}(\mathbf{y}|\mathbf{x})}{\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})}\right)\right]-\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(.|\mathbf{x})}\left[\log\left(\frac{\pi_{\theta}(\mathbf{y}|\mathbf{x})}{\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})}\right)\right]\right]$
$\displaystyle={}$
$\displaystyle-\beta\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(.|\mathbf{x})}\left[\log
Z(\mathbf{x})-\log\left(\frac{\pi_{\theta}(\mathbf{y}|\mathbf{x})}{\pi^{*}(\mathbf{y}|\mathbf{x})}\right)\right]\right]$
$\displaystyle={}$
$\displaystyle-\beta\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}[\log
Z(\mathbf{x})]+\beta\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}[\mathbb{D}_{\text{KL}}[\pi_{\theta}(.|\mathbf{x})||\pi^{*}(.|\mathbf{x})]]$

Note that $Z(\mathbf{x})$ does not depend on $\pi_{\theta}$. Therefore,
minimizing $\mathcal{L}_{\text{RL}}$ with respect to $\pi_{\theta}$ is
equivalent to optimizing the reverse KL-divergence. Since optimizing the
reverse KL-divergence is mode-seeking, we see that on-policy RL algorithms
have mode-seeking behavior. ∎

#####  C.1.2 Contrastive Approaches (e.g., DPO/IPO) are Mode-Seeking

Next, we show that this is also the case for contrastive approaches as we
prove Lemma 6.2.

###### Proof.

First consider an input $\mathbf{x}$. Consider the gradient update (with a
small enough learning rate):

$\theta_{t+1}\leftarrow\theta_{t}-\eta[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\cdot
c_{1}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\cdot
c_{2}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})]$

We shall prove that for all possible models $\theta$ and for all $t$, there
always exists appropriate pairing of positive and negative samples
$(\mathbf{y}_{w},\mathbf{y}_{l})$, such that after taking the gradient update,
we have:

$\displaystyle\omega_{t+1}\Big{|}_{{c_{2}>0}}\geq\omega_{t+1}\Big{|}_{{c_{2}=0}}$

The core idea behind this proof is the normalization of the probability
simplex. We proceed with a combination of mathematical inducation and
contradiction: assume that
$\omega_{t}\Big{|}_{{c_{2}>0}}\geq\omega_{t}\Big{|}_{{c_{2}=0}}$, but for all
possible pairings $(\mathbf{y}_{w},\mathbf{y}_{l})$, we have
$\omega_{t+1}\Big{|}_{{c_{2}>0}}<\omega_{t+1}\Big{|}_{{c_{2}=0}}$. We will
show that this is not possible. To do this, we first derive the expressions
for $\omega_{t+1}$ and then study under what conditions is it possible that
for any pairing of positives and negatives, $\omega_{t+1}$ is smaller when
$c_{2}>0$. The expression for $\omega_{t+1}$ is given by:

$\displaystyle\omega^{\prime}$
$\displaystyle=\omega+\eta\left(\theta^{\prime}-\theta\right)^{\top}\left(\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right)$
$\displaystyle=\omega+\eta\left(\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\cdot
c_{1}-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\cdot
c_{2}\right)^{\top}\left(\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right)$
$\displaystyle=\omega+\eta\left[c_{1}\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right|\right|^{2}+c_{2}\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}-(c_{1}+c_{2})\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right].$

Now, define:
$f(t+1;\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})=\omega_{t+1}\Big{|}_{{c_{2}>0}}-\omega_{t+1}\Big{|}_{{c_{2}=0}}$,
then we have:

$\displaystyle f(t+1;\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})$
$\displaystyle=f(t;\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})+\eta\underbrace{\left[c_{1}\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}-c_{2}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right]}_{\Delta(\mathbf{y}_{l},\mathbf{y}_{w},\mathbf{x})}.$

Suppose that for all negatives $\mathbf{y}_{l}$ for a given positive response
$\mathbf{y}_{w}$, $f(t+1;\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})<0$, then:

$\displaystyle\forall\mathbf{y}_{l}$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\ \Delta(\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})<0$
$\displaystyle\implies$
$\displaystyle\forall\mathbf{y}_{l},\leavevmode\nobreak\ \leavevmode\nobreak\
c_{2}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})>c_{1}\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}$
$\displaystyle\implies$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\
c_{2}\mathbb{E}_{\mathbf{y}_{l}\sim\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})}\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right]>c_{1}\mathbb{E}_{\mathbf{y}_{l}\sim\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})}\left[\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}\right]$
$\displaystyle\implies$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\
c_{2}\mathbb{E}_{\mathbf{y}_{l}\sim\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})}\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right]^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})>c_{1}\mathbb{E}_{\mathbf{y}_{l}\sim\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})}\left[\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}\right]$
$\displaystyle\implies$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\
0>c_{1}\mathbb{E}_{\mathbf{y}_{l}\sim\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})}\left[\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}\right],$

which is a contradiction since $c_{1}>0$. This means that there is at least
one choice of $\mathbf{y}_{l}$ for a given $\mathbf{y}_{w}$, for which
$\Delta(\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})\geq 0$. This means that if
$f(t;\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})>0$ then
$f(t+1;\mathbf{y}_{w},\mathbf{y}_{l},\mathbf{x})>0$. Averaging over
$\mathbf{x}$ for all iterations then gives us the desired result, when
starting from an initialization when starting from the same initialization for
both the cases when $c_{2}>0$ and $c_{2}=0$.

For the second part of this result, we note that when the gradient dot
products are negative and $c_{2}>0$, then by writing down the Taylor
expansion, we can note that the likelihood of the positive sample increases by
an additional
$-c_{2}\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})^{\top}\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})\right]$
and decreases by an additional amount given by
$c_{2}\mathbb{E}_{\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l}\sim\mathcal{D}}\left[\left|\left|\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right|\right|^{2}\right]$
for the negative response. This proves the second part of this statement. ∎

Gradients for both DPO and IPO exhibit the form in Lemma 6.2. We now show that
the gradient of both DPO and IPO takes the form shown in Equation 6.1. From
Rafailov et al. (2023), the gradient of the DPO loss is:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\mathrm{ref}})=-\beta\mathbb{E}_{(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\sim\mathcal{D}_{\text{pref}}}\left[c^{\text{DPO}}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\cdot\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})\right]\right]$

where
$c^{\text{DPO}}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})=\sigma\left(\beta\log\frac{\pi_{\theta}(\mathbf{y}_{l}|\mathbf{x})}{\pi_{\mathrm{ref}}\mathbf{y}_{l}|\mathbf{x})}-\beta\log\frac{\pi_{\theta}(\mathbf{y}_{w}|\mathbf{x})}{\pi_{\mathrm{ref}}\mathbf{y}_{w}|\mathbf{x})}\right)$.

Now we derive the gradient of the IPO loss. Define

$\displaystyle
c^{\text{IPO}}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})=2\cdot\left(\log\left(\frac{\pi_{\theta}(\mathbf{y}_{w}|x)}{\pi_{\mathrm{ref}}(\mathbf{y}_{w}|x)}\right)-\log\left(\frac{\pi_{\theta}(\mathbf{y}_{l}|x)}{\pi_{\mathrm{ref}}(\mathbf{y}_{l}|x)}\right)-\frac{\tau^{-1}}{2}\right)$

The gradient of the IPO loss is:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{IPO}}(\pi_{\theta};\pi_{\mathrm{ref}})={}$
$\displaystyle\nabla_{\theta}\mathbb{E}_{(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\sim\mathcal{D}_{\text{pref}}}\left[\left(\log\left(\frac{\pi_{\theta}(\mathbf{y}_{w}|x)\pi_{\mathrm{ref}}(\mathbf{y}_{l}|x)}{\pi_{\mathrm{ref}}(\mathbf{y}_{w}|x)\pi_{\theta}(\mathbf{y}_{l}|x)}\right)-\frac{\tau^{-1}}{2}\right)^{2}\right]$
$\displaystyle={}$
$\displaystyle\mathbb{E}_{(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\sim\mathcal{D}_{\text{pref}}}\left[c^{\text{IPO}}(\mathbf{x},\mathbf{y}_{w},\mathbf{y}_{l})\cdot\left[\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{w}|x)-\nabla_{\theta}\log\pi_{\theta}(\mathbf{y}_{l}|x)\right]\right]$

#####  C.1.3 Supervised Offline Algorithms are Mode-Covering

Now we prove Lemma 6.3, which shows that supervised offline methods that
optimize a maximum likelihood loss exhibit mode-covering behavior.

###### Proof.

Offline supervised methods optimize the following loss function:

$\displaystyle\mathcal{L}_{\text{off-
sup}}(\pi_{\theta};\pi_{\mathrm{ref}})={}$
$\displaystyle-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[\sum_{\mathbf{y}}\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})\log\pi_{\theta}(\mathbf{y}|\mathbf{x})\cdot
F(\mathbf{x},\mathbf{y})\right]$

Define a new distribution

$\tilde{\pi}(\mathbf{y}|\mathbf{x})=\frac{\pi_{\mathrm{ref}}(\mathbf{y}|\mathbf{x})\cdot
F(\mathbf{x},\mathbf{y})}{Z(\mathbf{x})}$

Here
$Z(\mathbf{x})=\sum_{\mathbf{z}}\pi_{\mathrm{ref}}(\mathbf{z}|\mathbf{x})\cdot
F(\mathbf{x},\mathbf{z})$ is the normalization constant. It is easy to check
that this a valid conditional distribution. This gives us:

$\displaystyle\mathcal{L}_{\text{off-
sup}}(\pi_{\theta};\pi_{\mathrm{ref}})={}$
$\displaystyle-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[Z(\mathbf{x})\sum_{\mathbf{y}}\tilde{\pi}(\mathbf{y}|\mathbf{x})\log\pi_{\theta}(\mathbf{y}|\mathbf{x})\right]$
$\displaystyle={}$
$\displaystyle\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[Z(\mathbf{x})\cdot\mathbb{E}_{\mathbf{y}\sim\tilde{\pi}(\mathbf{y}|\mathbf{x})}\left[\log\left(\frac{\tilde{\pi}(\mathbf{y}|\mathbf{x})}{\pi_{\theta}(\mathbf{y}|\mathbf{x})}\right)\right]\right]-\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[Z(\mathbf{x})\cdot\mathbb{E}_{\mathbf{y}\sim\tilde{\pi}(\mathbf{y}|\mathbf{x})}\left[\log\tilde{\pi}(\mathbf{y}|\mathbf{x})\right]\right]$
$\displaystyle={}$
$\displaystyle\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[Z(\mathbf{x})\cdot\mathbb{D}_{\text{KL}}(\tilde{\pi}(.|\mathbf{x})||\pi_{\theta}(.|\mathbf{x}))\right]+\mathbb{E}_{\mathbf{x}\sim\mathcal{D}_{\text{pref}}}\left[Z(\mathbf{x})\cdot
H(\tilde{\pi}(.|\mathbf{x}))\right]$

Hence offline supervised methods minimize the re-weighted forward KL-
divergence. ∎

####  C.2 Characterization of Gradients of Forward and Reverse KL

For simplicity, let $\mathcal{X}$ be our input space, and
$\mathcal{Y}=\\{1,\ldots,V\\}$ be the output space consisting of $V>1$
discrete tokens. Let $f:\mathcal{X}\rightarrow\mathbb{R}^{V}$ be our network
that outputs $V$ real values logits for inputs $x\in\mathcal{X}$. With respect
to the logits of $f$, we can define a probability distribution $p$ over
discrete tokens $1,\ldots,V$ using the softmax function, namely:

$\displaystyle p_{i}(x)=\frac{\exp(f_{i}(x))}{\sum_{k=1}^{V}\exp(f_{k}(x))}$
(C.2)

for any $x\in\mathcal{X}$.

Now assume that for some given input $x$, $q(x)$ is the true probability
distribution over tokens that we want to learn. One way to do this would be to
find $p$ that minimizes $\mathbb{D}_{\text{KL}}(q||p)$ via SGD:

$p_{0}(x)\leftarrow p_{\text{ref}}(x)$ $p_{t+1}(x)\leftarrow
p_{t}(x)-\eta\nabla_{f(x)}\mathbb{D}_{\text{KL}}(q(x)||p(x))\big{|}_{p=p_{t}}$

where $p_{\text{ref}}$ is reference distribution we start with, and $\eta$ is
the learning rate. $\mathbb{D}_{\text{KL}}(q||p)$ is called the forward-KL,
since $q$ is the true distribution of interest. In contrast,
$\mathbb{D}_{\text{KL}}(p||q)$ is called the reverse-KL, and one can try to
optimize $p$ by minimizing $\mathbb{D}_{\text{KL}}(p||q)$ in a similar
fashion.

We shall now prove Lemma 6.4. We break this lemma in two parts. First, let us
investigate how the gradients
$\nabla_{f(x)}\mathbb{D}_{\text{KL}}(q(x)||p(x))\big{|}_{p=p_{t}}$ and
$\nabla_{f(x)}\mathbb{D}_{\text{KL}}(p(x)||q(x))\big{|}_{p=p_{t}}$ look like:

######  Lemma C.1.

The gradients of forward and reverse KL are given by:

$\displaystyle\frac{\partial}{\partial
f_{j}}\mathbb{D}_{\text{KL}}(q(x)||p(x))=p_{j}(x)-q_{j}(x)$ (C.3)
$\displaystyle\frac{\partial}{\partial
f_{j}}\mathbb{D}_{\text{KL}}(p(x)||q(x))=p_{j}(x)\left[\log\frac{p_{j}(x)}{q_{j}(x)}-\mathbb{D}_{\text{KL}}(p(x)||q(x))\right]$
(C.4)

###### Proof.

We start with the definition of KL-divergence:

$\displaystyle\mathbb{D}_{\text{KL}}(q(x)||p(x))={}$
$\displaystyle\sum_{i}q_{i}(x)\log q_{i}(x)-\sum_{i}q_{i}(x)\log p_{i}(x)$
$\displaystyle={}$
$\displaystyle-H(q)-\sum_{i}q_{i}(x)\log\left(\frac{e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)$
$\displaystyle={}$
$\displaystyle-H(q)-\sum_{i}q_{i}(x)f_{i}(x)+\sum_{i}q_{i}(x)\log\left(\sum_{k}e^{f_{k}(x)}\right)$

Therefore, we have:

$\displaystyle\frac{\partial}{\partial
f_{j}}\mathbb{D}_{\text{KL}}(q(x)||p(x))={}$ $\displaystyle-
q_{j}(x)+\sum_{i}q_{i}(x)\left(\frac{e^{f_{j}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)$
$\displaystyle={}$ $\displaystyle-q_{j}(x)+\sum_{i}q_{i}(x)p_{j}(x)$
$\displaystyle={}$ $\displaystyle p_{j}(x)-q_{j}(x)$

This proves Equation C.3. Similarly, we can write:

$\displaystyle\mathbb{D}_{\text{KL}}(p(x)||q(x))={}$
$\displaystyle\sum_{i}p_{i}(x)\log p_{i}(x)-\sum_{i}p_{i}(x)\log q_{i}(x)$
$\displaystyle={}$
$\displaystyle\sum_{i}\left(\frac{e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)\left[f_{i}(x)-\log\left(\sum_{k}e^{f_{k}(x)}\right)\right]-\sum_{i}\log
q_{i}(x)\left(\frac{e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)$
$\displaystyle={}$
$\displaystyle\frac{\sum_{i}f_{i}(x)e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}-\left(\sum_{i}e^{f_{i}(x)}\right)\left(\frac{\log\left(\sum_{k}e^{f_{k}(x)}\right)}{\sum_{k}e^{f_{k}(x)}}\right)-\sum_{i}\log
q_{i}(x)\left(\frac{e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)$
$\displaystyle={}$
$\displaystyle\frac{\sum_{i}f_{i}(x)e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}-\log\left(\sum_{k}e^{f_{k}(x)}\right)-\frac{\sum_{i}\log
q_{i}(x)e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}$

Now we calculate the partial derivative with respect to $f_{j}$:

$\displaystyle\frac{\partial}{\partial
f_{j}}\frac{\sum_{i}f_{i}(x)e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}={}$
$\displaystyle\frac{\left(\sum_{k}e^{f_{k}(x)}\right)\frac{\partial}{\partial
f_{j}}\left(\sum_{i}f_{i}(x)e^{f_{i}(x)}\right)-\left(\sum_{i}f_{i}(x)e^{f_{i}(x)}\right)\left(\frac{\partial}{\partial
f_{j}}\sum_{k}e^{f_{k}(x)}\right)}{\left(\sum_{k}e^{f_{k}(x)}\right)^{2}}$
$\displaystyle={}$ $\displaystyle\frac{\frac{\partial}{\partial
f_{j}}\left(\sum_{i}f_{i}(x)e^{f_{i}(x)}\right)}{\sum_{k}e^{f_{k}(x)}}-\frac{e^{f_{j}(x)}\left(\sum_{i}f_{i}(x)e^{f_{i}(x)}\right)}{\left(\sum_{k}e^{f_{k}(x)}\right)^{2}}$
$\displaystyle={}$
$\displaystyle\frac{e^{f_{j}(x)}+f_{j}(x)e^{f_{j}(x)}}{\sum_{k}e^{f_{k}(x)}}-\frac{e^{f_{j}(x)}}{\sum_{k}e^{f_{k}(x)}}\left(\sum_{i}f_{i}(x)\left(\frac{e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)\right)$
$\displaystyle={}$ $\displaystyle
p_{j}(x)+f_{j}(x)p_{j}(x)-p_{j}(x)\left(\sum_{i}f_{i}(x)p_{i}(x)\right)$
$\frac{\partial}{\partial
f_{j}}\log\left(\sum_{k}e^{f_{k}(x)}\right)=\frac{e^{f_{j}(x)}}{\sum_{k}e^{f_{k}(x)}}=p_{j}(x)$

And for the third term,

$\displaystyle\frac{\partial}{\partial f_{j}}\frac{\sum_{i}\log
q_{i}(x)e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}={}$
$\displaystyle\frac{\left(\sum_{k}e^{f_{k}(x)}\right)\frac{\partial}{\partial
f_{j}}\left(\sum_{i}\log q_{i}(x)e^{f_{i}(x)}\right)-\left(\sum_{i}\log
q_{i}(x)e^{f_{i}(x)}\right)\frac{\partial}{\partial
f_{j}}\left(\sum_{k}e^{f_{k}(x)}\right)}{\left(\sum_{k}e^{f_{k}(x)}\right)^{2}}$
$\displaystyle={}$ $\displaystyle\frac{e^{f_{j}(x)}\log
q_{j}(x)}{\sum_{k}e^{f_{k}(x)}}-\left(\frac{e^{f_{j}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)\left(\sum_{i}\log
q_{i}(x)\frac{e^{f_{i}(x)}}{\sum_{k}e^{f_{k}(x)}}\right)$ $\displaystyle={}$
$\displaystyle p_{j}(x)\log q_{j}(x)-p_{j}(x)\sum_{i}p_{i}(x)\log q_{i}(x)$

Putting it all together, we obtain:

$\displaystyle\nabla_{f_{j}}\mathbb{D}_{\text{KL}}(p(x)||q(x))={}$
$\displaystyle p_{j}(x)\cdot\left(f_{j}(x)-\log
q_{j}(x)\right)-p_{j}(x)\cdot\left(\sum_{i}p_{i}(x)\cdot\left(f_{i}(x)-\log
q_{i}(x)\right)\right)$ $\displaystyle={}$ $\displaystyle
p_{j}(x)\cdot\log\frac{p_{j}(x)}{q_{j}(x)}-p_{j}(x)\sum_{i}p_{i}(x)\cdot\log\frac{p_{i}(x)}{q_{i}(x)}$
$\displaystyle={}$ $\displaystyle
p_{j}(x)\left[\log\frac{p_{j}(x)}{q_{j}(x)}-\mathbb{D}_{\text{KL}}(p(x)||q(x))\right]$

completing our proof. ∎

Proof for Lemma 6.4. Now, if the logits $f_{t}$ are being updated with
gradient descent on loss $\mathcal{L}$, the distribution at the next step
$p^{t+1}$ is given by:

$\displaystyle p^{t+1}_{j}(x)={}$
$\displaystyle\exp(f^{t+1}_{j}(x))/\sum_{i}\exp(f^{t+1}_{i}(x))$
$\displaystyle={}$
$\displaystyle\frac{\exp(f^{t}_{j}(x))-\eta\nabla_{f^{t}_{j}}\mathcal{L})}{\sum_{i}\exp(f^{t}_{i}(x))}\cdot\frac{\sum_{i}\exp(f^{t}_{i}(x))}{\sum_{i}\exp(f^{t}_{i}(x)-\eta\nabla_{f^{t}_{i}}\mathcal{L})}$
$\displaystyle={}$ $\displaystyle
p^{t}_{j}(x)\cdot\frac{\exp\left(-\eta\nabla_{f^{t}_{j}}\mathcal{L}\right)}{\sum_{i}p^{t}_{i}(x)\exp\left(-\eta\nabla_{f^{t}_{i}}\mathcal{L}\right)}$

Let’s consider what the characterization of $p^{t+1}$ for the forward kl:

$\displaystyle p^{t+1}_{j}(x)$
$\displaystyle=p^{t}_{j}(x)\cdot\frac{\exp\left(-\eta\left(p_{j}^{t}(x)-q_{j}(x)\right)\right)}{\sum_{i}p^{t}_{i}(x)\exp\left(-\eta\left(p_{i}^{t}(x)-q_{i}(x)\right)\right)}$

Noticing that the denominator is just a normalization constant, we can write
this as:

$\displaystyle\frac{p^{t+1}_{j}(x)}{p^{t}_{j}(x)}\propto\exp\left(-\eta\left(p_{j}^{t}(x)-q_{j}(x)\right)\right)$
(C.5)

Similarly the characterization of $p^{t+1}$ for the reverse KL looks like:

$\displaystyle\frac{p^{t+1}_{j}(x)}{p^{t}_{j}(x)}$
$\displaystyle\propto\exp\left(-\eta\left(p^{t}_{j}(x)\left[\log\frac{p_{j}^{t}(x)}{q^{t}(x)}-\mathbb{D}_{\text{KL}}(p^{t}(x)||q(x))\right]\right)\right)$
(C.6)

This completes the proof of Lemma 6.4.

####  C.3 Quantifying the Differences Between Forward and Reverse KL

In this section we will prove Theorem 6.5: specifically, we will study certain
special cases to explain the differences between approaches that optimize the
forward and reverse KL divergences. We drop the subscript $t$ from all terms
to prevent notational clutter.

###### Proof.

We prove these statements case by case. First we prove the result for Case 1.
In this scenario, we have the following:

$\displaystyle\Delta^{f}(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle=\eta\left(q(\mathbf{x}_{1})-q(\mathbf{x}_{2})\right)$
$\displaystyle\Delta^{r}(\mathbf{x}_{1},\mathbf{x}_{2})$ $\displaystyle=\eta
p(\mathbf{x}_{1})\left[\log q(\mathbf{x}_{1})-\log q(\mathbf{x}_{2})\right].$

The gap between $\Delta^{f}$ and $\Delta^{r}$ is now given by:

$\displaystyle\Delta^{r}(\mathbf{x}_{1},\mathbf{x}_{2})-\Delta^{f}(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle=\eta\left[\log q(\mathbf{x}_{1})-\log
q(\mathbf{x}_{2})-\frac{q(\mathbf{x}_{1})-q(\mathbf{x}_{2})}{p(\mathbf{x}_{1})}\right].$

Now, we note by mean-value theorem, that there exists a
$c_{0}\in[q(\mathbf{x}_{2}),q(\mathbf{x}_{1})]$ such that,

$\displaystyle\log q(\mathbf{x}_{1})-\log q(\mathbf{x}_{2})=\frac{d\log
p}{dp}\Big{|}_{p=c_{0}}\cdot\left(q(\mathbf{x}_{1})-q(\mathbf{x}_{2})\right).$

Since $d\log p/dp=1/p>1$ for $c_{0}\in(0,1)$, we have that:

$\displaystyle\Delta^{r}(\mathbf{x}_{1},\mathbf{x}_{2})-\Delta^{f}(\mathbf{x}_{1},\mathbf{x}_{2})=\eta\cdot\left(q(\mathbf{x}_{1})-q(\mathbf{x}_{2})\right)\cdot\left[\frac{1}{c_{0}}-\frac{1}{p(\mathbf{x}_{1})}\right].$

This quantity is positive when $p(\mathbf{x}_{1})>c_{0}=\delta_{1}$. This
shows the result for Case 1.