# Direct Alignment of Language Models
via Quality-Aware Self-Refinement

Runsheng Yu1 Yong Wang2 Xiaoqi Jiao2 Youzhi Zhang3 James T. Kwok1
1Hong Kong University of Science and Technology
2LightSpeed Studios, Tencent
3Centre for Artificial Intelligence and Robotics, Hong Kong Institute of
Science & Innovation, CAS
{runshengyu,seaywang}<EMAIL_ADDRESS>
youzhi.zhang<EMAIL_ADDRESS>Work done during the internship
at Tencent LightSpeed Studios.

###### Abstract

Reinforcement Learning from Human Feedback (RLHF) has been commonly used to
align the behaviors of Large Language Models (LLMs) with human preferences.
Recently, a popular alternative is Direct Policy Optimization (DPO), which
replaces an LLM-based reward model with the policy itself, thus obviating the
need for extra memory and training time to learn the reward model. However,
DPO does not consider the relative qualities of the positive and negative
responses, and can lead to sub-optimal training outcomes. To alleviate this
problem, we investigate the use of intrinsic knowledge within the on-the-fly
fine-tuning LLM to obtain relative qualities and help to refine the loss
function. Specifically, we leverage the knowledge of the LLM to design a
refinement function to estimate the quality of both the positive and negative
responses. We show that the constructed refinement function can help self-
refine the loss function under mild assumptions. The refinement function is
integrated into DPO and its variant Identity Policy Optimization (IPO).
Experiments across various evaluators indicate that they can improve the
performance of the fine-tuned models over DPO and IPO.

##  1 Introduction

Large Language Models (LLMs) have demonstrated significant capabilities across
various natural language processing tasks [34; 49; 42; 1]. Ensuring that these
LLMs produce the desired responses and behaviors that are aligned with human
preferences is crucial for safe and controllable AI systems [33]. To achieve
this, a popular method is Reinforcement Learning from Human Feedback (RLHF)
[13; 1; 5], which first trains a reward model using human-labeled response
pairs, and then uses this to adjust the policy parameters of the LLM [5; 33].
However, the reward model is often constructed by another LLM, which requires
further training and storage [3].

To reduce the storage and training time of the reward model, a variety of
methods have been proposed recently [17; 47; 3]. In particular, a prominent
solution is the Direct Policy Optimization (DPO) [3], which replaces the
reward model with the policy itself, thus obviating the need for an explicit
reward model. Recently, numerous variants of DPO have also been developed [44;
4; 3; 18; 39].

The objective of DPO is to consistently increase the likelihood of human-
preferred responses while reducing the likelihood of the undesired ones.
However, this strategy does not consider the relative qualities of the
positive and negative responses, and can lead to suboptimal training outcomes,
particularly when the preferred responses are not substantially superior, or
when the undesired responses are not adequately inferior [3; 43; 16].

To alleviate this issue, Amini et al. [3] and Zhou et al. [53] propose the use
of a score function to self-refine the objective. However, this approach
requires the availability of an ideal reward or score function, which may not
be always feasible. Similarly, Cui et al. [16] and Tunstall et al. [43] employ
GPT-4 [1] to select high-quality response pairs by scoring them. This method
requires a strong LLM to effectively filter the dataset, which again may not
always be practical. These considerations raise the question: _Can we achieve
this by using the inherent knowledge within the policy itself?_

Recently, self-alignment has attracted increasing attention due to its ability
to leverage the inherent knowledge of LLMs to enhance alignment capabilities,
obviating the necessity for additional human-annotated data [30; 2; 27; 46].
Inspired by this, we propose to utilize on-the-fly fine-tuning of an LLM’s
knowledge to help evaluate the quality of positive and negative responses. The
underlying premise is that even relatively weak LLMs possess some ability to
assess the quality of responses [23; 22]. Consequently, our objective is to
exploit this capability to more effectively evaluate response quality, thereby
enhancing the efficiency and accuracy of the fine-tuned model.

In this paper, we investigate the use of intrinsic knowledge within the LLM to
self-refine the loss function. In summary, the contributions of this work are
as follows:

  * •

We leverage the knowledge of the LLM to design a refinement function, which
estimates the quality of positive and negative responses.

  * •

We demonstrate that the constructed refinement function can help self-refine
the loss function under mild assumptions.

  * •

By utilizing the refinement function, we propose two novel approaches based on
DPO and its variant Identity Policy Optimization (IPO) [4].

Experimental results across various evaluators indicate that the proposed
self-refined methods improve the performance of the fine-tuned models compared
to their counterparts.

##  2 Preliminaries

###  2.1 Classical RLHF with Bradley-Terry Reward Model

Given a pre-trained large language model (LLM) $\pi_{\text{ref}}$ as
initialization, Reinforcement Learning from Human Feedback (RLHF) [13; 1; 5]
aims to learn an LLM $\pi$ that aligns with human values and preferences.
Specifically, let $x$ be the query, $y$ be the output of $\pi$, and $r$ be a
reward function that evaluates the performance of $y$ given $x$, RLHF tries to
maximize $r$ while ensuring that the trained LLM $\pi$ does not deviate
significantly from the pre-trained model $\pi_{\text{ref}}$. This can be
formulated as the following optimization problem [33; 31; 1]:

$\max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot|x)}r(y|x)-\beta\mathbb{D}_{\mathrm{KL}}[\pi(y|x)\|\pi_{\text{ref}}(y|x)],$
(1)

where $\mathbb{D}_{\mathrm{KL}}[\pi(y|x)\|\pi_{\text{ref}}(y|x)]$ is the
Kullback-Leibler divergence between $\pi$ and $\pi_{\text{ref}}$, and $\beta$
is a constant.

Since $r$ is unknown, the user needs to provide a set of preferences
$D\equiv\\{(x_{i},y^{+}_{i},y^{-}_{i})\\}^{N}_{i=1}$, where $y^{+}_{i}$ (resp.
$y^{-}_{i}$) is the positive (resp. negative) response for query $x_{i}$. A
suitable model, typically another LLM (parameterized by $\omega$) [26; 1],
then learns the reward function by maximizing the probability
$p(y_{i}^{+}\succ y_{i}^{-}|x)$ that $y_{i}^{+}$ is preferred over $y_{i}^{-}$
(denoted $y_{i}^{+}\succ y_{i}^{-}$) [33; 3; 4]. Typically, this probability
is defined by the Bradley-Terry preference model [8; 13] as:

$p(y_{i}^{+}\succ
y_{i}^{-}|x_{i})\equiv\sigma(r(y_{i}^{+}|x_{i})-r(y_{i}^{-}|x_{i})),$ (2)

where $\sigma(\cdot)$ is the logistic function. The optimal $r$ can then be
obtained by maximizing the log likelihood

$\max_{r}\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log[\sigma(r(y_{i}^{+}|x_{i})-r(y_{i}^{-}|x_{i}))].$
(3)

With the obtained $r$, we can then find $\pi$ by optimizing (1).

###  2.2 Direct Preference Optimization (DPO)

In RLHF, the reward model is represented by a LLM [33]. This can be time- and
memory-expensive. It is observed that the optimal $\pi$ in (1) indeed has the
closed form [35]:
$\pi(y|x)\propto\pi_{\text{ref}}(y|x)\exp\left(\frac{r(y|x)}{\beta}\right),$
and so

$r(y|x)=\beta\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}+c(x),$ (4)

where $c(x)\equiv\sum_{y\in\mathcal{Y}}\pi_{\text{ref}}(y\mid
x)\exp\left(\frac{1}{\beta}r(y|x)\right)$, and $\mathcal{Y}$ is the set of
responses. By plugging this into (3), the optimal policy can be found directly
by maximizing:

$\max\nolimits_{\pi}\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log\sigma\left(\beta\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\beta\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}\right).$
(5)

###  2.3 Identity Policy Optimisation (IPO)

DPO aims to maximize (1). However, our goal is to maximize the preference
rather than the reward [45]. Consequently, a better option is to maximize the
preference probabilities $p(y\succ y^{\prime}|x)$. To this end, IPO [4]
optimizes the following objective:

$\max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y^{+}\sim\pi(\cdot|x),y^{-}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ
y^{-}|x)-\beta\mathbb{D}_{\mathrm{KL}}[\pi(y^{+}|x)\|\pi_{\text{ref}}(y^{-}|x)].$
(6)

Similar to DPO, following [4], the optimal $\pi$ also has a closed-form
solution:

$\pi(y^{+}|x)\propto\pi_{\text{ref}}(y^{-}|x)\exp\left(\mathbb{E}_{y^{+}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ
y^{-}|x)/\beta\right).$ (7)

The loss for IPO is then defined as:

$\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\left[\left(\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}-\frac{1}{2\beta}\right)^{2}\right].$
(8)

###  2.4 Self-Alignment

The proposed method is also related to self-alignment [27; 46; 29], which uses
the LLM’s own knowledge to improve the alignment. For example, self-judgement
[27; 46; 40; 29; 50] uses the LLM as a judge to evaluate the generated answer.
Self-improving [30; 2; 11; 10; 9] improves LLM by playing game with another
LLM opponent. Our work uses the self-alignment ability of LLM to construct a
refinement function to adjust the quality of responses.

##  3 Proposed Method

###  3.1 Limitation of Maximizing Bradley-Terry Preference

Let $r^{*}$ and $r$ be the true and learned reward function respectively. For
any tuple $(x,y^{+},y^{-})\in D$, the true reward of the positive response
$y^{+}$ should exceed that of the negative response $y^{-}$, i.e., $y^{+}\succ
y^{-}\Leftrightarrow r^{*}(y^{+}|x)>r^{*}(y^{-}|x)$. Consider two tuples
$(x_{i},y_{i}^{+},y_{i}^{-})$ and $(x_{j},y_{j}^{+},y_{j}^{-})$. When
$r^{*}(y_{i}^{+}|x_{i})-r^{*}(y_{i}^{-}|x_{i})>r^{*}(y_{j}^{+}|x_{j})-r^{*}(y_{j}^{-}|x_{j})$,
we prefer $p(y_{i}^{+}\succ y_{i}^{-}|x_{i})>p(y_{j}^{+}\succ
y_{j}^{-}|x_{j})$, and so
$\sigma(r(y_{i}^{+}|x_{i})-r(y_{i}^{-}|x_{i}))>\sigma(r(y_{j}^{+}|x_{j})-r(y_{j}^{-}|x_{j}))$
from (2). In other words, the more informative tuple
$(x_{i},y_{i}^{+},y_{i}^{-})$ should be more important. However, RLHF simply
maximizes (3), which considers all tuples in the dataset $D$ equally.

###  3.2 Refining the Reward Difference between Positive and Negative
Responses

####  3.2.1 Intuition

To alleviate this problem, we propose adding a refinement function
$\Delta(y^{-},y^{+};x):\mathcal{Y}\times\mathcal{Y}\times\mathcal{X}\rightarrow\mathbb{R}$
(where $\mathcal{X}$ and $\mathcal{Y}$ are the sets of queries and responses,
respectively) to adjust $r(y^{+}|x)-r(y^{-}|x)$, so that problem (3) is
modified to:

$\max_{\pi}\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log[\sigma(r(y^{+}|x)-r(y^{-}|x)-\lambda\Delta(y^{-},y^{+};x))],$
(9)

where $\lambda$ is a positive constant. Intuitively, when
$\Delta(y^{-},y^{+};x)$ is large,
$\sigma(r(y^{+}|x)-r(y^{-}|x)-\lambda\Delta(y^{-},y^{+};x))$ becomes small,
and the optimization in (9) will tend to enlarge the “distance”
$r(y^{+}|x)-r(y^{-}|x)$ between the positive and negative responses. Thus, for
two tuples $(x,y^{+},y^{-})$ and $(x,\tilde{y}^{+},\tilde{y}^{-})$
corresponding to the same query $x$, we want to design a $\Delta$ such that
when $\Delta(y^{-},y^{+};x)>\Delta(\tilde{y}^{-},\tilde{y}^{+};x)$, their true
reward values satisfy:

$r^{*}(y^{+}|x)-r^{*}(y^{-}|x)>r^{*}(\tilde{y}^{+}|x)-r^{*}(\tilde{y}^{-}|x).$

In other words, a larger difference $r^{*}(y^{+}|x)-r^{*}(y^{-}|x)$ in the
true reward values between the positive and negative responses corresponds to
a larger $\Delta(y^{-},y^{+};x)$, and vice versa. However, obviously the
difficulty is that we do not have access to $r^{*}$.

####  3.2.2 Implementing $\Delta$ via Prompting

First, we assume that the LLM is capable of learning a reward function that
aligns with the true reward function. This premise is formalized as follows:

######  Assumption 3.1.

LLM $\pi$ can construct a reward model $r$ such that for any $(x,y^{+},y^{-})$
with $y^{+}\succ y^{-}$, the corresponding reward values satisfy
$r(y^{+}|x)>r(y^{-}|x)$.

As $r^{*}$ is unknown, with a capable LLM, a natural idea is to use $r$ as a
proxy of $r^{*}$. Recall from Section 3.2.1 that a large
$r^{*}(y^{+}|x)-r^{*}(y^{-}|x)$ should correspond to a large
$\Delta(y^{-},y^{+};x)$. Using (4), one can define $\Delta$ as

$\Delta_{\text{naive}}=r(y^{+}|x)-r(y^{-}|x)=\beta\left(\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}\right).$
(10)

Substituting this into the DPO objective (5), we have

$\displaystyle\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log\sigma\left(\beta\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\beta\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}-\lambda\Delta_{\text{naive}}(y^{-},y^{+};x)\right)$
$\displaystyle=$
$\displaystyle\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log\sigma\left((\beta-\lambda\beta)\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-(\beta-\lambda\beta)\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}\right).$

However, this is the same as the original DPO objective except for a scaling
of the regularization parameter $\beta$, and thus is not useful.

To alleviate this problem, our idea is to first improve the LLM performance by
concatenating the query $x$ with a prompt $p$, as $p\oplus x$, where $\oplus$
denotes concatenation. With (4) the reward with this prompt-augmented query
is:

$r(y|p\oplus x)=\beta\log\frac{\pi(y|p\oplus x)}{\pi_{\text{ref}}(y|p\oplus
x)}+c(x).$ (11)

With a good prompt, this $r$ is expected to be a good proxy of $r^{*}$.
Specifically, we use the following prompt $p$. Please generate a response with
a usefulness rating of 100 out of 100 for the following query. Note that the
response should be harmless. The term 100 indicates the level of usefulness,
where 100 is the maximum and 1 is the minimum. Query:

We assume that adding this prompt does not change the preference between the
positive and negative responses ($y^{+}$ and $y^{-}$).

######  Assumption 3.2.

For a given query $x$, if $y^{+}\succ y^{-}$, we still have $y^{+}\succ y^{-}$
with the prompt-augmented query $p\oplus x$.

Analogous to (10), we consider the following refinement function:

$\Delta_{\pi}\left(y^{-},y^{+};x\right)\equiv\beta\log\frac{\pi(y^{+}|p\oplus
x)\pi_{\text{ref}}(y^{-}|p\oplus x)}{\pi_{\text{ref}}(y^{+}|p\oplus
x)\pi(y^{-}|p\oplus x)}.$ (12)

Note that we have added subscript $\pi$ to explicitly indicate the dependence
of $\Delta$ on $\pi$. Obviously, when putting this $\Delta_{\pi}$ into the DPO
objective (5), it does not suffer from the same problem as the
$\Delta_{\text{naive}}$ discussed earlier.

The following Proposition shows that $\Delta_{\pi}$ in (12) satisfies two
important properties. First, $\Delta_{\pi}(y^{-},y^{+};x)$ can be measured
relative to the optimal response $y^{*}$, which allows to represent
$\Delta_{\pi}(y^{-},y^{+};x)$ in terms of $\Delta_{\pi}(y^{-},y^{*};x)$ and
$\Delta_{\pi}(y^{+},y^{*};x)$. Second, the positive response (which has a
higher true reward value) is “closer” to $y^{*}$ than the negative response,
and vice versa. All the proofs are in Appendix A.

######  Proposition 3.3.

With Assumptions 3.1 and 3.2, we have (i)
$\Delta_{\pi}(y^{-},y^{+};x)=\Delta_{\pi}(y^{-},y^{*};x)-\Delta_{\pi}(y^{+},y^{*};x)$,
where $y^{*}$ is the optimal $y$ for the given $x$; (ii) For any tuple
$(x,y^{+},y^{-})$,
$r^{*}(y^{+}|x)>r^{*}(y^{-}|x)\Leftrightarrow\Delta_{\pi}(y^{+},y^{*};x)<\Delta_{\pi}(y^{-},y^{*};x)$.

The following Corollary shows that this $\Delta_{\pi}(y^{-},y^{+};x)$
satisfies the desired property in Section 3.2.1, namely that a larger
difference $r^{*}(y^{+}|x)-r^{*}(y^{-}|x)$ in the true reward values between
the positive and negative responses corresponds to a larger
$\Delta_{\pi}(y^{-},y^{+};x)$.

######  Corollary 3.3.1.

For any $(x,y_{i}^{+},y_{i}^{-})$ and $(x,y_{j}^{+},y_{j}^{-})$, if
$r^{*}(y_{i}^{+}|x)>r^{*}(y_{j}^{+}|x)$, and
$r^{*}(y_{i}^{-}|x)<r^{*}(y_{j}^{-}|x)$ we have
$r^{*}(y_{i}^{+}|x)-r^{*}(y_{i}^{-}|x)>r^{*}(y_{j}^{+}|x)-r^{*}(y_{j}^{-}|x)\Leftrightarrow\Delta_{\pi}(y_{i}^{-},y_{i}^{+};x)>\Delta_{\pi}(y_{j}^{-},y_{j}^{+};x)$.

###  3.3 Integration with DPO and IPO

####  3.3.1 Integration with DPO

First, we show how to integrate the proposed refinement into DPO. Substituting
(4) and (12) into (9), we obtain:

$\displaystyle\max_{\pi}\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log\sigma(r(y^{+}|x)-r(y^{-}|x)-\lambda\Delta_{\pi}(y^{-},y^{+};x))$
(13) $\displaystyle=$
$\displaystyle\\!\\!\\!\\!\\!\\!\max_{\pi}\mathbb{E}_{(x,y^{+},y^{-})\sim\mathcal{D}}\log\sigma\left(\beta\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}\\!-\\!\beta\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}-\lambda\Delta_{\pi}(y^{-},y^{+};x)\right).$

Recall that $\Delta_{\pi}$ above depends on $\pi$. During learning, we use the
stop-gradient operator111In other words, $\nabla_{x}\perp\\!\\![g(x)]\equiv 0$
and $\perp\\!\\![g(x)]\equiv g(x)$ for any differentiable $g$.
$\perp\\!\\![\cdot]$ on $\Delta_{\pi}$to prevent it from being changed. The
whole procedure, which will be called Self-refined DPO (Sr-DPO), is shown in
Algorithm 1.

Input: Dataset $D=\\{(x,y^{+},y^{-})\\}$, a pre-trained LLM $\pi_{\text{ref}}$
with parameter $\bm{\theta}_{0}$, learning rate $\alpha$.

1 for  _$t=1,2,\dots,T$_ do

2 sample a minibatch $B$ of size $N$ from $D$;

3
$\bm{\theta}_{t}\\!\\!=\\!\\!\bm{\theta}_{t-1}+\alpha\nabla_{\bm{\theta}}\frac{1}{N}\sum^{N}_{i=1}\\!\log\sigma\left(\beta\left(\log\frac{\pi(y_{i}^{+}|x_{i})}{\pi_{\text{ref}}(y_{i}^{+}|x_{i})}\\!-\\!\log\frac{\pi(y_{i}^{-}|x_{i})}{\pi_{\text{ref}}(y_{i}^{-}|x_{i})}\right)\\!-\\!\lambda\\!\perp\\![\Delta_{\pi}(y_{i}^{-},y_{i}^{+};x_{i})]\right)$;

return $\bm{\theta}_{T}$.

Algorithm 1 Self-refined Direct Policy Optimization (Sr-DPO).

####  3.3.2 Integration with IPO

For IPO, we first construct a variant of the IPO objective in (6):

$\displaystyle\max_{\pi}$
$\displaystyle\\!\\!\\!\\!\\!\\!\\!\mathbb{E}_{x\sim\mathcal{D},y^{+}\sim\pi(\cdot|x),y^{-}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ
y^{-}|x)\\!\\!+\\!\\!\lambda r(y^{+}|p\oplus
x)\\!\\!-\\!\\!\beta\mathbb{D}_{\mathrm{KL}}[\pi(y^{+}|x)\|\pi_{\text{ref}}(y^{-}|x)],$
(14)

which adds an extra
$\lambda\mathbb{E}_{x\sim\mathcal{D},y^{+}\sim\pi(\cdot|x)}r(y^{+}|p\oplus x)$
term to maximize the expectation of reward $r(y^{+}|p\oplus x)$.

######  Proposition 3.4.

The optimal policy $\pi$ in (14) satisfies

$\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}=\frac{1}{2\beta}+\lambda\Delta_{\pi}(y^{-},y^{+};x).$
(15)

Note that when $\beta=0$,
$\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}-\lambda\Delta_{\pi}(y^{-},y^{+};x)$
should be infinite. This implies maximizing
$\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}-\lambda\Delta_{\pi}(y^{-},y^{+};x)$
w.r.t, $\pi$, which has a similar form to DPO. The procedure, which will be
called Self-Refined IPO (Sr-IPO), is shown in Algorithm 2. It uses stochastic
gradient descent to minimize the difference between the LHS and RHS in (15).

Input: Dataset $D=\\{(x,y^{+},y^{-})\\}$, a pre-trained LLM $\pi_{\text{ref}}$
with parameter $\bm{\theta}_{0}$, learning rate $\alpha$.

1 for  _$t=1,2,\dots,T$_ do

2 sample a minibatch $B$ of size $N$ from $D$;

3
$\bm{\theta}_{t}=\bm{\theta}_{t-1}\\!\\!-\alpha\nabla_{\bm{\theta}}\frac{1}{N}\sum^{N}_{i=1}\\!\\!\left[\left(\log\frac{\pi(y_{i}^{+}|x_{i})}{\pi_{\text{ref}}(y_{i}^{+}|x_{i})}\\!-\\!\log\frac{\pi(y_{i}^{-}|x_{i})}{\pi_{\text{ref}}(y_{i}^{-}|x_{i})}\right)\\!-\\!\lambda\perp\\![\Delta_{\pi}(y_{i}^{-},y_{i}^{+};x_{i})]-\frac{1}{2\beta}\right]^{2}$;

return $\bm{\theta}_{T}$.

Algorithm 2 Self-refined Identity Policy Optimization (Sr-IPO).

##  4 Experiments

###  4.1 Setup

Datasets. We evaluate the effectiveness of the proposed methods on three
widely-used benchmark datasets: (i) MT-Bench [52], which is a multi-turn
question set on writing, roleplay, extraction, reasoning, math, coding,
knowledge I (STEM), and knowledge II (humanities/social science). (ii) Vicuna-
Bench [12], which is a single-turn question set on writing, roleplay, generic,
fermi, counterfactual, coding, math, and knowledge. (iii) Open-LLM leader-
board [6], which includes (a) commonsense reasoning: Arc [14], HellaSwag [48],
WinoGrande [36]; (b) multi-task language understanding: MMLU [20]; (c) human
falsehood mimic: TruthfulQA [28]; and (d) math problem solving: GSM8k [15].
Table 1 shows more information on the Hugging-Face Open LLM Leaderboard
datasets.

Table 1: Information on the Hugging-Face Open LLM Leaderboard datasets.  | Arc | TruthfulQA | Winogrande | GSM8k | HellaSwag | MMLU
---|---|---|---|---|---|---
# few-shot | 25 | 0 | 5 | 5 | 10 | 5
performance metric | acc_norm | mc2 | acc | acc | acc_norm | acc

Setup for MT-Bench and Vicuna-Bench. As in [35], we use Pythia 2.8B [7], a
pretrained LLM without supervised fine-tuning and RLHF, as the backbone model.
Following [3], we first conduct supervised fine tuning (SFT) using the HH-RLHF
dataset [5], which is human preference data on helpfulness and harmlessness
based on positive feedbacks. We then perform direct alignment using the HH-
RLHF dataset on the SFT model. Finally, we follow [35] to use GPT-4 [1] as a
judge to evaluate the testing performance of the direct alignment trained
model. For performance evaluation, we use the win-rate, tie-rate, and lose
rate as in [44].

Setup for Open-LLM Leader-board. Following [43; 10], we use zephyr-7b-sft-
full222https://huggingface.co/alignment-handbook/zephyr-7b-sft-full, a
supervised fine-tuned version of Mistral 7B [24], as the basic model. We
directly perform direct alignment using a large-scale diverse preference
dataset Ultra-feedback [16]. The fine-tuned model is then evaluated on the
testing benchmarks via the platform [19].

Following [44], we use three performance metrics:

i) Average marginal:
$\frac{1}{N}\sum^{N}_{i=1}\log\frac{\pi\left(y_{i}^{+}\mid
x_{i}\right)}{\pi_{\mathrm{ref}}\left(y_{i}^{+}\mid
x_{i}\right)}-\log\frac{\pi\left(y_{i}^{-}\mid
x_{i}\right)}{\pi_{\mathrm{ref}}\left(y_{i}^{-}\mid x_{i}\right)}$, which
measures the gap between positive and negative responses.

(ii) Accuracy:
$\frac{1}{N}\sum^{N}_{i=1}\mathbbm{1}\left(\log\frac{\pi(y_{i}^{+}\mid
x_{i})}{\pi_{\mathrm{ref}}(y_{i}^{+}\mid x_{i})}>\log\frac{\pi(y_{i}^{-}\mid
x_{i})}{\pi_{\mathrm{ref}}(y_{i}^{-}\mid x_{i})}\right)$, which measures the
number of tuples with the reward of a positive response larger than that of
the negative response. Here, $\mathbbm{1}(\cdot)$ returns 1 when the argument
holds and 0 otherwise.

(iii) Accuracy defined on the prompt-augmented tuples:
$\frac{1}{N}\sum^{N}_{i=1}\mathbbm{1}\left(\log\frac{\pi(y_{i}^{+}\mid p\oplus
x_{i})}{\pi_{\mathrm{ref}}(y_{i}^{+}\mid p\oplus x_{i})}>\right.$
$\left.\log\frac{\pi(y_{i}^{-}\mid p\oplus
x_{i})}{\pi_{\mathrm{ref}}(y_{i}^{-}\mid p\oplus x_{i})}\right)$, which
measures the accuracy for tuples with input augmented by the prompt.

Baselines and Implmentation Details. We choose two widely-adopted direct
alignment baselines: (i) DPO [35], and (ii) IPO [4]. Following [35], $\beta$
is set to 0.1 for DPO and IPO. We also set $\beta=0.1$ for the proposed Sr-DPO
and Sr-IPO for fair comparison. We select the optimal $\lambda$ from
$\\{0.1,0.3,0.5,1\\}$ on the first 50 tuples from the HH-RLHF testing dataset.
For both the HH-RLHF and UltraChat datasets, following [35], the learning rate
is $5\times 10^{-7}$, the optimizer is RMSprop [21], the batch size is 64, and
we also use gradient normalization to help training. The maximal input token
size in training is 512. All experiments are performed on 8 A100 GPUs, and we
use fully sharded data parallel [51] for distributed training.

###  4.2 Performance Results

MT-Bench and Vicuna-Bench. Table 2 shows the testing win/tie/lose rates of the
various methods as evaluated by GPT-4. As can be seen, the proposed Sr-DPO and
Sr-IPO are effective and outperform DPO and IPO (in other words, the win rate
is larger than the lose rate) in both MT-bench and Vicuna-bench. This reveals
the effectiveness of our proposal methods. Some examples are also shown in
Appendix B.

Table 2: Testing performance on MT-Bench and Vicuna-Bench. | MT-Bench | Vicuna-Bench
---|---|---
| Win Rate | Tie Rate | Lose Rate | Win Rate | Tie Rate | Lose Rate
Sr-DPO vs DPO | 45.62% | 33.76% | 20.62% | 63.75% | 13.75% | 22.50%
Sr-IPO vs IPO | 38.75% | 25.00% | 26.25% | 60.00% | 8.75% | 31.25%
Sr-DPO vs IPO | 42.50% | 21.25% | 36.25% | 66.25% | 11.50% | 26.25%
Sr-IPO vs DPO | 38.61% | 28.89% | 32.50% | 61.25% | 5.00% | 33.75%

HH-RLHF. Figure 1 shows results on the training set. As seen, the proposed Sr-
DPO and Sr-IPO have lower marginals (Figures 1(a) and 1(d)), while maintaining
comparable accuracies (Figures 1(b) and 1(e)). This shows they avoid always
enlarging the reward difference between positive and negative pairs, yet still
achieving high accuracies. From Figures 1(c) and 1(f), we find that accuracies
for the prompt-augmented tuples are similar to those for the original tuples,
thus verifying Assumption 3.2.

(a) Marginal: DPO vs Sr-DPO.

(b) Accuracy: IPO vs Sr-IPO.

(c) Sr-DPO’s accuracies for augmented and original tuples.

(d) Marginal: IPO vs Sr-IPO.

(e) Accuracy: IPO vs Sr-IPO.

(f) Sr-IPO’s accuracies for augmented and original tuples.

Figure 1: Average marginal and accuracy on HH-RLHF dataset with different numbers of training tuples. Table 3: Testing performance of the various methods on Open-LLM leader-board. The best one is in bold. | Arc | TruthfulQA | WinoGrande | GSM8k | HellaSwag | MMLU | Average
---|---|---|---|---|---|---|---
Zephyr-7b-sft-full | 60.41 | 43.73 | 74.19 | 26.76 | 82.85 | 60.92 | 58.14
DPO | 63.05 | 59.30 | 78.53 | 31.01 | 84.89 | 61.44 | 63.04
IPO | 61.86 | 46.03 | 79.24 | 35.91 | 82.80 | 61.55 | 61.23
Sr-DPO | 65.27 | 60.30 | 80.82 | 33.36 | 85.52 | 61.63 | 64.48
Sr-IPO | 62.46 | 47.87 | 79.32 | 39.80 | 83.15 | 61.80 | 62.40

Open-LLM leader-board. Table 3 shows the testing performance across various
methods. The results indicate that Sr-DPO achieves superior performance on
Arc, TruthfulQA, WinoGrande; while Sr-IPO excels on GSM8k and MMLU. Overall,
Sr-DPO emerges as the most effective method. Furthermore, Sr-DPO (resp. Sr-
IPO) consistently outperforms DPO (resp. IPO) across all six datasets. These
results validate the effectiveness of the proposed approach.

###  4.3 Effects of $\lambda$ and Number of Training Tuples

Effect of $\lambda$. Figure 2(a) shows the testing win rate with varying
$\lambda$ in Sr-DPO (resp. Sr-IPO) on MT-Bench and Vicuna-Bench. In both cases
the win rate first increases with $\lambda$ and then decreases. In particular,
$\lambda=0$ (i.e., not using the proposed refinement) leads to the worst
performance. However, a $\lambda$ too large exaggerates the influence of
$\Delta$, which can also negatively impact performance. We do not study its
effect on Open-LLM leader-board because it is very time-consuming.

Effect of the number of training tuples. Figure 2(b) shows the testing win
rate with varying number of training tuples on Vicuna-Bench. As can be seen,
both Sr-DPO and Sr-IPO can benefit from the use of more training tuples.

(a) Variation with $\lambda$.

(b) Variation with #tuples.

Figure 2: Win rates of Sr-DPO (vs DPO) and Sr-IPO (vs IPO) w.r.t. $\lambda$
and number of training tuples on Vicuna-Bench.

Figure 3 shows the testing performance with varying number of training tuples
on Open-LLM leader-board. As can be seen, DPO and IPO exhibit performance drop
with the use of more training tuples on 4 of the 6 tasks (except on MMLU and
HellaSwag), suggesting potential overfitting. In contrast, Sr-DPO and Sr-IPO
are less prone to overfitting and exhibit improved performance with increased
training data on ARC, TruthfulQA, GSM8k, and MMLU.

(a) ARC.

(b) TruthfulQA.

(c) WinoGrande.

(d) GSM8k.

(e) HellaSwag.

(f) MMLU.

Figure 3: Performance on Open-LLM leader-board with different numbers of
training tuples.

###  4.4 Correlation between Marginal and Score

Recall that GPT-4 is used as the evaluator. For each instance $x_{i}$, we
define the score difference between the corresponding positive $y_{i}^{+}$ and
negative $y_{i}^{-}$ responses as $s^{+}(x_{i})-s^{-}(x_{i})$, where
$s^{+}(x_{i})$ (resp. $s^{-}(x_{i})$) is GPT-4’s evaluation score (0-5) of
$y_{i}^{+}$ (resp. $y_{i}^{-}$). Inspired by [25], which uses correlation to
measure the agreements between GPT-4 and testing LLMs, here we compute the
correlation between the marginal $\log\frac{\pi\left(y_{i}^{+}\mid
x_{i}\right)}{\pi_{\mathrm{ref}}\left(y_{i}^{+}\mid
x_{i}\right)}-\log\frac{\pi\left(y_{i}^{-}\mid
x_{i}\right)}{\pi_{\mathrm{ref}}\left(y_{i}^{-}\mid x_{i}\right)}$ and score
difference $s^{+}(x_{i})-s^{-}(x_{i})$ over 200 random tuples from the
Ultrafeedback-binarized-preferences
dataset333https://huggingface.co/datasets/argilla/ultrafeedback-binarized-
preferences (which already contains $s^{+}(x)$’s and $s^{-}(x)$’s). In
particular, we use (i) Pearson’s correlation [37], which measures linear
relationships; (ii) Spearman’s rank correlation [38], which measures monotonic
relationships; and (iii) Kendall’s Tau [32], which evaluates the strength and
direction of associations.

Table 4 shows the correlation. As can be seen, Sr-DPO (resp. Sr-IPO) exhibits
higher correlation values compared to DPO (resp. IPO). This suggests that Sr-
DPO and Sr-IPO can evaluate the qualities of the positive and negative
responses more accurately.

###  4.5 Training Time

Table 5 shows the direct alignment training time on the HH-RLHF dataset. As
can be seen, Sr-DPO and Sr-IPO are about $24\%$ slower than DPO and IPO. We
consider this acceptable given the performance improvements achieved by Sr-DPO
and Sr-IPO.

Table 4: Correlation between the marginal and GPT-4’s rating.

| Pearson’s | Spearman’s | Kendall’s
---|---|---|---
| correlation | rank correlation | Tau
DPO | 0.21 | 0.14 | 0.10
IPO | 0.11 | 0.05 | 0.03
Sr-DPO | 0.26 | 0.20 | 0.14
Sr-IPO | 0.13 | 0.08 | 0.05

Table 5: Training time (in GPU hours) on HH-RLHF.

DPO | IPO | Sr-DPO | Sr-IPO
---|---|---|---
16.8 | 16.8 | 20.8 | 20.8

##  5 Conclusion

In this paper, we find that the widely adopted DPO method falls short by not
accounting for the relative qualities of positive and negative samples, which
can lead to sub-optimal training outcomes. To address this issue, we propose
leveraging the intrinsic knowledge within LLMs to refine the loss function.
Our main contributions are three-fold: 1) We utilize the knowledge of LLMs to
create a refinement function that effectively estimates the quality of both
positive and negative responses. 2) We demonstrate that under mild
assumptions, the refinement function can enable the loss function to self-
refine, leading to better alignment with human preferences. 3) Based on the
refinement function, we develop two practical algorithms that enhance the
training process. Experimental results from various evaluators indicate that
the proposed self-refined methods significantly improve the performance of
fine-tuned models compared to existing approaches.

One limitation of our work is that we do not combine it with online policy-
based direct alignment [41]. We will investigate this in our future work.

##  6 Broader Impacts

Our work falls under one of the RLHF frameworks, and thus shares similar
social impacts, both positive and negative. On the positive side, our method
can help LLMs avoid generating harmful or unhelpful results by aligning with
human feedback. However, conversely, our method also has the potential to
enable LLMs to generate harmful or unhelpful results if it is used to align
with malicious preferences.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  Preprint arXiv:2303.08774, 2023.
  * [2] Reda Alami, Abdalgader Abubaker, Mastane Achab, Mohamed El Amine Seddik, and Salem Lahlou.  Investigating regularization of self-play language models.  Preprint arXiv:2404.04291, 2024.
  * [3] Afra Amini, Tim Vieira, and Ryan Cotterell.  Direct preference optimization with an offset.  Preprint arXiv:2402.10571, 2024.
  * [4] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.  A general theoretical paradigm to understand learning from human preferences.  In AISTATS, 2024.
  * [5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  Preprint arXiv:2204.05862, 2022.
  * [6] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open LLM leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * [7] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In ICML, 2023.
  * [8] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. The method of paired comparisons.  Biometrika, 39(3/4):324–345, 1952.
  * [9] Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al.  Human alignment of large language models through online preference optimisation.  Preprint arXiv:2403.08635, 2024.
  * [10] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  Preprint arXiv:2401.01335, 2024.
  * [11] Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du.  Self-playing adversarial language game enhances llm reasoning.  Preprint arXiv:2404.10642, 2024.
  * [12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023.
  * [13] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  NeurIPS, 2017.
  * [14] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try ARC, the AI2 reasoning challenge.  Preprint arXiv:1803.05457, 2018.
  * [15] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  Preprint arXiv:2110.14168, 2021.
  * [16] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback.  Preprint arXiv:2310.01377, 2023.
  * [17] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang.  Raft: Reward ranked finetuning for generative foundation model alignment.  Preprint arXiv:2304.06767, 2023.
  * [18] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  Kto: Model alignment as prospect theoretic optimization.  Preprint arXiv:2402.01306, 2024.
  * [19] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 2023.
  * [20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  ICLR, 2021.
  * [21] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.  Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.  CSC321, 2012.
  * [22] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang.  Aligner: Achieving efficient alignment through weak-to-strong correction.  Preprint arXiv:2402.02416, 2024.
  * [23] Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al.  AI alignment: A comprehensive survey.  Preprint arXiv:2310.19852, 2023.
  * [24] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  Preprint arXiv:2310.06825, 2023.
  * [25] Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo.  Prometheus 2: An open source language model specialized in evaluating other language models.  Preprint arXiv:2405.01535, 2024.
  * [26] Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla.  Illustrating reinforcement learning from human feedback (RLHF).  Hugging Face Blog, 2022.
  * [27] Sangkyu Lee, Sungdong Kim, Ashkan Yousefpour, Minjoon Seo, Kang Min Yoo, and Youngjae Yu.  Aligning large language models by on-policy self-judgment.  Preprint arXiv:2402.11253, 2024.
  * [28] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  Preprint arXiv:2109.07958, 2021.
  * [29] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.  Self-refine: Iterative refinement with self-feedback.  NeurIPS, 2024.
  * [30] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al.  Nash learning from human feedback.  Preprint arXiv:2312.00886, 2023.
  * [31] Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanović, and Adish Singla.  Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences.  Preprint arXiv:2403.01857, 2024.
  * [32] Gottfried E Noether.  Why kendall tau?  Teaching Statistics, 1981.
  * [33] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  NeurIPS, 2022.
  * [34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [35] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  NeurIPS, 2023.
  * [36] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  CACM, 2021.
  * [37] Philip Sedgwick.  Pearson’s correlation coefficient.  BMJ, 2012.
  * [38] Philip Sedgwick.  Spearman’s rank correlation coefficient.  BMJ, 2014.
  * [39] Feifan Song, Yuxuan Fan, Xin Zhang, Peiyi Wang, and Houfeng Wang.  Icdpo: Effectively borrowing alignment capability of others via in-context direct preference optimization.  Preprint arXiv:2402.09320, 2024.
  * [40] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  NeurIPS, 2024.
  * [41] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al.  Understanding the performance gap between online and offline alignment algorithms.  Preprint arXiv:2405.08448, 2024.
  * [42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  Preprint arXiv:2302.13971, 2023.
  * [43] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  Preprint arXiv:2310.16944, 2023.
  * [44] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen.  Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints.  In ICLR, 2024.
  * [45] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz.  A survey of preference-based reinforcement learning methods.  JMLR, 2017.
  * [46] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  Preprint arXiv:2401.10020, 2024.
  * [47] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  Rrhf: Rank responses to align language models with human feedback without tears.  Preprint arXiv:2304.05302, 2023.
  * [48] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  Preprint arXiv:1905.07830, 2019.
  * [49] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  Preprint arXiv:2205.01068, 2022.
  * [50] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  Preprint arXiv:2303.18223, 2023.
  * [51] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al.  Pytorch fsdp: experiences on scaling fully sharded data parallel.  Preprint arXiv:2304.11277, 2023.
  * [52] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging LLM-as-a-judge with mt-bench and chatbot arena.  NeurIPS, 2023.
  * [53] Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao.  Beyond one-preference-for-all: Multi-objective direct preference optimization.  Preprint arXiv:2310.03708, 2023.

##  Appendix A Proofs

###  A.1 Proof of Proposition 3.3

###### Proof.

(i) First property: By directly substituting
$\Delta_{\pi}(y^{-},y^{*};x)-\Delta_{\pi}(y^{+},y^{*};x)$ into the definition
of $\Delta$ in (12), we have

$\displaystyle\Delta_{\pi}(y^{-},y^{*};x)-\Delta_{\pi}(y^{+},y^{*};x)$
$\displaystyle=$ $\displaystyle\beta\left(\log\frac{\pi(y^{+}|p\oplus
x)}{\pi_{0}(y^{+}|p\oplus x)}-\log\frac{\pi(y^{-}|p\oplus
x)}{\pi_{0}(y^{-}|p\oplus x)}\right)$ $\displaystyle=$
$\displaystyle\beta\log\frac{\pi(y^{+}|p\oplus x)\pi_{0}(y^{-}|p\oplus
x)}{\pi(y^{-}|p\oplus x)\pi_{0}(y^{+}|p\oplus x)}$ $\displaystyle=$
$\displaystyle\Delta_{\pi}\left(y^{-},y^{+};x\right).$

(ii) Second property: From assumption 3.1, we have:
$r\left(y^{+}|x\right)>r\left(y^{-}|x\right),\forall(x,y^{+},y-)$ with
$y^{+}\succ y^{-}$. From (4),

$r(y^{+}|x)>r\left(y^{-}|x\right)\Leftrightarrow\log\frac{\pi\left(y^{+}\mid
x\right)}{\pi_{0}\left(y^{+}\mid x\right)}>\log\frac{\pi\left(y^{-}\mid
x\right)}{\pi_{0}\left(y^{-}\mid x\right)}.$

From Assumption 3.2, for the prompt-augmented query $p\oplus x$, we still have
$y^{+}\succ y^{-}$. By using Assumption 3.1 on $(p\oplus x,y^{+},y^{-})$, we
have:

$\displaystyle\log\frac{\pi(y^{+}|p\oplus x)}{\pi_{0}(y^{+}|p\oplus
x)}>\log\frac{\pi(y^{-}|p\oplus x)}{\pi_{0}(y^{-}|p\oplus x)}.$ (16)

Thus, from the first property,

$\Delta_{\pi}(y^{+},y^{*};x)-\Delta_{\pi}(y^{-},y^{*};x)=-\Delta_{\pi}\left(y^{-},y^{+};x\right)=-\beta\log\frac{\pi(y^{+}|p\oplus
x)\pi_{0}(y^{-}|p\oplus x)}{\pi_{0}(y^{+}|p\oplus x)\pi(y^{-}|p\oplus x)}<0$

because of (16), and so
$\Delta_{\pi}(y^{+},y^{*};x)<\Delta_{\pi}(y^{-},y^{*};x)$. Thus,

$y^{+}\succ
y^{-}\Leftrightarrow\Delta_{\pi}\left(y^{+},y^{*};x\right)<\Delta_{\pi}\left(y^{-},y^{*};x\right).$
(17)

Since $y^{+}\succ y^{-}\Leftrightarrow r^{*}(y^{+}|x)>r^{*}(y^{-}|x)$. We
obtain that
$r^{*}(y^{+}|x)>r^{*}(y^{-}|x)\Leftrightarrow\Delta_{\pi}\left(y^{+},y^{*};x\right)<\Delta_{\pi}\left(y^{-},y^{*};x\right)$.
∎

###  A.2 Proof of Corollary 3.3.1

###### Proof.

Using property (ii) in Proposition 3.3 on the assumption
$r^{*}(y_{i}^{+}|x)>r^{*}(y_{j}^{+}|x)$ in Corollary 3.3.1, we have

$r^{*}(y_{i}^{+}|x)>r^{*}(y_{j}^{+}|x)\Leftrightarrow-\Delta_{\pi}(y_{i}^{+},y^{*};x)>-\Delta_{\pi}(y_{j}^{+},y^{*};x).$
(18)

Similarly, using this on the assumption in Corollary 3.3.1 that
$r^{*}(y_{i}^{-}|x)<r^{*}(y_{j}^{-}|x)$, we have

$-r^{*}(y_{i}^{-}|x)>-r^{*}(y_{j}^{-}|x)\Leftrightarrow\Delta_{\pi}(y_{i}^{-},y^{*};x)>\Delta_{\pi}(y_{j}^{-},y^{*};x).$
(19)

Adding these two together, we obtain

$\displaystyle
r^{*}(y_{i}^{+}|x)-r^{*}(y_{i}^{-}|x)<r^{*}(y_{j}^{+}|x)-r^{*}(y_{j}^{-}|x)$
$\displaystyle\Leftrightarrow$
$\displaystyle\Delta_{\pi}(y_{i}^{-},y^{*};x)-\Delta_{\pi}(y_{i}^{+},y^{*};x)>\Delta_{\pi}(y_{j}^{-},y^{*};x)-\Delta_{\pi}(y_{j}^{+},y^{*};x)$
$\displaystyle\Leftrightarrow$
$\displaystyle\Delta_{\pi}(y_{i}^{-},y_{i}^{+};x)>\Delta_{\pi}(y_{j}^{-},y_{j}^{+};x),$

due to property (i) of Proposition 3.3. ∎

###  A.3 Proof of Proposition 3.4

As in [3, 4], the closed-form solution of (14) is:

$\pi(y^{+}|x)\propto\pi_{\text{ref}}(y^{+}|x)\exp\left(\beta^{-1}\mathbb{E}_{y^{+}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ\pi_{\text{ref}}|x)+\lambda
r(y^{+}|p\oplus x)\right).$

Similarly for $y^{-}$, we have

$\pi(y^{-}|x)\propto\pi_{\text{ref}}(y^{-}|x)\exp\left(\beta^{-1}\mathbb{E}_{y^{-}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{-}\succ\pi_{\text{ref}}|x)+\lambda
r(y^{-}|p\oplus x)\right).$

After simplification, we have:

$\displaystyle\frac{\pi(y^{+}|x)}{\pi(y^{-}|x)}$ $\displaystyle=$
$\displaystyle\frac{\pi_{\text{ref}}(y^{+}|x)}{\pi_{\text{ref}}(y^{-}|x)}\exp\left(\beta^{-1}[\mathbb{E}_{y^{+}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ\pi_{\text{ref}}|x)-\mathbb{E}_{y^{-}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{-}\succ\pi_{\text{ref}}|x)]\right.$
$\displaystyle\left.+\lambda[r(y^{+}|p\oplus x)-r(y^{-}|p\oplus x)]\right).$

Note that $r(y^{+}|p\oplus x)-r(y^{-}|p\oplus x)=\Delta_{\pi}(y^{-},y^{+};x)$.
After some simplifications, we have

$\displaystyle\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}$
$\displaystyle=$
$\displaystyle\frac{1}{\beta}\left[\mathbb{E}_{y^{+}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ\pi_{\text{ref}}|x)-\mathbb{E}_{y^{-}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{-}\succ\pi_{\text{ref}}|x)\right]+\lambda\Delta_{\pi}(y^{-},y^{+};x).$

As
$\mathbb{E}_{y^{+}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{+}\succ\pi_{\text{ref}}|x)-\mathbb{E}_{y^{-}\sim\pi_{\text{ref}}(\cdot|x)}p(y^{-}\succ\pi_{\text{ref}}|x)=\frac{1}{2}$
[4], after taking expectations of both sides, we obtain

$\displaystyle\log\frac{\pi(y^{+}|x)}{\pi_{\text{ref}}(y^{+}|x)}-\log\frac{\pi(y^{-}|x)}{\pi_{\text{ref}}(y^{-}|x)}=\frac{1}{2\beta}+\lambda\Delta_{\pi}(y^{-},y^{+};x).$
(20)

##  Appendix B Example Cases

Tables 6, 7, 8, and 9 show four examples of responses by Sr-DPO, Sr-IPO, DPO,
and IPO. As can be seen, the proposed Sr-DPO and Sr-IPO outperform DPO and IPO
under GPT4’s judgement.

Table 6: Example 1 for Sr-DPO and DPO. The first row is the query, the second and third rows are the responses generated by Sr-DPO and DPO, respectively, and the last row is the judgement made by GPT-4. Prompt |  Explain the process of natural selection and how it contributes to the evolution and adaptation of species.
---|---
Sr-DPO |  Natural selection is the process of survival of the fittest, which occurs when organisms with the most advantageous traits are more likely to reproduce and pass on their traits than organisms with less desirable traits. It is a key factor in the evolution of species, with the most successful species being those that are the most adaptable to the environment. Natural selection ensures that organisms with the most advantageous traits are more likely to survive and pass on their traits to the next generation. Here is a brief explanation of the process of natural selection: 1\. The most adaptable organisms with the most advantageous traits are more likely to survive and pass on their traits to the next generation. 2\. The most advantageous traits of organisms are those that are most effective at surviving and reproducing in their environment. 3\. Those organisms with the most advantageous traits are more likely to survive and reproduce than organisms with less advantageous traits.
DPO |  Natural selection is the process of survival of the fittest, which occurs when the genetic makeup of a population changes over time, favoring individuals that have beneficial traits such as increased physical strength, increased intelligence, or other adaptive features. This process allows for the survival of the species that is best able to adapt to its environment.
Judgment |  Both assistants provide a correct explanation of natural selection, describing it as a process that favors individuals with beneficial traits, leading to changes in the genetic makeup of a population over time. However, Assistant A’s response is more detailed and comprehensive. It explains how the probability of an individual’s survival and reproduction is influenced by how well adapted the organism is to its environment, and it also mentions the importance of natural selection in the field of biology and evolution. Therefore, Assistant A’s response is more informative and provides a deeper understanding of the topic. Final Verdict: [[A]]
Table 7: Example 2 for Sr-DPO and DPO. The first row is the query, the second and third rows are the responses generated by Sr-DPO and DPO, respectively, and the last row is the judgement made by GPT-4.  Prompt |  Draft an apology email to a customer who experienced a delay in their order, and provide reassurance that the issue has been resolved.
---|---
Sr-DPO |  Here’s an apology email draft for a customer who experienced a delay in their order: Dear Customer, Thank you for your patience during our recent order process. We recently experienced a delay with your order and apologize for the inconvenience. We are currently working to process your order, and we will provide a notification when it is ready. We sincerely apologize for the delay and appreciate your patience. Our team is working hard to resolve this issue as soon as possible. We look forward to welcoming your order and providing you with the best customer service experience possible. Best regards, Customer Support Team
DPO |  Here’s an apology email draft for a customer who experienced a delay in their order: Dear [Customer Name], apologize for the delay in your delivery of your order. I have worked with our team to resolve the issue, and I am very pleased to let you know that the issue has been resolved and your order is now ready for dispatch. I look forward to providing you with a great customer experience in the future. Sincerely, [Customer Name] Customer Support
Judgment |  Both assistants provided a good response to the user’s request. However, Assistant A’s response was more detailed and provided more reassurance to the customer. Assistant A mentioned that they reviewed the issue with their fulfillment partners and are now shipping the order as quickly as possible. This gives the customer a sense of urgency and care for their order. On the other hand, Assistant B’s response was also good but lacked the level of detail and reassurance that Assistant A’s response provided. Therefore, Assistant A’s response is better. Final Verdict: [[A]]
Table 8: Example 3 for Sr-IPO and IPO. The first row is the query, the second and third rows are the responses generated by Sr-IPO and IPO, respectively, and the last row is the judgement made by GPT-4.  Prompt |  Why might someone choose to use a paper map or ask for directions instead of relying on a GPS device or smartphone app?
---|---
Sr-IPO |  A paper map or paper printed directions can be a fun way to help you get from point A to point B. Paper maps are portable and easy to use, and they’re also a fun and easy way to explore your surroundings and get around town. They’re also less expensive than GPS systems or smartphone apps, and they’re very easy to use, so they’re a good option for older kids or adults who need to get by without a smartphone. Paper maps offer a more personal and hands-on experience than smartphone apps or GPS devices, and they can be a fun way to get around town or explore your surroundings.
IPO |  A paper map or paper printed directions can be a fun way to help you get from point A to point B. Paper maps are portable and easy to use, and they’re also a fun and easy way to explore your surroundings and get around town. They’re also less expensive than GPS systems or smartphone apps, and they’re very easy to carry around. If you are interested in using a paper map or directions, here are a few reasons why you might choose to do so:1. Paper maps are portable and easy to use. 2\. Paper maps are less expensive than GPS systems or smartphone apps. 3\. Paper maps are easy to carry around. 4\. Paper maps are easy to use and interpret. 5\. Paper maps can help you explore your surroundings and get around town. 6\. Paper maps are easily outdated or outdated, so it is important to be sure you have the latest version of the map to use. 7\. Paper maps are a fun way to explore your surroundings and get around town. 8\. Paper maps are easy to use and interpret. 9\. Paper maps can help you explore your surroundings and get around town. 10\. Paper maps make it easy to get around town and explore your surroundings. 11\. Paper maps are a fun and easy way to explore your surroundings and get around town. Remember to tailor the activities to the birthday childs interests and preferences. Have a great celebration!
Judgment |  Assistant A’s response is more concise and to the point, providing a clear explanation of why someone might choose to use a paper map or ask for directions instead of relying on a GPS device or smartphone app. Assistant B’s response, while it does provide some valid points, is repetitive and confusing, especially the last sentence about tailoring activities to a birthday child’s interests and preferences, which is completely irrelevant to the question asked. Therefore, Assistant A’s response is more helpful and relevant. Final Verdict: [[A]]
Table 9: Example 4 for Sr-IPO and IPO. The first row is the query, the second and third rows are the responses generated by Sr-IPO and IPO, respectively, and the last row is the judgement made by GPT-4.  Prompt |  How many lightning strikes occur on Earth each day? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step.
---|---
Sr-IPO |  There are about 2,500 lightning strikes each day on Earth. That’s about two lightning strikes for every minute! The most common type of lightning strikes occur in the tropics, where temperatures range from 70 to 90 degrees Fahrenheit. Lightning strikes are most common around noon and also around sunset. The next most common type of lightning occurs in the polar regions, where the temperatures can reach as low as -20 degrees Celsius. Lightning strikes are most common during the winter months in these regions. The most dangerous type of lightning occurs when the atmospheric pressure is extremely low, like when a storm is approaching. Low atmospheric pressure can cause the air to move faster and more violently, creating a stronger electric current through the air. When the air is moving quickly, it is also possible for an electrical charge to build up between the tips of a cloud or a thunderstorm. This electrical charge can then be released as lightning. Low atmospheric pressure can also increase the intensity of a thunderstorm, leading to more severe lightning storms.
IPO |  There are about 2,500 lightning strikes each day on Earth. That’s about two lightning strikes for every minute! The most common type of lightning strikes occur in the tropics, where temperatures range from 70 to 90 degrees Fahrenheit. Lightning strikes are most common around noon and also around sunset. The next most common type of lightning occurs in the polar regions, where the temperatures can reach as low as -20 degrees Celsius. Lightning strikes are most common during the winter months in these regions.
Judgment |  Both Assistant A and Assistant B start their responses with the same information, stating that there are about 2,500 lightning strikes each day on Earth. They both also provide similar information about where and when lightning strikes are most common. However, Assistant A goes a step further by explaining the conditions under which the most dangerous type of lightning occurs, providing a more detailed and comprehensive response to the user’s question. Therefore, Assistant A’s response is more informative and thorough. Final Verdict: [[A]]