# Aligning Large Language Models
with Self-generated Preference Data

Dongyoung Kim 1, Kimin Lee 1, Jinwoo Shin 1 Jaehyung Kim 2
1Korea Advanced Institute of Science and Technology , 2Carnegie Mellon
University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Aligning large language models (LLMs) with human preferences becomes a key
component to obtaining state-of-the-art performance, but it yields a huge cost
to construct a large human-annotated preference dataset. To tackle this
problem, we propose a new framework that boosts the alignment of LLMs through
Self-generated Preference data (Selfee) using only a very small amount of
human-annotated preference data. Our key idea is leveraging the human prior
knowledge within the small (seed) data and progressively improving the
alignment of LLM, by iteratively generating the responses and learning from
them with the self-annotated preference data. To be specific, we propose to
derive the preference label from the logits of LLM to explicitly extract the
model’s inherent preference. Compared to the previous approaches using
external reward models or implicit in-context learning, we observe that the
proposed approach is significantly more effective. In addition, we introduce a
noise-aware preference learning algorithm to mitigate the risk of low quality
within generated preference data. Our experimental results demonstrate that
the proposed framework significantly boosts the alignment of LLMs. For
example, we achieve superior alignment performance on AlpacaEval 2.0 with only
3.3% of the ground-truth preference labels in the Ultrafeedback data compared
to the cases using the entire data or state-of-the-art baselines.

##  1 Introduction

Recently, large language models (LLMs) have made huge progress in various NLP
tasks, leading to real-world applications that are used by millions of users,
such as coding assistants and chatbots [1, 27, 38]. Aligning LLMs with human
feedback, particularly through learning from human preferences, is widely
considered a crucial technique for their success [5, 18, 51]. To enhance this
alignment, various preference learning algorithms have been extensively
explored [29, 32]. Despite these advancements, one of the remaining challenges
is the reliance on large-scale human-annotated preference data. As the quality
and quantity of preference data are critical for the successful alignment of
LLMs [2, 7], the huge cost to acquire such data inevitably presents
significant obstacles.

To mitigate this challenge, engaging LLMs in constructing preference data and
improving their alignment using this data has recently gained attention. For
example, a representative way on this line is generating multiple responses
for the input prompts, and then approximating human preference between them
through LLM’s predictions, often referred to as LLM-as-judge [3, 48]. However,
these approaches are only effective when the given LLM is sufficiently large
and well-aligned to mimic human preference via in-context learning. On the
other hand, using an external reward model is considerable to substitute human
preference annotation efficiently [16, 35], but it is built on the
availability of large human preference data and also could be ineffective if
the distribution mismatch exists. Lastly, these approaches have a risk of
potential labeling noise from LLMs, but this aspect has not been explored yet.
Therefore, in this work, we aim to develop a method to effectively improve the
alignment of LLM by overcoming these limitations but only relying on the small
human annotation.

Figure 1: Illustration of the proposed Selfee framework. Selfee progressively
improves the alignment of LLMs by iterating (1) the generation of new
preference data and (2) the preference learning on the constructed data with
self-refinement. Technical details are presented in Section 4.

Contribution. We introduce a simple yet effective framework, coined Selfee, to
improve the alignment of LLMs via Self-generated Preference data, with only a
small amount of human-labeled preference data. Our key idea is to
progressively expand the knowledge of human preference within the small (seed)
data, by iteratively generating the responses and learning from them through
the self-annotated preference labels. Specifically, our technical
contributions are three-fold as described in what follows. First, we derive
the preference labels using the logits of LLM to explicitly extract the
model’s inherent preference. This approach is more effective compared to
previous methods that rely on external reward models or implicit in-context
learning. Second, we introduce a confidence-based refinement of preference
labels to reduce the risk of noise in preference learning with generated data.
Third, to further enhance the effectiveness of this refinement, we propose
using a linearly extrapolated prediction between the current and reference
models; this approach approximates the predictions of a more strongly aligned
model, leading to better noise identification.

Figure 2: Summary of main result. Evaluation results on AlpacaEval2.0
benchmark [20]. Our framework significantly improves the alignment of LLMs,
without additional human preference data. See detailed results in Section 5.

We demonstrate the effectiveness of the proposed Selfee by aligning recent
LLMs with small human-annotated preference data and evaluating their alignment
on the commonly used benchmarks. For example, using only 3.3% of ground-truth
preference in Ultrafeedback data [7] with the mistral-7b-0.1v SFT model [15],
our framework achieves over 16.4% increase in AlpacaEval2.0 [20] win rate
compared to the initial SFT model (see Figure 2). Additionally, the
AlpacaEval2.0 length-controlled win rate is improved from 7.58% to 15.39%, and
MT-bench score [50] increased from 6.38 to 6.94. Compared to preference
judgment methods like LLM-as-judge [50], and even strong reward models such as
PairRM [16], which have recently shown state-of-art performance in
AlpacaEval2.0 benchmark, our approach consistently outperforms them across all
metrics. More interestingly, the proposed Selfee successfully improves the
alignment of various LLMs, even without the initial human preference data.
These results demonstrate that our framework is highly competitive and
practical for real-world applications.

##  2 Related Work

Alignment of LLMs with human preference. Learning from human preferences now
serves as a core component for the state-of-the-art LLMs [1, 28, 38, 39] for
aligning their responses with users’ intent and values [29, 51]. Arguably, one
of the most popular frameworks is reinforcement learning with human preference
(RLHF) [5, 18], which first trains the reward model, and then fine-tunes LLM
to maximize that reward with KL divergence regularization to prevent the
reward over-optimization of LLM. On the other hand, various training
algorithms have recently been proposed to fine-tune LLMs with human preference
more efficiently [11, 14, 23, 32, 47, 49]. For example, Rafailov et al. [32]
proposes Direct Preference Optimization (DPO) which allows one to fine-tune
LLMs without a separate reward modeling stage, by deriving the training
objective mathematically equivalent to RLHF. Ethayarajh et al. [11] further
removes the reliance on pair-wise preference labels by formulating the
objective based on a human utility model. However, these methods assume that
large human-annotated preference data is available, which requires a huge data
acquisition cost.

Engagement of LLMs for constructing preference data. For an efficient and
scalable alignment procedure, engaging LLMs for preference dataset
construction has recently received attention. One common approach involves
generating multiple responses to input prompts from LLM, and using an LLM’s
predictions to approximate human preferences between them, a technique often
referred to as LLM-as-judge [2, 48]. However, this method is effective only
when the LLM is sufficiently large and well-aligned to mimic human preferences
through in-context learning. Alternatively, employing an external reward model
can efficiently replace human preference judgment [16, 35], but this approach
relies on the availability of extensive human preference data to pre-train
reward model and may be ineffective if there is a distribution mismatch. Some
concurrent works [34, 35, 45, 46] have proposed the alignment procedure with
iterative data expansion and preference learning. However, they use the
external reward model or stronger LLM for the preference judgment. In
contrast, we only utilize the intrinsic knowledge of training LLM with new
data expansion and preference learning methods.

##  3 Preliminaries

Let denote LLM as $\pi_{\theta}$, which generates an output sequence (e.g.,
response) $y$ for a given input sequence (e.g., prompt) $x$, i.e.,
$y\sim\pi_{\theta}(\cdot|x)$. Then, our goal is to make $\pi_{\theta}$ provide
human-aligned responses for the various input prompts. To this end, we
consider the popular framework of preference learning, which optimizes
$\pi_{\theta}$ to learn the human preferences between two different responses
[5, 18, 29]. Specifically, we assume that the preference dataset
$\mathcal{D}=\\{(x,y_{l},y_{w})\\}$ is available which consists of the
triplets of input prompt $x$, preferred response $y_{w}$, and dispreferred
response $y_{l}$. Here, the preference labels were annotated by a ground truth
annotator, that is usually a human expert.

Reward modeling and RL fine-tuning. Since a pairwise preference between
$y_{w}$ and $y_{l}$ is hard to model directly, one of the common practices is
introducing reward function $r(x,y)$ and modeling the preference based on this
using the Bradley-Terry model [4]:

$p(y_{w}\succ y_{l}\mid
x)=\frac{\exp\left(r(x,y_{w})\right)}{\exp\left(r(x,y_{w})\right)+\exp\left(r(x,y_{l})\right)}.$
(1)

From this formulation, one can introduce a parametrized reward model
$r_{\phi}(x,y)$ by estimating its parameters with the maximum-likelihood
objective:

$\mathcal{L}_{R}(r_{\phi})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\left(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l})\right)\right].$
(2)

where $\sigma$ is a sigmoid function. After this reward modeling procedure,
one could improve the alignment of LLM $\pi_{\theta}$ by optimizing it to
maximize the reward captured by $r_{\phi}$. Here, KL-distance from the
reference model $\pi_{\text{ref}}$ is usually incorporated as a regularization
to prevent the reward over-optimization of $\pi_{\theta}$, with a hyper-
parameter $\beta>0$ [29, 51]:111$\pi_{\text{ref}}$ is usually initialized with
supervised fine-tuned (SFT) LLM [6, 43]. Also, $\pi_{\theta}$ is initialized
with $\pi_{\text{ref}}$.

$\mathcal{L}_{\text{RLHF}}(\pi_{\theta})=-\mathbb{E}_{y\sim\pi_{\theta},x\sim\rho}\left[r_{\phi}(x,y)\right]+\beta\mathrm{D}_{\mathrm{KL}}\left(\pi_{\theta}(y|x)\parallel\pi_{\text{ref}}(y|x)\right).$
(3)

Direct preference modeling and optimization. Rafailov et al. [32] propose an
alternative approach to align LLM $\pi_{\theta}$ with preference dataset
$\mathcal{D}$, which is called Direct Preference Optimization (DPO). DPO
integrates a two-step alignment procedure with reward modeling and RL fine-
tuning into a single unified fine-tuning procedure. Specifically, the optimal
reward function is derived from RLHF objective (Eq. 3), with the target LLM
$\pi_{\theta}$ and reference model $\pi_{\text{ref}}$ [12, 30, 31].

$r(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid
x)}+\beta\log Z(x),~{}\text{where}~{}Z(x)=\sum_{y}\pi_{\text{ref}}(y\mid
x)\exp\left(\frac{1}{\beta}r(x,y)\right).$ (4)

Then, the preference between two responses could be measured using this reward
derivation, and $\pi_{\theta}$ is optimized to maximize this preference of
$y_{w}$ over $y_{l}$ using the preference dataset $\mathcal{D}$.

$p_{\theta}(y_{w}\succ
y_{l}|x)=\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right).$
(5)
$\mathcal{L}_{\text{DPO}}(\pi_{\theta})=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[-\log
p_{\theta}(y_{w}\succ y_{l}|x)\right].$ (6)

##  4 Selfee: Boosting Alignment of LLMs with Self-generated Preference

Overview. In this section, we present Selfee: Self-generated Preference data
to align LLMs while mitigating the huge cost for preference dataset
construction. Our main idea is to fully exploit the human prior knowledge
within the small (seed) data, and progressively update LLM to improve the
alignment. To be specific, Selfee iterates two steps: (1) data expansion with
self-generated preference (Section 4.1) and (2) fine-tuning LLM with self-
refined preference learning (Section 4.2). We present full procedure of Selfee
in Algorithm 1 (see Figure 1 for the overview).

Initial stage. We assume that a small (seed) preference dataset $D_{0}$ and an
initial LLM $\pi_{\text{init}}$ are given. Here, following the common practice
[29, 32, 51], we use $\pi_{\text{init}}$ which has been supervised fine-tuned
(SFT) LLM on the instruction dataset [6, 43], but not aligned with human
preference yet. Then, we first obtain weakly aligned LLM $\pi_{0}$ by fine-
tuning $\pi_{\text{init}}$ on $D_{0}$ using DPO [32] (Eq. 6). We adopt DPO
among various preference learning methods due to its simplicity and
effectiveness.

Algorithm 1 Selfee algorithm

Input: initial LLM $\pi_{\text{init}}$, seed preference dataset
$\mathcal{D}_{0}$, number of improving iterations $T$, new prompt sets
$\\{X_{i}\\}_{i=1}^{T}$,

Obtaining an initial weakly aligned model $\pi_{0}$ using DPO with
$\pi_{\text{init}}\text{ and }\mathcal{D}_{0}$ (Eq. 6)

for $t=1$ to $T$ do

Synthesizing preference data $\mathcal{D}_{t}$ with $\pi_{t-1}\text{ and
}{X}_{t}$ (Eq. 7 and 8)

Initialization of training and reference models
$\pi_{\theta}\leftarrow\pi_{t-1}$, $\pi_{\text{ref}}\leftarrow\pi_{t-1}$

for mini-batch $B\sim\mathcal{D}_{t}$ do

$z_{\widetilde{\theta}}\leftarrow\text{De-coupled noise detection for }B\text{
from }\pi_{\theta},\pi_{\text{ref}},{X}_{t}$ (Eq. 11 and 12)

Calculate training loss $\mathcal{L}_{\text{rf}}$ with refined preference
labels using $z_{\widetilde{\theta}}$ and $\pi_{\theta}$ (Eq. 10)

Update model parameter:
$\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{\text{rf}}$

end for

Initializing next iteration model $\pi_{t}$ with the updated parameters
$\theta$

end for

return $\pi_{T}$

###  4.1 Data expansion with self-generated preference data

For the $i$-th iteration ($i=1,\dots)$, we assume that the new prompt set
$X_{i}=\\{x\\}$ is available, i.e., $X_{i}\cap X_{j}=\emptyset$ for all
$j=0,\dots,i-1$.222$X_{0}=\\{x|(x,y_{l},y_{w})\in\mathcal{D}_{0}$} From
$X_{i}$, we construct $i$-th artificial preference dataset
$\mathcal{D}_{i}=\\{(x,y_{l},y_{w})|x\in X_{i}\\}$, by using LLM’s intrinsic
generation and reward modeling capabilities. Specifically, for each input
prompt $x\in X_{i}$, we sample two responses $y_{1}$ and $y_{2}$ from
$\pi_{i-1}$, i.e., $y_{1},y_{2}\sim\pi_{i-1}(x)$ where $\pi_{i-1}$ is the
resulting model from the previous iteration. Then, using the reward captured
with $\pi_{i-1}$ and $\pi_{\text{init}}$ (Eq. 4), we measure the preference of
$\pi_{i-1}$ between $y_{1}$ and $y_{2}$:

$p_{i-1}(y_{1}\succ
y_{2}|x)=\sigma\left(\beta\log\frac{\pi_{i-1}(y_{1}|x)}{\pi_{\text{init}}(y_{1}|x)}-\beta\log\frac{\pi_{i-1}(y_{2}|x)}{\pi_{\text{init}}(y_{2}|x)}\right).$
(7)

Then, we directly judge the preference label as below and construct
$\mathcal{D}_{i}$ through this:

$(y_{w},y_{l})=(y_{1},y_{2})~{}\text{ if }~{}p_{i-1}(y_{1}\succ
y_{2}|x)>0.5~{}\text{ else }~{}(y_{w},y_{l})=(y_{2},y_{1}).$ (8)

###  4.2 Self-refinement of generated preference data for effective learning

After the construction of $\mathcal{D}_{i}$, we conduct $i$-th preference
learning by fine-tuning $\pi_{\theta}$, which is initialized by $\pi_{i-1}$,
using DPO (here, we also use $\pi_{i-1}$ as $\pi_{\text{ref}}$ in Eq. 6).
Learning the self-generated preference data $\mathcal{D}_{i}$ could improve
the alignment by effectively spreading the human preference prior from
$\mathcal{D}_{0}$ using the power of LLM. However, it also has a risk of the
potential labeling noise which could occur from the distribution shift with
$X_{i}$ or insufficient reward modeling with $\pi_{i-1}$. Therefore, we
further propose an improved preference learning method by introducing a novel
denoising technique: self-refinement of preference labels with de-coupled
noise detection.

Self-refinement of preference label: Our key intuition is that one can view
the derived preference (Eq. 5) can be viewed as the confidence of the
currently training LLM $\pi_{\theta}$ for the labels assigned by $\pi_{i-1}$.
Then, $\pi_{\theta}$ would exhibit lower confidence if the given pair of
responses is uncertain to answer, indicating a higher probability of labeling
noise. Notably, we also remark that confidence is one of the most popular
metrics in the noisy label learning literature [13, 33, 36]. Under this
intuition, we first identify the $K$% least confident samples:

$z_{\theta}=1~{}\text{ if }~{}p_{\theta}(y_{w}\succ y_{l}|x)<\tau~{}\text{
else }~{}z_{\theta}=0,$ (9)

where $\tau$ is the confidence of $K$ percentile sample of $\mathcal{D}_{i}$.
Then, with this (potentially) noise identification label $z_{\theta}$, we
refine the assigned preference label using label smoothing [25], to train
$\pi_{\theta}$ less confidently when the risk of label noise is high (i.e.,
$z_{\theta}=1$):

$\mathcal{L}_{\text{rf}}(\pi_{\theta})=\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{i}}\left[-\big{(}(1-\alpha\ast
z_{\theta})\log p_{\theta}(y_{w}\succ y_{l}|x)+\alpha\ast z_{\theta}\log
p_{\theta}(y_{l}\succ y_{w}|x)\big{)}\right],$ (10)

where $\alpha$ is a hyper-parameter. Then, we train $\pi_{\theta}$ using
$\mathcal{L}_{\text{rf}}(\pi_{\theta})$ instead of naive DPO (Eq. 6).

De-coupled noise preference detection: While learning with the refined
preference label reduces the risk of learning $\pi_{\theta}$ the noisy
preference, its effectiveness could be limited as the model $\pi_{\theta}$ for
noise detection originated from the label generation model $\pi_{i-1}$.
Therefore, to further improve the effectiveness of our denoising framework
with preference label refinement, we introduce the de-coupled noise detection
[13, 19] technique for LLM alignment. Specifically, we identify the preference
noise by mimicking the preference prediction of a more strongly aligned LLM
$\pi_{\widetilde{\theta}}$: 333With $\lambda$ in Eq. 12,
$\pi_{\widetilde{\theta}}$ is equivalent to model trained with (1 + $\lambda$)
times smaller KL term than $\pi_{\theta}$ via Eq. 3.

$z_{\widetilde{\theta}}=1~{}\text{ if }~{}p_{\widetilde{\theta}}(y_{w}\succ
y_{l}|x)<\tau~{}\text{ else }~{}z_{\widetilde{\theta}}=0.$ (11)

With this de-coupled identification, $\pi_{\theta}$ is trained with refined
preference labels via Eq. 10. Here, we obtain the prediction of
$\pi_{\widetilde{\theta}}$ by approximating its logit $h_{\widetilde{\theta}}$
through the linear combination of the logits of $\pi_{\theta}$ and
$\pi_{\text{ref}}$.
444$\pi_{\theta}(\cdot|x,y_{1:t-1})=\text{Softmax}\big{(}h_{\theta}(x,y_{1:t-1})\big{)}$
It is motivated by the recent work [24] that shows the aligned models via RLHF
with varying $\beta$ are geometric mixtures of a reference model and a single
aligned model:

$h_{\widetilde{\theta}}(x,y_{1:t-1})=(1+\lambda)\ast
h_{\theta}(x,y_{1:t-1})-\lambda\ast h_{\text{ref}}(x,y_{1:t-1}),$ (12)

where $\lambda>0$ is a hyper-parameter and $y_{1:t-1}$ indicates the output
sequence before $t$-th output. We remark that this de-coupled noise
identification by approximating $p_{\widetilde{\theta}}(y_{w}\succ y_{l}|x)$
does not require additional computations compared to DPO, since the required
measurements $h_{\theta}$ and $h_{\text{ref}}$ are obtained during the
calculation of the original DPO objective (Eq. 6).

##  5 Experiments

In this section, we present our experimental results to answer the following
question:

  * $\circ$

Does Selfee improve the alignment of LLMs only using a small amount of human-
labeled preference data? (Table 1, Figure 4)

  * $\circ$

Does the proposed method outperform other preference labeling methods? (Table
2, Figure 3)

  * $\circ$

Is Selfee generalizable across various LLMs? (Table 4)

  * $\circ$

Is Selfee also effective to improve overall LLM’s capability? (Table 6)

###  5.1 Experimental Setups

Models. When there are no specific mentions, our experiments were conducted
using the supervised fine-tuned Mistral-7b-0.1 model [15], as the initial
model $\pi_{\text{init}}$ in Section 4. Specifically, we use the open-sourced
model555alignment-handbook/zephyr-7b-sft-full that follows the recipe of
zephyr [40] and fine-tuned on the instructions of Ultrachat [8]. More details
are presented in Appendix B.

Baselines. To evaluate the effectiveness of the proposed preference judgment
method (Eq. 7), we compare it with other preference judgment methods.
Specifically, we consider the baselines that train the model via iterative DPO
[35, 47], which iteratively generate preference data and update the model,
using LLM-as-judge [3, 50] (i.e., in-context learning) or an external powerful
reward model (PairRM [16]) for the preference judgment. Notably, these
approaches are the same in the case of changing the judgment method and
removing self-refinement in Selfee. Details are presented in Appendix B.

Datasets. For the preference learning, we utilized UltraFeedback dataset [7],
following the previous works [35, 34].666"argilla/ultrafeedback-binarized-
preferences-cleaned" To be specific, from this dataset, we first construct the
seed data, consisting of 2K samples with prompts, responses, and ground truth
preference labels. Then, the remaining samples are divided into subsets of 8K,
20K, and 30K samples, leaving only the prompts. These subsets were used as the
prompt sets for the iteration stages 1, 2, and 3, respectively.

Evaluations. Following the common practice, we mainly evaluate each model our
evaluations using AlpacaEval2.0 [9, 10, 20]. AlpacaEval2.0 approximately
evaluates human preference for instruction following. Using 805 instructions
from various datasets, the evaluation is conducted by comparing the response
of GPT-4 [28] and the testing model to measure win rates. To mitigate the
length bias of LLM’s preference [42, 50], both vanilla and length-controlled
win rates are simultaneously measured.

Implementation details. After the initialization stage, we conduct three
rounds of data expansion with self-generated preference data. For data
expansion, we sampled 2 responses independently per each prompt with a
temperature of 0.7. Then, using the SFT model as the reference model, we
assign the preference label (Eq. 7). The initial DPO training to obtain
$\pi_{0}$ was conducted for 3 epochs on the seed dataset. Training on each
subsequent iteration was carried out for 1 epoch. For the hyper-parameter
$\beta$ of DPO, we used a fixed value of $\beta=0.1$. The batch size was set
to 32, and the learning rate was $5\times 10^{-7}$. We employed adamw_torch
optimizer and a cosine learning rate scheduler with a warm-up phase
corresponding to 10% of the total training steps. For the hyper-parameters
$\alpha$ and $K\%$ for Selfee, we used fixed values of $\alpha=0.1$ and
$K=10$. Additionally, a warm-up phase was included in the denoising stage,
with denoising activated after 20% of the total training steps had been
completed. Regarding the hyper-parameters $\lambda$ for de-coupled noise
detection, we utilized the progressively reduced values of 1/2, 1/4, and 1/8
for iterations 1, 2, and 3, respectively.

Table 1: Main results. Evaluation results on AlpacaEval2.0 with different
variants of Mistral-7B-v0.1. Here, the human label (%) indicates the
proportion of used human preference data.

Models Gold Label (%) Len-control. Win Rate Win Rate vs. GPT-4 Avg. len (#
chars) Mistral-7B-v0.1 - 0.17 0.50 5692 Zephyr-7b-$\alpha$ 100 10.3 8.4 1302
Zephyr-7b-$\beta$ 100 13.2 11 1444 SFT - 7.58 4.72 901 DPO 3.3 9.03 7.68 1802
Selfee (Ours) 3.3 15.39 21.13 2884

Table 2: Comparison between judgment methods. Evaluation results on
AlpacaEval2.0 with iteratively trained models (from SFT model) under different
preference judgment methods.

Methods External Model Len-control. Win Rate Win Rate vs. GPT-4 Avg. len (#
chars) Iterative DPO (PairRM) ✓ 11.87 9.46 1292 Iterative DPO (LLM-as-judge) ✗
9.28 9.18 1711 Selfee (no refine) ✗ 14.41 19.91 3204

###  5.2 Main results

After completing 3 iterations of data expansion and fine-tuning, our model
achieved a 21.13% win rate against GPT-4 on the AlpacaEval2.0 benchmark, as
presented in Table 1. This represents a significant improvement compared to
the 7.68% (7.68% $\rightarrow$ 21.13%) win rate achieved when using only 3.3%
of labeled data with the standard DPO training, while the length-control win
rate is also improved. (9.03% $\rightarrow$ 15.39%). Comparing our model to
the Zephyr-7b series, which uses the same base model (Mistral-7B-0.1v) and is
trained on the same SFT dataset and the entire UltraFeedback preference
dataset, our framework achieved superior performance in both win rate (11% vs
21.13%) and length-control win rate (13.2% vs 15.39%). However, at the same
time, it is noteworthy that the average length of model outputs increased from
1802 to 2884 characters through the iterations. While such length increases
are commonly observed in DPO-based algorithms and likely contributed to the
higher win rate, the significant improvements in length-control win rate
affirm the overall enhancement in performance from Selfee.

Figure 3: Improvements during iterations. Length control (LC.) win rate (%)
measured by AlpacaEval2.0 is consistently improved by Selfee and it
outperforms other baselines.

Next, in Table 2, we present additional experimental results to validate the
proposed preference judgment method. For a fairer comparison, we provide the
result of the variant of Selfee (Selfee (no refine)), which removes the self-
refinement procedure (in Section 4.2) and makes it as Iterative DPO with
preference judgment via Eq. 7. Consequently, three experiments in Table 2
become iterative DPO variants with different preference judgment methods.
Here, one can observe that our approach showed significantly better
performance compared to other methods. Specifically, Selfee achieved a win
rate of 19.91% against GPT-4 on AlpacaEval2.0, compared to 9.46% for the
baseline with an external reward model, PairRM. In terms of length control win
rate, Selfee achieved 14.41%, surpassing the reward model’s 11.84%. On the
other hand, the in-context learning approach (LLM-as-judge) shows a similar
win rate compared to PairRM, but falls short in length control win rate
(11.87% vs 9.28%), showing the limitations of the LLM-as-judge approach.
Overall, the results reveal the superiority of our direct preference judgment
over other judgment methods. This superiority is consistently observed through
the iterations, as shown in Figure 3.

Table 3: Ablation study. Evaluation results on AlpacaEval2.0 with iteratively
trained models (from SFT) under different methodological configurations of
Selfee. DE, SR, DND are abbreviations of data expansion, self-refinement, and
de-coupled noise detection, respectively.

Methods DE SR DND Len-control. Win Rate Win Rate vs. GPT-4 Avg. len (# chars)
SFT - - - 7.58 4.72 901 DPO - - - 9.03 7.68 1802 ✓ ✗ ✗ 14.41 19.91 3204 Selfee
(Ours) ✓ ✓ ✗ 14.7 19.94 3043 ✓ ✓ ✓ 15.39 21.13 2884

###  5.3 More analyses

Ablation study. To evaluate the impact of the self-refinement components, we
conducted ablation experiments by excluding both self-refinement (SR) and
decoupled noise detection (DND) from the existing framework. The results are
presented in Table 3. With self-refinement without decoupled noise detection
(Eq. 10), we observed a slight performance improvement, with the win rate
against GPT-4 marginally increasing from 19.91% to 19.94%, and the length
control win rate rising from 14.41% to 14.7%. But, when the decoupled noise
detection is incorporated into the self-refinement (Eq. 11), we observed
significant improvements, with the win rate increasing from 19.91% to 21.13%
and the length control win rate improving from 14.41% to 15.39%. Furthermore,
the inclusion of the self-refinement components led to a reduction in the
average response length, decreasing from 3204 to 3043 and finally to 2884
characters (i.e., improvement in length bias). Also, these results confirm
that the self-refinement component is a crucial factor in enhancing
performance, contributing to both higher win rates and better length control.

Compatibility with different models. Next, to verify the compatibility of our
framework across different models, we conducted experiments using the Phi-2
model [21].777https://huggingface.co/microsoft/phi-2 Specifically, we
conducted experiments based on the supervised fine-tuned
Phi-2888lole25/phi-2-sft-ultrachat-full, which, like Mistral, has been
supervised fine-tuned on the UltraChat dataset. While maintaining the core
structure of our framework, we made slight adjustments to the learning rate to
accommodate the different characteristics of the Phi-2 ($5\times 10^{-6}$). In
addition, due to the smaller size of the Phi-2, we observed that performance
improvements were not evident beyond iteration 2. Therefore, we present the
results of iteration 1. As shown in Table 4, the experimental results showed
that applying Selfee to Phi-2 yields a significant increase in performance.
Specifically, the win rate improved from 5.67% to 9.43%, and the length
control win rate increased from 7.02% to 9.1%, compared to Phi-2 trained with
DPO. These results demonstrate that Selfee is effective across different
models, although its impact may be limited when applied to smaller models.

Figure 4: Improvements without seed data. Evaluation results on AlpacaEval2.0
with Mistral-7B-instruct-v0.1 and Selfee without seed preference dataset.

Dependency on seed data. In addition, we evaluated the feasibility of using
our method even without seed preference data. Namely, we want to answer
whether LLM can derive explicit human preference between responses, by
leveraging their intrinsic knowledge learned about humans, during the previous
training, such as pre-training or supervised instruction tuning (SFT). For
this experiment, we used the Mistral-7b-instruct-0.1v [15] as the initial
model (i.e., $\pi_{0}$) and the Mistral-7b-0.1v-base as the reference model
(i.e., $\pi_{\text{init}}$) (see the initial setup in Section 4). This setup
allows us to demonstrate that our framework can function effectively even in
the absence of seed preference data, when the model is sufficiently fine-tuned
with iterative data expansion and learning through self-refinement. As shown
in Figure 4, the win rate increased from 6.31% to 9.79%, and the length-
control win rate improved from 10.14% to 11.59%, after 3 iterations. Notably,
unlike experiments with seed preference data, there was no significant
increase in the average length of model outputs (1077 to 1461 characters).
This indicates that Selfee can effectively leverage internal information to be
aligned with human preference even without seed data.

Table 4: Compatibility across various LLMs. Evaluation results on
AlpacaEval2.0 with different variants fine-tuned from Phi-2.

Models Gold Label (%) Len-control. Win Rate Win Rate vs. GPT-4 Avg. len (#
chars) Phi-2 SFT - 5.88 3.78 978 Phi-2 DPO 3.3 7.02 5.67 1342 Selfee (Ours)
3.3 9.1 9.43 1861

Table 5: Additional analyses with Selfee. Evaluation results on AlpacaEval2.0
with models that fine-tuned with different judgment methods, from the
resulting model of 1st iteration of Selfee.

Models Len-control. Win Rate Win Rate vs. GPT-4 Avg. len (# chars) Selfee
after Iteration 1 10.57 11.89 2147 Eq. 7 with initial SFT model (Ours) 15.08
19.56 2772 Eq. 7 with previous model 13.73 17.66 2757 Judgment with PairRM
13.57 13.72 1917 Judgment without reference model 12.83 12.35 1768

Additional analysis with judgment methods. In Table 5, we further analyzed the
impact of the reference model in the preference judgment process in Eq. 7.
This analysis was conducted during the transition from iteration 1 to
iteration 2, where the most significant performance changes were observed
(i.e., we fine-tune from the resulting model of iteration 1). To isolate and
compare the effect of judgment methods, we followed the setup in Table 2 and
so excluded the influence of the self-refinement component. Then, we
experimented with three setups by varying the judgment method using (1) the
current policy from the previous iteration as the reference model, (2)
performing judgment without any reference model, and (3) using the PairRM for
judgment.

The results are presented in Table 5. Here, the experimental results
demonstrated that the method used in Selfee, where the SFT model was utilized
as the reference model for preference judgment, achieved the highest
performance increase. Specifically, using the model from the previous
iteration as the reference model showed lower performance, with a relatively
larger decrease in the length control win rate (15.08% vs 13.73%) compared to
the win rate (19.56% vs 17.66%). Despite these decreases, it still outperforms
using PairRM. These results may imply the importance of judging the preference
through the training LLM rather than the external model, as it is less
suffering from the distribution mismatch. However, without reference model
(i.e., only using the likelihood of the current model), the performance
increase was the lowest compared to all other cases. These findings underscore
the substantial impact of the choice of proper judgment method and reference
model.

Evaluation on MT-Bench. We further evaluate trained models using MT-Bench [16]
to assess different aspects of LLMs. Namely, MT-Bench evaluates a chatbot’s
overall abilities across multiple categories related to key LLM capabilities
such as math, coding, roleplay, writing, etc. The evaluation is conducted by
scoring responses to multi-turn questions using GPT-4. As shown in Table 6,
Selfee achieved a score of 6.94 on the MT-Bench, outperforming other baselines
and the model trained with DPO on 3.3% gold labeling data. In addition, when
we refine the preference using relatively less aligned predictions (i.e.,
$\lambda=0$), denoted by Selfee∗, it even outperforms the strong baseline with
the external reward model. Moreover, we observe that Selfee consistently
improves the MT-bench performance without seed preference data or with smaller
LLM, as presented in Table 7 in Appendix C. These results indicate that Selfee
not only excels in specific tasks but also contributes to the overall
performance enhancement of LLMs.

Table 6: MT-Bench. Evaluation results on MT-bench with different variants of
Mistral-7B-v0.1.

Models Gold Label (%) Avg Score Mistral-7B-v0.1 - 3.25 SFT - 6.34 DPO 3.3 6.81
Iterative DPO (PairRM) 3.3 6.98 Iterative DPO (LLM-as-judge) 3.3 6.67 Selfee
(Ours) 3.3 6.94 Selfee∗ (Ours) 3.3 7.09

##  6 Conclusion

In this paper, we proposed Selfee, a method that can efficiently improve the
alignment of LLMs using minimal human-labeled preference data. Our main
contributions include the development of an effective data expansion method
with the direct preference judgment method and preference learning algorithm
with the self-refinement of (potentially) noise preference. We demonstrate the
effectiveness of Selfee by fine-tuning the recent LLMs with the various
setups, and observing the significant improvements when evaluating them on the
commonly used benchmarks.

Limitation and future work. In the experiments, Selfee has shown the tendency
to increase the responses’ length. We demonstrated that the improvement by
Selfee is not a simple result of such length increase, by observing the
increase of win rate under a length length-controlled setup or MT-bench.
However, depending on the user, this behavior could be dispreferred. In this
sense, focusing on mitigating this bias during the self-improving alignment
will be an interesting future direction, and can enhance the robustness and
generalizability of Selfee across more diverse scenarios.

## References

  * Anthropic [2024] Anthropic.  Introducing the next generation of claude.  _https://www.anthropic.com/news/claude-3-family_ , 2024.
  * Bai et al. [2022a] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. [2022b] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Bradley and Terry [1952] R. A. Bradley and M. E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Christiano et al. [2017] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Chung et al. [2024] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al.  Scaling instruction-finetuned language models.  _Journal of Machine Learning Research_ , 25(70):1–53, 2024.
  * Cui et al. [2023] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun.  Ultrafeedback: Boosting language models with high-quality feedback.  _arXiv preprint arXiv:2310.01377_ , 2023.
  * Ding et al. [2023] N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  _arXiv preprint arXiv:2305.14233_ , 2023.
  * Dubois et al. [2023] Y. Dubois, C. X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. S. Liang, and T. B. Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Dubois et al. [2024] Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  _arXiv preprint arXiv:2404.04475_ , 2024.
  * Ethayarajh et al. [2024] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela.  Kto: Model alignment as prospect theoretic optimization.  _arXiv preprint arXiv:2402.01306_ , 2024.
  * Go et al. [2023] D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman.  Aligning language models with preferences through f-divergence minimization.  In _International Conference on Machine Learning_ , 2023.
  * Han et al. [2018] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Hong et al. [2024] J. Hong, N. Lee, and J. Thorne.  Orpo: Monolithic preference optimization without reference model.  _arXiv preprint arXiv:2403.07691_ , 2024.
  * Jiang et al. [2023a] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023a.
  * Jiang et al. [2023b] D. Jiang, X. Ren, and B. Y. Lin.  Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.  In _Annual Conference of the Association for Computational Linguistics_ , 2023b.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Lee et al. [2021] K. Lee, L. Smith, and P. Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  In _International Conference on Machine Learning_ , 2021.
  * Li et al. [2020] J. Li, R. Socher, and S. C. Hoi.  Dividemix: Learning with noisy labels as semi-supervised learning.  In _International Conference on Learning Representations_ , 2020.
  * Li et al. [2023a] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023a.
  * Li et al. [2023b] Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee.  Textbooks are all you need ii: phi-1.5 technical report.  _arXiv preprint arXiv:2309.05463_ , 2023b.
  * Lin et al. [2024] B. Y. Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, K. Chandu, C. Bhagavatula, and Y. Choi.  The unlocking spell on base llms: Rethinking alignment via in-context learning.  In _International Conference on Learning Representations_ , 2024.
  * Liu et al. [2023] T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, P. J. Liu, and J. Liu.  Statistical rejection sampling improves preference optimization.  _arXiv preprint arXiv:2309.06657_ , 2023.
  * Liu et al. [2024] T. Liu, S. Guo, L. Bianco, D. Calandriello, Q. Berthet, F. Llinares, J. Hoffmann, L. Dixon, M. Valko, and M. Blondel.  Decoding-time realignment of language models.  In _International Conference on Machine Learning_ , 2024.
  * Müller et al. [2019] R. Müller, S. Kornblith, and G. E. Hinton.  When does label smoothing help?  In _Advances in Neural Information Processing Systems_ , 2019.
  * Nakano et al. [2021] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman.  Webgpt: Browser-assisted question-answering with human feedback.  In _arXiv_ , 2021.
  * OpenAI [2022] OpenAI.  Introducing chatgpt.  _https://openai.com/blog/chatgpt_ , 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Ouyang et al. [2022] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Peng et al. [2019] X. B. Peng, A. Kumar, G. Zhang, and S. Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ , 2019.
  * Peters and Schaal [2007] J. Peters and S. Schaal.  Reinforcement learning by reward-weighted regression for operational space control.  In _International Conference on Machine Learning_ , 2007.
  * Rafailov et al. [2023] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn.  Direct preference optimization: Your language model is secretly a reward model.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Reed et al. [2014] S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich.  Training deep neural networks on noisy labels with bootstrapping.  _arXiv preprint arXiv:1412.6596_ , 2014.
  * Rosset et al. [2024] C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie.  Direct nash optimization: Teaching language models to self-improve with general preferences.  _arXiv preprint arXiv:2404.03715_ , 2024.
  * Snorkel [2024] Snorkel.  New benchmark results demonstrate value of snorkel ai approach to llm alignment.  _https://snorkel.ai/new-benchmark-results-demonstrate-value-of-snorkel-ai-approach-to-llm-alignment_ , 2024.
  * Sohn et al. [2020] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Stiennon et al. [2020] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano.  Learning to summarize from human feedback.  In _NeurIPS_ , 2020.
  * Team et al. [2023] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Touvron et al. [2023] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Tunstall et al. [2023] L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_ , 2023.
  * Wang et al. [2023a] P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui.  Math-shepherd: Verify and reinforce llms step-by-step without human annotations.  _CoRR, abs/2312.08935_ , 2023a.
  * Wang et al. [2023b] Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al.  How far can camels go? exploring the state of instruction tuning on open resources.  In _Advances in Neural Information Processing Systems_ , 2023b.
  * Wei et al. [2022a] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022a.
  * Wei et al. [2022b] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , 2022b.
  * Wu et al. [2024] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu.  Self-play preference optimization for language model alignment.  _arXiv preprint arXiv:2405.00675_ , 2024.
  * Xiong et al. [2024] W. Xiong, H. Dong, C. Ye, Z. Wang, H. Zhong, H. Ji, N. Jiang, and T. Zhang.  Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint.  In _ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models_ , 2024.
  * Xu et al. [2023] J. Xu, A. Lee, S. Sukhbaatar, and J. Weston.  Some things are more cringe than others: Preference optimization with the pairwise cringe loss.  _arXiv preprint arXiv:2312.16682_ , 2023.
  * Yuan et al. [2024] W. Yuan, R. Y. Pang, K. Cho, S. Sukhbaatar, J. Xu, and J. Weston.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_ , 2024.
  * Zhao et al. [2023] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.
  * Zheng et al. [2023] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

##  Appendix A Societal Impact

Selfee enables efficient human preference learning, allowing for cost-
effective training of models in data-scarce or domain-specific areas. Our
framework supports alignment learning in various fields, including
multilingual language learning and preferences beyond human helpfulness.
Consequently, it could contribute to facilitating the widespread adoption of
LLM technology across diverse sectors. By lowering the barriers to alignment
learning, Selfee makes it more accessible to a broader audience. However, the
widespread availability of this technology also brings potential risks. The
reduced cost of training models could enable malicious actors to misuse the
technology, leading to societal issues. Therefore, it is crucial to implement
ethical considerations and safety measures when deploying Selfee technology to
mitigate these risks.

##  Appendix B More Details of Experimental Setups

###  B.1 SFT model setup

Mistral. For supervised fine-tuning, Ultrachat dataset [8] is
used999https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k, batch
size was set 128, total epoch was 1, and the learning rate was $2\times
10^{-5}$. It employed Adam optimizer [17] and a cosine learning rate scheduler
with a warm-up phase corresponding to 10% of the total training steps.

Phi-2. For supervised fine-tuning, Ultrachat dataset is used, batch size was
set 64, total epoch was 3, and the learning rate was $2\times 10^{-5}$. It
employed Adam optimizer and a cosine learning rate scheduler with a warm-up
phase corresponding to 10% of the total training steps.

###  B.2 Baselines experiment setup

Zephr-7b-$\beta$. Zephr-7b-$\beta$ [40], which is compared in Table 1, was
trained using the same pre-trained model (mistral-7b-0.1v [15]) and the same
SFT data (Ultrachat [8]), but there are marginal differences compared to our
setup. Specifically, Zephr-7b-$\beta$ used the batch size of 512, but 128 was
used for the model in our experiments. As a result, the SFT model of
Zephr-7b-$\beta$101010https://huggingface.co/HuggingFaceH4/mistral-7b-sft-
beta, according to AlpacaEval2.0, showed a length control win rate of 8.14%
and a win rate of 5.26%, approximately 0.5% better than the SFT model used in
our experiments (8.14% vs 7.58%, 5.26% vs 4.72%). In addition, regarding the
preference dataset, Zephr-7b-$\beta$ was trained using 100% of the original
Ultrafeedback [7]
111111https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized
but we use 3.3% of its cleaned
version121212https://huggingface.co/datasets/argilla/ultrafeedback-binarized-
preferences-cleaned. Consequently, the results under the two datasets might be
slightly different, and the impact of these method differences has been
specifically denoted in [14]; here, Ultrafeedback clean shows approximately a
0.9% improvement in performance based on the AlpacaEval2.0 win rate (from
11.33% to 12.2%). Consequently, although Zephr-7b-$\beta$ is not a perfectly
strict comparison, we believe the minor variations are relatively small
compared to the differences between Selfee and Zephr-7b-$\beta$, and hence the
comparison results are meaningful.

LLM-as-Judgement. For LLM-as-judge, we used an SFT model to employ Consitual
AI’s pairwise comparison prompt for judging preferences [2]. Preference is
measured by comparing the logprob value of the token output as input to the
following prompt (Listing 5). To ensure fair comparison and prevent low
judgment performance, evaluation instructions were created using seed
preference data which is the same form as Consitual AI’s pairwise comparison.
(Listing 6) Using these, additional SFT learning is performed to obtain an
independent LLM-as-judge model. For this supervised fine-tuning, we set the
batch size 32, total epoch is 3, and the learning rate was $2\times 10^{-5}$.
We employed Adam optimizer and a cosine learning rate scheduler with a warm-up
phase corresponding to 10% of the total training steps.

Reward model judgment. For the reward model baseline, we selected PairRM [16]
due to its high performance on AlpacaEval2.0 [35, 45]. Unlike SELFEE, which
was trained on only 2K gold label data, PairRM was trained on a large-scale
dataset. The training data for PairRM includes the following:

  * •

openai/summarize_from_feedback [37]

  * •

openai/webgpt_comparisons [26]

  * •

Dahoas/synthetic-instruct-gptj-
pairwise131313https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-
pairwise

  * •

Anthropic/hh-rlhf [2]

  * •

lmsys/chatbot_arena_conversations [50]

  * •

openbmb/UltraFeedback [7]

The total number of pairwise samples in this training data is approximately
500K, compared to 2K for SELFEE. Specifically, the summarize_from_feedback
dataset contributes 179K samples, and the hh-rlhf dataset contributes 161K
samples, making up a significant portion of the total.

###  B.3 Implementation details

Resources and computation cost. For all experiments, we utilized 4 A6000 GPUs.
Under this computational resource, generating responses for 10K prompts takes
approximately 1 to 2 hour, and preference judging for generated responses also
takes about 1 to 2 hour. For training of model with Eq. 10, it takes about 1
to 2 hours per epoch. Therefore, the total time required to complete response
generation, preference judgment, and one epoch of training was between 5 to 6
hours per 10K prompt.

Response generation. To mitigate the length bias from training with Direct
Policy Optimization (DPO), we restricted the maximum token length for self-
generated responses to 300 tokens. This limit corresponds to approximately 900
characters.

Refinement. Using the simple moving average (SMA) method, we measured the
confidence over the most recent 50 samples. Based on this, we calculated the
boundary corresponding to the lower K% of the confidence distribution (Eq.
11).

Table 7: MT-Bench. Evaluation results on MT-bench with different models.
Selfeeinst and Selfeephi denotes the fine-tuned Mistral-7B-Instruct-v0.1 and
Phi-2 SFT with Selfee, respectively.

Models Gold Label (%) Avg Score Mistral-7B-Instruct-v0.1 - 6.78 Selfeeinst
(Ours) - 7.12 Phi-2 SFT - 5.35 Phi-2 DPO 3.3 6.16 Selfeephi (Ours) 3.3 6.33

⬇

f”’

Consider the following conversation between a human and an assistant:

[prompt]

Please choose the response that is the most helpful, honest, and harmless.

Options:

(A) [response 1]

(B) [response 2]

The answer is:

”’

Figure 5: Prompt used for judging preferences.

⬇

f”’

Consider the following conversation between a human and an assistant:

[Horses are hybrids of which two animals?]

Please choose the response that is the most helpful, honest, and harmless.

Options:

(A) [Horses, scientifically known as Equus caballus, are not the result of
hybridization between two other animals. They are considered a species of
their own within the Equidae family, which also includes other equines such as
donkeys, mules, and zebras. Horses are believed to have evolved from a small,
multi-toed forest-dwelling animal known as Eohippus, which lived around 60
million years ago. Over millions of years, these horses evolved into the
large, single-toed animals we know today.]

(B) [Horses are not hybrids of any other animals. They are a unique species
and are not a product of cross-breeding between different animals. Horses are
domesticated animals and have been bred and developed over thousands of years
by humans for various purposes such as transportation, riding, and
agriculture.]

The answer is: (A)

\par”’

Figure 6: Evaluation instruction.

##  Appendix C More Evaluation Results in MT-bench

In this section, we further provide the additional evaluation results on MT-
bench [16], which couldn’t be presented in the main text, due to the space
issue.

((a)) MT-bench task-wise evaluation results

((b)) Iteration-wise improvement with MT-bench

Figure 7: More evaluation results with MT-bench.

We first present (a) task-wise evaluation results and (b) iteration-wise
average improvement in Figure 7. As shown in Figure 7(a), Selfee consistently
improves the performance in various tasks. Notably, there is almost no gain in
Coding and degradation in Math. We remark that this phenomenon is commonly
observed in the relevant literature [22], which indicates that different
training [41] or inference [44] schemes might be necessary to improve the
performance in these tasks.

Next, in Figure 7(b), one can observe that the average performance on the MT-
bench is increased with more iterations. Specifically, while the iterative DPO
using PairRM shows the best performance until iteration 2, Selfee∗ outperforms
it in iteration 3. It demonstrates the effectiveness of our framework for
iteratively improving the alignment of LLM.

Lastly, we present additional evaluation results on MT-bench, not presented in
Table 6. Specifically, we measure the performances of Phi-2 variants and
Mistral-7B-Instruct-v0.1 variants, which presented in Table 4 and Figure 4.
The results are presented in Table 7. As one can see, Selfee consistently
yields the improvement across different backbones of Mistral-7B-Instruct-v0.1
and Phi-2.

##  Appendix D Qualitative Examples

In this section, we present examples of the responses from the models trained
under different methods (SFT, DPO, and Selfee), for a qualitative comparison.
The results are presented in Figures 8, 9, and 10. As one can see in the
examples, the model trained under Selfee provides more detailed and
informative responses to various questions.

Prompt:
---
Who is Larry Page?
SFT response:
Larry Page is the co-founder and CEO of Google, a multinational technology
company that provides various services such as search engine, online
advertising, and cloud computing. He was born on March 26, 1973, in East
Lansing, Michigan, and graduated from the University of Michigan with a degree
in computer engineering. Page and Sergey Brin founded Google in 1998, and
since then, it has become one of the most successful and influential companies
in the world.
DPO response:
Larry Page is an American computer scientist and entrepreneur who co-founded
Google, one of the world’s largest technology companies, with Sergey Brin in
1998. He is the CEO of Alphabet Inc., the parent company of Google, and has
been instrumental in the development and growth of the company. Page is known
for his innovative ideas and his focus on developing new technologies that can
improve people’s lives. He has been recognized for his contributions to the
field of computer science and has received numerous awards and honors,
including the Presidential Medal of Freedom in 2015.
SELFEE response:
Larry Page is an American entrepreneur, computer scientist, and co-founder of
Alphabet Inc., the parent company of Google, one of the world’s largest
technology conglomerates. Born on March 26, 1973, in East Lansing, Michigan,
Page grew up in a family of academics and engineers. He earned his Bachelor’s
degree in Computer Engineering and Master’s degree in Computer Science from
Stanford University in 1995 and 1998, respectively.
In 1998, Page teamed up with Sergey Brin, his classmate at Stanford, to
develop a new search engine called "Backrub," which later evolved into Google.
Initially funded by venture capitalists, Google quickly gained popularity for
its innovative search algorithm, user-friendly interface, and extensive
collection of online content. Page served as the CEO of Google from its
inception in 1999 until 2019, overseeing its rapid growth, expansion into new
markets, and development of groundbreaking products and services, such as
Android, Maps, YouTube, and Cloud Computing.
Under Page’s leadership, Google became a dominant player in the tech industry,
with a market capitalization of over $1 trillion and a global workforce of
over 135,000 employees. In 2019, Page stepped down as CEO of Google and
assumed the role of CEO of Alphabet, the holding company that reorganized
Google’s core businesses and diversified into new areas, including healthcare,
artificial intelligence, and sustainability.
Aside from his entrepreneurial achievements, Page is known for his
philanthropic efforts, particularly through the Larry Page Foundation, which
supports research in renewable energy, education, and medical innovation. He
has also been recognized for his contributions to the tech industry, receiving
numerous awards, honors, and recognitions, including the Presidential Medal of
Freedom, Time Person of the Year, and Forbes’ Person of the Year.
Figure 8: Response in AlpacaEval2.0 Prompt:
---
What causes the northern lights
SFT response:
The northern lights, also known as aurora borealis, are caused by the
interaction between charged particles from the sun and the Earth’s magnetic
field. When the charged particles from the sun enter the Earth’s atmosphere,
they collide with the gases in the atmosphere, causing them to emit light. The
colors of the northern lights are determined by the type of gas that is
colliding with the charged particles. Green is the most common color, but
other colors such as red, yellow, and blue can also be seen.
DPO response:
The northern lights, also known as aurora borealis, are caused by the
interaction between charged particles from the sun and the Earth’s magnetic
field and atmosphere.
When the sun emits charged particles, such as electrons and protons, they
travel through space towards the Earth. As they approach the Earth, they are
deflected by the Earth’s magnetic field, which creates a protective bubble
around the planet.
When these charged particles enter the Earth’s atmosphere, they collide with
the gases in the atmosphere, such as oxygen and nitrogen, causing them to
become excited and emit light. This light is what we see as the northern
lights.
The colors of the northern lights depend on the type of gas that is being
excited. For example, oxygen produces green and yellow lights, while nitrogen
produces blue and red lights.
The intensity and frequency of the northern lights depend on the activity of
the sun, which can vary over time. During periods of high solar activity, the
northern lights can be seen more frequently and with greater intensity.
SELFEE response:
The northern lights, also known as aurora borealis, are a natural phenomenon
caused by the interaction between charged particles from space and the Earth’s
magnetic field and atmosphere. The primary cause of the northern lights is the
solar wind, a stream of charged particles (primarily electrons, protons, and
alpha particles) emitted from the Sun’s corona and solar atmosphere.
As the solar wind approaches the Earth, it encounters the magnetic field,
which deflects and traps some of the charged particles in a region known as
the magnetosphere. The magnetosphere extends outward from the Earth’s surface
to a distance of several hundred thousand kilometers and is shaped like a
teardrop, with the magnetic field lines converging towards the polar regions.
When energetic particles from the solar wind enter the magnetosphere, they
collide with the atoms and molecules in the Earth’s upper atmosphere,
primarily in the polar regions, where the magnetic field lines are strongest.
These collisions ionize the atmospheric gases, creating a plasma region known
as the ionosphere.
The ionization process releases energy in the form of electromagnetic
radiation, including ultraviolet, visible, and radio waves. The visible
auroral emissions occur when the energetic particles excite the electrons in
the ionized gases, causing them to emit photons with specific wavelengths,
resulting in the characteristic colors of the northern lights.
The colors of the aurora depend on the type of gas molecule involved in the
collisions and the energy level of the excited states. Green and yellow
auroras are typically associated with oxygen atoms, while red and blue auroras
are caused by nitrogen molecules. Other gases, such as hydrogen, helium, and
carbon dioxide, can also contribute to the auroral spectrum, resulting in
additional colors like purple, pink, and white.
The intensity and frequency of auroral displays vary depending on solar
activity levels, geomagnetic conditions, and seasonal factors. Solar maximum
periods, when the Sun’s activity is at its highest, tend to produce more
frequent and intense auroral displays in both hemispheres. However, even
during solar minimum periods, when solar activity is low, occasional auroral
events can still occur in the polar regions.
Figure 9: Response in AlpacaEval2.0 Prompt:
---
What type of soil is suitable for cactus?
SFT response:
Cactus plants prefer well-draining soil that is slightly alkaline. A mixture
of sand, perlite, and peat moss is a good option for cactus soil. The soil
should be lightweight and allow excess water to drain quickly. It is also
important to avoid using soil that is too rich in nutrients, as cactus plants
do not require a lot of fertilizer.
DPO response:
Cacti prefer well-draining soil that is slightly alkaline to neutral. The
ideal soil for cacti is a mixture of sand, perlite, and peat moss. The sand
provides good drainage, while the perlite helps to aerate the soil and improve
water retention. The peat moss adds organic matter and helps to retain
moisture. The soil should have a pH level between 6.0 and 7.5. Avoid using
heavy, clay-based soils, as they can retain too much moisture and lead to root
rot. It is also important to ensure that the soil is not too rich in
nutrients, as cacti prefer a slightly nutrient-poor soil.
SELFEE response:
Cacti require a well-draining soil mix that is specifically formulated for
their unique needs. The ideal soil for cacti is a cactus soil mix, also known
as succulent soil or desert soil.
Cactus soil mixes typically contain a blend of coarse-textured, porous
materials such as:
1\. Coarse sand: This provides excellent drainage and aeration, preventing
waterlogging and root rot. Cacti prefer sandy soils with a particle size of
0.5-2.0 mm.
2\. Perlite or vermiculite: These lightweight, volcanic minerals improve soil
structure, increase water retention capacity, and promote air circulation
around the roots. They also help to break up compacted soil and improve
drainage.
3\. Pumice or expanded clay: Similar to perlite and vermiculite, these
materials improve soil porosity and aeration, while also adding nutrients and
buffering acidity levels.
4\. Peat moss or coconut coir: These organic components provide some moisture
retention and nutrient content, but in small quantities (<20%) to avoid
excessive water retention and potential fungal growth.
5\. Mineral aggregates: Optional additions such as crushed limestone,
dolomite, or volcanic ash can further enhance drainage, pH buffering, and
nutrient availability for cacti.
The ideal ratio of these components varies depending on the specific species
and growing conditions, but a common starting point is a mix of 60-70% sand,
20-30% perlitevermiculitepumice, and 5-10% organic matter. It is recommended
to purchase pre-mixed cactus soil blends from reputable nurseries or garden
centers, as they often include optimized ratios and additional amendments for
optimal cactus growth.
Figure 10: Response in AlpacaEval2.0