# It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

Taiming Lu♣
JHU
&Lingfeng Shen♣
Bytedance
&Xinyu Yang♣
CMU
&Weiting Tan
JHU
Beidi Chen
CMU
&Huaxiu Yao
UNC-Chapel Hill

###### Abstract

Reinforcement Learning from Human Feedback (RLHF) involves training policy
models (PMs) and reward models (RMs) to align language models with human
preferences. Instead of focusing solely on PMs and RMs independently, we
propose to examine their interactions during fine-tuning, introducing the
concept of seamlessness. Our study starts with observing the saturation
phenomenon, where continual improvements in RM and PM do not translate into
RLHF progress. Our analysis shows that RMs fail to assign proper scores to PM
responses, resulting in a 35% mismatch rate with human preferences,
highlighting a significant discrepancy between PM and RM. To measure
seamlessness between PM and RM without human effort, we propose an automatic
metric, SEAM. SEAM quantifies the discrepancies between PM and RM judgments
induced by data samples. We validate the effectiveness of SEAM in data
selection and model augmentation. Our experiments demonstrate that (1) using
SEAM-filtered data for RL training improves RLHF performance by 4.5%, and (2)
SEAM-guided model augmentation results in a 4% performance improvement over
standard augmentation methods. The code is accessible here:
https://github.com/TaiMingLu/seamless

00footnotetext: $\clubsuit$ indicates equal contributions.

##  1 Introduction

Reinforcement learning from human feedback (RLHF) has emerged as a popular
technique to optimize and align a language model with human preferences [43,
24, 22, 9, 28, 44, 1, 3, 32]. RLHF provides a natural solution for optimizing
non-differentiable, scalar objectives for language models and has been the
centerpiece of recent state-of-the-art large language models (LLMs) [21, 11,
10, 15, 1, 27]. In RLHF, a _reward model_ (RM) generates scalar rewards for a
_policy model_ (PM) generated outputs as supervision signals during
reinforcement learning. Since policy gradient methods [35] optimize based on
such signal, the PM and RM inevitably dictate the behavior of the resultant
RLHF model. As such, the properties of RMs (or PMs) and their impact on RLHF
models have become points of interest for the community [7, 49, 6, 7, 36].
Unlike prior work that examines the individual capabilities of each model, in
this work, we introduce and explore the concept of _seamlessness_ between the
PM and RM, focusing on their interactions.

Figure 1: We introduce the concept of Seamlessness to measure the
_discrepancies_ between reward and policy models as supported by human
evaluation. To automate measuring the Seamlessness, we propose SEAM, an
automated method for estimating seamlessness between PM and RM. We validate
its effectiveness through two experimental settings: data selection and
augmentation.

Our study begins with the observation of a saturation phenomenon in the RLHF
process (section 3): beyond a certain threshold, improvements in the quality
of the RM and PM do not translate into increased RLHF performance (Figure 1).
To understand this phenomenon, we explore whether the RM can assign
appropriate scalar rewards to responses $r$ generated by the PM prompted by
instruction $I$. This inquiry addresses the seamlessness between the RM and
PM. Although the RM performs well on standard preference benchmarks, it
struggles to evaluate PM-generated responses effectively. This is demonstrated
by a 35% mismatch rate between reward scores and human preferences, indicating
a significant, persistent discrepancy between the RM and PM as reflected in
the reinforcement learning (RL) training data. This discrepancy does not
diminish even as the PM and RM are individually optimized according to their
respective evaluation paradigms, thus disrupting their seamlessness.
Remarkably, when we remove instructions from the RL dataset that contribute to
this discrepancy and re-conduct RLHF, we observe an improvement in RLHF
performance. This outcome suggests that enhancing the seamlessness between PM
and RM benefits the overall RLHF process.

Based on these findings, we define the seamlessness between the PM and RM as
detailed in section 5 and introduce an automated estimation method, SEAM,
available in three computational variants: SEAM${}_{\text{Adv}}$,
SEAM${}_{\text{Contrast}}$, and SEAM${}_{\text{GPT}}$. Such methods remove the
reliance on manual effort traditionally required for measuring seamlessness.
Essentially, SEAM evaluates the risk associated with each data sample when
employed in RLHF processes, considering the specifics of the given PM and RM.
Additionally, we give two experimental scenarios to demonstrate how SEAM can
be effectively utilized to improve the real-world RLHF process. (1) Data
Selection: We compute the SEAM score for each sample and exclude those with
low scores for RL training data selection. This strategy underscores a “less
is more” phenomenon [48], whereby RLHF performance is enhanced when using this
filtered dataset compared to the unfiltered dataset. Additionally, removing
low-score samples helps mitigate the “saturation phenomenon”. (2) Model
augmentation: During RLHF, we explore the PM and RM failure modes and
subsequently strengthen them based on identified weaknesses. We calculate the
SEAM score for each data sample throughout the RL training phase. Samples
exhibiting low SEAM scores are then selected as targets for data augmentation
to enhance the capabilities of the PM and RM specifically for these
challenging samples. The results show that the SEAM score effectively
functions as a diagnostic metric within the RLHF framework. The primary
contributions of this paper are three-fold:

  * •

We shift focus from the individual capacities of the reward model (RM) and
policy model (PM) to explore their interplay and a noted saturation phenomenon
in RM/PM quality. Our analysis identifies a discrepancy between RM and PM that
cannot be resolved merely by scaling up.

  * •

We conceptualize the seamlessness between PM and RM and introduce SEAM, an
automatic estimation method that quantifies the seamlessness between PM and RM
in a data-centric manner.

  * •

We empirically design two experimental scenarios to demonstrate how SEAM can
be leveraged to improve RLHF training: (1) Data selection and (2) Model
augmentation. Our results validate the effectiveness of SEAM under such
scenarios.

##  2 Related Work and Background

RLHF in Language Models. In earlier studies, reinforcement learning (RL) has
been applied across various domains, such as machine translation [42, 16, 26],
dialogue generation [18, 46, 13], and text generation [20, 50, 39, 43], often
employing modeling reward as automatic evaluation metrics like BLEU [30] or
using simulated feedback [26, 13]. While integrating RL and language models
has been extensively explored, significant advancements in RLHF with LLMs for
general language tasks have only recently emerged [28, 44, 1, 3, 32]. In RLHF,
human feedback is collected to train a reward model, which then serves as a
surrogate for human feedback during the training process, providing scalar
evaluative feedback to the policy model (see detailed background of RLHF in
Appendix A). In RLHF, RL algorithms (e.g., PPO [35]) are particularly suitable
for training PM and RM.

Reward Hacking. In RLHF, a critical issue closely related to our research is
“reward hacking”, as identified in prior studies [2, 29, 41, 36]. This
phenomenon arises from discrepancies between the reward model (RM) and actual
human preferences [7, 17]. Although optimizing towards maximizing the rewards
may initially appear beneficial, it ultimately leads the trained policy to
exploit loopholes in the RM, securing high rewards without achieving the
intended objectives. This degrades performance, complicates the selection of
effective checkpoints, and may produce outputs that do not genuinely reflect
human preferences [40]. Such misalignments increase tendencies towards
sycophancy [31], reinforcing social biases [34, 51] and pose safety risks [25,
5, 38]. A key distinction of our work is its focus on the discrepancies
between RM and PM, which we term ‘seamlessness’, as opposed to the traditional
focus on discrepancies between reward models and human values.

Figure 2: We examine the relation between the RLHF performance and the quality
of PMs and RMs, measured by $\mathcal{P}_{in}$ and $\mathcal{A}_{in}$,
respectively. We can see a “saturation phenomeno”: the continual improvements
of RM/PM do not translate into RLHF improvements.

##  3 The Saturation Phenomenon Reflected in RLHF Quality

In this section, we conduct experiments to investigate the relationship
between the RLHF outcomes and the quality of PM/RM.

Experimental Setup. We follow the experimental configuration of StackLLaMa [4]
due to the proven success of its PPO and data settings for RLHF. Our framework
employs the LLaMa2-7B model as the base model for both the reward and policy
models. To explore the effects of the quality of RM and PM, we change the
volume of training data, enabling us to produce a spectrum of model strengths
for both PM and RM. We develop ten variants each for RMs and PMs. Each pairing
of PM and RM is then subjected to the RLHF technique, resulting in hundreds of
unique RLHF models. Further details on implementation and setup are provided
in Appendix C.

Quality Metrics. We employ two metrics111We do not use the KL divergence
between the outputs from the reference and policy models, as there is no clear
correlation between model quality and such KL divergence. to assess the
quality of the PM and RM: $\mathcal{Q}_{PM}$ (PM quality) and
$\mathcal{Q}_{RM}$ (RM quality). In our experiments on StackExchange,
$\mathcal{Q}_{PM}$ measures how well the policy model generates answers to
StackExchange questions. We use 1000 samples from the StackExchange test
split, with responses generated by the LLM evaluated by GPT-4 on a scale from
1 (worst) to 10 (best), similar to the MT-Bench scale. On the other hand,
$\mathcal{Q}_{RM}$ evaluates the accuracy of the reward model in predicting
human preferences on the StackExchange preference benchmark test split.
Additional details are provided in Appendix C.

Results. We show the correlation between the in-domain performance
($\mathcal{Q}_{PM}$ and $\mathcal{Q}_{RM}$) of RLHF models and the quality of
RMs and PMs, as illustrated in Figure 2. Our primary observation is that while
the quality of RMs and PMs generally positively correlates with the in-domain
performance of RLHF models, a saturation effect is evident. Beyond a certain
quality threshold, additional RM or PM quality improvements yield no further
enhancements in the in-domain performance of RLHF models.

Figure 3: Cross-validation of PM and RM quality using different datasets(3
random seeds). The performance of RM and PM remains consistent across
benchmarks. (e.g., on $\mathcal{D}{rl}$, the PM achieves 96% of its
performance on $\mathcal{D}_{p}$.)

##  4 Analyzing the Origin of Saturation Phenomenon

This section investigates the saturation phenomenon within RLHF, particularly
from the perspective of potentially noisy supervision signals. The RLHF
process comprises three primary stages: (1) policy modeling, (2) reward
modeling, and (3) RL training. Initially, we conduct a sanity check on our PM
and RM under our experimental settings to confirm their capacity for
transferability across different data subsets. As shown in section 4.1, the
results indicate that both the PM and the RM exhibit adequate generalization.
During the RL training stage, however, we observe that the RM struggles to
effectively evaluate many of the responses generated by the PM (section 4.2).
By removing data that reflects this discrepancy between RM and PM, we find
that RLHF performance improves (section 4.3).

###  4.1 A Sanity Check on PM and RM

We hypothesize that the observed saturation phenomenon may be due to the
capacity of RM or PM can not be transferred to data used in other stages
(e.g., the policy model can generate high-quality responses towards SFT
instructions but fails to respond to the RL instructions). Thus, we conducted
a sanity check on both models to answer the following two questions: (1) Q1:
whether the RM consistently distinguishes between better and worse responses
as per the instructions used in SFT and RL training and (2) Q2: whether the PM
sustains its generation quality with instructions from the RL dataset. We
prepare the SFT dataset $\mathcal{D}_{p}$, the preference benchmark
$\mathcal{D}_{r}$, and the RL dataset $\mathcal{D}_{rl}$. Specifically, the PM
and RM were trained on the train splits of $\mathcal{D}_{p}$ and
$\mathcal{D}_{r}$, respectively. We then employed cross-validation techniques
to assess the PM’s performance across the test split of the preference and RL
datasets. Similarly, we tested the RM on the test split of the SFT and RL
datasets. Experimental details are deferred to Appendix C.

The results are shown in Figure 3. We trained five models each for the PM and
RM, subsequently performing cross-validation. The key observation is that the
performance of both PM and RM remains consistent across various in-domain
datasets. This consistency indicates that PM and RM do not have significant
generalization issues under our experimental setup. Besides, it also answers
our two questions: (1) Given a well-trained PM that performs well on the
evaluation set of $\mathcal{D}_{p}$, it can also respond with similar quality
to the instructions in $\mathcal{D}_{rl}$; (2) Given a well-trained RM that
performs well on the evaluation set of $\mathcal{D}_{r}$, it can also perform
similarly well on distinguishing the golden and worse response in
$\mathcal{D}_{p}$ and $\mathcal{D}_{rl}$.

Figure 4: Agreement between reward and human preference is evaluated by
comparing two responses (A and B) from two different policy models. The blue
points indicate agreement between the reward and human preferences, while the
red points represent mismatches. However, the results show that the RM fails
to assign a proper score to the generation from PM.

###  4.2 Discrepancy between RM and PM during RL training

During the RL training stage, the PM is prompted by instructions from the RL
dataset $\mathcal{D}_{rl}$ to generate responses $r_{i}$. The RM then
evaluates these responses, which assigns reward scores to guide the RL
training process. Our empirical analysis reveals two key findings (Figure 3),
given a high-quality PM and RM: (1) the RM can effectively discriminate
between golden and suboptimal responses of instructions within
$\mathcal{D}_{rl}$, and (2) the PM can generate high-quality responses to
instructions from $\mathcal{D}_{rl}$. Thus, we investigate the RM’s capacity
to evaluate the PM’s responses to $\mathcal{D}_{rl}$ since there might be a
distribution shift between the responses generated from PM and those in the
dataset.

Directly evaluating the RM capability to accurately assign scores to responses
generated by the PM conditioned on an instruction $I_{i}$ has significant
challenges since the standard reward modeling cast the preference regression
problem into a classification problem. To address this, we employ a
comparative analysis. We select two PMs of differing qualities (ranked 1 and 5
in previous experiments section 3) and prompt each PM with instructions from
the dataset $\mathcal{D}_{rl}$ (we sample a total of 1,000 instructions). We
collect the responses and organize them into pairs for evaluation. Each pair
of responses is evaluated by two methods: (1) human judgment and (2) RM
evaluation using the rank 1 RM from section 3 to determine if even our best RM
faces issues. To investigate the matching degree between RM and human
preferences, we present pairs of responses (A and B) from the two PMs to human
annotators without revealing the originating model. Human annotators are asked
to annotate their preference between the two options. Similarly, we determine
RM preferences based on their assigned reward scores.

The results, as shown in Figure 4, reveal a mismatch rate of approximately
$40\%$, showing that the RM has some inability to accurately assign scores
that reflect the true quality of responses generated by the PM. Also, we can
observe a discrepancy between PM and RM - the RM can not well judge the
quality of the responses generated from PM. This discrepancy can introduce
noise into the RL training process, leading to the accumulation of incorrect
gradients during RL optimization. Besides, we show that such discrepancies can
not be resolved by scaling up the model (Appendix B). Consequently, a natural
strategy to enhance the RLHF process is removing instructions from
$\mathcal{D}_{rl}$ that exhibit discrepancies between the RM and PM. This
approach aims to reduce the noise in the RL training procedure, potentially
improving overall model performance.

Figure 5: Compared to the RLHF performance of the full dataset, filter low-
SEAM data further improves RLHF (3 random seeds).

###  4.3 Less Can Be More: A Case Study of Data Selection for RL Training

Based on the insights from section 4.2, we remove instructions that lead to
discrepancies between the PM and RM. We then use this refined dataset for RL
training and compare its performance against that achieved using the full
$\mathcal{D}_{rl}$ dataset. As per the experimental settings described in
section 4.2, we employ both models at rank$=1$ for RL training. The results,
presented in Figure 5, demonstrate a statistically significant improvement in
RLHF performance (p<0.05) after removing data that causes discrepancies
between the PM and RM. This case study illustrates a ‘less is more’ phenomenon
in RL training data: removing data that causes the discrepancy between PM and
RM can enhance overall RLHF performance. However, this selective data
filtering process is challenging to generalize due to its dependence on human
annotation. Currently, there is no formal concept to characterize such data-
driven discrepancies adequately. Consequently, we will discuss these in
section 5.

##  5 SEAM: An Automatic Estimation for Seamlessness

As shown in section 4, removing data that leads to discrepancies between the
PM and the RM improves RLHF performance. Currently, our approach depends on
manual human assessments to determine the alignment between the PM and RM for
specific datasets, a process that hinders full automation. This section first
explores the concept of ‘seamlessness’ in RL training data. Then, we propose
SEAM, an automated method designed to quantify the seamlessness of each data
point, potentially enabling a more efficient and systematic tool to enhance
RLHF training.

###  5.1 Concept of the Seamlessness

Generally, our concept of ‘seamlessness’ is proportional to the PM likelihood
of a data point that causes discrepancies between the policy and the reward
model. Therefore, seamlessness includes not only the probability of
misjudgment by the reward model but also the generative distribution of the
policy model when conditioned on given data. The formal definition of
seamlessness is provided in 1. Considering that it is implausible to iterate
the space of all responses $r$, we provide a discretization form for
seamlessness in Equation 2.

######  Definition 1.

(Definition of Seamlessness) Given an instruction ${I}\in\mathcal{D}_{rl}$, a
reward model $\mathcal{R_{\theta}}$ and a policy model $\pi^{SFT}$. We denote
the distribution of the response $r$ from $\pi^{SFT}$ as
$P_{r}(\cdot|{I},\pi^{SFT})$, we also denote the data distribution that hacks
$\mathcal{R_{\theta}}$ as $P_{h}(\cdot|\mathcal{R_{\theta}})$, which means the
data that leads to reward misjudgement. Then, the seamlessness of the
instruction $I$ is defined as follows:

$\mathcal{S}(I,{R_{\theta}},\pi^{SFT})=\int_{r\sim P_{h}}P_{r}\left(r\mid
I,\pi^{SFT}\right)\cdot\epsilon(r,R_{\theta})\,dP_{h}$ (1)

where $\epsilon(r,R_{\theta})$ denotes the magnitude of RM misjudgement.

Since the term defined in 1 is intractable, we propose SEAM, an estimation for
the seamlessness between RM and PM reflected through data. Following the
notations in 1, we define a sample set $\mathcal{X}$ that contains $N$ samples
$r_{i}\sim P_{h}(\cdot|\mathcal{R_{\theta}})$ to represent the hacking
distribution. Then, we present the discretization form of the seamlessness as
follows:

${{\color[rgb]{0.33984375,0.34375,0.734375}\definecolor[named]{pgfstrokecolor}{rgb}{0.33984375,0.34375,0.734375}\textsc{SEAM}}}{}(I,{R_{\theta}},\pi^{SFT})=\sum_{r_{i}\in\mathcal{X}}P_{r}\left(r_{i}\mid
I,\pi^{SFT}\right)\cdot\epsilon(r_{i},R_{\theta})$ (2)

In fact, our analyses in section 4 use a similar method to Equation 2 to
quantify the seamlessness between PM and RM. But under the formulation in
section 4, the $\epsilon(r_{i},R_{\theta})$ refers to the mismatch degree
between reward and human preferences, which inevitably incorporate the human
efforts.

###  5.2 Automatic Estimation for Seamlessness

A significant practical challenge in our previous method of measuring
seamlessness is the difficulty in automating the process. In this part, we
introduce several automated estimation methods designed to quantify the
seamlessness of data. Specifically, we propose three variants based on their
corresponding designs to construct the sample set $\mathcal{X}$ (Equation 2):
SEAM${}_{\text{Contrast}}$, SEAM${}_{\text{GPT}}$, SEAM${}_{\text{Adv}}$.

#### SEAM${}_{\text{Contrast}}$

In the SEAM${}_{\text{Contrast}}$ method, we implement the ‘Contrast
Instruction’ strategy [36] to automatically construct the sample set
$\mathcal{X}$. Specifically, for each instruction and its golden response pair
$(I,r)$ in the dataset $\mathcal{D}_{rl}$, we retrieve 30 semantically
relevant but distinct instructions $I^{*}$, along with their corresponding
golden responses $r^{*}$, from a large SFT dataset (each pair in this dataset
comprises an instruction and its golden response). We then use $r^{*}$ to form
new pairs, assessing whether the reward model can effectively distinguish
between the quality of the original pair $I\circ r$ and the newly constructed
pair $I\circ r^{*}$. It is guaranteed that the quality of $I\circ r$ is
superior to $I\circ r^{*}$, providing a reliable ground truth for evaluating
RM performance. We define the magnitude of RM misjudgments,
$\epsilon(r_{i},R_{\theta})$, as follows:

$\epsilon(r_{i},R_{\theta})=\text{max}\left\\{R_{\theta}(I\circ
r^{*})-R_{\theta}(I\circ r),0\right\\}$ (3)

#### SEAM${}_{\text{GPT}}$

In the SEAM${}_{\text{GPT}}$ method, we use GPT-4 [1] to construct the sample
set $\mathcal{X}$. Specifically, for each instruction and its golden response
pair $(I,r)$ in the dataset $\mathcal{D}_{rl}$, we prompt GPT-4 to produce
worse-quality responses $r^{*}$. Similarly, we use $r^{*}$ to form new pairs,
assessing whether the reward model can effectively distinguish between the
quality of the original pair $I\circ r$ and the newly constructed pair $I\circ
r^{*}$. We reuse the magnitude defined in Equation 3.

#### SEAM${}_{\text{Adv}}$

In the SEAM${}_{\text{Adv}}$ method, we use the adversarial attack to generate
adversarial sentences that construct the sample set $\mathcal{X}$.
Specifically, for each instruction and its golden response pair $(I,r)$ in the
dataset $\mathcal{D}_{rl}$, we use adversarial attacks [33] to produce
responses $r^{*}$ that hacks the reward model, such that $R_{\theta}(I\circ
r^{*})\textgreater R_{\theta}(I\circ r)$. Similarly, we follow the misjudgment
term defined in Equation 3.

#### Length penalty term

We introduce the operation to remove length bias. This operation targets the
bias introduced by the length of response $r$, primarily affected by the
exponential decrease in probability with increasing sequence length. To
mitigate this, we implement a length normalization operation on the log
probability of the response. This is formally represented as $\frac{\log
P_{r}\left(r_{i}\mid I,\pi^{SFT}\right)}{\operatorname{len}(r_{i})}$, where
$\log P_{r}(r_{i}\mid I,\pi^{SFT})$ denotes the logarithm of the probability
that the policy model assigns to generating the response $r_{i}$ given the
instruction $I$. It is also worth mentioning that, due to the logarithm, the
value of SEAM becomes negative.

##  6 SEAM for RL Training Data Selection

In this section, we employ three SEAM variants as indicators to filter RL
training data and evaluate the corresponding effectiveness.

###  6.1 Experimental setup

Since this is a data-centric experiment, we follow the previous RLHF setup
outlined in Appendix C. For SEAM${}_{\text{Contrast}}$, we utilize SimCSE [8]
as the embedding model to retrieve the top 30 instructions from a
StackExchange dataset containing over 1 million instruction-response pairs,
with cosine similarity values in the interval $[0.8,0.9]$. For
SEAM${}_{\text{GPT}}$, we select GPT-4-0613 to generate 30 lower-quality
responses using the prompt shown in 2. For SEAM${}_{\text{Adv}}$, we employ
TextAttack [23] to perform adversarial attacks on the reward model. For each
instruction, we generate 30 adversarial responses.

For the models, we reuse the policy model and reward model checkpoints from
section 3 to calculate each SEAM variant across the RL dataset. Subsequently,
we filter out 20% of the RL dataset based on the value of each SEAM variant,
respectively. We then compare the RLHF performance using the full and filtered
datasets based on the evaluation paradigm used in section 3. Specifically, we
add a baseline (LLaMa) that uses the perplexity computed by LLaMa2-7B and
filters the high perplexity data.

Figure 6: RLHF performance when using SEAM to filter 20% of the RL dataset
$\mathcal{D}_{rl}$. After filtering out the low-SEAM data, we observe an
improvement in RLHF performance compared to using the full $\mathcal{D}_{rl}$.
The effectiveness of the three SEAM variants is ranked as follows: GPT >
Contrast > Adv. Specifically, we also observe that randomly removing 20% RL
data does not bring statistically significant performance changes.

###  6.2 Results

The results are presented in Figure 6, showcasing performance based on the
top-5 RMs and PMs, where the saturation phenomenon occurs (section 3). The key
observations are as follows:

(1) Training on SEAM-filtered RL data further improves RLHF performance:
Compared to RLHF on the full $\mathcal{D}_{rl}$, conducting RL training on the
filtered $\mathcal{D}_{rl}$ enhances RLHF performance. This finding
empirically validates that data with low SEAM values negatively impacts the RL
training stage in RLHF. Additionally, randomly removing the same amount of RL
training data does not yield benefits, indicating that the effectiveness of
SEAM is not merely due to a reduction in data size.

(2) Training on SEAM${}_{\text{GPT}}$-filtered RL data alleviates the
saturation phenomenon: We observe that as the quality of RM (PM) increases,
conducting RLHF on the data filtered by SEAM${}_{\text{GPT}}$ continues to
improve performance to a certain extent. Compared to the case of full data
training, the saturation phenomenon is mitigated by filtering data with low
SEAM${}_{\text{GPT}}$ values.

In general, the performance of the three SEAM variants is ranked as follows:
GPT > Contrast > Adv. In this section, we analyze the limitations of these
variants through case studies and a straightforward analysis. Under the setup
in Equation 2, a low likelihood indicates that, given the instruction $I$, the
PM is unlikely to generate the response $r^{*}\in\mathcal{X}$, leading to
issues in estimating seamlessness.

SEAM | Attack | Likelihood
---|---|---
SEAM${}_{\text{GPT}}$ | - | -1.81
SEAM${}_{\text{Contrast}}$ | - | -3.07
SEAM${}_{\text{Adv}}$ | GA [45] | -9.32
BA [19] | -9.17
PWWS [33] | -9.87
Table 1: Per-sentence log-likelihood (with length penalty) from the top-ranked
PM (rank 1) for sentences in the sample set $\mathcal{X}$ (Equation 2)
computed using the three estimation variants of SEAM. The sentences created by
SEAM${}_{\text{Adv}}$ exhibit significantly lower likelihoods, indicating
their unnaturalness.

For SEAM${}_{\text{Adv}}$, we found that the adversarial sentences generated
for estimating SEAM have a much lower likelihood in the PM compared to the
other two methods, as shown in Table 1. Compared to the other two variants,
the sentences generated by SEAM${}_{\text{Adv}}$ are significantly less likely
to be sampled from the PM. Although such adversarial sentences can
consistently hack the RM, they do not represent the PM’s natural outputs,
indicating a lack of representativeness. This is because adversarial attacks
tend to introduce non-coherent perturbations to the response $r$,
significantly impacting fluency. We present typical cases in Appendix D. For
SEAM${}_{\text{Contrast}}$, a similar low-likelihood problem exists, although
it is less severe than with SEAM${}_{\text{Adv}}$.

##  7 SEAM for RLHF Model Augmentation

This section demonstrates how SEAM can augment models that target to increase
seamlessness between PM and RM.

###  7.1 Experimental Setup

We maintain our previous RLHF setup and use the same implementation of SEAM.
The key difference between this experiment and the one in section 6 is that,
after computing SEAM for the RL dataset, we augment the PM and RM by adding
the data augmented based on such low-SEAM data points in $\mathcal{D}_{rl}$,
rather than filtering them.

For each SEAM variant, we select the lowest 20% of the data based on their
SEAM scores and generate augmented data to enhance the RM and PM.
Specifically, for each low-SEAM instruction $I_{i}$ and its corresponding
golden response $r_{i}$, we apply the ’Contrast Instruction’ strategy [36] to
create five augmented data samples for each instruction-response pair. These
samples are then added to the training set of the PM. Similarly, we use the
same method for the RM to generate five augmented preference data samples for
each low-SEAM instruction $I_{i}$, which are incorporated into the RM’s
training data. We assess the RLHF performance using the augmented PM and RM.
To ensure a fair comparison, we add two baselines: (1) Random: we randomly
select 20% of $\mathcal{D}_{rl}$ and apply the same augmentation method. The
RLHF performance of the PM and RM augmented by both SEAM and random selection
is then evaluated. (2) Full Aug: For each data sample in $\mathcal{D}_{rl}$,
we apply augmentation methods based on all of them, and add the augmented data
to train PM and RM.

Figure 7: Performance comparison between model augmentation w/ and w/o SEAM.
‘Original’ means RLHF with no model augmentation.

###  7.2 Results

As shown in Figure 7, the results illustrate the effectiveness of using SEAM
to guide model augmentation. Augmenting the PM and RM with data specifically
selected by SEAM demonstrates greater benefits than augmentations using
randomly selected RL data and achieves comparable performance towards Full
Aug. This indicates that the RL data chosen by SEAM is closely related to the
weaknesses of the RM and PM combination during RLHF. Addressing these specific
weaknesses through targeted data augmentation effectively improves the
identified issues. Overall, this validates that SEAM can serve as a signal to
improve RM and PM in terms of their brittleness during RLHF.

##  8 Limitations

One limitation of our framework is that it is restricted to offline RLHF
experiments rather than being tested in an online RLHF scenario. In online
RLHF, the RM and PM are updated continuously based on real-time feedback from
user interactions, offering a more dynamic and realistic setting. Despite
this, we propose that SEAM can still be effectively utilized by segmenting the
online RLHF into a series of offline RLHF cycles. At the beginning of each
cycle, the same analyses for data selection and model augmentation could be
applied. This adaptation would allow us to extend the benefits of SEAM to more
practical, real-world applications. Another limitation of our framework is
inherent in the SEAM metric, which assesses the seamlessness of data only
comparatively rather than absolutely. Consequently, while we can selectively
filter portions of data (e.g., the lowest 20%), we cannot establish a
definitive threshold to categorize data as good or bad outright. However, to
understand the impact of different filtering rates more thoroughly, we have
conducted an analysis detailed in Appendix E, where we see 20% is a practical
choice.

##  9 Conclusion

In this paper, we explored the concept of seamlessness between policy and
reward models within Reinforcement Learning from Human Feedback (RLHF),
uncovering significant discrepancies between the models as reflected in the
data. We introduced SEAM, an automated method to quantify this seamlessness,
demonstrating its practical benefits for improving RLHF outcomes. Our findings
emphasize the critical interplay between policy and reward models,
contributing to a deeper understanding of RLHF dynamics. We hope our insights
will guide future research toward developing more effective and nuanced RLHF
strategies.

## References

  * Achiam et al., [2023] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023).  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774.
  * Askell et al., [2021] Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021).  A general language assistant as a laboratory for alignment.  arXiv preprint arXiv:2112.00861.
  * Bai et al., [2023] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023).  Qwen technical report.  arXiv preprint arXiv:2309.16609.
  * Beeching et al., [2023] Beeching, E., Belkada, Y., Rasul, K., Tunstall, L., von Werra, L., Rajani, N., and Lambert, N. (2023).  Stackllama: An rl fine-tuned llama model for stack exchange question and answering.
  * Carlini et al., [2024] Carlini, N., Nasr, M., Choquette-Choo, C. A., Jagielski, M., Gao, I., Koh, P. W. W., Ippolito, D., Tramer, F., and Schmidt, L. (2024).  Are aligned neural networks adversarially aligned?  Advances in Neural Information Processing Systems, 36.
  * Dong et al., [2023] Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. (2023).  Raft: reward ranked finetuning for generative foundation model alignment.  arXiv preprint arXiv:2304.06767.
  * Gao et al., [2023] Gao, L., Schulman, J., and Hilton, J. (2023).  Scaling laws for reward model overoptimization.  In International Conference on Machine Learning, pages 10835–10866. PMLR.
  * Gao et al., [2021] Gao, T., Yao, X., and Chen, D. (2021).  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * Glaese et al., [2022] Glaese, A., McAleese, N., Trbacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., et al. (2022).  Improving alignment of dialogue agents via targeted human judgements.  arXiv preprint arXiv:2209.14375.
  * Go et al., [2023] Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. (2023).  Aligning language models with preferences through f-divergence minimization.  arXiv preprint arXiv:2302.08215.
  * Hejna III and Sadigh, [2023] Hejna III, D. J. and Sadigh, D. (2023).  Few-shot preference learning for human-in-the-loop RL.  In Conference on Robot Learning (CoRL), pages 2014–2025.
  * Jin et al., [2020] Jin, D., Jin, Z., Zhou, J. T., and Szolovits, P. (2020).  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  In Conference on Artificial Intelligence (AAAI).
  * Keneshloo et al., [2019] Keneshloo, Y., Shi, T., Ramakrishnan, N., and Reddy, C. K. (2019).  Deep reinforcement learning for sequence-to-sequence models.  IEEE transactions on neural networks and learning systems, 31(7):2469–2489.
  * Kingma and Ba, [2014] Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Korbak et al., [2023] Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., and Perez, E. (2023).  Pretraining language models with human preferences.  arXiv preprint arXiv:2302.08582.
  * Kreutzer et al., [2018] Kreutzer, J., Khadivi, S., Matusov, E., and Riezler, S. (2018).  Can neural machine translation be improved with user feedback?  In Proceedings of NAACL-HLT, pages 92–105.
  * Lambert and Calandra, [2023] Lambert, N. and Calandra, R. (2023).  The alignment ceiling: Objective mismatch in reinforcement learning from human feedback.  arXiv preprint arXiv:2311.00168.
  * Li et al., [2016] Li, J., Miller, A. H., Chopra, S., Ranzato, M., and Weston, J. (2016).  Dialogue learning with human-in-the-loop.  In International Conference on Learning Representations.
  * Li et al., [2020] Li, L., Ma, R., Guo, Q., Xue, X., and Qiu, X. (2020).  Bert-attack: Adversarial attack against bert using bert.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202.
  * Li et al., [2018] Li, Z., Jiang, X., Shang, L., and Li, H. (2018).  Paraphrase generation with deep reinforcement learning.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865–3878.
  * Lu et al., [2022] Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. (2022).  Quark: controllable text generation with reinforced unlearning.  Advances in neural information processing systems, 35:27591–27609.
  * Menick et al., [2022] Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022).  Teaching language models to support answers with verified quotes.  arXiv preprint arXiv:2203.11147.
  * Morris et al., [2020] Morris, J., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y. (2020).  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126.
  * Nakano et al., [2021] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021).  WebGPT: Browser-assisted question-answering with human feedback.  arXiv preprint arXiv:2112.09332.
  * Ngo et al., [2022] Ngo, R., Chan, L., and Mindermann, S. (2022).  The alignment problem from a deep learning perspective.  arXiv preprint arXiv:2209.00626.
  * Nguyen et al., [2017] Nguyen, K., Daumé III, H., and Boyd-Graber, J. (2017).  Reinforcement learning for bandit neural machine translation with simulated human feedback.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1464–1474.
  * OpenAI, [2023] OpenAI (2023).  GPT-4 Technical Report.
  * Ouyang et al., [2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022).  Training Language Models to Follow Instructions with Human Feedback.  In Advances in Neural Information Processing Systems (NeurIPS).
  * Pan et al., [2021] Pan, A., Bhatia, K., and Steinhardt, J. (2021).  The effects of reward misspecification: Mapping and mitigating misaligned models.  In International Conference on Learning Representations.
  * Papineni et al., [2002] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002).  BLEU: a method for automatic evaluation of machine translation.  In Annual Meeting of the Association for Computational Linguistics (ACL).
  * Perez et al., [2023] Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2023).  Discovering language model behaviors with model-written evaluations.  In 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023, pages 13387–13434. Association for Computational Linguistics (ACL).
  * Rafailov et al., [2024] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2024).  Direct preference optimization: Your language model is secretly a reward model.  Advances in Neural Information Processing Systems, 36.
  * Ren et al., [2019] Ren, S., Deng, Y., He, K., and Che, W. (2019).  Generating natural language adversarial examples through probability weighted word saliency.  In Annual Meeting of the Association for Computational Linguistics (ACL).
  * Santurkar et al., [2023] Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., and Hashimoto, T. (2023).  Whose opinions do language models reflect?  In International Conference on Machine Learning, pages 29971–30004. PMLR.
  * Schulman et al., [2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017).  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347.
  * Shen et al., [2023] Shen, L., Chen, S., Song, L., Jin, L., Peng, B., Mi, H., Khashabi, D., and Yu, D. (2023).  The trickle-down impact of reward inconsistency on rlhf.  In The Twelfth International Conference on Learning Representations.
  * Shen et al., [2022] Shen, L., Li, S., and Chen, Y. (2022).  KATG: keyword-bias-aware adversarial text generation for text classification.  In Conference on Artificial Intelligence (AAAI).
  * Shen et al., [2024] Shen, L., Tan, W., Chen, S., Chen, Y., Zhang, J., Xu, H., Zheng, B., Koehn, P., and Khashabi, D. (2024).  The language barrier: Dissecting safety challenges of llms in multilingual contexts.  arXiv preprint arXiv:2401.13136.
  * Shi et al., [2018] Shi, Z., Chen, X., Qiu, X., and Huang, X. (2018).  Toward diverse text generation with inverse reinforcement learning.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4361–4367.
  * Singhal et al., [2023] Singhal, P., Goyal, T., Xu, J., and Durrett, G. (2023).  A long way to go: Investigating length correlations in rlhf.  arXiv preprint arXiv:2310.03716.
  * Skalse et al., [2022] Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D. (2022).  Defining and characterizing reward gaming.  Advances in Neural Information Processing Systems, 35:9460–9471.
  * Sokolov et al., [2016] Sokolov, A., Riezler, S., and Urvoy, T. (2016).  Bandit structured prediction for learning from partial feedback in statistical machine translation.  arXiv preprint arXiv:1601.04468.
  * Stiennon et al., [2020] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. (2020).  Learning to summarize with human feedback.  Advances in Neural Information Processing Systems, 33:3008–3021.
  * Touvron et al., [2023] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023).  LLaMA: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971.
  * Wang et al., [2019] Wang, X., Jin, H., and He, K. (2019).  Natural language adversarial attack and defense in word level.
  * Yi et al., [2019] Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., and Hakkani-Tur, D. (2019).  Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators.  In Proceedings of the 12th International Conference on Natural Language Generation, pages 65–75.
  * Zheng et al., [2023] Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023).  Judging llm-as-a-judge with mt-bench and chatbot arena.  arXiv preprint arXiv:2306.05685.
  * Zhou et al., [2024] Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2024).  Lima: Less is more for alignment.  Advances in Neural Information Processing Systems, 36.
  * Zhu et al., [2023] Zhu, B., Jiao, J., and Jordan, M. (2023).  Principled reinforcement learning with human feedback from pairwise or $k$-wise comparisons.  In ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models.
  * Ziegler et al., [2019] Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019).  Fine-tuning language models from human preferences.  arXiv preprint arXiv:1909.08593.
  * Ziems et al., [2024] Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., and Yang, D. (2024).  Can large language models transform computational social science?  Computational Linguistics, pages 1–55.

##  Appendix A Preliminaries: A three-stage paradigm for RLHF

A RLHF practice includes three stages: policy modeling, reward modeling, and
RL training, which involve three benchmarks: an SFT dataset $\mathcal{D}_{p}$,
a preference benchmark $\mathcal{D}_{r}$, and an RL dataset
$\mathcal{D}_{rl}$.

#### Policy model.

Following the setup [28], we obtain the policy model (PM) by supervised fine-
tuning (SFT) the base version of LLM. Given an SFT dataset $\mathcal{D}_{p}$,
each instance in the dataset consists of an instruction and its golden
response. Then, we train the LLM on $\mathcal{D}_{p}$ with language modeling
loss to obtain the PM: $\pi^{\mathrm{SFT}}$.

#### Reward model.

Following the conventional setup [28], we are given a dataset of human
preferences $\mathcal{D}_{r}$. Each instance in this dataset
${(I_{i},r_{i}^{+},r_{i}^{-})}$ is comprised of an instruction prompt $I_{i}$,
a pair of responses $r_{i}^{+},r_{i}^{-}$ where $r_{i}^{+}$ is preferred over
$r_{i}^{-}$ by humans. On this labeled data, RM $\mathcal{R}_{\theta}$ is
trained to assign a higher scalar reward to human-preferred $r_{i}^{+}$ over
non-preferred $r_{i}^{-}$ in the context of $I_{i}$. This can be achieved by
minimizing the ranking loss $\mathcal{L}$, where $\sigma$ is the sigmoid
function and ${I}_{i}\circ r_{i}^{+}$ is the concatenation of $I_{i}$ and
$r_{i}^{+}$.

$\mathcal{L}(\theta)=-\mathbb{E}_{\left({I_{i}},r_{i}^{+},r_{i}^{-}\right)\sim\mathcal{D}_{h}}\left[\log\left(\sigma\left(\mathcal{R}_{\theta}({I_{i}}\circ
r_{i}^{+}\right)-\mathcal{R}_{\theta}\left({I_{i}}\circ
r_{i}^{-})\right)\right)\right].$ (4)

#### Reinforcement Learning.

The last stage of RLHF is reinforcement learning. Specifically, a per-token KL
penalty from the SFT model at each token is used to mitigate over-optimization
of the reward model, and the value function is initialized from the RM. We
maximize the following combined objective function $\mathcal{J}(\phi)$ in RL
training based on PPO algorithm [35, 28], RL training dataset
$\mathcal{D}_{rl}$ and pre-training dataset $\mathcal{D}_{\text{pre}}$:

$\mathcal{J}(\phi)=\mathbb{E}_{(I,r)\sim\mathcal{D}_{\pi_{\phi}^{\mathrm{RL}}}}\left[\mathcal{R}_{\theta}(I\circ
r)-\beta\log\left(\pi_{\phi}^{\mathrm{RL}}(r\mid I)/\pi^{\mathrm{SFT}}(r\mid
I)\right)\right]$

where $\pi_{\phi}^{\mathrm{RL}}$ is the learned RL policy parameterized by
$\phi$ initialized from a pretrained supervised trained model
$\pi^{\mathrm{SFT}}$. The first term encourages the policy
$\pi_{\phi}^{\mathrm{RL}}$ to generate responses that have higher reward
scores. The second term represents a per-token KL reward controlled by
coefficient $\beta$ between $\pi_{\phi}^{\mathrm{RL}}$ and
$\pi^{\mathrm{SFT}}$ to mitigate over-optimization toward the reward.

##  Appendix B The discrepancy does not vanish as scaling up

Model | Match Rate | PM performance | RM performance
---|---|---|---
LLaMa2-7B | 60.5% | 66.1 | 5.24
LLaMa2-13B | 60.7% | 66.9 | 5.30
LLaMa2-70B | 60.4% | 67.6 | 5.35
Table 2: The scaling tendency of our base model for training PM and RM, from
7B to 70B. We observe that the performance of PM and RM improves as the model
scales up but find the match rate toward human preference remains nearly the
same.

As demonstrated in section 4.2, there is a notable discrepancy between the PM
and RM: the RM fails to appropriately assign reward scores to responses
generated by the PM. In this section, we explore the impact of scaling the
base model on these discrepancies by reanalyzing the data discussed in section
4.2. The findings, presented in Table 2, reveal that while the capacities of
the PM and RM improve with an increase in the size of the base model (LLaMa2),
the preference matching rate remains nearly consistent across different model
scales. These results confirm that merely scaling up the model size does not
address the underlying discrepancy between the RM and PM.

##  Appendix C Implementation details of RLHF

###  C.1 Training details

• Standard fine-tuning (SFT): The base model chosen is LLaMa2-7B. We created
10 PMs of increasing quality by varying the training data amounts at 50, 100,
250, 500, 800, 1500, 2500, 5000, and 10000, plus a baseline pretrained model
without SFT. The configuration employed includes the AdamW [14] optimizer with
a learning rate of 1e-4, 10 warmup steps, and training facilitated by LoRA.

• Reward model (RM): Training of the RM utilized the SFT model as the base
model. Depending on the SFT model’s quality rank, StackExchange pairwise
preference data of subset 50, 100, 500, 2500, 5000, 10000, 20000, 50000, and
100000 were employed to train 9 RMs. With an additional pretrained model
replaced with a randomly initialized classifier head, in total we create 10
RMs with increasing accuracy. Training employed LoRA, with AdamW optimizer and
learning rate 2e-5.

• Reinforcement learning with PPO: PPO is used for each PM-RM pairing,
generating hundreds of unique RLHF models. The RL prompts are from the
StackExchange question dataset and remain consistent across all RLHF
implementations. The SFT model served as the reference model, utilizing the
reward scores from the RM as supervision. All PPO training has the
configuration of LoRA with a learning rate of 1.4e-5, a batch size of 32, and
200 PPO steps.

######  Prompt 1.

(Prompt used in RLHF/PM evaluation)

[System]

Please act as an impartial judge and evaluate the quality of the response
provided by an AI assistant to the user question displayed below. Your
evaluation should consider factors such as the helpfulness, relevance,
accuracy, depth, creativity, and level of detail of the response. Begin your
evaluation by providing a short explanation. Be as objective as possible.
After providing your explanation, please rate the response on a scale of 1 to
10 by strictly following this format: "[[rating]]", for example: "Rating:
[[5]]".

[Question]

{question}

[The Start of Assistant’s Answer]

{answer}

[The End of Assistant’s Answer]

###  C.2 Evaluation details

For the evaluation details, we detail the setup of the generator (i.e., PM and
RLHF model) and classifier (i.e., RM), respectively.

  * •

Reward model: the reward model is evaluated on the corresponding test split of
the preference benchmark based on accuracy (i.e., whether the RM can
distinguish the better and worse response in the context of the given
instruction.)

  * •

Policy and RLHF model: we follow the general principle of MT-Bench [47].
Specifically, we use their instruction (1) to prompt GPT-4 for measuring the
quality of the responses from the policy and RLHF models. GPT-4 will assign a
quality score, ranging from 0 to 10, to measure the quality of the response.

###  C.3 Sanity check setup

In the sanity check for the capacity of the RM and PM, our primary objective
is to verify that both models maintain comparable performance across different
stages of the training process. Specifically, we aim to ensure that: (1) the
RM consistently distinguishes between better and worse responses as per the
instructions used in SFT and RL training; (2) the PM sustains its generation
quality with instructions from the RL training dataset.

To achieve this, we utilize the Stack-Exchange dataset’s three segments (SFT,
Preference, RL), dividing each into train, dev, and test splits. For the RM,
the data distribution is 100,000/20,000/20,000, and for the PM, it is
20,000/2,000/2,000. We prepare the dataset in a format where each instruction
is paired with a corresponding high-quality answer and a lower-quality
candidate, ensuring the data’s compatibility for training both the RM and PM.
The training configurations adhere to the setup described in Appendix C.

##  Appendix D Implementation details of SEAM

###  D.1 Prompt used in SEAM${}_{\text{GPT}}$

We use GPT-4 to generate worse-quality responses in SEAM${}_{\text{GPT}}$,
based on the prompt detailed in 2.

######  Prompt 2.

(Prompt used in SEAM${}_{\text{GPT}}$)

[System]

Using the question and its correct answer provided below, generate 30 distinct
answers that are of lower quality. Each response should include one or more of
the following characteristics: factual inaccuracies, misunderstandings of the
core question, irrelevant information, or grammatical errors. The answers
should vary in their mistakes to cover a range of common errors seen in
similar topics. Format the responses as separate paragraphs for each answer.

[Question]

{question}

[Answer]

{answer}

[The Start of Assistant’s Answer]

{answer}

[The End of Assistant’s Answer]

###  D.2 Cases of SEAM${}_{\text{Adv}}$

We employed several adversarial attack strategies to challenge the integrity
of the reward model (RM). Specifically, for each instruction along with its
corresponding better response $r^{+}$ and worse response $r^{-}$, these
adversarial attacks introduce a perturbation $\alpha$ to $r^{-}$. The goal is
for $r^{-}+\alpha$ to receive a higher reward score than $r^{+}$, thereby
compromising the RM. The attacks we utilized include GA [45], Bert-Attack
[19], PWWS [33], KATG [37], and TextFooler [12]. However, a common limitation
of these methods is that they tend to produce sentences with extremely low
likelihood according to the policy model. Below, we present some examples
illustrating the discrepancies between the original responses and those
generated by the adversarial attacks.

###  D.3 Setup of SEAM${}_{\text{Contrast}}$

Using a human preference dataset, we have divided it into training,
development, and testing sets. The reward model is trained on the training set
and ceases training once it attains optimal performance on the development
set. Subsequently, it is evaluated on the test set. Our Contrast Instructions
are built upon the test set in each benchmark. We establish a similarity
threshold range to ensure the retrieved instruction differs from the original
one ($[0.8,0.9]$). Only instructions falling within this similarity range are
retrieved.

###  D.4 Human evaluation

Since we aim to compute the degree of match between the reward outputs and
human preferences, we enlist multiple human annotators to assess the quality
of responses to Stack Exchange questions. Each annotator is kept unaware of
the model that generated the responses, and then they are asked to give the
index of the response with better quality based on tools like search engines.
Since the evaluation relates to Stack Exchange, each annotator has expertise
in computer science.

##  Appendix E Extra Analysis of low-SEAM data

###  E.1 The effects of the filtering rate

We vary the filter rate as follows $\\{10\%,20\%,30\%,40\%,60\%,80\%\\}$, and
re-conduct the experiments in section 6 with the rank 1 PM and RM. The
results, as shown in Figure 8, demonstrate the relationship between the filter
rate of data samples and the in-domain RLHF performance across various
thresholds. Notably, increasing the filter rate initially enhances RLHF
performance, with a peak observed at approximately 40%. Beyond this threshold,
further increases in the filter rate result in a gradual decline in
performance. This trend indicates an optimal range for filtering out low-seam
score samples to maximize RLHF effectiveness, thereby illustrating the
critical trade-off between data quantity and quality. Based on this
observation, we set the filtering rate as $20\%$.

Figure 8: The effects of filter rate in RL data selection.

###  E.2 The overlap rate between low-SEAM data on different combinations.

Following the previous setup, we examine the overlap rate of the 20% low-SEAM
data across three model combinations: (1) rank 5 PM with rank 5 RM, (2) rank 3
PM with rank 3 RM, and (3) rank 1 PM with rank 1 RM. We aim to assess whether
the low-SEAM data varies significantly among different model pairings. The
results, illustrated in Table 3, reveal that the overlap rate between model
combinations is generally high, exceeding 60%. Notably, the overlap rate
increases as the differences between the models decrease.

Model Combo | rank = 1 | rank = 3 | rank = 5
---|---|---|---
rank = 1 | - |  |
rank = 3 | $72\%$ | - |
rank = 5 | $64\%$ | $69\%$ | -
Table 3: The overlap rate between the 20% low-SEAM data on different model
combinations, where a rank of 1 denotes using the rank 1 PM and rank 1 RM in
the combination.

##  Appendix F Broader Impact

Improved Human Model Alignment: Integrating SEAM into RLHF techniques enhances
the alignment between machine outputs and human values, leading to AI systems
that are more ethical and responsive to user needs. This improvement is
critical for increasing trust and encouraging the adoption of AI technologies
across diverse sectors.

Increased Efficiency and Accessibility: Refining interactions between policy
and reward models optimizes the training processes and reduces the
computational resources required, making AI technologies more accessible and
affordable. This democratization of AI could lead to broader innovation and
application.

Misuse in Content Generation: The enhancements that improve model quality and
user experience can also be exploited to create misleading information. Such
misuse may pose risks of spreading misinformation and violating privacy.

Figure 9: Case comparisons between the original and adversarial responses
generated by text attacks. The differences are highlighted in RED.