# Reward Model Ensembles Help Mitigate Overoptimization

Thomas Coste ➀, Usman Anwar ➀, Robert Kirk ➁, David Krueger ➀
➀University of Cambridge, ➁University College London
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Reinforcement learning from human feedback (RLHF) is a standard approach for
fine-tuning large language models to follow instructions. As part of this
process, learned reward models are used to approximately model human
preferences. However, as imperfect representations of the “true” reward, these
learned reward models are susceptible to overoptimization. Gao et al. (2023)
studied this phenomenon in a synthetic human feedback setup with a
significantly larger “gold” reward model acting as the true reward (instead of
humans) and showed that overoptimization remains a persistent problem
regardless of the size of the proxy reward model and training data used. Using
a similar setup, we conduct a systematic study to evaluate the efficacy of
using ensemble-based conservative optimization objectives, specifically worst-
case optimization (WCO) and uncertainty-weighted optimization (UWO), for
mitigating reward model overoptimization when using two optimization methods:
(a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We
additionally extend the setup of Gao et al. (2023) to include $25\%$ label
noise to better mirror real-world conditions. Both with and without label
noise, we find that conservative optimization practically eliminates
overoptimization and improves performance by up to $70\%$ for BoN sampling.
For PPO, ensemble-based conservative optimization always reduces
overoptimization and outperforms single reward model optimization. Moreover,
combining it with a small KL penalty successfully prevents overoptimization at
no performance cost. Overall, our results demonstrate that ensemble-based
conservative optimization can effectively counter overoptimization.

##  1 Introduction

With the advent of large language models, reinforcement learning from human
feedback (RLHF) has emerged as a powerful technique to fine-tune and enhance
models’ behaviors (Ziegler et al., 2019; Ouyang et al., 2022; Bai et al.,
2022a). However, despite its empirical success, RLHF remains a fickle method
suffering from many failure modes (Casper et al., 2023). One such failure mode
is overoptimization, a phenomenon in which policy optimization appears to be
making progress according to the learned reward model, but in reality begins
to regress with respect to the true reward function (Ziegler et al., 2019;
Stiennon et al., 2020; Gao et al., 2023). While many works on RLHF contain
anecdotal evidence of overoptimization (Ziegler et al., 2019; Stiennon et al.,
2020; Dubois et al., 2023), Gao et al. (2023) is the only work that studies
overoptimization in a systematic way. As working directly with human labelers
is expensive, Gao et al. (2023) introduce a synthetic setup to study
overoptimization in which a much larger language model is first trained as a
“gold” reward model and is then used to generate preference labels for
training of proxy reward models.

In this work, we conduct a systematic study investigating whether combining
ensembles with conservative optimization can help mitigate overoptimization.
Our results indicate that not only does ensemble-based conservative
optimization help mitigate overoptimization, it also results in improved
performance. Our setup is identical to that of Gao et al. (2023) with one
modification: the addition of label noise. Gao et al. assume that the
preference labels used to train the proxy reward model do not contain any
noise. However, this does not mirror the real-world RLHF setup, in which
agreement rates among human annotators are typically between $60-75\%$
(Ziegler et al., 2019; Stiennon et al., 2020; Dubois et al., 2023). To
simulate that disagreement and to better reflect the real-world RLHF, we
extend the setup to include $25\%$ label noise as well. In both the cases of
no label noise and $25\%$ label noise, we provide strong evidence that
ensemble-based conservative optimization methods are effective in mitigating
overoptimization and improving performance.

Scaling laws for reward model overoptimization discovered by Gao et al. (2023)
indicate that increasing the size of the proxy reward model reduces
overoptimization as well. However, reward models are derived from pretrained
language models. Thus, acquiring a larger reward model would require
significant pretraining, which is not always feasible and can be very costly
(Morgan, 2022; Venigalla & Li, 2022). However, our approach, using ensembles
of reward models, only requires fine-tuning multiple copies of an already
pretrained reward model, which is relatively inexpensive. Moreover, our model
and data scaling results (Figures 8 and 9) indicate that the gains provided by
our method are orthogonal to the gains achieved by increasing the reward model
size; thus, the two approaches can be combined seamlessly for even better
results.

Our main contributions are as follows:

  * •

We conduct the first study of using ensembles to counter overoptimization in
RLHF-based fine-tuning of language models.

  * •

Our results indicate that using ensembles and conservative optimization
eliminates overoptimization for BoN and results in up to $70\%$ improvement in
some cases.

  * •

For PPO, ensemble-based conservative optimization typically outperforms single
reward model optimization, and when combined with a suitable KL penalty weight
successfully eliminates overoptimization.

  * •

We further conduct studies to establish the robustness of the ensemble-based
conservative optimization methods to any new hyperparameters it introduces
(e.g. size of the ensemble).

Figure 1: RLHF pipeline used in this work - our modifications on top of the
standard RLHF setup used in Gao et al. (2023) are highlighted in green.

##  2 Background

In this section, we briefly review two commonly used policy optimization
methods: best-of-n sampling (BoN) and proximal policy optimization (PPO),
followed by a discussion of overoptimization.

###  2.1 Best-of-n Sampling (BoN)

Best-of-$n$ (BoN) sampling, also called rejection sampling, is a simple
inference-time optimization method (Ouyang et al., 2022; Nakano et al., 2021).
For a given prompt, $n$ responses are generated from the policy model, and the
answer with the highest proxy reward model score is returned. To evaluate the
degree of optimization, the KL distance is defined analytically as a function
of $n$:

$\text{KL}_{\text{bon}}=\text{log}\,n-\frac{n-1}{n}$ (1)

###  2.2 Proximal Policy Optimization (PPO)

Proximal Policy Optimization (Schulman et al., 2017) is a policy-gradient-
based online reinforcement learning method that maximizes a given reward
function by repeatedly performing small incremental updates to the policy. PPO
is the standard algorithm used in fine-tuning language models based on human
feedback (Ouyang et al., 2022; Bai et al., 2022a; Stiennon et al., 2020; Zheng
et al., 2023). When using PPO to fine-tune a language model, a KL penalty term
is added during the reward calculation to regularize the policy by preventing
it from deviating far from the initial policy:

$\displaystyle
R^{\text{PPO}}(q,a)=R(q,a)-\beta\log\left[\frac{\pi^{\text{PPO}}(a|q)}{\pi^{\text{init}}(a|q)}\right]$
(2)

where $\pi^{\text{PPO}}$ is the policy being optimized and $\pi^{\text{init}}$
is the initial (pretrained) language model.

###  2.3 Overoptimization

In reinforcement learning from human feedback (RLHF) a reward model is used to
approximate human preferences, to remove the need for querying humans for
every policy generation. As the learned reward model is only a proxy for the
true reward function, optimizing it may not always result in an improvement
according to true human preferences. In practice, optimizing a (fixed) learned
reward model almost always leads to improvement according to this learned
reward model, but only improves according to the true reward model (i.e.
humans) for some initial period, after which performance often begins to
regress. This phenomenon is referred to as overoptimization, an example of
which is shown in Figure 2.

Figure 2: An example of overoptimization.

To study the problem of overoptimization, Gao et al. (2023) introduced a
synthetic setup in which, instead of human annotators, a gold reward model is
used to score responses and generate preferences to train proxy reward models.
The gold reward model is generally chosen to be much larger than the proxy
reward model, to simulate the fact that in a real setup, human preferences are
too complex to be captured by a neural network with a finite capacity. Within
this setup, Gao et al. (2023) discover overoptimization to be a persistent
issue, although they note that larger proxy reward model sizes and greater
amounts of training data can help reduce overoptimization. However, we note
that scaling up is not always a feasible solution as the proxy reward models
are generally derived from extensively pretrained language models.

##  3 Method

Standard RLHF often learns a single reward model to estimate the true reward,
which is then used to optimize the policy. However, many works in wider
machine learning have shown that learning multiple estimators and combining
them can help improve robustness (Lakshminarayanan et al., 2017; Ovadia et
al., 2019; Yu et al., 2020b). Taking inspiration from this insight, we propose
to learn an ensemble of reward models $\\{R_{1},...,R_{k}\\}$ in the reward
model training stage. During policy optimization, we combine the reward
estimates from different reward models within the ensemble according to the
following three methods.

Mean Optimization: Mean optimization (Boyd & Vandenberghe, 2004; Wright, 2006)
simply takes the mean of the outputs of the different ensemble members:

$R_{\mu}(q,a):=\frac{1}{k}\sum_{i=1}^{k}R_{i}(q,a)$ (3)

where $q$ is the prompt given to language model and $a$ is the corresponding
response sampled from the language model.

We note that mean optimization is not a conservative optimization technique;
if at least one member of the ensemble overestimates the reward (even while
other members accurately estimate it), mean optimization will not prevent the
policy from exploiting the faulty reward model.

Worst-Case Optimization: Worst-case optimization (WCO) (Boyd & Vandenberghe,
2004; Wright, 2006) creates a conservative estimate by choosing the lowest
reward from the ensemble at every step. Choosing the lowest reward from the
ensemble helps ensure that as long as at least one ensemble member does not
overestimate the true reward, policy optimization will not result in
overoptimization.

$R_{\text{WCO}}(q,a):=\min_{i}R_{i}(q,a)$ (4)

A major advantage of WCO is that it does not have any hyperparameters that
might require tuning. However, it can sometimes result in a performance
penalty due to its highly conservative nature.

Uncertainty-Weighted Optimization: In uncertainty-weighted optimization (UWO)
(Wu et al., 2021b; Yu et al., 2020b; Brantley et al., 2019), the reward for a
sample is calculated by combining the average reward across all models in an
ensemble with the intra-ensemble variance, weighted by a coefficient
$\lambda$. Intuitively, UWO works by penalizing the policy for generating
responses for which there is high disagreement among reward models within the
ensemble. This helps prevent the exploitation of a single faulty reward model
which might be erroneously assigning high rewards to incorrect or low-quality
responses. Mathematically, this objective is given by:

$R_{\text{UWO}}(q,a):=\underbrace{\frac{1}{k}\sum_{i}R_{i}(q,a)}_{\text{mean}}-\lambda\underbrace{\frac{1}{k}\sum_{i}\left(R_{i}(q,a)-\frac{1}{k}\sum_{i}R_{i}(q,a)\right)^{2}}_{\text{variance}}$
(5)

where $\lambda$ is a hyperparameter which controls the weight of the
uncertainty component.

##  4 Experimental Setup

We based our experiments on the setup proposed by Gao et al. (2023), however,
we use open source models (Biderman et al., 2023; Dubois et al., 2023), open
source datasets (Taori et al., 2023), and train our proxy reward models under
$25\%$ label noise. We also present qualitative reproduction of the results of
Gao et al. in Appendix E.

###  4.1 Data

In order to train proxy reward models, we use the Alpaca dataset (Taori et
al., 2023), which contains $52,000$ instructions covering a range of commands
and corresponding demonstrations generated by OpenAI’s text-davinci-003
(OpenAI, 2023a). Each entry is composed of an instruction, an optional
additional input, and a demonstrator output. More specifically, we use the
AlpacaFarm (Dubois et al., 2023) variant of this dataset, as it provides
splits for use in the different RLHF stages and for validation, as well as
human preference annotations. More details concerning the format used and
examples can be found in Appendix D.2.

###  4.2 Pretrained Models

For the policy model and (proxy) reward model, we use pretrained language
models provided in the Pythia suite (Biderman et al., 2023). The policy model
used everywhere in this work is the 1.4B Pythia model. For proxy reward
models, we remove the unembedding layers from Pythia models of sizes 14M, 70M,
and 1.4B and add a scalar head to output the reward to get proxy reward models
of sizes 7M, 44M, and 1.3B.

Finally, the 7B human preference reward model from AlpacaFarm (Dubois et al.,
2023) is used as the gold reward model. It is of very similar size to the
(closed-source) 6.9B gold reward model used in Gao et al. (2023) and is
significantly larger than any of our proxy RMs (with the largest being 1.3B)
and as such is a reasonable choice for a gold standard.

###  4.3 RLHF Pipeline

Our RLHF pipeline is similar to that of Gao et al. (2023) with several
modifications highlighted in Figure 1. We explain the full setup below for
completeness.

Supervised Fine-tuning: As the first step in our RLHF pipeline, both the
policy model and the proxy reward model undergo supervised fine-tuning using
10k instruction demonstrations from the “sft” split of the AlpacaFarm dataset
(see Section 4.1 for details). This fine-tunes the models to have better
instruction-following capabilities.

Preference Generation and Labeling: To generate a preference dataset, the SFT
model is prompted using instructions from the AlpacaFarm dataset, for which it
produces two responses per instruction. Relevant hyperparameters for sampling
these responses are given in Appendix D.1. The two responses are then scored
with the gold reward model, which assigns a score to each of them. Human
annotators tend to have high disagreement rates, often around $25\%$ (Stiennon
et al., 2020) or more (Ziegler et al., 2019; Dubois et al., 2023). To simulate
this, we optionally mislabel $25\%$ of the dataset. This results in two
datasets: one with no label noise and one with $25\%$ label noise.

Proxy Reward Model Training: We train our proxy reward model by minimizing
cross-entropy loss on the preference dataset generated in the previous
timestep. Unless mentioned otherwise, we use the complete dataset of $46,000$
prompts for reward model training and train all the reward models for $5$
epochs. We give other hyperparameters for reward model training in Appendix
D.1 and example validation loss curves for a set of reward models in Appendix
F.1.

Ensemble Creation: To create an ensemble, we train a fixed number of proxy
reward models using identical data and hyperparameters but with different
random seeds. This results in different random initialization for the scalar
reward head added on top of the pretrained language model and a different data
shuffling order. We use all the available training data for the training of
every reward model as training on lesser data results in higher validation
loss and poorer performance (Lakshminarayanan et al., 2017). Unless stated
otherwise, we train an ensemble consisting of five reward models.

Policy Optimization: Similar to Gao et al. (2023), we use BoN sampling and PPO
as optimization algorithms. For BoN, the evaluation cost for greater KL nats
increases exponentially. As a result, due to constraints on available compute,
we only evaluate BoN for a maximum of $n_{max}=12,500$ samples111Generating
the $n=12,500$ answers for 1000 prompts and then relabeling them with proxy
and gold reward model takes approximately 700 A100 GPU hours., which roughly
equals $8.4$ nats of KL. For PPO, we train for $3000$ PPO steps. We give
further details on implementation and other hyperparameters in Appendix D.1.

##  5 Results

In this section, we report the results of our experiments. To summarize, our
main findings are:

  * •

Using an ensemble, with any of the objectives highlighted in Section 3, almost
always helps over using a single reward model.

  * •

For BoN, we do not observe any overoptimization when using WCO and UWO as
optimization methods; however, in the $25\%$ label noise case, mean
optimization does overoptimize (Figure 3).

  * •

For BoN, all three ensemble-based optimization methods result in up to $\sim
30\%$ improvement in final performance over the average performance attained
by optimizing individual reward models in the no label noise case and up to
$\sim 75\%$ improvement in the $25\%$ label noise case (Figure 3).

  * •

For PPO, WCO, and UWO do reduce overoptimization (Figure 5), but completely
mitigating overoptimization requires combining them with a small KL penalty
(Figure 4).

  * •

For PPO, WCO, and UWO always match or outperform single reward model
optimization in terms of final performance for all values of KL penalties
(Figure 7).

  * •

Our findings are robust to scaling of model size and training dataset size
(Figures 8 and 9).

(a) With no label noise.

(b) With $25\%$ label noise.

Figure 3: BoN results for 44M reward model size.

(a) With no label noise. For all methods,
KL penalty of $0.01$ is used here.

(b) With $25\%$ label noise. KL penalty $0.1$ for mean and $0.01$ for all
other methods is used.

Figure 4: PPO results for $44$M reward model size. For each method, we choose
the KL penalty that gives the best final performance.

To present our results, we primarily rely on policy optimization curves where
we show the gold reward model score on the left y-axis and the proxy reward
model score on the right y-axis. For the x-axis, following the common practice
(Bai et al., 2022a; Gao et al., 2023), we use the square root of KL divergence
$\sqrt{D_{KL}(\pi|\pi_{0})}$ where $\pi$ is the current policy and $\pi_{0}$
is the initial policy. Additionally, when comparing many different experiment
results at once, we present bar plots showing the final performance on the
gold reward model and the variable of interest on the x-axis. Finally, we also
include some qualitative samples for different methods of interest in Appendix
F.5.

###  5.1 Best-of-N Sampling

In Figure 3, we present results for BoN sampling for a $44$M size reward
model. Across both noiseless and noisy settings, ensembles help improve
performance by $\sim 30\%$ and $\sim 75\%$ respectively and successfully avoid
overoptimization, except for mean optimization in the case of noisy labels.
For UWO, we show results for $\lambda=0.5$ which we found to be most
performant; however, in Section 5.4, we show that many reasonable values of
$\lambda$ work well. Additional results for different sizes of reward models
are presented in Appendices F.3 and F.4.

###  5.2 Proximal Policy Optimization

For PPO we observe that UWO and WCO do reduce overoptimization and outperform
other methods in terms of the final performance in the absence of KL penalty,
although they do not eliminate overoptimization completely (as shown in Figure
5). However, in Figure 4, we show that with a small KL penalty coefficient of
$0.01$, WCO and UWO both successfully prevent overoptimization, without any
notable penalty in terms of performance. On the other hand, when using KL
penalty on its own, a $20$x larger KL penalty weight of $0.2$ is required to
eliminate overoptimization which incurs a significant performance penalty
(Figure 7). In Figure 7, we show that for all KL values, UWO and WCO match or
outperform the average final performance achieved by optimizing a single
reward model. Moreover, for small KL penalties, they comprehensively
outperform single reward model optimization.

(a) With no label noise.

(b) With $25\%$ label noise.

Figure 5: PPO results for different optimization objectives without using a KL
penalty.

Figure 6: Individual reward model optimization (i.e. no ensemble) for
different values of KL penalty in the $25\%$ label noise case.

Figure 7: Effectiveness of conservative optimization methods across KL penalty
weights in the $25\%$ label noise case.

###  5.3 Model and Data Scaling Results

Prior work (Gao et al., 2023) has shown that increasing reward model size or
training dataset size helps improve performance as measured by the gold reward
model. In Figure 8, we vary the size of reward models (keeping the dataset
size fixed at $46$k samples and policy size fixed at $1.4$B) and plot the
final performance of each method for the fixed training budget. Recall that
for BoN, this is $n=12,500$ samples and for PPO, this is $3000$ timesteps. Our
results show that the improvement due to the use of ensembles is orthogonal to
the improvement in performance achieved by scaling the reward model size. In
particular, with the 1.3B reward model, we highlight that even when the policy
and reward model have a similar number of parameters, WCO and UWO still
provide non-trivial gains over using a single reward model. A more detailed
illustration of this example can be found in Appendix F.4.

Similarly, in Figure 9, we vary the size of the training dataset (keeping the
reward model size fixed at $44$M) and plot the final performance for each
method for the fixed training budget. The results again indicate that using
ensembles provides additional improvement in addition to increasing the size
of the training dataset.

(a) BoN

(b) PPO

Figure 8: Final gold reward model performance achieved by different objectives
when optimizing reward models with varying parameter sizes, but trained with
the same dataset containing $25\%$ label noise.

(a) BoN

(b) PPO

Figure 9: Final gold reward model performance achieved by different objectives
when optimizing (44M) reward models trained under varying amounts of data. A
label noise ratio of $25\%$ was maintained for all dataset sizes.

###  5.4 Evaluating Robustness to Hyperparameters

Using an ensemble-based approach introduces an additional hyperparameter: the
cardinality, or size, of the ensemble. In Figure 11, we plot the final
performance for BoN and PPO for the three ensemble-based approaches: mean
optimization, WCO, and UWO. We note that while there is a noticeable gap
between between $3$ member ensemble and $4$ member ensemble, in most cases the
performance is highly similar for the cases of $4$ member ensemble and $5$
member ensemble, indicating that $4$ or $5$ member ensembles are likely to
work best, and diminishing returns will be seen after this point.

For UWO, the value of the uncertainty penalty is another hyperparameter. Our
results in Figure 12 indicate that most reasonable values of uncertainty
penalty actually work well, indicating that there is potentially no need to
tune this hyperparameter.

###  5.5 Effects of Label Noise and Uncertainty Penalty

Figure 10: Intra ensemble variance for different optimization objectives under
different levels of noise for PPO.

In Figure 10, we show intra-ensemble variance for an ensemble of $44$M
parameter reward models optimized via UWO and mean optimization objectives
using PPO. The variance among the ensemble members starts at a small value in
the no label noise case, and increases by a relatively small amount during
training for UWO. However, the uncertainty increases by almost $3$ times for
mean optimization during training.

For the case of $25\%$ label noise, the variance starts much higher and during
the course of training, increases by about $2.5$ times for mean optimization.
However, the variance only increases by about $20\%$ for UWO (using
$\lambda=0.1$). Further, using a KL penalty of $0.01$ results in a slight
reduction in the variance at the end of training. This hints at the fact that
while mean optimization is able to exploit any reward model that is currently
overestimating the reward, the uncertainty penalty in UWO prevents that - thus
resulting in better final performance (as shown in Figure 7) and reduced over-
optimization.

(a) BoN.

(b) PPO.

Figure 11: Impact of the cardinality of the ensemble on the final performance
of mean, UWO, and WCO for the $25\%$ label noise case.

(a) BoN.

(b) PPO.

Figure 12: Impact of uncertainty penalty weight on final performance for
different numbers of reward models within the ensemble. We note that there is
considerable robustness to the value of the uncertainty penalty.

##  6 Discussion

In this work, we have demonstrated that ensemble-based conservative
optimization methods improve performance and are highly effective in combating
the overoptimization problem in RLHF. In particular, our positive results in
the setup with $25\%$ label noise are highly encouraging as this setup better
models the real-world RLHF where human annotators often have high disagreement
rates (Stiennon et al., 2020; Dubois et al., 2023). This work opens up the
possibility of several interesting follow-ups. Firstly, future work should
consider replicating our findings in other environments (i.e. other RLHF
datasets) and with larger-scale language models. Secondly, our setup is based
on offline RLHF, in which human feedback is collected upfront and there are no
updates made to the reward model throughout the policy optimization process.
This contrasts with online RLHF (Bai et al., 2022a; Christiano et al., 2017),
where reward models are periodically retrained on freshly collected data from
humans. Does ensembling result in similar gains in this setup as well or not?
Uncertainty estimates from the ensemble may also help improve sample
efficiency of human feedback in this setup (Lee et al., 2021).

##  7 Related Works

Overoptimization in RLHF: Several works (Ibarz et al., 2018; Ziegler et al.,
2019; Stiennon et al., 2020) have provided anecdotal evidence of
overoptimization in RLHF. However, to the best of our knowledge, Gao et al.
(2023) are the only ones who study it systematically within the setup of fine-
tuning of LLMs. Our work utilizes their setup, reproduces their results, and
performs a systematic study evaluating the effectiveness of using ensembles
for mitigating overoptimization.

Use of Ensembles in (Model-Based) RL: Ensembles are often used in deep
learning as a tool to estimate uncertainty (Lakshminarayanan et al., 2017;
Dietterich, 2000) and can often outperform more complex Bayesian deep learning
methods (Ovadia et al., 2019). Learning an ensemble of dynamics model is
especially popular in model-based reinforcement learning where often a
reliable uncertainty estimate over state space is critical for avoiding
distribution shift (Depeweg et al., 2016; Gal et al., 2016; Chua et al., 2018;
Yu et al., 2020a). Disagreement among ensemble members is also a popular
approach for driving the exploration of learning agents in an environment
(Henaff, 2019; Pathak et al., 2019; Shyam et al., 2019). Brantley et al.
(2019) uses an ensemble-based approach to estimate the support of expert
policies in an imitation learning setup. However, to the best of our
knowledge, there is no prior work that explores the use of ensembles for
improving the robustness of RLHF; especially in the setting of language models
fine-tuning.

We further provide additional related works related to reward hacking and RLHF
in general in Appendix A.

## References

  * Abramson et al. (2022) Josh Abramson, Arun Ahuja, Federico Carnevale, Petko Georgiev, Alex Goldin, Alden Hung, Jessica Landon, Jirka Lhotka, Timothy Lillicrap, Alistair Muldal, et al.  Improving multimodal interactive agents with reinforcement learning from human feedback.  _arXiv preprint arXiv:2211.11602_ , 2022.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022a.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022b.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pp. 2397–2430. PMLR, 2023.
  * Boyd & Vandenberghe (2004) Stephen P Boyd and Lieven Vandenberghe.  _Convex optimization_.  Cambridge university press, 2004.
  * Brantley et al. (2019) Kiante Brantley, Wen Sun, and Mikael Henaff.  Disagreement-regularized imitation learning.  In _International Conference on Learning Representations_ , 2019.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _arXiv preprint arXiv:2307.15217_ , 2023.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Chua et al. (2018) Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  _Advances in neural information processing systems_ , 31, 2018.
  * Clark & Amodei (2016) Jack Clark and Dario Amodei.  Faulty reward functions in the wild.  https://openai.com/research/faulty-reward-functions, 2016.
  * Depeweg et al. (2016) Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, and Steffen Udluft.  Learning and policy search in stochastic dynamical systems with bayesian neural networks.  _arXiv preprint arXiv:1605.07127_ , 2016.
  * Dietterich (2000) Thomas G Dietterich.  Ensemble methods in machine learning.  In _International workshop on multiple classifier systems_ , pp. 1–15. Springer, 2000.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_ , 2023.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _arXiv preprint arXiv:2305.14387_ , 2023.
  * Gal et al. (2016) Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen.  Improving PILCO with Bayesian neural network dynamics models.  In _Data-Efficient Machine Learning workshop, International Conference on Machine Learning_ , 2016.
  * Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton.  Scaling laws for reward model overoptimization.  In _International Conference on Machine Learning_ , pp. 10835–10866. PMLR, 2023.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.  Reinforced self-training (rest) for language modeling.  _arXiv preprint arXiv:2308.08998_ , 2023.
  * Henaff (2019) Mikael Henaff.  Explicit explore-exploit algorithms in continuous state spaces.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Ibarz et al. (2018) Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei.  Reward learning from human preferences and demonstrations in atari.  _Advances in neural information processing systems_ , 31, 2018.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al.  Openassistant conversations–democratizing large language model alignment.  _arXiv preprint arXiv:2304.07327_ , 2023.
  * Krakovna et al. (2020) Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg.  Specification gaming: The flip side of ai ingenuity.  April 2020.  URL https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30, 2017.
  * Lee et al. (2021) Kimin Lee, Laura Smith, and Pieter Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  _arXiv preprint arXiv:2106.05091_ , 2021.
  * Lehman et al. (2020) Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al.  The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.  _Artificial life_ , 26(2):274–306, 2020.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.  Let’s verify step by step.  _arXiv preprint arXiv:2305.20050_ , 2023.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ , 2023.
  * Morgan (2022) Timothy Prickett Morgan.  Counting the cost of training large language models, 2022.  URL https://www.nextplatform.com/2022/12/01/counting-the-cost-of-training-large-language-models/.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.
  * OpenAI (2023a) OpenAI.  Openai models.  https://platform.openai.com/docs/models/, 2023a.
  * OpenAI (2023b) OpenAI.  Gpt-4 technical report.  _arXiv_ , pp. 2303–08774, 2023b.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ , 32, 2019.
  * Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt.  The effects of reward misspecification: Mapping and mitigating misaligned models.  _arXiv preprint arXiv:2201.03544_ , 2022.
  * Pathak et al. (2019) Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta.  Self-supervised exploration via disagreement.  In _International conference on machine learning_ , pp. 5062–5071. PMLR, 2019.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_ , 2023.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike.  Self-critiquing models for assisting human evaluators.  _arXiv preprint arXiv:2206.05802_ , 2022.
  * Scheurer et al. (2023) Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez.  Training language models with language feedback at scale.  _arXiv preprint arXiv:2303.16755_ , 2023.
  * Schulman (2020) John Schulman.  Approximating kl divergence, 2020.  URL http://joschu.net/blog/kl-approx.html.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Shyam et al. (2019) Pranav Shyam, Wojciech Jaśkowski, and Faustino Gomez.  Model-based active exploration.  In _International conference on machine learning_ , pp. 5779–5788. PMLR, 2019.
  * Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger.  Defining and characterizing reward gaming.  _Advances in Neural Information Processing Systems_ , 35:9460–9471, 2022.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.  Solving math word problems with process-and outcome-based feedback.  _arXiv preprint arXiv:2211.14275_ , 2022.
  * Venigalla & Li (2022) Abhinav Venigalla and Linden Li.  Mosaic llms (part 2): Gpt-3 quality for $<500$ k, 2022.  URL https://www.mosaicml.com/blog/gpt-3-quality-for-500k.
  * Wright (2006) Stephen J Wright.  _Numerical optimization_.  2006\.
  * Wu et al. (2021a) Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano.  Recursively summarizing books with human feedback.  _arXiv preprint arXiv:2109.10862_ , 2021a.
  * Wu et al. (2021b) Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh.  Uncertainty weighted actor-critic for offline reinforcement learning.  _arXiv preprint arXiv:2105.08140_ , 2021b.
  * Yu et al. (2020a) Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.  Mopo: Model-based offline policy optimization.  _Advances in Neural Information Processing Systems_ , 33:14129–14142, 2020a.
  * Yu et al. (2020b) Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.  Mopo: Model-based offline policy optimization.  _Advances in Neural Information Processing Systems_ , 33:14129–14142, 2020b.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  Rrhf: Rank responses to align language models with human feedback without tears.  _arXiv preprint arXiv:2304.05302_ , 2023.
  * Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  _arXiv preprint arXiv:2305.10425_ , 2023.
  * Zheng et al. (2023) Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, et al.  Secrets of rlhf in large language models part i: Ppo.  _arXiv preprint arXiv:2307.04964_ , 2023.
  * Zhuang & Hadfield-Menell (2020) Simon Zhuang and Dylan Hadfield-Menell.  Consequences of misaligned ai.  _Advances in Neural Information Processing Systems_ , 33:15763–15773, 2020.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

## Appendix

##  Appendix A Additional Related Works

RLHF: Reinforcement learning from human feedback (RLHF), in its modern form,
was popularized by Christiano et al. (2017). Ziegler et al. (2019) was the
first work to demonstrate the effectiveness of RLHF for fine-tuning language
models. This led to several applications of RLHF to fine-tuning LLMs focused
on improving several aspects of language models e.g. instruction following
(Ouyang et al., 2022; Abramson et al., 2022), summarization capabilities
(Stiennon et al., 2020; Wu et al., 2021a), web browsing (Nakano et al., 2021),
translation (Gulcehre et al., 2023) and helpfulness & harmlessness (Bai et
al., 2022a). This has also caused a great amount of interest in improving RLHF
as a method. One main line of research here is methods that try to augment the
learning signal for the reward model in some form e.g. language feedback
(Scheurer et al., 2023) or process supervision (Lightman et al., 2023; Uesato
et al., 2022). This, however, generally incurs additional costs in labeling.
This has prompted research on extracting feedback signal from a pretrained
language model via appropriate prompting (Bai et al., 2022b; OpenAI, 2023b;
Madaan et al., 2023; Saunders et al., 2022), however, this requires effective
prompting and a highly capable language model to be successful. Another line
of research focuses on the use of alternative algorithms to PPO for policy
optimization step (Gulcehre et al., 2023; Rafailov et al., 2023; Zhao et al.,
2023; Yuan et al., 2023; Dong et al., 2023). Orthogonal to both these lines of
work, we explore using ensembles to improve RLHF.

Reward Hacking: Overotpimization in RLHF is an instance of reward hacking.
Many examples of reward hacking exist in literature (Clark & Amodei, 2016;
Lehman et al., 2020; Krakovna et al., 2020). Skalse et al. (2022) provide a
formal definition for reward hacking and theoretically study whether
unhackable proxies can exist or not under different conditions. Zhuang &
Hadfield-Menell (2020) study the reward hacking that might arise due to
partial observability. Pan et al. (2022) provide examples of proxy reward
functions in various reinforcement learning environments where low-capacity
policies do not elicit reward hacking but policies with greater capacity do.

##  Appendix B BoN Estimator

The naive way of estimating the average reward of the BoN policy under a gold
reward model, when using a proxy reward model as the ranking function, is to
use a Monte Carlo estimate in which $n$ answers $\\{A_{1},...,A_{2}\\}$ are
sampled from a language model $f$ and then scored for a given prompt $q$ as
follows:

$\displaystyle R_{n}^{\text{gold}}(q):=\mathbb{E}_{A_{1},A_{2},...,A_{n}\sim
f(q)}\left[R^{\text{gold}}\left(\operatorname*{arg\,max}_{a\in\\{A_{1},A_{2},...,A_{n}\\}}R^{\text{proxy}}(a|q)|q\right)\right]$
(6)

This is very wasteful as it does not reuse answers for different values of
$n$. Therefore, we instead use the following unbiased estimator (Nakano et
al., 2021) that samples $N\geq n_{max}$ outputs for a given input $q$:

$\displaystyle R_{n}^{\text{gold}}(q)=\frac{1}{\binom{N}{n}}\sum_{1\leq
i_{1}<\dots<i_{n}\leq
N}R^{\mathrm{gold}}\left(\underset{a\in\left\\{A_{i_{1}},\dots,A_{i_{n}}\right\\}}{\operatorname{argmax}}R^{\mathrm{proxy}}\left(a\mid
q\right)\mid q\right)$ (7)

This can be computed efficiently by first sorting all answers
$A_{1},...,A_{N}$ under the proxy reward model to obtain scores
$S_{1},...,S_{N}$ and then computing:

$\displaystyle\frac{1}{\binom{N}{n}}\sum_{1\leq i_{1}<\dots<i_{n}\leq
N}R^{\mathrm{gold}}\left(\underset{a\in\left\\{S_{i_{1}},\dots,S_{i_{n}}\right\\}}{\operatorname{argmax}}R^{\mathrm{proxy}}\left(a\mid
q\right)\mid
q\right)=\sum_{i=n}^{N}\frac{\binom{i-1}{n-1}}{\binom{N}{n}}R^{\mathrm{gold}}\left(S_{i}\mid
q\right)$ (8)

To get the gold reward model score for a given range of values
$n=1,2,...,n_{max}$, we simply evaluate the empirical average of the above
estimator for all questions for each $n$.

##  Appendix C KL Distance Calculation for PPO

The naive way to calculate KL distance between the PPO-optimized policy
$\pi_{\text{RL}}$ and the pretrained model is as follows:

$\text{KL}_{\text{RL}}(\pi_{\text{RL}},\pi_{\text{init}})=E_{x\sim\pi_{\text{RL}}}\left[\text{log}\,\frac{\pi_{\text{RL}}}{\pi_{\text{init}}}\right]$
(9)

However, this estimator has high variance and can be negative, unlike actual
KL. Therefore, we use the following estimator (Schulman, 2020):

$\text{KL}_{\text{RL}}(\pi_{\text{RL}},\pi_{\text{init}})=E_{x\sim\pi_{\text{RL}}}\left[\frac{1}{2}\left(\text{log}\,\frac{\pi_{\text{RL}}}{\pi_{\text{init}}}\right)^{2}\right]$
(10)

##  Appendix D Additional Experimental Details

###  D.1 Hyperparameters

We give the hyperparameters here for different components of our RLHF
pipeline:

Table 1: SFT hyperparameters. Parameter | Value
---|---
Learning rate | 8e-6
Epochs | 3
Batch size | 4
Table 2: RM hyperparameters. Parameter | Value
---|---
Learning rate | 1e-5
Epochs | 5
Batch size | 32
Table 3: PPO hyperparameters. Parameter | Value
---|---
Learning rate | 1e-6
Cosine annealing scheduler | 1e-7
PPO epochs | 4
Batch size | 32
Number of rollouts | 256
Chunk size | 32
Clipping range & value | 0.2
GAE lambda | 0.95
Table 4: Generation hyperparameters. Parameter | Value
---|---
Max instruction length | 520
Max new tokens (answer length) | 256
PPO epochs | 4
Top-p | 0.9 (1.0 for PPO training)
Top-k | 0
Temperature | 1.0

###  D.2 Prompt Format

Though we use instructions from the AlpacaFarm dataset, this only provides
content for prompting the LLM and still requires formatting. We opt for
minimalism, following the v2 format used in OpenAssistant (Köpf et al., 2023).
This format follows the GPTNeoXTokenizer class used to pretrain our LLMs and
introduces two special tokens: <|prompter|> and <|assistant|>.

Table 5: Example answer generation and reward modeling prompts with proper formatting. Answer generation prompt |  Reward modelling prompt
---|---
<|prompter|>What geometric shape has 5 sides and 5 angles?<|endoftext|> <|assistant|> |  <|prompter|>What geometric shape has 5 sides and 5 angles?<|endoftext|> <|assistant|>The geometric shape is a pentagon.<|endoftext|>
<|prompter|>Analyze the following sentence and identify the verb and its tense.$\backslash$nShe played the piano.<|endoftext|> <|assistant|> |  <|prompter|>Analyze the following sentence and identify the verb and its tense.$\backslash$nShe played the piano.<|endoftext|> <|assistant|>Verb: Played Tense: Past<|endoftext|>

For answer generation, the model should be prompted with the instruction and
the input. Inputs, should they be present, are appended to the instruction
after a new line to form the prompt. The prompt is then prepended with the
<|prompter|> token and closed off with an end-of-text (EOT) <|endoftext|>
token declaring the end of the instruction, and the <|assistant|> token to
start the answer. An example is shown in Table 5.

For reward modeling, the prompt should also contain an answer to be evaluated.
In this case, the answer text (from an AlpacaFarm dataset demonstration or a
previous answer generation) is appended to the initial prompt containing the
instruction and closed off with the EOT token. This forms the full RM prompt,
with an example shown in Table 5.

##  Appendix E Reproduction of Results Of Gao et al. (2023)

We also successfully reproduced the results of Gao et al. in our setup shown
in Figures 13 and 14. Note that our setup uses open-source models derived from
the Pythia suite (Biderman et al., 2023) as the base for policy and proxy
reward models and the AlpacaFarm human-preference reward model (Dubois et al.,
2023) is used as the gold reward model. In contrast, Gao et al. used closed-
source models based on GPT series. Our successful reproduction of their
results in our setup hints at the general nature of overoptimization.

(a) BoN results for various reward model sizes, with data size held constant
(46k).

(b) BoN results for various reward model training data sizes, with RM size
held constant (44M).

Figure 13: Gold reward model scores for BoN follow a similar trend as that
observed by Gao et al. (2023) when varying reward model and data sizes. Runs
are averaged over five seeds, with standard deviation additionally shown.
$25\%$ label noise is used here to highlight overoptimization, as we found it
difficult to observe with our limited BoN optimization capability. This is in
line with label noise ablation experiments from Dubois et al. (2023). Figure
14: PPO optimization results for various reward model training data sizes,
with reward model size held constant (44M). Average and standard deviation
over three runs are shown. A similar trend to Gao et al. (2023) is observed,
with greater overoptimization as optimization is pushed further.

##  Appendix F Additional Results

###  F.1 RM Validation Loss Curve

Figure 15: Reward model validation loss during training for 44M reward models
with no label noise. We note that while there is a minor variation in
validation loss, this variation is not predictive of whether the reward model
is robust to overoptimization or not. For example, in Figure 16, we see that
the reward model with the highest validation loss does not suffer
overoptimization but other reward models with lower validation losses do.

###  F.2 Individual Reward Model Training Optimization Plots

For the single reward model results, we optimize each reward model separately
and present their average in Figures 3 and 4. Here, we present the training
curve for each reward model separately.

(a) No label noise.

(b) $25\%$ label noise case.

Figure 16: BoN for $44$M reward model. The average of these curves is
presented in Figure 3.

(a) No label noise.

(b) $25\%$ label noise case.

Figure 17: PPO results for $44$M reward model. The average of these curves is
presented in Figure 4.

###  F.3 $7$M Reward Model Training Optimization Results

(a) BoN.

(b) PPO.

Figure 18: BoN and PPO results with the 7M reward models to supplement the bar
plot in Figure 8.

###  F.4 $1.3$B Reward Model Training Optimization Results

(a) BoN.

(b) PPO.

Figure 19: BoN and PPO results with the 1.3B reward models to supplement the
bar plot in Figure 8.

###  F.5 Qualitative Samples

Quantitative metrics are used throughout our experiments as they enable the
rapid and comprehensive evaluation of various methods. Here we provide a brief
insight into the qualitative aspect of the methods discussed in this work. For
a given prompt, answers from the final policy ($n=12500$ for BoN, and 3000
steps for PPO) of each method are provided. Policies from the runs in Figures
3 and 4, which provide the strongest final results, are used.

The main takeaways are as follows. First, the BoN answers are shorter, because
policy optimization does not occur in the same way it does for PPO. Second,
single reward models show signs of failure and overoptimization. For BoN, this
translates to poor answers and even hallucinations in the case of Table
LABEL:tab:set_one. For PPO, this also translates to poor answers, which are
very long and highly repetitive. These are clear instances of
overoptimization, where the reward model seems to have learned to produce long
answers regardless of the content, which offers good proxy reward model
performance but poor gold model performance. Finally, we note that the
conservative optimization methods produce strong qualitative results, even
when single reward models are struggling.

Table 6: First set of example answers to an evaluation prompt. Prompt |  Find the closest airport to the given location.
Seattle, Washington
---|---
Single RM (BoN) |  The closest airport to the given location is Sea-Tac Airport, with a direct flight from SFO airport in San Francisco, CA to Seattle-Tacoma International Airport available on up to dates.
Mean (BoN) |  The nearest airport to Seattle, Washington is Sea-Tac International Airport (SEA). It is located a distance of 17.6 miles away and has an estimated time of arrival of around 12:30 PM.
WCO (BoN) |  The closest airport to the given location is Seattle-Tacoma International Airport (SITE). It is located approximately 10.7 miles away from the given location.
UWO (BoN) |  The closest airport to the given location is Seattle-Tacoma International Airport. It is located 23.7 miles away, and the estimated time of arrival is 2 hours, 28 minutes.
Single RM (PPO) |  The closest airport to the given location is Seattle-Tacoma International Airport, located in the Seattle, Washington, region. It is a major airport with a major hub, serving over 15 million passengers each year. It is located just 10 miles from the city center and is connected to the city by the Bunker Hill and Space Needle commuter rail lines. It is the primary airport for the region and is one of the most important hubs in the country. It is a major hub for a variety of airlines, and is a popular destination for travelers from around the world. It is a hub for a variety of domestic and international flights, and is the main hub for the region. It is a major hub for several airlines, and is a popular destination for travelers from around the world. It is a major hub for a variety of domestic and international flights, and is the primary hub for the region. It is a major hub for a variety of airlines, and is a popular destination for travelers from around the world. It is a major hub for a variety of domestic and international flights, and is the main hub for the region. It is a major hub for a variety of airlines, and is a popular destination for travelers from around the world.
Mean (PPO) |  The closest airport to Seattle is Boeing Field, located in Seattle, Washington. It is served by several airlines, including Alaska Airlines, United Airlines, Delta Air Lines, and American Airlines. It is located approximately 35 miles away from the given location. The airport is also served by several car rental companies, including Hertz, Avis, and Budget.
WCO (PPO) |  The closest airport to the given location is Seattle-Tacoma International Airport (SEA). It is located approximately 43 miles from the location. The average time to get to the airport is around 2 hours. The cost to get to the airport is around $110 for a single-occupancy car. It is the most popular airport in the area and is served by many airlines. The airport offers a variety of services, including a baggage claim, a car rental counter, and a currency exchange. There are also a number of attractions nearby, including the Space Needle, the Aquarium of the Pacific, and the Seattle Center. The airport is well-served by public transportation, making it easy to get around.
UWO (PPO) |  The closest airport to the given location is Seattle-Tacoma International Airport (SEA). The airport is located in the Seattle-Tacoma, Washington, area and is served by many domestic and international airlines. It is located about 70 miles east of the city of Seattle and is accessible by car, bus, train, and ferry. The airport has several terminals, including a domestic terminal, an international terminal, and a car rental center. The airport has two runways, allowing for both domestic and international flights. The airport is served by many international carriers, including Alaska, American, Delta, United, and Virgin. It is also home to many domestic carriers, including American, Delta, and United.
Table 7: Second set of example answers to an evaluation prompt. Prompt |  Create an analogy to explain cloud computing.
---|---
Single RM (BoN) |  Cloud computing is like the sky; it is a vast, untamed expanse filled with a variety of colorful shapes and textures. Just like clouds move and shift based on the weather, so too does cloud computing. It is constantly evolving and adapting to meet the demands of modern technology.
Mean (BoN) |  Cloud computing is like a sprawling house, with multiple rooms and floors that each represent a specific type of resource. Each room serves a specific purpose, such as storage, processing, or network access, and each floor has its own features such as storage, computing power, or network connectivity. The cloud provides a central hub where resources can be accessed from anywhere in the house, and enables users to access the resources they need quickly and easily.
WCO (BoN) |  Cloud computing is like a giant server room in your home, with multiple computers connected together to store, process, and retrieve data. The computers are like small servers in the server room, performing different tasks such as running virtual machines, performing analytics, and performing other tasks. Each computer is connected to the server, and the data stored in the server is retrieved and used by the computers. The data is stored in a database, allowing data to be quickly and easily accessed, updated, and processed.
UWO (BoN) |  Cloud computing is like a stack of pancakes; each pancake is a machine that processes data and provides computing power. When you want to make pancakes, you simply select the ingredients, put them in a pan, and flip them to make pancakes. Similarly, when you use cloud computing, you select the services you need to make use of, then pay a monthly fee to use them. When you’re done with your work, you can shut down the machine and delete the files, or if you need more power, you can simply purchase more power and pay a recurring fee to make use of it.
Single RM (PPO) |  The analogy is that of a computer being a network of interconnected servers, each with its own processing power and memory, and the cloud as the central hub where the data is stored, allowing the various servers to access the data in a more efficient and cost-effective manner. It is the cloud that allows the various servers to be able to access the same data in the same way, and the data is stored and managed in the cloud, thus allowing the various servers to work together to provide a more efficient and cost-effective solution. It is the cloud that is the backbone of the modern technology and it is the ability to access the data from the various servers, in a more cost-effective and efficient manner, that makes cloud computing the ideal solution for the modern technology. It is the cloud that is the future of technology and it is the ability to access the data from the various servers, in a more cost-effective and efficient manner, that is the foundation of the modern technology. It is the cloud that is the new standard and it is the ability to access the data from the various servers, in a more cost-effective and efficient manner, that is the foundation of the future of technology. It is the cloud that is the future of technology, and it is the ability
Mean (PPO) |  Cloud computing is like a giant cloud of air that provides all of the computing power needed to run a business. The cloud provides access to the necessary computing resources to enable businesses to remain competitive and stay ahead of the competition. Cloud computing is a cost-effective way to access the computing power needed to run a business, and it can reduce the time it takes to develop and deploy new software. It also allows businesses to scale their operations quickly, with the ability to quickly add or remove resources as needed. Cloud computing is a great way for businesses to remain competitive and stay ahead of the competition.
WCO (PPO) |  Cloud computing is like a huge, open room with many different walls and floors. The walls provide security and privacy, while the floors provide resources and data access. The ceiling is the sky, and the sky is the sky, providing a vast amount of resources, power, and opportunities. The cloud is a room full of resources, and the resources are the people, providing a diverse community of expertise and resources. The walls, floors, and sky are all important, and together they provide an invaluable resource. This analogy helps to explain the importance of cloud computing, and how it can be beneficial to organizations of all sizes. It shows the power of the cloud and the opportunities it provides, allowing organizations to scale quickly and efficiently. It also illustrates the importance of providing a secure, reliable, and cost-effective infrastructure.
UWO (PPO) |  Cloud computing is like a modern-day library, where data is stored and processed on a centralized server. The data is accessed and manipulated from any device connected to the internet, allowing for faster and more efficient services. The data is stored in the cloud and is accessible from any device, allowing for the user to access the data quickly and efficiently. The cloud computing platform is easy to use, provides the user with the flexibility to access the data quickly and easily, and is cost effective. The cloud computing platform provides the user with the ability to access the data quickly and easily, allowing them to make decisions based on the data. The cloud computing platform also provides the user with the flexibility to access the data quickly and easily, allowing them to make decisions based on the data. This allows the cloud computing platform to be a cost effective solution for the user, while providing them with the ability to access the data quickly and easily.


# The TDHF code Sky3D version 1.2

Abhishek  Paul Stevenson  Yue Shi  Esra Yüksel  A. S. Umar  School of
Mathematics and Physics, University of Surrey, Guildford, GU2 7XH  Department
of Physics, Harbin Institute of Technology, Harbin 150001, China  Department
of Physics and Astronomy, Vanderbilt University, Nashville, TN 37235, USA

###### Abstract

The Sky3D code has been widely used to describe nuclear ground states,
collective vibrational excitations, and heavy-ion collisions. The approach is
based on Skyrme forces or related energy density functionals. The static and
dynamic equations are solved on a three-dimensional grid, and pairing is been
implemented in the BCS approximation. This updated version of the code aims to
facilitate the calculation of nuclear strength functions in the regime of
linear response theory, while retaining all existing functionality and use
cases. The strength functions are benchmarked against available RPA codes, and
the user has the freedom of choice when selecting the nature of external
excitation (from monopole to hexadecapole and more). Some utility programs are
also provided that calculate the strength function from the time-dependent
output of the dynamic calculations of the Sky3D code.

††journal: Computer Physics Communications

NEW VERSION PROGRAM SUMMARY

Program Title: Sky3D
CPC Library link to program files: (to be added by Technical Editor)
Developer’s repository link:
https://gitlab.surrey.ac.uk/phs3ps/Sky3D/-/tree/v1.2?ref_type=tags
Code Ocean capsule: (to be added by Technical Editor)
Licensing provisions: GPLv3
Programming language: Fortran, with one post-processing utility in Python
Supplementary material:
Journal reference of previous version: Schuetrumpf, B., Reinhard, P. G.,
Stevenson, P. D., Umar, A. S., and Maruhn, J. A. (2018). The TDHF code Sky3D
version 1.1. Computer Physics Communications, 229, 211-213.
Does the new version supersede the previous version?: Yes
Reasons for the new version: The capability of reproducing the nuclear
strength function for a variety of newly-coded external boosts has been added.

Nature of the problem: Calculating nuclear multipole strength functions is a
crucial probe that can help model the nuclear system and its structure
properties. A variety of models exist for this task, such as QRPA
(Quasiparticle Random Phase Approximation) and its variants, but such
approaches are often limited due to symmetry constraints. Time-dependent
Hartree Fock (TDHF) has been used to simulate nuclear vibrations and
collisions between nuclei for low energies without assuming any symmetry in
the system. This code extends the TDHF to calculate the multi-pole strength
functions of atomic nuclei. We showcase its reliability by comparing it with
the established RPA codes for the calculation of such strength functions.

Solution method: We extended previous versions of the Sky3D code [1, 2] to
include an external boost of multipole type where the user can provide custom
input that decides the nature of the multipole (monopole, quadrupole,
octupole, and so on) boost. The principal aim is to calculate the multipole
strength function, which is the Fourier transform of the time-dependent
expectation value of the multipole operator, which has the same form as the
external boost. The proper unit conversion is done so that one can extract the
exact unit of the thus calculated strength function, which is comparable to
the available literature in the field. The boundary conditions are chosen such
that a Woods-Saxon-like function cuts off the external field, driving it to
zero at the boundary.

Summary of revisions: To generate the documentation, go into the Doc-doxygen
subdirectory and execute make HTML, make latex, or make all to produce the
corresponding version or both of them. The following is the exhaustive list of
changes to the code.

## Makefile

The Makefile has been updated to include targets apple and apple_omp to use
gfortran targetting Apple silicon (ARM-based M1, M2 etc) in which the Lapack
dependencies are satisfied with the Apple Accelerate framework and fftw is
assumed to be installed via Homebrew111http://brew.sh.

## External Field

For the dynamic case, the system can be coupled to an external excitation
field of general type

$\displaystyle F_{LM}^{IS}(\vec{r})$ $\displaystyle=\sqrt{2L+1}r^{L}Y_{LM}$
(1) $\displaystyle F_{LM}^{IV}(\vec{r})$
$\displaystyle=\sqrt{2L+1}r^{L}Y_{LM}f_{c},$ (2)

where IS and IV represent the isoscalar and isovector operators, with some
exceptions as noted below. For the IV case, the factor $f_{c}$ is multiplied
with the operator. $f_{c}=\tau_{z}$ for all the multipole operators except the
dipole operator. Here $\tau_{z}$ is the is the $z$-component of the isospin
operator. For the dipole case, it is customary to remove the contribution from
the center of mass by setting the $f_{c}$ to $N/A$ for protons and $-Z/A$ for
neutrons. For the monopole case, $L=0$ case, the non-vanishing excitations are
produced using

$\displaystyle F_{00}^{IS}$ $\displaystyle=r^{2}Y_{00}$ (3) $\displaystyle
F_{00}^{IV}$ $\displaystyle=r^{2}Y_{00}\tau_{z}$ (4)

We have added this external field in the external.f90 file and a new ylm.f90
file has been added to calculate the spherical harmonics ($Y_{LM}$) used to
define the external field.

To avoid spurious states in the strength function and provide a comparison
with the available RPA results, the definition of the external field for the
isoscalar dipole excitation is different and defined as [3]

$F_{q}(\vec{r})=\sqrt{3}(r^{3}-\frac{5\left\langle
r^{2}\right\rangle}{3}r)Y_{10}\;,$ (5)

where $\left\langle r^{2}\right\rangle$ is the expectation value of $r^{2}$
calculated in the static run. This includes a higher order ($r^{3}$ ) term as
the natural dipole order induces a shift of the center of mass of the nucleus
and not an internal excitation. The input file contains the variable isoext
which decides the nature of the external boost, it should be set to a value of
1 for an isovector type external boost and 0 for the isoscalar type. The code
calculates both the isovector and isoscalar multipole response for the either
kind of external boost.

## Mix Pairing

The definition of pairing force is changed slightly to accommodate the
implementation of mixed-type pairing by introducing a parameter, $x$, in the
case of DDDI input for the ipair=6 value. The new definition, in this case, is

$V_{P}(\vec{r})=V_{0}\cdot\xi(\vec{r})\cdot\left(1-\frac{x\cdot\rho}{\rho_{0}}\right).$
(6)

The isospin-dependent pairing strength $V_{0}$ is obtained from the force
definition as before and

$\xi(\vec{r})=\sum_{k}u_{k}v_{k}|\phi_{k}(\vec{r})|^{2}$ (7)

where $\phi_{k}(\vec{r})$ are the single-particle wave functions. The value of
$x$ is now contained in the forces.data file for the case of DDDI, and by
default, the value is given as one to generate the original DDDI case, and a
value of 0.5 defines a mix of volume and surface-type pairing. Parameter $x$
is denoted as ‘mixture’ in the code and given as input in the forces.data file
along with the v0prot (the strength of pairing for protons in MeV) v0neut (the
strength of pairing for neutrons in MeV) and rho0pr (the value of $\rho_{0}$).
Although the same effect of creating a mixed type pairing can be achieved by
changing the value of $\rho_{0}$, e.g., $x=0.5$, is the same as taking
$\rho_{0}=0.32$ without introducing the parameter $x$, but using a parameter
like $x$ is easier to read and convey the meaning of a mixture of surface and
volume pairing. It is a more common method in the literature [4] to use $x$ as
a switch, which can go from 0 (volume pairing) to 1 (surface pairing) and
fixing the $\rho_{0}$ to 0.16 fm-3.

## Input Files

The input file for the dynamic run to calculate the time-dependent multipole
moments now requires the $L$ and $M$ values used in the definition of the
external field [Eq. (1)]. The main changes are made to the namelist extrn as
it now includes

    NAMELIST/extern/ ampl_ext,L_val,M_val,radext,widext,isoext,&
        ipulse,omega,tau0,taut,textfield_periodic,r_avg,only_P

Where ampl_ext is the strength of external boost, L_val and M_val are the $L$
and $M$ values and $r\\_avg$ is the value of $\sqrt{\left\langle
r^{2}\right\rangle}$ also known as the root mean square value (rms) and is
used in the isoscalar dipole case [Eq. (5)].
The variable only_P is introduced in order to calculate the electromagnetic
response, which is obtained by only boosting the proton wavefunctions with the
external field (if only_P = 1). The default value is set to zero. This can be
helpful in the evaluation of $B(E\lambda)$ values. However, caution is needed
as it can shift the system’s center of mass, leading to unreliable results.

## Output Files

For every dynamic run, the code now prints the expectation values of all the
multipole operators as a function of time in the  .res files. For example, the
monopole moment will be printed in monopoles.res, dipole moment in
dipoles.res, and so on.

The corresponding multipole moment file should be considered to calculate the
strength function for the given multipole boost. By analysing a different
multipole response to that applied as a boost, one can also explore mode
coupling, which appears naturally in a TDHF framework [5].

## Utilities

A new Python program, Fourier.py, is added in the utility section in the
‘utils/strength_calculation’ directory. The numpy package is required to run
this utility program. It calculates the strength function by performing the
Fourier transform ($\hat{O}(\omega)$) of the time-dependent multipole moment
generated after a boost with a strength $\eta$. The strength function is
defined as

$S(E)=\frac{1}{\eta\hbar c\pi}\text{Im}[\hat{O}(\omega)]$ (8)

A finite time simulation introduces some artifacts in the Fourier transform if
the time signal doesn’t die at the end of the simulation [6]. This can be
overcome by manually damping the signal by introducing a filter. The filtering
can be done in many ways and we provide two filters in this utility program.
One is named the Cos filter, and the other is the Exponential (Exp) filter,
whose definitions are

Cos Filter $\displaystyle=\mathrm{cos}^{n}\left(\frac{\pi t}{2T_{0}}\right),$
(9) Exp Filter $\displaystyle=\exp{\left(\frac{-\Gamma_{0}t}{2\hbar}\right)},$
(10)

where $\Gamma_{0}$ and $n$ are parameters whose default values are chosen as 1
MeV and 6, respectively. The Cos filter was initially provided in the previous
version of the code. In the present work, the exponential filter is added to
help compare the results with the other codes. The output of this strength
calculation program is saved in two files, one of which contains the isoscalar
strength function and other contains the isovector strength function. The
multipole is selected when running the code.

## Benchmarks

This section compares the isoscalar (IS) and isovector (IV) strength functions
obtained for different multipole operators with the RPA results obtained using
the hfbcs-qrpa (v1) code[4]. We compare the results for 208Pb. In all cases,
TDHF satisfies the energy-weighted sum rules (EWSR) with a percentage score of
100$\pm$3 %.

The M is set to zero in Eq. (1) for all multipole operators in this
comparison. A sample of the input file for the dynamic run looks like

     &files wffile=‘restart’/
     &force name=‘Sly5’, pairing=‘NONE’ /
     &main mprint=10,mplot=0,
      mrest=500000,writeselect=‘rc’,
      imode=2,tfft=T,nof=1 /
     &grid nx=24,ny=24,nz=24,dx=1,dy=1,dz=1,
    Ψperiodic=F /
     &dynamic nt=50000, dt=0.2, mxpact=4, texternal=T/
     &extern ipulse=0,isoext=0,ampl_ext=5D-5,L_val=0,M_val=0,
       radext=200D0, widext=1D0,textfield_periodic=F /
     &fragments filename=1‘./Pb208’,fix_boost=T,
          fcent(1,1)=0,0,0    /

Figure 1: Comparison between the IS strength obtained with SLy5 force using
TDHF (Solid Black) and RPA strength functions (Red dashed) associated with the
isoscalar operators, in the case of 208Pb . Figure 2: Same as the Fig. 1 but
with Isovector strength functions.

Fig. 1, shows the TDHF strength functions for isoscalar monopole to isoscalar
octupole boosts obtained with the present version of Sky3D compared with the
RPA strength functions in the case of 208Pb. For the isovector cases, the
results are shown in Fig. 2. All the results are obtained using the SLy5 force
and the Exp. filter [Eq. 10] with $\Gamma_{0}=1$ MeV is used. Overall, the
TDHF results show a good agreement with the QRPA results in all the multipole
responses in both isoscalar and isovector cases. Some differences in which the
discretisation of broad peaks above particle decay threshold can be seen. This
is presumably due to boundary conditions at large $r$, which differ between
TDHF (wave functions go to zero at the edge of coordinate space box) and RPA
(truncated oscillator basis).

## Acknowledgements

This work has been supported by the UK Science and Technology Facilities
Council (STFC) under grants ST/V001108/1. This work was in part performed
using the DiRAC Data Intensive service at Leicester, operated by the
University of Leicester IT Services, which forms part of the STFC DiRAC HPC
Facility (www.dirac.ac.uk). The equipment was funded by BEIS capital funding
via STFC capital grants ST/K000373/1 and ST/R002363/1 and STFC DiRAC
Operations grant ST/R001014/1. DiRAC is part of the National e-Infrastructure.
This work is also Sspported by the National Natural Science Foundation of
China (12075068, 11705038), the Fundamental Research Funds for the Central
Universities (HIT.BRET.2021003). This work is also supported by the US DOE
under grant No. DE-SC0013847

## References

  * [1] J. Maruhn, P.-G. Reinhard, P. Stevenson, A. Umar, The tdhf code sky3d, Computer Physics Communications 185 (7) (2014) 2195–2216.  doi:https://doi.org/10.1016/j.cpc.2014.04.008.
  * [2] B. Schuetrumpf, P.-G. Reinhard, P. Stevenson, A. Umar, J. Maruhn, The tdhf code sky3d version 1.1, Computer Physics Communications 229 (2018) 211–213.  doi:10.1016/j.cpc.2018.03.012.
  * [3] G. Colò, L. Cao, N. Van Giai, L. Capelli, Self-consistent rpa calculations with skyrme-type interactions: The skyrme_rpa program, Computer Physics Communications 184 (1) (2013) 142–161.  doi:10.1016/j.cpc.2012.07.016.
  * [4] G. Colò, X. Roca-Maza, User guide for the hfbcs-qrpa(v1) code (2021).  arXiv:2102.06562.
  * [5] Y. Shi, P. D. Stevenson, Isoscalar monopole strength in 100Mo: an indicator for static triaxial deformation in the ground state, Chinese Physics C 47 (3) (2023) 034105.  doi:10.1088/1674-1137/acac6b.
  * [6] W. Press, S. Teukolsky, W. Vetterling, Numerical Recipes with Source Code CD-ROM 3rd Edition: The Art of Scientific Computing, Cambridge University Press, 2007.