# Shattering the Agent-Environment Interface for Fine-Tuning Inclusive
Language Models

Wanqiao Xu
Department of Management Science & Engineering
Stanford University
<EMAIL_ADDRESS>
&Shi Dong
Microsoft
<EMAIL_ADDRESS>
Dilip Arumugam
Department of Computer Science
Stanford University
<EMAIL_ADDRESS>
&Benjamin Van Roy
Department of Electrical Engineering
Department of Management Science & Engineering
Stanford University
<EMAIL_ADDRESS>
###### Abstract

A centerpiece of the ever-popular reinforcement learning from human feedback
(RLHF) approach to fine-tuning autoregressive language models is the explicit
training of a reward model to emulate human feedback, distinct from the
language model itself. This reward model is then coupled with policy-gradient
methods to dramatically improve the alignment between language model outputs
and desired responses. In this work, we adopt a novel perspective wherein a
pre-trained language model is itself simultaneously a policy, reward function,
and transition function. An immediate consequence of this is that reward
learning and language model fine-tuning can be performed jointly and directly,
without requiring any further downstream policy optimization. While this
perspective does indeed break the traditional agent-environment interface, we
nevertheless maintain that there can be enormous statistical benefits afforded
by bringing to bear traditional algorithmic concepts from reinforcement
learning. Our experiments demonstrate one concrete instance of this through
efficient exploration based on the representation and resolution of epistemic
uncertainty. In order to illustrate these ideas in a transparent manner, we
restrict attention to a simple didactic data generating process and leave for
future work extension to systems of practical scale.

##  1 Introduction

While recent years have witnessed a dramatic shift in the capabilities of
generative AIs across numerous data modalities, excitement and discourse
surrounding natural language processing (NLP) and large language models (LLMs)
in particular has become near-ubiquitous within just the last few months [55,
84], leading to an unprecedented proliferation of daily users probing and
exploring these models’ impressive capabilities through prolonged, interactive
dialogues. With this attention has also come an onslaught of challenges for
the AI and machine learning research communities, ranging from the rigorous
benchmarking of capabilities [47], adherence to copyright law [30], concerns
for privacy [46, 45], and insight into the key methodologies for training
these models [55], to name just a few. One question that lies at the heart of
the last issue revolves around how much the successes of fine-tuned,
autoregressive LLMs are driven by the reinforcement learning from human
feedback (RLHF) [76, 66] pipeline?

While the classic NLP task of language modeling is easily formulated and
solved through traditional supervised-learning techniques [15, 52], the RLHF
paradigm has found great empirical success by interpreting this as merely a
preliminary pretraining phase and further incorporating a subsequent fine-
tuning phase that leverages human feedback when refining responses to be more
accurate and more preferable. From the perspective of a sequential decision-
making process, two hallmark characteristics of this pipeline include (1)
viewing a language model as a policy, mapping a rich, pretrained
representation of a sequence of tokens along with a partial response to a
next-token distribution and (2) interpreting human feedback as identifying a
terminal reward function that assigns scalar feedback to completed prompt-
response pairs so as to incentivize preferred responses. In this work, we
introduce a novel perspective on LLMs that extends (1) and renders (2) moot,
giving rise to a novel and statistically-efficient fine-tuning method.

We recognize that by virtue of vast amounts of unstructured Web data, a
pretrained LLM can be simultaneously viewed as a policy, a reward function,
and an environment simulator. Traditionally, a policy is implemented within a
decision-making agent whereas the reward function and simulator are properties
of the environment and, therefore, reside external to the agent. Thus, our
novel perspective blurs the traditional boundary between agent and environment
found throughout the reinforcement-learning literature [79]. Nevertheless, in
this paper we demonstrate the value of this triumvirate through a meticulous
collection of simple yet illustrative experiments, designed to highlight how
foundational concepts from reinforcement learning can still be successfully
brought to bear for fine-tuning LLMs.

Concretely, through the lens of viewing a pretrained LLM as a reward function,
we propose a new fine-tuning algorithm, Inclusive Learning From Human Feedback
(ILHF), that offers two key advantages over current RLHF approaches. Firstly,
from a computational perspective, ILHF avoids the need for further downstream
application of policy-gradient methods [75] in order to align LLM responses to
human preferences. Secondly, from a statistical perspective and as our
method’s name suggests, the LLMs resulting from ILHF are inclusive [8] and,
therefore, demonstrably converge to the preferred population response
distribution over the course of fine-tuning; this stands in stark contrast to
the agglomerative models that arise from the standard RLHF approach, which are
encouraged to place all probability mass on a singular, “best” response that
is preferred by the majority of the population. Beyond empirical results that
validate the emergence of such inclusive and agglomerative models, we further
demonstrate how ILHF, a supervised-learning approach, can still be made more
statistically efficient by leveraging judicious exploration strategies borne
out in the reinforcement-learning literature [48].

The paper proceeds as follows: in Section 2 we establish notation, review the
current RLHF pipeline, and briefly present empirical results on a didactic
example to highlight the difference between inclusive and agglomerative LLMs.
We then proceed to outline ILHF in Section 3 followed by details of our
experimental protocol in Section 4 and a discussion of empirical results in
Section 5.

##  2 Preliminaries

###  2.1 Notation

For any natural number $N\in\mathbb{N}$, we denote the index set as
$[N]\triangleq\\{1,2,\ldots,N\\}$. For any arbitrary set $\mathcal{Z}$, we use
the Kleene plus $\mathcal{Z}^{+}$ to denote the set of all sequences of length
at least one formed by elements of $\mathcal{Z}$. Orthogonally, we use
$\Delta(\mathcal{Z})$ to denote the set of all probability distributions
supported on $\mathcal{Z}$. At the most abstract level, a LLM is an
autoregressive mapping that, given a current sequence of tokens, generates a
probability distribution over next tokens; modern applications of LLMs often
elide this low-level mechanistic view of these models and instead adopt the
more holistic perspective that a LLM maps an input prompt to a response, both
of which are variable-length sequences of tokens. If $\mathcal{V}$ is the
vocabulary or set of all possible tokens, then a LLM is represented as a
mapping $\pi_{\phi}:\mathcal{V}^{+}\rightarrow\Delta(\mathcal{V})$
parameterized by $\phi\in\Re^{d}$ where some initial, non-empty sequence of
tokens from $\mathcal{V}$ (constituting a prompt) and subsequently sampled
tokens for the response are autoregressively passed back into $\pi$ as inputs
to generate the next-token distribution. With a slight abuse of notation, we
use $\overline{\pi}_{\phi}:\mathcal{V}^{+}\rightarrow\Delta(\mathcal{V}^{+})$
to denote the associated mapping from an input prompt to a distribution over
full, complete responses.

###  2.2 Reinforcement Learning from Human Feedback (RLHF)

Current approaches to RLHF [76, 66] are characterized by three distinct
phases: pretraining, reward model learning, and fine-tuning. Pretraining is
facilitated by curating a large dataset of $N\in\mathbb{N}$ prompt-response
pairs $\mathcal{D}=\\{(\overline{X}_{i},\overline{Y}_{i})\\}_{i=1}^{N}$ where
$\overline{X}_{i},\overline{Y}_{i}\in\mathcal{V}^{+}$, typically representing
unstructured text data scraped from the Web. Pretrained LLM parameters
$\phi^{\mathrm{pre}}\in\Re^{d}$ are obtained through the standard supervised
language-modeling objective which, in this context, aligns with the classic
behavioral cloning [11, 68] loss function used widely in imitation learning:
$\mathcal{L}^{\mathrm{pre}}(\phi)=-\frac{1}{N}\sum\limits_{i=1}^{N}\sum\limits_{j=1}^{\lambda(i)}\log\left(\overline{\pi}_{\phi}(\overline{Y}_{i,j}\mid\overline{X}_{i},\overline{Y}_{i,1:j-1})\right),$
where $\lambda(i)$ denotes the length of the $i$th response,
$\overline{Y}_{i}$. The challenge posed after the completion of pretraining is
that the unstructured text data curated in $\mathcal{D}$ is only an
approximation to proper, natural text data that end users want and expect from
a LLM; this is a direct consequence of quickly collating $\mathcal{D}$ by
scraping the Internet. Moreover, beyond these initial syntactic issues, such
Web sources are also fraught with errors and factual inaccuracies that need to
be corrected as well. Oftentimes, these errors can be easily identified and
remedied by human evaluators though the challenge lies in propagating such
corrections completely throughout the vast space of possible prompts and
responses.

Since the acquisition of feedback is the limiting reactant that inhibits
scalability, a reward model is trained over the course of $H\in\mathbb{N}$
rounds to emulate human feedback obtained by iteratively fetching a single
prompt $X_{i}$, sampling two random responses
$Y_{i}^{A},Y_{i}^{B}\sim\overline{\pi}_{\phi^{\mathrm{pre}}}(\cdot\mid
X_{i})$, and then querying a human evaluator for a binary indicator
$L_{i}\in\\{0,1\\}$ that communicates their preference (or lack thereof) for
the first response $Y_{i}^{A}$. An external reward model
$r_{\psi}:\mathcal{V}^{+}\times\mathcal{V}^{+}\rightarrow\Re$ parameterized by
$\psi\in\Re^{m}$ can then be trained via supervised learning by minimizing
$\mathcal{L}^{\mathrm{reward}}(\psi)=-\frac{1}{H}\sum\limits_{i=1}^{H}\mathrm{LogSigmoid}\Big{(}R_{\psi}\big{(}X,Y_{i}^{A},Y_{i}^{B},L_{i}\big{)}\Big{)},$
where

$R_{\psi}\big{(}X,Y_{i}^{A},Y_{i}^{B},L_{i}\big{)}=\begin{cases}r_{\psi}(X,Y_{i}^{A})-r_{\psi}(X,Y_{i}^{B})&\text{if
$L_{i}=1$}\\\
r_{\psi}(X,Y_{i}^{B})-r_{\psi}(X,Y_{i}^{A})&\text{otherwise}\end{cases}.$

With the fully-trained reward model $r_{\psi^{\star}}$ in hand, subsequent
prompts can have their corresponding responses aligned to human preferences
via reinforcement learning but without the need for laboriously querying a
live human labeler. Specifically, for each of $T\in\mathbb{N}$ fine-tuning
prompts $X_{1},\ldots,X_{T}$, one can sample two responses
$Y_{t,A},Y_{t,B}\sim\overline{\pi}_{\phi}(\cdot\mid X_{t})$, obtain a
synthetic human feedback signal $L_{t}(A,B)$ based on
$r_{\psi^{\star}}(X_{t},Y_{t,A})$ as well as
$r_{\psi^{\star}}(X_{t},Y_{t,B})$, and apply policy-gradient methods [86, 80,
40, 53, 75] in order to maximize
$\mathcal{J}(\phi)=\mathbb{E}_{X_{t},Y_{t,A},Y_{t,B}}\left[L_{t}(A,B)\right],\forall
t\in[T]$. Naturally, as this is an objective for fine-tuning the LLM, initial
policy parameters are set to be $\phi^{\mathrm{pre}}$. The default standard
choice for carrying out this policy optimization in the existing literature is
Proximal Policy Optimization (PPO) [75].

###  2.3 Inclusive vs. Agglomerative AIs

Figure 1: ILHF succeeds and REINFORCE+KL penalty fails to match human response
probability.

By design, a LLM trained via the RLHF pipeline as outlined in the previous
section learns to emit responses that maximize the likelihood of being
preferred by human evaluators. Consequently, a RLHF model generating responses
$Y_{t,A}$ across $T$ evaluation prompts $X_{1},\ldots,X_{T}$ would likely be
preferred (or be designated at least as good) as the alternative responses
$Y_{t,B}$ of some other model $B$ according to the criterion
$\sum\limits_{t=1}^{T}L_{t}(A,B)$. Unfortunately, as discussed and analyzed in
Arumugam et al. [8], LLMs that are preferred under this criterion are
agglomerative and allow for per-prompt response distributions that place all
probability mass on a hypothetical “best” response which is simply preferred
by the majority of human evaluators sampled from a given population (see
Theorem 2 of Arumugam et al. [8]). Without delving into the details of the
theoretical argument for this result, a simple intuition is that the fine-
tuning phase of the RLHF pipeline operates as a contextual bandit [42], for
which there always exists an optimal policy that is a Dirac delta distribution
on the optimal arm with highest expected mean reward, for each context. Not
only does such a degenerate response distribution qualitatively fail to
reflect the diverse interests and preferences of the overall population, but
it also quantitatively precludes further downstream gradient-based
optimization to redress the issue or cater to any shifts in the desired
response distribution altogether. In contrast to an agglomerative model, one
might instead favor an inclusive model that strives to preserve the full
population response distribution. The primary contribution of this paper is a
fine-tuning algorithm that leverages feedback signals derived from this
preferred response distribution to yield such inclusive models.

To concretize the issues surrounding agglomerative models and to verify that
such problems do manifest from current RLHF practice, we report results for a
toy experiment using an extension of the simple, didactic example discussed in
Section 2.2 of Arumugam et al. [8]. The example consists of multiple rounds of
fine-tuning performed on exactly one prompt and where a response is a single
token from $\\{-1,1\\}$. Initially, the model was pretrained to emit $-1$ with
probability 0.77 and $1$ with probability 0.23. For fine-tuning, the desired
population response distribution prefers response $-1$ with probability
$\frac{2}{3}$ while response $1$ is favored with probability $\frac{1}{3}$.

For such a small-scale and simple problem, we capture the essence of the RLHF
fine-tuning process through policy search by using REINFORCE [86], rather than
PPO, along with KL-regularization towards the pretrained model response
distribution; this implies that we explicitly forego the benefits of variance
reduction that come from a critic as well as the potential for faster
convergence through off-policy policy gradient updates. Results are provided
in Figure 1 varying the value of the KL-regularization coefficient. The
primary observation is that as the REINFORCE fine-tuning updates induce an
agglomerative model that emits the preferred token near-deterministically, the
less-preferred token probability decays under the fine-tuned response
distribution. Moreover, since the initial pretraining distribution
underestimates the preference of the less-desired token, intensifying the KL-
regularization can only halt this undesired behavior by pinning the model to
the pretraining response distribution, but cannot otherwise alleviate the
issue and recover the desired response distribution. In the next section, we
introduce our ILHF fine-tuning approach that is also pictured in Figure 1 and
learns an inclusive model to emit the less-preferred token with the correct
probability.

##  3 Approach

In this section, we outline the precise manner in which LLMs violate the
agent-environment interface typically observed throughout the reinforcement-
learning literature before introducing an alternative approach to RLHF fine-
tuning that obviates the need for separate reward learning and policy
optimization phases. While the resulting ILHF optimization does constitute a
supervised learning problem, we proceed to introduce an augmented fine-tuning
procedure that leverages solution concepts for efficient exploration in
reinforcement learning.

###  3.1 Shattering the Agent-Environment Interface

We may formalize the sequential decision-making problem encapsulated by a LLM
as a finite-horizon, episodic Markov Decision Process (MDP) [14, 69] defined
by
$\mathcal{M}=\langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{T},\beta,H\rangle$.
Specializing these MDP components to the language modeling problem, we may
observe that the state space $\mathcal{S}=\mathcal{V}^{+}$ represents a
sampled prompt as well as the current response generated thus far, the action
space $\mathcal{A}=\mathcal{V}$ is the vocabulary of all possible tokens the
LLM may generate, the initial state distribution
$\beta\in\Delta(\mathcal{V}^{+})$ represents an arbitrary distribution over
prompts whose responses are adjusted over the course of the fine-tuning
process, and the horizon $H\in\mathbb{N}$ is the maximum allowed response
length which still enables variable-length responses shorter than $H$, akin to
an episode of an MDP where the agent transitions to an absorbing terminal
state before exhausting all $H$ steps. Naturally, a particular LLM with
parameters $\phi\in\Re^{d}$ embodies a policy of this MDP
$\pi_{\phi}:\mathcal{S}\rightarrow\Delta(\mathcal{A})$. All that remains is to
define the reward function
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\Re$ providing evaluative
feedback signals to the agent and the deterministic transition function
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ yielding next
states for each state-action pair.

Notably, the mechanics by which a single episode unfolds in this MDP violates
the standard agent-environment interface, as the agent itself is a simulator
that can sample rollouts for any given prompt. At the start of each episode, a
new prompt is sampled $s_{1}\sim\beta$ and, for each timestep $h\in[H]$, a LLM
samples a next token $a_{h}\sim\pi_{\phi}(\cdot\mid s_{h})$ before appending
it to the current response yielding a deterministic next state
$s_{h+1}=\mathcal{T}(s_{h},a_{h})$. More importantly, the fine-tuning approach
introduced in the next section capitalizes on the realization that a suitable
reward function for MDP $\mathcal{M}$ can be induced directly from the policy
itself as $\mathcal{R}_{\phi}(s,a)=\log\left(\pi_{\phi}(a\mid s)\right)$. This
again breaks the standard interface whereby rewards are computed within the
confines of the environment and direct updates to policy parameters $\phi$ do
not explicitly change the underlying reward function. While the idea of
inducing a reward function (or, more generally, a cumulant [81]) from a policy
and vice versa is not new [13], the implications for LLMs in particular stand
to be quite profound; namely, it establishes a direct relationship between
reward learning and policy optimization that the current RLHF paradigm
segregates into distinct phases. In the next section, we provide a novel fine-
tuning algorithm that leverages the equivalence between reward learning and
policy optimization to consolidate these latter two stages of the RLHF
pipeline.

###  3.2 ILHF: A New Fine-Tuning Algorithm

The previous section sets the stage for interpreting the output of the LLM
pretraining phase as producing reward function parameters
$\phi^{\mathrm{pre}}$ which analogously function as policy parameters. As
pretraining typically occurs with a dataset that represents a crude
approximation to proper written language (such as text scraped widely from the
Internet), the corresponding reward function
$\mathcal{R}_{\phi^{\mathrm{pre}}}(s,a)=\log\left(\pi_{\phi^{\mathrm{pre}}}(a\mid
s)\right)$ is misspecified and the associated reward-maximizing policy
$\overline{\pi}_{\phi^{\mathrm{pre}}}$ does not accurately reflect the desired
response distribution. This begets the need for a loss function that refines
reward function parameters to more accurately depict response preferences and,
in doing so, refine the LLM policy parameters to induce a response
distribution reflective of those preferences.

To that end, we offer the following loss function for optimizing reward
function parameters and refining the LLM response distribution jointly. For
any sampled prompt $X\sim\beta$, denote two i.i.d. sampled responses as
$Y_{i}^{A},Y_{i}^{B}\sim\overline{\pi}_{\phi}(\cdot\mid X)$ which are judged
by a human evaluator according to $L_{i}\in\\{0,1\\}$. Define the binary
probability distribution
$\mathcal{P}_{i}=\begin{bmatrix}L_{i},&1-L_{i}\end{bmatrix}$ induced from the
human evaluator. Then, we may induce a complementary distribution over the two
sampled LLM responses as

$\displaystyle\mathcal{Q}_{i}^{\phi}$
$\displaystyle=\text{Softmax}\left(\left[\mathcal{R}_{\phi}(X,Y_{i}^{A}),\mathcal{R}_{\phi}(X,Y_{i}^{B})\right]\right)=\begin{bmatrix}\frac{\pi_{\phi}(Y_{i}^{A}\mid
X)}{\pi_{\phi}(Y_{i}^{A}\mid X)+\pi_{\phi}(Y_{i}^{B}\mid
X)},&\frac{\pi_{\phi}(Y_{i}^{B}\mid X)}{\pi_{\phi}(Y_{i}^{A}\mid
X)+\pi_{\phi}(Y_{i}^{B}\mid X)}\end{bmatrix},$

in accordance with the Bradley-Terry model for pairwise comparisons [16].
Then, our proposed fine-tuning loss aims to minimize the KL-divergence between
the induced human label distribution $\mathcal{P}$ and the current LLM
response preference distribution $\mathcal{Q}_{\phi}$:
$\mathcal{L}^{\mathrm{ILHF}}(\phi)=\mathbb{E}_{X_{i}}\left[D_{\mathrm{KL}}\left(\mathcal{P}_{i}\mid\mid\mathcal{Q}_{i}^{\phi}\right)\right]$111We
use the standard convention that $0\cdot\log(0)=0$.. In Section 4, we provide
an empirical confirmation that fine-tuning via ILHF does indeed yield an
inclusive model by converging to the desired response distribution.

###  3.3 Efficient Exploration

While our proposed ILHF loss function can be optimized via traditional
supervised-learning techniques, a LLM model can only utilize human feedback
for the responses it generates, akin to a reinforcement-learning agent that
may only perceive reward signals for the actions executed under its own
policy. Given the vastness of the space of possible responses for each prompt,
this implies that a LLM model must also contend with the challenge of
exploration in its MDP. Fortunately, the reinforcement-learning literature has
long-studied the problem of exploration and developed a wide range of solution
concepts with varying degrees of statistical efficiency and computational
tractability [36, 17, 35, 9, 77, 33, 59, 58, 3, 34, 49]. While future work
will likely benefit from a deep and meticulous investigation of which concepts
from reinforcement learning might fruitfully transfer over to improve the
efficiency of LLM fine-tuning, we here offer one concrete suggestion through
the use of uncertainty-based exploration.

Briefly, one principled exploration strategy represents and maintains an
agent’s epistemic uncertainty [21] in the underlying MDP or value function and
uses it as a quantitative signal to foster exploratory behaviors in a manner
that is both provably-efficient [56, 61, 58, 3, 63, 50] and computationally-
scalable [60, 62, 54, 23, 64, 65]. While the numerous flavors of uncertainty-
based explorations schemes also appear with varying degrees of sophistication
[72, 73, 50], we leave an investigation of more complex candidates to future
work and, instead, focus our attention on those grounded in Thompson sampling
[83, 74], which is both computationally simple and widely used in practice
[44, 18, 28]. Posterior-sampling methods that employ Thompson sampling for
reinforcement learning [78, 59, 57, 1, 3, 58, 49] operate in each time period
by drawing a single statistically-plausible hypothesis for optimal behavior
from the agent’s beliefs and proceed to act optimally with respect to the
single sample as if it reflects reality. The simplest candidate for
maintaining an agent’s beliefs and refining them as data accumulates is via
ensemble sampling [48] which maintains a finite number of randomly-initialized
models that can be sampled in each time period and optimized via bootstrapped
mini-batches of data [24]. Specifically, for each prompt, an Ensemble-ILHF
agent samples one model from its ensemble to generate the two responses which
induce human feedback.

##  4 Experiment Setup

We discuss in this section how we simulate agent-human interactions for all
agents used in our empirical evaluation. A reader familiar with reinforcement
learning should interpret this section as describing a particular choice of
MDP. While language models typically involve a large numbers of tokens in the
vocabulary, we will restrict our scope to exactly two: $-1$ and $1$. Typical
language models also include a STOP token in the vocabulary that allows for
variable-length responses; instead, our synthetic language simply assumes that
all response lengths are homogeneous. This is intended to offer a microcosm
for studying methods that process and generate tokenized language data. While
a reader may feel discouraged at the prospect of results obtained at such a
scale orders of magnitude smaller than what is currently driving practical and
deployed models in this space, our goal throughout this work is to leverage
such simplicity in order to convey maximal clarity. Moreover, if ILHF agents
can be shown to bear fruit in such a basic setting, the potential benefits of
tackling the same challenges we outline at a larger scale could be far more
substantial.

###  4.1 A Token-Generating Process & Pretraining

Consider a synthetic, stateful token-generating process that is governed by a
vector $\mu\in\Re^{d}$, a matrix $W\in\Re^{d\times d}$, and a vector
$U\in\Re^{d}$. The process begins in an initial state $S_{0}\in\Re^{d}$
sampled from a fixed distribution and, at any time $t$, given the state
$S_{t}\in\Re^{d}$ of this process, a next token $X_{t+1}\in\\{-1,1\\}$ and
state $S_{t+1}\in\Re^{d}$ are generated according to

$\displaystyle X_{t+1}$ $\displaystyle=\left\\{\begin{array}[]{ll}1&\text{w.p.
}\frac{\exp\left(\mu^{\top}S_{t}\right)}{1+\exp\left(\mu^{\top}S_{t}\right)}\\\
-1&\text{otherwise}\end{array}\right.,\qquad S_{t+1}=\tanh(WS_{t}+UX_{t+1}).$

Note that $\tanh$ is applied component-wise. This generating process can be
interpreted as a recurrent neural network with a single hidden layer and a
softmax output; indeed, as discussed in the Appendix, all the agents we
evaluate adhere to this exact network architecture, only with a larger hidden
dimension. To keep things simple, all prompts and responses will be of a fixed
length $\tau\in\mathbb{N}$. One might wonder why this particular token
generating process is worth further study. While it is true that there are
numerous stateful stochastic processes one could use to model token
generation, the one presented above is clearly among the simpler choices while
still retaining a sufficient degree of nontrivial structure.

We offer a preliminary experiment to make the preceding statement precise;
namely, that the output token $X_{t}$ at each time $t$ exhibits long-term
dependencies on the history of tokens. Consequently, it suffices for the next
output logit, i.e., the probability of the next token being 1, to depend on a
relatively long history of logits, since the distribution of the next token is
completely determined by the next logit. To demonstrate the dependence, we
plot the autocorrelation function of logits in Figure 2 (please see the
Appendix for the exact autocorrelation formula).

Figure 2: Long-tailed dependence

As the goal of our token generating process is to serve as a simplified
surrogate to natural language, an important distinction arises between ideal
text and shadow text. One should think of ideal text as exemplary of well-
written text that perfectly aligns with the standards of human evaluators.
This excludes, at the bare minimum, any garbled or malformed token sequences,
and more broadly, those that are in discordance with ideal human-level
responses; such maligned token sequences are instances of shadow text,
(sometimes crude) approximations of ideal text which, for example, appear
frequently throughout the Internet and are often intertwined or alongside
ideal text.

We will think of our aforementioned token generating process as one that
yields ideal text so as to be emblematic of proper, linguistically-correct
natural language reflective of human preferences. Modern approaches to
pretraining, however, through their reliance on textual data scraped from the
Internet, do not rely on text generated by this process but instead on shadow
text, an approximation which a downstream agent must use for learning. For
example, the text may not be written by an eloquent writer, may express
harmful thoughts, or espouse erroneous responses. Maintaining fidelity to this
reality, we assume that this shadow process uses the same matrix
$W\in\Re^{d\times d}$ and vector $U\in\Re^{d}$ but an approximation
$\theta\in\Re^{d}$ of the vector $\mu$.

We conclude this section with a brief discussion of how pretraining with
shadow text transpires, clarifying what information all agents in our
evaluation will be initialized with at the start of fine-tuning.The
pretraining dataset $\mathcal{D}=\\{X_{i,1:2\tau}\\}_{i=1}^{N}$ consists of
documents generated according to the shadow process. Each document can be seen
as a concatenated prompt-response pair. Using this dataset, an initial policy
$\phi$ is pretrained to maximize the log-probability of predicting the next
tokens given the corresponding preceding strings
$\min_{\phi}\mathcal{L}^{\text{pre}}(\phi)=\frac{1}{N}\sum_{i=1}^{N}\left(\log\pi_{\phi}(X_{i,1})+\sum_{t=2}^{2\tau}\log\pi_{\phi}(X_{i,t}|X_{i,1:t-1})\right),$
in a manner that resembles the behavioral cloning [11, 68] algorithm used for
imitation learning. We denote $\phi^{\text{pre}}$ as the policy parameters
obtained by pretraining.

###  4.2 Simulating Agent-Human Interactions

We take each $i$th prompt $X_{i,1:\tau}$ to be sampled independently from our
token generating process but with $\theta$ taking the place of $\mu$. During
training, prompts to language models are often truncated from streams of
shadow text; when deployed in practice, one also does not assume that prompts
are ideal text free of typographical errors. These are echoed by our use of
the approximate parameter $\theta$ in prompt generation. Yet another reason
that motivates using the shadow process to generate prompts is that, the agent
ought to obtain all relevant information about the error $\theta-\mu$ from
human feedback. Thus, prompts should not reveal information about the
parameter $\mu$ of the ideal process.

We consider binary human feedback, where each bit indicates a preference
between two responses. Given the $i$th prompt $X_{i,1:\tau}$, we generate an
associated state sequence $S_{i,1:\tau}$ according to
$S_{i,t+1}=\tanh(WS_{i,t}+UX_{i,t})$. Similarly, for each $i$ and
$b\in\\{0,1\\}$, we let $S^{(b)}_{i,0}=S_{i,\tau}$ and generate a state
sequence $S^{(b)}_{i,1:\tau}$ associated with response $b$ according to
$S^{(b)}_{i,t+1}=\tanh(WS^{(b)}_{i,t}+UX^{(b)}_{i,t})$.

For each $i$, denote the likelihood function that the ideal text generating
process assigns to each response $b\in\\{0,1\\}$ by
$\ell_{i,b}(\mu)=\prod_{t=1}^{\tau}\left((1-X^{(b)}_{i,t})\frac{1}{1+\exp(\mu^{\top}S^{(b)}_{i,t})}+X^{(b)}_{i,t}\frac{\exp(\mu^{\top}S^{(b)}_{i,t})}{1+\exp(\mu^{\top}S^{(b)}_{i,t})}\right).$

Then, the binary preference $B_{i}$ of a random individual is sampled
according to $B_{i}=\begin{cases}0&\text{with probability
}\frac{\ell_{i,0}(\mu)}{\ell_{i,0}(\mu)+\ell_{i,1}(\mu)}\\\ 1&\text{with
probability
}\frac{\ell_{i,1}(\mu)}{\ell_{i,0}(\mu)+\ell_{i,1}(\mu)}\end{cases},$ which is
the classic Bradley-Terry model for the human preference between response
$X^{(1)}_{i,t}$ and $X^{(0)}_{i,t}$. The human feedback we consider is based
on the ideal process that employs parameter $\mu$, in spirit with the goal of
aligning language model outputs to human preference. We assume that the human
annotator communicates relatively high-quality signals, with the only error
stemming from random sampling.

###  4.3 Evaluation Metrics

A key advantage of studying a simple data-generating process is the ability
to, for any prompt generated as described above, tractably compute the KL-
divergence between the ideal distribution of responses and distribution of
responses produced by any agent as an evaluation metric. Concretely, the ideal
process generates a set of independently sampled sequences
$\\{X_{i,1:2\tau}\\}_{i=1}^{N}$, which can also be viewed as concatenated
prompt-response pairs, as well as the corresponding set of state sequences
$S_{i,1:2\tau}$. Like the token generating process, a stateful agent then
generates corresponding agent state sequences
$\\{\widehat{S}_{i,1:2\tau}\\}_{i=1}^{N}$. For each $i$, let
$\widehat{\ell}_{i}(\mu)=\prod_{t=\tau+1}^{2\tau}\left((1-X_{i,t})\frac{1}{1+\exp(\mu^{\top}S_{i,t})}+X_{i,t}\frac{\exp(\mu^{\top}S_{i,t})}{1+\exp(\mu^{\top}S_{i,t})}\right)$
denote the likelihood function that the ideal text generating process assigns
to the $i$-th response, and let $\widehat{\ell}_{i}(\phi)$ denote the
likelihood function that the agent’s model assigns to the $i$-th response
which is identical to $\widehat{\ell}_{i}(\mu)$ with $\phi$ swapped for $\mu$
and $\widehat{S}_{i,t}$ in place of $S_{i,t}$. The Monte-Carlo estimation of
the KL-divergence can then be expressed as
$\mathcal{S}^{\text{KL}}(\phi)=\frac{1}{N}\sum_{i=1}^{N}\left(\ln\widehat{\ell}_{i}(\mu)-\ln\widehat{\ell}_{i}(\phi)\right).$
Note that with the token-generating process introduced in Section 4.1, there
exist an agent that attains zero KL-divergence.

###  4.4 Inclusive Agents

As a concrete instantiation of our ILHF fine-tuning algorithm, we consider an
agent that computes a maximum a posteriori (MAP) estimate of $\phi$ at each
round of human interaction. The agent initializes its parameter $\phi_{0}$
with $\phi^{\text{pre}}$; for the $k$th interaction, it first samples
responses from the token generating process with parameter $\phi_{k}$ and then
aims to minimize $\mathcal{L}^{\mathrm{ILHF}}(\phi)$ which, in the context of
our particular token-generating process, simplifies as
$\mathcal{L}^{\mathrm{ILHF}}(\phi)=-\frac{1}{N}\sum_{i=1}^{N}\left(\ln\ell_{i,b_{i}}(\phi)-\ln(\ell_{i,0}(\phi)+\ell_{i,1}(\phi))\right).$
This agent operates in a greedy fashion, using the MAP estimate of parameters
to generate responses that humans can subsequently rate. Note that in our
simplified token generating process, we take only one optimization step
between interactions, whereas one may engage a significantly larger number of
updates in larger models. The full procedure is shown as Algorithm 1 in the
Appendix.

As an alternative ILHF agent design that leverages exploration strategies to
accelerate convergence, we consider a second fine-tuning algorithm, Ensemble-
ILHF, that fits an ensemble of models that approximates a posterior
distribution. Before fine-tuning starts, an ensemble of parameters are drawn
independently from a normal distribution centered around the pretrained
parameter $\phi^{\text{pre}}$. The agent’s belief about the variance of the
posterior distribution is captured by a covariance matrix $\Sigma$ which, over
the course of fine-tuning, is further conditioned on the observed human
feedback data. The full procedure for this ensemble agent is shown as
Algorithm 2 in the Appendix.

##  5 Results and Discussion

In this section, we present two sets of computational studies that compare
REINFORCE, ILHF, and Ensemble-ILHF agents. The first set of experiments aim to
illustrate how our approach produces an inclusive agent for the didactic
example introduced in Section 2.3. The second set of experiments centers
around the token generating process introduced in the previous section, again
demonstrating how our fine-tuning procedure yields an inclusive model that
captures the desired response distribution while also highlighting the
benefits of efficient, uncertainty-based exploration schemes.

In both experiments, all agents follow the same pretraining protocol with
1,000 pretraining samples generated using a shadow process with perturbation
variance of 0.3 from the ideal process, yielding identical parameters
$\phi^{\mathrm{pre}}$ at the start of fine-tuning for all agents. All agents
use the Adam optimizer [37] with a learning rate of 0.001 for both pretraining
and finetuning. In each finetuning episode, exactly 64 prompts are provided to
the agents to respond to and gain feedback from our synthetic human labels.
All error bars and shaded areas correspond to 1 standard error over 20 seeds.

###  5.1 Didactic Experiment

Figure 3 provides a continuation of the preliminary results shown in Figure 1
for the didactic example of Section 2.3 only now showing the KL-divergence
metric introduced in Section 4.3. We first pretrain for 200 epochs before
starting the finetuning phase. Our KL-divergence metric shows that ILHF is
able to learn the ground-truth token distribution, whereas all REINFORCE
agents (acting as proxies for the current RLHF paradigm) fail regardless of
the KL penalty scale. Notably, the gap between these RLHF agents and our
proposed ILHF agent is entirely a function of the divergence between the
pretraining and desired response distributions; thus, whenever fine-tuning
occurs to correct a significant shift between the two response distributions
than what is shown here, the gap between ILHF and RLHF could be significantly
larger.

Figure 3: Reinforce+KL penalty is not inclusive

###  5.2 Efficient Exploration via Ensemble Sampling

Our next set of experiments follows the experiment setup introduced in Section
4. All agents are first pretrained over 20 episodes, then finetuned over 100
episodes of human interactions, as represented by the ideal data generating
process. Note that due to the online nature of the REINFORCE algorithm, during
each interaction episode, only one gradient update is performed for each agent
model. In Figure 4, we compare the KL-divergence between the response
distribution of the ideal process and the response distribution of various
agents. In addition to the KL-penalized REINFORCE agents and ILHF, we also
examine the performance of ILHF equipped with an ensemble of models (also
called particles) to facilitate exploration, as is introduced in Section 3.3.
The figure shows that only ILHF and Ensemble-ILHF can learn the ideal process,
while all REINFORCE agents diverge from it regardless of the KL penalty scale.
At the same time, using an ensemble to account for epistemic uncertainty in an
inclusive agent significantly accelerates learning. We provide an ablation
study on problem-specific parameters in the Appendix.

Although our synthetic data-generating process allows us to exactly evaluate
the KL-divergence between the response distribution of the ideal process and
that of an agent, it is not viable to compute in general. In practice, a head-
to-head competition between two agents is typically involved to determine
which one performs better. In a spirit to echo such a procedure, we also carry
out a head-to-head competition between our best Ensemble-ILHF agent with 50
particles against all other agents considered in this experiment after
finetuning. Since our goal is to select inclusive agents, instead of employing
the agglomerative criterion discussed in Section 2.3, we consider the
inclusive score introduced in [8] which selects the agent that better
represents the distribution of human preferences. For each agent being
compared to the Ensemble-ILHF agent with 50 particles, we perform a
normalization procedure over the inclusive scores that produces a single
statistic, inclusive score ratio. The competing agent wins if the ratio is
greater than 1 and loses otherwise. Figure 5 indicates that Ensemble-ILHF 50
consistently outperforms all other agents in head-to-head combat with
statistically significant differences.

|
---|---

Figure 4: KL-divergence vanishes with ILHF but remains large with REINFORCE
Figure 5: Head-to-head comparison of various agents against Ensemble-ILHF 50

Figure 6: Ensembles improve ILHF, which greatly improves on REINFORCE with KL
penalty

##  6 Conclusion

Excitement around the capabilities and prospects of LLMs is a burgeoning area
of machine learning, whose prevalence will only continue to grow. Notably,
these successes are driven by the RLHF paradigm, which hinges on learning a
separate reward model that sits distinct from the LLM itself. To mitigate the
challenges of such agglomerative models that collapse towards a single best
response, we have proposed Inclusive Learning from Human Feedback (ILHF) as an
alternative LLM fine-tuning approach which leverages insights from the field
of reinforcement learning to produce inclusive models that preserve the
population response distribution. Future work in this area may benefit from
incorporating other reinforcement learning ideas to design and optimize LLMs
in a more statistically-efficient manner.

## References

  * Abbasi-Yadkori and Szepesvari [2014] Yasin Abbasi-Yadkori and Csaba Szepesvari.  Bayesian optimal control of smoothly parameterized systems: The lazy posterior sampling algorithm.  _ArXiv preprint arXiv:1406.3926_ , 2014.
  * Abel et al. [2017] David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans.  Agent-agnostic human-in-the-loop reinforcement learning.  _arXiv preprint arXiv:1701.04079_ , 2017.
  * Agrawal and Jia [2017] Shipra Agrawal and Randy Jia.  Optimistic posterior sampling for reinforcement learning: Worst-case regret bounds.  In _Advances in Neural Information Processing Systems_ , pages 1184–1194, 2017.
  * Akrour et al. [2011] Riad Akrour, Marc Schoenauer, and Michèle Sebag.  Preference-based policy learning.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 12–27. Springer, 2011.
  * Akrour et al. [2012] Riad Akrour, Marc Schoenauer, and Michèle Sebag.  APRIL: Active preference learning-based reinforcement learning.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 116–131. Springer, 2012.
  * Akrour et al. [2014] Riad Akrour, Marc Schoenauer, Michèle Sebag, and Jean-Christophe Souplet.  Programming by feedback.  In _International Conference on Machine Learning_ , pages 1503–1511. JMLR. org, 2014.
  * Arumugam et al. [2019] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman.  Deep reinforcement learning from policy-dependent human feedback.  _arXiv preprint arXiv:1902.04257_ , 2019.
  * Arumugam et al. [2022] Dilip Arumugam, Shi Dong, and Benjamin Van Roy.  Inclusive Artificial Intelligence.  _arXiv preprint arXiv:2212.12633_ , 2022.
  * Auer et al. [2009] Peter Auer, Thomas Jaksch, and Ronald Ortner.  Near-optimal regret bounds for reinforcement learning.  In _Advances in Neural Information Processing Systems_ , pages 89–96, 2009.
  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_ , 2022.
  * Bain and Sommut [1999] Michael Bain and Claude Sommut.  A framework for behavioural cloning.  _Machine Intelligence_ , 15(15):103, 1999.
  * Bakker et al. [2022] Michiel A Bakker, Martin J Chadwick, Hannah R Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, and Matthew M Botvinick.  Fine-tuning language models to find agreement among humans with diverse preferences.  _arXiv preprint arXiv:2211.15006_ , 2022.
  * Barreto et al. [2019] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, et al.  The Option Keyboard: Combining skills in reinforcement learning.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Bellman [1957] Richard Bellman.  A Markovian Decision Process.  _Journal of Mathematics and Mechanics_ , pages 679–684, 1957.
  * Bengio et al. [2000] Yoshua Bengio, Réjean Ducharme, and Pascal Vincent.  A Neural Probabilistic Language Model.  _Advances in Neural Information Processing Systems_ , 13, 2000.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E Terry.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Brafman and Tennenholtz [2002] Ronen I Brafman and Moshe Tennenholtz.  R-MAX - A general polynomial time algorithm for near-optimal reinforcement learning.  _Journal of Machine Learning Research_ , 3(Oct):213–231, 2002.
  * Chapelle and Li [2011] Olivier Chapelle and Lihong Li.  An empirical evaluation of Thompson sampling.  In _Advances in Neural Information Processing Systems_ , pages 2249–2257, 2011.
  * Christiano et al. [2018] Paul Christiano, Buck Shlegeris, and Dario Amodei.  Supervising strong learners by amplifying weak experts.  _arXiv preprint arXiv:1810.08575_ , 2018.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Der Kiureghian and Ditlevsen [2009] Armen Der Kiureghian and Ove Ditlevsen.  Aleatory or epistemic? Does it matter?  _Structural Safety_ , 31(2):105–112, 2009.
  * Dragan and Srinivasa [2013] Anca D Dragan and Siddhartha S Srinivasa.  A policy-blending formalism for shared control.  _The International Journal of Robotics Research_ , 32(7):790–805, 2013.
  * Dwaracherla et al. [2020] Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Ian Osband, Zheng Wen, and Benjamin Van Roy.  Hypermodels for exploration.  In _International Conference on Learning Representations_ , 2020.
  * Efron [1982] Bradley Efron.  _The Jackknife, the Bootstrap and Other Resampling Plans_.  SIAM, 1982.
  * El Asri et al. [2016] Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin.  Score-based inverse reinforcement learning.  In _International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016)_ , 2016.
  * Fürnkranz et al. [2012] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park.  Preference-based reinforcement learning: A formal framework and a policy iteration algorithm.  _Machine learning_ , 89(1):123–156, 2012.
  * Glaese et al. [2022] Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_ , 2022.
  * Gopalan et al. [2014] Aditya Gopalan, Shie Mannor, and Yishay Mansour.  Thompson sampling for complex online problems.  In _International Conference on Machine Learning_ , pages 100–108. PMLR, 2014.
  * Griffith et al. [2013] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz.  Policy shaping: Integrating human feedback with reinforcement learning.  _Advances in Neural Information Processing Systems_ , 26, 2013.
  * Henderson et al. [2023] Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang.  Foundation models and fair use.  _arXiv preprint arXiv:2303.15715_ , 2023.
  * Ibarz et al. [2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei.  Reward learning from human preferences and demonstrations in atari.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Isbell et al. [2001] Charles Lee Isbell, Christian R. Shelton, Michael Kearns, Satinder Singh, and Peter Stone.  A social reinforcement learning agent.  In _International Conference on Autonomous Agents_ , 2001.
  * Jaksch et al. [2010] Thomas Jaksch, Ronald Ortner, and Peter Auer.  Near-Optimal Regret Bounds for Reinforcement Learning.  _Journal of Machine Learning Research_ , 11(4), 2010.
  * Jin et al. [2018] Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan.  Is $Q$-learning provably efficient?  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 4868–4878, 2018.
  * Kakade [2003] Sham Machandranath Kakade.  _On the Sample Complexity of Reinforcement Learning_.  PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.
  * Kearns and Singh [2002] Michael Kearns and Satinder Singh.  Near-optimal reinforcement learning in polynomial time.  _Machine Learning_ , 49(2-3):209–232, 2002.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Knox and Stone [2008] W Bradley Knox and Peter Stone.  TAMER: Training an agent manually via evaluative reinforcement.  In _2008 7th IEEE International Conference on Development and Learning_ , pages 292–297. IEEE, 2008.
  * Knox [2012] William Bradley Knox.  _Learning from Human-Generated Reward_.  PhD thesis, The University of Texas at Austin, 2012.
  * Konda and Tsitsiklis [1999] Vijay Konda and John Tsitsiklis.  Actor-Critic Algorithms.  _Advances in Neural Information Processing Systems_ , 12, 1999.
  * Lambert et al. [2022] Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla.  Illustrating Reinforcement Learning from Human Feedback (RLHF), 2022.  URL https://huggingface.co/blog/rlhf.
  * Lattimore and Szepesvári [2020] Tor Lattimore and Csaba Szepesvári.  _Bandit Algorithms_.  Cambridge University Press, 2020.
  * Leike et al. [2018] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg.  Scalable agent alignment via reward modeling: a research direction.  _arXiv preprint arXiv:1811.07871_ , 2018.
  * Li et al. [2010] Lihong Li, Wei Chu, John Langford, and Robert E Schapire.  A contextual-bandit approach to personalized news article recommendation.  In _Proceedings of the 19th international conference on World wide web_ , pages 661–670, 2010.
  * Li et al. [2022a] Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta.  When does differentially private learning not suffer in high dimensions?  _Advances in Neural Information Processing Systems_ , 35:28616–28630, 2022a.
  * Li et al. [2022b] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto.  Large language models can be strong differentially private learners.  In _International Conference on Learning Representations_ , 2022b.
  * Liang et al. [2022] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ , 2022.
  * Lu and Van Roy [2017] Xiuyuan Lu and Benjamin Van Roy.  Ensemble Sampling.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 3260–3268, 2017.
  * Lu and Van Roy [2019] Xiuyuan Lu and Benjamin Van Roy.  Information-Theoretic Confidence Bounds for Reinforcement Learning.  _Advances in Neural Information Processing Systems_ , 32:2461–2470, 2019.
  * Lu et al. [2021] Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, and Zheng Wen.  Reinforcement Learning, Bit by Bit.  _ArXiv preprint arXiv:2103.04047_ , 2021.
  * MacGlashan et al. [2017] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman.  Interactive learning from policy-dependent human feedback.  In _International Conference on Machine Learning_ , pages 2285–2294. PMLR, 2017.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Mnih et al. [2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * O’Donoghue et al. [2018] Brendan O’Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih.  The Uncertainty Bellman Equation and Exploration.  In _International Conference on Machine Learning_ , pages 3836–3845, 2018.
  * OpenAI [2023] OpenAI.  GPT-4 Technical Report.  Technical report, OpenAI, 2023.
  * Osband [2016] Ian Osband.  _Deep Exploration via Randomized Value Functions_.  PhD thesis, Stanford University, 2016.
  * Osband and Van Roy [2014] Ian Osband and Benjamin Van Roy.  Model-based reinforcement learning and the Eluder dimension.  _Advances in Neural Information Processing Systems_ , 27, 2014.
  * Osband and Van Roy [2017] Ian Osband and Benjamin Van Roy.  Why is posterior sampling better than optimism for reinforcement learning?  In _International Conference on Machine Learning_ , pages 2701–2710. PMLR, 2017.
  * Osband et al. [2013] Ian Osband, Daniel Russo, and Benjamin Van Roy.  (More) efficient reinforcement learning via posterior sampling.  _Advances in Neural Information Processing Systems_ , 26:3003–3011, 2013.
  * Osband et al. [2016a] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy.  Deep exploration via Bootstrapped DQN.  In _Advances in Neural Information Processing Systems_ , pages 4026–4034, 2016a.
  * Osband et al. [2016b] Ian Osband, Benjamin Van Roy, and Zheng Wen.  Generalization and exploration via randomized value functions.  In _International Conference on Machine Learning_ , pages 2377–2386, 2016b.
  * Osband et al. [2018] Ian Osband, John Aslanides, and Albin Cassirer.  Randomized prior functions for deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Osband et al. [2019] Ian Osband, Benjamin Van Roy, Daniel J Russo, and Zheng Wen.  Deep Exploration via Randomized Value Functions.  _Journal of Machine Learning Research_ , 20(124):1–62, 2019.
  * Osband et al. [2021] Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, and Benjamin Van Roy.  Epistemic Neural Networks.  _arXiv preprint arXiv:2107.08924_ , 2021.
  * Osband et al. [2023] Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy.  Approximate Thompson Sampling via Epistemic Neural Networks.  _arXiv preprint arXiv:2302.09205_ , 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Pilarski et al. [2011] Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton.  Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning.  In _2011 IEEE International Conference on Rehabilitation Robotics_ , pages 1–7. IEEE, 2011.
  * Pomerleau [1989] Dean A Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In _Advances in neural information processing systems_ , pages 305–313, 1989.
  * Puterman [1994] Martin L. Puterman.  _Markov Decision Processes—Discrete Stochastic Dynamic Programming_.  John Wiley & Sons, Inc., New York, NY, 1994.
  * Ross and Bagnell [2014] Stephane Ross and J Andrew Bagnell.  Reinforcement and imitation learning via interactive no-regret learning.  _arXiv preprint arXiv:1406.5979_ , 2014.
  * Ross et al. [2011] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  * Russo and Van Roy [2014] Daniel Russo and Benjamin Van Roy.  Learning to optimize via information-directed sampling.  _Advances in Neural Information Processing Systems_ , 27:1583–1591, 2014.
  * Russo and Van Roy [2018] Daniel Russo and Benjamin Van Roy.  Learning to optimize via information-directed sampling.  _Operations Research_ , 66(1):230–252, 2018.
  * Russo et al. [2018] Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen.  A Tutorial on Thompson Sampling.  _Foundations and Trends® in Machine Learning_ , 11(1):1–96, 2018.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021, 2020.
  * Strehl et al. [2009] Alexander L Strehl, Lihong Li, and Michael L Littman.  Reinforcement learning in finite MDPs: PAC analysis.  _Journal of Machine Learning Research_ , 10(Nov):2413–2444, 2009.
  * Strens [2000] Malcolm JA Strens.  A Bayesian Framework for Reinforcement Learning.  In _Proceedings of the Seventeenth International Conference on Machine Learning_ , pages 943–950, 2000.
  * Sutton and Barto [1998] Richard S Sutton and Andrew G Barto.  _Introduction to Reinforcement Learning_.  MIT Press, 1998.
  * Sutton et al. [1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.  Policy Gradient Methods for Reinforcement Learning with Function Approximation.  _Advances in Neural Information Processing Systems_ , 12, 1999.
  * Sutton et al. [2011] Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup.  Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction.  In _The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2_ , pages 761–768, 2011.
  * Thomaz and Breazeal [2006] Andrea Lockerd Thomaz and Cynthia Breazeal.  Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance.  In _AAAI_ , volume 6, pages 1000–1005. Boston, MA, 2006.
  * Thompson [1933] William R Thompson.  On the likelihood that one unknown probability exceeds another in view of the evidence of two samples.  _Biometrika_ , 25(3/4):285–294, 1933.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Warnell et al. [2018] Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone.  Deep TAMER: Interactive agent shaping in high-dimensional state spaces.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Williams [1992] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ , 8(3):229–256, 1992.
  * Wilson et al. [2012] Aaron Wilson, Alan Fern, and Prasad Tadepalli.  A Bayesian approach for policy learning from trajectory preference queries.  _Advances in Neural Information Processing Systems_ , 25, 2012.
  * Wirth et al. [2016] Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann.  Model-free preference-based reinforcement learning.  In _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.
  * Ziegler et al. [2019] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_ , 2019.

##  Appendix A Related Work

The wealth of knowledge instilled within a generative AI during pretraining is
only as good as the procedure developed to judiciously and selectively
operationalize that information for downstream tasks of interest.
Reinforcement learning [79] has emerged as the critical conduit between these
pretrained and finetuned models, with dedicated procedures for bridging
between them appearing most recently under the name Reinforcement Learning
from Human Feedback (RLHF) [66, 41].

Of course, there is a longstanding history around the incorporation of
individual humans into the reinforcement-learning process, spanning a
multitude of possible entry points into the underlying formalism [2],
traditionally represented as a Markov Decision Process (MDP)[14, 69]. While
many rich forms of human interaction and intervention are possible, like
action labeling [71, 29, 70] or shared autonomy [22], the RLHF paradigm falls
in with a broader group of work that tethers human feedback to the reward
signal observed by an agent. Preliminary work on this topic exclusively
considers human feedback as a exchangeable proxy to the reward signal [32, 82,
38, 67, 39, 85], while subsequent work has gone on to question this base
premise and explored alternative characterizations through the policy-
dependent advantage function [51, 7] or as an indicator of human preferences
[4, 87, 5, 26, 6, 88, 25]. With recent progress around generative AIs being
driven largely by the capacity for successful deployment of these agents in
applications, the latter paradigm has emerged as a default choice for the
alignment of complex, monolithic models to human preferences [20, 31, 43, 19,
89, 66, 10, 27, 12].

##  Appendix B Didactic Example

In an effort to echo the prevailing practice in RLHF, we compare to an
alternative baseline, namely the REINFORCE agent [86]. For the $i$-th prompt
and its response labeled $b\in\\{0,1\\}$, the usual Monte-Carlo policy
gradient is given by

$\nabla_{\phi}\ln\ell_{i,b}(\phi)\cdot R_{i,b},$ (1)

where $R_{i,b}$ is the reward associated with this prompt-response pair. To
keep things simple, we sidestep training a separate reward model, and simply
supply the agent with an oracle reward model. This reward model is represented
by the ideal process that produces human feedback. Specifically, the $i$-th
prompt and its response labeled $b\in\\{0,1\\}$ earns reward

$R_{i,b}=\ln\ell_{i,b}(\mu),$

where $\ell_{i,b}(\mu)$ is the likelihood function the ideal process assigns
to response $b$ given prompt $i$. The training procedure for this agent is
similar to Algorithm 1, where the loss function is replaced by

$\displaystyle\mathcal{L}^{\text{REINFORCE}}(\phi)$
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\left(\ln\ell_{i,0}(\phi)\cdot
R_{i,0}+\ln\ell_{i,1}(\phi)\cdot R_{i,1}\right)$
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\left(\ln\ell_{i,0}(\phi)\cdot\ln\ell_{i,0}(\mu)+\ln\ell_{i,1}(\phi)\cdot\ln\ell_{i,1}(\mu)\right).$

Note that the appearance of $\mu$ in the agent’s loss function is by design.
Normally, without access to the oracle reward, $R_{i,b}$ would be replaced by
the output $R_{i,b}(\psi)$ of a learned reward model separately parameterized
by $\psi$.

A toy example helps demonstrate how this loss function leads to an
agglomerative agent, as opposed to the inclusive agent produced by ILHF in
Algorithm 1. The example is inspired by the didactic example discussed in
Section 2.2 of [8]. Consider a finetuning dataset consisting of the exact same
prompt $X_{1}=X$, $X_{2}=X$, …, $X_{N}=X$. Suppose each response contains a
single token in $\\{-1,1\\}$, with $-1$ preferred by two-thirds of the human
labelers and $1$ by one-third. The corresponding logits generating such human
feedback are $\\{-\ln\sqrt{2},\ln\sqrt{2}\\}$. While our sequential reward
agent only gets to see the 0-1 labels, the REINFORCE agent is endowed with the
logits that generate these labels. As shown in Table 1, ILHF is able to learn
the correct distribution of human labels, whereas REINFORCE over-concentrates
onto the preferred response, which is $-1$ in this case. Notably, this issue
is not mitigated by adding a KL-penalty to the policy gradient loss.

The agent models we use in this example is exactly the same as that explained
in Appendix D. In Table 1, we specify the parameters for this didactic example
in our experiments.

Table 1: Parameters for the didactic experiment Problem parameter | Value
---|---
prompt $X$ | $[1,\ldots,1]$
prompt length | 10
evaluation batch size | 3,000
Hyperparameter | Value
ensemble prior scale | 0.01
ensemble $L_{2}$-regularization scale $\eta$ | 1.0
agent hidden size $D$ | 2

##  Appendix C Token-Generating Process

Recall that our token-generating process is governed by a vector
$\mu\in\Re^{d}$, a matrix $W\in\Re^{d\times d}$, and a vector $U\in\Re^{d}$.
Given these parameters and a fixed initial state distribution, sequences are
generated in an autoregressive fashion, as defined in Section 4.1. We
instantiate this abstract formulation on specific parameter distributions, and
showcase some structural properties implied by this generation process.

The reason for using a recurrent neural network is so that there is a notion
of latent representation that could be useful for predicting the future
trajectory. We choose the underlying data-generating process to be relatively
simple, with a low-dimensional latent space. Specifically, we take $d=2$ in
all our subsequent simulations. The parameters are randomly drawn according to
the following distributions:

  * •

each entry of the weight matrix $W\in\Re^{2\times 2}$ is drawn i.i.d. from
$\mathcal{U}(-\sqrt{\frac{1}{2}},\sqrt{\frac{1}{2}})$,

  * •

each entry of the weight vector $U\in\Re^{1\times 2}$ is drawn i.i.d. from
$\mathcal{U}(-1,1)$,

  * •

the vector $\mu\in\Re^{2}$ is drawn from 2-dimensional standard Gaussian,

  * •

the perturbed vector $\theta\in\Re^{2}$ is drawn from
$\mathcal{N}(\mu,0.3\cdot I)$,

  * •

the initial hidden state $S_{0}$ is drawn from 2-dimensional standard
Gaussian.

Despite the simplicity of this construction, the resulting autoregressive
process exhibits nontrivial stationary behavior. For one, it does not
eventually produce all $-1$s or all $1$s or settle on a simple repeated
pattern. For another, each next token depends on a relatively long history of
tokens. To verify that the latter is indeed the case, we consider the
autocorrelation of a series of logits. Let $(a_{i})_{i=1}^{\tau}$ denote the
logits at the last layer of the model for $\tau$ outputs. For each
$j,k\in\\{1,\ldots,\tau\\}$ such that $j<k$, let

$\overline{a_{j:k}}=\frac{1}{k-j}\sum_{i=j}^{k}a_{i}$

denote the mean of the logits $a_{j},a_{j+1},\ldots,a_{k}$. For $k\geq 0$, the
autocorrelation function at lag $k$ is given by

$\displaystyle\alpha(k)$
$\displaystyle=\frac{1}{\tau-k}\sum_{i=1}^{\tau-k}\frac{(a_{i}-\overline{a_{1:\tau-k}})(a_{i+k}-\overline{a_{k+1:\tau}})}{\sqrt{\sum_{i=1}^{\tau-k}(a_{i}-\overline{a_{1:\tau-k}})^{2}\sum_{i=1}^{\tau-k}(a_{i+k}-\overline{a_{k+1:\tau}})^{2}}}.$

We plot the function $\alpha$ against lag in Figure 2.

##  Appendix D Agent Designs

###  D.1 Agents

In Section 4.1, we introduced the architecture for the token-generating
process in our experiments. Our agents are equipped with similar architectures
as the token generating process, but with high dimensional hidden states. To
distinguish learned from data generating models, recall that we denote the
parameters of the latter by $\theta\in\Re^{d}$, $W\in\Re^{d\times d}$, and
$U\in\Re^{d}$, whereas we denote the agent’s parameters by a vector
$\phi\in\Re^{D}$, a matrix $\widehat{W}\in\Re^{D\times D}$, and a vector
$\widehat{U}\in\Re^{D}$. As before, the process begins in an initial state
$\widehat{S}^{\prime}_{0}\in\Re^{D}$ sampled from a fixed distribution. Given
a prompt $X_{1:\tau}$, the agent generates an associated agent state sequence
$\widehat{S}^{\prime}_{1:\tau}$ according to
$\widehat{S}^{\prime}_{t+1}=\tanh(\widehat{W}\widehat{S}^{\prime}_{t}+\widehat{U}X_{t})$.
We let $\widehat{S}_{0}=\widehat{S}^{\prime}_{\tau}$. Then at any time $t$,
given the state $\widehat{S}_{t}\in\Re^{D}$ of this process, a next token
$\widehat{X}_{t+1}\in\\{-1,1\\}$ and state $\widehat{S}_{t+1}\in\Re^{D}$ are
generated according to

$\displaystyle\widehat{X}_{t+1}$
$\displaystyle=\left\\{\begin{array}[]{ll}1&\text{w.p.
}\frac{e^{\phi^{\top}\widehat{S}_{t}}}{1+e^{\phi^{\top}\widehat{S}_{t}}}\\\
-1&\text{otherwise.}\end{array}\right.$ $\displaystyle\widehat{S}_{t+1}$
$\displaystyle=\tanh(\widehat{W}\widehat{S}_{t}+\widehat{U}\widehat{X}_{t+1}).$

A more complex agent model than the true data generating process affords the
agent to a rich latent state representation that is likely to encode a close
approximation to the true latent state, which can be useful when fine-tuning
for various downstream tasks. The agent’s larger model grants it the ability
to fit to the data generating process almost perfectly without advanced
knowledge of the latent features driving it. This is akin to the manner in
which overparameterized neural networks are able to discover latent spaces.

All our agents follow the same pretraining procedure described in Section 4.1
that fits a model to the pretraining data to produce a point estimate, which
is then fine-tuned using the human feedback data. Thus, before fine-tuning,
our agents are endowed with knowledge learned in pretraining, represented by
$\widehat{W}$, $\widehat{U}$, and $\phi^{\text{pre}}$. During fine-tuning, an
agent progresses over rounds of human interaction. Each round begins with the
agent observing a prompt. The agent then produces two responses, which could
depend on the knowledge from pretraining as well as human feedback received
over previous rounds. Finally, the agent observes an indication of preference
provided by a random individual, and moves on to the next round. The problem
parameters and agent hyperparameters used throughout the experiments are
presented in Table 2.

Problem parameter | Value
---|---
prompt length $\tau$ | 64
softmax temperature | 3.0
evaluation batch size | 500
Hyperparameter | Value
ensemble prior scale | 0.01
ensemble $L_{2}$-regularization scale $\eta$ | 1.0
agent hidden size $D$ | 10
Table 2: Parameters for agents

##  Appendix E Algorithms

Algorithm 1 Inclusive Learning from Human Feedback (ILHF)

Inputs: | parameters | pretrained parameters $\widehat{W},\widehat{U},\phi^{\text{pre}}$
---|---|---
| data | prompt generator $\mathtt{PromptGen}(\tau)$, batch size $N$
| human labeler |  $\mathtt{Human(\cdot,\cdot)}$, number of interaction episodes $K$
| loss function |  $\mathcal{L}^{\text{ILHF}}(\phi;\mathcal{D})$ evaluates parameter $\phi$ on dataset $\mathcal{D}$
| optimization | update rule $\mathtt{Optimizer}$, minibatch loader $\mathtt{DataLoader}$
Returns: | $\phi_{K}$ | parameters for the trained language model

1:Initialize: initial agent parameters $\phi_{0}\leftarrow\phi^{\text{pre}}$

2:for epoch $k=0,\dots,K$ do

3: for $i=1,\ldots,N$ do

4: sample prompt $x_{i,1:\tau}\leftarrow\mathtt{PromptGen}(\tau)$

5: generate response pair $(x_{i,1:\tau}^{(0)},x_{i,1:\tau}^{(1)})$

6: observe human label
$b_{i}\leftarrow\mathtt{Human}(x_{i,1:\tau}^{(0)},x_{i,1:\tau}^{(1)})$

7: end for

8:
$\mathcal{D}^{k}\leftarrow(x_{i,1:\tau},x_{i,1:\tau}^{(0)},x_{i,1:\tau}^{(1)},b_{i})_{i=1}^{N}$

9: for minibatch $\mathcal{D}$ in $\mathtt{DataLoader}(\mathcal{D}^{k})$ do

10: compute
$\mathtt{gradient}\leftarrow\nabla_{\phi|\phi=\phi_{k}}\mathcal{L}^{\text{ILHF}}(\phi;\mathcal{D})$

11: update
$\phi_{k+1}\leftarrow\mathtt{Optimizer}(\phi_{k},\mathtt{gradient})$

12: end for

13:end for

Algorithm 2 ILHF with Ensemble Models (Ensemble-ILHF)

Inputs: | parameters | pretrained parameters $\widehat{W},\widehat{U},\phi^{\text{pre}}$
---|---|---
| data | prompt generator $\mathtt{PromptGen}(\tau)$, batch size $N$
| ensemble | ensemble size $M$, prior scale $\Sigma$, prior regularization scale $\eta$
| human labeler |  $\mathtt{Human(\cdot,\cdot)}$, number of interaction episodes $K$
| loss function |  $\mathcal{L}^{\text{ILHF}}(\phi;\mathcal{D})$ evaluates parameter $\phi$ on dataset $\mathcal{D}$
| optimization | update rule $\mathtt{Optimizer}$, minibatch loader $\mathtt{DataLoader}$
Returns: | $(\phi_{K,m})_{m=1}^{M}$ | ensemble parameters for the trained language model

1:Initialize: initial ensemble parameters
$\phi_{0,m}\leftarrow\mathcal{N}(\phi^{\text{pre}},\Sigma)$, $m=1,\ldots,M$

2:for epoch $k=0,\dots,K$ do

3: for $i=1,\ldots,N$ do

4: sample prompt $x_{i,1:\tau}\leftarrow\mathtt{PromptGen}(\tau)$

5: sample ensemble index $z_{i}\sim\text{Unif}\\{1,\ldots,M\\}$

6: generate response pair $(x_{i,1:\tau}^{(0)},x_{i,1:\tau}^{(1)})$ using
$\phi_{k,z_{i}}$

7: observe human label
$b_{i}\leftarrow\mathtt{Human}(x_{i,1:\tau}^{(0)},x_{i,1:\tau}^{(1)})$

8: end for

9:
$\mathcal{D}^{k}\leftarrow(x_{i,1:\tau},x_{i,1:\tau}^{(0)},x_{i,1:\tau}^{(1)},b_{i})_{i=1}^{N}$

10: for $m=1,\ldots,M$ do

11: for minibatch $\mathcal{D}$ in $\mathtt{DataLoader}(\mathcal{D}^{k})$ do

12: randomly perturb $\mathcal{D}$ (double or nothing)

13: compute
$\mathtt{gradient}\leftarrow\nabla_{\phi|\phi=\phi_{k,m}}\mathcal{L}^{\text{ILHF}}(\phi;\mathcal{D})+\eta\|\phi-\phi_{0}\|_{2}^{2}$

14: update
$\phi_{k+1,m}\leftarrow\mathtt{Optimizer}(\phi_{k,m},\mathtt{gradient})$

15: end for

16: end for

17:end for

##  Appendix F Ablations

In addition to the experiments in the main paper, we perform ablation studies
on problem parameters and agent hyperparameters. All error bars and confidence
intervals are 1 standard error over 20 seeds.

#### Ensemble size ablations.

We further vary the number of ensembles in Ensemble-ILHF and plot the KL-
divergence in Figure 7. As we increase the number of ensemble particles, the
convergence speed increases, but the marginal gain decreases. Table 3 shows
the inclusive score ratio, a metric explained in Section 5.2, of different
Ensemble-ILHF agents compared against a ILHF agent after 100 human
interactions.

Figure 7: ILHF with ensembles converges faster as the number of particles
increases.

Agent | Inclusive score ratio
---|---
Ensemble-ILHF 10 | $1.0384\pm 0.0061$
Ensemble-ILHF 15 | $1.0467\pm 0.0084$
Ensemble-ILHF 30 | $1.0338\pm 0.0062$
Ensemble-ILHF 50 | $1.0365\pm 0.0088$

Table 3: All Ensemble-ILHF agents beat ILHF.

#### Sequence length ablations.

We vary the sequence length from $\tau=64$ to $\tau\in\\{128,256\\}$ in our
experiments and observe similar qualitative behavior, as shown in Figures 8
and 9. Since increasing $\tau$ in general requires a longer horizon for the
agents to learn, we also increase the number of human interactions to $150$
for these ablations.

Figure 8: Comparisons of REINFORCE with various KL penalty scales, ILHF, and
Ensemble-ILHF with various ensemble sizes when $\tau=128$.
Figure 9: Comparisons of REINFORCE with various KL penalty scales, ILHF, and
Ensemble-ILHF with various ensemble sizes when $\tau=256$.